| *** hamzaachi has quit IRC | 00:01 | |
| *** tosky has quit IRC | 00:02 | |
| cmart | Hi OSA. I just upgraded my production cluster from Pike to Queens. I ran the playbook to remove the old Neutron agents containers, and when I did this (a few hours ago), networking for the entire cloud broke | 00:07 |
|---|---|---|
| cmart | The Neutron agents are running on bare metal of the control plane, the linuxbridge agent claims to be creating bridges etc, but I can't pass any traffic. | 00:08 |
| cmart | Has anyone encountered a similar issue, or have any ideas/suggestions? | 00:08 |
| cmart | following this now https://docs.openstack.org/openstack-ansible/queens/admin/troubleshooting.html#troubleshooting-instance-connectivity-issues | 00:09 |
| *** ansmith has joined #openstack-ansible | 00:09 | |
| *** sdake has quit IRC | 00:19 | |
| *** ansmith has quit IRC | 00:23 | |
| *** sdake has joined #openstack-ansible | 00:27 | |
| *** ansmith has joined #openstack-ansible | 00:46 | |
| *** macza has quit IRC | 00:53 | |
| *** nurdie has joined #openstack-ansible | 00:59 | |
| *** cmart has quit IRC | 01:03 | |
| *** nurdie has quit IRC | 01:03 | |
| *** hamzy has joined #openstack-ansible | 01:04 | |
| *** ansmith has quit IRC | 01:11 | |
| *** sdake has quit IRC | 01:19 | |
| *** sdake has joined #openstack-ansible | 01:22 | |
| *** markvoelker has joined #openstack-ansible | 01:36 | |
| jamesdenton | may need to run thru that scenario again and see if something is wonky | 01:40 |
| *** markvoelker has quit IRC | 01:41 | |
| *** cmart has joined #openstack-ansible | 01:56 | |
| *** tinwood has quit IRC | 02:09 | |
| *** tinwood has joined #openstack-ansible | 02:11 | |
| jamesdenton | cmart did you manage to get it resolved? | 02:23 |
| cmart | hi jamesdenton! not yet. :( | 02:23 |
| cmart | i'm hand-editing inventory to re-deploy Neutron agents in containers | 02:24 |
| cmart | kind of backpedaling at this point. 12 hours in | 02:24 |
| jamesdenton | are you seeing anything of interest in the linuxbridge agent log? | 02:24 |
| jamesdenton | on the infras? | 02:24 |
| cmart | nope, i've pored over those. not a hint | 02:25 |
| cmart | instances can talk to each other on private networks | 02:26 |
| cmart | so some kind of networking is working | 02:26 |
| jamesdenton | Alright - so can you describe in a little more detail whats happening? | 02:26 |
| jamesdenton | what kind of networks are you using? vxlan? vlan? are you using routers and floating ips? | 02:26 |
| cmart | we're using vxlan I believe. and yes, external routers and floating IPs | 02:27 |
| cmart | instances can talk to each other on private networks, but can't ping their default gateways on same private networks | 02:27 |
| cmart | so it's almost like something with the routers is busted | 02:27 |
| jamesdenton | ok gotcha. If you spin up a new instance does DHCP work? | 02:27 |
| cmart | checking! | 02:27 |
| jamesdenton | assuming DHCP runs on the infras (now baremetal) | 02:27 |
| cmart | (yes, I would assume that too) | 02:28 |
| *** TxGirlGe_ has quit IRC | 02:28 | |
| jamesdenton | when you get a chance, please show me the 'brctl show' output from just one of the compute nodes and one of the infras. You can use paste.openstack.org and then share the link | 02:28 |
| cmart | sure! give me just a minute | 02:28 |
| jamesdenton | no worries. I'm gonna ask questions but take your time. Did you already delete the old neutron agent containers? And have you rerun any playbooks with any modifications to your inventory? | 02:29 |
| jamesdenton | when you get a chance, also share the output of 'openstack network agent list' | 02:30 |
| cmart | `brctl-show` on an infra node: http://paste.openstack.org/show/744220/ | 02:35 |
| cmart | `brctl-show` on a compute node: http://paste.openstack.org/show/744221/ | 02:35 |
| cmart | output of `openstack network agent list`: http://paste.openstack.org/show/744222/ | 02:36 |
| jamesdenton | looking | 02:36 |
| cmart | answering other questions: I have already deleted the old neutron agent containers :( | 02:36 |
| jamesdenton | no worries | 02:36 |
| jamesdenton | safe to assume t01, t05, and t09 are infras? | 02:36 |
| cmart | I have NOT yet re-run playbooks with the most recent modifications to inventory (hand-adding the neutron agent containers back in) | 02:36 |
| jamesdenton | baby duty - brb | 02:37 |
| cmart | yessir | 02:37 |
| *** markvoelker has joined #openstack-ansible | 02:37 | |
| jamesdenton | on your infras are you seeing qdhcp and qrouter namespaces? | 02:38 |
| jamesdenton | i would assume so based on the # of interfaces i'm seeing | 02:39 |
| cmart | One slightly weird thing I did, mid-upgrade I switched from the 17.1.6 tag of OSA. to the stable/queens tag. The diff set is fairly small, but I did it to get odyssey4me's commit that supports the automated Neutron agent migration from containers to bare metal | 02:39 |
| cmart | sorry, I'm unsure how to look for these Q namespaces. googling | 02:40 |
| jamesdenton | 'ip netns' | 02:40 |
| jamesdenton | when you switched... how far along? | 02:40 |
| cmart | 'ip netns' returns about 20-some qdhcp namespaces, and a similar number of qrouter namespaces, roughly equivalent to the number of projects / private networks we have | 02:41 |
| jamesdenton | right, ok | 02:41 |
| jamesdenton | on the infra - does br-vxlan have an ip? | 02:42 |
| jamesdenton | and can you ping the ip of that compute? | 02:42 |
| cmart | I switched branches after running everything (including setup-openstack.yml completing without error). Since switching branches, I re-ran "${UPGRADE_PLAYBOOKS}/neutron-tmp-inventory.yml", "repo-install.yml", "bootstrap-ansible.sh" (to pull in some updated roles), then I re-ran both Neutron and Nova playbooks. | 02:43 |
| jamesdenton | if you can, show me 'ip addr show br-vxlan' and 'ip -d link show vxlan-62' on both of those machines | 02:43 |
| cmart | "on both of those machines" meaning the three infra hosts, or an infra and a compute host? | 02:44 |
| jamesdenton | just the two you sent me | 02:44 |
| cmart | output on infra node (tombstone01): http://paste.openstack.org/show/744223/ | 02:46 |
| cmart | output on compute node (tombstone22): http://paste.openstack.org/show/744224/ | 02:46 |
| jamesdenton | ok yeah, i see it | 02:47 |
| jamesdenton | one sec | 02:47 |
| jamesdenton | let me put the baby back to bed and we'll see what we can do | 02:47 |
| cmart | OK. thank you very much! a bunch of researchers in Arizona may wish to buy you a beverage if we can get them access to their servers again :) | 02:48 |
| jamesdenton | hah well let's get you fixed up first | 02:49 |
| jamesdenton | so, the short of it appears to be that the infra nodes are using a different vtep interface than the computes. | 02:49 |
| jamesdenton | compute: vxlan id 62 group 239.1.1.1 dev br-vxlan | 02:50 |
| jamesdenton | infra: vxlan id 62 group 239.1.1.1 dev br-mgmt | 02:50 |
| jamesdenton | i've seen this, just need to find out to remedy. brb | 02:50 |
| cmart | aha. different bridge, meaning different layer 2 fabric | 02:50 |
| jamesdenton | yeah the bridge itself is just a placeholder for the interface where the VTEP addr is applied | 02:51 |
| jamesdenton | so one if using the IP configured on br-mgmt while the other, br-vxlan (correct) | 02:51 |
| jamesdenton | this likely means that the infras are missing a block in the inventory relating to br-vxlan or the tunnel_address | 02:51 |
| jamesdenton | just need to find my notes or bug on this.. | 02:52 |
| cmart | I didn't make any changes to my `provider_networks` in openstack_user_config.yml before deploying Queens | 02:53 |
| jamesdenton | yeah i don't think it's anything you did or didn't do, necessarily | 02:54 |
| jamesdenton | did br-vxlan always have an IP on it, or was it something you added recently? | 02:54 |
| jamesdenton | on the infra nodes that is | 02:54 |
| cmart | it's not something that I knowingly added | 02:54 |
| jamesdenton | kk | 02:54 |
| cmart | looking at our other production cloud (which is still on Pike), br-vxlan does have an IP on all infra nodes. | 02:56 |
| jamesdenton | ok, can you do me a favor and post the contents of /opt/openstack-ansible/playbooks/common-tasks/dynamic-address-fact.yml | 02:58 |
| cmart | here 'tis http://paste.openstack.org/show/744225/ | 02:58 |
| jamesdenton | Oh right, you're on Queens now. n/m on that | 03:00 |
| cmart | would a diff of openstack_inventory.json between now and the backup from Pike be helpful? | 03:00 |
| jamesdenton | um, actually, can you post the inventory sections for the two hosts you're working with? | 03:00 |
| *** sdake has quit IRC | 03:00 | |
| cmart | sure. tombstone01 (infra): http://paste.openstack.org/show/744226/ | 03:02 |
| cmart | tombstone22 (compute): http://paste.openstack.org/show/744227/ | 03:02 |
| jamesdenton | Also... in either the linuxbridge_agent.ini or ml2_conf.ini you'll find a var called 'local_ip'. Can you share that for each host? | 03:02 |
| cmart | that's 172.29.224.201 on the infra host and 172.29.228.222 on the compute host | 03:04 |
| *** jpward1981 has quit IRC | 03:04 | |
| jamesdenton | k | 03:06 |
| *** markvoelker has quit IRC | 03:08 | |
| jamesdenton | On your deploy host, there is a directory at /etc/openstack_deploy/ansible_facts. Can you open up the file for tombstone22 and share the entire block for 'ansible_br_vxlan'? | 03:10 |
| jamesdenton | and the same thing for tombstone01. | 03:11 |
| jamesdenton | The IP address that populates 'local_ip' in those config files is used as the VTEP. And that address is determined dynamically during the neutron playbook run based on the ansible facts for a given host. If the IP doesn't exist (in facts) then it defaults to ansible_host, which is the br-mgmt IP we saw | 03:12 |
| cmart | ansible_br_vxlan for tombstone22: http://paste.openstack.org/show/744228/ | 03:12 |
| jamesdenton | i'm trying to figure out a) if the info doesn't exist in facts, and b) how to get it recognized so we can fix it properly | 03:12 |
| cmart | ansible_br_vxlan for tombstone01: http://paste.openstack.org/show/744229/ | 03:13 |
| jamesdenton | OK, so it looks like the facts contains the correct IP for tombstone01 172.29.228.201 | 03:14 |
| jamesdenton | so what you might consider is to run the os-neutron-install.yml playbook only against tombstone01. "openstack-ansible os-neutron-install.yml --limit localhost,tombstone01" | 03:15 |
| cmart | yep. but we saw 172.29.224.201 for the `local_ip` value, which is on the management subnet rather than the tunnel subnet | 03:15 |
| jamesdenton | right | 03:15 |
| jamesdenton | two things need to happen - we need to make sure the config file gets the right IP and then we will need to blow away the vxlan interfaces and restart linuxbridge/dhcp/l3 agents on the node | 03:16 |
| jamesdenton | so those interfaces can be rebuilt | 03:16 |
| jamesdenton | but the key is making sure that the config file gets the right IP. | 03:16 |
| jamesdenton | not sure why that didn't happen here | 03:17 |
| cmart | sure. happy to do that. can't hurt anything. i'm reverting my most recent changes to openstack_inventory.json to bring it back to what OSA set up during the queens upgrade. | 03:17 |
| jamesdenton | ok yes | 03:17 |
| cmart | here goes, just ran playbook | 03:19 |
| jamesdenton | that was fast | 03:19 |
| cmart | i mean, it's running :) just started it | 03:20 |
| jamesdenton | ahh ok | 03:20 |
| jamesdenton | :) | 03:20 |
| jamesdenton | when it's done, please take a peek at those neutron config files, whichever had local_ip, and see if it changed to 172.29.228.201 | 03:20 |
| cmart | yeah linuxbridge_agent.ini now shows `local_ip = 172.29.228.201` | 03:21 |
| jamesdenton | fantastic | 03:21 |
| jamesdenton | let me know when it's done | 03:22 |
| cmart | playbook is finished | 03:22 |
| jamesdenton | ok cool | 03:22 |
| cmart | i'm taking a peek at the other infra hosts to see if they have the same issue... | 03:22 |
| jamesdenton | they likely do | 03:22 |
| cmart | yeah they do | 03:23 |
| cmart | so running playbook against those as well... | 03:23 |
| jamesdenton | you can run it against them, too. cool. | 03:23 |
| jamesdenton | make sure to include localhost | 03:23 |
| *** jpward1981 has joined #openstack-ansible | 03:24 | |
| cmart | sure enough, it changed local_ip for the other infra hosts too. | 03:27 |
| cmart | playbook finished. now, question is do we still need to kick those services... | 03:27 |
| jamesdenton | ok cool. so starting with tombstone01, we need to isolate the vxlan-* interfaces so they can be deleted and then rebuilt. One way to do that is with something like this: | 03:28 |
| jamesdenton | brctl show | grep vxlan | grep -v br-vxlan | awk {'print $4'} | wc -l | 03:29 |
| jamesdenton | and then maybe: "for i in $(brctl show | grep vxlan | grep -v br-vxlan | awk {'print $4'}); do ip link delete $i; done" | 03:29 |
| cmart | the first of your commands returns 47 | 03:29 |
| jamesdenton | and then "systemctl restart neutron-linuxbridge-agent neutron-dhcp-agent neutron-l3-agent" | 03:29 |
| jamesdenton | k | 03:29 |
| cmart | running these in pieces so I understand what they do | 03:30 |
| jamesdenton | can you run it without the wc -l and make sure it only contains vxlan-* | 03:30 |
| jamesdenton | so the idea is to delete all of those vxlan interfaces, since the vtep address is wrong. when you restart those agents )likely only linuxbridge-agent, they will be rebuilt with the proper vtep addr | 03:30 |
| jamesdenton | how you do the delete is up to you - no warranty with my commands | 03:31 |
| cmart | understood :) your first command works up until the awk | 03:31 |
| cmart | i'll just munge it into `ip link delete` followed by the vxlan-## | 03:32 |
| jamesdenton | right on | 03:32 |
| cmart | ok everything is now gone except for br-vxlan which we apparently want to keep | 03:32 |
| cmart | now restarting the agent(s) | 03:33 |
| jamesdenton | ok great | 03:33 |
| jamesdenton | if you -- watch 'brctl show | grep vxlan' -- you should see then come back | 03:34 |
| jamesdenton | ideally we end up with 47 | 03:34 |
| cmart | they're repopulating! | 03:34 |
| jamesdenton | and then 'ip -d link show vxlan-62' should show 'br-vxlan' instead of 'br-mgmt' | 03:34 |
| cmart | it does indeed | 03:35 |
| jamesdenton | good deal | 03:35 |
| cmart | holy crap some of my servers are now back online | 03:35 |
| jamesdenton | you should be good to repeat that on the other hosts, just be sure to rebuild those ip link delete commands for each host, as they could be a little different | 03:36 |
| cmart | right, because different vxlan numbers | 03:36 |
| jamesdenton | well, same vxlan numbers, but sometimes they don't exist everywhere | 03:36 |
| cmart | ok, rinse & repeating | 03:36 |
| jamesdenton | vxlan-62 represents the same 'network' across this cloud no matter what | 03:37 |
| jamesdenton | the 62 is arbitrary, really. | 03:37 |
| cmart | right, and there's a different one for each network | 03:37 |
| jamesdenton | yep | 03:37 |
| jamesdenton | it is connected to a brq bridge that is named consistently across all hosts that have an object in that network, be it a dhcp server, router, vm, whatever | 03:38 |
| jamesdenton | that brq bridge name, the 9 chars, should be the first 9 chars of the UUID of the respective network | 03:38 |
| jamesdenton | just FYI | 03:38 |
| jamesdenton | and the connected tap interfaces, the chars line up with the port UUID for whatever its connected to | 03:38 |
| jamesdenton | how are things looking? | 03:43 |
| cmart | really good! just finished rebuilding the vxlan links on the last infra host. everything appears to be back online | 03:44 |
| cmart | I feel humbled and grateful, kind stranger | 03:44 |
| jamesdenton | very glad to hear that | 03:45 |
| jamesdenton | i know how those 12 hr upgrades go, believe me | 03:45 |
| cmart | yeah. my pencil is now very dull | 03:46 |
| jamesdenton | when you're feeling up for it, can you fill out a bug describing your upgrade process, including the change in the middle, and if you have them, the playbook commands you ran? | 03:46 |
| jamesdenton | seems to me like it was using stale facts and applying the wrong IP for tunnel_address, either because it didn't have one in facts or it did and ignored it or something | 03:47 |
| cmart | yes, I should have all that. do you suspect the change in the middle may have caused the misconfigured "local_ip"? | 03:47 |
| jamesdenton | i don't think so | 03:47 |
| jamesdenton | i have heard of folks (maybe even today) having a similar issues because br-vxlan did not have an address on it, and when they upgrading, things didn't work. In the old container, the VTEP addr was configured on eth10. | 03:49 |
| jamesdenton | And the playbooks will happily default to what is effectively br-mgmt IP, and you won't be able to communicate between the infras and the computes | 03:49 |
| jamesdenton | and even if you fix local_ip after the fact, things still won't work until you delete the vxlan-* interfaces and rebuild them. Or restart the box | 03:50 |
| jamesdenton | So, if there's anything productive to come out of this exercise maybe it's some additional troubleshooting steps to at least identify or rule out that particular issue | 03:50 |
| jamesdenton | sorry you had to go thru that | 03:50 |
| cmart | right. I did clear all my Ansible facts at the start of the upgrade, but not after switching branches | 03:51 |
| jamesdenton | k | 03:51 |
| jamesdenton | who knows :/ | 03:52 |
| cmart | and no apology needed :) if this is the price we occasionally pay for what is otherwise very solid infrastructure for us, then I will take that deal | 03:54 |
| cmart | but clearly I need to learn more about vxlan and friends | 03:54 |
| cmart | if you tell me the second edition of your book is still reasonably current and applicable, then you just sold at least one book, my friend | 03:55 |
| jamesdenton | Well, it's just one of many little observations I've made over time in these kinds of cases. | 03:55 |
| jamesdenton | well, it isn't really, to be honest. There is a third edition that's pretty relevant (Pike/Queens, IIRC) but isn't OSA-based. Packt runs $5 eBook specials all the time, i would wait :D | 03:57 |
| cmart | heh okay | 03:57 |
| cmart | but yes, ruling out the afore-troubleshot issue seems like one for this doc https://docs.openstack.org/openstack-ansible/queens/admin/troubleshooting.html#troubleshooting-instance-connectivity-issues | 03:58 |
| jamesdenton | nice ascii art. lol | 03:59 |
| jamesdenton | but yes, would be a good spot indeed | 03:59 |
| jamesdenton | looking forward to your bug report. thanks for hanging in there. and don't be a stranger here | 03:59 |
| jamesdenton | get some rest. i'm sure you need it | 03:59 |
| cmart | yes. I'll do the writeup after a big food and a long sleep | 04:00 |
| cmart | Thank you very much! | 04:00 |
| jamesdenton | perfect | 04:00 |
| jamesdenton | you're very welcome. glad we figured it out | 04:00 |
| * jamesdenton wipes brow | 04:00 | |
| *** nurdie has joined #openstack-ansible | 04:36 | |
| *** chkumar|out is now known as chandankumar | 04:38 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Update cirros from 3.5 to 3.6 https://review.openstack.org/633208 | 04:41 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added dependencies of os_tempest role https://review.openstack.org/632726 | 04:42 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Disable nova-lxd tempest plugin https://review.openstack.org/633711 | 04:43 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Use the correct heat tests https://review.openstack.org/630695 | 04:50 |
| *** aedc has quit IRC | 05:01 | |
| *** ArchiFleKs has quit IRC | 05:18 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Always generate stackviz irrespective of tests pass or fail https://review.openstack.org/631967 | 05:25 |
| *** ArchiFleKs has joined #openstack-ansible | 05:28 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Use tempest_cloud_name in tempestconf https://review.openstack.org/631708 | 05:32 |
| *** ArchiFleKs has quit IRC | 05:34 | |
| *** udesale has joined #openstack-ansible | 05:34 | |
| *** ArchiFleKs has joined #openstack-ansible | 05:35 | |
| *** devx has quit IRC | 05:44 | |
| *** udesale has quit IRC | 05:46 | |
| *** udesale has joined #openstack-ansible | 05:48 | |
| chandankumar | jrosser: morning | 05:52 |
| chandankumar | jrosser: https://review.openstack.org/#/c/632726/ is failing http://logs.openstack.org/26/632726/6/check/openstack-ansible-functional-centos-7/1fdf846/job-output.txt.gz#_2019-01-30_05_28_44_981018 | 05:52 |
| *** udesale has quit IRC | 05:53 | |
| *** udesale has joined #openstack-ansible | 05:54 | |
| chandankumar | jrosser: {"changed": false, "msg": "The variable venv_install_destination_path is required and\nhas not been set.\n" | 05:55 |
| *** cmart has quit IRC | 06:01 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_nova master: Use venv_packages_to_symlink to symlink to import libvirt-python https://review.openstack.org/633474 | 06:01 |
| *** cmart has joined #openstack-ansible | 06:02 | |
| *** cmart has quit IRC | 06:08 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Update cirros from 3.5 to 3.6 https://review.openstack.org/633208 | 06:28 |
| *** hamzaachi_ has quit IRC | 06:29 | |
| *** nurdie has quit IRC | 06:35 | |
| *** nurdie has joined #openstack-ansible | 06:35 | |
| *** nurdie has quit IRC | 06:40 | |
| *** PTO has quit IRC | 06:53 | |
| *** udesale has quit IRC | 07:02 | |
| *** udesale has joined #openstack-ansible | 07:05 | |
| *** udesale has quit IRC | 07:06 | |
| *** udesale has joined #openstack-ansible | 07:08 | |
| *** udesale has quit IRC | 07:09 | |
| *** udesale has joined #openstack-ansible | 07:09 | |
| chandankumar | jrosser: odyssey4me https://review.openstack.org/#/c/631967/ and https://review.openstack.org/#/c/631708/ are good to go, needed for final tripleo ci job | 07:10 |
| *** radeks_ has joined #openstack-ansible | 07:11 | |
| *** jawad_axd has joined #openstack-ansible | 07:13 | |
| *** radeks_ has quit IRC | 07:17 | |
| *** radeks_ has joined #openstack-ansible | 07:23 | |
| *** kopecmartin|off is now known as kopecmartin | 07:26 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-tests master: [WIP] Add delorean-deps.repo in OSA https://review.openstack.org/633882 | 07:35 |
| chandankumar | jrosser: I think we need to fix vxlan issue on os_nova side also | 07:37 |
| chandankumar | jrosser: Same scneario tests are failing there | 07:37 |
| jrosser | chandankumar: yes you are right | 07:38 |
| chandankumar | jrosser: let me propose a patch based on yours | 07:38 |
| jrosser | This appears to be happening after the host network was changed to use systemd_networkd. Most likely due to systemd version on centos being different/older than the other distro | 07:39 |
| chandankumar | jrosser: may be using delorean-deps.repo will the issue | 07:40 |
| *** fnpanic has joined #openstack-ansible | 07:40 | |
| jrosser | Ideally we understand what it is there that is broken, and then there is a fix possible in one place, rather than in all the roles individually | 07:40 |
| chandankumar | jrosser: do we collect list of installed rpms in each hosts? | 07:40 |
| jrosser | I think so yes | 07:40 |
| chandankumar | jrosser: yes we collect | 07:40 |
| chandankumar | http://logs.openstack.org/74/633474/4/check/openstack-ansible-functional-centos-7/4335862/logs/redhat-rpm-list-installed-host.txt.gz | 07:41 |
| chandankumar | systemd-networkd.x86_64 219-62.el7_6.2 @updates | 07:41 |
| chandankumar | jrosser: may be cloudnull can help here to fix it at one place | 07:43 |
| *** jawad_axd has quit IRC | 07:43 | |
| prometheanfire | jrosser: you don't sleep either? | 07:43 |
| jrosser | It may appear that way, I guess :) | 07:43 |
| jrosser | chandankumar: yes it would be good for cloudnull to take a peek now we have ide tidied exactly what is broken | 07:44 |
| jrosser | chandankumar: and this is all about to turn into a big bonfire on centos anyway sadly https://bugzilla.redhat.com/show_bug.cgi?id=1650342 | 07:44 |
| openstack | bugzilla.redhat.com bug 1650342 in systemd "systemd-networkd support in RHEL 8" [Unspecified,New] - Assigned to systemd-maint | 07:44 |
| prometheanfire | link to failure? | 07:45 |
| prometheanfire | and ya, redhat pissed me off because of that | 07:46 |
| prometheanfire | push it on everyone and then re-invent | 07:46 |
| jrosser | prometheanfire: this is a workaround, hopefully the commit msg explains https://review.openstack.org/#/c/633732/ | 07:47 |
| chandankumar | prometheanfire: it is related to tempest scenario tests failure on os_nova side http://logs.openstack.org/74/633474/4/check/openstack-ansible-functional-centos-7/4335862/job-output.txt.gz#_2019-01-30_07_29_46_086083 | 07:47 |
| prometheanfire | ah | 07:48 |
| *** gkadam has joined #openstack-ansible | 07:51 | |
| *** slaweq has joined #openstack-ansible | 07:58 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Ping router once it is created https://review.openstack.org/633883 | 08:02 |
| chandankumar | jrosser: ^^ regarding pinging router not sure too much perfect | 08:03 |
| *** mkuf has quit IRC | 08:10 | |
| *** pcaruana has joined #openstack-ansible | 08:10 | |
| *** markvoelker has joined #openstack-ansible | 08:16 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added tempest.conf for heat_plugin https://review.openstack.org/632021 | 08:24 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Use the correct heat tests https://review.openstack.org/630695 | 08:25 |
| *** aedc has joined #openstack-ansible | 08:32 | |
| *** rgogunskiy has joined #openstack-ansible | 08:32 | |
| *** mkuf has joined #openstack-ansible | 08:33 | |
| *** gkadam has quit IRC | 08:42 | |
| *** markvoelker has quit IRC | 08:49 | |
| chandankumar | jrosser: how to kick this environment locally? | 08:51 |
| chandankumar | jrosser: http://git.openstack.org/cgit/openstack/openstack-ansible-os_heat/tree/zuul.d/jobs.yaml#n21 | 08:51 |
| *** electrofelix has joined #openstack-ansible | 08:51 | |
| chandankumar | jrosser: scenario: aio_metal_heat | 08:51 |
| *** tosky has joined #openstack-ansible | 08:55 | |
| jrosser | chandankumar: here are the AIO instructions https://docs.openstack.org/openstack-ansible/rocky/user/aio/quickstart.html | 09:03 |
| jrosser | look at the bit where the SCENARIO environment variable is set | 09:03 |
| fnpanic | hi | 09:04 |
| fnpanic | i am currently trying to find where lxc_net_gateway is used in rocky? | 09:06 |
| fnpanic | the docs say how to set it but i cannot find where this is used... | 09:06 |
| fnpanic | https://docs.openstack.org/openstack-ansible-lxc_hosts/rocky/ | 09:06 |
| jrosser | fnpanic: two ways to find where it is used, first in that role https://github.com/openstack/openstack-ansible-lxc_hosts/search?q=lxc_net_gateway&unscoped_q=lxc_net_gateway | 09:10 |
| jrosser | and second in all the repos http://codesearch.openstack.org/?q=lxc_net_gateway&i=nope&files=&repos= | 09:10 |
| jrosser | which in this case yield pretty similar results | 09:10 |
| *** DanyC has joined #openstack-ansible | 09:16 | |
| *** DanyC has quit IRC | 09:24 | |
| *** DanyC has joined #openstack-ansible | 09:25 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Always generate stackviz irrespective of tests pass or fail https://review.openstack.org/631967 | 09:33 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Always generate stackviz irrespective of tests pass or fail https://review.openstack.org/631967 | 09:33 |
| chandankumar | odyssey4me: jrosser https://review.openstack.org/#/c/631708/ cloudname fix to merge tripleo ci patch thanks ! | 09:35 |
| *** markvoelker has joined #openstack-ansible | 09:47 | |
| *** shyamb has joined #openstack-ansible | 09:58 | |
| *** DanyC has quit IRC | 09:58 | |
| *** DanyC has joined #openstack-ansible | 10:05 | |
| odyssey4me | jamesdenton you are magic, sir - thanks for helping cmart out so quickly and precisely... I, too, a interested to figure out what went wrong there | 10:06 |
| chandankumar | jrosser: https://review.openstack.org/#/c/633711/ this one also | 10:08 |
| jrosser | odyssey4me: ++ on that, we should make sure that the steps are captured because its a great example of systematic debugging and fixing up | 10:09 |
| *** DanyC has quit IRC | 10:11 | |
| *** DanyC has joined #openstack-ansible | 10:12 | |
| *** shyamb has quit IRC | 10:13 | |
| *** shyamb has joined #openstack-ansible | 10:13 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-tests master: Add delorean-deps.repo in OSA https://review.openstack.org/633882 | 10:18 |
| *** markvoelker has quit IRC | 10:19 | |
| odyssey4me | chandankumar are you aware of where that var gets used, because just that alone will do nothing at all | 10:20 |
| chandankumar | odyssey4me: one more change coming | 10:20 |
| chandankumar | in openstack_hosts | 10:20 |
| odyssey4me | oh, what do you know - there's already a var for deps: https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/defaults/main.yml#L142 | 10:21 |
| chandankumar | odyssey4me: https://trunk.rdoproject.org/centos7-master/delorean-deps.repo | 10:21 |
| *** DanyC has quit IRC | 10:22 | |
| odyssey4me | hmm, it looks like we've been using the deps repo the whole time | 10:22 |
| *** DanyC has joined #openstack-ansible | 10:23 | |
| chandankumar | odyssey4me: http://logs.openstack.org/83/633883/1/check/openstack-ansible-functional-centos-7/b052465/logs/ara-report/result/12e6100a-c09a-42ac-bbc7-67a1f679a436/ | 10:25 |
| *** DanyC has quit IRC | 10:25 | |
| chandankumar | odyssey4me: deps is there "baseurl": "https://trunk.rdoproject.org/centos7-master/deps/latest/", | 10:25 |
| *** DanyC has joined #openstack-ansible | 10:26 | |
| odyssey4me | yep, so what exactly are we changing then? | 10:26 |
| chandankumar | odyssey4me: from code search, I found only delorean.repo | 10:26 |
| chandankumar | so that we can enable delorean-deps.repo (which is missibg) | 10:26 |
| chandankumar | *missing | 10:27 |
| odyssey4me | oh ok | 10:30 |
| chandankumar | odyssey4me: does downloading repo file directly in /etc/yum.repos.d does not uses the repo? why we have to enable/disable each repo using ansible? | 10:30 |
| chandankumar | I mean in CI | 10:31 |
| odyssey4me | chandankumar I honestly have no idea. | 10:31 |
| odyssey4me | As far as I recall, we had issues with repositories built in the the images at some point, and wanted to change them to be the right ones. | 10:32 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Ping router once it is created https://review.openstack.org/633883 | 10:34 |
| *** priteau has joined #openstack-ansible | 10:46 | |
| chandankumar | odyssey4me: enabling and disabling stackviz is a nice idea for a normal user | 10:54 |
| odyssey4me | chandankumar yeah, it should likely be disabled by default so it's opt-in - then enabled for CI? | 10:56 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Use tempest_cloud_name in tempestconf https://review.openstack.org/631708 | 11:09 |
| *** mkuf has quit IRC | 11:14 | |
| *** shyamb has quit IRC | 11:15 | |
| *** markvoelker has joined #openstack-ansible | 11:16 | |
| fnpanic | jrosser: thanks! works great! | 11:18 |
| fnpanic | btw will the proxy patches make it inot 18.1.2? | 11:19 |
| *** udesale has quit IRC | 11:26 | |
| chandankumar | odyssey4me: yes, I will propose a patch for the stackviz enable | 11:38 |
| chandankumar | odyssey4me: http://logs.openstack.org/26/632726/6/check/openstack-ansible-functional-centos-7/1fdf846/logs/ara-report/result/10aaa101-aab4-40c4-9fcc-6a554fcb1790/ | 11:38 |
| chandankumar | odyssey4me: I am not sure what is wrong with this patch https://review.openstack.org/#/c/632726/ | 11:38 |
| odyssey4me | fnpanic I think they're all in, yes. | 11:40 |
| fnpanic | so as soon as 18.1.2 is released i should be fine right? | 11:41 |
| jamesdenton | odyssey4me i'm hoping there's something in the bug report that can shed light on what happened. i'll be out today, as the kids are all home from school due to the cold, cold temps | 11:41 |
| odyssey4me | chandankumar adding it as a dependency, makes python_venv_build run prior to the os_tempest role - but the metal dep does not have the included vars, so it fails | 11:41 |
| odyssey4me | chandankumar the role dep will work for config_template, I think (is that right evrardjp?) - but not for python_venv_build | 11:42 |
| odyssey4me | jamesdenton oh dear - hope they mend well, enjoy making the chicken soup :) | 11:43 |
| chandankumar | odyssey4me: let me try the reverseoricess | 11:43 |
| jamesdenton | :) | 11:43 |
| evrardjp | adding dependencies on projects that have a tasks/main.yml is generally a bad idea in the long run (if conditionals are added for example) | 11:44 |
| evrardjp | adding dependencies to roles with module_utils , modules or various plugins make sense to me | 11:44 |
| evrardjp | if that was the question | 11:45 |
| fnpanic | btw when i want to use the dynamic inventory to do a matrix ping check for example, how can i use it? the json file is not accepted by ansible. The docs are not clear to me.. :-( | 11:45 |
| odyssey4me | evrardjp yes, I was asking if adding config_template as a meta-dep will work as-is, or whether anything else is needed | 11:45 |
| odyssey4me | fnpanic just do: cd /opt/openstack-ansible; ansible -m ping all | 11:46 |
| odyssey4me | fnpanic as long as you're in the subdirectory /opt/openstack-ansible, the wrapper will include the inventory | 11:46 |
| fnpanic | i wanted to use fping for a matrix like ping | 11:46 |
| odyssey4me | fnpanic otherwise you can also do: ansible -i /opt/openstack-ansible/inventory -m ping all | 11:47 |
| fnpanic | not the ansible ping ;-) | 11:47 |
| fnpanic | ah ok... | 11:47 |
| odyssey4me | fnpanic well, I was illustrating how to use the inventory, not how to write the whole task :p | 11:47 |
| * jrosser has a jenkins job doing matrix ping | 11:48 | |
| *** mkuf has joined #openstack-ansible | 11:48 | |
| jrosser | for bonus points you should also do the inverse, check that the things that shouldnt be able to talk to each other cant | 11:48 |
| fnpanic | - /opt/openstack-ansible/inventory does not exist :-) | 11:48 |
| evrardjp | odyssey4me: nothing is required as far as I know, because I am using it outside OSA right now | 11:48 |
| evrardjp | :) | 11:49 |
| odyssey4me | fnpanic which series are you using? | 11:49 |
| fnpanic | jrosser: yeah, that is why i am disabling nat now in the lxcbridge | 11:49 |
| fnpanic | the one i tried is pike | 11:49 |
| odyssey4me | fnpanic assuming you cloned openstack-ansible to /opt, then https://github.com/openstack/openstack-ansible/tree/stable/rocky/inventory should be there | 11:49 |
| fnpanic | i will switch to rocky system | 11:50 |
| *** markvoelker has quit IRC | 11:50 | |
| odyssey4me | fnpanic the proxy fixes only went back to rocky IIRC | 11:50 |
| fnpanic | odyssey4me: the inventory is in rocky | 11:50 |
| fnpanic | odyssey4me: the pike one was just a system running here which i was logged into | 11:51 |
| fnpanic | will reinstall it with rocky | 11:51 |
| fnpanic | so the fixes will be in 18.1.2 when it is released, r8? | 11:51 |
| odyssey4me | fnpanic for pike, the path is /opt/openstack-ansible/playbooks/inventory | 11:52 |
| fnpanic | yeah, works for both now. Thanks! | 11:53 |
| odyssey4me | fnpanic fnpanic 18.1.3 is the next scheduled release - and yes, the fixes will be in it | 11:53 |
| fnpanic | ok | 11:53 |
| odyssey4me | 18.1.2 is out already, but it didn't have one of them IIRC | 11:53 |
| fnpanic | ok, so i will cherrypick the fixes for 18.1.2. | 11:54 |
| fnpanic | any eta for 1.3? | 11:54 |
| odyssey4me | fnpanic no need - just use stable/rocky | 11:54 |
| fnpanic | ok | 11:54 |
| odyssey4me | or if you want a fixed point for docs or something, use the SHA at the current head of stable/rocky | 11:54 |
| odyssey4me | ie, git checkout a603874cc54e62ccb3ac290443b59f242c5df84c | 11:55 |
| fnpanic | thanks | 11:55 |
| *** sdake has joined #openstack-ansible | 12:02 | |
| *** shyamb has joined #openstack-ansible | 12:07 | |
| *** gkadam has joined #openstack-ansible | 12:08 | |
| jrosser | fnpanic: if you are disabling nat on the container eth0 interface you may as well just remove that interface entirely? | 12:11 |
| jrosser | fnpanic: see what is done here https://review.openstack.org/#/c/625523/9/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2 | 12:11 |
| *** ansmith has joined #openstack-ansible | 12:17 | |
| fnpanic | jrosser: thanks! where is this set in the docs? | 12:18 |
| *** gkadam is now known as gkadam-bmgr | 12:19 | |
| jrosser | fnpanic: it's not in the docs per-se, this really an example of where you are able to override any of the role defaults you like in your user_variables, to customise the deployment as you like | 12:20 |
| fnpanic | ok, got it. | 12:21 |
| fnpanic | so then there is no default GW in the containers i guess.What about the dnsmasq part? | 12:21 |
| fnpanic | because what if the containers need to talk to ldap for example | 12:22 |
| jrosser | it depends where the ldap server is and if the containers have a route to it via the mgmt network | 12:22 |
| jrosser | thats still there on eth1 | 12:22 |
| jrosser | but really again you get to choose how you architect that | 12:23 |
| fnpanic | then i need to put in route on the mgmt network. i will use the syntax from the routed deployment example | 12:24 |
| *** shyamb has quit IRC | 12:24 | |
| *** shyamb has joined #openstack-ansible | 12:25 | |
| jrosser | fnpanic: it's worth studying the defaults/main.yml for the roles, they really are the documentation | 12:26 |
| jrosser | you can do this for example https://github.com/openstack/openstack-ansible-lxc_container_create/blob/33268989de7d271a28386c9f57fb6c295baad0b0/releasenotes/notes/container-extra-networks-c74119ba6a559a59.yaml | 12:26 |
| *** Pbing has joined #openstack-ansible | 12:28 | |
| *** apevec has joined #openstack-ansible | 12:28 | |
| Pbing | When I am trying to create a instance on compute node. i am facing below error | 12:29 |
| Pbing | ResourceProviderCreationFailed: Failed to create resource provider | 12:30 |
| Pbing | i am using devstack | 12:30 |
| *** shyamb has quit IRC | 12:30 | |
| fnpanic | i was thinking of doing this with provider_networks: in openstack_user_config and adding this to the keystone containers for example | 12:31 |
| CeeMac | afternoon all | 12:33 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Add tripleo-ci-centos-7-standalone-os-tempest job https://review.openstack.org/633931 | 12:35 |
| CeeMac | is anyone able to help with a PortBindingFailed issue when spinning up an instance? | 12:37 |
| jrosser | fnpanic: there a lots of different ways depending on what you want - provider_networks on o_u_c is fine if you are happy with the dynamic inventory assigning the addresses | 12:37 |
| jrosser | container_extra_networks is a bit more low level than that, you can specify exact IP you want per container, if that is what you need. Pick the one that suits I guess | 12:38 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added dependency of os_tempest role https://review.openstack.org/632726 | 12:40 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Add tripleo-ci-centos-7-standalone-os-tempest job https://review.openstack.org/633931 | 12:42 |
| chandankumar | odyssey4me: hello | 12:42 |
| *** sdake has quit IRC | 12:43 | |
| chandankumar | odyssey4me: I need to reuse this var file https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/vars/tempest_skip_master.yml in the existing tripleo os_tempest job so that if we start moving other scenario job it does not explode | 12:43 |
| chandankumar | odyssey4me: will I directly call the var file there? | 12:43 |
| chandankumar | jrosser: need some help here on router ping stuff http://logs.openstack.org/83/633883/2/check/openstack-ansible-functional-centos-7/75ddf44/logs/ara-report/result/e79dbc75-2c5d-46c7-8263-77c1b9934256/ | 12:45 |
| *** sdake has joined #openstack-ansible | 12:45 | |
| chandankumar | jrosser: https://review.openstack.org/633883 | 12:45 |
| jrosser | oh wow | 12:45 |
| odyssey4me | chandankumar where's the test playbook you want to apply that vars file to the os_tempest role? | 12:45 |
| jrosser | you need to pick the right fields out of the registered variable | 12:46 |
| chandankumar | odyssey4me: https://github.com/openstack/tripleo-quickstart-extras/blob/master/playbooks/multinode-standalone.yml#L48 | 12:46 |
| chandankumar | odyssey4me: I might need to copy that var file in the playbook directory | 12:47 |
| *** markvoelker has joined #openstack-ansible | 12:47 | |
| odyssey4me | chandankumar nope, not necessary - I'll work up a gist for you | 12:47 |
| odyssey4me | lemme grab some coffee first | 12:47 |
| chandankumar | odyssey4me: sure thanks :-) | 12:47 |
| jrosser | chandankumar: i think you can make that much simpler, the 10.1.3.x address should be accessible directly with ping, no need to use the netns | 12:48 |
| jrosser | the external IP of the router is public | 12:48 |
| chandankumar | jrosser: ok just ping module will do the job | 12:48 |
| jrosser | no, thats ansible ping not ip ping | 12:49 |
| jrosser | ^ confusing :/ | 12:49 |
| chandankumar | ok got it | 12:49 |
| chandankumar | jrosser: let me update the patch | 12:49 |
| jrosser | you see the field with the ip there in the error? just extract the right part of that dict | 12:49 |
| jrosser | chandankumar: to be even more excellent you could assert that 'admin_state_up': True | 12:51 |
| chandankumar | ok, nice | 12:51 |
| jrosser | router up, ping fails is even more useful debug | 12:54 |
| jrosser | chandankumar: maybe -c1 is a bit fail prone, particularly if arp tables are empty, the first one might go missing | 12:55 |
| odyssey4me | chandankumar https://gist.github.com/odyssey4me/e1bd3b3bea5851a988ac015c94b4b52e should do the trick | 12:57 |
| odyssey4me | (for the tripleo changes) | 12:58 |
| openstackgerrit | Martin Kopec proposed openstack/openstack-ansible-os_tempest master: Improve overview subpage https://review.openstack.org/633934 | 12:58 |
| openstackgerrit | Martin Kopec proposed openstack/openstack-ansible-os_tempest master: Improve overview subpage https://review.openstack.org/633934 | 13:00 |
| *** Pbing has quit IRC | 13:10 | |
| chandankumar | odyssey4me: thanks , will try that | 13:12 |
| *** markvoelker has quit IRC | 13:13 | |
| mnaser | morning all :) | 13:38 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_heat master: [DNM] heat tempest tests finding https://review.openstack.org/630694 | 13:39 |
| jrosser | o/ mnaser | 13:40 |
| mnaser | how are ya jrosser | 13:42 |
| jrosser | good, busy but good | 13:42 |
| jrosser | did you have any more thoughts on that heat internal vs external thing? | 13:42 |
| mnaser | jrosser: i haven't dug as much, i am concerned because i think that the url presented for os-collect-config might be one of those | 13:45 |
| mnaser | which means that a vm gets deployed would get an internal url | 13:45 |
| jrosser | i was worried about the original patch really too | 13:46 |
| jrosser | but i'm not using heat for anything so don't have a reference right now | 13:46 |
| *** rgogunskiy has quit IRC | 13:47 | |
| mnaser | jrosser: i mean i know without that patch, magnum doesn't work if your internal net isnt accessible for sure | 13:48 |
| *** pcaruana has quit IRC | 13:50 | |
| jrosser | our tests are unhelpful, because the service containers all get to see the inside and outside trivially | 13:51 |
| mnaser | jrosser: yeah we don't do a good job on that part | 13:51 |
| jrosser | i want to revisit the patch i did for tempest yesterday because in hindsight thats just made it worse | 13:52 |
| jrosser | as in, more short-circuiting of inside/outside | 13:52 |
| mnaser | i think so | 13:54 |
| mnaser | i mean i think its a big major effort we need to look into | 13:54 |
| jrosser | we should check this with cloudnull - he did a bunch of stuff on the host networking recently | 13:54 |
| *** strattao has joined #openstack-ansible | 13:55 | |
| *** pcaruana has joined #openstack-ansible | 13:57 | |
| mnaser | yeah, i also need to get around pulling containers out of c7 | 13:57 |
| mnaser | jrosser: lol, guilhermesp literally just got blocked because an internal path was just pulled in a magnum deployment | 14:01 |
| guilhermesp | hhahaha mnaser | 14:01 |
| mnaser | guilhermesp: do you have this patch in your checkout? https://review.openstack.org/#/c/619355/ | 14:01 |
| guilhermesp | mnaser: checking | 14:02 |
| guilhermesp | all right, I don't | 14:03 |
| guilhermesp | it is using the internal endpoint | 14:03 |
| guilhermesp | I can check it out a re-execute the heat playbook to see how it goes | 14:04 |
| jrosser | mnaser: hrrm - in your deploys do you allow your internal things to nat out and hit the external endpoint? | 14:04 |
| mnaser | guilhermesp: yes, try again with the patch | 14:06 |
| mnaser | jrosser: yes, they can technically do that | 14:06 |
| mnaser | its not firewalled off | 14:07 |
| jrosser | that might be what allows it to work for you, when it failed for ThiagoCMC yesterday | 14:07 |
| jrosser | heat, that is | 14:07 |
| mnaser | well heat works fine in this scenario | 14:14 |
| mnaser | even after that patch | 14:14 |
| *** sdake has quit IRC | 14:20 | |
| guilhermesp | owo mnaser jrosser the cluster was created successfuly | 14:20 |
| guilhermesp | :D | 14:21 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-tests master: Add delorean-deps.repo in OSA https://review.openstack.org/633882 | 14:24 |
| mnaser | so yeah.. it's the fix | 14:25 |
| mnaser | and heat works fine with it applied too, right guilhermesp ? | 14:26 |
| guilhermesp | seems to be mnaser I didn't test heat specifically but I assume is working because the creation was completed | 14:27 |
| guilhermesp | cluster creation* | 14:27 |
| *** jpward1981 has quit IRC | 14:28 | |
| *** jpward1981 has joined #openstack-ansible | 14:29 | |
| *** sdake has joined #openstack-ansible | 14:34 | |
| mgariepy | can I have some review on https://review.openstack.org/#/c/632907/ and https://review.openstack.org/#/c/632908/ please :) | 14:43 |
| jrosser | guilhermesp: whats the state of the OSA magnum bits? "just works"? | 14:43 |
| *** cmart has joined #openstack-ansible | 14:45 | |
| *** jpward1981 has quit IRC | 14:47 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Always generate stackviz irrespective of tests pass or fail https://review.openstack.org/631967 | 14:47 |
| *** sdake has quit IRC | 14:49 | |
| guilhermesp | jrosser: in my 18.1.1 deployment, now it is working. I needed a few workarounds | 14:49 |
| guilhermesp | one of them, is on task that I'm going to do asap: the heat user in heat domain didn't get an admin role | 14:49 |
| guilhermesp | second: that heat patch mnaser provided | 14:50 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Update cirros from 3.5 to 3.6 https://review.openstack.org/633208 | 14:51 |
| *** sdake has joined #openstack-ansible | 14:55 | |
| gshippey | https://review.openstack.org/#/c/608031/ has anyone seen this issue on a fresh rocky AIO? | 15:05 |
| gshippey | specifically with octavia deployed | 15:07 |
| jrosser | odyssey4me: do we need to backport that ^ ? | 15:08 |
| odyssey4me | jrosser it would appear so, given it's master only | 15:08 |
| *** udesale has joined #openstack-ansible | 15:08 | |
| jrosser | ok, we'll do it | 15:09 |
| odyssey4me | thanks jrosser | 15:09 |
| openstackgerrit | Georgina Shippey proposed openstack/openstack-ansible-os_neutron stable/rocky: Fix whitespace in neutron.conf template https://review.openstack.org/633973 | 15:09 |
| *** sdake has quit IRC | 15:11 | |
| odyssey4me | jrosser also interesting is https://github.com/openstack/openstack-ansible-os_neutron/commit/30e20d6cd7a6a248aa2336f501e3de9b47f263be#diff-574b76c26842c4c0c2607fc38bf7e90d | 15:11 |
| *** sdake has joined #openstack-ansible | 15:13 | |
| mnaser | isnt that whole layer deprecated in rocky btw? | 15:15 |
| ThiagoCMC | Good morning guys! =P | 15:15 |
| *** TxGirlGeek has joined #openstack-ansible | 15:16 | |
| ThiagoCMC | I'm curious about something... Yesterday I finally managed to deploy OSA/Rocky/Ubuntu with Ceph, Glance, Cinder, Nova, Neutron and Heat working! Then, I tried the Object Storage RadosGW, it was deployed successfully but, when I access Horizon, I'm seeing the following errors: | 15:17 |
| ThiagoCMC | This error is seen when 'Object Storage' is selected: Error: Unable to get the Swift container listing. | 15:17 |
| ThiagoCMC | This error is seen when I attempt to create containers: Unable to create containers. | 15:17 |
| ThiagoCMC | Any idea? | 15:18 |
| ThiagoCMC | I don't need Swift to use RadosGW, right> | 15:18 |
| ThiagoCMC | ? | 15:18 |
| cloudnull | mornings | 15:19 |
| ThiagoCMC | Morning! =) | 15:19 |
| *** ostackz has joined #openstack-ansible | 15:22 | |
| odyssey4me | ThiagoCMC Nope, you don't need swift - what's happening, though, is that radosgw is pretending to be swift - which is why the horizon plugin is there, I think. | 15:23 |
| odyssey4me | ThiagoCMC There may be something broken in the rados config - I dunno. | 15:23 |
| jrosser | we have that working here - horizon shows us the radosgw contents | 15:24 |
| jrosser | stuartgr might be able to help | 15:24 |
| stuartgr | ThiagoCMC: the object storage pages in horizon work for us. We don''t have swift deployed, just a rados gateway providing the swift API listed in the service catalog | 15:27 |
| *** TxGirlGeek has quit IRC | 15:28 | |
| stuartgr | ThiagoCMC: we disable the S3 API and put the following setting in ceph.conf on the rados gateway: rgw swift url prefix = / | 15:32 |
| odyssey4me | stuartgr ThiagoCMC I know that fghaas also did a bunch of work recently to make it work with swift enough to pass all the appropriate swift API standard tests. | 15:33 |
| ThiagoCMC | Oh, wow! Thank you guys so much for this help! :-D | 15:38 |
| ThiagoCMC | I'll try that prefix thing... | 15:38 |
| ostackz | cloudnull hi, could you help to understand what are correct git commands to issue regarding systemd? | 15:46 |
| cloudnull | whats going on ? | 15:46 |
| ostackz | We recently had a chat on topic that syslog-remote did not receive data from other hosts, you had fix for it http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2019-01-22.log.html#t2019-01-22T16:32:12 | 15:46 |
| cloudnull | yes | 15:47 |
| ostackz | I have fresh Rocky 18.1.2 install after that, but Im afraid to run following - not to switch my git to Stein: | 15:47 |
| ostackz | cd /opt/openstack-ansible; git fetch https://git.openstack.org/openstack/openstack-ansible refs/changes/05/632505/1 && git checkout FETCH_HEAD; | 15:47 |
| odyssey4me | ostackz that fix is in rocky already - just checkout stable/rocky | 15:47 |
| ostackz | openstack-ansible infra-journal-remote.yml | 15:47 |
| ThiagoCMC | stuartgr, I'm reading that if S3 is disabled, then, it's impossible to use multi-site configuration! This sounds undesirable but, I'll try it anyway! Also, do you add that line under [global] in ceph.conf? | 15:47 |
| odyssey4me | ostackz also, that would have you checking out master - not rocky! | 15:48 |
| cloudnull | ostackz what odyssey4me said, but you could also run the git command with "cherry-pick" instead of "checkout" | 15:48 |
| ostackz | odyssey4me, cloudnull what I would be happy to do is have this systemd patched on top of 18.1.2 but not switch away from it so I can for sure upgrade to 18.1.3 later | 15:50 |
| ostackz | not master of git yet so not sure if I can get patches in rocky/stable if Im in 18.1.2 | 15:51 |
| odyssey4me | ostackz if you checkout stable/rocky now then you will have the currently proposed 18.1.3... and if there are further changes, you will be able to upgrade to whatever is actually released as 18.1.3 | 15:54 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add the ability to set the JVM heap size https://review.openstack.org/633984 | 15:54 |
| odyssey4me | as far as I know, the request to release went in yesterday | 15:54 |
| mnaser | jrosser: will you be at denver? | 15:55 |
| mnaser | also asking same for cloudnull odyssey4me evrardjp and others :) | 15:55 |
| mnaser | unless thats too early to ask now | 15:55 |
| odyssey4me | ostackz yep, the release request went in yesterday: https://review.openstack.org/633719 | 15:56 |
| odyssey4me | ostackz if you wish to checkout that particular SHA to match it before the tag goes out, then do: git checkout 93bcc1fd24ae1d46c8773cd218f0b63531f273b0 | 15:57 |
| ThiagoCMC | stuartgr, I added the " rgw swift url prefix = /" to my 3 radosgw containers and restarted them, however, Horizon still can't list containers, neither create them. :-( | 15:57 |
| *** gkadam-bmgr has quit IRC | 15:57 | |
| odyssey4me | ostackz that gets you the journal fix: https://github.com/openstack/openstack-ansible/commits/93bcc1fd24ae1d46c8773cd218f0b63531f273b0 | 15:57 |
| ThiagoCMC | So I need to explicity disable S3 somewhere as well? | 15:58 |
| *** arxcruz|ruck is now known as arxcruz | 15:58 | |
| ostackz | odyssey4me ok, starting to get it! Only thing remaining is actual commands to run, my "git tag" only shows 18.1.2 so I would do something like "cd /opt/openstack-ansible/ ; git checkout ...?" to get current stage of 18.1.3 ? | 15:58 |
| odyssey4me | ostackz - cd /opt/openstack-ansible; git checkout 93bcc1fd24ae1d46c8773cd218f0b63531f273b0 | 16:00 |
| *** pcaruana has quit IRC | 16:01 | |
| odyssey4me | well, you might need to do a little more, so let's be sure | 16:01 |
| odyssey4me | cd /opt/openstack-ansible; git fetch --all; git checkout 93bcc1fd24ae1d46c8773cd218f0b63531f273b0 | 16:01 |
| *** udesale has quit IRC | 16:02 | |
| ostackz | odyssey4me thanks! git part worked fine, now will rerun playbooks. After these commands I understand Im somewhere between 18.1.2 - 18.1.3 and when 18.1.3 comes out it should be ok to apply that? | 16:09 |
| stuartgr | ThiagoCMC: disable S3 API by omitting it from this list in ceph.conf: rgw enable apis = swift, swift_auth, admin | 16:09 |
| evrardjp | ostackz: https://review.openstack.org/#/c/633719/ | 16:11 |
| odyssey4me | ostackz unless https://review.openstack.org/633719 changes, that SHA is 18.1.3 | 16:11 |
| jrosser | ThiagoCMC: regarding having to turn S3 off - we have had to deploy different radosgw instances to serve S3 and swift, otherwise you can't pass the validation tests. Both API really need to be served from / | 16:13 |
| *** sdake has quit IRC | 16:14 | |
| ostackz | evrardjp odyssey4me well, then if I do cd /opt/openstack-ansible; git fetch --all; git checkout I4933d91cb989b3fa3010db8528c56b616f99ce7c I actually would be in 18.1.3? | 16:15 |
| ostackz | When will that change marked as tag 18.1.3? | 16:15 |
| *** Pbing has joined #openstack-ansible | 16:16 | |
| odyssey4me | ostackz when the above-mentioned review is merged | 16:16 |
| openstackgerrit | Juri Hudolejev proposed openstack/openstack-ansible-os_glance stable/rocky: Fix Glance NFS mount point ownership https://review.openstack.org/633993 | 16:17 |
| odyssey4me | usually that takes no more than 2-3 days from the release team | 16:17 |
| *** pcaruana has joined #openstack-ansible | 16:17 | |
| *** kopecmartin is now known as kopecmartin|off | 16:20 | |
| *** sdake has joined #openstack-ansible | 16:21 | |
| ostackz | odyssey4me, cloudnull, evrardjp thanks, got more clear this git checkout stuff! My syslogs start to flow now and will wait for 18.1.3 comming soon. | 16:22 |
| odyssey4me | ostackz a tag is just a human-friendly way to refer to a SHA - it's a pointer from the tag to the SHA | 16:27 |
| *** nurdie has joined #openstack-ansible | 16:28 | |
| *** TxGirlGeek has joined #openstack-ansible | 16:31 | |
| *** macza has joined #openstack-ansible | 16:36 | |
| *** macza has quit IRC | 16:37 | |
| ThiagoCMC | jrosser, is this going to be like this forever? I mean, RadosGW = no S3 and no multi-site / you need two kinds of RadosGW? | 16:41 |
| jrosser | that is what stuartgr and I have built, yes | 16:41 |
| ThiagoCMC | Oh, I see... Well, ok then! =) | 16:41 |
| odyssey4me | ThiagoCMC I think he's suggesting to go with a known good config, then once that's working you can experiment to get to where you want to go. | 16:42 |
| ThiagoCMC | Sure! | 16:42 |
| *** gyee has joined #openstack-ansible | 16:42 | |
| jrosser | start with radosgw doing swift and get that to pass tempest/refstack/whatever and the horizon integration going | 16:42 |
| jrosser | if you want S3 as well thats not really an OSA thing, you need to extend your ceph deployment somehow | 16:43 |
| ThiagoCMC | Okdok! | 16:43 |
| ThiagoCMC | Thanks for clarifying that! | 16:43 |
| ThiagoCMC | Maybe for S3, go with Swift instead of Ceph? To be an OSA thing then? | 16:43 |
| jrosser | you can probably construct enough extra config for some radosgw-s3 containers and haproxy endpoints to do that | 16:43 |
| jrosser | it would all just be extra config data | 16:43 |
| ThiagoCMC | ok | 16:44 |
| ThiagoCMC | Is it possible to use Swift for S3 while hosting data on Ceph somehow? | 16:44 |
| ThiagoCMC | Or it doesn't make any sense? lol | 16:44 |
| jrosser | ^ that, i think :) | 16:44 |
| odyssey4me | ThiagoCMC ceph-rgw's advantage is using a common back-end for storage with your block storage (cinder) too... so same hardware = more flexibility | 16:44 |
| odyssey4me | swift does things that ceph doesn't, and ceph does things swift doesn't | 16:45 |
| odyssey4me | you really have to figure out what's important to you for your use-case | 16:45 |
| odyssey4me | you can also have both as far as I know, although then you'd have to double up storage servers which probably doesn't make much sense | 16:45 |
| jrosser | we also use a dedicated haproxy for this becasue we have massive object traffic, so odyssey4me is right you need to tune up your deployment to match your use case | 16:45 |
| *** pcaruana has quit IRC | 16:45 | |
| ThiagoCMC | Nice! What's important for me is S3 and multi-site... | 16:48 |
| ThiagoCMC | But we're using Ceph for everything | 16:48 |
| jrosser | the same here, minus the multi-site | 16:49 |
| ThiagoCMC | cool =) | 16:49 |
| *** electrofelix has quit IRC | 16:49 | |
| jrosser | it would be nice to come up with an example to document this in the ops repo, if you choose to do it all with the OSA/ceph integration | 16:49 |
| *** sdake has quit IRC | 16:51 | |
| *** TxGirlGeek has quit IRC | 16:56 | |
| *** TxGirlGeek has joined #openstack-ansible | 16:57 | |
| jrosser | cloudnull: this you just rechecked https://review.openstack.org/#/c/633104/ | 17:00 |
| cloudnull | yes? | 17:01 |
| jrosser | take a read of this https://review.openstack.org/#/c/633732/ which i think is the same reason it is failing, but in a different role | 17:01 |
| *** sdake has joined #openstack-ansible | 17:01 | |
| cloudnull | looks like that merged yesterday? | 17:02 |
| cloudnull | so this recheck should work, I assume? | 17:02 |
| *** radeks_ has quit IRC | 17:02 | |
| *** radeks_ has joined #openstack-ansible | 17:02 | |
| jrosser | so, i have a theory that changing the host networking to systemd_network has brok things on centos | 17:03 |
| jrosser | which shows itself as container traffic in role tests heading out eth0 / default route can no longer get to 10.1.3.x on br-vlan | 17:03 |
| jrosser | we held a node the other day and poked it after one broke, and that was certainly the case | 17:04 |
| cloudnull | did the ip exist on the interface? | 17:04 |
| jrosser | so either we go round all the role tests and fix them up, or there is something subtle happening with systemd_networkd on centos which breaks the previous behaviour | 17:05 |
| jrosser | no it didnt | 17:05 |
| cloudnull | was the ip in config though ? | 17:05 |
| jrosser | before the tests relied on the default route and host networking for tempest to ssh to the instances | 17:05 |
| cloudnull | or was it all together missing | 17:05 |
| jrosser | i added one, and it started working https://review.openstack.org/#/c/633732/2/tests/host_vars/tempest1.yml | 17:06 |
| cloudnull | ah | 17:06 |
| jrosser | but it's a bit ewwwwwww becasue thats now on all the containers, not just tempest | 17:06 |
| jrosser | and thats very much crossed the streams of internal/external networking, which i don't really like | 17:06 |
| *** priteau has quit IRC | 17:08 | |
| cloudnull | we could add a local only route instead of an IP adderss? | 17:08 |
| jrosser | well, ideally there is something we can do in the systemd_networkd role to mend whatever is wrong on centos | 17:09 |
| jrosser | otherwise we have to visit all the role tests and fix them up | 17:09 |
| cloudnull | something like `ip route add 10.1.3.103/24 dev br-vlan table local` | 17:09 |
| cloudnull | well. `ip route add 10.1.3.0/24 dev br-vlan table local` | 17:09 |
| jrosser | on the host? | 17:10 |
| cloudnull | yes | 17:10 |
| cloudnull | isn't br-vlan supposed to have an IP on it ? | 17:10 |
| jrosser | yes it has one, the .1 is on there | 17:10 |
| cloudnull | ok | 17:10 |
| cloudnull | but in the container theres no address | 17:11 |
| jrosser | right - eth12 was there and wired, and hooked to br-vlan | 17:12 |
| cloudnull | ok | 17:12 |
| jrosser | so the "fix" for tempest was to stick and IP on it, and to keep the test host_vars simple to all the other containers too | 17:12 |
| jrosser | but this is only centos brokenness | 17:12 |
| cloudnull | "\ | 17:13 |
| * cloudnull <begins rant about centos> | 17:13 | |
| cloudnull | do we need to implement that fix in the tempest role generally ? | 17:13 |
| jrosser | we can do, but it's a lot of patches, so if there is a route we can add somewhere globally that would be prefereably | 17:14 |
| cloudnull | so in the event we have br-vlan and eth12 we need a route added to the cidr used by br-vlan ? | 17:17 |
| mnaser | cloudnull: i'll be removing container support in centos-7 -- which i believe largely affects/caused by this | 17:18 |
| cloudnull | mnaser it sounds like an issue regardless of containers. | 17:18 |
| mnaser | sorry i see bridges and assume containers :) | 17:18 |
| mnaser | i didn't read context | 17:19 |
| *** nurdie has quit IRC | 17:19 | |
| * cloudnull just getting up to speed myself | 17:19 | |
| cloudnull | so maybe it is | 17:19 |
| jrosser | cloudnull: adding the ip to eth12 automatically gives the route in the container and everything works | 17:19 |
| *** nurdie has joined #openstack-ansible | 17:19 | |
| jrosser | but it does leave open what is actually brok on the host compared to bionic, for example | 17:20 |
| jrosser | smells like some systemd version nonsense | 17:20 |
| mnaser | ok y'all | 17:20 |
| cloudnull | could be | 17:20 |
| mnaser | when are we just going to start rolling out our own distro | 17:21 |
| mnaser | im down for it at this point | 17:21 |
| cloudnull | gentoo ? | 17:21 |
| mnaser | latest kernels to support latest tech | 17:21 |
| cloudnull | -cc prometheanfire :) | 17:21 |
| mnaser | one common deployment target | 17:21 |
| cloudnull | jrosser itd be awesome to compare the routing table of cent vs bionic | 17:22 |
| cloudnull | if its just something missing then we can simply make sure the missing bits are added | 17:22 |
| cloudnull | which test did this crop up on ? | 17:22 |
| cloudnull | I can spin a cent and bionic test instance of the same thing to try and recreate it | 17:23 |
| mnaser | i know the systemd_networkd bits arent super reliable (the ones that creates routes and all) | 17:23 |
| jrosser | os_tempest, but that got attention becasue there is quite a lot of activity there and it was blocking stuff merging | 17:23 |
| mnaser | like it has a lot of "|| true" or things like that | 17:23 |
| mnaser | so lots of things can break and it just noops | 17:23 |
| bgmccollum | is the current discussion why my flat external network wasn't working properly? | 17:23 |
| prometheanfire | wat | 17:23 |
| prometheanfire | cloudnull: cent7 non-networkd issues? | 17:24 |
| jrosser | cloudnull: if you could compare centos vs. bionic routing now the pressure is off unblocking tempest that would be ace | 17:24 |
| cloudnull | bgmccollum kinda. | 17:24 |
| mnaser | prometheanfire: we're just deciding we're dropping support for all operating systems and just doing gentoo from now on | 17:24 |
| mnaser | with our own hand rolled kernels and os | 17:24 |
| prometheanfire | mnaser: sgtm | 17:24 |
| prometheanfire | I've been hand-rolling my kernels forever :P | 17:25 |
| odyssey4me | mnaser cloudnull so... we're just gonna turn ourselves into Piston? | 17:25 |
| prometheanfire | lol | 17:25 |
| cloudnull | hahahaha | 17:25 |
| bgmccollum | pxe -> initrd -> dd -> done? | 17:25 |
| mnaser | minus the part where we disappear :( | 17:25 |
| cloudnull | odyssey4me exactly :) | 17:25 |
| mnaser | rip piston | 17:25 |
| cloudnull | they had the best parties | 17:25 |
| mnaser | maybe that contributed to their disappearance | 17:25 |
| mnaser | ahaha | 17:25 |
| cloudnull | hahaha | 17:25 |
| cloudnull | could be | 17:25 |
| odyssey4me | lol | 17:26 |
| prometheanfire | you can always judge a project by their parties | 17:26 |
| mnaser | i always say hp died off | 17:26 |
| mnaser | because of those crazy parties | 17:26 |
| mnaser | lol | 17:26 |
| prometheanfire | possible | 17:26 |
| cloudnull | im sure that was part of it | 17:26 |
| cloudnull | jrosser so just run tests from os_tempst? | 17:26 |
| jrosser | do whatever the role test in the gate does | 17:26 |
| cloudnull | ok. | 17:27 |
| odyssey4me | cloudnull you'd need to revert the fix jrosser made - then do run_tests.sh functional | 17:30 |
| mnaser | btw, cores -- just a heads up | 17:31 |
| mnaser | http://lists.openstack.org/pipermail/openstack-discuss/2019-January/002176.html and http://lists.openstack.org/pipermail/openstack-discuss/2019-January/002237.html | 17:31 |
| *** DanyC has quit IRC | 17:31 | |
| odyssey4me | that reminds me, I am just not getting those emails - I need to figure out why | 17:31 |
| odyssey4me | oh yes, very welcome jamesdenton | 17:31 |
| *** DanyC has joined #openstack-ansible | 17:31 | |
| jrosser | oh thats cool - i'm already plotting an adventure with networking-vpp | 17:33 |
| cloudnull | YES! welcome jamesdenton! | 17:35 |
| * cloudnull goes to update my list subscription ... | 17:35 | |
| mnaser | odyssey4me: are you subscribed to openstack-discuss? | 17:36 |
| *** DanyC_ has joined #openstack-ansible | 17:37 | |
| *** DanyC has quit IRC | 17:37 | |
| mnaser | openstack-dev and openstack are not longer a thing, moved to openstack-discuss | 17:37 |
| odyssey4me | mnaser I have requested it several times, but I see that nothing's ever come back to me. I'll work out what's going on tomorrow. | 17:37 |
| odyssey4me | Time for me to exit for the day. Cheers folks! | 17:38 |
| mnaser | later odyssey4me | 17:40 |
| *** cmart has quit IRC | 17:43 | |
| jamesdenton | weeeeee | 17:43 |
| jamesdenton | https://www.youtube.com/watch?v=wSqWc88Qj4U | 17:44 |
| jamesdenton | thanks mnaser odyssey4me cloudnull. happy to help! | 17:45 |
| mnaser | lols | 17:46 |
| *** DanyC_ has quit IRC | 17:52 | |
| ThiagoCMC | stuartgr, I only have "rgw_enable_apis = swift", so I believe that S3 isn't there but, I don't have swift_auth, neither admin. Under which config [group] you have yours "rgw swift url prefix = /"? | 18:02 |
| ThiagoCMC | Here is my ceph.conf from rgw container (from OSA): http://paste.openstack.org/show/744271/ | 18:03 |
| ThiagoCMC | Horizon still can't list object containers, neither create them. | 18:03 |
| ThiagoCMC | :- | 18:03 |
| stuartgr | ThiagoCMC: "rgw swift url prefix = /" should be under [client.rgw.vucmon-1-ceph-rgw-container-875ddb12] | 18:08 |
| ThiagoCMC | Oh, thanks! | 18:08 |
| ThiagoCMC | Just on first container? | 18:09 |
| ThiagoCMC | better replicate 3 * 3, right? | 18:09 |
| stuartgr | yes, on all containers | 18:09 |
| ThiagoCMC | ok | 18:09 |
| nsmeds | hey guys - if we're going to upgrade from Queens to Rocky, do we need to be on latest version of Queens before upgrading? | 18:12 |
| nsmeds | using 17.1.2, looks like newest is 17.1.7 | 18:12 |
| nsmeds | err, 17.1.6* | 18:13 |
| *** shardy has quit IRC | 18:15 | |
| ThiagoCMC | stuartgr, still error: "Unable to get the Swift container listing" :-( | 18:16 |
| *** pcaruana has joined #openstack-ansible | 18:17 | |
| ThiagoCMC | could you share a working ceph.conf from a ceph-rgw container? | 18:19 |
| stuartgr | swift API expected to be on port 7980 I think, so where you have "rgw frontends = civetweb port=172.29.236.172:8080 num_threads=100" that should be "rgw frontends = civetweb port=172.29.236.172:7980 num_threads=100" | 18:22 |
| stuartgr | and similar on other containers | 18:22 |
| jrosser | That needs to match up with the haproxy config | 18:23 |
| ThiagoCMC | Yeah, I'm just using the OSA/Rocky defauls... | 18:23 |
| jrosser | And the object storage entry in the device catalogue | 18:23 |
| ThiagoCMC | sounds hard to deploy it via openstack-ansible and then, change a bunch of things... | 18:23 |
| jrosser | *service | 18:23 |
| ThiagoCMC | I'm deploying both OpenStack and Ceph (osd/mon/mgr/rgw) via OSA. | 18:24 |
| ThiagoCMC | I was expecting that it would just works out of the box :-P | 18:24 |
| jrosser | osa just wraps ceph-ansible | 18:24 |
| ThiagoCMC | sure | 18:24 |
| ThiagoCMC | I can tell that my ceph cluster os healthy, glance works, nova boot and cinder create/attache, all using rbd | 18:25 |
| jrosser | As far as out of the box goes there is a ceph test in the osa gate which should be your reference | 18:25 |
| ThiagoCMC | Hmmm | 18:25 |
| jrosser | That deploys all the bits enough to pass tempest with object storage | 18:25 |
| ThiagoCMC | I see... Well, it isn't working for me... :-( | 18:28 |
| ThiagoCMC | But, okay | 18:28 |
| stuartgr | ThiagoCMC: ceph.conf from one of our rgw containers: http://paste.openstack.org/show/744272/ | 18:30 |
| ThiagoCMC | Thank you!!! | 18:30 |
| ThiagoCMC | You had to change the haproxy settings for that port, right? | 18:32 |
| ThiagoCMC | Is there a OSA-way of doing this? Or you just manually changed it after ansible? | 18:32 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Disable nova-lxd tempest plugin https://review.openstack.org/633711 | 18:33 |
| ThiagoCMC | Ok, I'm also manually updating the port to 7980 at /etc/haproxy/conf.d/ceph-rgw | 18:43 |
| ThiagoCMC | Same problem... =/ | 18:44 |
| ThiagoCMC | Never mind... | 18:44 |
| ThiagoCMC | I gave up on OSA / radosgw / horizon for now | 18:44 |
| jrosser | The haproxy config is generates from here https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/haproxy/haproxy.yml#L343 | 18:47 |
| jrosser | ThiagoCMC: the thing here is that the OSA role defaults/main.yml and the host/group vars set “sensible defaults” | 18:48 |
| jrosser | You are free then to use the user_variabkes.yml or your own host/group bars to override any of that as you need | 18:49 |
| jrosser | *vars | 18:49 |
| *** TxGirlGeek has quit IRC | 18:50 | |
| *** TxGirlGeek has joined #openstack-ansible | 18:51 | |
| *** Darcidride_ has joined #openstack-ansible | 18:52 | |
| *** Darcidride_ has quit IRC | 18:54 | |
| *** priteau has joined #openstack-ansible | 18:54 | |
| *** Darcidride_ has joined #openstack-ansible | 18:55 | |
| *** Darcidride_ has quit IRC | 18:55 | |
| ThiagoCMC | Ok! I manually changed the rgw's ceph.conf and haproxy but, Horizon still can't list/create new containers on it. | 19:00 |
| *** TxGirlGeek has quit IRC | 19:00 | |
| ThiagoCMC | And I'm just shooting in the dark now, which isn't good. I'll try again later on, maybe on next OSA/Stein release and hope that it will just works | 19:00 |
| *** TxGirlGeek has joined #openstack-ansible | 19:02 | |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_heat master: Add heat user to heat domain admin role https://review.openstack.org/634032 | 19:06 |
| ThiagoCMC | guilhermesp, is this missing on OSA/Rocky stable branch? | 19:09 |
| ThiagoCMC | Quick question... 1 TASK from os-neutron-install.yml is failing only in 1 of 3 machines, error: | 19:26 |
| ThiagoCMC | fatal: [c3bb-os-cmpt]: FAILED! => {"attempts": 5, "changed": false, "cmd": "/openstack/venvs/neutron-18.1.3/bin/pip2 install -U --constraint http://172.29.239.250:8181/os-releases/18.1.3/ubuntu-18.04-x86_64/requirements_absolute_requirements.txt neutron_dynamic_routing | 19:26 |
| ThiagoCMC | neutron_fwaas neutron_lbaas", "msg": "\n:stderr: Traceback (most recent call last):\n File \"/openstack/venvs/neutron-18.1.3/bin/pip2\", line 7, in <module>\n from pip._internal import main\nImportError: No module named pip._internal\n"} | 19:26 |
| ThiagoCMC | TASK: os_neutron : Install optional pip packages | 19:27 |
| *** cmart has joined #openstack-ansible | 19:51 | |
| guilhermesp | ThiagoCMC: yes. We need to wait it to be merged in master to backport it | 19:59 |
| *** priteau has quit IRC | 20:01 | |
| *** sdake has quit IRC | 20:06 | |
| *** sdake has joined #openstack-ansible | 20:06 | |
| ThiagoCMC | Ok! | 20:08 |
| cmart | will a lot of you be in Denver at the end of April? strongly considering actually going to one of these | 20:08 |
| mnaser | cmart: i will be personally :) | 20:08 |
| cmart | forgive the dumb question, will there be an OSA gathering / track? | 20:09 |
| mnaser | cmart: we have the ptg which is the project team gathering where the OSA team has a few days where we sit and scope up the next cycle | 20:09 |
| mnaser | and we also do some hacking | 20:09 |
| mnaser | and some hanging out :) | 20:10 |
| cmart | right on. is that at the summit or before? | 20:10 |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_tempest master: Only init a workspace if doesn't exists https://review.openstack.org/633549 | 20:11 |
| *** sdake has quit IRC | 20:39 | |
| *** sdake has joined #openstack-ansible | 20:43 | |
| *** sdake has quit IRC | 20:44 | |
| *** mgariepy has quit IRC | 20:45 | |
| *** sdake has joined #openstack-ansible | 20:46 | |
| *** hwoarang has quit IRC | 20:50 | |
| *** hwoarang has joined #openstack-ansible | 20:51 | |
| *** sdake has quit IRC | 21:01 | |
| *** sdake has joined #openstack-ansible | 21:02 | |
| cloudnull | jrosser mnaser odyssey4me - I spun up a test instance and run the tempest tests (cloned then executed run-tests.sh) for the os_tempest repo | 21:04 |
| cloudnull | ubuntu worked, cent failed. | 21:04 |
| cloudnull | and it failed on the python venv build | 21:04 |
| cloudnull | is that something we're aware of , or maybe its just a transient thing | 21:04 |
| cloudnull | that said, the routing table looks the same across distros | 21:04 |
| jrosser | things have passed recently https://review.openstack.org/#/q/project:openstack/openstack-ansible-os_tempest | 21:05 |
| jrosser | if you enter the tempest1 container can you ping 10.1.3.x things? | 21:05 |
| cloudnull | yes https://pasted.tech/pastes/c77ef1cb7f59953c7b7ef45d5f55c2fbc9a71ee9 | 21:07 |
| jrosser | this was the symptom of it http://logs.openstack.org/08/633208/9/check/openstack-ansible-functional-centos-7/95a334e/logs/openstack/tempest1/stestr_results.html | 21:07 |
| * cloudnull looking | 21:07 | |
| jrosser | and this is with my patch reverted? | 21:09 |
| cloudnull | oh no. | 21:12 |
| cloudnull | that was clone of master | 21:12 |
| * cloudnull can do another one | 21:12 | |
| *** radeks_ has quit IRC | 21:19 | |
| cloudnull | jrosser my test didnt get that far | 21:22 |
| cloudnull | for cent | 21:22 |
| *** zul has quit IRC | 21:27 | |
| jrosser | Right, but it made the containers, and added ip to eth12? | 21:28 |
| cloudnull | neither deployment has an IP on eth12 | 21:34 |
| cloudnull | though it does exist | 21:34 |
| cloudnull | https://pasted.tech/pastes/5003f296d04e28653d705f1a0e9170e7fd50b5d5 | 21:35 |
| cloudnull | thats what i see in br-vlan | 21:35 |
| cloudnull | these are the interface files I see /etc/systemd/network/6-general-eth12-veth.link /etc/systemd/network/6-general-eth12-veth.network | 21:36 |
| *** sdake has quit IRC | 21:38 | |
| *** nurdie has quit IRC | 21:47 | |
| *** nurdie has joined #openstack-ansible | 21:48 | |
| *** sdake has joined #openstack-ansible | 21:49 | |
| *** openstackgerrit has quit IRC | 21:50 | |
| *** nurdie has quit IRC | 21:52 | |
| *** Pbing has quit IRC | 22:10 | |
| cloudnull | welp. i think i found the difference | 22:12 |
| cloudnull | xenial and bionic use `/sbin/ip` while centos uses `/usr/sbin/ip` and suse uses `/bin/ip`. | 22:13 |
| cloudnull | and the tests role just uses /usr/sbin/ip | 22:14 |
| * cloudnull shakes fist at ALL THE LINUXES! | 22:14 | |
| cloudnull | mnaser now im ready to build a distro | 22:14 |
| cloudnull | :D | 22:14 |
| *** ostackz has quit IRC | 22:15 | |
| apevec | FWIW /usr/sbin/ is correct and for backward compat centos has symlink /sbin -> usr/sbin | 22:18 |
| *** apevec has quit IRC | 22:18 | |
| ThiagoCMC | cloudnull, Linux from Scratch? lol | 22:19 |
| cloudnull | apevec +1 agree /usr/sbin is correct | 22:19 |
| cloudnull | so this case I agree with centos ") | 22:19 |
| cloudnull | ^ a first I'm sure | 22:20 |
| *** DanyC has joined #openstack-ansible | 22:40 | |
| *** strattao has quit IRC | 22:41 | |
| *** openstackgerrit has joined #openstack-ansible | 22:45 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-tests master: Set the ip tool path for each OS https://review.openstack.org/634057 | 22:45 |
| *** sdake has quit IRC | 23:01 | |
| *** hwoarang has quit IRC | 23:04 | |
| *** hwoarang has joined #openstack-ansible | 23:05 | |
| *** hwoarang has quit IRC | 23:10 | |
| *** hwoarang has joined #openstack-ansible | 23:11 | |
| *** nurdie has joined #openstack-ansible | 23:24 | |
| *** sdake has joined #openstack-ansible | 23:29 | |
| *** sdake has quit IRC | 23:33 | |
| *** slaweq has quit IRC | 23:38 | |
| *** nurdie_ has joined #openstack-ansible | 23:49 | |
| *** nurdie has quit IRC | 23:50 | |
| *** nurdie_ has quit IRC | 23:56 | |
| *** nurdie has joined #openstack-ansible | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!