| *** v1k0d3n has quit IRC | 00:02 | |
| Pavo | hmmm seems like something changed | 00:10 |
|---|---|---|
| Pavo | getting error that seems to be related to ports_checks.yml located in /usr/share/kolla/ansible/roles/prechecks/tasks/ | 00:10 |
| Pavo | http://pastebin.com/qYbyXzzA | 00:11 |
| *** diogogmt has joined #openstack-kolla | 00:11 | |
| Pavo | nvm figured it out, 1 of the interfaces on each node wasn't up lol | 00:18 |
| *** signed8bit is now known as signed8bit_Zzz | 00:30 | |
| *** hieulq has joined #openstack-kolla | 00:36 | |
| *** duonghq has joined #openstack-kolla | 00:40 | |
| Pavo | anyone know what this error means? | 00:46 |
| Pavo | http://pastebin.com/8cYVYE3d | 00:46 |
| Pavo | this is the same spot I got stuck at before | 00:48 |
| Pavo | but also getting this TASK [rabbitmq : Check if all rabbit hostnames are resolvable] ***************** | 00:50 |
| Pavo | ok: [192.168.1.11] => (item=192.168.1.11) | 00:50 |
| Pavo | so.. I have no clue were to start debugging | 00:50 |
| *** phuongnh has joined #openstack-kolla | 00:58 | |
| openstackgerrit | Wei Cao proposed openstack/kolla: Add solum container https://review.openstack.org/355408 | 01:00 |
| sbezverk | Pavo: you need to make sure that destination host where you deploy kolla to, its name should be dns resolved to 192.168.1.11 | 01:04 |
| Pavo | they are | 01:04 |
| Pavo | have all IPs and names in all nodes /etc/hosts | 01:05 |
| Pavo | did a ping of IPs, hostnames and even fqdn on all nodes | 01:05 |
| sbezverk | Pavo: ping that host by name from your deployment station | 01:05 |
| Pavo | thats why I am so confused | 01:05 |
| *** zhurong has joined #openstack-kolla | 01:05 | |
| Pavo | [root@deployer kolla]# ping node2 | 01:06 |
| Pavo | PING node2 (192.168.1.11) 56(84) bytes of data. | 01:06 |
| Pavo | 64 bytes from node2 (192.168.1.11): icmp_seq=1 ttl=64 time=0.631 ms | 01:06 |
| Pavo | 64 bytes from node2 (192.168.1.11): icmp_seq=2 ttl=64 time=0.377 ms | 01:06 |
| Pavo | 64 bytes from node2 (192.168.1.11): icmp_seq=3 ttl=64 time=0.499 ms | 01:06 |
| sbezverk | do you have different ip assigned to api_interface in globals? | 01:06 |
| Pavo | I only have VIP internal IP set in globals | 01:07 |
| Pavo | the one that is not pingable from all nodes | 01:07 |
| sbezverk | Pavo: what is this "ahostsv4" | 01:11 |
| sbezverk | ? | 01:11 |
| Pavo | have no clue | 01:11 |
| Pavo | here is my globals | 01:11 |
| Pavo | https://www.dropbox.com/s/2694rq4mkojt9ef/globals.yml?dl=0 | 01:11 |
| sbezverk | so it is not one of names or anything like that? | 01:11 |
| Pavo | no | 01:11 |
| Pavo | here is my multinode | 01:12 |
| Pavo | https://www.dropbox.com/s/xkhanhb0ysq6gym/multinode?dl=0 | 01:12 |
| Pavo | I hope they are correct | 01:12 |
| Pavo | oops I think I see the issue | 01:13 |
| Pavo | I have the network interfaces flipped I think on globals | 01:13 |
| sbezverk | Pavo: that is the good possibility to cause this issue | 01:14 |
| Pavo | trying again | 01:14 |
| *** haplo37__ has joined #openstack-kolla | 01:24 | |
| openstackgerrit | Wei Cao proposed openstack/kolla: Ubuntu with systemd support https://review.openstack.org/346900 | 01:26 |
| Pavo | looks like I am getting stuck again | 01:27 |
| Pavo | TASK [mariadb : Waiting for MariaDB service to be ready through VIP] *********** | 01:28 |
| Pavo | FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready through VIP (3 retries left). | 01:28 |
| *** rstarmer has quit IRC | 01:39 | |
| Pavo | where does kolla-ansible genconfig put the generated configs at? | 01:43 |
| openstackgerrit | Wei Cao proposed openstack/kolla: Ubuntu with systemd support https://review.openstack.org/346900 | 01:51 |
| *** haplo37__ has quit IRC | 01:53 | |
| openstackgerrit | Lu lei proposed openstack/kolla: Fix oslo.i18n in kolla project https://review.openstack.org/355342 | 01:56 |
| *** jtriley has joined #openstack-kolla | 02:02 | |
| *** inc0 has joined #openstack-kolla | 02:03 | |
| Pavo | ok I think I am getting really confised on what interface I need to use for a IP for kolla_internal_vip_address | 02:08 |
| Pavo | I have 2 interfaces eth0 and eth1 on each node, I am using eth0 for public network ie... routable IPs, and eth1 as non routable IPs | 02:09 |
| inc0 | Pavo, one of management iface | 02:10 |
| Pavo | then that would be eth0 | 02:10 |
| inc0 | one that you can access servers over | 02:10 |
| Pavo | I gave it an IP out of the IP range on that network that is not being used and it doesn't seem that when I deploy that its giving any ports or IPs to the containers | 02:11 |
| Pavo | I keep getting stuck at TASK [mariadb : Waiting for MariaDB service to be ready through VIP] *********** | 02:12 |
| inc0 | ip range? | 02:12 |
| Pavo | this is the inspect of that container on that node | 02:12 |
| Pavo | http://pastebin.com/jVMy8HSJ | 02:12 |
| inc0 | it should be single ip | 02:12 |
| Pavo | eth0 is on 192.168.1.0/24 network, eth1 is on 192.168.188.0/24 network | 02:13 |
| Pavo | eth0 is management/internet, eth1 is only between servers | 02:13 |
| Pavo | my nodes have deployer node eth0 192.168.1.33, node1 eth0 192.168.1.45 eth1 192.168.188.129 and node2 has eth0 192.168.1.11, eth1 192.168.188.130 | 02:15 |
| Pavo | I gave the VIP IP 192.168.1.90 | 02:15 |
| Pavo | in the /etc/kolla/globals.yml file | 02:15 |
| inc0 | so this seems ok | 02:18 |
| Pavo | I thought so to but as you can see from the inspect on the mariadb container there isn't an IP getting added | 02:18 |
| Pavo | unless I understand the deployment wrong | 02:19 |
| *** jtriley has quit IRC | 02:20 | |
| openstackgerrit | Wei Cao proposed openstack/kolla: Ubuntu with systemd support https://review.openstack.org/346900 | 02:21 |
| openstackgerrit | Wei Cao proposed openstack/kolla: Remove duplicate items https://review.openstack.org/355377 | 02:25 |
| inc0 | Pavo, that's because all of our containers are net=host | 02:26 |
| inc0 | mariadb inside will have exactly same networking stack as host | 02:26 |
| Pavo | then what IP should I give kolla_internal_vip_address? | 02:26 |
| Pavo | the host IP? | 02:26 |
| inc0 | no, not used IP:) | 02:27 |
| inc0 | it's for keepalived | 02:27 |
| Pavo | ok thats what I gave it | 02:27 |
| Pavo | a IP thats on the same network but not being used | 02:27 |
| inc0 | if your keepalived is running in container | 02:27 |
| inc0 | ip a should show IP you gave it | 02:27 |
| inc0 | ip a on host | 02:27 |
| duonghq | anybody has good news about the yesterday neutron_server error? | 02:31 |
| Pavo | I have kepalive on node2 and mariadb on node1 | 02:33 |
| Pavo | not seeing an IP for the keepalive either | 02:33 |
| inc0 | hmm, can you show me your inventory? | 02:34 |
| Pavo | https://www.dropbox.com/s/xkhanhb0ysq6gym/multinode?dl=0 | 02:34 |
| inc0 | can you paste it somewhere?;) | 02:36 |
| Pavo | sure | 02:36 |
| inc0 | thanks | 02:36 |
| inc0 | dropbox would want me to download it | 02:36 |
| inc0 | and as I know myself it would stay there forever | 02:36 |
| *** signed8bit_Zzz is now known as signed8bit | 02:37 | |
| Pavo | https://paste.pound-python.org/show/6XsAEDMTG6JSzinplXQC/ | 02:37 |
| inc0 | so 1.45 should have the IP | 02:38 |
| inc0 | network node should run haproxy container and keepalived container | 02:38 |
| Pavo | where do you see that? | 02:38 |
| inc0 | just ip addr on host | 02:38 |
| Pavo | oh ok | 02:38 |
| Pavo | so I have my inventory messed up | 02:38 |
| inc0 | well not that bad | 02:39 |
| inc0 | but I'd suggest putting network and control on same node | 02:39 |
| inc0 | for first deployments;) | 02:39 |
| Pavo | you mind helping me out, this is the first time I have tried this and don't understand it all | 02:39 |
| Pavo | oh ok | 02:39 |
| inc0 | so put control and network on .11, compute on .45 | 02:40 |
| inc0 | and storage whatever, unless you enable ceph/cinder | 02:40 |
| inc0 | I suggest starting without these | 02:40 |
| inc0 | as both require a bit of additional setup | 02:40 |
| Pavo | ok | 02:41 |
| Pavo | so I need to remove contol from [cinder:children? | 02:41 |
| inc0 | no | 02:41 |
| *** fragatin_ has joined #openstack-kolla | 02:42 | |
| inc0 | you specify enable_cinder: true in globals | 02:42 |
| inc0 | unless you set that, it should be ok | 02:42 |
| inc0 | just put control and network on .11 and compute on .45 | 02:42 |
| inc0 | or you can put control on both | 02:43 |
| Pavo | this is globals | 02:43 |
| Pavo | https://paste.pound-python.org/show/ikw04SvnaHQZDpmsp7wa/ | 02:43 |
| inc0 | just to see ha work | 02:43 |
| *** zhubingbing has joined #openstack-kolla | 02:43 | |
| zhubingbing | hello | 02:43 |
| inc0 | ok | 02:43 |
| inc0 | so I see the issue | 02:43 |
| Pavo | ? | 02:43 |
| inc0 | network_interface => management | 02:43 |
| inc0 | neutron_external_interface => where vm land | 02:44 |
| inc0 | so if you set up network_interface to eth1 | 02:44 |
| inc0 | you need to put kolla_internal_vim_address in same network as eth1 | 02:44 |
| inc0 | ok it seems you did that | 02:45 |
| Pavo | ok so use eth0 for both since my 192.168.1.0/24 network is being used for both management and internet then right | 02:45 |
| *** fragatina has quit IRC | 02:45 | |
| inc0 | no, eth0 will get owned by neutron | 02:45 |
| inc0 | which means you shouldn't use it for anything else really | 02:45 |
| Pavo | the 192.168.188.0/24 network is only used for communication between the nodes | 02:45 |
| inc0 | as neutron will put bridge there | 02:46 |
| Pavo | ah | 02:46 |
| inc0 | but that's later | 02:46 |
| *** klint has joined #openstack-kolla | 02:46 | |
| *** fragatin_ has quit IRC | 02:46 | |
| inc0 | keepalived will try to setup 192.168.188.254 on eth1 | 02:46 |
| inc0 | of controller | 02:46 |
| inc0 | besides usual address | 02:46 |
| irtermite | inc0: did you figure out your spanning tree issue? | 02:47 |
| inc0 | then haproxy will listen on this address for service traffic | 02:47 |
| inc0 | irtermite, not yet, I got pulled out | 02:47 |
| *** klint has quit IRC | 02:47 | |
| inc0 | will get back to it tomorrow | 02:47 |
| inc0 | after I repave whole thing, it seems to behave better | 02:48 |
| *** klint has joined #openstack-kolla | 02:48 | |
| inc0 | it might be something on our end | 02:48 |
| irtermite | from what I am told, it sounds like an issue within, yes | 02:48 |
| irtermite | was it traffic between hosts, vms, both? | 02:48 |
| irtermite | over VPN? | 02:48 |
| inc0 | we wanted to put vlan network for vms | 02:49 |
| *** yuanying has quit IRC | 02:49 | |
| inc0 | and add tagged iface on deploy host so we can access vms over vlan | 02:49 |
| inc0 | but I think we messed up something and put vlan in vlan | 02:49 |
| inc0 | anyway, going off | 02:50 |
| inc0 | enough of no-lifing in irc | 02:50 |
| *** jamesdenton has joined #openstack-kolla | 02:50 | |
| inc0 | cya all tomorrow | 02:50 |
| Pavo | thanks for the help inc0 | 02:51 |
| inc0 | np | 02:51 |
| *** yuanying has joined #openstack-kolla | 02:52 | |
| jamesdenton | hi irtermite | 02:53 |
| irtermite | inc0: ^^^ meet the guy that wrote the book on Openstack Neutron | 02:53 |
| irtermite | literally | 02:53 |
| irtermite | and go | 02:53 |
| irtermite | Michal -> James -> Michal | 02:54 |
| jamesdenton | hey inc0 | 02:54 |
| *** inc0 has quit IRC | 02:55 | |
| irtermite | *sadtrombone* | 02:55 |
| zhubingbing | Shaun Smekel | 02:56 |
| *** klint has quit IRC | 02:56 | |
| zhubingbing | hello | 02:56 |
| *** klint has joined #openstack-kolla | 02:57 | |
| zhubingbing | Are you there | 02:57 |
| *** klint has quit IRC | 02:58 | |
| *** klint has joined #openstack-kolla | 02:58 | |
| *** klint has quit IRC | 02:59 | |
| *** signed8bit is now known as signed8bit_Zzz | 03:00 | |
| *** rstarmer has joined #openstack-kolla | 03:00 | |
| openstackgerrit | Wei Cao proposed openstack/kolla: Ubuntu with systemd support https://review.openstack.org/346900 | 03:01 |
| openstackgerrit | Wei Cao proposed openstack/kolla: Ubuntu with systemd support https://review.openstack.org/346900 | 03:03 |
| *** signed8bit_Zzz is now known as signed8bit | 03:09 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add aodh role https://review.openstack.org/351027 | 03:16 |
| *** haplo37__ has joined #openstack-kolla | 03:22 | |
| *** signed8bit has quit IRC | 03:34 | |
| *** diogogmt has quit IRC | 03:34 | |
| *** signed8bit has joined #openstack-kolla | 03:34 | |
| *** signed8bit has quit IRC | 03:38 | |
| *** signed8b_ has joined #openstack-kolla | 03:39 | |
| *** yuanying has quit IRC | 03:48 | |
| *** yuanying has joined #openstack-kolla | 03:51 | |
| *** signed8bit has joined #openstack-kolla | 03:56 | |
| *** jamesdenton has quit IRC | 03:56 | |
| *** signed8b_ has quit IRC | 03:58 | |
| duonghq | when I checkout my git repository to a revision (and pip install...), and do kolla-build --push, does the repo get the version? | 04:13 |
| duonghq | Daviey: do you have any experience | 04:17 |
| *** fragatina has joined #openstack-kolla | 04:24 | |
| openstackgerrit | Tin Lam proposed openstack/kolla: Add support for building vmtp container https://review.openstack.org/354362 | 04:26 |
| *** fragatina has quit IRC | 04:29 | |
| *** hieulq has quit IRC | 04:39 | |
| *** Pavo_ has joined #openstack-kolla | 04:42 | |
| openstackgerrit | Wei Cao proposed openstack/kolla: Add solum container https://review.openstack.org/355408 | 04:43 |
| *** hieulq has joined #openstack-kolla | 04:45 | |
| openstackgerrit | Wei Cao proposed openstack/kolla: Update instructions for ubuntu with systemd https://review.openstack.org/346900 | 04:47 |
| openstackgerrit | Wei Cao proposed openstack/kolla: Remove duplicate items https://review.openstack.org/355377 | 04:50 |
| *** signed8bit has quit IRC | 04:52 | |
| *** zhubingbing has quit IRC | 04:56 | |
| Pavo_ | is there a place where I can see all the options that can be used in the globals.yml? | 05:06 |
| *** Pavo is now known as Guest55739 | 05:08 | |
| *** Pavo_ has quit IRC | 05:08 | |
| *** senk_ has joined #openstack-kolla | 05:16 | |
| *** zhubingbing has joined #openstack-kolla | 05:25 | |
| *** fragatina has joined #openstack-kolla | 05:40 | |
| *** haplo37__ has quit IRC | 05:41 | |
| *** zhubingbing has quit IRC | 05:44 | |
| *** senk_ has quit IRC | 05:45 | |
| *** fragatina has quit IRC | 05:45 | |
| *** zhubingbing has joined #openstack-kolla | 05:46 | |
| zhubingbing | Shaun Smekel | 06:12 |
| zhubingbing | hello | 06:12 |
| duonghq | zhubingbing: did you get any error related to neutron-l3-agent? | 06:13 |
| zhubingbing | netron-l3-agent？ | 06:13 |
| duonghq | iup | 06:13 |
| duonghq | yup | 06:13 |
| zhubingbing | NO， | 06:14 |
| zhubingbing | I'm not wrong for neutron-l3-agent | 06:15 |
| zhubingbing | What is your fault detail？ | 06:15 |
| duonghq | I got kolla_start not found or does not exist | 06:16 |
| zhubingbing | sure，you setting enable it? | 06:17 |
| duonghq | no, just default settings | 06:17 |
| zhubingbing | can you share VM？ | 06:19 |
| duonghq | just fresh install of ubuntu 16.04 | 06:19 |
| zhubingbing | ok | 06:19 |
| zhubingbing | you should run this command mount --make-shared /run | 06:20 |
| duonghq | ah, do you get any problem with neutron_server in master branch? | 06:20 |
| zhubingbing | http://docs.openstack.org/developer/kolla/quickstart.html | 06:21 |
| duonghq | I remembered that I ran this command | 06:22 |
| zhubingbing | the system have reboot？ | 06:22 |
| duonghq | yeah, but for 16.04, it is in systemd | 06:22 |
| zhubingbing | If you want to re execute this command after the reboot of the system | 06:23 |
| duonghq | so reboot doesn't affect | 06:23 |
| zhubingbing | ok | 06:23 |
| zhubingbing | sure | 06:23 |
| zhubingbing | then I don't know. | 06:23 |
| zhubingbing | - sorry | 06:23 |
| duonghq | which revision are you runnig zhubingbing? | 06:24 |
| zhubingbing | ubuntu 14.04 ，kolla is 3.0.0 | 06:25 |
| duonghq | I mean the git revision | 06:25 |
| zhubingbing | git | 06:26 |
| zhubingbing | git version 1.9.1 | 06:26 |
| duonghq | no, the commit id of kolla | 06:26 |
| zhubingbing | this my name zhubingbing the commit id | 06:27 |
| duonghq | so, you build image from latest kolla source code? | 06:29 |
| *** Jeffrey4l has joined #openstack-kolla | 06:30 | |
| *** unicell has joined #openstack-kolla | 06:30 | |
| zhubingbing | my vm is 52.196.228.195: | 06:30 |
| zhubingbing | you can ssh it | 06:30 |
| duonghq | you can cd in to the kolla source and exec git status? | 06:31 |
| zhubingbing | root@ip-172-31-22-78:~/kolla# git status | 06:32 |
| zhubingbing | On branch master | 06:32 |
| zhubingbing | Your branch is up-to-date with 'origin/master'. | 06:32 |
| zhubingbing | nothing to commit, working directory cle | 06:32 |
| duonghq | ah, ok | 06:32 |
| duonghq | but do you run kolla-build recently | 06:33 |
| zhubingbing | yes | 06:33 |
| duonghq | ok, thank you | 06:33 |
| zhubingbing | ok | 06:34 |
| zhubingbing | That's OK. | 06:34 |
| *** diga_ has joined #openstack-kolla | 06:41 | |
| *** b_bezak has joined #openstack-kolla | 07:06 | |
| *** rstarmer_ has quit IRC | 07:09 | |
| *** bootsha has joined #openstack-kolla | 07:21 | |
| *** imcsk8 is now known as imcsk8|zZz | 07:37 | |
| *** bjolo has joined #openstack-kolla | 07:41 | |
| *** fragatina has joined #openstack-kolla | 07:41 | |
| *** belmoreira has joined #openstack-kolla | 07:46 | |
| *** fragatina has quit IRC | 07:46 | |
| *** diga_ has left #openstack-kolla | 08:07 | |
| *** athomas has joined #openstack-kolla | 08:14 | |
| *** daneyon has joined #openstack-kolla | 08:16 | |
| *** Serlex has joined #openstack-kolla | 08:18 | |
| *** matrohon has joined #openstack-kolla | 08:19 | |
| *** daneyon has quit IRC | 08:21 | |
| duonghq | mntdevops: did you get kolla-start command not found anywhere? | 08:25 |
| duonghq | Client Error: Not Found (\"Container command \\'kolla_start\\' not found or does not exist.\")\\n' | 08:25 |
| openstackgerrit | Mathias proposed openstack/kolla: Added grafana container https://review.openstack.org/346447 | 08:35 |
| *** mewald has joined #openstack-kolla | 08:36 | |
| mewald | Do people still care about performance monitoring? The activity in reviewing is rather frustrating. I am waiting for https://review.openstack.org/#/c/346448/ It has no comments, no +1 nor -1 | 08:45 |
| patchbot | mewald: patch 346448 - kolla - Added telegraf role | 08:45 |
| *** cristicalin has joined #openstack-kolla | 08:53 | |
| duonghq | mewald: did you get command not found error when start container? | 08:58 |
| mewald | duonghq: which container are we talking about? | 09:03 |
| mewald | telegraf? | 09:03 |
| *** gfidente has joined #openstack-kolla | 09:03 | |
| *** bootsha has quit IRC | 09:04 | |
| duonghq | mewald: memcached | 09:04 |
| mewald | my patches aren't concerned with memcache at all - are we talking about the monitoring patches or is this a general question? | 09:05 |
| duonghq | ah, general question | 09:05 |
| duonghq | just master branch | 09:05 |
| mewald | ah, no I haven't seen anything like this | 09:05 |
| *** cristicalin has quit IRC | 09:06 | |
| duonghq | thanks | 09:06 |
| *** MrDan has joined #openstack-kolla | 09:06 | |
| *** bootsha has joined #openstack-kolla | 09:07 | |
| *** bootsha has quit IRC | 09:14 | |
| openstackgerrit | Mathias proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 | 09:14 |
| openstackgerrit | Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 | 09:16 |
| openstackgerrit | Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 | 09:17 |
| openstackgerrit | Paul Bourke proposed openstack/kolla: WIP: attempt to fix gate https://review.openstack.org/355808 | 09:20 |
| *** kwazar is now known as quasar | 09:23 | |
| *** rstarmer_ has joined #openstack-kolla | 09:26 | |
| openstackgerrit | Sajauddin proposed openstack/kolla: Author: Sajauddin https://review.openstack.org/351108 | 09:28 |
| *** rstarmer_ has quit IRC | 09:29 | |
| zhubingbing | hello | 09:32 |
| duonghq | hello | 09:40 |
| duonghq | hope that this time kolla will up and running | 09:40 |
| duonghq | It already toke me 2 day | 09:40 |
| mewald | crazy - first time I had it up and running after 20min :D | 09:41 |
| duonghq | mewald: me too | 09:41 |
| duonghq | but after some try it screw up | 09:41 |
| duonghq | no idea | 09:41 |
| *** fragatina has joined #openstack-kolla | 09:43 | |
| duonghq | mewald: do you try recent version on git? | 09:44 |
| mewald | yeah I am on master | 09:44 |
| mewald | I am building new images right now, I'll let you know if it worked | 09:45 |
| duonghq | thanks | 09:45 |
| *** egonzalez90 has joined #openstack-kolla | 09:46 | |
| *** fragatina has quit IRC | 09:48 | |
| *** bootsha has joined #openstack-kolla | 09:57 | |
| *** zhurong has quit IRC | 10:01 | |
| *** daneyon has joined #openstack-kolla | 10:05 | |
| *** daneyon has quit IRC | 10:09 | |
| egonzalez90 | Hi there! | 10:14 |
| MrDan | hello, I have an issue when i use kolla-ansible pull | 10:14 |
| duonghq | MrDan: can you give me more detail | 10:15 |
| MrDan | fails at pulling kolla-toolbox, but it is a code issue, not by failing to pull it | 10:15 |
| MrDan | An exception occurred during task execution. The full traceback is: | 10:15 |
| MrDan | Traceback (most recent call last): | 10:15 |
| MrDan | File "/tmp/ansible_kklOeD/ansible_module_kolla_docker.py", line 703, in <module> | 10:15 |
| MrDan | main() | 10:15 |
| MrDan | File "/tmp/ansible_kklOeD/ansible_module_kolla_docker.py", line 673, in main | 10:15 |
| MrDan | module = generate_nested_module() | 10:15 |
| MrDan | File "/tmp/ansible_kklOeD/ansible_module_kolla_docker.py", line 659, in generate_nested_module | 10:15 |
| MrDan | new_args.update(module._load_params()[0]) | 10:15 |
| MrDan | TypeError: 'NoneType' object has no attribute '__getitem__' | 10:16 |
| MrDan | fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_name": "kolla_docker"}, "module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible_kklOeD/ansible_module_kolla_docker.py\", line 703, in <module>\n main()\n File \"/tmp/ansible_kklOeD/ansible_module_kolla_docker.py\", line 673, in main\n module = generate_nested_module()\n File | 10:16 |
| MrDan | \"/tmp/ansible_kklOeD/ansible_module_kolla_docker.py\", line 659, in generate_nested_module\n new_args.update(module._load_params()[0])\nTypeError: 'NoneType' object has no attribute '__getitem__'\n", "module_stdout": "", "msg": "MODULE FAILURE", "parsed": false} | 10:16 |
| MrDan | kolla at 2.0.2 | 10:16 |
| duonghq | sound familar | 10:16 |
| openstackgerrit | Prithiv proposed openstack/kolla: Add Networking-SFC roles in ansible https://review.openstack.org/346842 | 10:22 |
| *** duonghq has quit IRC | 10:27 | |
| *** ntpttr has quit IRC | 10:30 | |
| rstarmer | wrong ansible versin | 10:32 |
| rstarmer | try with ansible 1.9.4 | 10:32 |
| rstarmer | MrDan: if you're using the master branch, you should be using ansible 2.0.0 if you're using stable/mitaka, ansible 1.9.4 | 10:33 |
| openstackgerrit | Sajauddin proposed openstack/kolla: Skips a debug log during image build https://review.openstack.org/351108 | 10:35 |
| *** ntpttr has joined #openstack-kolla | 10:35 | |
| *** kbaegis has joined #openstack-kolla | 10:41 | |
| *** fragatina has joined #openstack-kolla | 10:47 | |
| *** rhallisey has joined #openstack-kolla | 10:49 | |
| *** phuongnh has quit IRC | 10:49 | |
| *** dmueller-evoila has joined #openstack-kolla | 10:56 | |
| *** dmueller-evoila has quit IRC | 11:00 | |
| *** vbel has quit IRC | 11:01 | |
| *** vbel has joined #openstack-kolla | 11:01 | |
| *** ad_rien_ has joined #openstack-kolla | 11:02 | |
| *** dmsimard is now known as dmsimard|afk | 11:03 | |
| *** ad_rien_ has quit IRC | 11:06 | |
| *** bootsha has quit IRC | 11:07 | |
| MrDan | back, i was on lunch | 11:15 |
| MrDan | i am using the default build | 11:16 |
| MrDan | let me check it | 11:16 |
| MrDan | where I can find it, in /etc/kolla/global.yml? | 11:16 |
| MrDan | ok, nvm | 11:16 |
| *** bootsha has joined #openstack-kolla | 11:17 | |
| *** fragatin_ has joined #openstack-kolla | 11:19 | |
| openstackgerrit | Merged openstack/kolla: Customizations for Swift https://review.openstack.org/354405 | 11:20 |
| egonzalez90 | MrDan ansible --version | 11:20 |
| *** haplo37__ has joined #openstack-kolla | 11:20 | |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Fix Neutron bootstrap failure due to missing fixtures dependency https://review.openstack.org/355808 | 11:21 |
| *** fragatina has quit IRC | 11:23 | |
| MrDan | i am trying with 2.0.0.0 | 11:23 |
| MrDan | failed again, trying with 1.9.4 | 11:24 |
| egonzalez90 | as rstarmer said, with stable/mitaka ansible 1.9.4 is needed, migration to ansible 2 is being made at master | 11:29 |
| *** haplo37__ has quit IRC | 11:31 | |
| *** mbound has joined #openstack-kolla | 11:34 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Missing hosts at haproxy in site.yml https://review.openstack.org/355861 | 11:34 |
| *** sdake has joined #openstack-kolla | 11:36 | |
| *** bjolo has quit IRC | 11:36 | |
| MrDan | where can I set the openstack version, in /etc/kolla/globals.yml? | 11:38 |
| MrDan | or ar kolla build with an aoption | 11:38 |
| *** berendt has joined #openstack-kolla | 11:39 | |
| *** dwalsh has joined #openstack-kolla | 11:41 | |
| *** duonghq has joined #openstack-kolla | 11:42 | |
| sdake | morning cats | 11:44 |
| sdake | version 2.0.2 = mitaka | 11:44 |
| sdake | 1.1.2 = liberty | 11:44 |
| duonghq | mewald: did you finish building image? | 11:44 |
| sdake | openstaack_version: "2.0.2" | 11:44 |
| sdake | check out the empel in globals.yml | 11:44 |
| sdake | MrDan this will control which vrsion is deployed - not built | 11:46 |
| sdake | to control the built version you can use the --tag option | 11:46 |
| sdake | from the appropriate repositroy | 11:46 |
| duonghq | sdake: today is hard to me, building, deploying and failing :(, hope that tomorrow is better | 11:48 |
| sdake | duonghq sorryto hear it | 11:48 |
| sdake | need any help? | 11:48 |
| duonghq | at least, I found working commit at end of day | 11:49 |
| sdake | duonghq your probably rying to deploy master since your developing | 11:49 |
| sdake | master is not super stable atm | 11:49 |
| pbourke | sdake: hey | 11:49 |
| duonghq | something screw up my physical dev machine, so I use it only to build and deploy to VM | 11:49 |
| sdake | morning paul | 11:49 |
| sdake | pbourke any wordo n the networking situation in osic? | 11:50 |
| sdake | i went to bed early yesterday | 11:50 |
| sdake | after cratering my $1,800 cartridge | 11:50 |
| pbourke | sdake: was about to ask you the same :) Im a bit unclear on the status the past few days, some of the guys in other TZs have been working on it but not got an update | 11:50 |
| sdake | i couldn't bear to stay up any longer :( | 11:50 |
| sdake | pbourke - email in order i think to help ix that problem | 11:51 |
| pbourke | ok thanks I'll draft something up | 11:51 |
| pbourke | as right now time is slipping away and we're not progressing | 11:51 |
| pbourke | or maybe we are and I dont know | 11:51 |
| sdake | i agree totally blocked | 11:51 |
| sdake | i grilled inc0 yesterday about it so hee could get it working for us | 11:51 |
| sdake | as we have been blocked ofr a week | 11:51 |
| sdake | at this point i am almsot ready to abandon rally testing | 11:52 |
| pbourke | sdake: also gates are not good lately, I have two patches up to help fix | 11:52 |
| sdake | we can do the remainder of our testing without totally functional netowrking | 11:52 |
| sdake | pbourke cool thanks :) | 11:52 |
| pbourke | sdake: but there's something wrong with the keys for the package mirrors | 11:52 |
| sdake | which mirrors specifically | 11:52 |
| pbourke | the ubuntu xenial ones | 11:52 |
| kbaegis | I'm curious about this too | 11:53 |
| sdake | when the US wakes up i'll bring it up | 11:53 |
| duonghq | hey kbaegis | 11:53 |
| sdake | do you have evidence of such | 11:53 |
| kbaegis | morning! | 11:53 |
| *** daneyon has joined #openstack-kolla | 11:53 | |
| sdake | the best way to get that fixed would be an emil to openstack-infra | 11:53 |
| pbourke | have we always had xenial gates or are these new | 11:53 |
| sdake | they are new - jeffrey added them | 11:53 |
| pbourke | ok that makes more sense then | 11:53 |
| sdake | oh the keys in our tree! | 11:53 |
| sdake | not the mirrors | 11:53 |
| sdake | i know the problem | 11:53 |
| sdake | possibly | 11:54 |
| sdake | we don't inject the correct mirror information into the build | 11:54 |
| sdake | for xenial | 11:54 |
| sdake | give me 20 misn to get warmed up and i'll look at it | 11:54 |
| sdake | kbaegis are you ahving trouble with xenial as well? | 11:54 |
| pbourke | good stuff sdake thanks | 11:55 |
| sdake | pbourke after today - if no progress on networking, i think best courseof action is to abandon rally testing | 11:55 |
| sdake | or atleast the parts of it requring vm access | 11:56 |
| sdake | and continue on our merry way | 11:56 |
| pbourke | sdake: ok | 11:56 |
| sdake | we can't stay blocked and have zero output for the osic work for 4 weeks because we don't know how to operate our own software ;) | 11:56 |
| kbaegis | sdake Not per-se | 11:56 |
| sdake | fwiw - got this gem - stp is not enabledon the TORs, so when deploying do not enable stp in linuxbridge | 11:56 |
| sdake | otherwise you will get a stp storm | 11:57 |
| sdake | whatever all that means ;) | 11:57 |
| sdake | Jeffrey4l ^^ | 11:57 |
| sdake | sean-k-mooney was around yesterday - maye he has more data from what happened | 11:57 |
| sdake | i was on a 40 hour bender doing tonearm setup | 11:57 |
| *** daneyon has quit IRC | 11:58 | |
| duonghq | sdake: I thought you made it work before? | 11:58 |
| kbaegis | I can give folks access to my lab if you need it. | 11:58 |
| kbaegis | I think I have spreed around still :) | 11:58 |
| sdake | kbaegis thanks for the offer, but osic has granted us access to 131 nodes for scale testing | 11:58 |
| sdake | that is what we are talking about here | 11:59 |
| duonghq | kbaegis: finally, I found 1 working commit for Ubuntu host | 11:59 |
| kbaegis | sdake Oh now that's cool | 11:59 |
| sdake | pbourke also another gem - ping to the vms works, but ssh does not | 11:59 |
| kbaegis | sdake ooc, does ansible scale that effectively? I saw the appeal of headless mcollective/puppet | 12:00 |
| kbaegis | Curious how that's going | 12:00 |
| sdake | it takes 20 minutes to deploy 131 nodes | 12:00 |
| sdake | so I'd say so :) | 12:00 |
| pbourke | sdake: yeah it sounds messy | 12:00 |
| kbaegis | Wow, awesome | 12:00 |
| sdake | which images areyou guys tryign to ssh into btw | 12:00 |
| kbaegis | That's w/ 10GBASE links? | 12:01 |
| sdake | iirc cirros does not include sshd | 12:01 |
| pbourke | sdake: Id like to know where we think the problem is, if it's an osic thing we can file a ticket with them, if its an openstack thing should be plenty of people to help on the ML | 12:01 |
| sdake | 10gig yes | 12:01 |
| sdake | pbourke i am not sure on that point - i'm not sure if linuxbridge improved the situation compared to ovs | 12:01 |
| kbaegis | Makes sense. My labs still on 1gbase. 10 GBASE switches are expensive at 48 port densitites | 12:01 |
| sdake | pbourke i never go tthat feedback and we need to know that information sap | 12:01 |
| sdake | my personal lab has 10gig at home :) | 12:01 |
| pbourke | sdake: email sent to ML | 12:01 |
| kbaegis | How much did you have to drop on your switch? | 12:02 |
| sdake | 800 bucks | 12:02 |
| kbaegis | wtf? | 12:02 |
| sdake | need link? :) | 12:02 |
| kbaegis | Yes | 12:02 |
| sdake | i think its a really good value personally | 12:02 |
| sdake | https://www.amazon.com/NETGEAR-ProSAFE-XS708E-Ethernet-XS708E-100NES/dp/B00B46AEE6 | 12:03 |
| sdake | copper 10gig | 12:03 |
| sdake | requires cat7 cabling which is about 10-15 bucks a cable | 12:03 |
| kbaegis | Oh, 8 port | 12:03 |
| sdake | they make bigger ones | 12:03 |
| kbaegis | Yeah, | 12:03 |
| sdake | but more expensive | 12:03 |
| kbaegis | That's what I was saying | 12:03 |
| sdake | in retrospect i wish i bought thee 12 port | 12:03 |
| kbaegis | I have too many systems for an 8 port | 12:03 |
| kbaegis | I'd need 24/48 | 12:03 |
| kbaegis | And not ready to plunge in for 5k | 12:04 |
| sdake | still chepeast switch you can buy for the port count | 12:04 |
| sdake | cisco gear is far better perf/design | 12:04 |
| sdake | but the netgear stuff is functional for testing | 12:04 |
| kbaegis | Yeah, but at that density I may as well daisy chain | 12:04 |
| kbaegis | You're right, it's a good value | 12:04 |
| *** kbaegis has quit IRC | 12:08 | |
| *** dmueller-evoila has joined #openstack-kolla | 12:08 | |
| *** haplo37__ has joined #openstack-kolla | 12:09 | |
| *** kbaegis has joined #openstack-kolla | 12:09 | |
| kbaegis | I mention equationgroup and I get mysteriously dropped :) | 12:09 |
| sean-k-mooney | sdake: you mentioned me | 12:12 |
| sean-k-mooney | osic? | 12:12 |
| sdake | roger | 12:13 |
| sdake | pbourke would like an update if you know anything :) | 12:13 |
| *** haplo37__ has quit IRC | 12:13 | |
| sean-k-mooney | i was around for a few hours but when i left the issue was still not resolve. | 12:14 |
| sean-k-mooney | we were still seeing some stp traffic on the node. | 12:14 |
| *** srwilkers has joined #openstack-kolla | 12:15 | |
| sean-k-mooney | inc0 tried a few things | 12:15 |
| sean-k-mooney | im not sure if he found a way to fix it | 12:15 |
| sean-k-mooney | we tried simplying the network config by using just bond 0 for neutron and sending all other traffic over a single interface but still had issues | 12:16 |
| sean-k-mooney | inc0 was able to ping the spawned vm but there was a lot of packet loss still | 12:16 |
| sean-k-mooney | unfortunetlly i dont know what the outcome was so i guess we will have to ask inc0 when he comes online | 12:21 |
| kbaegis | You guys were using linuxbridge at scale? | 12:24 |
| sean-k-mooney | ovs | 12:24 |
| kbaegis | Oh, good :) | 12:24 |
| sean-k-mooney | linux brige is a fallback if we can sort out the networking issue we are having with osic but i dont think they are ovs related | 12:25 |
| kbaegis | lot of components created. Docker relies on a bridge doesn't it? (I'm newer to docker than openstack) | 12:25 |
| kbaegis | If there's a way to have docker terminate its veths on ovs, I'd be all ears on how | 12:26 |
| sean-k-mooney | kbaegis: we use --net=host for our container so they do not use ovs | 12:28 |
| sean-k-mooney | ovs is used by the vms spawned by openstack | 12:28 |
| sean-k-mooney | but yes you can use docker with ovs also | 12:29 |
| *** b_bezak_ has joined #openstack-kolla | 12:29 | |
| sean-k-mooney | i just not sure if docker supprot adding the port to ovs itself. | 12:30 |
| sdake | thanks sean-k-mooney | 12:32 |
| sdake | pbourke ^^ - i think there is nothign new there that i didn't already explain | 12:32 |
| *** b_bezak has quit IRC | 12:33 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Missing hosts at haproxy in site.yml https://review.openstack.org/355861 | 12:34 |
| kbaegis | Ha! Kolla deployed from centos-source:3.0.0 on gentoo! Now to see if I can actually create VMs this time. | 12:39 |
| sdake | sean-k-mooney iiuc this is what kuryr is all about | 12:43 |
| sdake | but huikang wouldl know for certain | 12:43 |
| sdake | (the ovs networkign link to docker0 | 12:43 |
| sdake | docker0 | 12:43 |
| sdake | docker) | 12:43 |
| sdake | keyboard fail | 12:44 |
| sean-k-mooney | kbaegis: interesting... if it has systemd it should work i guess but that is definetly not a tested combination | 12:44 |
| sdake | sean-k-mooney - yup i warned him :) | 12:44 |
| sdake | also not a supported combination | 12:44 |
| sdake | we expect the container distro to match the host os | 12:45 |
| sean-k-mooney | sdake: kuryr likns docker networks with neutron so with out the ovs neutron agent to wire up the openflow rules im not sure if it will work | 12:45 |
| sdake | sean-k-mooney cool thanks for the clarification and enlightenment :) | 12:45 |
| zhubingbing | hello | 12:45 |
| sdake | we have kuryr on the way | 12:46 |
| sdake | waiting on keystone v3 support | 12:46 |
| sean-k-mooney | kuryr with kolla | 12:46 |
| sean-k-mooney | i think i remember reviewing that at some point | 12:46 |
| *** ad_rien_ has joined #openstack-kolla | 12:46 | |
| kbaegis | sean-k-mooney: Why systemd? I can't use sysvinit? | 12:46 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Missing hosts at haproxy in site.yml https://review.openstack.org/355861 | 12:46 |
| sean-k-mooney | i was a little uncomfortable about the docker on the host useing someting form a docker container it was running | 12:47 |
| sean-k-mooney | kbaegis: you can but we have had issues in the past using a container os that expect systemd with a host os that does not use it and vice versa | 12:47 |
| *** mbound has quit IRC | 12:48 | |
| sean-k-mooney | kbaegis: there can also be issue when one distro uses apparmor and the other uses selinux | 12:48 |
| kbaegis | I can recompile or change the security model | 12:48 |
| sdake | turn off is probably the only answer there | 12:48 |
| kbaegis | I can even alter my init. I just like sysv better :) | 12:48 |
| sdake | (turn off security) | 12:49 |
| kbaegis | I'll play with it. Thanks for the warning | 12:49 |
| sean-k-mooney | kbaegis: well the issue is the patch for the profiels dont match the patch for the same apllication in side the contaienr | 12:49 |
| sdake | and also syscall interface + rtnetllink + ioctl = not compatible betwen distros | 12:49 |
| kbaegis | Interesting | 12:50 |
| kbaegis | Well I do know that horizon, as deployed, is giving me 500s | 12:50 |
| sean-k-mooney | sdake: there are not completely compatible because the tools are built against different kernel headers but somehow they do kind of work | 12:50 |
| kbaegis | I was falling back to the cli | 12:50 |
| sdake | sean-k-mooney right - sort of work being the key thing there ;) | 12:51 |
| sdake | mostly things work | 12:51 |
| sean-k-mooney | sdake: at leaset when the kernels they are expecting are somewhat close to the kernel you are running | 12:51 |
| sdake | but can't be guaranteed | 12:51 |
| sean-k-mooney | ya | 12:51 |
| sdake | roger on kernel versions | 12:51 |
| sean-k-mooney | kbaegis: in any case if you do get it working let us know im sure it will be interesting testing it out | 12:52 |
| kbaegis | 4.8.0-rc1 :) | 12:52 |
| kbaegis | Yeah, it's a good way to learn | 12:52 |
| kbaegis | I think | 12:52 |
| sean-k-mooney | kbaegis: that might be a little newer then the "3.10" kernel Centos is expecting | 12:53 |
| sean-k-mooney | the centos kernel isnt really 3.10 as it has backports from 4.x but who am i to argure with there nameing scheme | 12:53 |
| kbaegis | I guess I could try my hand at setting up the containers on gentoo | 12:54 |
| kbaegis | Square block, round hole | 12:55 |
| kbaegis | But it would allow you guys to test with the bleeding edge | 12:55 |
| kbaegis | So I've seen this horizon error before: Dashboard with slug "developer" is not registered | 12:58 |
| sdake | kbaegis our testing is in teh gate - getting new images in the gate is exceedingly difficult | 12:58 |
| sdake | kbaegis create a _member_ role | 12:58 |
| sdake | I was pretty sure this was fixed - perhaps its not merged yet | 12:58 |
| kbaegis | That fixes horizon? | 12:59 |
| sdake | yes | 12:59 |
| kbaegis | I just created the role from cli | 12:59 |
| kbaegis | Maybe it's a cache issue | 12:59 |
| sdake | Jeffrey4l is the patch author | 12:59 |
| sdake | its possible you found a new problem | 13:00 |
| sdake | nto certain - i don't recall seeing that specific slug | 13:00 |
| egonzalez90 | was not merged yet https://review.openstack.org/#/c/349198/ | 13:00 |
| patchbot | egonzalez90: patch 349198 - kolla - Create the _member_ role in the horizon role | 13:00 |
| kbaegis | http://hastebin.com/iveweruwex.rb | 13:00 |
| kbaegis | I'll try restarting the container | 13:00 |
| sdake | kbaegis cherry-pick that patch on your repo | 13:01 |
| *** ad_rien_ has quit IRC | 13:01 | |
| sdake | you may need to rebuild horizon - not sure | 13:01 |
| sdake | thanks egonzalez90 ! | 13:01 |
| Jeffrey4l | kbaegis, you also need https://review.openstack.org/349197 | 13:01 |
| Jeffrey4l | run deploy again will work. no need remove the container. | 13:02 |
| sdake | pbourke can you review https://review.openstack.org/#/c/349197/ plz | 13:03 |
| patchbot | sdake: patch 349197 - kolla - Implement kolla_keystone_role module | 13:03 |
| *** bjolo has joined #openstack-kolla | 13:04 | |
| kbaegis | Are there any docs that show how gate works? | 13:04 |
| sdake | its all in our heads :) | 13:05 |
| sdake | there are some docs | 13:05 |
| sdake | the best place to get a primer is in the #openstack-infra channel | 13:05 |
| sdake | i can introduce you there if you like | 13:05 |
| pbourke | Jeffrey4l: can we not use ansible 2.1 in toolbox on the stable branch? | 13:05 |
| sdake | although its a bit early - many folks may not be around | 13:05 |
| pbourke | Jeffrey4l: in fact I thought we did already | 13:05 |
| Jeffrey4l | pbourke, we are using ansible>2 in toolbox. | 13:06 |
| pbourke | Jeffrey4l: why not update the pin | 13:06 |
| kbaegis | Neat | 13:06 |
| Jeffrey4l | i.e. we are using the latest version. | 13:06 |
| sdake | pbourke good question | 13:06 |
| *** dmueller-evoila has quit IRC | 13:06 | |
| Jeffrey4l | why not? | 13:06 |
| pbourke | why not use ansible 2.1 in toolbox on the stable branch | 13:06 |
| sdake | that does make ssense to me | 13:07 |
| Jeffrey4l | hmm. stable branch? i have no idea about that :( | 13:07 |
| sdake | what we hae now in the stable versions is essentially a beta version of fansible | 13:07 |
| Jeffrey4l | when i joined kolla. it use 2.1 already. | 13:07 |
| Jeffrey4l | a rc tag. | 13:07 |
| sdake | (just speaking about kolla-toolbox) | 13:07 |
| sdake | oh maybe someone fixed that | 13:08 |
| Jeffrey4l | OK. it is me change it from installing from source to installing from pypi ( pip install ansible>2). | 13:10 |
| Jeffrey4l | But you are right, we should fix it version number. like pip install ansible<2.1 or ansible<=2.1 | 13:10 |
| Jeffrey4l | it will be more stable. pbourke | 13:10 |
| kbaegis | well, the vm creation works | 13:11 |
| pbourke | Jeffrey4l: this is what's on stable/mitaka right now https://github.com/openstack/kolla/blob/stable/mitaka/docker/kolla-toolbox/Dockerfile.j2#L43-L47 | 13:11 |
| sdake | ya - thats a beta version :) | 13:12 |
| sdake | an ansible engineer told us "use this git commit for now" | 13:12 |
| sdake | an we never got back to it | 13:12 |
| sdake | but it does work | 13:12 |
| Jeffrey4l | pbourke, yes. ansible>2 is bad. | 13:13 |
| sdake | but it may notwork with your patches backported Jeffrey4l | 13:13 |
| sdake | Jeffrey4l that is a 2.x beta hash | 13:13 |
| sdake | Jeffrey4l you mean ansible < 2? | 13:13 |
| Jeffrey4l | beta hash? what does it mean? | 13:14 |
| Jeffrey4l | ansible 2.1 is beta version and shouldn't be used? | 13:14 |
| sdake | the git commit hash in that link from pbourke is 2.x beta | 13:14 |
| pbourke | i know | 13:14 |
| sdake | i.e. we are using beta software in stable branches for kolla-toolbox | 13:14 |
| Jeffrey4l | it is. | 13:15 |
| sdake | i think pbourke 's question is why not just unpin it or pin it to a non-beta version | 13:15 |
| sdake | or maybe pbourke has something else in mind ? | 13:15 |
| pbourke | no that's what I meant sdake | 13:15 |
| sdake | sorry - i'll gtfootw :) | 13:16 |
| pbourke | im saying you are correct :) | 13:16 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Missing hosts at haproxy in site.yml https://review.openstack.org/355861 | 13:16 |
| Jeffrey4l | so you mean: we should move the ansible 2.1 on in kolla-toolbox on mitaka branch. right? | 13:16 |
| sdake | oh somehow i read "not" in there | 13:16 |
| sdake | Jeffrey4l yes | 13:16 |
| sdake | Jeffrey4l so your backports work properly | 13:17 |
| Jeffrey4l | I think it is OK. | 13:17 |
| sdake | xenial hashes | 13:17 |
| sdake | Jeffrey4l did you do the work o fadding xenail to kolla? | 13:18 |
| Jeffrey4l | to move to ansible 2.1. using a hash commit tag is really bad. | 13:18 |
| sdake | Jeffrey4l it was the best option we had available at th time | 13:18 |
| sdake | we have better options now | 13:18 |
| sdake | it took alot of hunting to find a git commit that would work at all with shade in the early days of ansible 2.x | 13:19 |
| Jeffrey4l | sdake, i fix the xenial build gate. the deploy gate is still failing, which is wired. | 13:19 |
| Jeffrey4l | https://review.openstack.org/355274 | 13:19 |
| sdake | which is wired? | 13:19 |
| sdake | weird? | 13:19 |
| *** jtriley has joined #openstack-kolla | 13:20 | |
| sdake | agree - its odd | 13:20 |
| sdake | wtb solution to deploy gate | 13:20 |
| sdake | pbourke looks like a review is outstanding for xenial building | 13:20 |
| Jeffrey4l | ya weird. | 13:20 |
| Jeffrey4l | here : http://logs.openstack.org/74/355274/1/check/gate-kolla-dsvm-deploy-ubuntu-source-ubuntu-xenial-nv/2fc5b3c/console.html#_2016-08-15_02_33_08_726480 | 13:20 |
| Jeffrey4l | \"msg\": \"the python mysqldb module is required\"" | 13:20 |
| *** imcsk8|zZz is now known as imcsk8 | 13:21 | |
| *** zhurong has joined #openstack-kolla | 13:21 | |
| Jeffrey4l | the kolla-toolbox has this module( installing by pip ). But it still complain the python-mysqldb do not exist. | 13:21 |
| sdake | that has to do with the mixing of source and binary installs iirc | 13:22 |
| Jeffrey4l | in another patchset, i install the python-mysqldb by using `apt-get install python-mysqldb`, then it works. weird. | 13:22 |
| sdake | kolla-toolbox should be source only - no binary at all | 13:23 |
| sdake | in other words - based upon -base, rather then openstack-base | 13:23 |
| Jeffrey4l | yes. | 13:23 |
| Jeffrey4l | it installed the latest pip. | 13:23 |
| sdake | but that doesn't work without alot of effort | 13:23 |
| sdake | what we really need is a new base openstack-source-base | 13:23 |
| sdake | (I think but not 100% certain:) | 13:23 |
| Jeffrey4l | move the ansible base to openstack-source-base? | 13:24 |
| Jeffrey4l | why? | 13:24 |
| sdake | Jeffrey4l no that isn't what i mean | 13:24 |
| sdake | kolla-toolbox inherits from openstack-base | 13:24 |
| sdake | openstack-base no mattr how you dice it, installs packges either via rpm or apt | 13:24 |
| sdake | this is incompiatble with source-only containers | 13:25 |
| sdake | installing with yum then later with pip resultss in teh mysql behavior you see | 13:25 |
| Jeffrey4l | it inherits from base | 13:25 |
| Jeffrey4l | FROM {{ namespace }}/{{ image_prefix }}base:{{ tag }} | 13:25 |
| Jeffrey4l | https://github.com/openstack/kolla/blob/master/docker/kolla-toolbox/Dockerfile.j2#L1 | 13:25 |
| sdake | oh | 13:25 |
| Jeffrey4l | not openstack-base | 13:25 |
| sdake | maybe i didn't get enough sleep ;) | 13:25 |
| *** dmueller-evoila has joined #openstack-kolla | 13:25 | |
| Jeffrey4l | :p | 13:25 |
| sdake | Jeffrey4l cool thanks for the clarification - pretty sure that was not always the case | 13:26 |
| Jeffrey4l | we installed MySQL-python https://github.com/openstack/kolla/blob/master/docker/kolla-toolbox/Dockerfile.j2#L49 | 13:26 |
| Jeffrey4l | but it complain still. whereas, when i tested it locally, it can not be reproduced. :( | 13:27 |
| kbaegis | Feel dumb for asking, but how does one git-cherry pick when those aren't in the master branch? | 13:27 |
| sdake | upper right hand corner | 13:27 |
| sdake | click download | 13:27 |
| sdake | one of hte options is cherry-pick | 13:27 |
| sdake | (upper right hand corner of gerrit) | 13:28 |
| kbaegis | I'm using git from cli | 13:28 |
| sdake | clic cherry pick - itwil copy to clipboard | 13:28 |
| sdake | paste it | 13:28 |
| kbaegis | Oh | 13:28 |
| kbaegis | Neat | 13:28 |
| sdake | it will cherry-pick from gerrit | 13:28 |
| kbaegis | ty :) | 13:28 |
| sdake | it/git | 13:28 |
| Jeffrey4l | kbaegis, sdake please use checkout if you do not familiar with git. | 13:28 |
| sdake | yup | 13:28 |
| Jeffrey4l | cherry-pick is more tricky and powerful :) | 13:29 |
| *** ad_rien_ has joined #openstack-kolla | 13:29 | |
| sdake | i like cherry-pick | 13:29 |
| kbaegis | I'm sort of familiar with git. I use pull/reset/clean all the time | 13:30 |
| kbaegis | I'm not much of a developer though | 13:30 |
| Jeffrey4l | yes. cherry-pick is the best solution. | 13:30 |
| Jeffrey4l | but you need cherry-pick two SP. | 13:31 |
| Jeffrey4l | SP/PS | 13:31 |
| kbaegis | SP? | 13:33 |
| *** diogogmt has joined #openstack-kolla | 13:33 | |
| kbaegis | I did both commits | 13:33 |
| *** inc0 has joined #openstack-kolla | 13:35 | |
| sdake | pbourke inc0 has arrived | 13:36 |
| inc0 | "carrier has arrived" | 13:36 |
| inc0 | wassup | 13:36 |
| inc0 | I played with this stuff yesterday | 13:37 |
| inc0 | repaved for ovs, deployed flat network | 13:37 |
| inc0 | infra networking is ok | 13:37 |
| inc0 | for some reason neutron added tagged iface | 13:37 |
| inc0 | to flat network | 13:37 |
| sdake | inc0 so the adding of tagged iface is the problem? | 13:38 |
| kbaegis | Sharded interfaces will break ovs | 13:38 |
| inc0 | no, vm doesn't have dhcp | 13:41 |
| kbaegis | Well, I cherry picked those commits and horizon is still 500'ing after redeploy | 13:42 |
| pbourke | smekel_: ping | 13:42 |
| *** diogogmt has quit IRC | 13:42 | |
| sbezverk | sdake: have you manage to get docker to use directly LVM instead of loop device? | 13:42 |
| sdake | sbezverk i always setup for btrfs so no | 13:43 |
| *** zhurong has quit IRC | 13:43 | |
| sbezverk | I see some issues with using loop on my kube cluster | 13:44 |
| sbezverk | sdake: ok will google it | 13:44 |
| *** zhurong has joined #openstack-kolla | 13:44 | |
| *** MrDan has quit IRC | 13:44 | |
| Jeffrey4l | i think the root cause should be the physical swither. | 13:44 |
| Jeffrey4l | switcher. | 13:44 |
| sdake | Jeffrey4l we dont have access to the tor | 13:45 |
| sdake | Jeffrey4l by we, i mean osic in general can't modify settings | 13:45 |
| Jeffrey4l | i set up a ovs bridge and add a vlan port on it manual. when ping the gateway, it has lots of loss. | 13:45 |
| Jeffrey4l | OK. | 13:45 |
| kbaegis | Rebuilding horizon. Just for kicks | 13:46 |
| sdake | ok well t-1 day until lwe abandon vm related testing with any of our tools | 13:46 |
| Jeffrey4l | btw, the centos gate failed due to this neutron bug: https://review.openstack.org/#/c/355185/ | 13:48 |
| patchbot | Jeffrey4l: patch 355185 - neutron - Fixed neutron-db-manage without neutron/tests inst... (MERGED) | 13:48 |
| Jeffrey4l | and it is merged already. So you can recheck you PS> | 13:48 |
| *** mbound has joined #openstack-kolla | 13:48 | |
| pbourke | inc0: do you know if its a kolla problem or something else? | 13:48 |
| sdake | Jeffrey4l i did recheck it | 13:49 |
| Jeffrey4l | is it passed? | 13:49 |
| sdake | in t-1 day, we need to take a look at abaonding any vm related tests in rally and just focus on non-network tests | 13:49 |
| sdake | Jeffrey4l not sure - i just did it a bout 1 hour ago for that patch you linked that needs merging | 13:49 |
| Jeffrey4l | OK. thanks. | 13:49 |
| sdake | because otherwise we will miss ALL data | 13:50 |
| sdake | some data is better then none | 13:50 |
| sdake | but lets try to spend this last day sorting out this problem | 13:50 |
| pbourke | Jeffrey4l: sdake: can you guys review https://review.openstack.org/#/c/355808/ | 13:50 |
| patchbot | pbourke: patch 355808 - kolla - Fix Neutron bootstrap failure due to missing fixtu... | 13:50 |
| sdake | so we can get vm related performance data out of rally | 13:50 |
| sdake | pbourke on it | 13:50 |
| pbourke | with that + the bandit + Jeffrey4l's patch all gates should be green again | 13:50 |
| Jeffrey4l | no... deploy ubuntu source is still buggy. | 13:50 |
| Jeffrey4l | about this https://review.openstack.org/349197 | 13:51 |
| Jeffrey4l | pbourke, you want to use ansible side keystone_role ? ^^ | 13:51 |
| pbourke | Jeffrey4l: unless you see a reason not to | 13:52 |
| sdake | pbourke we are using our own modules to decouple kolla from the ansible release cycle | 13:52 |
| sdake | or lack thereof ;) | 13:52 |
| Jeffrey4l | we are using lost of our side ansible module. depend on the ansible project is not stable. | 13:52 |
| sdake | no 1.8.3 disaster in the future | 13:52 |
| pbourke | we only do that for kolla_docker | 13:53 |
| pbourke | we can't do that for all modules | 13:53 |
| Jeffrey4l | pbourke, no | 13:53 |
| Jeffrey4l | kolla_keystone_service | 13:53 |
| sdake | the reason its targetedd at master is agreement between Jeffrey4l and i to do the member role adding work incrementally | 13:53 |
| inc0 | sdake, t01 day? | 13:53 |
| inc0 | we still have 2 weeks | 13:53 |
| *** mbound has quit IRC | 13:53 | |
| Jeffrey4l | kolla_keystone_user | 13:53 |
| pbourke | sdake: Jeffrey4l: this role is in ansible 2.1 right? | 13:53 |
| sdake | inc0 what i mean is we will abandon vm related networking tests in 1 day - to unblock our other work | 13:53 |
| inc0 | ok | 13:54 |
| inc0 | we don't need tpo | 13:54 |
| Jeffrey4l | it is. | 13:54 |
| pbourke | inc0: can you please summarise the issue? I've lost track of it :( | 13:54 |
| inc0 | worst case scenerio I'll hack our way into network ns | 13:54 |
| sdake | i hate needless wheel reinvention, but in this case it is needful | 13:54 |
| inc0 | hold on will log onto tmux | 13:54 |
| pbourke | Jeffrey4l: so just pip install ansible==2.1 on the stable branches | 13:54 |
| Jeffrey4l | But we need upgrade the kolla-tools's ansible version to 2.1 in a stable version. | 13:54 |
| sdake | the need is to decouple kolla from the ansible release cycle | 13:54 |
| pbourke | we can't do that for all modules | 13:55 |
| *** haplo37__ has joined #openstack-kolla | 13:55 | |
| sdake | unless folks feel this module is stable enough | 13:55 |
| pbourke | we've had no trouble with this module im aware of | 13:55 |
| sdake | ok wfm then | 13:55 |
| Jeffrey4l | let's me check the ansible's code. | 13:55 |
| sdake | Jeffrey4l ya - give it a good review and make sure its up to snuff :) | 13:55 |
| sdake | thanks | 13:55 |
| pbourke | maybe im missing something here | 13:55 |
| inc0 | pbourke, come to tmux | 13:56 |
| sdake | pbourke iiuc we have never used the ansible module before | 13:56 |
| sdake | pbourke gets yanked around ;) | 13:56 |
| Jeffrey4l | btw, i remove all the container in osic inc0 | 13:56 |
| Jeffrey4l | when try to fix the network issue. | 13:56 |
| inc0 | well, they're there, looking at ovs now;) | 13:56 |
| inc0 | I repaved | 13:56 |
| Jeffrey4l | cool | 13:56 |
| pbourke | inc0: so correct me if im wrong, it seems to be a neutron config issue | 14:00 |
| *** b_bezak_ has quit IRC | 14:00 | |
| inc0 | Jeffrey4l, btw I think we messed up interfaces in previous conf | 14:00 |
| inc0 | pbourke, correct | 14:00 |
| *** b_bezak has joined #openstack-kolla | 14:00 | |
| Jeffrey4l | inc0, does it work now? | 14:02 |
| Jeffrey4l | i am i | 14:02 |
| Jeffrey4l | i am coming | 14:02 |
| inc0 | well, we get all traffic through cables | 14:02 |
| inc0 | it's ovs issue now | 14:02 |
| *** jamesdenton has joined #openstack-kolla | 14:03 | |
| *** ad_rien_ has quit IRC | 14:04 | |
| *** Guest55739 is now known as Pavo | 14:05 | |
| Pavo | I think I finally might have figured this out | 14:05 |
| pbourke | inc0: im not good with this stuff, but is there any point in creating the provider network with type vlan | 14:05 |
| inc0 | if you want multiple of them, yes | 14:05 |
| *** zhurong has quit IRC | 14:06 | |
| Pavo | also if you want some provider vlans to interact with some services that you host in Openstack | 14:06 |
| *** zhurong has joined #openstack-kolla | 14:06 | |
| Jeffrey4l | we need make it work first with any configuration. | 14:07 |
| Jeffrey4l | now, in the dhcp namespace, we can not ping the gateway. here is the root issue. | 14:08 |
| sdake | i hate that problem | 14:08 |
| sdake | it usually means sthere is a lack of route | 14:09 |
| *** bootsha has quit IRC | 14:09 | |
| sdake | i.e. network config problem | 14:09 |
| sdake | in the hardware.. | 14:09 |
| sdake | but keep at it guys :) | 14:09 |
| sdake | elrog 845s on the way -> http://www.6moons.com/industryfeatures/elrog/1.html | 14:12 |
| sdake | yay :) | 14:12 |
| Pavo | how long does it normally take for neutron-server container to start? | 14:12 |
| Pavo | during deployment | 14:12 |
| sdake | pavo it should be millseconds | 14:12 |
| sdake | docker ps -a | 14:12 |
| Pavo | hmmm, been setting here for about 5 minutes now | 14:12 |
| sdake | if you see something in restarting state, thats bad | 14:12 |
| sdake | could you expandon "sitting here" | 14:13 |
| kbaegis | Hmm | 14:13 |
| sdake | you mean ansible is blocked? | 14:13 |
| kbaegis | Rebuilt horizon. It's still not working, even after cherry picks | 14:13 |
| sdake | kbaegis syptoms of your failure are what? | 14:13 |
| Jeffrey4l | kbaegis, hmm you need rebuild kolla-toolbox, tool | 14:13 |
| Jeffrey4l | the re-deploy | 14:14 |
| sdake | kbaegis my guess is you would need to cleanu pand deploy | 14:14 |
| kbaegis | 500 is the symptom | 14:14 |
| kbaegis | Okay | 14:14 |
| sdake | rather then simply redeploy | 14:14 |
| Pavo | been setting at this state for about 5 minutes sd | 14:14 |
| sdake | after applying jeffrey's suggestion of a toolbox rebuild | 14:14 |
| Pavo | sdake | 14:14 |
| Pavo | TASK [neutron : Starting neutron-server container] ***************************** | 14:14 |
| Pavo | skipping: [node2] | 14:14 |
| sdake | how many nodes do you have? | 14:14 |
| sdake | pavo ^ | 14:14 |
| *** bjolo has quit IRC | 14:15 | |
| *** narasimha_SV has joined #openstack-kolla | 14:15 | |
| Pavo | 2 | 14:15 |
| sdake | what is the network node? | 14:15 |
| Pavo | deployer plus 2 target nodes | 14:15 |
| sdake | what is the name of the network node | 14:15 |
| narasimha_SV | I am getting this error for openvswitch agent when I check the logs of the container | 14:15 |
| narasimha_SV | ]# docker logs 4711df58d518 INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json ERROR:__main__:Invalid json file found at /var/lib/kolla/config_files/config.json | 14:15 |
| Pavo | node1 | 14:16 |
| sdake | narasimha_SV are you using kolla's json file or one you created? | 14:16 |
| sdake | can the deployer ssh into node1? | 14:16 |
| sdake | from whatever user context your running kolla-ansible deploy? | 14:16 |
| Pavo | not currently | 14:16 |
| Pavo | I think it building the bridges now | 14:17 |
| sdake | well thats a problem | 14:17 |
| Pavo | unless I have to do that manually | 14:17 |
| sdake | your nuetron network must be a seprate network interface | 14:17 |
| narasimha_SV | sdake: kolla's json file ?? I didn't understand about this | 14:17 |
| sdake | the management network and neutron network cannot be shared | 14:17 |
| sdake | what is happening is your sharing the neutron and managmeenet network in some way, and the bridge setup craters connectivity to node1 | 14:18 |
| sdake | narasimha_SV are you working on magnum integration or are you deploying directlly from kolla? | 14:18 |
| narasimha_SV | I am using mitaka stable branch of kolla | 14:18 |
| narasimha_SV | i am deploying directly from kolla | 14:18 |
| narasimha_SV | magnum is not involved in this | 14:18 |
| sdake | narasimha_SV - we have an integration effort happenign with upstream magnum is why i asked | 14:19 |
| Pavo | this is my globals https://paste.pound-python.org/show/C1jADxwasIZEmSF3jmJZ/ | 14:19 |
| sdake | narasimha_SV i'm a bit stumped on that one - can you paste the json in /var/lib/kolla/config-files? | 14:19 |
| narasimha_SV | sdake: fine . is the any problem with any configuration which I placed | 14:20 |
| Pavo | this is my inventory https://paste.pound-python.org/show/54zkBzjsPwqKclkI7dzz/ | 14:20 |
| sdake | pavo: neutron_external_interface: "eth0" | 14:20 |
| sdake | this needs to be a unique interface | 14:20 |
| Pavo | so target host need 3 nics? | 14:20 |
| sdake | 2 nics | 14:20 |
| Pavo | I have 2 nics | 14:20 |
| sdake | it is possible you ahve eth1 and eth0 switched around | 14:20 |
| narasimha_SV | sdake: # ls -l /var/lib/kolla/ total 0 drwxr-xr-x 1 root root 10 Aug 16 10:16 usr | 14:21 |
| Pavo | let draw a network diagram really quick | 14:21 |
| narasimha_SV | apart from usr folder i dont see any other files in /var/lib/kolla | 14:21 |
| pbourke | inc0: Jeffrey4l: can we not create flat networks | 14:21 |
| sdake | narasimha_SV run ansible --version | 14:21 |
| Jeffrey4l | pbourke, we can. But i tried that. not work | 14:22 |
| pbourke | same problem? | 14:22 |
| sdake | narasimha_SV it should report 1.9.4 | 14:22 |
| Jeffrey4l | pbourke, yep | 14:22 |
| Jeffrey4l | linuxbridge should has the same issue, right? inc0 ? | 14:22 |
| narasimha_SV | sdake: ansible 1.9.6 | 14:23 |
| sdake | narasimha_SV run pip show jinja2 | 14:23 |
| narasimha_SV | I am seeing all these files in /etc/kolla folder that config.json | 14:23 |
| narasimha_SV | http://paste.openstack.org/show/558401/ | 14:24 |
| sdake | narasimha_SV run pip show kolla as well | 14:24 |
| sdake | we require jinja 2.8.0 or later | 14:24 |
| sdake | need to update that | 14:24 |
| sdake | run pip show kolla though as well | 14:24 |
| sdake | rhallisey ping | 14:28 |
| sdake | re https://review.openstack.org/#/c/348029/3/doc/deployment-philosophy/deployment-philosophy.rst | 14:28 |
| patchbot | sdake: patch 348029 - kolla - Build a more organized doc structure | 14:28 |
| Jeffrey4l | could we get some hep from osic guys? i still think it should physical switcher issue. inc0 sdake | 14:29 |
| Pavo | sdake this it the kind of network I am running right now https://www.dropbox.com/s/phx32ggavh1uc5p/Screenshot%202016-08-16%2010.28.57.png?dl=0 | 14:29 |
| Pavo | probably not ideal | 14:29 |
| sdake | Pavo can you put somewhere where I dont have to sign in pls :) | 14:30 |
| sdake | google drive would work for me | 14:30 |
| Pavo | you shouldn't have to but sure | 14:30 |
| sdake | nm | 14:30 |
| sdake | your right don't have to sign in | 14:30 |
| Pavo | yeah didn't think you did | 14:30 |
| sdake | Pavo are you running on virtual machines here? | 14:31 |
| sdake | pavo if so, the nat in that diagram will crater your network - you mustuse host networking feature instead (atelast in virtualbox) | 14:31 |
| Pavo | basically I am trying to use eth0 network as management and internet access for openstack and using eth1 network for communication between the nodes | 14:31 |
| Pavo | yeah | 14:31 |
| sdake | pavo the way you ahve it setup now in your globals.yml | 14:31 |
| sdake | eth0 = neutron interface | 14:32 |
| sdake | eth1=mgmt interface | 14:32 |
| sdake | try switching those around first | 14:32 |
| sdake | reboot vms | 14:32 |
| sdake | and probably your host as well | 14:32 |
| Pavo | ok | 14:32 |
| Jeffrey4l | sdake, pbourke the os_keystone_role in ansible 2.1 seems OK. | 14:32 |
| Jeffrey4l | But are u really think it is OK to move 2.1 in the stable/mitaka branch? | 14:33 |
| sdake | we are not moving all of our deployment to 2.1 | 14:33 |
| sdake | just the toolbox | 14:33 |
| pbourke | Jeffrey4l: ok again unless Im missing something Id just use that. we only created our own docker module as the its so central to our project and ansible didnt commit to maintaining it properly | 14:33 |
| sdake | just the toolbox is okk | 14:33 |
| Jeffrey4l | OK. | 14:33 |
| pbourke | Jeffrey4l: try it and if gates pass should be ok | 14:33 |
| Jeffrey4l | I will try. | 14:34 |
| Jeffrey4l | but we need another PS to upgrade the toolbox first, it it work | 14:34 |
| *** satya has joined #openstack-kolla | 14:34 | |
| Pavo | wait I already have that sdake | 14:34 |
| *** b_bezak has quit IRC | 14:35 | |
| *** michauds has joined #openstack-kolla | 14:35 | |
| sdake | link globals.yml again - sorry too much conversation goiing on at once | 14:35 |
| Pavo | neutron int is eth0 and network int is eth1 | 14:35 |
| sdake | yes | 14:35 |
| sdake | but you just said above | 14:35 |
| sdake | you want eth0 to be your management interface | 14:35 |
| sdake | and eth1 to be your neutron network | 14:35 |
| *** daneyon has joined #openstack-kolla | 14:35 | |
| sdake | pavo ^ | 14:35 |
| Pavo | ah ok I get you, but doesn't the kolla_internal_vip_address have to be on the same int as neutron? | 14:36 |
| satya | I have deployed openstack kolla multinade . while creating the instance most of the time i am getting the error: ERROR neutron.api.v2.resource InvalidRequestError: This Session's transaction has been rolled back by a nested rollback() call. To begin a new transaction, issue Session.rollback() first. | 14:36 |
| sdake | pavo no it needs to be ont he management network | 14:36 |
| Jeffrey4l | inc0, sdake pbourke how can we ask help from osic? | 14:36 |
| satya | Could you please help me in finding the root cause of the issue | 14:36 |
| Pavo | ok lol I will try it again | 14:36 |
| sdake | hey satya - weclome to the part y:) | 14:36 |
| openstackgerrit | Alexander Reunov proposed openstack/kolla: Fix murano notification driver https://review.openstack.org/355979 | 14:36 |
| sdake | satya doing an eval or planning to do kolla development? | 14:37 |
| satya | planning kolla deployment | 14:37 |
| *** diogogmt has joined #openstack-kolla | 14:37 | |
| sdake | satya are you running master or a stable branch? | 14:37 |
| Pavo | sdake but isn't the neutron_external_interface the int for internet access? | 14:37 |
| sdake | pip show kolla will tell yoou teh answer | 14:37 |
| sdake | Pavo yes | 14:37 |
| sdake | oh vip_external | 14:37 |
| *** lrensing_ has joined #openstack-kolla | 14:37 | |
| sdake | Pavo you only need vip_external for TLS | 14:38 |
| sdake | pavo do you have TlS? | 14:38 |
| Pavo | then I am totally confused lol | 14:38 |
| sdake | if not, then comment it out | 14:38 |
| satya | i am running master branch | 14:38 |
| satya | the stacktrace of the error is given below | 14:38 |
| sdake | satya i'd highly reecommend running a stable branch for eval | 14:38 |
| satya | 2016-08-16 12:42:57.991 50 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 79, in resource 2016-08-16 12:42:57.991 50 ERROR neutron.api.v2.resource result = method(request=request, **args) 2016-08-16 12:42:57.991 50 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/base.py", line 571, in update 2016-08 | 14:38 |
| sdake | satya openstack is in a constant state of broken/unbroken and an eval of master will just make you sad ^^ | 14:38 |
| Pavo | ok in global/yml network interface is what? | 14:39 |
| satya | 2016-08-16 12:42:57.991 50 ERROR neutron.api.v2.resource obj = obj_updater(request.context, id, **kwargs) 2016-08-16 12:42:57.991 50 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 618, in inner 2016-08-16 12:42:57.991 50 ERROR neutron.api.v2.resource return f(self, context, *args, **kwargs) 2016-08-16 12:42:57.991 50 ERROR neutron.api.v2.resource F | 14:39 |
| Pavo | management or communication between host | 14:39 |
| sdake | network_interface is the management wnetwork | 14:39 |
| Pavo | ok which is on the same network as my internet | 14:39 |
| sdake | the management network includes the deployment network | 14:39 |
| pbourke | Jeffrey4l: inc0: "These NICs have not been well tested in Ubuntu and as such the upstream i40e driver in the default 14.04.3 Linux kernel will begin showing issues when you setup VLAN tagged interfaces and bridges." | 14:39 |
| satya | did anybody observed this issue with master branch | 14:40 |
| sdake | pavo by internet, do you mean 192.168 address or somethign ele? | 14:40 |
| pbourke | Jeffrey4l: inc0: "If you will be installing Ubuntu Server 14.04 on these servers, I highly recommend you use an i40e driver no older than 1.3.47." | 14:40 |
| sdake | satya not yet - our gates are passing | 14:40 |
| Jeffrey4l | pbourke, where did u get the info? | 14:40 |
| *** daneyon has quit IRC | 14:40 | |
| pbourke | Jeffrey4l: its in the original email and also the osic guide | 14:40 |
| Jeffrey4l | pbourke, OK. | 14:40 |
| Pavo | the network I want to be able to access horizon and the instances inside openstack to use as External network | 14:40 |
| sdake | satya if your doing an eval or deploy, i highly recommend stable/mitaka or stable/liberty | 14:40 |
| pbourke | Jeffrey4l: inc0: we currently seem to have 1.2.2 | 14:40 |
| sdake | they start with stable/ for a reason ;) | 14:40 |
| Jeffrey4l | cool found the root cause | 14:40 |
| Jeffrey4l | yes. it is. | 14:40 |
| Jeffrey4l | how about upgrade the kernel? | 14:41 |
| satya | our stable/mitaka is running fine | 14:41 |
| Jeffrey4l | or using ubuntu 16.04? | 14:41 |
| satya | now we want to use master branch | 14:41 |
| duonghq | staya" commit a2e369982d is quite good for test | 14:41 |
| sdake | satya roger | 14:41 |
| pbourke | Jeffrey4l: im not sure but lets update the kernel to start with | 14:41 |
| Jeffrey4l | OK | 14:41 |
| *** bootsha has joined #openstack-kolla | 14:41 | |
| duonghq | satya: commit a2e369982d is quite good for test | 14:41 |
| sdake | satya a big chunk of our team is a bit busy atm working on osic scale testing (limited time window of 2 weeks left) so don't have direct time to debug thatissue at this moment | 14:41 |
| sdake | satya i'd recommend filing a bug | 14:42 |
| satya | could you please provide any pointers in identifying root cause of this issue | 14:42 |
| sdake | link it - i'll mark it as critical and confirmed | 14:42 |
| sdake | satya no clue what would cause that | 14:42 |
| Jeffrey4l | pbourke, how about try just one machine first? | 14:42 |
| sdake | satya it will require coordination between our team and neutron team and possibly nova | 14:42 |
| pbourke | egonzalez90: thats boring :p | 14:42 |
| pbourke | Jeffrey4l: ^ | 14:42 |
| satya | nova does not have any issue | 14:42 |
| pbourke | Jeffrey4l: joking ok will do | 14:42 |
| Jeffrey4l | we can test it by using `ping the gateway from the dhcp namespace` | 14:42 |
| Jeffrey4l | pbourke, ugprade the control first? | 14:43 |
| satya | if i try to create vm 7 times. 6 times this exception is thrown. only one time it is succeeding. | 14:43 |
| sdake | pavo - please paste your globals.yml again | 14:43 |
| Pavo | sdake https://paste.pound-python.org/show/C1jADxwasIZEmSF3jmJZ/ | 14:44 |
| sdake | satya google search turn up anything? | 14:44 |
| satya | nova-compute is not able to get network from neutron-server | 14:44 |
| satya | https://bugs.launchpad.net/neutron/+bug/1409774 | 14:44 |
| openstack | Launchpad bug 1409774 in neutron "sqlalchemy session needs to be rolled back after catching a db exception" [Undecided,Expired] | 14:44 |
| satya | my issue is same as above bug | 14:44 |
| sdake | pavo you do not have the external vip configured - also don't have tls configured | 14:45 |
| sdake | so ther ewill be no external vip | 14:45 |
| sdake | so your good to go | 14:45 |
| sdake | just switch eth1 and eth0 around | 14:45 |
| Pavo | ok | 14:45 |
| satya | but in my case it is happeing when port_update is called | 14:46 |
| satya | can I reopen the bug 1409774 | 14:46 |
| openstack | bug 1409774 in neutron "sqlalchemy session needs to be rolled back after catching a db exception" [Undecided,Expired] https://launchpad.net/bugs/1409774 | 14:46 |
| sdake | satya - not sure - ask the neutron cats - tell them master is busted | 14:47 |
| satya | ok thanks. | 14:47 |
| sdake | satya filing a bug would hep :) | 14:48 |
| satya | it should be a neutron bug or kolla bug | 14:49 |
| sdake | kolla could possibly be misconfiguring some new change in neutron | 14:49 |
| *** mbound has joined #openstack-kolla | 14:49 | |
| satya | will create a new bug | 14:49 |
| sdake | or more likely neutron is broke n:) | 14:49 |
| sdake | Jeffrey4l did you say you got to the root cause of the connectivity problems? | 14:50 |
| Jeffrey4l | sdake, seem so . pbourke found some useful info. we are testing it. | 14:50 |
| sdake | mind sharing a tldr? | 14:50 |
| Jeffrey4l | sdake, https://github.com/osic/ref-impl/blob/master/documents/bare_metal_provisioning.md#update-linux-kernel-1 | 14:50 |
| Jeffrey4l | the nic driver issue | 14:51 |
| pbourke | sdake: the network driver that comes in 14.04 is buggy | 14:51 |
| sdake | right we were warned about the nic drivers | 14:51 |
| sdake | thanks! | 14:51 |
| satya | ok thanks sdake | 14:51 |
| *** mbound has quit IRC | 14:54 | |
| *** narasimha_SV has quit IRC | 14:55 | |
| Pavo | ok it finished, what IP do I access horizon with now | 14:57 |
| duonghq | the internal vip | 14:57 |
| Pavo | ok I can ping that but not working for web | 14:58 |
| sdake | pavo - cool - see you ahd your networks backwards :) | 14:59 |
| sdake | brb | 14:59 |
| duonghq | Pavo: can you telnet the web service port? | 14:59 |
| *** ad_rien_ has joined #openstack-kolla | 14:59 | |
| Jeffrey4l | seems the solution(upgrade the kernel) works. | 14:59 |
| sdake | hotness! | 15:00 |
| Pavo | telnet 192.168.1.254 | 15:00 |
| Pavo | Trying 192.168.1.254... | 15:00 |
| Pavo | telnet: connect to address 192.168.1.254: Connection refused | 15:00 |
| Pavo | telnet: Unable to connect to remote host | 15:00 |
| sdake | try telnet to port 80 on that ip | 15:01 |
| pbourke | inc0: got a minute for quick question wrt macros | 15:04 |
| Pavo | been awhile since I telnet lol, the cmd to use a port is just telnet 192.168.1.254 80 right | 15:04 |
| sdake | pbourke Jeffrey4l inc0 are we good to go with sshing into vms now? | 15:04 |
| duonghq | Pavo: you deploy on your VM or your lab? iirc the horizon is enabled by default | 15:04 |
| sdake | with both linuxbridge and ovs? | 15:04 |
| pbourke | sdake: still updating rest of the nodes | 15:04 |
| pbourke | sdake: we'll let you know :) | 15:04 |
| Jeffrey4l | sdake, we should. Just make a small test. | 15:04 |
| sdake | please do | 15:04 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Customizations for Senlin https://review.openstack.org/352013 | 15:04 |
| duonghq | Pavo: can you check horizon log? | 15:04 |
| pbourke | Jeffrey4l: it didnt seem like therew as 130 nodes in that output | 15:05 |
| *** zhubingbing has quit IRC | 15:06 | |
| Pavo | duonghq horizon logs https://paste.pound-python.org/show/JUgJyZE0I4DtHEICvY1I/ | 15:06 |
| Jeffrey4l | pbourke, let's check the result when the nodes is up. | 15:06 |
| sdake | pavo telnet 192.168.1.254 80 | 15:06 |
| pbourke | Jeffrey4l: ok cool | 15:06 |
| Pavo | sdake yeah can not telnet to it | 15:07 |
| sdake | pavo those logs look ok | 15:07 |
| sdake | can you ssh to the hosts at all? | 15:07 |
| Jeffrey4l | pbourke, about the `mount --make-shared /run` in ubuntu. Does it work if add this to the fstab? | 15:07 |
| pbourke | Jeffrey4l: not sure | 15:07 |
| Pavo | sdake yeah | 15:07 |
| sdake | Jeffrey4l you can't just add that command right in fstab | 15:07 |
| sdake | but it may be possible to automate that via some other mechanism | 15:07 |
| inc0 | pbourke, shoot | 15:09 |
| pbourke | inc0: smekel_ raised the case where we want to handle an empty package list | 15:09 |
| pbourke | https://review.openstack.org/#/c/354429/ | 15:09 |
| patchbot | pbourke: patch 354429 - kolla - Handle empty package list for install_packages | 15:09 |
| sdake | need to handle that | 15:09 |
| *** rstarmer has quit IRC | 15:09 | |
| pbourke | inc0: problem with the patch currently is if the macro returns nothing, we still have a "RUN" hardcoded into the Dockerfile | 15:10 |
| pbourke | inc0: so I propose moving the "RUN" into the macro | 15:10 |
| inc0 | that's ok | 15:10 |
| inc0 | makes sense | 15:10 |
| pbourke | inc0: I think it makes sense, just said Id run it by you as it means refactoring all the existing customisations | 15:10 |
| pbourke | not by much mind you | 15:10 |
| sdake | should take him 30 minutes max :) | 15:10 |
| inc0 | yeah, we'll need to remove all the RUN's | 15:10 |
| inc0 | but that's one grep | 15:10 |
| pbourke | grand I'll update the patch and see if everything builds | 15:10 |
| pbourke | yup already done | 15:11 |
| inc0 | cool, thanks | 15:11 |
| inc0 | hmm you updated kenrle everywhere Jeffrey4l | 15:11 |
| inc0 | ? | 15:11 |
| sdake | sean-k-mooney - just a heads up - you may not get the full week you were promised originally on osic cluster | 15:12 |
| Jeffrey4l | inc0, should pbourke does. | 15:12 |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Handle empty package list for install_packages https://review.openstack.org/354429 | 15:12 |
| sdake | priorities are our currently defined scenarios with tempest+rally, then bifrost scale out testing | 15:12 |
| pbourke | inc0: any reason we should not have? | 15:12 |
| Jeffrey4l | inc0, and it works. | 15:12 |
| Jeffrey4l | ya, why not? | 15:13 |
| inc0 | no, just wondering if you're onto anything:_) | 15:13 |
| *** vhosakot has joined #openstack-kolla | 15:13 | |
| sdake | i think kernel update everywhere is mandatory since that nicc is on all the gear | 15:13 |
| sdake | morning vhosakot | 15:13 |
| vhosakot | sdake: morning! | 15:14 |
| *** michauds has quit IRC | 15:15 | |
| *** belmoreira has quit IRC | 15:17 | |
| Pavo | yeah keep getting no route to host but can ping the internal ip | 15:17 |
| Pavo | at least I got this close | 15:18 |
| sdake | pavo - i need to head to dr soon | 15:18 |
| sdake | and will be out of office for ~2 hrs | 15:18 |
| sdake | pavo duonghq may be able to help you | 15:18 |
| sdake | pavo otherwise, just a heads up in case I dont respond :) | 15:18 |
| Pavo | no problem, I will continue TSing | 15:18 |
| duonghq | Pavo: what is your network topo? | 15:19 |
| Pavo | and I thought openstack networking was a nightmare to understand, now kolla giving me the headache lol | 15:19 |
| Pavo | duonghq https://www.dropbox.com/s/phx32ggavh1uc5p/Screenshot%202016-08-16%2010.28.57.png?dl=0 | 15:19 |
| sdake | pavo not kolla - its neutron - blame neutron :) | 15:19 |
| duonghq | so you setup 3 VMs? | 15:20 |
| *** athomas has quit IRC | 15:20 | |
| sdake | duonghq he has his network setup properly but he is using a nat on a virtual machine | 15:20 |
| Pavo | lol yeah but how kolla defines what neutron uses is what I need to understand I think | 15:20 |
| Pavo | duonghq yes | 15:20 |
| sdake | i think you can't access nat from a desktop | 15:20 |
| sdake | that is why you need that host networking bridged mode thing in virtualbox | 15:20 |
| duonghq | which hypervisor do you use | 15:20 |
| Pavo | I have all the eth0 bridged in VMWare fusion | 15:21 |
| Pavo | I have all the eth1s shared ie... NAT | 15:21 |
| sdake | pavo that should work then | 15:21 |
| sdake | although i dont have experience with fusion | 15:21 |
| sdake | only virtualbox | 15:21 |
| Pavo | eth0 https://www.dropbox.com/s/w0ctrs0uephe7j3/Screenshot%202016-08-16%2011.21.45.png?dl=0 | 15:21 |
| duonghq | so you set network_interface to eth0? | 15:22 |
| Pavo | eth1 https://www.dropbox.com/s/cbw9pp6706006xh/Screenshot%202016-08-16%2011.22.03.png?dl=0 | 15:22 |
| Pavo | duonghq yeap | 15:22 |
| sdake | wifi.. | 15:22 |
| openstackgerrit | Li Yingjun proposed openstack/kolla: Fix no tar command for magnum conductor using ubuntu https://review.openstack.org/356008 | 15:22 |
| sdake | your on a laptop? | 15:22 |
| Pavo | duonghq kolla_internal_vip_address: "192.168.1.254" | 15:23 |
| Pavo | duonghq network_interface: "eth0" | 15:23 |
| Pavo | duonghq neutron_external_interface: "eth1" | 15:23 |
| *** athomas has joined #openstack-kolla | 15:24 | |
| duonghq | I'm not pretty sure which node got the VIP | 15:24 |
| duonghq | what is your nodes' ip | 15:24 |
| Pavo | node1 192.168.1.45, node2 192.168.1.11 | 15:25 |
| Pavo | for eth0 | 15:25 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Add Senlin Ansible role https://review.openstack.org/339725 | 15:25 |
| Pavo | node1 192.168.188.129, node2 192.168.188.130 eth1 | 15:25 |
| openstackgerrit | Li Yingjun proposed openstack/kolla: Fix no tar command for magnum conductor using ubuntu https://review.openstack.org/356008 | 15:26 |
| sdake | pavo - idea - instead of accessing vip | 15:26 |
| sdake | try accessing node directly on port 80 | 15:26 |
| Pavo | ok | 15:26 |
| pbourke | inc0: you around to do that dhcp test again? | 15:26 |
| Pavo | negative for both | 15:26 |
| sdake | ping works for both? | 15:27 |
| Pavo | roger | 15:27 |
| sdake | disable iptables on the two machines | 15:27 |
| sdake | reboot the vms | 15:27 |
| sdake | cleanup then reboot i mean | 15:27 |
| Pavo | ok so use the tools files? | 15:28 |
| sdake | pavo right | 15:28 |
| Pavo | cleanup-containers and cleanup-host | 15:28 |
| Pavo | roger | 15:28 |
| sdake | pavo are you running centos or ubuntu vms? | 15:28 |
| Pavo | centos 7 for all 3 | 15:28 |
| sdake | systemctl disable firewalld | 15:28 |
| *** daneyon has joined #openstack-kolla | 15:28 | |
| Pavo | roger | 15:28 |
| duonghq | Pavo: after that, can you inspect horozion container on those VMs? | 15:29 |
| Pavo | roger | 15:29 |
| sdake | those arguments may be backwards | 15:29 |
| sdake | on all 3 nodes | 15:29 |
| sdake | reboot even your deployer node | 15:29 |
| *** bootsha has quit IRC | 15:29 | |
| Pavo | will have to redeploy just ran the cleanup scripts | 15:29 |
| Pavo | lol | 15:29 |
| inc0 | pbourke, give me 10min | 15:29 |
| sdake | after that change | 15:29 |
| pbourke | inc0: cool | 15:29 |
| *** michauds has joined #openstack-kolla | 15:30 | |
| *** daneyon_ has quit IRC | 15:32 | |
| Pavo | ok did cleanup scripts on target nodes, disabled firewalld on all three nodes and rebooted, now redeploying | 15:32 |
| *** matrohon has quit IRC | 15:32 | |
| sdake | Jeffrey4l - I may havea conflict during our team meeting wednesday - any chance you would like to take on facilitating the team meeting? | 15:33 |
| sdake | (as in charing it) | 15:33 |
| sdake | chairing it | 15:33 |
| pbourke | Jeffrey4l: this is weird, never seen before | 15:35 |
| pbourke | Jeffrey4l: that said dont reboot often | 15:36 |
| Jeffrey4l | pbourke, yes. | 15:36 |
| Jeffrey4l | the rabbitmq cluster seem not joined up, then failed. | 15:36 |
| sdake | but it was working prior? | 15:37 |
| *** zhurong has quit IRC | 15:37 | |
| pbourke | inc0: sdake working!! | 15:40 |
| pbourke | WOOT | 15:40 |
| duonghq | Pavo: anything news? | 15:40 |
| sdake | pbourke as in ssh into vm works? | 15:41 |
| sean-k-mooney | sdake: re osic cluster that fine. if we need to push or drop the bifrost testing to complete the other work items i think that makes sense | 15:41 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Add a note about the sandbox environment to the CONTRIBUTING doc https://review.openstack.org/356017 | 15:41 |
| sdake | sean-k-mooney cool - but i'd like to give you atleast a cuple days crack at the gear | 15:42 |
| sdake | sean-k-mooney i don't know how you feel about weekends, but the gear seems to be available then :) | 15:42 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Add a note about the sandbox environment to the CONTRIBUTING doc https://review.openstack.org/356017 | 15:42 |
| pbourke | sdake: actually celebrated too soon, ping works but ssh not yet | 15:42 |
| sdake | sean-k-mooney but you would probably need help of others who have ramped up all the network setup issues | 15:42 |
| pbourke | sdake: very close though I think | 15:42 |
| Pavo | still deploying | 15:43 |
| *** daneyon_ has joined #openstack-kolla | 15:43 | |
| pbourke | sdake: ssh working | 15:45 |
| sdake | hot - so root cause was 710 driver? | 15:45 |
| *** daneyon has quit IRC | 15:45 | |
| sdake | what else did you do? | 15:45 |
| Jeffrey4l | sdake, nothing. it it the driver issue. | 15:46 |
| pbourke | sdake: that's it | 15:46 |
| sdake | cool | 15:46 |
| sdake | thanks! | 15:46 |
| *** dmueller-evoila has quit IRC | 15:47 | |
| pbourke | i have a script to run all rally tests we're intrested in in another tmux | 15:47 |
| Pavo | ok done and now getting alittle closer | 15:47 |
| Pavo | https://www.dropbox.com/s/3wugmb0nyrgu10k/Screenshot%202016-08-16%2011.47.03.png?dl=0 | 15:47 |
| pbourke | right now I have to do some internal stuff but if someone wants to continue with that let me know | 15:47 |
| *** haplo37__ has quit IRC | 15:47 | |
| sdake | i think we are ready for coolsvap to do the magic with rally | 15:48 |
| sdake | wouldn't hurt if someone else learned dthat as well | 15:48 |
| sdake | i'd take it on, but ogtta jet in 15-20 mins | 15:48 |
| duonghq | Pavo: can you check the haproxy/keepalived? | 15:48 |
| Pavo | logs? | 15:48 |
| duonghq | yup | 15:48 |
| sdake | vhosakot - available for scale testing ? | 15:48 |
| sdake | vhosakot - specifically learning and configuring our rally setup so it actually tests something useful :) | 15:49 |
| pbourke | im familiar enough with the rally stuff | 15:49 |
| pbourke | so please ping me if you want in | 15:49 |
| Pavo | haproxy logs https://paste.pound-python.org/show/4ybRwDCld95WC6FI2tJa/ | 15:49 |
| Pavo | keepalive logs https://paste.pound-python.org/show/fDnYGgrXgEy2eTdpJaRU/ | 15:50 |
| sdake | pavo those are not the logs you are looking for | 15:50 |
| *** athomas has quit IRC | 15:50 | |
| sdake | docker exec heka ls -l /var/log/kolla | 15:50 |
| Pavo | ah | 15:50 |
| Pavo | on the networking node? | 15:51 |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Handle empty package list for install_packages https://review.openstack.org/354429 | 15:52 |
| Jeffrey4l | this block the horizon deployment https://review.openstack.org/355232 | 15:53 |
| Jeffrey4l | on master. | 15:53 |
| Pavo | duonghq what is the cmd to see the logs you are looking for? | 15:53 |
| *** hieulq_ has joined #openstack-kolla | 15:55 | |
| duonghq | actually, I want to see which node get the VIP, and then horizon log in this node | 15:55 |
| berendt | Jeffrey4l horizon change looks good, use of an array for multiple memcache servers for django is correct | 15:55 |
| Pavo | how to tell which one got the VIP | 15:56 |
| Jeffrey4l | berendt, thanks. | 15:56 |
| duonghq | I think "ip addr" in each node is ok | 15:56 |
| openstackgerrit | Larry Rensing proposed openstack/kolla: Add support for building vmtp container https://review.openstack.org/354362 | 15:56 |
| *** athomas has joined #openstack-kolla | 15:56 | |
| NachoDuck | Hey, sorry to interrupt. If I want to setup a VLAN provider network (i.e edit ml2_conf.ini), how do I approach this in Kolla? | 15:57 |
| berendt | at all i am not sure if it makes sense to specify multiple memcached servers, in my environment i overwrite all memcached settings to only use the local available memcachd server | 15:57 |
| duonghq | but which node you get the log above? | 15:57 |
| Pavo | node1 https://paste.pound-python.org/show/ZQGULjphXaJmfjoMbHMC/ | 15:57 |
| Jeffrey4l | berendt, how about when local memcached crashed? | 15:57 |
| Pavo | node2 https://paste.pound-python.org/show/GaSfXSSFh9IicMIuJwq3/ | 15:57 |
| berendt | Jeffrey4l than caching is not working, that is fine for me | 15:57 |
| duonghq | so, check node1 horizon log | 15:57 |
| *** huikang has joined #openstack-kolla | 15:57 | |
| duonghq | apache2 log, too | 15:57 |
| berendt | Jeffrey4l but i think it makes no sense when a service connects to a memcached service running on an other host | 15:58 |
| Pavo | the container log ie... docker logs blah? | 15:58 |
| berendt | Jeffrey4l but i have not benchmarked it... | 15:58 |
| Jeffrey4l | berendt, yes, it is. but using multi memcached will more robust. | 15:58 |
| duonghq | just use heka container like sdake mentioned | 15:59 |
| inc0 | NachoDuck, https://github.com/openstack/kolla/blob/master/doc/advanced-configuration.rst#openstack-service-configuration-in-kolla | 15:59 |
| duonghq | something like /var/log/kolla/apache2 | 15:59 |
| duonghq | is there anything like that? | 15:59 |
| Jeffrey4l | berendt, in some case, ( like murano dashboard ), it require memcached as session storage. In this case, just use local memcached will cause some issue. | 15:59 |
| sbezverk | inc0: do you have a second for qq? | 15:59 |
| inc0 | sure sbezverk | 16:00 |
| berendt | Jeffrey4l ok, i do not provide murano at the moment, i will re-evaluate this in the future, but at the moment it is working quite good with only one memcached process | 16:01 |
| Pavo | duonghq cat: /var/log/kolla/horizon: Is a directory | 16:01 |
| sbezverk | inc0: since new variable got introduced orchestration_engine, can we then customing services configuration so they would be more consumable by kube? | 16:01 |
| Pavo | duonghq cat: /var/log/kolla/apache2: No such file or directory | 16:01 |
| inc0 | sbezverk, so what we wanted to do is to put all of configs to shared dir as images | 16:02 |
| inc0 | so I'd say yeah, we want same conf files | 16:02 |
| duonghq | Pavo so, let me the horizon log | 16:02 |
| NachoDuck | inc0: I've had a read through that but changes I made don't seem to apply, or rather the file is reset after a reconfigure. I've tried creating it under /etc/kolla/config/neutron/plugins/ml2 and also /etc/kolla/neutron-server/ml2_conf.ini I take I'm doing something wrong here but no too sure what | 16:02 |
| sbezverk | inc0: example glance-api for kolla points to VIP, I need to point it to a specific dns name, can I use if statement to check for orchestration engine and if it is kubernetes then use value I need | 16:02 |
| sbezverk | > | 16:02 |
| berendt | Pavo docker exec -it horizon ls /var/log/kolla/horizon | 16:02 |
| Pavo | ok figured it out | 16:02 |
| Pavo | duonghq horizon log | 16:02 |
| Pavo | https://paste.pound-python.org/show/8p9K7LPbtPNwgv84JoKW/ | 16:03 |
| sbezverk | inc0: thanks | 16:03 |
| berendt | Pavo you can also directly access the kolla-logs volume, cd /var/lib/docker/volumes/kolla_logs/_data | 16:03 |
| *** srwilkers has quit IRC | 16:03 | |
| sdake | https://twitter.com/steven_dake | 16:03 |
| Pavo | ok | 16:04 |
| sdake | enjoy :) | 16:04 |
| inc0 | NachoDuck, and what you should do is /etc/kolla/config/neutron/ml2_conf.ini | 16:04 |
| inc0 | https://github.com/openstack/kolla/blob/master/ansible/roles/neutron/tasks/config.yml#L65 | 16:04 |
| sdake | i wonder if I beg ortofon enough if they will rebuild my cadenza black for no cost ;) | 16:04 |
| inc0 | this is code that will deploy the conf | 16:04 |
| duonghq | Pavo: ahah, so horizon fails you | 16:04 |
| inc0 | sbezverk, so how about something different | 16:05 |
| Pavo | says its up | 16:05 |
| Pavo | 575ebd442305 deployer.local:4000/kolla/centos-binary-horizon:3.0.0 "kolla_start" 19 minutes ago Up 9 minutes horizon | 16:05 |
| inc0 | just thinking | 16:05 |
| inc0 | see how we use merge_config in ansible? | 16:05 |
| NachoDuck | inc0: I see, thanks for your help :) | 16:05 |
| inc0 | maybe provide this kind of mechanism in k8s | 16:05 |
| openstackgerrit | Merged openstack/kolla: Make the enable_isolated_metadata and force_metadata to yes in default https://review.openstack.org/352709 | 16:05 |
| inc0 | NachoDuck, no problem, let us know if you have any other issues | 16:06 |
| inc0 | sbezverk, but don't let that stop you. I think proper shared configuration would be good work for Ocata | 16:06 |
| inc0 | we should discuss that on summit, how exactly we want to do it | 16:07 |
| inc0 | we could for example have this block mechanism we use now for dockerfiles | 16:07 |
| sbezverk | inc0: well it makes life very difficult, as everytime I genconfig, I need to go and manually changes 10k of values | 16:07 |
| duonghq | Pavo: can you upload full horizon log? | 16:07 |
| Pavo | yeah | 16:07 |
| inc0 | yeah sbezverk sure, we need to figure out good way, all I'm saying | 16:07 |
| sbezverk | inc0: it introduces 20k of type errors and delay develoopment | 16:08 |
| inc0 | what I don't want to have at the end of the day is 100 ifs in configs | 16:08 |
| inc0 | if k8s then elif ansible elif chef... | 16:08 |
| inc0 | we need to figure out good way to work with that stuff | 16:08 |
| sbezverk | inc0: got it | 16:08 |
| sbezverk | next engine will be added as elif | 16:08 |
| *** sdake has quit IRC | 16:09 | |
| inc0 | at some point it would be great to be able to add new engine without any changes in kolla | 16:09 |
| Pavo | duonghq this is the full log https://paste.pound-python.org/show/YdbzI1xSZnbffUEiMqyG/ | 16:09 |
| inc0 | kolla will just be consumed | 16:09 |
| duonghq | pavo: let see | 16:10 |
| Pavo | I scp it from /var/lib/docker/volumes/kolla_logs/_data/horizon/horizon.log | 16:10 |
| Pavo | on node1 | 16:10 |
| Pavo | looks like a lot of python errors | 16:10 |
| duonghq | yup | 16:10 |
| duonghq | Django, indeed | 16:11 |
| Pavo | rebuild images? | 16:11 |
| duonghq | not sure, let me search around | 16:12 |
| Pavo | roger just checked the python version on all nodes and its 2.7.5 | 16:12 |
| duonghq | I think it is fine | 16:13 |
| Pavo | you gonna be on for a bit, gotta run to the store for about 30 minutes | 16:13 |
| inc0 | Jeffrey4l, pbourke deployment finished | 16:13 |
| inc0 | Jeffrey4l, what kind of network you did when it worked? | 16:13 |
| pbourke | inc0: we didnt change any of the settings | 16:14 |
| Jeffrey4l | inc0, current works | 16:14 |
| pbourke | just rebooted the VM | 16:14 |
| pbourke | (the guest) | 16:14 |
| Jeffrey4l | i am fixing the metadata issue. | 16:14 |
| Jeffrey4l | works. :D | 16:15 |
| Pavo | be back in like 30 duonghq | 16:15 |
| duonghq | Pavo: tommorrow I'll recheck the error when I reach my dev machine (if I have time), but I think kbaegis has experience | 16:16 |
| duonghq | so, how can I reach you in case I found something? | 16:16 |
| Pavo | I'll PM you my email | 16:16 |
| duonghq | okay | 16:16 |
| duonghq | Pavo: I think you can try rollback the docker/horizon some commit and try build and deploy horizon (just kolla-ansible upgrade) | 16:33 |
| *** unicell has quit IRC | 16:34 | |
| * duonghq head to my bed now | 16:34 | |
| *** duonghq has left #openstack-kolla | 16:34 | |
| kbaegis | Hmm | 16:36 |
| kbaegis | Still getting 500 from proxy after patch | 16:36 |
| kbaegis | 2.0.1 works, 3.0.0 horizon does not | 16:36 |
| kbaegis | cleaned and rebuilt | 16:37 |
| kbaegis | more accurately patched, reinstalled, cleaned, rebuilt, redeployed | 16:37 |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Fix neutron-server section in AIO sample inventory https://review.openstack.org/356056 | 16:38 |
| sean-k-mooney | pbourke: inc0 so out of interest is the networking issue in the osic cluster relsolved | 16:40 |
| pbourke | sean-k-mooney: I think so | 16:41 |
| sean-k-mooney | pbourke: ok cool | 16:41 |
| pbourke | sean-k-mooney: we can boot guests on the VLAN provider net and ssh to them | 16:41 |
| sean-k-mooney | pbourke: what was the issue in the end | 16:41 |
| kbaegis | Going to roll back just the horizon | 16:41 |
| inc0 | pbourke, so at the end of the day...it was kernel issue right? | 16:41 |
| pbourke | sean-k-mooney: the network driver is buggy on the switches osic use | 16:41 |
| pbourke | sean-k-mooney: so we needed to update. The osic docs say this but we were using an ISO which someone told me was up to date | 16:42 |
| inc0 | I'm to blame probably | 16:42 |
| pbourke | kind of crazy that was it really, was sure it would be some configuration issue | 16:42 |
| inc0 | well | 16:42 |
| sean-k-mooney | pbourke: on the switches? do you mean nics or the TORs? | 16:42 |
| inc0 | irtermite, btw ^ | 16:43 |
| *** thumpba has joined #openstack-kolla | 16:43 | |
| inc0 | sean-k-mooney, nics in servers | 16:43 |
| pbourke | sean-k-mooney: whats a TOR again? | 16:43 |
| inc0 | top of the rack switch pbourke | 16:43 |
| pbourke | inc0: I thought it was the TORs | 16:43 |
| inc0 | it were nics | 16:43 |
| inc0 | we don't have access to TOR at all | 16:43 |
| pbourke | ah sorry you're right | 16:44 |
| *** berendt has quit IRC | 16:44 | |
| pbourke | " All servers contain two Intel X710 10 GbE NICs. This is a relatively new NIC that has caused us a lot of problems during the setup of the OSIC environment. If you will be installing Ubuntu Server 14.04 on these servers, I highly recommend you use an i40e driver no older than 1.3.47." | 16:44 |
| pbourke | at least we solved it :) | 16:44 |
| sean-k-mooney | oh wait your are using ubuntu 14.04. did you upgrade the kernel or use the default kernel | 16:44 |
| *** ad_rien_ has quit IRC | 16:45 | |
| pbourke | sean-k-mooney: we upgraded the kernel to xenial | 16:45 |
| pbourke | i thought the iso we were using had an up to date kernel, so kind of my fault :( | 16:45 |
| sean-k-mooney | i taught you were using niantics (x520) but yes the x710 dirver in the default ubuntu 14.04 kernel is broken | 16:45 |
| sean-k-mooney | i should have copped that. i knew that driver was broken but it should be fine if you were useing the vivid kernel or newer but i didnt know you were using those nics | 16:46 |
| *** egonzalez90 has quit IRC | 16:47 | |
| openstackgerrit | Merged openstack/kolla: enhancements to install_packages macro https://review.openstack.org/348567 | 17:02 |
| *** srwilkers has joined #openstack-kolla | 17:05 | |
| pbourke | im off guys - please check latest update that will be arriving to the ML any minute now, add to it if you need | 17:07 |
| *** david-lyle_ has joined #openstack-kolla | 17:09 | |
| *** srwilkers_ has joined #openstack-kolla | 17:11 | |
| irtermite | inc0: what am I looking at above? the update part? right on | 17:12 |
| *** david-lyle has quit IRC | 17:13 | |
| irtermite | glad to see someone reads my emails ( thanks pbourke ) | 17:13 |
| *** david-lyle_ is now known as david-lyle | 17:13 | |
| openstackgerrit | Joseph Horne proposed openstack/kolla: Minor doc fix https://review.openstack.org/356068 | 17:14 |
| irtermite | ;) | 17:14 |
| *** unicell has joined #openstack-kolla | 17:14 | |
| *** srwilkers has quit IRC | 17:15 | |
| *** srwilkers_ is now known as srwilkers | 17:15 | |
| ntpttr___ | inc0: ping | 17:15 |
| irtermite | ntpttr___: I THINK he is at the food trucks | 17:15 |
| ntpttr___ | ah that would make sense, it's 12:15 in the great country of texas | 17:16 |
| irtermite | hahaha @ country | 17:16 |
| irtermite | yea | 17:16 |
| *** srwilkers_ has joined #openstack-kolla | 17:16 | |
| ntpttr___ | maybe someone else can help me, I'm having a bit of trouble with a deploy involving ceph | 17:16 |
| irtermite | need me to go smack him for you? | 17:16 |
| ntpttr___ | idk if the problem is ceph itself or not, because the error is happening in the neutron playbooks | 17:17 |
| irtermite | ah... <not it> | 17:17 |
| ntpttr___ | the error is here, it's only failing on the storage servers running ceph http://paste.openstack.org/show/557765/ | 17:17 |
| *** daneyon has joined #openstack-kolla | 17:18 | |
| *** srwilkers has quit IRC | 17:19 | |
| *** srwilkers_ is now known as srwilkers | 17:19 | |
| *** daneyon has quit IRC | 17:23 | |
| *** Serlex has quit IRC | 17:23 | |
| openstackgerrit | Josh Lothian proposed openstack/kolla: Customizations for heka https://review.openstack.org/346861 | 17:25 |
| *** mark-casey has joined #openstack-kolla | 17:26 | |
| *** dasTor has quit IRC | 17:27 | |
| sean-k-mooney | ntpttr___: its just a guess but i think one of those groups is not defined. | 17:27 |
| sean-k-mooney | ntpttr___: can you check if all the groups are in the inventory file? | 17:28 |
| kbaegis | Do you have to destroy/deploy every time you make a change? | 17:28 |
| *** hieulq_ has quit IRC | 17:29 | |
| sean-k-mooney | kbaegis: it depend on the change. if its a configfile change you can run the reconfigure playbook instead | 17:30 |
| sean-k-mooney | kbaegis: if you are enableing a new service it might be easier to do a cleanup and a redeploy | 17:30 |
| sean-k-mooney | ntpttr___: the failing section is this include https://github.com/openstack/kolla/blob/68ca8942cb1828e801540088e7de9d3dce34f23e/ansible/roles/neutron/tasks/deploy.yml#L8-L14 | 17:31 |
| ntpttr___ | sean-k-mooney: yeah I'll double check | 17:33 |
| *** hieulq_ has joined #openstack-kolla | 17:36 | |
| ntpttr___ | sean-k-mooney: ah I think that neutron-lbaas-agent was missing | 17:36 |
| ntpttr___ | I had the inv from mitaka but had upgraded my kolla dir to master | 17:37 |
| ntpttr___ | thanks! | 17:37 |
| sean-k-mooney | no worries hopefully adding that in will fix it :) | 17:37 |
| *** rstarmer has joined #openstack-kolla | 17:40 | |
| *** athomas has quit IRC | 17:42 | |
| jrich523 | whats the username used for rabbit? i guess, whats the username that is mapped to the passwords.yml | 17:42 |
| kbaegis | Has anyone been able to get horizon working from master? I'm still getting 500s | 17:42 |
| jrich523 | i was able to, sorta | 17:43 |
| jrich523 | set it up last week, off master (V3.0) | 17:43 |
| Pavo | I am gettng the same kbaegis | 17:43 |
| Pavo | so its just not me then, thats good to know | 17:43 |
| sean-k-mooney | kbaegis: is it not deploying or are you not able to reach it? | 17:44 |
| kbaegis | They gave me two cherrypicks, but they're not working | 17:44 |
| kbaegis | It's deploying just fine | 17:44 |
| kbaegis | It just doesn't give me a site. 500 | 17:44 |
| Pavo | my horizon logs show a total mess of python errors | 17:44 |
| sean-k-mooney | oh ok strange am if you are deploying from source maybe go back two-three weeks in the logs and pick an older commit | 17:45 |
| sean-k-mooney | Pavo: do you think the error is on the kolla side or horizon? | 17:45 |
| kbaegis | Pavo So they recommended two cherrypicks: https://review.openstack.org/#/c/349197/, https://review.openstack.org/#/c/349198 | 17:45 |
| Pavo | horizon I think | 17:45 |
| patchbot | kbaegis: patch 349197 - kolla - Implement kolla_keystone_role module | 17:46 |
| patchbot | kbaegis: patch 349198 - kolla - Create the _member_ role in the horizon role | 17:46 |
| kbaegis | Umm | 17:46 |
| kbaegis | I didn't accidentally do anything w/ patchbot did I? | 17:46 |
| sean-k-mooney | wait is the _member_ createtion not merged? | 17:46 |
| kbaegis | No | 17:46 |
| kbaegis | It's pushed, not merged | 17:46 |
| Pavo | I get Internal Server Error | 17:46 |
| Pavo | The server encountered an internal error or misconfiguration and was unable to complete your request. | 17:46 |
| Pavo | Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error. | 17:46 |
| Pavo | More information about this error may be available in the server error log. | 17:46 |
| kbaegis | Yeah, that's an HTTP 500 error :) | 17:47 |
| kbaegis | Same thing | 17:47 |
| sean-k-mooney | can ye both check if the _member_ role exists | 17:48 |
| Pavo | how do I do that? | 17:49 |
| sean-k-mooney | you need to use the openstack client. let me see if i can find the command | 17:49 |
| kbaegis | openstack role list | 17:50 |
| *** hieulq_ has quit IRC | 17:50 | |
| Pavo | https://paste.pound-python.org/show/lmtozB30baPUWed3fYwD/ | 17:51 |
| kbaegis | I already tried $(openstack roll create _member_) | 17:51 |
| Pavo | doesn't look liek _member_ role exist | 17:51 |
| kbaegis | Doesn't solve the issue | 17:51 |
| kbaegis | I didn't think to try kolla-ansible reconfigure afterwards though | 17:53 |
| inc0 | ntpttr___, you've summoned me | 17:53 |
| inc0 | . | 17:53 |
| inc0 | or at least you've been summoning me | 17:54 |
| inc0 | why did you summon me? speak mortal! | 17:54 |
| sean-k-mooney | kbaegis: you could try to restart horizon container | 17:54 |
| sean-k-mooney | inc0: he had a bug in his deployment | 17:54 |
| sean-k-mooney | inc0: i think i answered his question | 17:54 |
| kbaegis | I tried that too | 17:55 |
| *** bootsha has joined #openstack-kolla | 17:55 | |
| inc0 | thanks Sean | 17:55 |
| *** sdake has joined #openstack-kolla | 17:55 | |
| inc0 | I'm going to look for SamYaple, apperantly he's in building | 17:55 |
| inc0 | brb | 17:55 |
| sean-k-mooney | inc0: since your here any idea what is wrong with horizon | 17:55 |
| sdake | yo | 17:55 |
| sean-k-mooney | inc0: oh ok say hi to sam for me | 17:56 |
| sdake | sean-k-mooney is horizon broken foryou as well? | 17:56 |
| sean-k-mooney | no im not useing master right now | 17:56 |
| sean-k-mooney | but its broken for kbaegis and Pavo | 17:56 |
| kbaegis | Yes. Log? | 17:56 |
| Pavo | I an trying reconfigure right now | 17:56 |
| jrich523 | it blows my mind that you guys dev on master | 17:57 |
| sdake | i'll give master a spin in a few hours | 17:57 |
| sdake | jrich523 normally master is stable enough for us to get real work done :) | 17:57 |
| sean-k-mooney | jrich523: why i generally have no issue with master | 17:57 |
| sdake | master is stable enoug hto ddev, not stable enough to eval or deploy imo | 17:57 |
| jrich523 | well im normally a windows guy, and have worked in a few dev shops (not linux stuff) | 17:57 |
| Pavo | I just follwed the doc on openstacks wiki, don't know what branch I am using lol | 17:57 |
| ntpttr___ | inc0: it's been handled, thanks tho | 17:58 |
| ntpttr___ | inc0: hope the food carts were tasty | 17:58 |
| jrich523 | and its considered bad to use master for anything but stable (released to prod) | 17:58 |
| jrich523 | but as i deal with more linux projects i see a lot of folks doing dev on master | 17:58 |
| sean-k-mooney | jrich523: openstack use a contiusly gated trunk modle | 17:58 |
| sdake | we dev against master | 17:58 |
| jrich523 | i guess the only reason not to, is a lot of people dont understand git/code repos | 17:58 |
| sdake | and then backport | 17:58 |
| sean-k-mooney | jrich523: in theroy it should alway be deployable to production | 17:58 |
| sdake | that is how softwrae is created :) | 17:58 |
| sean-k-mooney | jrich523: the important word is "should" even with all of openstack ci systems bugs slip through from time to time | 17:59 |
| jrich523 | yeah, so large its tough to test it all | 18:00 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Create the _member_ role in the horizon role https://review.openstack.org/356082 | 18:00 |
| jrich523 | Atlassian had an awesome article on git flow, however gated trunk is a new one to me | 18:00 |
| *** dmsimard|afk is now known as dmsimard | 18:01 | |
| *** satya has quit IRC | 18:01 | |
| sdake | jrich523 gated trunk = ci | 18:01 |
| jrich523 | however i have a great excuse to not be an expert on that, im an ops guy :) | 18:01 |
| sean-k-mooney | what the difference between https://review.openstack.org/#/c/349198 and https://review.openstack.org/#/c/356082/ | 18:01 |
| patchbot | sean-k-mooney: patch 349198 - kolla - Create the _member_ role in the horizon role | 18:01 |
| patchbot | sean-k-mooney: patch 356082 - kolla - Create the _member_ role in the horizon role | 18:01 |
| jrich523 | err well with CI you still use the master as stable, the idea is to branch for feature (agile sprint) and then merge back to master when its completed and passes unit tests, which then the CD comes in and its auto deployed off master | 18:01 |
| sean-k-mooney | jrich523: we do that in openstack but it does not merge to master untill it pass unitest + intergration tests | 18:02 |
| jrich523 | oh so then master counts as stable | 18:03 |
| jrich523 | or, should | 18:03 |
| sean-k-mooney | jrich523: yes if our test coverage is good enaough you should always be able to deploy master to production | 18:03 |
| jrich523 | yeah.... pesky test coverage | 18:04 |
| jrich523 | lol you could probably use a whole team of folks just doing that | 18:04 |
| jrich523 | i really hate writing tests | 18:04 |
| kbaegis | Pavo: did adding _member_ work for you? | 18:05 |
| sean-k-mooney | the bit i dislike is when may change is 20% code and 80% tests but if it means i dont have to go back and fix something again later i would perfer to have the tests | 18:05 |
| Pavo | haven't tried that yet, still waiting on reconfigure to finsh | 18:05 |
| jrich523 | the last project i worked on was TDD, and well... great idea, just tough to adjust to | 18:06 |
| *** tummy has joined #openstack-kolla | 18:08 | |
| sean-k-mooney | Jeffrey4l: why do you have two identical patches https://review.openstack.org/#/c/349198 and https://review.openstack.org/#/c/356082/ | 18:08 |
| patchbot | sean-k-mooney: patch 349198 - kolla - Create the _member_ role in the horizon role | 18:08 |
| patchbot | sean-k-mooney: patch 356082 - kolla - Create the _member_ role in the horizon role | 18:08 |
| kbaegis | I have the _member_ and horizon is still broken | 18:09 |
| Jeffrey4l | sean-k-mooney, yes it is. two fix way. the later one may be better. I will abandon one later | 18:09 |
| sean-k-mooney | Jeffrey4l: well one of them is approved and waiting to merge | 18:10 |
| Pavo | kbaegis whats the cmd to add the _member_ role? | 18:10 |
| sean-k-mooney | Jeffrey4l: https://review.openstack.org/#/c/349198/ | 18:10 |
| patchbot | sean-k-mooney: patch 349198 - kolla - Create the _member_ role in the horizon role | 18:10 |
| Jeffrey4l | sean-k-mooney, pbourke do not like it. and it is block by its parent PS. | 18:10 |
| sean-k-mooney | Jeffrey4l: shoudl it merge or do you have more changes? | 18:10 |
| Jeffrey4l | sean-k-mooney, i will -1 WIP for that PS. | 18:10 |
| kbaegis | http://hastebin.com/ijefoqeyov.rb | 18:11 |
| Pavo | ok after I did reconfigure I am able to telnet into VIP on port 80 | 18:11 |
| Pavo | but web not working still | 18:11 |
| kbaegis | Pavo you have to have an adminrc. $(kolla-ansible post-deploy; source /etc/kolla/adminrc.sh; openstack role create _member) | 18:11 |
| Jeffrey4l | sean-k-mooney, I -2 for my former PS. could u remove your -1? | 18:12 |
| kbaegis | Yes, horizon is broken on master. IDK how, but troubleshooting now. | 18:12 |
| sean-k-mooney | Jeffrey4l: sure will do | 18:12 |
| Jeffrey4l | thanks. | 18:12 |
| kbaegis | images_panel is not set | 18:13 |
| kbaegis | I don't know django well enough to fix | 18:13 |
| kbaegis | Frustrating. I can actually create VMs and networks with 3.0.0 | 18:16 |
| kbaegis | 2.0.1 not so much | 18:16 |
| kbaegis | VM creation is broken | 18:16 |
| kbaegis | So I get to choose between a cli which works or a gui which doesn't | 18:16 |
| kbaegis | Mixing 2.0.1 in gives 404s | 18:19 |
| kbaegis | Hmm. Maybe I can use docker inspect and manually mount the older image | 18:20 |
| kbaegis | Rather than going through kolla for it | 18:20 |
| Pavo | kbaegis is it _member or _member_? | 18:21 |
| kbaegis | _member_ | 18:22 |
| Pavo | ok | 18:22 |
| Pavo | [root@deployer kolla]# openstack role list | 18:22 |
| Pavo | +----------------------------------+------------------+ | 18:22 |
| Pavo | | ID | Name | | 18:22 |
| Pavo | +----------------------------------+------------------+ | 18:22 |
| Pavo | | 2e0dfa24d69e4e23a9db1c54f6ae721e | admin | | 18:22 |
| Pavo | | 5cb448aeae8c4aab96dbf2a262fe713d | heat_stack_owner | | 18:22 |
| Pavo | | 9fe2ff9ee4384b1894a90878d3e92bab | _member_ | | 18:22 |
| Pavo | | bda3422929494996a957fd00ca48f5e0 | heat_stack_user | | 18:22 |
| Pavo | +----------------------------------+------------------+ | 18:22 |
| kbaegis | use hastebin | 18:22 |
| Pavo | mybad | 18:23 |
| kbaegis | It exists for you | 18:23 |
| kbaegis | Yeah, you're good | 18:23 |
| Pavo | yeah still web not working | 18:23 |
| Pavo | but can still telnet into the VIP and port 80 | 18:23 |
| *** fragatin_ has quit IRC | 18:24 | |
| *** fragatina has joined #openstack-kolla | 18:24 | |
| Pavo | logs still show a ton of python errors | 18:26 |
| Pavo | https://paste.pound-python.org/show/DN4CJtgwXOcJnVzPgO3l/ | 18:26 |
| Pavo | and seems it all related to KeyError: 'images_panel' | 18:27 |
| kbaegis | try this, if you have a 2.0.1 image handy | 18:27 |
| kbaegis | docker run -v /etc/localtime:/etc/localtime -v /var/lib/docker/volumes/kolla_logs/_data:/var/log/kolla -v /etc/kolla/horizon:/var/lib/kolla/config_files -e KOLLA_SERVICE_NAME=horizon -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS -e PATH=/var/lib/kolla/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -e KOLLA_BASE_DISTRO=centos -e 'PS1=$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ ' 29c891d46b46 | 18:27 |
| Pavo | yeah I would have to build 2.0.2 image | 18:27 |
| kbaegis | your image id should be whorizon | 18:27 |
| Pavo | oops | 18:27 |
| kbaegis | umm | 18:27 |
| kbaegis | lol | 18:27 |
| kbaegis | *horizon | 18:27 |
| Pavo | I mean 2.0.1 | 18:27 |
| Pavo | all my images are 3.0.0 | 18:27 |
| kbaegis | kolla-build horizon will only build that one image | 18:28 |
| Pavo | yeah how do I set only 2.0.1 | 18:28 |
| *** berendt has joined #openstack-kolla | 18:28 | |
| *** berendt has quit IRC | 18:28 | |
| kbaegis | kolla-build.conf | 18:29 |
| kbaegis | Forget where that's documented | 18:30 |
| Pavo | I remember seeing it somewhere just can't find it again lol | 18:31 |
| *** Jeffrey4l has quit IRC | 18:31 | |
| kbaegis | http://docs.openstack.org/developer/kolla/image-building.html | 18:31 |
| Pavo | --base-tag maybe | 18:31 |
| sdake | pavo to get an openrc file run kolla-ansible post-deploy | 18:32 |
| Pavo | I have that | 18:32 |
| Pavo | trying to figure out how to build a 2.0.1 horizon image | 18:32 |
| Pavo | instead of 3.0.0 | 18:32 |
| Pavo | yeah --base-tag it is | 18:33 |
| Pavo | kolla-build --base-tag 2.0.1 horizon | 18:34 |
| Pavo | well nvm | 18:34 |
| Pavo | ERROR:kolla.image.build.base:Tag 2.0.1 not found in repository docker.io/library/centos | 18:34 |
| *** michauds has quit IRC | 18:34 | |
| sdake | pavo do following | 18:34 |
| *** fragatina has quit IRC | 18:35 | |
| sdake | docker pull kolla/horizon:2.0.2 | 18:35 |
| kbaegis | sdake I'm trying to run the image manually from docker | 18:35 |
| kbaegis | sdake could use some help with the specifics | 18:35 |
| sdake | good luck with that one :) | 18:35 |
| kbaegis | only thing I keep getting an error on: make_sock: could not bind to address 192.168.254.3:80 | 18:36 |
| kbaegis | Nothing bound to 80 | 18:36 |
| sdake | run it as root | 18:36 |
| kbaegis | I am | 18:36 |
| sdake | -u root iirc | 18:36 |
| sdake | netstat -nl | grep 80 | 18:36 |
| kbaegis | uid 0 | 18:37 |
| Pavo | Error: image kolla/horizon:2.0.2 not found | 18:37 |
| kbaegis | No matches | 18:37 |
| kbaegis | no matches on | 18:37 |
| kbaegis | netstat -nl | grep 80 | 18:37 |
| sdake | pavo thta may not be the exeact syntx | 18:37 |
| kbaegis | Should be 2.0.1 | 18:37 |
| sdake | pavo try docker.io/kolla/centos-binary-horizon:2.0.1 | 18:37 |
| sdake | or remoe the docker.io | 18:37 |
| sdake | my tests machines are not functional atm because of a rebuild | 18:38 |
| sdake | so i can't get the exact command or you atm | 18:38 |
| sdake | someone else can probby help wit h that | 18:38 |
| *** dwalsh has quit IRC | 18:40 | |
| Pavo | yeah that was it, pulling both 2.0.1 and 2.0.2 | 18:40 |
| *** dwalsh has joined #openstack-kolla | 18:41 | |
| ntpttr___ | hey inc0, question about upgrades | 18:43 |
| inc0 | go ahead | 18:44 |
| ntpttr___ | inc0: I'm running cinder on mitaka, do I need to change any configs before running kolla-ansible upgrade to tell it I want it to be master? | 18:44 |
| ntpttr___ | where do I tell it what to upgrade to? | 18:44 |
| inc0 | ntpttr___, so you need to have images of master built | 18:44 |
| ntpttr___ | inc0: ah okay that makes sense | 18:44 |
| inc0 | in globals.yml you have openstack_release where you setup a tag | 18:45 |
| ntpttr___ | inc0: I actually had it set in kolla-build.conf | 18:45 |
| inc0 | also, keep in mind that kolla-master plays should be used | 18:45 |
| ntpttr___ | inc0: if I change that and rebuild is that enough? | 18:45 |
| ntpttr___ | inc0: yeah I have kolla on master | 18:45 |
| inc0 | no, you need both | 18:45 |
| inc0 | kolla-build in mitaka will tag your images as 2.0.2 | 18:45 |
| inc0 | kolla-build with master will tag them as 3.0.0 | 18:46 |
| inc0 | and in globals you set it up, which ones | 18:46 |
| ntpttr___ | inc0: but in kolla-build.conf I have the source for cinder set to a mitaka repo | 18:46 |
| inc0 | that being said, defaults should work | 18:46 |
| ntpttr___ | why wouldn't that be enough? | 18:46 |
| *** michauds has joined #openstack-kolla | 18:46 | |
| inc0 | then you need to change this to master and rebuild with different tag | 18:46 |
| inc0 | well, you can just rebuild | 18:46 |
| inc0 | technically | 18:46 |
| ntpttr___ | inc0: sorry I'm not sure I understand the tags | 18:47 |
| inc0 | docker images, there is tag there | 18:47 |
| inc0 | do docker images | 18:47 |
| ntpttr___ | is that a way for having multiple built images at different versions in my registry? | 18:47 |
| inc0 | yes | 18:47 |
| ntpttr___ | inc0: ah yes I see | 18:48 |
| ntpttr___ | they're all at 3.0.0 | 18:48 |
| inc0 | duh...it's a damn bug | 18:48 |
| ntpttr___ | I'll just destroy them and rebuild | 18:48 |
| ntpttr___ | with master cinder | 18:48 |
| inc0 | no, don't need to destroyu | 18:48 |
| inc0 | just rebuild with kolla master | 18:48 |
| inc0 | kolla-build | 18:49 |
| ntpttr___ | inc0: okay | 18:49 |
| ntpttr___ | It's running, I'll let you know what happens | 18:49 |
| ntpttr___ | the only difference then is that I changed the cinder source to point to master | 18:49 |
| Pavo | if you don't set a tag what does it default to kolla-master? | 18:49 |
| Pavo | because it built all my images as 3.0.0 | 18:50 |
| sdake | pavo yup | 18:50 |
| sdake | it defaults to that | 18:50 |
| Pavo | ok | 18:50 |
| sdake | really depends if you are running rom git or have kolla pip installed :) | 18:50 |
| Pavo | from git | 18:50 |
| sdake | do you have kolla pip installed as well? | 18:51 |
| Pavo | grr I did a kolla-ansible upgrade and it used 3.0.0 horizon | 18:51 |
| Pavo | sdake I just did what the docs say | 18:51 |
| sdake | kolla defaults to looking at containers/ansible (if installed via pip which is what the docs say) rather then your cwd | 18:51 |
| sdake | this means your not using whateverr changes you have made in git.. | 18:52 |
| sdake | sorry loooking at the pip installed containers/ansible | 18:52 |
| Pavo | I did a git clone https://git.openstack.org/openstack/kolla and a pip install kolla/ | 18:52 |
| sdake | right - pip remove kolla | 18:52 |
| sdake | and your good to go | 18:52 |
| sdake | pip installed? kolla uses that - even if you have a git checout | 18:53 |
| inc0 | sdake, even mitaka tags 3.0 | 18:53 |
| sdake | it is fantastically frustrating that nobody has fixed that yet | 18:53 |
| inc0 | which is bug | 18:53 |
| sdake | no its not a bug | 18:53 |
| Pavo | so that means I need to do a rebuild of the images? | 18:53 |
| sdake | it is using code from the pip install | 18:53 |
| inc0 | ehh | 18:53 |
| sdake | pavo depends what you changed | 18:53 |
| Pavo | I just did a pipi uninstall kolla | 18:54 |
| sdake | the pip install = 3.0.0 | 18:54 |
| sdake | it may be remove not uninstall | 18:54 |
| sdake | not certain | 18:54 |
| Pavo | tried remove and said unknow cmd | 18:54 |
| sdake | did uninstall work? | 18:54 |
| Pavo | had to do uninstall | 18:54 |
| Pavo | yeah | 18:54 |
| sdake | they are the sam ehting | 18:54 |
| sdake | i cant' ever remember which one it is ;) | 18:54 |
| Pavo | lol | 18:54 |
| Pavo | anyways.... | 18:55 |
| sdake | why dont you tell me what your trying to do | 18:55 |
| Pavo | probably best to rebuild images then | 18:55 |
| sdake | and i can guide from there | 18:55 |
| sdake | if you followed the instructions, you are using master | 18:55 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Modifying Keystone bootstrap to use secrets https://review.openstack.org/356098 | 18:56 |
| Pavo | well horizon is all mesed up, so I tried to redeploy using kolla-ansible upgrade to deploy horizon 2.0.2 but it still used 3.0.0 | 18:56 |
| sdake | ok to do that is easy | 18:56 |
| sdake | there are a couple options | 18:56 |
| sdake | actually one option | 18:56 |
| sdake | you need to get that horizon 2.0.2 version in your registry first step | 18:56 |
| Pavo | ok I did a pull from docker.io and have it on deployer images list | 18:57 |
| Pavo | how do I get it pushed to my registry | 18:57 |
| Pavo | local registry | 18:57 |
| Pavo | as remove 3.0.0 from registry | 18:57 |
| Pavo | and I mean | 18:58 |
| sdake | https://github.com/openstack/kolla/blob/master/ansible/roles/horizon/defaults/main.yml#L8 | 18:58 |
| sdake | the second step is to set horizon_tag: "2.0.2" in your globals.yml | 18:58 |
| sdake | pavo docker push will do it - but I don't recall exact syntax needed | 18:59 |
| Pavo | what section does that go in? | 18:59 |
| sdake | and as I said, my machines are down and out | 18:59 |
| sdake | pavo there are no sections in globals.yml - its a flat file | 18:59 |
| Pavo | so.... horizon_tag:"2.0.2" anywhere in globals.yml? | 19:00 |
| sdake | pavo if its on the deployer images list it doesn't need to be in the registry | 19:00 |
| sdake | as its already pre-cached now | 19:00 |
| sdake | pavo roger | 19:00 |
| sdake | then run cleanup scripts (but not cleanup-images) | 19:00 |
| *** srwilkers has quit IRC | 19:00 | |
| sdake | make sure horizon 2.0.2 is on every node in your cluster | 19:00 |
| sdake | I think that should work | 19:00 |
| sdake | but not certain | 19:00 |
| Pavo | ok | 19:00 |
| sdake | you may have to rename it | 19:01 |
| sdake | to match that of your registry | 19:01 |
| sdake | another option is to set horizon_image: "whatever-its-called-from-docker-images" | 19:01 |
| sdake | this last option is what i'd do | 19:02 |
| sdake | so two globals.yml overrides | 19:02 |
| sdake | horizon_tag: "2.0.2" | 19:02 |
| sdake | horizon_image: "docker.io/kolla/centos-source-horizon" | 19:02 |
| sdake | or whatever it was called :) | 19:02 |
| sdake | or maybe its just kolla/centos-source-horizon | 19:03 |
| sdake | you will have to play around with it a bit | 19:03 |
| Pavo | this si what I have in deployer | 19:03 |
| Pavo | https://paste.pound-python.org/show/9EnSVblpCLEM3QMmv7pK/ | 19:03 |
| Pavo | how do I get the kolla/centos-binary-horizon into my registry? | 19:04 |
| sdake | cool | 19:04 |
| sdake | pavo I don't have dockr in front of me | 19:04 |
| sdake | so I can't tell you the command to type | 19:04 |
| sdake | you can either read the internets or how to push | 19:04 |
| Pavo | ok np | 19:04 |
| sdake | or follow this other direction i provide | 19:04 |
| sdake | which is set horizon_image: "kolla/centos-binary-horizon" | 19:04 |
| sdake | in globals.yml | 19:05 |
| sdake | and make sure you have pulled it on all nodes | 19:05 |
| sdake | not just the deploy node | 19:05 |
| sdake | when you deploy, docker pulls from the registry to the local machine and stores in teh "docker cache" | 19:05 |
| sdake | when you do a docker pull the image also gets stored in the docker cache | 19:05 |
| sdake | if its there its there - no sense pulling it twice ;) | 19:06 |
| sdake | understand or have further questions? | 19:07 |
| Pavo | I think I understand | 19:07 |
| sdake | sweet | 19:08 |
| *** tummy has quit IRC | 19:12 | |
| *** fragatina has joined #openstack-kolla | 19:13 | |
| *** fragatina has quit IRC | 19:13 | |
| *** fragatina has joined #openstack-kolla | 19:13 | |
| *** sdake_ has joined #openstack-kolla | 19:14 | |
| *** sdake has quit IRC | 19:14 | |
| jrich523 | so... i am trying to build a compute node following these directions, http://docs.openstack.org/mitaka/install-guide-rdo/nova-compute-install.html | 19:15 |
| jrich523 | however its having issues connecting to RabbitMQ, 3116 ERROR oslo.messaging._drivers.impl_rabbit [req-82caf02c-a17a-4abd-b247-87e37de81b9d - - - - -] AMQP server on 192.168.11.30:5672 is unreachable: | 19:15 |
| *** sdake_ has quit IRC | 19:15 | |
| jrich523 | from what i can tell (netstat -anp|grep 5672) thats the correct IP, 30 is the controller host, but the VIP is 31, however 31 doesnt have port 5672 open, so i used 30 | 19:16 |
| *** sdake has joined #openstack-kolla | 19:16 | |
| Pavo | ok I am gonna rebuild my deployer from the beginning | 19:16 |
| Pavo | gonna only do what this site says http://docs.openstack.org/developer/kolla/quickstart.html | 19:16 |
| jrich523 | it wont work | 19:17 |
| jrich523 | unless it was very recently updated | 19:17 |
| Pavo | then what guide to use? git? | 19:17 |
| jrich523 | well, there isnt a guide | 19:17 |
| jrich523 | the only reason i got any of it working was because i gave zhugaoxiao ssh access to my VM | 19:17 |
| jrich523 | im fairly certain i couldnt reproduce what he did | 19:18 |
| jrich523 | even tho i watched all of it and took notes | 19:18 |
| jrich523 | he was extremely helpful, however, bit of a language barrier | 19:18 |
| jrich523 | i recall a bit issue being the ansible version (1.9.4 is ideal) | 19:20 |
| jrich523 | also changing the branch (git checkout stable/mitaka) | 19:20 |
| Pavo | CentOS and RHEL package Ansible >2.0 which is suitable for use with Kolla. | 19:20 |
| jrich523 | thats what i used at first | 19:21 |
| jrich523 | didnt get far | 19:21 |
| Pavo | hmm ok | 19:21 |
| jrich523 | it was all a bit of a blur, just telling you what i recall that varied from the directions | 19:21 |
| Pavo | how about this guide http://egonzalez.org/openstack-kolla-deployment-from-rdo-packages/ | 19:22 |
| jrich523 | i think outside of that it came down to modifications to globals.yml | 19:22 |
| jrich523 | actually | 19:23 |
| jrich523 | that looks good | 19:23 |
| jrich523 | seems mostly the same | 19:23 |
| jrich523 | but it def takes in to account the branch change, which is huge | 19:23 |
| jrich523 | he uses the most current ansible, but, who knows what version was out when he did this | 19:24 |
| Pavo | says April 24th | 19:24 |
| sdake | pavo wait | 19:25 |
| Pavo | ? | 19:25 |
| sdake | here is the deal guys | 19:25 |
| sdake | we are super busy with alot of stuff and the documentation is deficient | 19:25 |
| sdake | I get it, and I am working on getting it fixed | 19:25 |
| jrich523 | not complaining, just trying to work through it ) | 19:26 |
| jrich523 | :) | 19:26 |
| sdake | pavo - your goals are to develop for kolla - then use masater | 19:26 |
| Pavo | yeah me too, its completely understandable | 19:26 |
| jrich523 | actually after i get this setup, i was going to build from scratch again, and take better notes, and perhaps update the docs | 19:26 |
| sdake | pavo you said you wanted to learn about openstack programming | 19:26 |
| sdake | by using msater some things will be broken from time to time | 19:26 |
| sdake | the core team is here to help smooth over the rough spots for new devs coming up to speed on the code base | 19:27 |
| sdake | and get their first deployment done | 19:27 |
| Pavo | umm don't think that was me, I know openstack pretty well, just trying to learn how kolla does it, we currently use packstack to deploy openstack but seems like deploying as containers is more efficent | 19:27 |
| sdake | repaving the deployment host wont help in this case | 19:27 |
| sdake | it sounds like horizon has a legitimate defect | 19:27 |
| sdake | i don't know the status of this problem - but jeffrey4l submitted a NEW patch for this _member_ thing | 19:27 |
| jrich523 | yeah i will say, when i did horizon it was super easy | 19:27 |
| sdake | pavo roger | 19:28 |
| sdake | pavo sorry hard to keep eveyrone straight :) | 19:28 |
| Pavo | its all good | 19:28 |
| Pavo | I will provide feedback on what I learn | 19:28 |
| jrich523 | if you're just looking to get a feel for it, then using stable/mitaka might help with that | 19:28 |
| sdake | so if the problem is now "everything works except horizon" | 19:28 |
| sdake | let me help you with that | 19:29 |
| *** srwilkers has joined #openstack-kolla | 19:29 | |
| sdake | pavo if you don't plan to develop for kolla but only want to eval, I'd highly recommend using stable/mitaka | 19:34 |
| sdake | for which there are no instructions | 19:34 |
| Pavo | well I can start dev after I get the complete understanding of how it works | 19:34 |
| Pavo | probably would be best route | 19:35 |
| Pavo | route | 19:35 |
| sdake | well if you want to get master rolling let me knwo | 19:35 |
| sdake | if you want to get stable/mitaka rolling let me know | 19:35 |
| Pavo | will do | 19:35 |
| sdake | its all pretty easy | 19:35 |
| sdake | I don't know what the deal is with this horizon problem | 19:36 |
| sdake | its neews to me as of today | 19:36 |
| sdake | so probably an upstream regression | 19:36 |
| sdake | i know starting from scratch can be a best practice in many cases, but not in this case;-) | 19:36 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP- openvswitch ip https://review.openstack.org/356114 | 19:39 |
| *** bootsha has quit IRC | 19:42 | |
| kbaegis | Well with gentoo, I can't use mitaka release | 19:43 |
| kbaegis | So I'm stuck with you on master sdake :) | 19:43 |
| *** tummy has joined #openstack-kolla | 19:43 | |
| kbaegis | 3.0.0 is working perfectly- except for horizon | 19:43 |
| sdake | mitaka broke n on gentoo? | 19:43 |
| kbaegis | The CLI is good enough for me | 19:43 |
| kbaegis | Yes | 19:43 |
| kbaegis | The libvirt wasn't working at all. Kept breaking on vm create | 19:44 |
| kbaegis | deploys fine | 19:44 |
| *** tummy has quit IRC | 19:44 | |
| kbaegis | But can't create anything | 19:44 |
| kbaegis | Also ovs had issues | 19:44 |
| *** bootsha has joined #openstack-kolla | 19:44 | |
| kbaegis | I also am using the 4.8 kernel though :) | 19:44 |
| kbaegis | It'd detect my node w/ 32 available cores, but wouldn't schedule/create | 19:45 |
| kbaegis | retry invoked after first failure | 19:45 |
| sdake | does anyone have a backtrace of this horizon problem ? | 19:46 |
| kbaegis | pasted it in a while back | 19:46 |
| sdake | mind pasting again - i dont keep a large scroll buffer and dc often because of vpn nonsene | 19:47 |
| kbaegis | http://hastebin.com/ijefoqeyov.rb | 19:47 |
| kbaegis | images_panel not set | 19:48 |
| Pavo | here ya go sdake https://paste.pound-python.org/show/JhzdyKGGs7XL6srQ6xNJ/ | 19:48 |
| *** bootsha has quit IRC | 19:49 | |
| sdake | a key error means there is no key like that in the dictionary | 19:49 |
| sdake | moment let me ask on #openstack-horizon | 19:49 |
| ntpttr___ | has anyone here who has deployed cinder w/ ceph using kolla run into an issue where when you ask to create a backup it hangs in 'creating' forever? | 19:51 |
| *** nihilifer has quit IRC | 19:54 | |
| *** nihilifer has joined #openstack-kolla | 19:56 | |
| *** nihilifer has quit IRC | 20:00 | |
| sdake | ok so this horizon problem looks like a kolla usage error | 20:03 |
| sdake | apparently we are doing it wrong | 20:03 |
| sdake | i dont have enough details to fix it | 20:03 |
| sdake | and i dont know why it just started happening | 20:03 |
| sdake | but it will get fixed | 20:04 |
| sdake | until then, hang tight and try out other things :) | 20:04 |
| *** srwilkers has quit IRC | 20:04 | |
| *** matrohon has joined #openstack-kolla | 20:09 | |
| *** vhosakot has quit IRC | 20:10 | |
| kbaegis | Will do :) TY sdake! | 20:11 |
| *** vhosakot has joined #openstack-kolla | 20:11 | |
| ntpttr___ | sdake: hey is there an option to turn on debugging in the logs in globals.yml? | 20:18 |
| ntpttr___ | nvm found it | 20:20 |
| *** inc0 has quit IRC | 20:27 | |
| Mech422 | sdake: well poop - ticketmaster managed to screw me out of $320 worth of Adele tickets tonight | 20:30 |
| sdake | too hot anyway :) | 20:30 |
| Mech422 | sdake: heh - well, beware - if you get any tickets from them that are 'tied' to your credit card...they have no process for dealing with 're-issued' cards | 20:31 |
| kbaegis | Yeah in Phoenix | 20:31 |
| sdake | wasn't that at talking stick anyway? | 20:32 |
| sdake | that place would be a mad house to have a concern in | 20:33 |
| Mech422 | sdake: yeah - talking stick | 20:33 |
| sdake | conceert in | 20:33 |
| Mech422 | sdake: yeah? I've never actually gone... the tickets were an xmas present for the gf. | 20:33 |
| sdake | oh nice | 20:33 |
| Mech422 | yeah :-/ | 20:33 |
| sdake | xmas early this year :) | 20:33 |
| Mech422 | sdake: nah - the tickets go on sale like 9 months before the concert, then they don't know what to do if your bank issues you a new card | 20:34 |
| Mech422 | B.o.A. had a breach like 6 months ago and re-issued my card | 20:34 |
| *** huikang has quit IRC | 20:35 | |
| *** jtriley has quit IRC | 20:39 | |
| *** rstarmer has quit IRC | 20:40 | |
| *** lrensing_ has quit IRC | 20:41 | |
| *** jamesden_ has joined #openstack-kolla | 20:41 | |
| *** jeh has quit IRC | 20:44 | |
| *** jamesdenton has quit IRC | 20:44 | |
| *** kbaegis has quit IRC | 20:48 | |
| *** jeh has joined #openstack-kolla | 20:49 | |
| *** jeh has quit IRC | 20:49 | |
| *** jeh has joined #openstack-kolla | 20:49 | |
| *** lyncos has joined #openstack-kolla | 20:52 | |
| *** dwalsh has quit IRC | 20:53 | |
| *** daneyon has joined #openstack-kolla | 20:55 | |
| *** kbaegis has joined #openstack-kolla | 20:55 | |
| *** matrohon has quit IRC | 20:55 | |
| *** jeh has quit IRC | 20:56 | |
| kbaegis | Step one, Fire BoA and get a credit union account | 20:57 |
| mark-casey | step two. pay fees :D | 20:59 |
| *** daneyon has quit IRC | 20:59 | |
| *** jeh has joined #openstack-kolla | 21:01 | |
| *** jeh has quit IRC | 21:01 | |
| *** jeh has joined #openstack-kolla | 21:01 | |
| *** ad_rien_ has joined #openstack-kolla | 21:07 | |
| *** bootsha has joined #openstack-kolla | 21:08 | |
| kbaegis | Credit union fees? | 21:13 |
| kbaegis | I'm a member of two and I've never paid a fee. I had an account at wells fargo for a year though, and they have tons of fees | 21:14 |
| kbaegis | sdake any luck from the horizon team? | 21:18 |
| *** matrohon has joined #openstack-kolla | 21:18 | |
| kbaegis | We're copying from devstack as a short-term hack? :) lol | 21:18 |
| kbaegis | I like it | 21:18 |
| kbaegis | Oh I see | 21:20 |
| kbaegis | it was a copy-replace rather than update | 21:20 |
| *** egonzalez90 has joined #openstack-kolla | 21:24 | |
| *** egonzalez90 has quit IRC | 21:24 | |
| kbaegis | Hmm | 21:28 |
| kbaegis | I guess I can just edit /etc/kolla/horizon/local_settings in the interrim | 21:29 |
| imcsk8 | Pavo: i was reading the backlog, i was recently playing with kolla. i was very impressed | 21:29 |
| Pavo | lol did a rebuild of my deployer node now getting something weird | 21:29 |
| Pavo | https://paste.pound-python.org/show/6U1sDSYedrSSMDaDIKXG/ | 21:29 |
| *** ad_rien_ has quit IRC | 21:29 | |
| kbaegis | Pavo looks like dhcp failed on your local network Pavo | 21:30 |
| Pavo | on what though? | 21:30 |
| Pavo | there is no ''169.254 anywhere | 21:30 |
| kbaegis | Whatever interface you configured in global | 21:30 |
| Pavo | in the global.yml | 21:30 |
| kbaegis | It may be a secondary IP | 21:30 |
| kbaegis | Oh, that's fine | 21:30 |
| kbaegis | It's probably grabbing the actual configured address | 21:31 |
| kbaegis | do ifconfig/ip addr show | 21:31 |
| Pavo | I'll reboot and retry prechecks | 21:31 |
| kbaegis | Don't reboot | 21:31 |
| Pavo | ok | 21:31 |
| Pavo | https://paste.pound-python.org/show/MwPpvSF67ysgx7CLkxqD/ | 21:31 |
| kbaegis | just do ifconfig 0.0.0.0 up for your neutron | 21:31 |
| kbaegis | neutron interface | 21:31 |
| kbaegis | I bet if you do ip addr show dev eth0; ip addr show dev eth1 | 21:32 |
| Pavo | I don't have anything deployed yet, that error was from prechecks | 21:32 |
| kbaegis | it will show up with a 169 address | 21:32 |
| *** matrohon has quit IRC | 21:32 | |
| Pavo | I just checked all nodes ip addr and they all have a IP on eth0 and eth1 | 21:33 |
| Pavo | what I just posted was on deployer node | 21:33 |
| kbaegis | And it's on the same subnet? | 21:33 |
| imcsk8 | Pavo: did you se kolla_internal_vip_address in /etc/kolla/globals.yml? | 21:33 |
| Pavo | node2 https://paste.pound-python.org/show/k9Vo64r4E4K0kJNcqyD0/ | 21:33 |
| kbaegis | Well that's either coming from an interface or it's coming from globals | 21:33 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP- openvswitch ip https://review.openstack.org/356114 | 21:34 |
| Pavo | yeah I set kolla_internal_vip_address in globals to 192.168.1.254 | 21:34 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP- openvswitch ip https://review.openstack.org/356114 | 21:35 |
| *** rhallisey has quit IRC | 21:41 | |
| *** bootsha has quit IRC | 21:43 | |
| Pavo | any luck with your horizon kbaegis | 21:44 |
| kbaegis | yes and no | 21:45 |
| kbaegis | As we learned on #openstack-horizon, kolla had been overwriting the previous config with a statically generated one | 21:46 |
| imcsk8 | Pavo: you can run kolla-ansible -v precheck to get more info | 21:46 |
| kbaegis | So it's a matter of merging ~300 functional lines from https://github.com/openstack/horizon/blob/master/openstack_dashboard/local/local_settings.py.example with the local_settings | 21:46 |
| Pavo | ok | 21:46 |
| Pavo | ouch | 21:46 |
| *** matrohon has joined #openstack-kolla | 21:46 | |
| kbaegis | I believe sdake is working on the necessary scripts now | 21:47 |
| *** jamesden_ has quit IRC | 21:47 | |
| kbaegis | I'm running reconfigure, will do the merge and share | 21:47 |
| kbaegis | You should be able to dump some text into /etc/kolla/horizon/local_settings to fix this | 21:47 |
| kbaegis | really wish kolla-ansible reconfigure supported individual containers | 21:49 |
| kbaegis | oh well :) | 21:49 |
| Pavo | ok looks like there is something wrong with the precheck script for this section of the precheck | 21:50 |
| Pavo | https://paste.pound-python.org/show/jz1qqMIX2xAV5nusrkAD/ | 21:50 |
| *** bootsha has joined #openstack-kolla | 21:51 | |
| kbaegis | Well, I don't know how to read that | 21:51 |
| kbaegis | But it's possible that you have multiple IPs assigned | 21:51 |
| kbaegis | which is why you use the ip addr show dev <if> command rather than ifconfig | 21:52 |
| kbaegis | ifconfig is simpler/less functional than full iproute2 | 21:52 |
| kbaegis | That's what I get from "non-iteratable" | 21:52 |
| Pavo | I don't have net-tools installed so no ifconfig | 21:52 |
| Pavo | using ip addr cmd | 21:52 |
| kbaegis | Huh | 21:53 |
| kbaegis | Well then I don't know :) | 21:53 |
| kbaegis | The most absurd solution I can think of is to rename all devs w/ udev rules and reinput the values in global | 21:54 |
| Pavo | this is what node1 shows | 21:54 |
| Pavo | https://paste.pound-python.org/show/aJyZJo3llYWSXyYT4JgE/ | 21:54 |
| kbaegis | Yup. All matches up. 192.168.1/24 192.168.188/24 | 21:54 |
| Pavo | have no clue where that 169.254 is coming from on prechecks | 21:55 |
| Pavo | not anywhere in globals or inventory | 21:55 |
| *** michauds has quit IRC | 22:02 | |
| *** chrichip has quit IRC | 22:13 | |
| *** bootsha has quit IRC | 22:25 | |
| *** sdake has quit IRC | 22:27 | |
| *** dwalsh has joined #openstack-kolla | 22:27 | |
| *** sdake has joined #openstack-kolla | 22:28 | |
| kbaegis | Well, my merge with vimdiff didn't work | 22:33 |
| *** rstarmer has joined #openstack-kolla | 22:33 | |
| sdake | kbaegis working on the horizon problem you mean? | 22:33 |
| kbaegis | Yes | 22:35 |
| kbaegis | used vimdiff with https://github.com/openstack/horizon/blob/master/openstack_dashboard/local/local_settings.py.example and /etc/kolla/horizon/local_settings | 22:35 |
| kbaegis | manually overwrote local_settings and restarted the horizon container | 22:36 |
| sdake | kbaegis does that fix the eproblem? | 22:37 |
| sdake | I think there should be something more surgical here :) | 22:37 |
| kbaegis | It does not. :) | 22:37 |
| Pavo | during the image build do it use the host repo files for yum cmds or does it use default ones? | 22:40 |
| *** matrohon has quit IRC | 22:40 | |
| *** daneyon has joined #openstack-kolla | 22:43 | |
| *** thumpba has quit IRC | 22:46 | |
| sdake | pavo kolla knows nothing about a rpm or deb host repo except one master location | 22:46 |
| sdake | pavo - that can be reconfigured - but not trivial | 22:46 |
| Pavo | ok just wondering | 22:46 |
| sdake | it was designed to be reconfigured | 22:46 |
| sdake | atleast the from src part | 22:47 |
| sdake | from bin part - i think only oen mastaer repo | 22:47 |
| *** daneyon has quit IRC | 22:48 | |
| *** dave-mccowan has quit IRC | 22:48 | |
| *** thumpba has joined #openstack-kolla | 22:49 | |
| *** thumpba has quit IRC | 22:49 | |
| *** vhosakot has quit IRC | 22:56 | |
| *** chrichip has joined #openstack-kolla | 22:58 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla: Add crudini https://review.openstack.org/356155 | 22:58 |
| *** thumpba has joined #openstack-kolla | 22:59 | |
| *** thumpba has quit IRC | 22:59 | |
| *** thumpba has joined #openstack-kolla | 22:59 | |
| *** chrichip has quit IRC | 23:07 | |
| kbaegis | sdake Do you think you'll need a full day for that change? | 23:19 |
| *** jtriley has joined #openstack-kolla | 23:20 | |
| kbaegis | Longer? :) | 23:27 |
| kbaegis | I'm just curious | 23:27 |
| *** mbound has joined #openstack-kolla | 23:34 | |
| *** thumpba_ has joined #openstack-kolla | 23:37 | |
| imcsk8 | Pavo: https://bugs.launchpad.net/kolla/+bug/1571612 check comment #6 | 23:37 |
| openstack | Launchpad bug 1571612 in kolla "Prechecks fails when checking if kolla_internal_vip_address is in the same network as network_interface on all nodes" [High,Triaged] - Assigned to Hiroki Ito (hrito) | 23:37 |
| *** thumpba has quit IRC | 23:39 | |
| *** mbound has quit IRC | 23:39 | |
| *** jamesdenton has joined #openstack-kolla | 23:44 | |
| *** zhurong has joined #openstack-kolla | 23:49 | |
| openstackgerrit | Li Yingjun proposed openstack/kolla: Fix tar command path for magnum conductor using ubuntu https://review.openstack.org/356008 | 23:53 |
| *** kbaegis has quit IRC | 23:55 | |
| openstackgerrit | Li Yingjun proposed openstack/kolla: Fix tar command path for magnum conductor using ubuntu https://review.openstack.org/356008 | 23:55 |
| *** kbaegis has joined #openstack-kolla | 23:56 | |
| Pavo | thanks imcsk8 | 23:57 |
| kbaegis | Pavo | 23:57 |
| kbaegis | Did you figure it out? | 23:57 |
| kbaegis | The horizon bug? | 23:57 |
| Pavo | not yet | 23:57 |
| kbaegis | Oh :) | 23:57 |
| kbaegis | Just checking | 23:57 |
| Pavo | still waiting on images to finish building lol | 23:57 |
| Pavo | thats probably the longest part of this whole thing | 23:57 |
| mark-casey | Pavo: if this is for specific issues and you haven't seen it, look into just rebuilding one image or a few images | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
