| SamYaple | but yea anyway if you run `ceph-authtool -p -C /dev/null -g` | 00:00 |
|---|---|---|
| SamYaple | it will pump out a key thats note tied to the cluster | 00:00 |
| SamYaple | just the raw key | 00:00 |
| SamYaple | its a struct with a timestamp and then the random data | 00:01 |
| SamYaple | you can actually make it by hand without issue | 00:01 |
| *** goldyfruit has joined #openstack-kolla | 00:05 | |
| inc0 | SamYaple: I'm thinking we could do it next release | 00:10 |
| SamYaple | i think to deprecate you need a migration path in place. so if you get taht work done by queens, then yea | 00:11 |
| SamYaple | basically in queens people would need to be able to migrate to the new method | 00:12 |
| *** yangyapeng has quit IRC | 00:12 | |
| inc0 | well if we deprecate in queens, then by rocky we need to have migration path | 00:12 |
| inc0 | by deprecate in queens I mean queens still have fully functional L | 00:13 |
| *** rwsu has joined #openstack-kolla | 00:13 | |
| inc0 | but rocky have migration path | 00:13 |
| *** yangyapeng has joined #openstack-kolla | 00:13 | |
| inc0 | and requirement for upgrade Q->R would be to externalize ceph | 00:13 |
| SamYaple | inc0: you need the migration path for a full cycle though | 00:13 |
| SamYaple | nah you cant do that | 00:13 |
| inc0 | why tho? | 00:14 |
| SamYaple | policy? | 00:14 |
| SamYaple | i mean theres no *technical* reason | 00:14 |
| inc0 | well in any case we can still have ceph in Rocky | 00:14 |
| inc0 | I mean if we upgrade to L it's just gonna be bugfix | 00:14 |
| SamYaple | but you need a time period (a cycle) of old way depcreated with new way working | 00:14 |
| SamYaple | so if thats not there by Q, you cant remove til S | 00:15 |
| inc0 | so keeping ceph code for till S isn't issue | 00:15 |
| inc0 | I mean we can do it | 00:15 |
| SamYaple | as long as you have a migration path for a cycle before you remove the code you can call it whatever you want | 00:16 |
| inc0 | that was the plan | 00:16 |
| inc0 | if we get migration path in Q, all the better | 00:16 |
| SamYaple | 00:13:50 < inc0> and requirement for upgrade Q->R would be to externalize ceph | 00:16 |
| SamYaple | thats whats confusing me there | 00:16 |
| inc0 | well | 00:16 |
| inc0 | you have Q | 00:16 |
| inc0 | you want to upgrade to R | 00:16 |
| inc0 | since R is already released | 00:17 |
| inc0 | part of R is this magical playbook to move ceph from kolla to external | 00:17 |
| SamYaple | yea thats not proper deprecation | 00:17 |
| inc0 | run this magical playbook before you call kolla-ansible upgrade | 00:17 |
| *** jtriley has joined #openstack-kolla | 00:17 | |
| *** yangyapeng has quit IRC | 00:17 | |
| SamYaple | in that case you couldnt call it deprecated until R | 00:17 |
| SamYaple | with an S removal | 00:17 |
| inc0 | we can do it | 00:17 |
| inc0 | although you could argue that external ceph is already working | 00:18 |
| inc0 | because it is | 00:18 |
| SamYaple | ok let me rephrase, policy says you cant do that | 00:18 |
| SamYaple | migration path | 00:18 |
| inc0 | you need release of both working, external and kolla-supported | 00:18 |
| inc0 | and that's reality today | 00:18 |
| SamYaple | with a migration path | 00:18 |
| SamYaple | all in one release | 00:18 |
| inc0 | yeah I guess | 00:18 |
| inc0 | anyway, we can keep ceph inR | 00:18 |
| SamYaple | listen you do what you want, but its policy because otherwise you break people | 00:18 |
| inc0 | L is lts anyway | 00:18 |
| inc0 | I don't think you'll break in our case - it's not like you need to cause downtime for your cloud | 00:19 |
| inc0 | but we'll do what will be best for operators | 00:19 |
| inc0 | if that means keeping ceph code for one more release, we'll keep code for one more release | 00:20 |
| openstackgerrit | Heetae Ahn proposed openstack/kolla master: Add RHEL subscription registration https://review.openstack.org/495148 | 00:40 |
| *** huanxie has joined #openstack-kolla | 00:44 | |
| *** tovin07_ has joined #openstack-kolla | 00:47 | |
| *** kolla-slack has quit IRC | 01:00 | |
| *** kolla-slack has joined #openstack-kolla | 01:00 | |
| *** huanxie has quit IRC | 01:01 | |
| openstackgerrit | yanpeifei proposed openstack/kolla master: update "copy_links:true" ARG. https://review.openstack.org/514493 | 01:02 |
| *** kiennt26 has joined #openstack-kolla | 01:02 | |
| *** dave-mccowan has joined #openstack-kolla | 01:02 | |
| *** sbezverk has quit IRC | 01:06 | |
| *** sbezverk has joined #openstack-kolla | 01:07 | |
| *** yangyapeng has joined #openstack-kolla | 01:12 | |
| *** yangyapeng has quit IRC | 01:12 | |
| *** yangyapeng has joined #openstack-kolla | 01:13 | |
| *** daidv has joined #openstack-kolla | 01:16 | |
| *** jinke_ has joined #openstack-kolla | 01:19 | |
| *** hrw has joined #openstack-kolla | 01:19 | |
| *** Mratrend has joined #openstack-kolla | 01:24 | |
| *** zhubingbing has joined #openstack-kolla | 01:26 | |
| *** pramodrj07 has quit IRC | 01:34 | |
| *** MasterOfBugs has quit IRC | 01:34 | |
| *** salv-orlando has joined #openstack-kolla | 01:36 | |
| *** salv-orlando has quit IRC | 01:40 | |
| *** jtriley has quit IRC | 01:44 | |
| Mratrend | Hi guys! I have a question, How to use kolla-ansible to deploy neutron-metering-agent containers? globals.yml havent enable_neutron_metering_agent option | 01:53 |
| *** zhangfei has joined #openstack-kolla | 01:58 | |
| *** yingjun has joined #openstack-kolla | 02:00 | |
| Jeffrey4l | inc0, around? | 02:04 |
| *** yangyape_ has joined #openstack-kolla | 02:07 | |
| *** yangyapeng has quit IRC | 02:10 | |
| *** jtriley has joined #openstack-kolla | 02:12 | |
| *** zshi has quit IRC | 02:14 | |
| *** zshi has joined #openstack-kolla | 02:16 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Remove check flat network type task in ironic pre-check.yml https://review.openstack.org/513651 | 02:24 |
| inc0 | Jeffrey4l: now I am | 02:37 |
| Jeffrey4l | inc0, what kinds of issue did u see? | 02:37 |
| inc0 | so masber notices something that I tried to debug in gates | 02:37 |
| inc0 | mariadb failed to bootstrap | 02:37 |
| inc0 | but for him (and gates as it turns out) it was issue that keepalived didn't setup floating ip | 02:38 |
| Jeffrey4l | did u get any error message? | 02:38 |
| inc0 | I don't have env to reproduce | 02:38 |
| Jeffrey4l | ok. | 02:38 |
| inc0 | timeouts on mariadb bootsrap | 02:38 |
| inc0 | http://logs.openstack.org/79/512779/13/check/kolla-ansible-ubuntu-source-ceph/d609bd6/ <- look here | 02:39 |
| Jeffrey4l | so this happens in master? | 02:39 |
| inc0 | yeah, and possibly 5.0 | 02:40 |
| Jeffrey4l | hrm, allow address pair issue? | 02:41 |
| Jeffrey4l | vip can not be used in openstack directly. need configure through --all-address-pair | 02:41 |
| *** calebb has quit IRC | 02:41 | |
| inc0 | I know, in gates I used vxlan tunnel | 02:42 |
| Jeffrey4l | btw, deploy jobs is working now. https://review.openstack.org/513971 | 02:42 |
| inc0 | nice | 02:43 |
| Jeffrey4l | seems both keepalived run into master state. | 02:43 |
| inc0 | right | 02:43 |
| Jeffrey4l | so i still doubt the underlay openstack issue ( address pair ( | 02:44 |
| inc0 | well, no, I dont think so | 02:44 |
| Jeffrey4l | ok. i will check this today. | 02:45 |
| inc0 | I've setup vxlan tunnel spanning across nodes | 02:45 |
| inc0 | and used it for api_interface | 02:45 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Optimize zuul v3 jobs https://review.openstack.org/513612 | 02:45 |
| inc0 | that wou;dn't be affected by address pairs | 02:45 |
| Jeffrey4l | yes. | 02:45 |
| inc0 | double check it, and I'll try it tomorrow too | 02:45 |
| Jeffrey4l | i will. | 02:46 |
| inc0 | https://review.openstack.org/#/c/512779/19 also, if you have time, see if you can figure out what's wrong here | 02:48 |
| inc0 | I disabled keepalived, but mariadb still fails to bootstrap | 02:49 |
| inc0 | so it might not be keepalived after all... | 02:49 |
| *** heigao has joined #openstack-kolla | 02:50 | |
| Jeffrey4l | roger. | 02:51 |
| inc0 | thanks | 02:51 |
| *** calebb has joined #openstack-kolla | 02:51 | |
| inc0 | I'll look at that tomorrow, have a great day! | 02:51 |
| *** heigao has quit IRC | 02:52 | |
| *** heigao has joined #openstack-kolla | 02:52 | |
| Jeffrey4l | thanks. good night. | 02:54 |
| masber | I am going to deploy ceph using ceph-ansible and then I would like to deploy kolla and connect it to my ceph cluster as described here https://docs.openstack.org/kolla-ansible/latest/reference/external-ceph-guide.html | 02:58 |
| masber | my question is, who is going to deploy ceph clients? ceph-ansible or kolla? | 02:58 |
| masber | *who should | 02:59 |
| SamYaple | masber: the clients are baked into the images | 03:02 |
| SamYaple | masber: so you need to build your images with the clients | 03:03 |
| SamYaple | if you just want the clients on hosts outside of kolla, ceph-ansible can handle that for you | 03:03 |
| masber | SamYaple, ok, I guess prefer to deploy ceph client through kolla as I have no idea how to connect cinder/swift with the ceph clients... | 03:04 |
| masber | SamYaple, just curious, what is your preferred way of deploying ceph? though kolla or ceph-ansible? | 03:05 |
| masber | SamYaple, and have you noticed performance degradation by running ceph and compute on same nodes? | 03:05 |
| *** jinke_ has quit IRC | 03:07 | |
| *** jinke_ has joined #openstack-kolla | 03:08 | |
| *** yangyape_ has quit IRC | 03:08 | |
| *** yangyapeng has joined #openstack-kolla | 03:09 | |
| *** yangyape_ has joined #openstack-kolla | 03:10 | |
| spsurya | morning all | 03:12 |
| *** yangyapeng has quit IRC | 03:16 | |
| SamYaple | masber: hyperconveraged can impact performance yes. if you have enough nodes, seperating everything out is, im sure, prefered. | 03:21 |
| *** mdnadeem has joined #openstack-kolla | 03:27 | |
| *** lukepatrick has joined #openstack-kolla | 03:37 | |
| *** salv-orlando has joined #openstack-kolla | 03:37 | |
| openstackgerrit | Hanxi Liu proposed openstack/kolla-kubernetes master: Remove extra characters https://review.openstack.org/514511 | 03:39 |
| *** salv-orlando has quit IRC | 03:42 | |
| *** dave-mccowan has quit IRC | 03:47 | |
| *** goldyfruit has quit IRC | 03:53 | |
| *** heigao has quit IRC | 03:54 | |
| *** gkadam_ has joined #openstack-kolla | 03:56 | |
| *** newmember has joined #openstack-kolla | 04:07 | |
| *** yangyape_ has quit IRC | 04:24 | |
| *** yangyapeng has joined #openstack-kolla | 04:25 | |
| *** gnupyx has quit IRC | 04:45 | |
| *** skramaja has joined #openstack-kolla | 04:47 | |
| *** kiennt26 has quit IRC | 04:51 | |
| *** lukepatrick has quit IRC | 04:53 | |
| *** jtriley has quit IRC | 04:54 | |
| *** janki has joined #openstack-kolla | 04:55 | |
| *** salv-orlando has joined #openstack-kolla | 05:05 | |
| *** jinke_ has quit IRC | 05:12 | |
| *** dmellado has quit IRC | 05:21 | |
| *** dmellado has joined #openstack-kolla | 05:22 | |
| *** janki has quit IRC | 05:23 | |
| *** janki has joined #openstack-kolla | 05:23 | |
| *** janki has quit IRC | 05:28 | |
| kolla-slack | <egonzalez> @masber reading scroll back to your keepaalived issue, can you check keepalived logs, also ensure no other vrrp vid is not used in any other place in the same network https://github.com/openstack/kolla-ansible/blob/175c65e4900c9cc68917242fd7c9677fc32a2087/ansible/group_vars/all.yml#L118 | 05:29 |
| *** janki has joined #openstack-kolla | 05:30 | |
| kolla-slack | <masber> @egonzalez what's vrrp? | 05:31 |
| kolla-slack | <egonzalez> Thw protocol used by keepalived | 05:32 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible stable/pike: Optimize zuul v3 jobs https://review.openstack.org/514517 | 05:32 |
| *** lpetrut has joined #openstack-kolla | 05:33 | |
| kolla-slack | <masber> @egonzalez ok, I checked and the vid assigned is free in the network | 05:34 |
| pratapagoutham | kolla-slack: I could clear all the prechecks tasks but i have an issue in pulling the images present in docker-registry | 05:35 |
| pratapagoutham | command i used is : kolla-ansible pull -i <multi_node_file> and error it is throwing up is https://hastebin.com/dugimabaha.py | 05:35 |
| pratapagoutham | kolla-slack: any help ?? please | 05:35 |
| *** jtriley has joined #openstack-kolla | 05:37 | |
| *** dmellado has quit IRC | 05:37 | |
| SamYaple | pratapagoutham: the 5.0.0 images dont exist in dockerhub | 05:38 |
| SamYaple | youll need to build them yourself | 05:38 |
| *** egonzalez has joined #openstack-kolla | 05:39 | |
| pratapagoutham | SamYaple: yes i have built it with this "kolla-build --base ubuntu --type source --registry 172.16.69.71:5443 --push " | 05:40 |
| *** dmellado has joined #openstack-kolla | 05:40 | |
| SamYaple | ah i see that now | 05:41 |
| SamYaple | does `docker pull 172.16.69.71:5443/kolla/ubuntu-binary-kolla-toolbox:5.0.0` work for you? | 05:42 |
| SamYaple | oh wait, there is the issue | 05:42 |
| SamYaple | binary vs source | 05:42 |
| SamYaple | you built from source images (good) but you didnt set the type in globals.yaml for deploying with kolla-ansible | 05:42 |
| SamYaple | pratapagoutham: ^ | 05:42 |
| pratapagoutham | haan yes i have that to source in /etc/kolla/globals.yml | 05:44 |
| pratapagoutham | SamYaple: ^ | 05:44 |
| SamYaple | its definetely trying to pull binary images, so perhaps your value is getting overridden somewhere? | 05:45 |
| pratapagoutham | SamYaple: yes seems this solved the issue | 05:45 |
| SamYaple | pratapagoutham: awesome! good luck | 05:45 |
| SamYaple | nighttime for me | 05:46 |
| pratapagoutham | SamYaple: https://hastebin.com/exoweqakon.py but then this?? | 05:46 |
| egonzalez | pratapagaoutham, check docker py version installed. No docker-py should be present in the deploy host | 05:49 |
| egonzalez | Only docker and docker-pycreds | 05:49 |
| pratapagoutham | egonzalez: only docker is there | 05:51 |
| egonzalez | Hrm, maybe is insecure registey not set in docker systemd service | 05:51 |
| pratapagoutham | i installed it with pip install docker | 05:51 |
| *** vharsh has joined #openstack-kolla | 05:59 | |
| *** kevinz has quit IRC | 06:00 | |
| *** kevinz has joined #openstack-kolla | 06:00 | |
| *** tovin07_ has quit IRC | 06:01 | |
| spsurya | pbourke: can you please review https://review.openstack.org/#/c/425446/ | 06:06 |
| *** coolsvap has joined #openstack-kolla | 06:06 | |
| pratapagoutham | any one who can help me out with this error while pulling images "https://hastebin.com/exoweqakon.py " | 06:07 |
| *** zhangfei has quit IRC | 06:09 | |
| *** jtriley has quit IRC | 06:14 | |
| *** brad[] has quit IRC | 06:18 | |
| *** brad[] has joined #openstack-kolla | 06:19 | |
| *** Mratrend has quit IRC | 06:19 | |
| *** lpetrut has quit IRC | 06:21 | |
| *** egonzalez has quit IRC | 06:21 | |
| *** zhangfei has joined #openstack-kolla | 06:22 | |
| *** egonzalez has joined #openstack-kolla | 06:25 | |
| *** zhubingbing_ has joined #openstack-kolla | 06:26 | |
| korean101 | ERROR 2003 (HY000): Can't connect to MySQL server on '172.16.0.99' (113 \"No route to host\") | 06:27 |
| *** zhubingbing has quit IRC | 06:27 | |
| korean101 | hi. why my multinode deployment is wrong? | 06:29 |
| korean101 | got an ERRORs | 06:29 |
| *** tvignaud has joined #openstack-kolla | 06:30 | |
| masber | just double checking, if I decide to install ceph by myself, should I deploy jewel or kraken? | 06:31 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Optimize zuul v3 jobs https://review.openstack.org/513612 | 06:33 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible stable/pike: Optimize zuul v3 jobs https://review.openstack.org/514517 | 06:37 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible stable/pike: Move to zuul v3 in project jobs https://review.openstack.org/513608 | 06:41 |
| *** jbadiapa_ has joined #openstack-kolla | 06:41 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible stable/pike: Optimize zuul v3 jobs https://review.openstack.org/514517 | 06:41 |
| *** jbadiapa has quit IRC | 06:44 | |
| *** danardelean has joined #openstack-kolla | 06:45 | |
| *** zhangfei has quit IRC | 06:46 | |
| *** danardelean has quit IRC | 06:50 | |
| *** dardelean has joined #openstack-kolla | 06:53 | |
| *** gfidente has joined #openstack-kolla | 06:55 | |
| *** gfidente has quit IRC | 06:55 | |
| *** gfidente has joined #openstack-kolla | 06:55 | |
| *** kevinz has quit IRC | 06:59 | |
| *** kevinz has joined #openstack-kolla | 07:00 | |
| *** cah_link has joined #openstack-kolla | 07:00 | |
| korean101 | kolla_internal_vip_address, kolla_external_vip_address not on arp table | 07:01 |
| korean101 | (incomplete) | 07:01 |
| korean101 | someone knows this? | 07:01 |
| *** pcaruana has joined #openstack-kolla | 07:02 | |
| *** zhangfei has joined #openstack-kolla | 07:03 | |
| *** jtriley has joined #openstack-kolla | 07:04 | |
| *** cah_link has quit IRC | 07:04 | |
| *** kiennt26 has joined #openstack-kolla | 07:12 | |
| *** cah_link has joined #openstack-kolla | 07:14 | |
| *** yangyape_ has joined #openstack-kolla | 07:14 | |
| *** yangyapeng has quit IRC | 07:16 | |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible master: Add content about cluster_network interface https://review.openstack.org/514544 | 07:18 |
| *** cah_link has quit IRC | 07:18 | |
| *** danardelean has joined #openstack-kolla | 07:19 | |
| *** vharsh has quit IRC | 07:22 | |
| *** rmart04 has joined #openstack-kolla | 07:25 | |
| *** zhangfei has quit IRC | 07:25 | |
| *** vharsh has joined #openstack-kolla | 07:26 | |
| *** yangyape_ has quit IRC | 07:27 | |
| *** yangyapeng has joined #openstack-kolla | 07:27 | |
| *** egonzalez_afk has joined #openstack-kolla | 07:30 | |
| *** yangyape_ has joined #openstack-kolla | 07:30 | |
| *** egonzalez_afk has quit IRC | 07:31 | |
| *** egonzalez has quit IRC | 07:31 | |
| *** yangyapeng has quit IRC | 07:32 | |
| *** jaosorior has quit IRC | 07:33 | |
| *** egonzalez has joined #openstack-kolla | 07:37 | |
| *** zhangfei has joined #openstack-kolla | 07:38 | |
| korean101 | http://paste.openstack.org/show/624426/ ERROR about multinode deploy | 07:44 |
| *** dasTor has quit IRC | 07:44 | |
| egonzalez | korean101, check if mariadb is running in the hosts, also ensure the VIP is bind into correct interface in network nodes | 07:45 |
| korean101 | egonzalez: mariadb in node-001 | 07:46 |
| korean101 | egonzalez: haproxy in node-002 | 07:46 |
| korean101 | egonzalez: node-001 --> haproxy ping fails | 07:46 |
| korean101 | [root@node-001 ~]# docker exec mariadb mysql -h 10.0.0.99 -P 3306 -u haproxy -e show databases; | 07:48 |
| korean101 | ERROR 2003 (HY000): Can't connect to MySQL server on '10.0.0.99' (113 "No route to host") | 07:48 |
| openstackgerrit | Chason Chan proposed openstack/kolla master: Upgrade the rst convention of the Admin Guide https://review.openstack.org/511659 | 07:48 |
| korean101 | can't reach haproxy (10.0.0.99) | 07:49 |
| korean101 | OMG... so hard | 07:49 |
| egonzalez | korean101, is 10.0.0.99 the IP of the haproxy host or the VIP? | 07:49 |
| korean101 | egonzalez: yes | 07:50 |
| korean101 | egonzalez: kolla_internal_vip_address: "10.0.0.99" | 07:50 |
| egonzalez | check if the vip is bind in the interface | 07:50 |
| korean101 | egonzalez: arp -a node-002 (10.0.0.99) at <incomplete> on eth0 | 07:50 |
| korean101 | egonzalez: inet 10.0.0.99/32 scope global eth0 | 07:50 |
| korean101 | bind is ok | 07:50 |
| korean101 | egonzalez: but can't ping | 07:51 |
| korean101 | other nodes... | 07:51 |
| egonzalez | korean101, from the eth0 IP can ping to mariadb hosts? | 07:51 |
| korean101 | egonzalez: yes! | 07:51 |
| korean101 | 10.0.0.101 (mariadb) 10.0.0.102 (haproxy) | 07:51 |
| korean101 | 10.0.0.99 (haproxy VIP) | 07:52 |
| korean101 | .101 <--> .102 ping each other | 07:52 |
| korean101 | egonzalez: but can't ping 10.0.0.99 | 07:52 |
| korean101 | egonzalez: but ip addr command show 10.0.0.99 (eth0) | 07:52 |
| korean101 | egonzalez: but arp incomplete | 07:53 |
| korean101 | i use 3 openstack VMs | 07:53 |
| egonzalez | hrm, is port security disabled or allowed-addresses in the VIP? | 07:53 |
| egonzalez | neutron will block any conn not matching mac addr | 07:53 |
| bjolo | egonzalez: was just about to ask the same :) | 07:54 |
| korean101 | egonzalez: http://paste.openstack.org/show/624427/ | 07:55 |
| korean101 | bjolo: same error? | 07:55 |
| egonzalez | korean101, if not have access to enable allowed addresses and only have one haproxy node, just disable haproxy and set vip to the host IP | 07:55 |
| korean101 | egonzalez: all secgroup is allowed | 07:56 |
| korean101 | | tcp | 1 | 65535 | 0.0.0.0/0 | | | 07:57 |
| korean101 | | udp | 1 | 65535 | 0.0.0.0/0 | | | 07:57 |
| korean101 | | icmp | -1 | -1 | 0.0.0.0/0 | | | 07:57 |
| *** jaosorior has joined #openstack-kolla | 07:57 | |
| korean101 | egonzalez: prohibit use openstack VMs for target hosts? | 07:57 |
| korean101 | only use baremetal? | 07:57 |
| *** dougsz has joined #openstack-kolla | 08:00 | |
| egonzalez | korean101, neutron will block the mac if does not match with the port to avoid arp spoofing, will need to disable or port-security or add allowed-addresses-pair to the eth0 port | 08:02 |
| *** vharsh has quit IRC | 08:10 | |
| *** ktibi has joined #openstack-kolla | 08:24 | |
| *** danardelean has quit IRC | 08:26 | |
| *** danardelean has joined #openstack-kolla | 08:27 | |
| *** danardelean has quit IRC | 08:31 | |
| *** dougsz1 has joined #openstack-kolla | 08:37 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible master: Use internal_protocol variable to replace http https://review.openstack.org/514561 | 08:38 |
| *** dougsz has quit IRC | 08:40 | |
| *** cah_link has joined #openstack-kolla | 08:40 | |
| *** danardelean has joined #openstack-kolla | 08:41 | |
| korean101 | egonzalez: OK. got it | 08:43 |
| *** cah_link has quit IRC | 08:47 | |
| *** danardel_ has joined #openstack-kolla | 08:49 | |
| openstackgerrit | Dai Dang Van proposed openstack/kolla-ansible master: [WIP] Add support policy.yaml file - Aodh https://review.openstack.org/514565 | 08:52 |
| *** danardelean has quit IRC | 08:53 | |
| openstackgerrit | Dai Dang Van proposed openstack/kolla-ansible master: [WIP] Add support policy.yaml file - Aodh https://review.openstack.org/514565 | 08:54 |
| openstackgerrit | Dai Dang Van proposed openstack/kolla-ansible master: [WIP] Add support policy.yaml file - Aodh https://review.openstack.org/514565 | 08:55 |
| korean101 | jesus | 08:55 |
| korean101 | egonzalez: you're genius | 08:56 |
| korean101 | egonzalez: ping's OK! | 08:56 |
| korean101 | egonzalez: oh god... thanks | 08:56 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/pike: cinder-api: handle Debian too https://review.openstack.org/514567 | 08:56 |
| hrw | morning | 08:57 |
| *** yangyapeng has joined #openstack-kolla | 08:57 | |
| hrw | egonzalez: can you approve https://review.openstack.org/#/c/514567/ one? It is backport for stable/pike to make cinder-api working with Debian. bug was reported as it should | 08:57 |
| *** yangyape_ has quit IRC | 09:01 | |
| egonzalez | hrw, please, next time do a cherry-pick for backports | 09:03 |
| *** daidv has quit IRC | 09:03 | |
| hrw | egonzalez: to keep same changeid? | 09:03 |
| egonzalez | yep | 09:03 |
| hrw | ok, will do | 09:03 |
| *** daidv has joined #openstack-kolla | 09:03 | |
| hrw | will push cherrypick one then | 09:03 |
| *** daidv_ has joined #openstack-kolla | 09:04 | |
| *** daidv_ has quit IRC | 09:04 | |
| *** daidv has quit IRC | 09:04 | |
| *** daidv has joined #openstack-kolla | 09:04 | |
| *** daidv has quit IRC | 09:05 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/pike: cinder-api: handle Debian too https://review.openstack.org/514573 | 09:06 |
| *** daidv has joined #openstack-kolla | 09:06 | |
| Zophar | Hello... when enabling Designate the task is hanging at "Update DNS pools" (designate-manage pool update).... in designate_manage.log i got "2017-10-24 10:56:38.501 18 ERROR oslo.messaging._drivers.impl_rabbit [designate-manage - - - - -] [31d094af-b00d-42b3-ba2b-6c128daf652b] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED" | 09:07 |
| hrw | egonzalez: resent | 09:07 |
| Zophar | But the transport_url in designate point on the right ip.. not on localhost | 09:07 |
| Zophar | any id? | 09:07 |
| *** lxdong has quit IRC | 09:07 | |
| ktibi | Zophar send all you designate conf file | 09:08 |
| Zophar | ok give me 1 min | 09:08 |
| Zophar | http://paste.openstack.org/show/624443/ | 09:10 |
| *** gkadam_ has quit IRC | 09:11 | |
| Zophar | this is designate.conf of designate_worker (where the pool update happens)... don't know if other conf files are relevant here | 09:12 |
| *** gkadam has joined #openstack-kolla | 09:13 | |
| Zophar | rabbitmq is listening on 10.56.18.192:5672 | 09:13 |
| Zophar | tested with telnet and i see messages comming from other service | 09:13 |
| *** dasTor has joined #openstack-kolla | 09:14 | |
| *** dasTor_ has joined #openstack-kolla | 09:14 | |
| *** dasTor_ has quit IRC | 09:14 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla master: Make env variables accessible in template files https://review.openstack.org/508459 | 09:15 |
| hrw | does someone know how to manage automatic push to hub.docker.com from CI without adding username/pass into job config? | 09:15 |
| openstackgerrit | Merged openstack/kolla-ansible master: Unify global syntax in baremetal role https://review.openstack.org/505132 | 09:17 |
| *** athomas has quit IRC | 09:32 | |
| *** dasTor has quit IRC | 09:33 | |
| *** dasTor has joined #openstack-kolla | 09:33 | |
| *** danardel_ has quit IRC | 09:38 | |
| *** danardelean has joined #openstack-kolla | 09:39 | |
| openstackgerrit | Dai Dang Van proposed openstack/kolla-ansible master: Add support policy.yaml file - Aodh https://review.openstack.org/514565 | 09:40 |
| *** yangyape_ has joined #openstack-kolla | 09:40 | |
| *** danardelean has quit IRC | 09:40 | |
| openstackgerrit | Dai Dang Van proposed openstack/kolla-ansible master: Add support policy.yaml file - Aodh https://review.openstack.org/514565 | 09:41 |
| *** dasTor has quit IRC | 09:41 | |
| *** zhubingbing_ has quit IRC | 09:43 | |
| *** sambetts|afk is now known as sambetts | 09:43 | |
| *** yangyapeng has quit IRC | 09:44 | |
| *** yingjun has quit IRC | 09:45 | |
| Zophar | anyone has an idea on designate rabbitmq issue? | 09:48 |
| Zophar | why the hell is trying to connect on 127.0.0.1? | 09:49 |
| masber | is it documented somewhere how tell nova to use local storage for ephemeral while using ceph for volumes? | 09:49 |
| *** danardelean has joined #openstack-kolla | 09:50 | |
| *** danardelean has quit IRC | 09:50 | |
| *** danardelean has joined #openstack-kolla | 09:50 | |
| *** huanxie has joined #openstack-kolla | 09:51 | |
| *** hachi_ has joined #openstack-kolla | 09:52 | |
| kolla-slack | <aslancimbom100> there should be an option in globals | 09:52 |
| kolla-slack | <aslancimbom100> masber...something like #nova_backend_ceph: "{{ enable_ceph }}" | 09:52 |
| Zophar | Ok strange i found what going wrong on designate manage | 09:56 |
| Zophar | in designate.conf the rabbitmq is set on transport_url (which should be fine) as documented | 09:57 |
| Zophar | i try to set it in oslo_messaging_rabbit section | 09:57 |
| Zophar | and it works | 09:57 |
| Zophar | Why!?!? | 09:57 |
| Zophar | the log says it's depracated... but now the designate-manage pool update process run (and doesn't hang) | 09:58 |
| egonzalez | Zophar, is a known bug in designate | 10:00 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Revert "Remove deprecated oslo_messaging_rabbit options" https://review.openstack.org/514585 | 10:00 |
| egonzalez | is not fixed yet afik | 10:01 |
| *** yangyapeng has joined #openstack-kolla | 10:02 | |
| Zophar | Edouardo you're my hero ;) | 10:03 |
| *** yangyapeng has quit IRC | 10:03 | |
| *** yangyapeng has joined #openstack-kolla | 10:03 | |
| *** yangyape_ has quit IRC | 10:06 | |
| *** egonzalez has quit IRC | 10:06 | |
| *** yangyapeng has quit IRC | 10:08 | |
| Zophar | btw it has been fixed here: https://bugs.launchpad.net/designate/+bug/1651576 | 10:09 |
| openstack | Launchpad bug 1651576 in Designate "designate-manage does not properly support DEFAULT/transport_url" [Medium,Fix released] - Assigned to Dr. Jens Harbott (j-harbott) | 10:09 |
| Zophar | 16th september | 10:09 |
| Zophar | will do a review to push it in stable/pike | 10:11 |
| Zophar | on designate repo... thank for your help | 10:11 |
| masber | #nova_backend_ceph will tell nova to use ceph for ephemeral, I want tell nova to use local storage as ephemeral | 10:14 |
| *** daidv has quit IRC | 10:14 | |
| berendt | masber: use overlay files to override the nova compute configuration | 10:27 |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Add hostname to all log messages https://review.openstack.org/514593 | 10:30 |
| *** jinke_ has joined #openstack-kolla | 10:37 | |
| *** pbourke has quit IRC | 10:38 | |
| *** pbourke has joined #openstack-kolla | 10:40 | |
| *** dasTor has joined #openstack-kolla | 10:42 | |
| *** lpetrut has joined #openstack-kolla | 10:42 | |
| kolla-slack | <aslancimbom100> what happens if you set #nova_backend_ceph to no? doesn't it fallback? | 10:44 |
| spsurya | krtaylor: around ? | 10:52 |
| *** egonzalez has joined #openstack-kolla | 10:54 | |
| kevinz | kfox1111: ping | 10:59 |
| *** zhangfei has quit IRC | 10:59 | |
| *** kevinz has quit IRC | 11:01 | |
| *** kevinz has joined #openstack-kolla | 11:02 | |
| kevinz | Hi kolla, I have a problem when deploying with Kolla-kubernetes. I've followed the "development guide", run the command: kollakube res create configmap \ | 11:06 |
| kevinz | mariadb | 11:06 |
| kevinz | but when I run helm to enable mariadb: helm install --debug kolla-kubernetes/helm/service/mariadb --namespace kolla --name mariadb --values ./cloud.yaml | 11:07 |
| *** serlex has joined #openstack-kolla | 11:07 | |
| openstackgerrit | Dai Dang Van proposed openstack/kolla-ansible master: Add support policy.yaml file - Aodh https://review.openstack.org/514565 | 11:08 |
| openstackgerrit | Dai Dang Van proposed openstack/kolla-ansible master: Add support policy.yaml file - Aodh https://review.openstack.org/514565 | 11:08 |
| kevinz | Got issue: http://paste.openstack.org/show/624452/ | 11:09 |
| *** newmember has quit IRC | 11:10 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack/kolla master: Remove redundant statement in help string of config option 'base' https://review.openstack.org/514608 | 11:22 |
| *** huanxie has quit IRC | 11:29 | |
| *** yangyapeng has joined #openstack-kolla | 11:32 | |
| *** yangyapeng has quit IRC | 11:33 | |
| *** yangyapeng has joined #openstack-kolla | 11:33 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/pike: base: use new Linaro repo https://review.openstack.org/514614 | 11:38 |
| *** jinke_ has quit IRC | 11:38 | |
| *** lhx__ has joined #openstack-kolla | 11:42 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack/kolla master: Remove duplicated statement of parameter choices of config options https://review.openstack.org/514608 | 11:44 |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack/kolla master: Fix docstring of method run_build https://review.openstack.org/514618 | 11:53 |
| *** ansmith has quit IRC | 11:56 | |
| hrw | any update on 6.0.0 tag? | 11:57 |
| *** lhx__ has quit IRC | 12:18 | |
| *** hachi_ has quit IRC | 12:24 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/kolla master: Add rsyslog image https://review.openstack.org/514632 | 12:25 |
| *** tvignaud has quit IRC | 12:25 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/kolla master: Add rsyslog image https://review.openstack.org/514632 | 12:25 |
| *** dave-mccowan has joined #openstack-kolla | 12:31 | |
| *** tvignaud has joined #openstack-kolla | 12:34 | |
| *** tvignaud has quit IRC | 12:34 | |
| *** tvignaud has joined #openstack-kolla | 12:35 | |
| *** goldyfruit has joined #openstack-kolla | 12:41 | |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Extract log level from syslog https://review.openstack.org/514637 | 12:45 |
| *** sbezverk has quit IRC | 12:45 | |
| openstackgerrit | Vladislav Belogrudov proposed openstack/kolla-ansible master: Cinder fails some operations with secure NAS features https://review.openstack.org/514639 | 12:46 |
| *** tvignaud has quit IRC | 12:52 | |
| *** tvignaud_ has joined #openstack-kolla | 12:52 | |
| *** tvignaud_ is now known as tvignaud | 12:54 | |
| rwellum | hi kevinz : did you get past your mariadb issue? | 12:55 |
| *** Sean-md has joined #openstack-kolla | 13:00 | |
| *** ansmith has joined #openstack-kolla | 13:02 | |
| *** jtriley has quit IRC | 13:03 | |
| krtaylor | morning kolla | 13:07 |
| krtaylor | spsurya, you rang? | 13:08 |
| *** athomas has joined #openstack-kolla | 13:12 | |
| Sean-md | Hi all. I get "error" messages as below when trying to connect cinder-api using "curl" or "haproxy readiness" | 13:16 |
| Sean-md | "mod_wsgi (pid=74): Target WSGI script '/var/www/cgi-bin/cinder/cinder-wsgi' cannot be loaded as Python module." | 13:16 |
| Sean-md | All messages are https://paste.ubuntu.com/25809148/ | 13:16 |
| Sean-md | env = kolla:5.0.1, centos7 | 13:16 |
| Sean-md | Dose anyone know a solution? | 13:16 |
| egonzalez | Sean-md, IOError: [Errno 13] Permission denied: '/var/log/kolla/cinder/cinder-wsgi.log' | 13:18 |
| *** goldyfruit has quit IRC | 13:20 | |
| *** skramaja has quit IRC | 13:21 | |
| *** goldyfruit has joined #openstack-kolla | 13:24 | |
| *** jtriley has joined #openstack-kolla | 13:27 | |
| *** Sean-md_ has joined #openstack-kolla | 13:30 | |
| openstackgerrit | Pierre Hanselmann proposed openstack/kolla-ansible master: Add horizon domain dropdown https://review.openstack.org/514676 | 13:40 |
| *** Sean-md_ has quit IRC | 13:41 | |
| openstackgerrit | Pierre Hanselmann proposed openstack/kolla-ansible master: Add horizon domain dropdown https://review.openstack.org/514676 | 13:41 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Optimize zuul v3 jobs https://review.openstack.org/513612 | 13:46 |
| *** rmart04 has quit IRC | 13:49 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Optimize zuul v3 jobs https://review.openstack.org/513612 | 13:51 |
| *** coolsvap has quit IRC | 13:53 | |
| *** sbezverk has joined #openstack-kolla | 13:54 | |
| *** sbezverk has quit IRC | 13:59 | |
| *** lukepatrick has joined #openstack-kolla | 14:01 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Optimize zuul v3 jobs https://review.openstack.org/513612 | 14:03 |
| *** lukepatrick has quit IRC | 14:04 | |
| *** Sean-md_ has joined #openstack-kolla | 14:04 | |
| Sean-md_ | egonzalez, The log file dose not exist, I thought I could overlook, I will try after creating the the file. Thanks | 14:05 |
| *** mdnadeem has quit IRC | 14:07 | |
| *** Sean-md_ has quit IRC | 14:07 | |
| masber | good afternoon memcached container fails, this is the error from container logs http://paste.openstack.org/raw/624483/ | 14:09 |
| masber | I am using 5.0.1 centos binary | 14:10 |
| masber | also, I am testing ceph integrated in kolla, uploading an image freezes | 14:12 |
| *** kiennt26 has quit IRC | 14:14 | |
| *** shardy has joined #openstack-kolla | 14:15 | |
| *** hongbin has joined #openstack-kolla | 14:18 | |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Add missing hostnames to fluentd log messages https://review.openstack.org/514693 | 14:24 |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Add missing hostnames to fluentd log messages https://review.openstack.org/514593 | 14:26 |
| *** jamesbenson has joined #openstack-kolla | 14:39 | |
| jamesbenson | morning all :-) | 14:41 |
| *** xion- has joined #openstack-kolla | 14:53 | |
| *** egonzalez has quit IRC | 15:03 | |
| *** salv-orlando has quit IRC | 15:03 | |
| *** salv-orlando has joined #openstack-kolla | 15:04 | |
| *** salv-orlando has quit IRC | 15:08 | |
| *** lrensing has joined #openstack-kolla | 15:09 | |
| *** danardelean has quit IRC | 15:12 | |
| *** danardelean has joined #openstack-kolla | 15:13 | |
| *** mummer has joined #openstack-kolla | 15:17 | |
| *** danardelean has quit IRC | 15:17 | |
| *** magicboiz has quit IRC | 15:22 | |
| *** magicboiz has joined #openstack-kolla | 15:22 | |
| *** TxGirlGeek has joined #openstack-kolla | 15:23 | |
| *** lpetrut has quit IRC | 15:23 | |
| *** lpetrut has joined #openstack-kolla | 15:23 | |
| *** athomas has quit IRC | 15:25 | |
| *** serlex has quit IRC | 15:25 | |
| *** magicboiz has quit IRC | 15:31 | |
| kfox1111 | morning. | 15:34 |
| xion- | evening :) | 15:37 |
| kfox1111 | evening. :) | 15:39 |
| jamesbenson | evening ;-) | 15:42 |
| ktibi | hi inc0 | 15:45 |
| ktibi | I resolve my issue with vxlan port, missing l2population ;) | 15:45 |
| *** newmember has joined #openstack-kolla | 15:46 | |
| inc0 | ktibi: cool | 15:50 |
| rwellum | kfox1111: do you have a little time to look at my libvirt issue with Pike? Serguei said he could later this week but I'm impatient. :) | 15:50 |
| kfox1111 | sure. | 15:51 |
| kfox1111 | I can look a bit now, or later today. | 15:51 |
| rwellum | kfox1111: btw - I don't get this issue with ocata - which makes me very suspicious about the environment being the issue. | 15:51 |
| kfox1111 | yeah. | 15:51 |
| kfox1111 | we don't test pike much yet. :( | 15:51 |
| kfox1111 | can you dump/upload libvirt's configmap? | 15:52 |
| rwellum | yes 1s | 15:52 |
| *** jmccarthy has joined #openstack-kolla | 15:54 | |
| rwellum | https://www.irccloud.com/pastebin/CKP4mxx5/ | 15:54 |
| rwellum | There's a big TODO for pike in that comment | 15:55 |
| *** magicboiz has joined #openstack-kolla | 15:55 | |
| kfox1111 | can you edit the nova-libvirt daemonset and do the bin/bash sleep thing to it so we can dig in? | 15:55 |
| rwellum | yes | 15:56 |
| ktibi | I want to use vlan for my external network. So I need to conf network_vlan_ranges in my ml2 conf file. with kolla, I can modify this value with "{% if enable_ironic | bool %}" | 15:57 |
| *** klindgren has quit IRC | 15:57 | |
| *** klindgren has joined #openstack-kolla | 15:57 | |
| ktibi | possible to override this value ? | 15:58 |
| inc0 | rwellum: hmm, where did you get your pike images? | 15:59 |
| rwellum | kfox1111: so I edit the config map for nova-libvirt and there's no kolla_start - correct config map? | 15:59 |
| rwellum | inc0: I'm using the registry method to download from tarball | 16:00 |
| kfox1111 | rwellum: I think it is probably ok, but we'll have to debug. | 16:00 |
| rwellum | kfox1111: I recall the kolla_start before so not sure. | 16:00 |
| kfox1111 | the kolla_start thing is in the daemonset | 16:00 |
| kfox1111 | kubectl edit daemonset nova-libvirt -n kolla | 16:01 |
| rwellum | kfox1111: ah it's edit daemonset not edit configmap :) | 16:01 |
| * kfox1111 nods | 16:01 | |
| ktibi | ok I found https://review.openstack.org/#/c/464675/ | 16:02 |
| rwellum | kfox1111: ok edited - now restart the container? | 16:02 |
| *** magicboiz has quit IRC | 16:02 | |
| kfox1111 | yeah. just nuke it and it hsould come back. | 16:02 |
| *** salv-orlando has joined #openstack-kolla | 16:05 | |
| inc0 | rwellum: so I remember how this user (todo) thing would show - you couldn't access console log | 16:05 |
| inc0 | whcih is not the case for you right? libvirt doesn't even start | 16:05 |
| rwellum | ok done, it's up and I am exec'ed in | 16:05 |
| rwellum | inc0: yes correct - container crashing | 16:05 |
| inc0 | any logs? | 16:05 |
| kfox1111 | do a kolla_start. see if it dies. | 16:05 |
| kfox1111 | if it does, pastebin /var/log/kolla/libvirt/* | 16:06 |
| rwellum | inc0: I'm on the container now - debugging with kfox1111 | 16:06 |
| inc0 | yeah do what kfox1111 said:) | 16:06 |
| rwellum | :) | 16:06 |
| * rwellum doing | 16:06 | |
| kfox1111 | :) | 16:06 |
| inc0 | kfox1111: you pass envs with k8s right? | 16:06 |
| kfox1111 | inc0: environment variables? | 16:07 |
| rwellum | kfox1111 ^^^ https://www.irccloud.com/pastebin/L2NjBzdS/ | 16:07 |
| inc0 | ktibi: yeah you can override | 16:07 |
| inc0 | kfox1111: right | 16:07 |
| kfox1111 | inc0: as few as I can get away with. :) | 16:07 |
| kfox1111 | inc0: generally configmaps are better. | 16:08 |
| kfox1111 | rwellum: ok, thats interesting. | 16:08 |
| inc0 | what I'm saying kolla_start needs it | 16:08 |
| inc0 | ktibi: https://docs.openstack.org/kolla-ansible/latest/admin/advanced-configuration.html#openstack-service-configuration-in-kolla | 16:08 |
| kfox1111 | inc0: we pass the stuff kolla_start needs. | 16:08 |
| ktibi | inc0 thx | 16:08 |
| inc0 | rwellum: well..we have error | 16:09 |
| inc0 | but I've never seen it | 16:09 |
| kfox1111 | https://github.com/openstack/kolla-kubernetes/blob/master/helm/kolla-common/templates/_common_lib.yaml#L26 | 16:09 |
| *** salv-orlando has quit IRC | 16:09 | |
| kfox1111 | inc0: I think it may have to do with sbezverk enabling tcp on libvirt? | 16:09 |
| inc0 | maybe | 16:10 |
| kfox1111 | looks like it may try and use tls by default. | 16:10 |
| *** magicboiz has joined #openstack-kolla | 16:10 | |
| kfox1111 | I preffer using the socket anyway. | 16:10 |
| kfox1111 | no reason to open up libvirt to the world. | 16:10 |
| rwellum | I see a conversation with other people seeing the same here: http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2017-09-18.log.html | 16:10 |
| jamesbenson | still trying to figure out where the breakdown is in our network.... everything looks good. :-( | 16:11 |
| kfox1111 | so kolla-ansible's seeing it too probably. | 16:11 |
| inc0 | maybe something new in 2.4? | 16:11 |
| rwellum | inc0: kfox1111 but our gates work for pike? | 16:12 |
| kfox1111 | could be. | 16:12 |
| kfox1111 | rwellum: do we have pike gates? | 16:12 |
| rwellum | kfox1111: we have master - I think Serguei set that up - but uses label 5.0.0 - which grabs the master tarball | 16:13 |
| kfox1111 | we don't have voting pike I guess. | 16:13 |
| kfox1111 | only experimental. | 16:13 |
| inc0 | try to rollback tcp patch? | 16:13 |
| kfox1111 | I think there is a way to disable tcp.... lets see... | 16:14 |
| inc0 | it's config option | 16:14 |
| kfox1111 | enable_libvirt_tcp = false | 16:14 |
| inc0 | rwellum: try to edit listen_tcp in libvirtd.conf | 16:16 |
| inc0 | and restart container | 16:16 |
| inc0 | listen_tcp = 0 | 16:16 |
| kfox1111 | ugg. its hardcoded in ansible roles. :( | 16:16 |
| *** jascott1 has joined #openstack-kolla | 16:17 | |
| jmccarthy | Anyone run into an issue where with cirros, console logging works in horizon, but with something else, it doesn't ? | 16:17 |
| rwellum | inc0: just to make sure - on the libvirt daemonset container, edit the libvirt.conf there? | 16:18 |
| inc0 | rwellum: edit configmap | 16:18 |
| kfox1111 | rwellum: inside the container, vi /etc/libvirt/libvirt.conf | 16:18 |
| kfox1111 | then kolla_start quick. | 16:18 |
| kfox1111 | shoudl be ok. | 16:18 |
| inc0 | I guess that's fastest | 16:18 |
| rwellum | kfox1111: done but same message/error in the log | 16:19 |
| kfox1111 | hmm.... | 16:20 |
| inc0 | libvirtd --version | 16:20 |
| kfox1111 | maybe get rid of auth_tcp and ca_file too? | 16:20 |
| rwellum | ok well there's nothing in this .conf file | 16:20 |
| rwellum | https://www.irccloud.com/pastebin/wXmtOEVn/ | 16:21 |
| kfox1111 | does it start? | 16:21 |
| inc0 | ahh so probably it's wrong conf file | 16:21 |
| kfox1111 | add: | 16:21 |
| kfox1111 | log_level = 3 | 16:21 |
| kfox1111 | log_outputs = "3:file:/var/log/kolla/libvirt/libvirtd.log" | 16:21 |
| kfox1111 | oh.. | 16:21 |
| kfox1111 | hmm. | 16:21 |
| rwellum | oh it does not - missed that... | 16:21 |
| rwellum | https://www.irccloud.com/pastebin/0khyyerf/ | 16:21 |
| kfox1111 | OH. | 16:22 |
| kfox1111 | libvirtd.conf, not libvirt.conf | 16:22 |
| rwellum | urgh! | 16:22 |
| rwellum | still getting the same start error though - I did remove the line from libvirt.conf | 16:23 |
| inc0 | set it explicitly to 0 | 16:23 |
| inc0 | I don't know what's default | 16:23 |
| inc0 | and try to remove all other notions of listening on network | 16:24 |
| kfox1111 | +1. remove ca_file and auth_tcp, and listen_addr. | 16:24 |
| kfox1111 | so shold only be 3 lines. listen_tcp = 0, log_level = 3, log_outputs = "3:file:/var/log/kolla/libvirt/libvirtd.log" | 16:25 |
| *** pcaruana has quit IRC | 16:25 | |
| rwellum | https://www.irccloud.com/pastebin/Y8nDkoES/ | 16:26 |
| inc0 | so libvirt runs right? | 16:27 |
| rwellum | No I don't think so - see that ERROR? | 16:27 |
| inc0 | that shouldn't matter | 16:27 |
| inc0 | check /var/log/kolla... | 16:27 |
| rwellum | https://www.irccloud.com/pastebin/dUFgjIQi/ | 16:28 |
| inc0 | interesting | 16:28 |
| inc0 | seems like your host has broken certificates | 16:29 |
| kfox1111 | inc0: they come from the container though. | 16:29 |
| inc0 | hmm, you're using ubuntu container on top of centos? | 16:29 |
| kfox1111 | inc0: we don't pass host certs to the container, so that shouldn't matter? | 16:29 |
| inc0 | I think ca certs are taken from host | 16:29 |
| inc0 | by docker | 16:29 |
| kfox1111 | gotta go in 5. | 16:29 |
| kfox1111 | uhhh... what? | 16:30 |
| rwellum | My host VM is ubuntu, but I see the same issues with centos host vm | 16:30 |
| kfox1111 | I don't think I've seen docker do that. unless its in a much newer version. | 16:30 |
| inc0 | rwellum: do you mix container distro and host distro? | 16:30 |
| inc0 | because one line shows something about centos, second shows ubuntu | 16:31 |
| rwellum | Yeah I think so in this case | 16:31 |
| inc0 | I've seen libvirt misbehave when that happens | 16:31 |
| inc0 | (in ansible tho) | 16:31 |
| rwellum | base_distro: "centos" | 16:32 |
| rwellum | So yeah this is centos on ubuntu - but tbh never seen any issues. | 16:32 |
| rwellum | before | 16:32 |
| *** shardy has quit IRC | 16:33 | |
| rwellum | I can respin centos-centos - and make that config change | 16:33 |
| kfox1111 | generally its ok. but I too have seen some issues with libvirt in a mixed env. | 16:33 |
| inc0 | yeah it was just libvirt that had issues as I remember, but there were those | 16:34 |
| rwellum | What is the change I need? Because kfox1111 you said hard coded somewhere? | 16:34 |
| *** jascott1 has quit IRC | 16:35 | |
| inc0 | https://github.com/openstack/kolla-kubernetes/blob/master/ansible/roles/nova/templates/libvirtd.conf.j2 <- try this file | 16:36 |
| *** janki has quit IRC | 16:37 | |
| hrw | inc0, rwellum: any idea when 6.0.0 tag will be done? iirc schedule mentioned 19th ;( | 16:37 |
| rwellum | Ok and remove those three lines | 16:37 |
| rwellum | hrw: I saw your question earlier but not sure if it's wrapped up in inc0 infra changes | 16:37 |
| inc0 | hrw: yeah, I guess zuulv3 chaos made us miss it;) | 16:38 |
| hrw | inc0: ;) | 16:38 |
| inc0 | check with Jeffrey4l tomorrow yours morning | 16:38 |
| inc0 | he does releases | 16:38 |
| hrw | inc0: will do. thanks | 16:38 |
| inc0 | thanks for keeping an eye for it | 16:38 |
| hrw | just built 5.0.1 today instead. and this one is not tagged too | 16:38 |
| hrw | atleast not at http://git.openstack.org/cgit/openstack/kolla/refs/tags | 16:39 |
| inc0 | no, we didn't tag yet | 16:39 |
| hrw | and (while we are at pike) I have two backports in queue: https://review.openstack.org/#/c/514573/ (make cinder-api work in Debian) https://review.openstack.org/#/c/514614/ (add Linaro kolla:pike repo for Debian to have some extra backports required for aarch64 (x86-64 binaries also provided)) | 16:41 |
| *** lukepatrick has joined #openstack-kolla | 16:45 | |
| inc0 | sooo keepalived works SamYaple -.- | 16:52 |
| inc0 | for me | 16:52 |
| *** jascott1 has joined #openstack-kolla | 16:53 | |
| *** unicell has joined #openstack-kolla | 16:55 | |
| *** jascott1 has quit IRC | 16:57 | |
| *** jascott1 has joined #openstack-kolla | 16:59 | |
| *** sambetts is now known as sambetts|afk | 17:00 | |
| inc0 | masber: any progress on keepalived issue? | 17:00 |
| *** jascott1 has quit IRC | 17:02 | |
| *** itlinux has joined #openstack-kolla | 17:02 | |
| *** jascott1 has joined #openstack-kolla | 17:03 | |
| *** jmccarthy has left #openstack-kolla | 17:07 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla master: WIP Dockerhub publisher job https://review.openstack.org/513855 | 17:07 |
| *** lpetrut_ has joined #openstack-kolla | 17:11 | |
| kfox1111 | fasinating.... | 17:12 |
| kfox1111 | https://github.com/sapcc/kubernetes-operators/tree/master/openstack-operator | 17:12 |
| *** lpetrut has quit IRC | 17:14 | |
| *** lpetrut_ has quit IRC | 17:21 | |
| *** dougsz1 has quit IRC | 17:22 | |
| *** magicboiz has quit IRC | 17:22 | |
| jascott1 | kfox1111 yeah saw that... early days i think | 17:25 |
| jascott1 | this was news to me: go codegen required for operators in 1.8? https://blog.openshift.com/kubernetes-deep-dive-code-generation-customresources/ | 17:25 |
| *** salv-orlando has joined #openstack-kolla | 17:27 | |
| *** gkadam has quit IRC | 17:30 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla master: WIP Dockerhub publisher job https://review.openstack.org/513855 | 17:32 |
| openstackgerrit | chandu proposed openstack/kolla-ansible stable/ocata: Removing groups scope from all to baremetal in baremetal pre-install task under bootstrap-server https://review.openstack.org/514774 | 17:32 |
| *** lpetrut has joined #openstack-kolla | 17:35 | |
| *** jinke_ has joined #openstack-kolla | 17:35 | |
| *** dougsz has joined #openstack-kolla | 17:37 | |
| *** jinke_ has quit IRC | 17:39 | |
| *** jascott1 has quit IRC | 17:45 | |
| *** jascott1 has joined #openstack-kolla | 17:46 | |
| *** lrensing has quit IRC | 17:48 | |
| *** jascott1 has quit IRC | 17:49 | |
| *** tvignaud has quit IRC | 17:53 | |
| *** jascott1 has joined #openstack-kolla | 17:54 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla master: WIP Dockerhub publisher job https://review.openstack.org/513855 | 17:58 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla master: WIP Dockerhub publisher job https://review.openstack.org/513855 | 18:00 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla master: WIP Dockerhub publisher job https://review.openstack.org/513855 | 18:09 |
| *** danardelean has joined #openstack-kolla | 18:13 | |
| *** danardelean has quit IRC | 18:17 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla master: WIP Dockerhub publisher job https://review.openstack.org/513855 | 18:21 |
| rwellum | kfox1111: when a pod is in init state, and it's downloading - is there are more granular way of tracking progress? | 18:21 |
| kfox1111 | like, you wondering how much left it has to download, etc? | 18:22 |
| rwellum | yes exactly | 18:22 |
| kfox1111 | I don't know of a way to do that currently. | 18:22 |
| rwellum | The registry can vary from 20m to 2 hours annoyingly | 18:23 |
| kfox1111 | would like there to be a feature like that myself. :) | 18:23 |
| rwellum | Ah :) | 18:23 |
| kfox1111 | wow. | 18:23 |
| kfox1111 | need to cache that locally somehow I guess. | 18:23 |
| *** salv-orlando has quit IRC | 18:25 | |
| *** salv-orlando has joined #openstack-kolla | 18:25 | |
| rwellum | I notice that the registry uses a 'curl' - wondering if an scp or better rsync would work better | 18:27 |
| kfox1111 | which registry? | 18:28 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla master: WIP Dockerhub publisher job https://review.openstack.org/513855 | 18:29 |
| *** salv-orlando has quit IRC | 18:30 | |
| inc0 | I wrote quick playbook that will setup local registry for you if you want | 18:32 |
| rwellum | Sorry - Serguie's registry for deploying Pike/Master that he's using in the gate. | 18:34 |
| *** lpetrut has quit IRC | 18:34 | |
| kfox1111 | oh. docker registry | 18:34 |
| rwellum | yup | 18:34 |
| *** lpetrut has joined #openstack-kolla | 18:34 | |
| jascott1 | sbezverks registry chart works for me | 18:34 |
| rwellum | kfox1111: that's a work-around correct? Until we sort the publishing | 18:35 |
| *** TxGirlGeek has quit IRC | 18:35 | |
| rwellum | jascott1: yeah I'm just complaining that it's slow and unpredictable :) | 18:35 |
| inc0 | https://github.com/inc0/kolla-registry | 18:35 |
| jascott1 | ah | 18:35 |
| inc0 | don't overuse it please tho | 18:35 |
| kfox1111 | rwellum: probably should tweak that and add an option to use a pv. | 18:35 |
| kfox1111 | rwellum: then you could set it to use a host path and point it at a previously downloaded tar. | 18:35 |
| jamesbenson | where is the qemu instance being hosted in kolla? /etc/libvirt/qemu/instance....xml file? | 18:36 |
| jascott1 | man i need a ceph survival guide | 18:36 |
| kfox1111 | jascott1: whats up? | 18:36 |
| rwellum | kfox1111: tweak inc0 's playbook? | 18:36 |
| inc0 | jascott1: /var/lib/docker/volumes/nova_compute ... | 18:36 |
| kfox1111 | rwellum: sbezverk's chart | 18:37 |
| rwellum | Oh right. | 18:37 |
| kfox1111 | kube, not ansible. ;) | 18:37 |
| jascott1 | inc0 ? | 18:37 |
| inc0 | jamesbenson: * ;) | 18:37 |
| rwellum | jascott1: ceph took me 3 months to get working bare-metal. | 18:37 |
| jamesbenson | yep caught that, thanks inc0 | 18:37 |
| jascott1 | so I have everything installed from the etherpad but something is missing | 18:37 |
| jamesbenson | rwellum, wow! with ceph-deploy or something else? | 18:38 |
| jascott1 | rwellum woah! | 18:38 |
| kfox1111 | that seems like a long time. wall clock or real effort? | 18:38 |
| *** danardelean has joined #openstack-kolla | 18:38 | |
| rwellum | Yeah.. Well a lot of networking issues. | 18:38 |
| rwellum | wall clock - I was doing other things. | 18:38 |
| kfox1111 | jascott1: I ran into a weird issue where the instructions I wrote layed down the wrong version of ceph. | 18:39 |
| rwellum | But tweaking it for performance took all the time. | 18:39 |
| kfox1111 | jascott1: did you end up with a luminious or jewel version? | 18:39 |
| jascott1 | ceph version 10.2.7 | 18:39 |
| kfox1111 | yeah. you can tweak it for a long time. | 18:39 |
| rwellum | inc0: that pb looks cool - will try it out | 18:39 |
| inc0 | tweak dockerfile to use ubuntu or centos | 18:39 |
| kfox1111 | weird. 10.2.7 is jewel too. | 18:40 |
| kfox1111 | ok. | 18:40 |
| kfox1111 | jascott1: so, what seems to be the problem? | 18:40 |
| jascott1 | so after everyting is installed in including your provisioner chart, I can create a PV and its seems ok but pvc stays pending | 18:41 |
| inc0 | jascott1: describe da pvc! | 18:41 |
| jascott1 | well before i create PV I created a ceph colyme | 18:41 |
| jascott1 | volume | 18:41 |
| jascott1 | thats not in the instructions | 18:42 |
| kfox1111 | jascott1: ah. so probably a problem with the provisioner not being fully working, not ceph itself. | 18:42 |
| jascott1 | idk | 18:42 |
| jamesbenson | ah yeah, I didn't really tweak it at all.. just got it running and go with it ;-) Do you have a github/post that shares the configs? | 18:42 |
| kfox1111 | yeah. that part of the meeting was a little rough, and I don't think I got all of the instructions. | 18:42 |
| kfox1111 | there could very well be a step or two missing still. | 18:42 |
| jascott1 | but i should be creating a rbd volune? | 18:42 |
| *** salv-orlando has joined #openstack-kolla | 18:42 | |
| *** danardelean has quit IRC | 18:42 | |
| kfox1111 | the provisioner shoudl create the rbd volume. | 18:42 |
| kfox1111 | but we need to ensure the ceph user and rbd pool are created. | 18:43 |
| kfox1111 | got a link to the etherpad handy? | 18:43 |
| jascott1 | https://etherpad.openstack.org/p/kolla-k8s-braindump | 18:43 |
| jascott1 | oh ok that helps, didnt know the provisioner should create the rbd vol | 18:43 |
| inc0 | what it needs tho is rbd pool | 18:44 |
| inc0 | that you need to create yourself | 18:44 |
| inc0 | also call "ceph -s" please | 18:44 |
| jamesbenson | missing this too in the etherpad: --release luminous | 18:45 |
| jamesbenson | otherwise the default is 10.2.7 | 18:45 |
| jamesbenson | ceph-deploy install --release luminous $ALL_NODES | 18:45 |
| rwellum | kfox1111: inc0 unfortunately libvirt still crashing even with TCP listen turned off. Centos host - centos containers. | 18:45 |
| jascott1 | i have a kollavolumes pool | 18:45 |
| inc0 | rwellum: crap... | 18:45 |
| inc0 | ok, back to debugging | 18:45 |
| inc0 | same error as before? | 18:46 |
| rwellum | Yes - I'll do the same debug as before and see what's going on. Have to pop home but will be back on later. | 18:46 |
| jascott1 | ceph -s says HEALTH_OK | 18:46 |
| inc0 | how about describe of pvc? | 18:46 |
| inc0 | also ceph_provisioner pod is healthy? | 18:47 |
| jascott1 | yup | 18:47 |
| inc0 | describe da pvc! | 18:47 |
| inc0 | kubectl describe pvc ... | 18:47 |
| jascott1 | to be clear, dont make a rbd vol but do make a k8s pv first right? | 18:48 |
| jascott1 | or no? | 18:48 |
| inc0 | you don't make vol, you make pvc first and then pv | 18:48 |
| kfox1111 | rwellum: yeah, I'm thinking there is something that has to be explicitly turned off. | 18:49 |
| jascott1 | where is example for pv? I have one but could be wrong | 18:49 |
| inc0 | jascott1: it's pod thing | 18:49 |
| jascott1 | what? | 18:50 |
| kfox1111 | jascott1: yeah, looking at the docs, we still need the bit to create the user, | 18:50 |
| kfox1111 | to create the pool, | 18:50 |
| inc0 | jascott1: I used these http://paste.openstack.org/show/624512/ | 18:51 |
| jascott1 | i have the kolla user and the kollavolumes pool | 18:51 |
| kfox1111 | and to pass the ceph admin key into the provisioner. | 18:51 |
| kfox1111 | inc0: do you still have notes for those actions, or should I try to dig up the right commands again? | 18:51 |
| inc0 | kfox1111: I don't think so | 18:53 |
| inc0 | I have machine tho | 18:53 |
| jascott1 | inc0 describing the pvc just says its waitin on external provisioner to create volume | 18:54 |
| inc0 | then kubectl logs ceph_provision | 18:55 |
| jascott1 | kfox1111 where is $key in following? helm install stable/ceph-provisioner --name ceph-provisioner --namespace kube-system --set "adminSecret.key=$key,monitors[0]=172.17.0.1" --values ... | 18:55 |
| inc0 | ahh | 18:55 |
| inc0 | right, that's probably what you're missing | 18:56 |
| inc0 | when you created kolla used you should create key file too | 18:56 |
| jascott1 | yeah just realized that I templated the values heredoc but neglected the --set art | 18:56 |
| jascott1 | part | 18:56 |
| inc0 | make sure to point monitors to actual monitor | 18:57 |
| kfox1111 | jascott1: needs to be set to the admin key | 18:57 |
| inc0 | s | 18:57 |
| jascott1 | admin in key created in ~/mycluster? | 18:57 |
| kfox1111 | yeah. but just the base64 part. so something like: | 18:57 |
| kfox1111 | key=$(grep key | awk '{print $3}') | 18:57 |
| kfox1111 | grep key <filename> | 18:58 |
| jascott1 | cool thanks.. got someone calling me brb | 18:58 |
| kfox1111 | should be able to create the pool with something like: | 18:59 |
| kfox1111 | ceph osd pool create kubevolumes 16 16 | 19:00 |
| kfox1111 | ceph auth get-or-create client.kube mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=kubevolumes' | 19:01 |
| kfox1111 | or something like that for the user. | 19:01 |
| kfox1111 | then need to put that secret into a kube secret. | 19:01 |
| jascott1 | yeah i think I got all that from the doc but has missed the --set line and defining the $key | 19:08 |
| jascott1 | going to grab some grub and then will try it | 19:08 |
| kfox1111 | k. | 19:10 |
| sdake | toys: https://twitter.com/steven_dake/status/922896978383593472 | 19:13 |
| sdake | jascott1 ^^ | 19:13 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla master: WIP Dockerhub publisher job https://review.openstack.org/513855 | 19:17 |
| *** lrensing has joined #openstack-kolla | 19:20 | |
| *** jascott1 has quit IRC | 19:20 | |
| *** jascott1 has joined #openstack-kolla | 19:21 | |
| *** TxGirlGeek has joined #openstack-kolla | 19:23 | |
| *** jascott1 has quit IRC | 19:26 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla master: WIP Dockerhub publisher job https://review.openstack.org/513855 | 19:27 |
| *** TxGirlGeek has quit IRC | 19:30 | |
| *** TxGirlGeek has joined #openstack-kolla | 19:30 | |
| *** jascott1 has joined #openstack-kolla | 19:33 | |
| *** TxGirlGeek has quit IRC | 19:33 | |
| *** TxGirlGeek has joined #openstack-kolla | 19:34 | |
| *** gfidente has quit IRC | 19:35 | |
| *** jascott1 has quit IRC | 19:35 | |
| *** TxGirlGeek has quit IRC | 19:38 | |
| *** hrw has quit IRC | 19:39 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Add prometheus output to telegraf https://review.openstack.org/514806 | 19:39 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla master: WIP Dockerhub publisher job https://review.openstack.org/513855 | 19:40 |
| *** egonzalez has joined #openstack-kolla | 19:40 | |
| *** hrw has joined #openstack-kolla | 19:40 | |
| *** jascott1 has joined #openstack-kolla | 19:45 | |
| *** jascott1 has quit IRC | 19:48 | |
| *** TxGirlGeek has joined #openstack-kolla | 19:51 | |
| openstackgerrit | chandu proposed openstack/kolla-ansible stable/ocata: Removing groups scope from all to baremetal in baremetal pre-install task under bootstrap-server https://review.openstack.org/514774 | 19:52 |
| *** TxGirlGeek has quit IRC | 19:56 | |
| *** TxGirlGeek has joined #openstack-kolla | 19:57 | |
| *** jascott1 has joined #openstack-kolla | 19:58 | |
| *** jascott1 has quit IRC | 19:59 | |
| *** jascott1 has joined #openstack-kolla | 20:00 | |
| rwellum | sdake: you try-ode them out? | 20:00 |
| jascott1 | sdake neato! what are they for? | 20:00 |
| *** rmart04 has joined #openstack-kolla | 20:03 | |
| *** rmart04 has quit IRC | 20:03 | |
| *** TxGirlGeek has quit IRC | 20:04 | |
| *** bmace has quit IRC | 20:06 | |
| *** bmace has joined #openstack-kolla | 20:06 | |
| rwellum | inc0: kfox1111 same error centos on centos: | 20:14 |
| rwellum | 2017-10-24 20:14:01.941+0000: 20735: error : virNetTLSContextCheckCertFile:120 : Cannot read CA certificate '/etc/pki/CA/cacert.pem': No such file or directory | 20:14 |
| *** ktibi has quit IRC | 20:16 | |
| jamesbenson | sdake you've got some old servers there ;-) | 20:18 |
| jamesbenson | http://1.bp.blogspot.com/-wtVCNAAt3E0/Ve98ynS4kbI/AAAAAAAABhI/i5RaddQel-M/s1600/eniac_vacuum_tubes.png | 20:19 |
| *** TxGirlGeek has joined #openstack-kolla | 20:20 | |
| *** itlinux has quit IRC | 20:22 | |
| *** TxGirlGeek has quit IRC | 20:29 | |
| *** itlinux has joined #openstack-kolla | 20:30 | |
| *** jascott1 has quit IRC | 20:31 | |
| *** jascott1 has joined #openstack-kolla | 20:31 | |
| *** TxGirlGeek has joined #openstack-kolla | 20:32 | |
| *** jascott1 has quit IRC | 20:32 | |
| *** jascott1 has joined #openstack-kolla | 20:33 | |
| *** jascott1 has quit IRC | 20:35 | |
| *** ansmith has quit IRC | 20:35 | |
| *** lukepatrick has quit IRC | 20:42 | |
| *** danardelean has joined #openstack-kolla | 20:43 | |
| *** paramite has quit IRC | 20:46 | |
| openstackgerrit | Merged openstack/kolla master: Create murano applications package for import https://review.openstack.org/513632 | 20:47 |
| *** TxGirlGeek has quit IRC | 20:47 | |
| *** danardelean has quit IRC | 20:47 | |
| *** TxGirlGeek has joined #openstack-kolla | 20:47 | |
| inc0 | rwellum: do you have /etc/pki/CA in your container?:) | 20:50 |
| *** egonzalez has quit IRC | 21:03 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla master: WIP Dockerhub publisher job https://review.openstack.org/513855 | 21:06 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla master: WIP Dockerhub publisher job https://review.openstack.org/513855 | 21:07 |
| *** TxGirlGeek has quit IRC | 21:11 | |
| *** TxGirlGeek has joined #openstack-kolla | 21:12 | |
| *** jtriley has quit IRC | 21:12 | |
| *** TxGirlGeek has quit IRC | 21:16 | |
| *** jascott1 has joined #openstack-kolla | 21:17 | |
| *** itlinux has quit IRC | 21:18 | |
| rwellum | inc0: I do but not the cacert.pem | 21:18 |
| jascott1 | hmm my pvc is still pending... | 21:19 |
| jascott1 | just to triple check, I dont have to create a vol with ceoh AND i dont have to create a pv? | 21:19 |
| jascott1 | i get | 21:20 |
| jascott1 | rbd-test Pending rbd 2m | 21:20 |
| rwellum | inc0: so possibly an issue with my host vm? It's an openstack vm - so maybe there's some setup missing. | 21:20 |
| rwellum | 3 months jascott1 ... | 21:20 |
| jascott1 | haha | 21:22 |
| jascott1 | maybe I can shave some time off that... only looking for dev quality | 21:22 |
| *** danardelean has joined #openstack-kolla | 21:23 | |
| jascott1 | rwellum ah so you are expert now :) | 21:23 |
| jascott1 | how do I troubleshoot? | 21:23 |
| *** tvignaud has joined #openstack-kolla | 21:24 | |
| *** TxGirlGeek has joined #openstack-kolla | 21:24 | |
| rwellum | I went through the redhat ceph webpage many times..... But this is with added complexity - containers. | 21:25 |
| *** danardelean has quit IRC | 21:26 | |
| *** itlinux has joined #openstack-kolla | 21:26 | |
| *** ansmith has joined #openstack-kolla | 21:26 | |
| inc0 | rwellum: can't say, it's strange tho | 21:27 |
| inc0 | I normally deploy without any additional conf needed | 21:27 |
| inc0 | on top of openstack vm | 21:27 |
| rwellum | Also weird that same VM, same infra - no problem with libvirt and ocata | 21:27 |
| inc0 | then it's not vm | 21:27 |
| inc0 | how about you override just libvirt container | 21:28 |
| inc0 | and use it with ocata? | 21:28 |
| inc0 | to confirm it's pike-libvirt issue | 21:28 |
| inc0 | also, ocata from tarballs? | 21:28 |
| rwellum | Ah good call - to debug | 21:28 |
| rwellum | No ocata is from dockerhub | 21:28 |
| inc0 | right, so it will be older version of libvirt | 21:28 |
| inc0 | try ocata from tarballs plz | 21:28 |
| inc0 | dockerhub images weren't updated | 21:29 |
| rwellum | Ok will do | 21:29 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla master: WIP Dockerhub publisher job https://review.openstack.org/513855 | 21:30 |
| rwellum | So to use ocata libvirt I'll need to think this out, because I'm using the docker registry | 21:30 |
| inc0 | rwellum: just setup second registry on different port | 21:31 |
| inc0 | or deploy all ocata but from tarballs | 21:31 |
| inc0 | seems like new libvirt have some strange issues with certs | 21:31 |
| jamesbenson | inc0: I deployed but for some reason some didn't ssh and failed on the node, can I just hit deploy again to finish up that node or two that it failed on? | 21:32 |
| jamesbenson | or does it need to destroy/deploy again? | 21:32 |
| inc0 | depends, what task failed? | 21:33 |
| inc0 | which* | 21:33 |
| jamesbenson | glance : Ensuring config directories exist | 21:33 |
| inc0 | just hit deploy | 21:33 |
| inc0 | if it's after rabbitmq/mariadb you should be fine | 21:33 |
| jamesbenson | and the setup failed for one node: fatal: [r2-710-07]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: mux_client_hello_exchange: write packet: Broken pipe\r\n", "unreachable": true} | 21:34 |
| inc0 | if it's mariadb, you still should be fine with rerunning deploy, but since it has all the clustering gimmics, I'd suggest clean redeploy | 21:34 |
| jamesbenson | yeah, the "dirty" deploy is still going.... hasn't errored out | 21:35 |
| jascott1 | hmm found this | 21:36 |
| jascott1 | librados: client.admin authentication error (110) Connection timed out rbd: couldn't connect to the cluster! | 21:37 |
| *** TxGirlGeek has quit IRC | 21:37 | |
| *** goldyfruit has quit IRC | 21:39 | |
| inc0 | jascott1: check security groups in cluster | 21:42 |
| jascott1 | will do | 21:42 |
| *** jamesbenson has quit IRC | 21:43 | |
| *** TxGirlGeek has joined #openstack-kolla | 21:45 | |
| rwellum | inc0: you mentioned remove ca_file="" - perhaps I should keep this line? | 21:46 |
| rwellum | actually maybe you didn't mentioned removing that line... | 21:47 |
| inc0 | or... | 21:48 |
| inc0 | let me check somethign | 21:48 |
| inc0 | hmf | 21:49 |
| rwellum | kk | 21:49 |
| inc0 | maybe play with this conf option? | 21:49 |
| rwellum | jascott1 https://www.irccloud.com/pastebin/lREw6TKt/ | 21:49 |
| *** TxGirlGeek has quit IRC | 21:49 | |
| *** TxGirlGeek has joined #openstack-kolla | 21:49 | |
| jascott1 | rwellum thanks. all that looks good | 21:51 |
| jascott1 | (afaik) | 21:51 |
| rwellum | That's cool then | 21:51 |
| jascott1 | seems like its the linkage from the provisioner to ceph | 21:52 |
| *** TxGirlGeek has quit IRC | 21:53 | |
| *** TxGirlGeek has joined #openstack-kolla | 21:54 | |
| rwellum | Yeah - I know the OSD's have their own private n/w to communicate - wonder if this is a problem. Have you/can you try running the rdb command manually? | 21:55 |
| rwellum | inc0: ? what conf option? | 21:55 |
| jascott1 | rwellum I can manually create a volume with `rbd create myvol --size 1G --pool kollavolumes` | 21:58 |
| kfox1111 | back. | 21:58 |
| rwellum | Huh that's odd jascott1 - seems to be very healthy | 21:59 |
| jascott1 | kfox1111 when you get a sec can you sanity check this? | 21:59 |
| jascott1 | http://paste.openstack.org/show/624523/ | 21:59 |
| kfox1111 | jascott1: looks prety sane. is your mon's ip really 172.17.0.1? | 22:00 |
| jascott1 | thats docker0 | 22:01 |
| jascott1 | was in the etherpad | 22:01 |
| kfox1111 | hmm... I think in the etherpad I used $(hostname) | 22:01 |
| kfox1111 | hmm... I think in the etherpad I used $(hostname -s) | 22:01 |
| kfox1111 | I wonder what the mon is listenign on though. | 22:02 |
| *** danardelean has joined #openstack-kolla | 22:02 | |
| jascott1 | let me see | 22:02 |
| *** danardelean has quit IRC | 22:03 | |
| jascott1 | this etherpad had that 172.x https://etherpad.openstack.org/p/kolla-k8s-braindump | 22:03 |
| inc0 | jascott1: use regular mon | 22:03 |
| *** danardelean has joined #openstack-kolla | 22:03 | |
| jascott1 | probably the 192.x from ens3? | 22:03 |
| kfox1111 | jascott1: yeah, but not sure that exact arg ever got tested. | 22:03 |
| jascott1 | gotcha | 22:03 |
| kfox1111 | netstat -pnl | grep mon | 22:03 |
| *** TxGirlGeek has quit IRC | 22:04 | |
| jascott1 | ya its the 192.x let me fix that | 22:04 |
| jascott1 | thanks | 22:04 |
| kfox1111 | np. | 22:04 |
| *** lpetrut has quit IRC | 22:05 | |
| rwellum | Good catch | 22:05 |
| *** TxGirlGeek has joined #openstack-kolla | 22:05 | |
| *** kolla-slack has quit IRC | 22:06 | |
| *** kolla-slack has joined #openstack-kolla | 22:06 | |
| *** danardelean has quit IRC | 22:07 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Telegraf should only output to influxdb when influxdb is enabled https://review.openstack.org/514851 | 22:09 |
| jascott1 | inc0 i wrapped the terraform and ansible in a little bash script so I have a one liner to destroy and rebuild everything from scratch. its uh very nice-uh! | 22:09 |
| SamYaple | jascott1: now alias it to `JUSTWORKALREADY` | 22:09 |
| jascott1 | haha | 22:10 |
| jascott1 | aliases for the win \o/ | 22:10 |
| *** goldyfruit has joined #openstack-kolla | 22:12 | |
| rwellum | jascott1: shouldn't you have an ansible playbook to wrap the ansinble playbook. :) | 22:12 |
| SamYaple | rwellum: soon youll be asking for a kolla api | 22:12 |
| rwellum | Or like some code I was looking at ruby->calling bash->calling python | 22:12 |
| inc0 | rwellum: that's what we're doing in gates | 22:12 |
| jascott1 | yass! | 22:12 |
| rwellum | SamYaple: +1 :) | 22:13 |
| rwellum | inc0: - you said above 'try this config: | 22:13 |
| jascott1 | SamYapple ive got a `export all="crd,endpoints,all --all-namespaces"` so I can do `k get $all` and acutally see everyting | 22:15 |
| jascott1 | instead of the lies k8s tells by default | 22:15 |
| kfox1111 | hehe. | 22:15 |
| jascott1 | well its not everyting, cause some like sc,pv wont work with namspace | 22:15 |
| jascott1 | this is interesting but i think k8s bash completions are equally useful https://medium.com/@Mikulas/kubectl-repl-e6f86cac8626 | 22:17 |
| SamYaple | jascott1: that just means youll be screwed when youre not on that box | 22:18 |
| jascott1 | haha exactly | 22:18 |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Add prometheus output to telegraf https://review.openstack.org/514806 | 22:19 |
| inc0 | ok, I'm done with dockerhub publisher it seems | 22:19 |
| jascott1 | woot sir | 22:19 |
| jascott1 | maximum woot. | 22:20 |
| inc0 | well now I need to merge it and hope for the best | 22:20 |
| inc0 | because I can't *really* test it | 22:20 |
| SamYaple | inc0: it wont work | 22:20 |
| inc0 | y tho | 22:20 |
| SamYaple | not all comments in first PS were addressed | 22:21 |
| rwellum | Even worse when you have it aliased in your bashrc and then you forget and spend weeks wondering why grep doesn't work... 🙋♂️ | 22:21 |
| inc0 | yeah I still need to fix .zuul | 22:21 |
| inc0 | but what I'm saying is logic is as close to ready as I can get it | 22:22 |
| jascott1 | ah the sweet taste of success | 22:22 |
| jascott1 | it worked | 22:22 |
| inc0 | and I can't do it in post because rather than build->push I'll just use --push | 22:22 |
| rwellum | jascott1: +1! | 22:22 |
| jascott1 | thanks everyone | 22:22 |
| inc0 | jascott1: multinode worky? | 22:22 |
| jascott1 | haha no | 22:22 |
| jascott1 | BUT | 22:22 |
| inc0 | tomorrow | 22:22 |
| jascott1 | should be a piece of cake | 22:23 |
| jascott1 | what? the night is young and i am not | 22:23 |
| rwellum | jascott1: ceph happy with just one node? | 22:23 |
| jascott1 | ya | 22:23 |
| inc0 | yeah, we just need to figure out this cluster networking issue | 22:23 |
| SamYaple | inc0: im a bit worried about https://review.openstack.org/#/c/513855/18/tests/templates/kolla-build.conf.j2 | 22:23 |
| SamYaple | inc0: that was implemented for a reason | 22:23 |
| SamYaple | because the gates were pulling in upstream images | 22:23 |
| inc0 | lokolla thing? | 22:23 |
| SamYaple | why not just do a post job and retag it? | 22:23 |
| rwellum | Pretty sure that is not supposed to work | 22:23 |
| SamYaple | yes | 22:23 |
| SamYaple | rwellum: its not but here we are | 22:23 |
| jascott1 | dont suppose we have ceph multinode example in gates or somewher? | 22:23 |
| *** goldyfruit has quit IRC | 22:23 | |
| inc0 | well, plan is that gates *will* pull upstream images | 22:24 |
| inc0 | with cache proxy | 22:24 |
| rwellum | Well is ceph single node works why the heck am I messing with LVM... | 22:24 |
| SamYaple | right but the way you have the patch written that only does it for "publisher" | 22:24 |
| rwellum | (if) | 22:24 |
| SamYaple | when it explicitly needs to *not* do that for publisher | 22:25 |
| inc0 | so for first stab, I'm ok with publisher pushing images to dockerhub and ansible gates pulling images from dockerhub+cache | 22:25 |
| inc0 | after that we'll deal with better testing pipeline | 22:25 |
| inc0 | but as it stands today, we need some sort of publishing images and tarballs won't work | 22:26 |
| SamYaple | i dont think we are tlaking about the same thing exactly | 22:26 |
| inc0 | maybe, what are you talking about? | 22:26 |
| *** salv-orlando has quit IRC | 22:26 | |
| *** salv-orlando has joined #openstack-kolla | 22:27 | |
| SamYaple | when building your images, you should *keep* the lokolla name and retag right before you push the images. otherwise you run the risk of not building and only pulling images or pulling then pushing the same image (the reason "lokolla" was added in the first place to prevent) | 22:27 |
| SamYaple | right now you are doing the opposite of that | 22:27 |
| inc0 | ahh | 22:27 |
| inc0 | I see what you're saying | 22:27 |
| kfox1111 | more years go by.... diagraming stuff still sucks.... | 22:28 |
| jascott1 | no doubt | 22:28 |
| jascott1 | i like plantUML | 22:28 |
| jascott1 | definitely rules for seq diagrams | 22:29 |
| jascott1 | kinda hit or miss for other ones | 22:29 |
| inc0 | let me think about it for a moment, I don't want to push in post job because it's going to be nightmarish long | 22:29 |
| kfox1111 | yeah. diagraming uml these days isn't too bad. | 22:29 |
| inc0 | since bash is not multithreaded | 22:29 |
| kfox1111 | diagraming other things, not as much. :( | 22:29 |
| jascott1 | they added deployment diagrams if thats what you are doing | 22:30 |
| jascott1 | its ok | 22:30 |
| kfox1111 | nodes, racks, switches, networks, services, etc. | 22:30 |
| SamYaple | inc0: where do you think you are going to push from? | 22:31 |
| SamYaple | inc0: you cant push from the gate | 22:31 |
| jascott1 | god their website is hideous http://plantuml.com/deployment-diagram | 22:31 |
| inc0 | SamYaple: why? | 22:31 |
| inc0 | well | 22:31 |
| SamYaple | has to be post (or other protected) pipelines | 22:31 |
| inc0 | periodic | 22:31 |
| *** salv-orlando has quit IRC | 22:31 | |
| SamYaple | hmmm let me find the list | 22:31 |
| inc0 | it's going to be periodic job | 22:31 |
| inc0 | periodic is fine I think | 22:31 |
| rwellum | jascott1: looks like my kids drawings | 22:31 |
| SamYaple | btw, you keep misunderstanding what i mean by post | 22:32 |
| *** TxGirlGeek has quit IRC | 22:32 | |
| SamYaple | i mean post-run not post job | 22:32 |
| inc0 | I know | 22:32 |
| inc0 | post playbook | 22:32 |
| SamYaple | yea, so you can move everythign there | 22:32 |
| SamYaple | that runs on the job | 22:32 |
| SamYaple | not just the post pipeline | 22:32 |
| inc0 | tagging and pushing, yeah, but if we figure out how to use --push safely it's going to be just fine | 22:33 |
| jascott1 | rwellum ya you can mess with the style. a lot of IDEs support it so it makes it easy to work with. I just like I can keep it in git | 22:33 |
| rwellum | Yeah that's cool. I have played with https://www.draw.io/ - and that's pretty good too. | 22:33 |
| SamYaple | ah yea inc0 even gate is good | 22:34 |
| jascott1 | saw a project the other day that was outputing some network tracing to automatically create plantuml seq diagrams | 22:34 |
| jascott1 | pretty neat | 22:34 |
| rwellum | I've used it with customers before for an on-the-fly demo and then used gdrive - they can interact and get a copy etc | 22:34 |
| *** TxGirlGeek has joined #openstack-kolla | 22:34 | |
| SamYaple | inc0: im of the opinion that you should change kolla-build a bit to allow a second push-only call | 22:35 |
| SamYaple | trying to get them to work toeghther in the gate is going to be a problem | 22:35 |
| inc0 | that might be option | 22:35 |
| SamYaple | i forget, do you only push *after* all images build successfully? | 22:36 |
| SamYaple | or do you push as you go? | 22:36 |
| inc0 | I think you build as they go | 22:36 |
| inc0 | so it's quite optimal | 22:36 |
| inc0 | also multithreaded | 22:36 |
| SamYaple | yea thats bad. what if an image fails to build, then youll have pushed a bunch of images out of sync | 22:36 |
| *** mummer has quit IRC | 22:37 | |
| inc0 | I'll need to think about it | 22:38 |
| kfox1111 | draw.io is quite interesting.... | 22:38 |
| inc0 | if we build->push we probably gonna hit timeout | 22:38 |
| inc0 | unless it's multithreaded | 22:38 |
| kfox1111 | the source is even open it seems. | 22:38 |
| SamYaple | if build AND push and it fails you break your images | 22:38 |
| inc0 | maybe it's worth to just write some python to retag and push multithreaded | 22:39 |
| *** TxGirlGeek has quit IRC | 22:39 | |
| inc0 | yeha I hear you | 22:39 |
| rwellum | yes kfox1111 - you can suggest changes etc | 22:40 |
| rwellum | And you can export in html etc | 22:40 |
| rwellum | XML | 22:41 |
| *** TxGirlGeek has joined #openstack-kolla | 22:42 | |
| *** TxGirlGeek has quit IRC | 22:49 | |
| *** TxGirlGeek has joined #openstack-kolla | 22:51 | |
| *** itlinux has quit IRC | 22:59 | |
| sdake | jascott1 for my tube amp :) | 22:59 |
| *** TxGirlGe_ has joined #openstack-kolla | 23:00 | |
| *** TxGirlGeek has quit IRC | 23:01 | |
| *** TxGirlGe_ has quit IRC | 23:11 | |
| *** hongbin has quit IRC | 23:14 | |
| *** thomasem has quit IRC | 23:21 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Optimize zuul v3 jobs https://review.openstack.org/513612 | 23:22 |
| *** salv-orlando has joined #openstack-kolla | 23:28 | |
| *** lrensing has quit IRC | 23:28 | |
| *** thomasem has joined #openstack-kolla | 23:28 | |
| *** TxGirlGeek has joined #openstack-kolla | 23:31 | |
| *** TxGirlGeek has quit IRC | 23:31 | |
| *** salv-orlando has quit IRC | 23:32 | |
| jascott1 | sdake what kind? | 23:32 |
| sdake | @jascott1 one of these http://www.stevehuffphoto.com/line-magnetic-219ia-integrated-tube-amp-review-300b-and-845-tube-magic/ | 23:33 |
| jascott1 | sweet | 23:35 |
| * jascott1 updates festivus wishlist | 23:36 | |
| jascott1 | kfox1111 can you tell me purpose of this in etherpad: swapoff /dev/dm-1 | 23:39 |
| jascott1 | which component is that for? | 23:40 |
| kfox1111 | jascott1: k8s 1.8+ doesn't like swap on. it fails a check in kubelet startup. | 23:40 |
| jascott1 | ah thanks | 23:40 |
| kfox1111 | could probably change that to 'swapoff -a' I think. | 23:40 |
| jascott1 | thats before kubeadm init i think> | 23:41 |
| jascott1 | ? | 23:41 |
| kfox1111 | yeah. | 23:41 |
| kfox1111 | should be one of the first things I think. | 23:41 |
| jascott1 | k i will stick it with the firewall off and selinux etc | 23:41 |
| kfox1111 | yeah. that works. | 23:41 |
| jascott1 | all nodes? | 23:42 |
| kfox1111 | yeah. | 23:44 |
| jascott1 | kfox1111 just for install or persist after reboot in fstab? | 23:47 |
| kfox1111 | probably need to remove it out of fstab too. | 23:47 |
| jascott1 | got it. thanks! youve save me much googlin' | 23:47 |
| kfox1111 | np. glad to help. :) | 23:48 |
| kfox1111 | rarrg. | 23:51 |
| kfox1111 | I really like graphviz, right up until I dont. :( | 23:52 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/509443 | 23:52 |
| kfox1111 | Its super helpful most of the time. but then when you get to a certain point in the graph, it just falls all apart. | 23:52 |
| jascott1 | graphviz: when it just has to be ugly | 23:52 |
| kfox1111 | Its super easy to do a lot of things with it though. far too few graphing tools just let you put in relations and have it auto build your layout. | 23:53 |
| kfox1111 | "here's some raw data. graph it." | 23:53 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
