| *** hogepodge has joined #openstack-kolla | 00:00 | |
| *** sacharya has joined #openstack-kolla | 00:04 | |
| *** hogepodge has quit IRC | 00:06 | |
| *** hogepodge has joined #openstack-kolla | 00:06 | |
| *** hogepodge has quit IRC | 00:08 | |
| *** hogepodge has joined #openstack-kolla | 00:09 | |
| *** sacharya has quit IRC | 00:09 | |
| *** hogepodge has quit IRC | 00:11 | |
| *** goldyfruit has quit IRC | 00:12 | |
| *** hogepodge has joined #openstack-kolla | 00:12 | |
| *** eaguilar_ has joined #openstack-kolla | 00:21 | |
| *** eaguilar has quit IRC | 00:22 | |
| openstackgerrit | Matt McEuen proposed openstack/kolla: Configure local registries to proxy docker hub https://review.openstack.org/413806 | 00:24 |
|---|---|---|
| *** tonyb has quit IRC | 00:43 | |
| *** tonyb has joined #openstack-kolla | 00:44 | |
| *** jmccarthy has quit IRC | 00:46 | |
| *** jmccarthy has joined #openstack-kolla | 00:47 | |
| *** saneax is now known as saneax-_-|AFK | 00:48 | |
| *** saneax-_-|AFK is now known as saneax | 00:51 | |
| *** saneax is now known as saneax-_-|AFK | 00:55 | |
| *** williamcaban has joined #openstack-kolla | 00:59 | |
| *** schwicht_at_work has quit IRC | 01:02 | |
| *** williamcaban has quit IRC | 01:03 | |
| *** schwicht has joined #openstack-kolla | 01:03 | |
| *** sacharya has joined #openstack-kolla | 01:05 | |
| *** gomarivera has joined #openstack-kolla | 01:09 | |
| *** sacharya has quit IRC | 01:10 | |
| *** zhubingbing has joined #openstack-kolla | 01:11 | |
| *** schwicht has quit IRC | 01:19 | |
| openstackgerrit | jimmygc proposed openstack/kolla: Add sanity check in macros.j2 https://review.openstack.org/414848 | 01:21 |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible: Update README.rst in kolla-ansible https://review.openstack.org/410528 | 01:25 |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible: Fix wrong register in ceph reconfigure task https://review.openstack.org/415150 | 01:25 |
| *** schwicht has joined #openstack-kolla | 01:26 | |
| *** sayantan_ has quit IRC | 01:27 | |
| *** sayantan_ has joined #openstack-kolla | 01:30 | |
| Pavo | seems like all my issues have been SELinux related | 01:32 |
| *** sacharya has joined #openstack-kolla | 01:33 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Spec - Zero-downtime upgrade in Kolla https://review.openstack.org/409598 | 01:35 |
| *** sayantan_ has quit IRC | 01:38 | |
| *** eaguilar_ has quit IRC | 01:38 | |
| *** pramodrj07 has quit IRC | 01:39 | |
| *** duonghq has joined #openstack-kolla | 01:42 | |
| duonghq | morning guys | 01:44 |
| *** tovin07_ has joined #openstack-kolla | 01:45 | |
| duonghq | sup tovin07_ | 01:46 |
| tovin07_ | sup duonghq | 01:47 |
| *** schwicht has quit IRC | 01:48 | |
| *** eaguilar has joined #openstack-kolla | 01:48 | |
| *** dave-mccowan has joined #openstack-kolla | 01:48 | |
| *** Pavo has quit IRC | 01:51 | |
| *** gomarivera has quit IRC | 02:03 | |
| *** sacharya has quit IRC | 02:04 | |
| *** duonghq has quit IRC | 02:10 | |
| *** duonghq has joined #openstack-kolla | 02:28 | |
| *** schwicht has joined #openstack-kolla | 02:30 | |
| openstackgerrit | jimmygc proposed openstack/kolla: Add sanity check in macros.j2 https://review.openstack.org/414848 | 02:31 |
| *** mdnadeem has joined #openstack-kolla | 02:32 | |
| *** dave-mccowan has quit IRC | 02:38 | |
| openstackgerrit | Kuo-tung Kao proposed openstack/kolla-ansible: provide a universal collectd template https://review.openstack.org/414499 | 02:38 |
| *** huzhengchuan has joined #openstack-kolla | 02:45 | |
| *** yuanying has quit IRC | 02:47 | |
| *** eaguilar has quit IRC | 02:47 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Add empty precheck.yml file for tacker and octavia https://review.openstack.org/415120 | 02:51 |
| *** williamcaban has joined #openstack-kolla | 03:01 | |
| *** sayantan_ has joined #openstack-kolla | 03:03 | |
| *** l4yerffeJ_ has joined #openstack-kolla | 03:04 | |
| *** Jeffrey4l has joined #openstack-kolla | 03:04 | |
| *** williamcaban has quit IRC | 03:05 | |
| *** Jeffrey4l has quit IRC | 03:07 | |
| *** Jeffrey4l has joined #openstack-kolla | 03:07 | |
| *** tovin07 has quit IRC | 03:17 | |
| *** hfu has quit IRC | 03:22 | |
| *** severion has quit IRC | 03:22 | |
| *** v1k0d3n has joined #openstack-kolla | 03:22 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Move gnocchi precheck into its own role https://review.openstack.org/415326 | 03:23 |
| *** hfu has joined #openstack-kolla | 03:23 | |
| *** yuanying has joined #openstack-kolla | 03:24 | |
| openstackgerrit | jimmygc proposed openstack/kolla: Add sanity check in macros.j2 https://review.openstack.org/414848 | 03:25 |
| *** jtriley has joined #openstack-kolla | 03:26 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible: WIP add fluentd role https://review.openstack.org/407392 | 03:28 |
| *** tovin07 has joined #openstack-kolla | 03:33 | |
| *** yuanying has quit IRC | 03:48 | |
| *** yuanying has joined #openstack-kolla | 03:51 | |
| *** prameswar has joined #openstack-kolla | 03:52 | |
| *** v1k0d3n has quit IRC | 03:53 | |
| *** v1k0d3n has joined #openstack-kolla | 03:53 | |
| *** v1k0d3n has quit IRC | 03:57 | |
| *** v1k0d3n has joined #openstack-kolla | 03:57 | |
| *** sacharya has joined #openstack-kolla | 04:04 | |
| *** v1k0d3n has quit IRC | 04:05 | |
| *** mdnadeem has quit IRC | 04:05 | |
| *** v1k0d3n has joined #openstack-kolla | 04:06 | |
| *** sacharya has quit IRC | 04:09 | |
| *** v1k0d3n has quit IRC | 04:11 | |
| *** tovin07 has quit IRC | 04:16 | |
| *** skramaja_ has quit IRC | 04:25 | |
| *** Satya_ has joined #openstack-kolla | 04:27 | |
| *** v1k0d3n has joined #openstack-kolla | 04:27 | |
| Satya_ | We deployed openstack using latest master branch kolla trunk code. Cinder volume is trying to make socket connection with ceph node every few seconds after few hours rabiitmq cinder volume connection is getting disconnected | 04:29 |
| Satya_ | opening connection to ceph cluster (timeout=-1). _connect_to_rados /var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:220 | 04:30 |
| Satya_ | 2016-12-27 19:47:12.983 63 ERROR oslo.messaging._drivers.impl_rabbit [-] [db4b0016-15e2-486d-831f-2bce7e396a1b] AMQP server x.x.x.x:5672 closed the connection. Check login credentials: Socket closed | 04:31 |
| Satya_ | Did anyone observe this issue with latest trunk code | 04:31 |
| Satya_ | and also I can see this error in cinder-volume log file | 04:32 |
| Satya_ | ImageNotFound: error opening image volume-dde8f952-ad88-4028-ad58-2937a118b001 at snapshot None | 04:32 |
| sp_ | Jeffrey4l: hi... | 04:32 |
| Jeffrey4l | sup sp_ | 04:32 |
| sp_ | Jeffrey4l: please confirm this bug https://bugs.launchpad.net/kolla/+bug/1650269 | 04:32 |
| openstack | Launchpad bug 1650269 in kolla "cover environment is not get created" [Wishlist,In progress] - Assigned to Surya Prakash Singh (confisurya) | 04:32 |
| Jeffrey4l | sp_, it is in progress. no need confirm ; | 04:33 |
| sp_ | Jeffrey4l: thanks | 04:33 |
| Jeffrey4l | but the fix patch need some improve. here https://review.openstack.org/#/c/411313/' | 04:33 |
| Satya_ | Hi Jeffrey41 have you observed above issue | 04:34 |
| * Jeffrey4l is scroll back | 04:34 | |
| *** v1k0d3n has quit IRC | 04:34 | |
| Jeffrey4l | Satya_, no. never saw such error. | 04:36 |
| Satya_ | ok thanks | 04:36 |
| Jeffrey4l | are u meaning at first cinder-volume works. and *after hours* it raise some error? Satya_ | 04:37 |
| Satya_ | yes | 04:37 |
| Satya_ | from my initial analysis it seems once in 30 seconds cinder volume is trying to make a socket connection with ceph node | 04:38 |
| Jeffrey4l | wired. did u make anything to the rabbitmq? the "check login credentials" is weird. | 04:38 |
| Satya_ | in one hour it will make huge number of connection | 04:38 |
| Jeffrey4l | cinder-volume <-> rabbitmq? | 04:38 |
| Satya_ | cinder-volume - rabbitmq connections is getting effected after 1 hour of deployment | 04:39 |
| Jeffrey4l | can u create create new volume and attache it to instance when it works? Satya_ | 04:39 |
| Satya_ | in initial few hours i can create the volume | 04:39 |
| Jeffrey4l | then what's the "huge number of connection"? | 04:39 |
| Satya_ | after few hours it goes in to error state | 04:39 |
| *** Pavo has joined #openstack-kolla | 04:40 | |
| Satya_ | cinder-volume we are running on controller node | 04:40 |
| Satya_ | cinder volume on controller node to ceph node | 04:40 |
| Jeffrey4l | how did u get the number of the connections? | 04:41 |
| Satya_ | cinder volume is trying to make a connection after every 30 seconds | 04:41 |
| Pavo | evening | 04:41 |
| Satya_ | i can see that in the debug logs and lsof commands shows many connections | 04:41 |
| Jeffrey4l | evening Pavo. | 04:41 |
| Jeffrey4l | Satya_, could u post these logs? or better file a bug in kolla? | 04:41 |
| Pavo | Jeffrey4l you want me to enable SELinux again and get audit.log? | 04:42 |
| Satya_ | i will create a bug | 04:42 |
| Jeffrey4l | Pavo, yep. I want make sure why selinux block the image upload. | 04:42 |
| Pavo | roger | 04:42 |
| Pavo | doing it now | 04:42 |
| Pavo | how about permissive? | 04:43 |
| Satya_ | logs are give here | 04:43 |
| Pavo | it should still log | 04:43 |
| *** goldyfruit has joined #openstack-kolla | 04:43 | |
| Jeffrey4l | kolla will announce that it supports selinux in the future. so we need fix selinux issue. | 04:43 |
| Jeffrey4l | Pavo, thanks. | 04:43 |
| Jeffrey4l | "it should still log" sorry? | 04:43 |
| Jeffrey4l | launch time. brb. | 04:44 |
| Pavo | yeah set to permissive should still log but allow the action | 04:44 |
| Jeffrey4l | lunch* ;( | 04:45 |
| *** goldyfruit has quit IRC | 04:45 | |
| Jeffrey4l | Pavo, roger. work for me. please post the audit.log to the bug. thanks. | 04:45 |
| Pavo | roger | 04:45 |
| Satya_ | Jeffrey41 , You can find the logs in the below path | 04:49 |
| Satya_ | http://paste.openstack.org/show/593494/ | 04:49 |
| Satya_ | this is for quick reference, I will create a bug for this issue | 04:49 |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible: WIP add fluentd role https://review.openstack.org/407392 | 04:50 |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible: WIP add fluentd role https://review.openstack.org/407392 | 04:58 |
| *** williamcaban has joined #openstack-kolla | 05:02 | |
| *** skramaja has joined #openstack-kolla | 05:03 | |
| *** sacharya has joined #openstack-kolla | 05:06 | |
| *** williamcaban has quit IRC | 05:06 | |
| *** sacharya has quit IRC | 05:10 | |
| *** unicell has joined #openstack-kolla | 05:10 | |
| *** unicell has joined #openstack-kolla | 05:10 | |
| *** hfu has quit IRC | 05:10 | |
| *** hfu has joined #openstack-kolla | 05:13 | |
| Pavo | Jeffrey4l you sure you want this audit log? | 05:15 |
| Pavo | its a ton of info | 05:15 |
| Jeffrey4l | Pavo, yep. | 05:15 |
| Jeffrey4l | what's the size? | 05:16 |
| Jeffrey4l | could u find the part when u create images in horizon? | 05:16 |
| Pavo | I have 3 of them | 05:16 |
| Pavo | audit.log, audit.log1, audit.log2 | 05:16 |
| Jeffrey4l | or just reproduce this bug again and only post the logs in the latest few hours? | 05:17 |
| Jeffrey4l | it is hard for me to find which part is what i want. | 05:17 |
| Pavo | ok | 05:18 |
| Pavo | I tried grep horizon and nothing | 05:18 |
| Pavo | tried grep glance and nothing | 05:18 |
| Jeffrey4l | can u reproduce this issue and post the audit.log file out? | 05:19 |
| Pavo | gonna try | 05:19 |
| Jeffrey4l | cool. | 05:19 |
| Pavo | uploading image now, if that doesn't reproduce the issue, wil enable SELinux like normal | 05:20 |
| Pavo | right now its in permissive | 05:20 |
| Pavo | would it be logging to the original log or the .2 one? | 05:21 |
| Jeffrey4l | roger. | 05:21 |
| Jeffrey4l | Pavo, audit.log should be OK. | 05:21 |
| Pavo | ok | 05:21 |
| Pavo | well looks like I have to enable SELinux permissive didn't log anything | 05:23 |
| *** lamt has joined #openstack-kolla | 05:23 | |
| openstackgerrit | Surya Prakash Singh proposed openstack/kolla-kubernetes: An improvement as per global-requirements https://review.openstack.org/415342 | 05:24 |
| *** hfu has quit IRC | 05:25 | |
| Pavo | Jeffrey4l do you know when the latest murano dashboard fix will be sent to pip kolla package? | 05:25 |
| Jeffrey4l | Pavo, until kolla 4.0b3, if it is merged before that. | 05:26 |
| Pavo | how long before that happens? | 05:27 |
| Jeffrey4l | around Jan 25 | 05:27 |
| Jeffrey4l | you can try my patch direct. ( it almost done ) | 05:28 |
| Pavo | I tried but missing the config.py file | 05:29 |
| *** lamt has quit IRC | 05:29 | |
| Pavo | the pip package install of kolla doesn't include the config.py file | 05:29 |
| Jeffrey4l | what do u mean by "mission"? | 05:29 |
| Jeffrey4l | try install from git repo ;) | 05:29 |
| Pavo | and the exec-startup.sh file has a ton more than the one from pip kolla 3.0.1 | 05:30 |
| Jeffrey4l | yep. lots of change are made between 3.0 and 4.0. it is impossible to backport the murano ui to 3.0 | 05:30 |
| Jeffrey4l | i think you have to try 4.0. | 05:30 |
| Pavo | wouldn't I have to uninstall the pip version and rebuild images then? | 05:31 |
| Jeffrey4l | yep. | 05:31 |
| Pavo | I might just do that but there are a ton of known bugs with master through right | 05:33 |
| Jeffrey4l | yes. you can try it. but it may break easily. | 05:41 |
| *** Pavo has quit IRC | 05:53 | |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible: Update README.rst in kolla-ansible https://review.openstack.org/410528 | 05:55 |
| *** Pavo has joined #openstack-kolla | 05:57 | |
| *** hfu has joined #openstack-kolla | 05:59 | |
| *** kristian__ has joined #openstack-kolla | 06:00 | |
| *** kristian__ has quit IRC | 06:00 | |
| *** kristian__ has joined #openstack-kolla | 06:00 | |
| *** hfu has quit IRC | 06:01 | |
| *** kristian__ has quit IRC | 06:04 | |
| *** Pavo has quit IRC | 06:07 | |
| *** zhugaoxiao has quit IRC | 06:08 | |
| *** hfu has joined #openstack-kolla | 06:08 | |
| *** saneax-_-|AFK is now known as saneax | 06:14 | |
| *** tovin07 has joined #openstack-kolla | 06:18 | |
| *** Satya__ has joined #openstack-kolla | 06:19 | |
| Satya__ | Hi jeffrey4l | 06:19 |
| *** Satya__ has quit IRC | 06:20 | |
| *** Satya__ has joined #openstack-kolla | 06:20 | |
| *** Satya__ has quit IRC | 06:22 | |
| *** Satya_ has left #openstack-kolla | 06:22 | |
| *** Satya__ has joined #openstack-kolla | 06:23 | |
| *** Satya__ has quit IRC | 06:23 | |
| *** Satya__ has joined #openstack-kolla | 06:23 | |
| *** Satya__ is now known as satroutr | 06:25 | |
| satroutr | Hi jeffrey4l | 06:26 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Move heat precheck into its own role https://review.openstack.org/415359 | 06:26 |
| *** satroutr has quit IRC | 06:29 | |
| *** Satya__ has joined #openstack-kolla | 06:30 | |
| openstackgerrit | Surya Prakash Singh proposed openstack/kolla: Cover environment for kolla https://review.openstack.org/411313 | 06:32 |
| *** hfu has quit IRC | 06:39 | |
| openstackgerrit | Surya Prakash Singh proposed openstack/kolla-kubernetes: An improvement as per global-requirements https://review.openstack.org/415342 | 06:39 |
| *** hfu has joined #openstack-kolla | 06:40 | |
| *** hfu has quit IRC | 06:40 | |
| *** hfu has joined #openstack-kolla | 06:41 | |
| *** l4yerffeJ_ has quit IRC | 06:49 | |
| *** l4yerffeJ_ has joined #openstack-kolla | 06:50 | |
| *** david-lyle has joined #openstack-kolla | 06:50 | |
| *** david-lyle has quit IRC | 06:55 | |
| *** duonghq has quit IRC | 07:03 | |
| *** williamcaban has joined #openstack-kolla | 07:04 | |
| *** williamcaban has quit IRC | 07:09 | |
| *** Satya_ has joined #openstack-kolla | 07:24 | |
| *** sayantan_ has quit IRC | 07:31 | |
| *** Satya_ is now known as satyar | 07:31 | |
| *** Satya_ has joined #openstack-kolla | 07:33 | |
| *** Satya_ has quit IRC | 07:33 | |
| *** satyar has quit IRC | 07:33 | |
| *** satyar has joined #openstack-kolla | 07:34 | |
| *** b_bezak has joined #openstack-kolla | 08:15 | |
| *** prameswar has quit IRC | 08:26 | |
| *** Serlex has joined #openstack-kolla | 08:27 | |
| *** portdirect__ has quit IRC | 08:28 | |
| *** v1k0d3n has joined #openstack-kolla | 08:31 | |
| openstackgerrit | Surya Prakash Singh proposed openstack/kolla-ansible: Removing unsed directory "etc/oslo-config-generator" from kolla-ansible https://review.openstack.org/415399 | 08:35 |
| *** v1k0d3n has quit IRC | 08:36 | |
| *** kristian__ has joined #openstack-kolla | 08:39 | |
| openstackgerrit | Surya Prakash Singh proposed openstack/kolla-ansible: Removing unsed directory "etc/oslo-config-generator" from kolla-ansible https://review.openstack.org/415399 | 08:43 |
| *** david-lyle has joined #openstack-kolla | 08:52 | |
| *** david-lyle has quit IRC | 08:56 | |
| *** gomarivera has joined #openstack-kolla | 09:00 | |
| *** msimonin has joined #openstack-kolla | 09:01 | |
| *** gomarivera has quit IRC | 09:04 | |
| *** williamcaban has joined #openstack-kolla | 09:05 | |
| *** sacharya has joined #openstack-kolla | 09:07 | |
| *** williamcaban has quit IRC | 09:10 | |
| *** sacharya has quit IRC | 09:11 | |
| satyar | SETNAME Satyasanjibani Routray | 09:13 |
| *** l4yerffeJ_ has quit IRC | 09:16 | |
| *** l4yerffeJ_ has joined #openstack-kolla | 09:16 | |
| *** mdnadeem has joined #openstack-kolla | 09:17 | |
| *** satyar has quit IRC | 09:30 | |
| *** negronjl has quit IRC | 09:31 | |
| *** satyar has joined #openstack-kolla | 09:31 | |
| *** negronjl has joined #openstack-kolla | 09:37 | |
| *** ocaner has joined #openstack-kolla | 09:38 | |
| *** pcaruana has joined #openstack-kolla | 09:42 | |
| *** b_bezak has quit IRC | 09:48 | |
| *** l4yerffeJ_ has quit IRC | 09:51 | |
| *** l4yerffeJ_ has joined #openstack-kolla | 09:51 | |
| *** david-lyle has joined #openstack-kolla | 09:52 | |
| *** hfu has quit IRC | 09:56 | |
| *** prameswar has joined #openstack-kolla | 09:56 | |
| *** david-lyle has quit IRC | 09:57 | |
| *** tovin07_ has quit IRC | 10:04 | |
| *** matrohon has joined #openstack-kolla | 10:05 | |
| *** pcaruana has quit IRC | 10:06 | |
| *** satyar has quit IRC | 10:31 | |
| *** matrohon has quit IRC | 10:46 | |
| *** huzhengchuan has quit IRC | 11:01 | |
| *** huzhengchuan has joined #openstack-kolla | 11:02 | |
| *** williamcaban has joined #openstack-kolla | 11:07 | |
| *** sacharya has joined #openstack-kolla | 11:08 | |
| *** sacharya has quit IRC | 11:12 | |
| *** kristian__ has quit IRC | 11:20 | |
| *** kristian__ has joined #openstack-kolla | 11:27 | |
| *** Bico_Fino has joined #openstack-kolla | 11:30 | |
| *** kristian__ has quit IRC | 11:36 | |
| *** Bico_Fino has quit IRC | 11:38 | |
| *** kristian__ has joined #openstack-kolla | 11:43 | |
| *** v1k0d3n has joined #openstack-kolla | 11:59 | |
| *** v1k0d3n has quit IRC | 12:04 | |
| *** williamcaban has joined #openstack-kolla | 12:07 | |
| *** williamcaban has quit IRC | 12:08 | |
| *** neophy has joined #openstack-kolla | 12:27 | |
| *** msimonin has quit IRC | 12:31 | |
| *** zhubingbing has quit IRC | 12:32 | |
| *** zhurong has joined #openstack-kolla | 12:46 | |
| *** huzhengchuan has quit IRC | 12:51 | |
| *** saneax is now known as saneax-_-|AFK | 12:52 | |
| *** david-lyle has joined #openstack-kolla | 12:55 | |
| *** v1k0d3n has joined #openstack-kolla | 12:59 | |
| *** david-lyle has quit IRC | 12:59 | |
| *** sacharya has joined #openstack-kolla | 13:08 | |
| *** williamcaban has joined #openstack-kolla | 13:09 | |
| *** williamcaban has quit IRC | 13:09 | |
| *** mdnadeem has quit IRC | 13:12 | |
| *** sacharya has quit IRC | 13:13 | |
| *** msimonin has joined #openstack-kolla | 13:13 | |
| *** eaguilar has joined #openstack-kolla | 13:20 | |
| *** schwicht has quit IRC | 13:21 | |
| *** kristian__ has quit IRC | 13:21 | |
| *** huzhengchuan has joined #openstack-kolla | 13:23 | |
| *** markus-k has quit IRC | 13:28 | |
| * portdirect those days when the docker-engine flakes out on you every 10 mins :( | 13:29 | |
| *** msimonin has quit IRC | 13:29 | |
| *** Bico_Fino has joined #openstack-kolla | 13:42 | |
| *** markus-k has joined #openstack-kolla | 13:45 | |
| *** huzhengchuan has quit IRC | 13:45 | |
| *** kristian__ has joined #openstack-kolla | 13:46 | |
| *** Bico_Fino has quit IRC | 13:52 | |
| *** jtriley has quit IRC | 13:55 | |
| *** david-lyle has joined #openstack-kolla | 13:56 | |
| *** zhubingbing has joined #openstack-kolla | 13:56 | |
| *** david-lyle has quit IRC | 14:00 | |
| *** Bico_Fino has joined #openstack-kolla | 14:02 | |
| *** fguillot has joined #openstack-kolla | 14:06 | |
| *** williamcaban has joined #openstack-kolla | 14:10 | |
| *** williamcaban has quit IRC | 14:15 | |
| *** kristian__ has quit IRC | 14:22 | |
| *** lamt has joined #openstack-kolla | 14:27 | |
| openstackgerrit | Merged openstack/kolla-ansible: Move heat precheck into its own role https://review.openstack.org/415359 | 14:29 |
| *** lamt has quit IRC | 14:30 | |
| *** palexster has quit IRC | 14:30 | |
| *** palexster has joined #openstack-kolla | 14:31 | |
| *** lamt has joined #openstack-kolla | 14:32 | |
| openstackgerrit | Surya Prakash Singh proposed openstack/kolla-ansible: Removing unsed directory "etc/oslo-config-generator" from kolla-ansible https://review.openstack.org/415399 | 14:35 |
| *** satyar has joined #openstack-kolla | 14:35 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: Run nova-manage simple_cell_setup during deploying and upgrading https://review.openstack.org/415472 | 14:36 |
| *** palexster has quit IRC | 14:36 | |
| *** srwilkers has joined #openstack-kolla | 14:41 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Move gnocchi precheck into its own role https://review.openstack.org/415326 | 14:41 |
| *** Bico_Fino has quit IRC | 14:44 | |
| openstackgerrit | Steve Wilkerson proposed openstack/kolla-kubernetes: Helm charts for cinder keystone v2 endpoints https://review.openstack.org/412615 | 14:44 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Move searchlight precheck into its own role https://review.openstack.org/415135 | 14:45 |
| *** Bico_Fino has joined #openstack-kolla | 14:52 | |
| satyar | Hi jeffrey4l | 14:54 |
| Jeffrey4l | sup satyar | 14:54 |
| satyar | regarding the review https://review.openstack.org/#/c/415472 | 14:55 |
| Jeffrey4l | yep. | 14:55 |
| satyar | i doubt it will work for someone who is moving from newton to ocata | 14:55 |
| Jeffrey4l | why/ | 14:55 |
| Jeffrey4l | it should works. | 14:55 |
| Jeffrey4l | which part are u doubting? | 14:56 |
| satyar | putting the db creation inside bootstrap | 14:56 |
| Jeffrey4l | are u meaning the nova_api_cell0 database? in nova upgrade.yml, it is including bootstrap.yml file now. ( may need add more comments ) but it works. | 14:58 |
| satyar | yes | 14:59 |
| Jeffrey4l | then is my explanation reasonable? | 14:59 |
| satyar | i was thinking to put that inside the simple_cell_setup | 14:59 |
| satyar | and call that inside upgrade.yml | 15:00 |
| satyar | what you think? | 15:00 |
| satyar | may be i am missing something | 15:01 |
| Jeffrey4l | then the nova_api_cell0 will not be created during first deploying, unless you put the task into bootstrap.yml and simple_cell_setup. | 15:01 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Move ironic precheck into its own role https://review.openstack.org/415481 | 15:01 |
| Jeffrey4l | you missed something. | 15:01 |
| satyar | yes | 15:02 |
| satyar | we need that only for upgrade | 15:02 |
| satyar | ok got it :) | 15:03 |
| Jeffrey4l | no. we need that both first deploy and ugprade. | 15:03 |
| satyar | yes | 15:03 |
| satyar | i got my mistake | 15:03 |
| Jeffrey4l | ;) | 15:03 |
| satyar | do you have the kolla code also? | 15:03 |
| *** TxGirlGeek has joined #openstack-kolla | 15:03 | |
| Jeffrey4l | i do not think so. I was thinking it should. but now. it shouldn't. when the patch passed the gate. i will mark the bug as invalid in kolla project. it should be only a kolla-ansible bug. | 15:04 |
| satyar | kolla need the cell_v2 simple_cell_setup run | 15:08 |
| satyar | from extended start | 15:08 |
| satyar | that needs a change in extended_start.sh i guess | 15:08 |
| Jeffrey4l | satyar, it will raise error if run simple_cell_setup in extend_start.sh | 15:08 |
| satyar | ok thats why i was etting the error :( | 15:09 |
| Jeffrey4l | simple_cell_setup need the nova-compute is up. ( which is weird, but it does ) | 15:09 |
| *** sacharya has joined #openstack-kolla | 15:09 | |
| Jeffrey4l | in my patch, the simple_cell_setup will be called in upgrade.yml file. | 15:09 |
| satyar | i had the same code but the new db creation part is in a different file | 15:09 |
| satyar | ok | 15:09 |
| Jeffrey4l | it will fix your upgrade issue. | 15:09 |
| satyar | i was calling it inside the the extended start | 15:10 |
| satyar | may be thats why i was getting the issue | 15:10 |
| satyar | will be looking for your patch :) | 15:10 |
| satyar | Thanks | 15:10 |
| Jeffrey4l | and for first deployment, the simple_cell_setup will be called after nova-compute service is running. | 15:10 |
| satyar | yes | 15:10 |
| Jeffrey4l | put it into extend_start will fix the upgrade issue. but it will raise error for first deployment. | 15:11 |
| satyar | after the patch pass the gate | 15:11 |
| *** zhurong has quit IRC | 15:11 | |
| satyar | i am going to cherrypick it | 15:11 |
| Jeffrey4l | cool. | 15:11 |
| satyar | and deploy the 05/12/2016 code | 15:11 |
| Jeffrey4l | i tried first deploy locally, which works. not test the upgrade. | 15:11 |
| satyar | and will update the cherry pick patch | 15:11 |
| Jeffrey4l | thanks for testing that ;) | 15:11 |
| satyar | then i will try to upgrade | 15:11 |
| satyar | :) | 15:12 |
| *** sacharya has quit IRC | 15:14 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Move senlin precheck into its own role https://review.openstack.org/415486 | 15:16 |
| *** goldyfruit has joined #openstack-kolla | 15:20 | |
| *** Bico_Fino has quit IRC | 15:21 | |
| *** schwicht has joined #openstack-kolla | 15:22 | |
| *** kristian__ has joined #openstack-kolla | 15:22 | |
| *** zhangshuai has quit IRC | 15:23 | |
| *** zhangshuai has joined #openstack-kolla | 15:24 | |
| *** Bico_Fino has joined #openstack-kolla | 15:27 | |
| *** kristian__ has quit IRC | 15:27 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Move cinder precheck into its own role https://review.openstack.org/415492 | 15:32 |
| *** goldyfruit has quit IRC | 15:32 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Update the horizon port into all.yml https://review.openstack.org/415495 | 15:40 |
| *** Bico_Fino has quit IRC | 15:42 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Update the horizon port into all.yml https://review.openstack.org/415495 | 15:45 |
| *** Pavo has joined #openstack-kolla | 15:47 | |
| openstackgerrit | Matt McEuen proposed openstack/kolla: Quickstart guide: disable libvirt apparmor https://review.openstack.org/415501 | 15:48 |
| *** schwicht has quit IRC | 15:51 | |
| *** kristian__ has joined #openstack-kolla | 15:52 | |
| *** david-lyle has joined #openstack-kolla | 15:57 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: optimize bootstrap.yml file https://review.openstack.org/415504 | 15:59 |
| *** schwicht has joined #openstack-kolla | 15:59 | |
| *** david-lyle has quit IRC | 16:01 | |
| sbezverk_ | srwilkers: ping | 16:04 |
| srwilkers | sbezverk_, pong | 16:04 |
| *** ocaner has quit IRC | 16:04 | |
| sbezverk_ | could you try to add this: {{ printf "%s%s" .serviceType $postfix | quote }} | 16:04 |
| srwilkers | yep, just saw your comment | 16:04 |
| *** TxGirlGeek has quit IRC | 16:05 | |
| sbezverk_ | srwilkers: thanks, I think that should fix the issue or at least part ;-) | 16:05 |
| *** TxGirlGeek has joined #openstack-kolla | 16:06 | |
| openstackgerrit | Steve Wilkerson proposed openstack/kolla-kubernetes: Helm charts for cinder keystone v2 endpoints https://review.openstack.org/412615 | 16:07 |
| *** goldyfruit has joined #openstack-kolla | 16:10 | |
| *** williamcaban has joined #openstack-kolla | 16:12 | |
| *** TxGirlGeek has quit IRC | 16:12 | |
| *** TxGirlGeek has joined #openstack-kolla | 16:16 | |
| *** williamcaban has quit IRC | 16:16 | |
| *** goldyfruit has quit IRC | 16:18 | |
| *** sayantani has joined #openstack-kolla | 16:19 | |
| *** TxGirlGeek has quit IRC | 16:20 | |
| *** TxGirlGeek has joined #openstack-kolla | 16:20 | |
| Pavo | morning everyone | 16:24 |
| *** eaguilar has quit IRC | 16:26 | |
| sbezverk_ | Jeffrey4l: ping | 16:27 |
| Jeffrey4l | sbezverk_, pong | 16:28 |
| sbezverk_ | Jeffrey4l: it seems cinder version for centos binary and ubuntu binary for 3.0.1 tag is different? Are yo uaware of it? | 16:29 |
| Jeffrey4l | sbezverk_, no. i never tried binary type. what's the different? | 16:29 |
| Pavo | is that because of upstream package sbezverk_? | 16:30 |
| Jeffrey4l | Pavo, i guess so. | 16:30 |
| sbezverk_ | Jeffrey4l: there was a cinder bug I reported which seems fixed in centos' cinder version but still not fixed in ubunut version | 16:30 |
| Jeffrey4l | sbezverk_, could u print the cinder version name in centos and ubuntu? | 16:31 |
| sbezverk_ | Jeffrey4l: it is a bit difficult since I run it in kolla-kubernetes and at the gate.. | 16:32 |
| Jeffrey4l | but i guess the issue is caused by what Pavo mentioned. kolla can do nothing for this . | 16:33 |
| Jeffrey4l | sbezverk_, ^ | 16:33 |
| *** kristian__ has quit IRC | 16:33 | |
| sbezverk_ | Jeffrey4l: agree, I was just checking if you knew.. | 16:33 |
| *** TxGirlGeek has quit IRC | 16:34 | |
| *** kelv has joined #openstack-kolla | 16:34 | |
| Pavo | binary is controlled by distro upstream and source is controlled by openstack git repo correct? | 16:34 |
| *** TxGirlGeek has joined #openstack-kolla | 16:34 | |
| Jeffrey4l | source is controlled by kolla actually. but it is easy for following the openstack( change the kolla-build.conf file manually. ) Pavo | 16:36 |
| kelv | We use Ubuntu primarily and likely have changes/fixes to propose. Are contributors expected to submit patches for ubuntu _and_ centos in general? | 16:36 |
| Pavo | what do you mean controlled by kolla? I thought source downloaded the openstack git repo source | 16:36 |
| Jeffrey4l | Pavo, in default, the soruce is grabbed from tarballs.openstack.org site. | 16:37 |
| Pavo | guess I mis-read the kolla docs then | 16:37 |
| Pavo | ah ok | 16:37 |
| Pavo | now I understand | 16:37 |
| Pavo | so the version of services are whatever kolla has set | 16:37 |
| Pavo | if using source | 16:37 |
| Jeffrey4l | Pavo, yep. https://github.com/openstack/kolla/blob/master/kolla/common/config.py#L266 | 16:37 |
| Jeffrey4l | but it is easy to change, based on what u want. | 16:38 |
| Pavo | yeah I get that | 16:38 |
| Jeffrey4l | kelv, yep. any contribution is welcome. openstack project is very open. | 16:38 |
| *** williamcaban has joined #openstack-kolla | 16:38 | |
| kelv | Right, just wondering if contributors are (implicitly?) expected to make their patches work for CentOS to satisfy the gate tests even though they may not use it | 16:40 |
| *** schwicht has quit IRC | 16:41 | |
| kelv | My environment is quite locked down, so making Ubuntu build work consumed some time. If the above is expected, we will need to invest some time in making CentOS build work in our environment before any patch submissions | 16:41 |
| Pavo | I wouldn't say expected | 16:41 |
| Pavo | I would say appreciated | 16:41 |
| Pavo | like my environment, we only can use centos or "REHL" | 16:42 |
| Pavo | I do nothing in ubuntu | 16:42 |
| *** lbeliveau has quit IRC | 16:42 | |
| kelv | Do you happen to have horizon built in yours? | 16:42 |
| *** huzhengchuan has joined #openstack-kolla | 16:43 | |
| Pavo | yes | 16:43 |
| Pavo | currently in 3.0.1 though none of the other plugins for horizon work though | 16:43 |
| kelv | Mine returns 404 on Ubuntu. What is the full "WSGIScriptAlias / ..." in your Apache configs? | 16:43 |
| Pavo | let me take a look kelv | 16:43 |
| Pavo | kelv were would that be located? | 16:45 |
| kelv | On Ubuntu, /etc/apache2/conf-enabled/000-default.conf: WSGIScriptAlias / /usr/lib/python2.7/site-packages/openstack_dashboard/wsgi/django.wsgi | 16:46 |
| Pavo | so I would assume in centos its using httpd | 16:46 |
| Pavo | looking for Alias? | 16:48 |
| kelv | WSGIScriptAlias | 16:50 |
| *** mdnadeem has joined #openstack-kolla | 16:50 | |
| Pavo | Not seeing WSGIScriptAlias but do see a ScriptAlias | 16:50 |
| Pavo | here is my whole httpd.conf | 16:51 |
| Pavo | https://www.dropbox.com/s/77ewotv84zu41m2/httpd.conf?dl=0 | 16:51 |
| Pavo | ScriptAlias /cgi-bin/ "/var/www/cgi-bin/" | 16:52 |
| kelv | do a recursive grep: grep -R -i wsgiscriptalias /path/to/httpd/etc/dir | 16:52 |
| *** kristian__ has joined #openstack-kolla | 16:52 | |
| Pavo | ah ok | 16:53 |
| Pavo | which one you want | 16:53 |
| Pavo | I have /etc/httpd/conf.d/openstack-dashboard.conf:WSGIScriptAlias /dashboard /usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi and /etc/httpd/conf.d/horizon.conf: WSGIScriptAlias / /usr/lib/python2.7/site-packages/openstack_dashboard/wsgi/django.wsgi | 16:53 |
| kelv | ls -l /usr/lib/python2.7/site-packages/openstack_dashboard/wsgi/django.wsgi <- does it exist? | 16:54 |
| Pavo | yes | 16:55 |
| *** zhubingbing has quit IRC | 16:55 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Unify haproxy prechecks condition order https://review.openstack.org/415510 | 16:55 |
| Pavo | is that what you are wanting? | 16:55 |
| kelv | That's interesting. I guess I will attempt to build CentOS and figure why it exists and not in Ubuntu | 16:55 |
| kelv | Yes, thanks for looking | 16:56 |
| Pavo | here you go a copy of it | 16:56 |
| Pavo | https://www.dropbox.com/s/fmgi5866v3d7rvw/django.wsgi?dl=0 | 16:56 |
| Pavo | maybe if you add itmanually it will work for ya | 16:56 |
| *** schwicht has joined #openstack-kolla | 17:02 | |
| openstackgerrit | Steve Wilkerson proposed openstack/kolla-kubernetes: Helm charts for cinder keystone v2 endpoints https://review.openstack.org/412615 | 17:02 |
| satyar | Hi jeffrey4l | 17:04 |
| satyar | for simple_cell_setup.yml dont we need to pass the transport url? | 17:04 |
| Jeffrey4l | satyar, no. cell0's tranport url is none:/// | 17:05 |
| Jeffrey4l | it do not require any transport url. | 17:05 |
| satyar | while i run the command manually it ask for transport url | 17:05 |
| Jeffrey4l | gate is red because openstack client is not installed. | 17:05 |
| satyar | hmmm ok | 17:05 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Move magnum precheck into its own role https://review.openstack.org/415513 | 17:05 |
| Jeffrey4l | need move that task into kolla_toolbox. | 17:07 |
| Jeffrey4l | satyar, could u post the logs? | 17:07 |
| Jeffrey4l | i never saw that. | 17:07 |
| *** schwicht has quit IRC | 17:07 | |
| *** krtaylor has quit IRC | 17:08 | |
| *** jtriley has joined #openstack-kolla | 17:08 | |
| satyar | sure | 17:09 |
| *** sacharya has joined #openstack-kolla | 17:10 | |
| *** unicell has quit IRC | 17:10 | |
| satyar | http://paste.openstack.org/show/593552/ | 17:11 |
| *** eaguilar has joined #openstack-kolla | 17:15 | |
| *** sacharya has quit IRC | 17:15 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Move mistral precheck into its own role https://review.openstack.org/415515 | 17:17 |
| satyar | jeffrey4l "http://paste.openstack.org/show/593552/" | 17:21 |
| *** krtaylor has joined #openstack-kolla | 17:21 | |
| Jeffrey4l | what nova version are u using ? satyar | 17:22 |
| Jeffrey4l | satyar, you may need configure nova.conf https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1213 | 17:22 |
| satyar | i guess thats the latest code | 17:24 |
| satyar | we need to support the upgrade from newton/mitaka to ocata | 17:24 |
| *** msimonin has joined #openstack-kolla | 17:25 | |
| satyar | checking the changes if its working :) | 17:25 |
| Jeffrey4l | right. i got what u mean. | 17:25 |
| satyar | BTW satyar is satya_ :) | 17:25 |
| satyar | registered the nick name ;) | 17:26 |
| Jeffrey4l | ahh | 17:26 |
| Jeffrey4l | i will fix it. | 17:26 |
| satyar | Thanks :) | 17:26 |
| Jeffrey4l | np. thanks for pointing it out ;) | 17:26 |
| satyar | :) | 17:28 |
| srwilkers | sbezverk_, ping | 17:33 |
| sbezverk_ | srwilkers: pong | 17:38 |
| *** mdnadeem has quit IRC | 17:38 | |
| *** haplo37_ has quit IRC | 17:48 | |
| *** kristian__ has quit IRC | 17:49 | |
| *** schwicht has joined #openstack-kolla | 17:56 | |
| *** fguillot has quit IRC | 17:57 | |
| *** haplo37_ has joined #openstack-kolla | 17:58 | |
| SamYaple | well after extensive testing i can say the autorecovery of galera is pretty sweet | 17:58 |
| *** fguillot has joined #openstack-kolla | 17:59 | |
| *** srwilkers has quit IRC | 18:00 | |
| *** srwilkers has joined #openstack-kolla | 18:01 | |
| *** prameswar has quit IRC | 18:01 | |
| kfox1111 | morning. | 18:07 |
| SamYaple | kfox1111: morning | 18:08 |
| SamYaple | btw. so good news for galera and kubernetes | 18:08 |
| kfox1111 | yeah? | 18:08 |
| SamYaple | should be much easier to do now | 18:08 |
| kfox1111 | cool. | 18:08 |
| kfox1111 | the containers? | 18:08 |
| SamYaple | the orchsertation | 18:08 |
| SamYaple | so ryan brought up some autorecovery stuff from galera the other day | 18:08 |
| *** schwicht has quit IRC | 18:08 | |
| SamYaple | and ive been hammering it pretty hard, and its holding up | 18:09 |
| *** schwicht has joined #openstack-kolla | 18:09 | |
| SamYaple | even with hard, power-failed, crashed on all nodes at the same time, it can auto recover | 18:09 |
| SamYaple | it will check all the ids and start the newest node, in the case of no node having the full picture, it will do galeras normal write merging | 18:09 |
| SamYaple | the only thing that is required is all the nodes in the gcomm list to be started within 30 seconds of each other (configurable) | 18:10 |
| SamYaple | so, you only need to bootstrap a container as a one time action, and you pretty much dont have to worry about recovery at all in kubernetes | 18:10 |
| kfox1111 | sweet. | 18:11 |
| kfox1111 | what orcestrates the recovery? | 18:11 |
| kfox1111 | mariadb or something in the container outside of mariadb? | 18:12 |
| SamYaple | galera! | 18:12 |
| kfox1111 | they got galera to the point where it can autorecover by itself? | 18:12 |
| srwilkers | sbezverk_, cinder v2 endpoints are green | 18:12 |
| kfox1111 | without the premote one to primary first dance? | 18:12 |
| SamYaple | kfox1111: in most situtations ive put it thorugh, yes | 18:13 |
| srwilkers | oh, you already saw. :) | 18:13 |
| SamYaple | kfox1111: and this code as been functional for almost a year | 18:13 |
| kfox1111 | what about the one situation it doesnt? still a recovery path. | 18:13 |
| SamYaple | kfox1111: kind of. but not neccessarily different code than youll alreayd have | 18:13 |
| SamYaple | so if you have a hard stop of the cluster and while the cluster is down, you lose a node, then it can't autorecover | 18:14 |
| SamYaple | _unless_ you remove the lost node from the gcomm list | 18:14 |
| kfox1111 | the kolla-kube code doesn't have a galera mode yet. | 18:14 |
| kfox1111 | (unless someone added it in the last couple days) | 18:14 |
| SamYaple | kfox1111: kolla-kube should be able to run single node galera no different than single-node anything else | 18:14 |
| *** jtriley has quit IRC | 18:15 | |
| kfox1111 | true. | 18:15 |
| SamYaple | you know kolla-ansible has no difference between AIO and multinode database | 18:15 |
| SamYaple | in fact you can grow from AIO to multinode | 18:15 |
| SamYaple | you can also shrink from multinode to AIO | 18:15 |
| *** sacharya has joined #openstack-kolla | 18:15 | |
| kfox1111 | need a 100% reliable recovery though, even if it has to be manual. | 18:15 |
| SamYaple | right. thats what i just said | 18:15 |
| SamYaple | all you have to do is remove the missing node form the config and then its fine | 18:16 |
| kfox1111 | kolla-kubernetes doesn't have plumbing at the moment though to do so. | 18:16 |
| SamYaple | yea but youll have to have the ability to remove nodes in the future | 18:16 |
| kfox1111 | not really an easy way to tweak the config of just one instance. | 18:16 |
| SamYaple | kfox1111: not one, all | 18:16 |
| SamYaple | youre thinking too singular about galera | 18:16 |
| sbezverk_ | kfox1111: morning | 18:17 |
| kfox1111 | you mean, scale it back to 1 pod, and change the config to be one master? | 18:17 |
| kfox1111 | sbezverk_: morning. :) | 18:17 |
| SamYaple | kfox1111: nope. thats still to singular | 18:17 |
| sbezverk_ | kfox1111: I finished iscsi gate need your input | 18:17 |
| kfox1111 | sbezverk_: k. I'll have a look. got a link handy? | 18:17 |
| SamYaple | kfox1111: if you have 3 nodes, they go hard down, you lose one node, then the lost node needs to be removed from all configs | 18:18 |
| sbezverk_ | kfox1111: https://review.openstack.org/#/c/414348/ | 18:18 |
| kfox1111 | SamYaple: hmm. k. still not entirely sure how that will map into kolla-kubernetes though. | 18:19 |
| sbezverk_ | srwilkers: I saw that, you got my +2 :-) good job.. | 18:19 |
| kfox1111 | depends on if we use stateful sets with pv's or what | 18:19 |
| SamYaple | kfox1111: i dont know how you plan on growing and repairing the lost services. but im sure it will map somehow | 18:20 |
| SamYaple | kfox1111: its very flexible from what ive seen | 18:20 |
| kfox1111 | SamYaple: yeah, I havent played with 10+ too much. the 5x series was a very manual recovery. | 18:21 |
| sbezverk_ | kfox1111: Ubuntu will fail until it gets the same version of cinder packaged as centos | 18:21 |
| *** neophy has quit IRC | 18:21 | |
| SamYaple | kfox1111: thats a different ballgame, yea | 18:21 |
| SamYaple | kfox1111: though this has nothing to do with 5x vs 10x | 18:21 |
| kfox1111 | sbezverk_: k. | 18:21 |
| SamYaple | this is strictly galera version | 18:22 |
| *** inc0 has joined #openstack-kolla | 18:22 | |
| kfox1111 | sbezverk_: been playing with config some more on the plane. trying to port various things over to the new config way. | 18:22 |
| inc0 | good morning | 18:22 |
| sbezverk_ | kfox1111: cool, let's see it :-) | 18:22 |
| kfox1111 | sbezverk_: my aproach was to try and standardize some vars, and then make it more simple for generic value functions to work. | 18:22 |
| srwilkers | hello inc0 | 18:22 |
| SamYaple | inc0: morning. you just missed me gushing about galera auto-recovery | 18:23 |
| kfox1111 | sbezverk_: but after doing a whole bunch, I'm finding enough exceptions to the rules that it will be hard to do it that way. | 18:23 |
| inc0 | damn SamYaple | 18:23 |
| inc0 | I missed your gushings, there is nothing like good gushing in the morning | 18:23 |
| sbezverk_ | kfox1111: so what do you propose then? | 18:23 |
| kfox1111 | sbezverk_: thinking now that we should have a seperate set of values from the main template that specifies where to look for config values. | 18:23 |
| inc0 | wassup with it? | 18:23 |
| SamYaple | inc0: automagically beautifulness! | 18:24 |
| kfox1111 | rather then try and reuse serviceName, serviceType, etc. | 18:24 |
| inc0 | I guess yay is in order? | 18:24 |
| SamYaple | inc0: it can self-recover itself from a hard-down, crashed outage | 18:24 |
| inc0 | salt? | 18:24 |
| kfox1111 | I've got a rough prototype of that working on the plane. so will plumb that up and see if that will fly. | 18:24 |
| SamYaple | inc0: galera | 18:24 |
| kfox1111 | if it works, it will also require a lot less code refactoring of the existing stuff. | 18:24 |
| inc0 | with salt or ansible or totally automatic? | 18:24 |
| SamYaple | inc0: outside of salt, just tested it thoruhgly in my salt implemntation though | 18:24 |
| SamYaple | inc0: outside of orchestration | 18:24 |
| inc0 | interesting | 18:24 |
| sbezverk_ | kfox1111: I like the idea of less refactoring ;-) | 18:25 |
| SamYaple | galera itself. | 18:25 |
| inc0 | didn't know galera can do it | 18:25 |
| *** Serlex has quit IRC | 18:25 | |
| inc0 | will make our job in k8s whole lot easier | 18:25 |
| SamYaple | inc0: ryan brought it up a few weeks back, and we started digging into it | 18:25 |
| kfox1111 | SamYaple: so, is there a document that explains the self reassembly? | 18:25 |
| SamYaple | kfox1111: https://github.com/MariaDB/server | 18:25 |
| kfox1111 | sbezverk_: it also uses the idea you had about using a range. | 18:26 |
| kfox1111 | managed to get it to work. | 18:26 |
| kfox1111 | the new 'set' function allows sooo much more to be done in helm. | 18:26 |
| SamYaple | the details are suprisingly not talked about much, but you can watch what the recovery scripts are doing | 18:26 |
| kfox1111 | SamYaple: cool. thanks. | 18:27 |
| kfox1111 | sbezverk_: with a set, you can use a hash to return values from macros. :) | 18:27 |
| kfox1111 | opens up a world of new possibilites. :) | 18:27 |
| inc0 | anyone tried vagrant + libvirt? | 18:27 |
| sbezverk_ | kfox1111: cannot wait to see what you have cooked up now :-) | 18:28 |
| kfox1111 | inc0: last I tried it was a wold of pain. but I've heard its better now. | 18:29 |
| kfox1111 | sbezverk_: I finally got internet access. I'll try and post something tonight. | 18:29 |
| kfox1111 | don't know how long I have though towork on stuff. we'll see. | 18:30 |
| portdirect | inc0: using vagrant and libvirt for the k8s at the moment | 18:30 |
| sbezverk_ | kfox1111: cool, please review that iscsi gate thing too.. that was my toy to play with during chrismas break.. | 18:30 |
| *** jtriley has joined #openstack-kolla | 18:31 | |
| *** zhangshuai has quit IRC | 18:31 | |
| *** zhangshuai has joined #openstack-kolla | 18:31 | |
| portdirect | inc0: be good to get your feedback on this if you try it: http://docs.openstack.org/developer/kolla-kubernetes/development-environment.html#ubuntu-16-04-with-libvirt | 18:31 |
| inc0 | well, looks good | 18:32 |
| inc0 | just my own libvirt is acting up | 18:32 |
| inc0 | networking is no bueno | 18:33 |
| *** sayantani has quit IRC | 18:33 | |
| *** sayantani has joined #openstack-kolla | 18:34 | |
| kfox1111 | sbezverk_: yeah. looking thorugh it now. so far, looking good. :) | 18:35 |
| *** kristian__ has joined #openstack-kolla | 18:39 | |
| *** jtriley has quit IRC | 18:42 | |
| *** kristian__ has quit IRC | 18:43 | |
| *** jtriley has joined #openstack-kolla | 18:56 | |
| portdirect | been having a load of issues with docker today :( the daemon seems really unhappy under load in my VM's: | 18:57 |
| portdirect | https://www.irccloud.com/pastebin/R7DFrIPP/ | 18:57 |
| portdirect | any ideas? | 18:58 |
| *** david-lyle has joined #openstack-kolla | 18:59 | |
| sbezverk_ | portdirect: I saw something similar in the past when I was running kolla's build. but after I switched to btrfs from loopback, I have not seen it again.. | 18:59 |
| inc0 | portdirect, docker images | wc -l | 19:00 |
| portdirect | 4 images | 19:01 |
| *** david-lyle has quit IRC | 19:03 | |
| *** klindgren__ has quit IRC | 19:08 | |
| *** klindgren has joined #openstack-kolla | 19:09 | |
| *** jtriley has quit IRC | 19:11 | |
| *** MasterOfBugs has joined #openstack-kolla | 19:13 | |
| *** pc_m has quit IRC | 19:13 | |
| inc0 | so that's not it | 19:13 |
| inc0 | I can have thousands after multiple rebuilds | 19:14 |
| *** pc_m has joined #openstack-kolla | 19:14 | |
| SamYaple | portdirect: thats caused by load. sbezverk_ btrfs is likely causing far less load than the loopback did | 19:16 |
| portdirect | SamYaple: yeah thats what I feared :( there must be a bug/regression somewhere as its pretty crazy -getting it with overlay and btrfs as well -think the kubelet is just pushing the daemon too hard as this just trying to launch the k8s control plane containers after restarting the daemon, the VM's I'm running are hardly anemic :/ | 19:21 |
| *** eaguilar has quit IRC | 19:22 | |
| *** pramodrj07 has joined #openstack-kolla | 19:24 | |
| *** MasterOfBugs has quit IRC | 19:24 | |
| *** pramodrj07 has quit IRC | 19:24 | |
| *** MasterOfBugs has joined #openstack-kolla | 19:24 | |
| *** pramodrj07 has joined #openstack-kolla | 19:24 | |
| inc0 | ok I give up with vagrant | 19:29 |
| inc0 | I'll spawn my own vms and use ansible to deploy kube | 19:29 |
| portdirect | I give up with kube for the day :( https://www.irccloud.com/pastebin/Y5lxyyAA/ | 19:30 |
| sbezverk_ | portdirect: SamYaple: after I switched to use btrfs most of hicups I hit in the past were gone, so far I am really happy with btrfs.. | 19:30 |
| *** kristian__ has joined #openstack-kolla | 19:31 | |
| kfox1111 | portdirect: I agree with sbezverk_. had docker eat itself 3 times. then moved to btrfs and no issues so far. | 19:31 |
| sbezverk_ | portdirect: check the controller manager log, see what it is not happy about | 19:31 |
| inc0 | loopback dm is a pita | 19:31 |
| inc0 | on so many levels | 19:31 |
| kfox1111 | inc0: yeah. it shoudl work, but just isn't mature enough yet. :/ | 19:32 |
| portdirect | sbezverk_: docker deamon issues, it's the healthckecks causing the pods to commit suicide :( | 19:33 |
| *** pcaruana has joined #openstack-kolla | 19:33 | |
| sbezverk_ | portdirect: ahhh, got it.. have you tried to restart docker daemon? | 19:34 |
| kfox1111 | portdirect: one of the big issues with using ha features like liveness hooks. | 19:34 |
| kfox1111 | when things go slow, the problem can compound rather then "self heal" :/ | 19:34 |
| portdirect | kfox1111, yeah its pretty ugly - I've seen a 50 node cluster blow itself up because of ddosing the k8s api before :D | 19:35 |
| kfox1111 | portdirect: yeah. I'm a bit worried kubernetes-entrypoint will do the same. | 19:35 |
| kfox1111 | had ceilometer blowing up neutron/nova before too. its touchy. :/ | 19:35 |
| portdirect | kfox1111: now its in an init container it'll be fine | 19:36 |
| *** kristian__ has quit IRC | 19:36 | |
| portdirect | (I hope...) | 19:36 |
| kfox1111 | no garantee of that. :/ | 19:36 |
| portdirect | sbezverk_: restarting just compounds the issue - I'm gonna try again with centos's docker and see if there is any improvement - I've got reports that since chistmas this has hit a few people - on ubuntu as well - so there's probably somthing deeper going on :( | 19:38 |
| portdirect | sbezverk_: was running pretty much this exact setup 4 days ago with no issues | 19:38 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adds cinder volume lvm microservice https://review.openstack.org/414348 | 19:40 |
| *** eaguilar has joined #openstack-kolla | 19:42 | |
| sbezverk_ | portdirect: sorry man, I guess the stars are not at the right places for your docker today ;-) | 19:44 |
| srwilkers | go home docker, youre drunk | 19:45 |
| sbezverk_ | srwilkers: wonder if it is somehow related to this docker's master ;-) | 19:46 |
| srwilkers | dockering under the influence | 19:47 |
| sbezverk_ | srwilkers: lol | 19:49 |
| satyar | Hi jeffrey4l | 19:51 |
| *** kristian__ has joined #openstack-kolla | 19:58 | |
| *** pcaruana has quit IRC | 20:01 | |
| *** kristian__ has quit IRC | 20:03 | |
| *** kristian__ has joined #openstack-kolla | 20:04 | |
| SamYaple | can I get an ack on this please https://review.openstack.org/#/c/413143 ? | 20:04 |
| SamYaple | also, _please_ someone merge this to fix the ubuntu binary gate https://review.openstack.org/#/c/412452/ | 20:04 |
| SamYaple | inc0: ^ | 20:05 |
| jrich523 | yeah inc0, get to work! holidays are over! | 20:05 |
| SamYaple | jrich523: dont think i didnt notice you not doing reviews over there | 20:06 |
| jrich523 | lol i dont have anywhere near enough knowledge to even begin to attempt revewing | 20:06 |
| SamYaple | jrich523: neither do we. we just merge things that sounds pretty | 20:07 |
| jrich523 | lol | 20:07 |
| Pavo | lol | 20:07 |
| jrich523 | if only you had some test suites to validate a lot of this stuff for you | 20:07 |
| jrich523 | sadly, thats a TON of work | 20:07 |
| SamYaple | i dont know if youre joking | 20:09 |
| jrich523 | i am not | 20:09 |
| SamYaple | we do | 20:09 |
| jrich523 | then auto merge :) | 20:09 |
| SamYaple | it exists for every commit | 20:09 |
| jrich523 | CI/CD! | 20:09 |
| SamYaple | that assumes just because a patch passes a gate, it should be merged | 20:09 |
| SamYaple | we have to agree as a community on what merges | 20:09 |
| jrich523 | i knew there were a good chunk of tests, from what i gathered it wasnt really extensive enough, but, im not of user, and try not to worry about such things :) | 20:10 |
| jrich523 | maybe after i figure out how to use it, then i can worry about helping with changes.. it will be a while tho lol | 20:10 |
| jrich523 | i guess thats a fair statement, so shouldnt it be a vote based acceptance? | 20:10 |
| SamYaple | jrich523: it is | 20:11 |
| SamYaple | but the final "vote" must come from a core member | 20:11 |
| SamYaple | i only tagged inc0 because as PTL i know he cares when a patch that fixes the gate tests is available | 20:11 |
| jrich523 | gotcha, so user created, commity nominated, board member approved | 20:11 |
| jrich523 | or, sorta like that.. since im sure the core group is doing many of those parts now | 20:12 |
| SamYaple | something like that jrich523 | 20:12 |
| jrich523 | hmm cool | 20:12 |
| SamYaple | and importantly, we cant approve our own patches | 20:13 |
| jrich523 | i would, one day, like to be able to contribute... but... well... i probably need to learn linux first :) | 20:13 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adds cinder volume lvm microservice https://review.openstack.org/414348 | 20:13 |
| jrich523 | yeah, i assumed that check was in there | 20:13 |
| SamYaple | jrich523: if you arent familiar with CLI stuff in linux, there will be a much higher learning curve, yes. but its a great place to start! | 20:14 |
| jrich523 | oh im well aware | 20:14 |
| jrich523 | im more of a windows guy.. powershell "expert".. that sorta stuff | 20:15 |
| jrich523 | but because of my automation background i get dumped on to all the openstack stuff in the company | 20:15 |
| jrich523 | actually attempting to create a powershell module for OS (mostly for my own use tho :) | 20:15 |
| SamYaple | i thnk that exists jrich523 | 20:15 |
| jrich523 | ugh, it does | 20:16 |
| jrich523 | HORRIBLE | 20:16 |
| jrich523 | also its .net based, i hate having requrired dlls for modules... it causes nightmares | 20:16 |
| jrich523 | i mean, it uses the .net openstack dll | 20:16 |
| jrich523 | obviously my powershell version would be .net based too, just in a different way :) | 20:16 |
| jrich523 | the module i found seems to have stopped development.. it follows no best practices at all... and... i wouldnt be surprised if it didnt work any more | 20:17 |
| sbezverk_ | kfox1111: basic_tests.sh line 137 and 138 could you double check the logic? | 20:17 |
| SamYaple | jrich523: yea i wouldnt recommend using it. just pointing it out | 20:20 |
| jrich523 | i was going to fork it, but because its based off the dll, and its fairly messy, seemed easier to start over... and it should be actually pretty easy if i can get the token stuff to work correctly | 20:21 |
| *** matrohon has joined #openstack-kolla | 20:36 | |
| kfox1111 | sbezverk_: which version should I look at? | 20:40 |
| kfox1111 | (wana make sure we're both looking at the same 137-138) | 20:41 |
| *** satyar has quit IRC | 20:50 | |
| *** sacharya has quit IRC | 20:50 | |
| kfox1111 | hmm... no array append. :/ | 20:57 |
| *** sayantan_ has joined #openstack-kolla | 20:57 | |
| *** david-lyle has joined #openstack-kolla | 21:00 | |
| *** david-lyle has quit IRC | 21:05 | |
| *** srwilkers has quit IRC | 21:05 | |
| sbezverk_ | kfox1111: sorry had to step out.. I was refering the latest PS I pushed | 21:07 |
| kfox1111 | k. I'll have a look. | 21:07 |
| inc0 | jrich523, how about cygwin? | 21:07 |
| kfox1111 | (no worries. I'm in and out fro the next few days too. | 21:08 |
| sbezverk_ | it is the code you use for barriers | 21:08 |
| inc0 | should be enough for CLI stuff | 21:08 |
| jrich523 | inc0: what about it? | 21:08 |
| jrich523 | oh i like objects, and pipes | 21:08 |
| jrich523 | :) | 21:08 |
| jrich523 | i like to be able to control the output (property selection/filtering of output objects) | 21:08 |
| inc0 | what I'm saying you can use openstack clients from comfort of your windows | 21:09 |
| jrich523 | and i do | 21:09 |
| jrich523 | but, i hate it | 21:09 |
| jrich523 | :) | 21:09 |
| inc0 | I see...yeah it's really like learning linux | 21:09 |
| jrich523 | its my only option at the moment.. so... im doing what i can with it.. but powershell would be a million times better for this | 21:09 |
| kfox1111 | sbezverk_: Thats: | 21:09 |
| kfox1111 | wait_for_cinder test detaching | 21:09 |
| kfox1111 | openstack server add volume test2 test | 21:09 |
| kfox1111 | ? | 21:09 |
| jrich523 | yeah, my linux is getting better, but, it still requires a lot of google-fu :) | 21:09 |
| sbezverk_ | kfox1111: right | 21:10 |
| jrich523 | and i really hate having to use grep on text output | 21:10 |
| sbezverk_ | what I noticed is volume stays in attaching state but does not have enugh time to become attached | 21:11 |
| sbezverk_ | when VM check for vdb, it is not there and it fails | 21:11 |
| sbezverk_ | by some reason extending waitfor cinder function does not have any impact but adding sleep does | 21:12 |
| sbezverk_ | I think it should be "wait_for_cinder test attaching" | 21:13 |
| sbezverk_ | instead of available.. | 21:13 |
| *** eaguilar has quit IRC | 21:14 | |
| *** kristian__ has quit IRC | 21:14 | |
| kfox1111 | sbezverk_: sec. let me finish reading through the logic. | 21:14 |
| kfox1111 | so... the loop continues until the volume leaves the specified state. | 21:14 |
| kfox1111 | so initially, it should be in-use as its attached to the first vm. | 21:15 |
| kfox1111 | then enters detaching. | 21:15 |
| kfox1111 | then available. when ready for attaching to another. | 21:15 |
| kfox1111 | so, there could be one more check that may be missing... | 21:16 |
| kfox1111 | line 139 could be "wait_for_cinder test attaching" (or whatever the attaching state is) | 21:17 |
| sbezverk_ | right | 21:17 |
| kfox1111 | probably never saw it get stuck in attaching before. | 21:17 |
| sbezverk_ | that is what I am thinking too | 21:17 |
| kfox1111 | yeah. I think that might fix the issue. | 21:18 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adds cinder volume lvm microservice https://review.openstack.org/414348 | 21:18 |
| kfox1111 | if it really takes multiple minutes to attach, we might want to add a $3 to wait_for_cinder and default it to 30 so we can override the timeout per call if needed. | 21:19 |
| sbezverk_ | kfox1111: I reverted back the times and added that barrier.. | 21:19 |
| kfox1111 | k. cool. | 21:19 |
| SamYaple | jrich523: agreed. powershell is pretty awesome | 21:25 |
| jrich523 | and now that its on linux (and working fairly well) i suspect it will (hopefully) gain popularity | 21:25 |
| jrich523 | also, if i make sure my module doenst use a DLL, it should, technically, jsut work on linux too | 21:25 |
| SamYaple | jrich523: ive not used it on linux because im a bash guy. but ive wanted to get back into it | 21:25 |
| SamYaple | havent messed with it since 1.0 days | 21:26 |
| jrich523 | well, i can confidently say im an expert in that area, so if you've got questions i'd be more than happy to help out, since its something i can actually help with :) | 21:26 |
| SamYaple | jrich523: awesome. i appreciate it. i used to be an expert once upon a time too | 21:26 |
| SamYaple | had an MCITP in server admin 2008 | 21:27 |
| SamYaple | i guess i still do but i havent used it in almost a decade | 21:27 |
| jrich523 | moved on over huh | 21:27 |
| jrich523 | seems like my career might be doing that to me as well... but hopefully i can drag powershell with me :D | 21:27 |
| SamYaple | i did windows because i had to, as soon as i got a job that paid me to linux i switched | 21:27 |
| jrich523 | my experience with linux has generally been painful, and has encouraged me to stick with windows :) | 21:28 |
| SamYaple | new things normally are painful | 21:28 |
| jrich523 | yeah, it jsut seems that (more so back in the day) installing apps was generally very painful based on dependencies that were either not clearly documented or assumed etc | 21:29 |
| inc0 | I use windows even now - wintendo | 21:29 |
| jrich523 | those things have gotten way better, and its much easier to just install and app and get going | 21:29 |
| *** eaguilar has joined #openstack-kolla | 21:30 | |
| *** l4yerffeJ has joined #openstack-kolla | 21:32 | |
| *** schwicht has quit IRC | 21:33 | |
| *** Jeffrey4l has quit IRC | 21:34 | |
| *** l4yerffeJ_ has quit IRC | 21:35 | |
| *** jtriley has joined #openstack-kolla | 21:40 | |
| sbezverk_ | kfox1111: yep it worked.. | 21:42 |
| *** lamt has quit IRC | 21:46 | |
| *** Jeffrey4l has joined #openstack-kolla | 21:47 | |
| *** sacharya has joined #openstack-kolla | 21:51 | |
| *** msimonin has quit IRC | 21:53 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Helm charts for cinder keystone v2 endpoints https://review.openstack.org/412615 | 21:54 |
| *** msimonin has joined #openstack-kolla | 21:54 | |
| *** sacharya has quit IRC | 21:56 | |
| *** msimonin has quit IRC | 21:58 | |
| *** msimonin has joined #openstack-kolla | 21:58 | |
| *** msimonin has quit IRC | 21:58 | |
| *** david-lyle has joined #openstack-kolla | 22:01 | |
| *** schwicht has joined #openstack-kolla | 22:02 | |
| openstackgerrit | Merged openstack/kolla: Removed percona from ubuntu repos https://review.openstack.org/413143 | 22:05 |
| *** david-lyle has quit IRC | 22:06 | |
| openstackgerrit | Merged openstack/kolla: Install correct version of arping https://review.openstack.org/414806 | 22:06 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adds cinder volume lvm microservice https://review.openstack.org/414348 | 22:06 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding iscsi support to the gate. https://review.openstack.org/414713 | 22:06 |
| *** TxGirlGeek has quit IRC | 22:15 | |
| *** srwilkers has joined #openstack-kolla | 22:19 | |
| *** eaguilar has quit IRC | 22:23 | |
| *** sayantani has quit IRC | 22:28 | |
| *** Pavo has quit IRC | 22:29 | |
| *** schwicht has quit IRC | 22:37 | |
| inc0 | kfox1111, you'll go to PTG I hope? | 22:38 |
| *** jtriley has quit IRC | 22:43 | |
| srwilkers | i think he mentioned he was | 22:44 |
| *** fguillot has quit IRC | 22:52 | |
| *** jrich523_ has joined #openstack-kolla | 23:01 | |
| *** jrich523 has quit IRC | 23:04 | |
| *** jrich523_ is now known as jrich523 | 23:04 | |
| *** sayantan_ has quit IRC | 23:08 | |
| *** huzhengchuan has quit IRC | 23:21 | |
| *** sayantan_ has joined #openstack-kolla | 23:25 | |
| *** lamt has joined #openstack-kolla | 23:30 | |
| *** jtriley has joined #openstack-kolla | 23:36 | |
| *** jtriley has quit IRC | 23:40 | |
| *** fguillot has joined #openstack-kolla | 23:41 | |
| *** fguillot has quit IRC | 23:53 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!