Monday, 2017-05-22

*** Pavo has joined #openstack-kolla00:02
spsuryamorning guys00:03
*** Pavo has quit IRC00:06
kbaegisMorning00:10
*** Seyong has joined #openstack-kolla00:11
*** yingjun has joined #openstack-kolla00:13
*** kbaegis has quit IRC00:17
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate  https://review.openstack.org/46600700:18
*** srwilkers has joined #openstack-kolla00:23
masuberugood morning all00:24
masuberuaccording to the kolla documentation - •api_interface - This interface is used for the management network. The management network is the network OpenStack services uses to communicate to each other and the databases. There are known security risks here, so it’s recommended to make this network internal, not accessible from outside. Defaults to network_interface00:24
masuberuhowever the op for horizon is within the api interface, so that means I will have to setup a NAT between the api interface and the normal users network?00:25
*** stefan_bo has joined #openstack-kolla00:38
*** stefan_bo has quit IRC00:43
*** vhosakot has joined #openstack-kolla00:44
*** ssurana has quit IRC00:45
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate  https://review.openstack.org/46600700:46
*** vhosakot has quit IRC00:47
*** duonghq has joined #openstack-kolla00:47
duonghqmorning guys00:49
*** cuongnv has joined #openstack-kolla00:56
*** Pavo has joined #openstack-kolla00:57
*** tovin07_ has joined #openstack-kolla01:04
*** inc0 has quit IRC01:07
*** inc0 has joined #openstack-kolla01:07
*** stefan_bo has joined #openstack-kolla01:17
*** Pavo has quit IRC01:23
*** huzhengchuan has joined #openstack-kolla01:30
*** hrw has quit IRC01:31
spsuryasup duonghq01:32
*** hrw has joined #openstack-kolla01:33
duonghqhi spsurya01:34
*** zul has quit IRC01:44
openstackgerritpangliye proposed openstack/kolla master: Install docker_client and groupadd  https://review.openstack.org/46588002:17
*** salv-orlando has joined #openstack-kolla02:54
*** srwilkers has quit IRC03:06
*** kbaegis has joined #openstack-kolla03:09
spsuryaduonghq: https://review.openstack.org/#/c/465474/03:16
spsuryahanging fruit03:16
spsurya;)03:16
*** caowei has joined #openstack-kolla03:23
*** goldyfruit has joined #openstack-kolla03:24
*** stefan_bo has quit IRC03:33
openstackgerritSurya Prakash (spsurya) proposed openstack/kolla-ansible master: [WIP] Add dragonflow ansible role  https://review.openstack.org/46586203:36
*** klindgren has joined #openstack-kolla03:43
*** klindgren_ has quit IRC03:44
*** goldyfruit has quit IRC03:44
*** kbaegis has quit IRC03:51
*** xinliang has quit IRC03:54
*** xinliang has joined #openstack-kolla03:55
*** xinliang has quit IRC03:55
*** xinliang has joined #openstack-kolla03:55
*** salv-orlando has quit IRC03:58
*** caowei has quit IRC04:06
*** rwsu has joined #openstack-kolla04:18
*** stefan_bo has joined #openstack-kolla04:24
duonghqspsurya, done04:36
*** caowei has joined #openstack-kolla04:37
openstackgerritMerged openstack/kolla-ansible master: Updated the quick start guide  https://review.openstack.org/46547404:37
*** caowei has quit IRC04:41
*** stefan_bo has quit IRC04:41
*** caowei has joined #openstack-kolla04:45
*** chandankumar has quit IRC04:46
*** jaosorior has joined #openstack-kolla04:52
spsuryaduonghq: +104:53
*** janki has joined #openstack-kolla04:59
*** stefan_bo has joined #openstack-kolla05:06
*** caowei has quit IRC05:08
*** skramaja has joined #openstack-kolla05:11
*** salv-orlando has joined #openstack-kolla05:17
*** cuongnv has quit IRC05:25
*** caowei has joined #openstack-kolla05:34
*** cuongnv has joined #openstack-kolla05:38
openstackgerritJuan Antonio Osorio Robles proposed openstack/kolla master: Add images for novajoin services  https://review.openstack.org/46626805:38
*** yingjun has quit IRC05:54
sdakesbezverk i heard from sam that insepctor is working06:01
*** jascott1 has joined #openstack-kolla06:02
*** salv-orlando has quit IRC06:04
*** unicell has joined #openstack-kolla06:05
*** salv-orlando has joined #openstack-kolla06:07
openstackgerritJuan Antonio Osorio Robles proposed openstack/kolla master: Add images for novajoin services  https://review.openstack.org/46626806:13
*** chas has joined #openstack-kolla06:19
*** athomas has joined #openstack-kolla06:29
*** chas has quit IRC06:37
*** chas has joined #openstack-kolla06:37
*** chas has quit IRC06:41
*** chas has joined #openstack-kolla06:41
*** zhubingbing has joined #openstack-kolla06:49
*** chas has quit IRC06:55
*** shardy has joined #openstack-kolla06:55
*** chas has joined #openstack-kolla06:55
*** chas has quit IRC06:57
*** chas has joined #openstack-kolla06:57
*** matrohon has joined #openstack-kolla06:57
*** salv-orlando has quit IRC07:01
*** salv-orlando has joined #openstack-kolla07:03
*** jascott1 has quit IRC07:03
*** chas has quit IRC07:06
*** chas has joined #openstack-kolla07:06
*** salv-orlando has quit IRC07:08
*** chas has quit IRC07:11
*** pcaruana has joined #openstack-kolla07:11
*** pcaruana has quit IRC07:12
openstackgerritcaoyuan proposed openstack/kolla-ansible master: Remove the unnecessary jinja2 templating delimiters  https://review.openstack.org/46661207:13
*** pcaruana has joined #openstack-kolla07:13
*** korzen has joined #openstack-kolla07:15
*** chenn has joined #openstack-kolla07:27
openstackgerritcaoyuan proposed openstack/kolla-ansible master: Use the different nova_cell0 database password from nova  https://review.openstack.org/46661507:35
masuberugood afternoon all, I am running into an issue when trying to use kolla-ansible to deploy openstack https://pastebin.com/raw/wZp6SXgV07:36
masuberubasically it fails during the ceph setup07:36
*** david-lyle has quit IRC07:36
masuberucph-mon container does not start07:36
masuberuapparently a file is missing07:36
openstackgerritcaoyuan proposed openstack/kolla-ansible master: Use the different nova_cell0 database password from nova  https://review.openstack.org/46661507:36
*** Serlex has joined #openstack-kolla07:38
masuberuI also tried destroying and redeploying the container but still didn't fix the issue07:38
hrwmorning07:41
*** david-lyle has joined #openstack-kolla07:42
*** david-lyle has quit IRC07:46
*** egonzalez has joined #openstack-kolla07:46
*** chas has joined #openstack-kolla07:47
*** zhuzeyu has joined #openstack-kolla07:51
*** blallau has joined #openstack-kolla07:54
*** manheim_ has joined #openstack-kolla07:58
*** jmccarthy has joined #openstack-kolla08:02
manheim_hello08:03
*** david-lyle has joined #openstack-kolla08:03
egonzalezsum manheim08:04
*** jmccarthy has quit IRC08:07
*** jmccarthy has joined #openstack-kolla08:08
*** huzhengchuan_ has joined #openstack-kolla08:09
jaosoriormandre: hey dude, can you help me out figuring out what went wrong here https://review.openstack.org/#/c/466268/ ?08:17
egonzalezjaosorior, No package python-keystone-middleware available.08:20
egonzalezhttp://logs.openstack.org/68/466268/3/check/gate-kolla-dsvm-build-centos-source-centos-7/946d410/logs/build/novajoin-base.txt.gz08:20
*** caowei has quit IRC08:21
jaosorioregonzalez: hey! thanks! sorry about the poking, still figuring out where logs are and such.08:21
openstackgerritJuan Antonio Osorio Robles proposed openstack/kolla master: Add images for novajoin services  https://review.openstack.org/46626808:22
egonzalezjaosorior, open the console.html log and scroll down, there should be something like 2017-05-22 06:59:51.688101 | CRITICAL:tests.test_build:>>> Expected image 'novajoin-base' to succeed!08:22
jaosorioregonzalez: so I saw that. But didn't know novajoin-base had it's own log file.08:23
egonzalezonce you know the image, open logs/build/<failed_image_logs>08:23
*** caowei has joined #openstack-kolla08:24
jaosorioregonzalez: that's good info. Thanks08:25
*** n0isyn0i1e is now known as n0isyn0ise08:25
*** lpetrut has joined #openstack-kolla08:31
masuberugood afternoon all, kolla-ansible deploy fails and ceph-mon does't start. https://pastebin.com/raw/wZp6SXgV08:32
openstackgerritMerged openstack/kolla-ansible stable/ocata: Fix log rotation issue on services  https://review.openstack.org/46653708:33
*** janki has quit IRC08:35
*** janki has joined #openstack-kolla08:36
*** jchhatbar has joined #openstack-kolla08:37
*** dasTor has quit IRC08:37
*** dasTor has joined #openstack-kolla08:39
*** janki has quit IRC08:40
*** pbourke has joined #openstack-kolla08:43
egonzalezmasuberu, can you share previous ceph tasks, docker ps -a, docker logs ceph_mon and cat /etc/kolla/ceph-mon/config.json? made any customization in the files?08:44
*** jchhatbar is now known as janki08:44
*** feisky has joined #openstack-kolla08:47
manheim_caowei you on?08:51
jaosorioris opendaylight having issues?08:52
egonzalezjaosorior, what kind of issues? currently only images are built but not deployed by ansible or k8s08:52
jaosorioregonzalez: oh, nevermind, it was a red herring on the log08:53
egonzalezjaosorior, build errors for images at https://github.com/openstack/kolla/blob/master/tests/test_build.py#L75 are ignored by the tests08:55
jaosorioregonzalez: it seems that the novajoin build failed http://logs.openstack.org/68/466268/4/check/gate-kolla-dsvm-build-ubuntu-source-ubuntu-xenial/9d6047e/console.html but I added it to the ignored_images08:55
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Add VMware DataStore support to glance  https://review.openstack.org/45217608:55
jaosoriorI mean08:55
jaosoriorexluded_images08:55
jaosorioregonzalez: right, I added them https://review.openstack.org/#/c/466268/4/tests/test_build.py08:56
egonzalezactually is failing aodh-base08:56
jaosorioroh08:56
egonzalezaodh removed etc/aodh folder and now uses config generator...08:57
openstackgerritEduardo Gonzalez proposed openstack/kolla master: Fix aodh-base image  https://review.openstack.org/46663909:03
hrwrestarted kolla builds on aarch64 (moonshot in mlab), ppc64le (power8 at Red Hat), x86_64 (my work laptop)09:05
masuberuegonzalez, ceph-mon container doesn't start, can't check logs09:07
egonzalezmasuberu, before fetching keyrings mons are started, any other task failed before? `docker logs ceph_mon` work on failed containers09:10
matrohonegonzalez, hi09:12
egonzalezmatrohon, sup09:12
matrohonegonzalez, inc0 : I fanally wrokaround my issues concerning the nova upgrade from newton to ocata09:12
matrohonhttps://bugs.launchpad.net/kolla-ansible/+bug/169246409:12
openstackLaunchpad bug 1692464 in kolla-ansible "nova upgrade failures from newton to ocata" [Undecided,New]09:12
matrohonegonzalez, did you succeed at reproducing my bug?09:13
egonzalezmatrohon, my upgrade succedd, although the workaround fix in the bug report should not fix the issue09:13
matrohonegonzalez, why do you say it doesn't fix the issue09:14
matrohon?09:14
egonzalezmatrohon, tested 3.0.3 to 4.0.0 not 4.0.3(do you have any public 4.0.3 images in some kind of repo to test with that version?)09:15
matrohonegonzalez, it's in a private repo, sorry09:15
matrohonegonzalez, but there is no custom config on the built images09:16
egonzalezmatrohon, because cell0 dont have a transport url setting, you hide the issue and "mounted" None cell (legacy) into cell009:16
matrohonegonzalez, so my upgraded cloud is not operational because of the lack of the transport-url in cell0 config?09:18
matrohonI didn't try it yet :(09:18
egonzalezcell0 just points to a nova_cell0 db for failed and unscheduled instances, real instances are mapped into a cell called None (this have transport_url and points to nova db)09:18
egonzalezmatrohon, if you use nova-manage cell_v2 list_cells --verbose will see connections for both cells09:19
matrohonegonzalez, I do, but I'm not sure to perfectly understand the cell magic behing... I need to read more docs!09:22
egonzalezmatrohon, https://docs.openstack.org/developer/nova/cells.html#cells-v209:22
egonzalezyep, is quite hard to understand at the beginning09:22
openstackgerritChen proposed openstack/kolla-ansible master: Fix some wording issues on a comment  https://review.openstack.org/46664609:22
egonzalezmatrohon, i'm going to upgrade again to latest ocata images09:26
egonzalezmatrohon, 3.0.3 from dockerhub to 4.0.3(latest ocata)09:27
zhubingbinghi gus09:27
zhubingbinghi guys09:27
matrohonegonzalez, great09:27
egonzalezsup zhubingbing09:27
zhubingbinghi egonzalez09:27
matrohonegonzalez, the cell setup issue might be a subsequent issue of the first failure described in my bug report09:28
egonzalezmatrohon, yep, is that what i want to re-verify, but may be an issue with other service09:29
egonzalezis fluent deployed and running in your nodes, from newton to ocata heka is replaced with fluentd09:30
matrohonI've run an upgrade to 4.0.2, it seems I'v to run a new upgrade to 4.0.3 :)09:30
egonzalezto 4.0.2 worked?09:31
matrohonegonzalez, with the workaround in teh bug report, the upgrade worked09:32
masuberuegonzalez, this is the only thing I can see form the logs --> https://pastebin.com/raw/wZp6SXgV09:38
egonzalezmasuberu, resumed the cell creation and upgrade process http://paste.openstack.org/show/610138/09:41
egonzalezmatrohon, ^^ wrong name09:42
egonzalezmasuberu, not use docker run into a failed container, use docker logs to see why the container failed to start09:42
matrohonegonzalez, ack, that what I saw for the upgrade, but I had to run another command for the upgrade to work : nova-manage --verbose cell_v2 map_cell_and_hosts09:45
*** cuongnv has quit IRC09:49
openstackgerritMartin Mágr proposed openstack/kolla master: Sensu images  https://review.openstack.org/46359309:51
egonzalezmatrohon, map_hosts is done by simple_cell-setup https://github.com/openstack/nova/blob/stable/ocata/nova/cmd/manage.py#L111209:51
*** sambetts_ is now known as sambetts09:51
*** zhuzeyu has quit IRC09:52
*** tovin07_ has quit IRC09:55
egonzalezmatrohon, can you share all nova upgrade tasks output with -vvv?10:01
egonzalezbtw, map_cell_and_hosts is needed to finish upgrade or to allow create new instances after upgrade?10:03
openstackgerritDai Dang Van proposed openstack/kolla-ansible master: Mixing binary and source images (2)  https://review.openstack.org/46666110:07
*** zhuzeyu has joined #openstack-kolla10:08
*** chas has quit IRC10:08
*** chas has joined #openstack-kolla10:09
matrohonegonzalez, my upgrade ha worked, I wo't be able to downgrade and reupgrade to get logs :)10:11
matrohonmap_cell_and_hosts is needed to finish the upgrade10:12
egonzalezmatrohon, can you share cell_v2 list_cells --verbose (hide your passwords ;)10:12
egonzalezmatrohon, to ensure cells mapping are properly created10:12
matrohonotherwise, the "Running nova bootstrap" taskfails10:13
matrohonsure10:13
*** chas has quit IRC10:13
matrohonegonzalez, http://paste.openstack.org/show/610146/10:15
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla master: Clean up unused sanity check modules  https://review.openstack.org/46551510:15
egonzalezmatrohon, my upgrade worked, going to share logs to check the outputs10:15
egonzalezmatrohon, looks good that10:15
matrohonegonzalez, ok, the issue might come from my config then...10:16
matrohonegonzalez, I did previous trial that failed, before having a correct inventory, those upgrade trial might have moved my cloud in a messy state10:18
openstackgerritDai Dang Van proposed openstack/kolla-ansible master: Mixing binary and source images (2)  https://review.openstack.org/46666110:19
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'outward' rabbitmq instance  https://review.openstack.org/45373810:22
*** stefan_bo has quit IRC10:24
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Mount cinder volume for cinder-volume and cinder-backup all the time  https://review.openstack.org/46666610:25
*** gfidente has joined #openstack-kolla10:26
openstackgerritDai Dang Van proposed openstack/kolla-ansible master: Mixing binary and source images (3)  https://review.openstack.org/46666710:26
egonzalezmatrohon, those are the steps I took for the upgrade http://paste.openstack.org/show/610147/10:26
egonzalezand the outputs from the upgrade tasks and list cells10:27
*** caowei has quit IRC10:28
egonzalezmatrohon, maybe if you played with various upgrade tries over the same environment broke something, many nova tasks first checks if some resource exists10:30
openstackgerritDai Dang Van proposed openstack/kolla-ansible master: Mixing binary and source images (3)  https://review.openstack.org/46666910:32
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a Kolla 'devstack' mode  https://review.openstack.org/45469010:32
openstackgerritDai Dang Van proposed openstack/kolla-ansible master: Mixing binary and source images (4)  https://review.openstack.org/46666910:33
openstackgerritChristian Berendt proposed openstack/kolla-ansible master: Use internal interface to connect with keystone_ssh  https://review.openstack.org/46667410:35
*** chas has joined #openstack-kolla10:43
*** chas has quit IRC10:47
*** chas has joined #openstack-kolla10:47
*** kbaegis has joined #openstack-kolla10:48
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Fix nova cell release note  https://review.openstack.org/46667610:53
openstackgerritPavel Gluschak (scsnow) proposed openstack/kolla-ansible master: Make cinder-backup service optional  https://review.openstack.org/46462010:54
*** numans has joined #openstack-kolla10:59
*** emccormick has joined #openstack-kolla11:01
*** bjolo has joined #openstack-kolla11:03
*** rwellum has joined #openstack-kolla11:04
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible stable/ocata: Mount cinder volume for cinder-volume and cinder-backup all the time  https://review.openstack.org/46668311:16
openstackgerritMerged openstack/kolla-ansible master: Mount cinder volume for cinder-volume and cinder-backup all the time  https://review.openstack.org/46666611:17
*** bigkol has joined #openstack-kolla11:24
bigkolHELP11:24
*** shardy has quit IRC11:27
*** dave-mccowan has joined #openstack-kolla11:30
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Fix nova cell release note  https://review.openstack.org/46667611:31
*** emccormick has quit IRC11:31
*** zul has joined #openstack-kolla11:36
*** athomas has quit IRC11:38
bjolohowdy11:45
bjoloJeffrey4l, around?11:47
*** jchhatbar has joined #openstack-kolla11:47
*** janki has quit IRC11:49
*** bigkol has quit IRC11:53
*** srwilkers has joined #openstack-kolla11:57
*** rhallisey has joined #openstack-kolla11:58
kbaegisHi all. I'm failing prechecks on the MountFlags=shared. I'm also no longer using systemd11:59
kbaegisCan anyone help me configure sysvinit to enable the shared flag? I can only seem to find instructions for the systemd setup12:00
egonzalezkbaegis, https://github.com/openstack/kolla-ansible/blob/stable/ocata/doc/quickstart.rst#additional-steps-for-upstart-and-other-non-systemd-distros12:02
*** bjolo has quit IRC12:05
kbaegisegonzalez: Thanks! I've actually done all that already. Still complains about MountFlags=shared not being set12:07
*** bjolo has joined #openstack-kolla12:07
kbaegisspecifically neutron12:07
kbaegisI'll post the -vvv dump in a minute. It's running12:08
egonzalezkbaegis, hrm, support for sysv was removed and now only checks for systemd(master) https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/tasks/precheck.yml#L2912:09
egonzalezin ocata still checking for sysv12:09
kbaegisI'm on stable/ocata12:13
kbaegisAlso systemd is an awful demonspawn of an init process12:13
kbaegisCan we add support for sysv again?12:13
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Fix nova cell release note  https://review.openstack.org/46667612:14
kbaegisHere's the -vvv for stable/ocata https://hastebin.com/ekosabowuw.sql12:16
egonzalezkbaegis, this task assume that ubuntu > 14.04 uses systemd ansible_distribution == 'Ubuntu' and ansible_distribution_version > '14.04'12:17
*** chenn has quit IRC12:19
*** chenn has joined #openstack-kolla12:20
kbaegisegonzalez: Interesting choice. :)12:20
egonzalezanyone with galera knowledge have a hint on what can be the issue while upgrading from ocata->master? https://bugs.launchpad.net/kolla-ansible/+bug/169250712:23
openstackLaunchpad bug 1692507 in kolla-ansible "mariadb fail to upgrade ocata->master" [Undecided,New]12:23
egonzalezimages and templates didnt change since ocata12:24
*** blallau has quit IRC12:25
*** bjolo has quit IRC12:37
kbaegisSo when switching to sysvinit deploy now hangs on "Waiting for nova-compute service up"12:40
kbaegisnova_compute container is up and functional, but it can't be reached12:41
egonzalezkbaegis, if executes nova service-list is there a compute service?12:42
kbaegisSec. Have to write an admin-rc by hand12:43
egonzalezkbaegis, nope you can use post-deploy12:43
egonzalezthat task will only copy a template with adminrc info12:44
kbaegisegonzalez: even if deploy fails?12:44
egonzalezyep, is just a copy file12:44
kbaegisegonzalez: no services show up12:44
egonzalezkbaegis, is rabbitmq running? what branch are you deploying?12:45
kbaegisstable/ocata12:45
kbaegisrabbitmq is certainly working12:45
egonzalezcan you get logs from nova-compute?12:46
kbaegisSure. https://hastebin.com/iqegizawaz.sql12:46
egonzalezkbaegis, logs from /var/lib/docker/volumes/kolla_logs/_data/kolla_logs/nova/nova-compute.log12:47
kbaegisSure, sec12:47
egonzalezsecond kolla_logs not needed12:47
kbaegishttps://hastebin.com/ebucapenuf.xml12:48
*** Pavo has joined #openstack-kolla12:48
kbaegiscinder service-list works fine btw12:50
egonzalezkbaegis, can you check this docker exec -ti kolla_toolbox mysql -unova -p<NOVA_DB_PASSWORD> -h <VIP> nova -e "SELECT * from services;"12:53
kbaegisSure12:54
kbaegissec12:54
kbaegisWhy don't I just run that straight in the db container?12:55
*** ArminderSingh has joined #openstack-kolla12:55
*** ArminderSingh- has quit IRC12:55
*** chenn has quit IRC12:56
kbaegisEither way. Ran successfully12:56
egonzalezkbaegis, you can, what is easier for you, i dont use to enter into container or remote hosts, all executed from ansible12:56
egonzalezwhats the output?12:56
*** chenn has joined #openstack-kolla12:57
kbaegishttps://hastebin.com/ururixanof.rb12:57
egonzalezkbaegis, you are using master images with ocata code12:57
kbaegisNo. I kolla-build myself12:57
egonzalezkbaegis, run docker exec nova_api pip freeze and share output12:58
egonzalezbut this issue was due a change in nova master for multi cells12:59
kbaegishttps://hastebin.com/rawehoticu.makefile12:59
*** ArminderSingh has quit IRC13:00
egonzalezkbaegis, nova==16.0.0.0b2.dev339 is master13:00
egonzalezocata is 15*13:00
kbaegisegonzalez: Should I kolla-build after pulling 4.0.1?13:00
*** manheim_ has quit IRC13:01
egonzalezkbaegis, nope, pull just download images from repository13:01
*** jheroux has joined #openstack-kolla13:01
*** salv-orlando has joined #openstack-kolla13:01
kbaegisegonzalez: You mean from docker.io?13:01
*** manheim_ has joined #openstack-kolla13:02
kbaegisI'm rebuilding from 14.0.1 rather than master. Feeling kind of silly about that, but I didn't realize it would cause issues like this13:03
kbaegisThanks egonzalez13:03
egonzalezkbaegis, is an issue of master, openstack projects changes things that can break our deployment13:04
egonzalezin this case ocata ansible dont work to deploy master/pike13:04
*** zul has quit IRC13:05
*** manheim_ has quit IRC13:06
*** salv-orlando has quit IRC13:07
*** salv-orlando has joined #openstack-kolla13:07
*** huzhengchuan_ has quit IRC13:11
*** huzhengchuan has quit IRC13:12
*** zul has joined #openstack-kolla13:12
*** chenn has quit IRC13:14
*** chenn has joined #openstack-kolla13:15
*** salv-orlando has quit IRC13:21
*** eanylin has quit IRC13:22
*** manheim_ has joined #openstack-kolla13:32
*** jchhatbar has quit IRC13:33
*** ducttape_ has joined #openstack-kolla13:35
*** myoung|pto is now known as myoung_13:48
Jeffrey4lbjolo__, yep13:50
*** schwicht has joined #openstack-kolla13:52
*** korzen has quit IRC13:53
openstackgerritKurt Taylor proposed openstack/kolla master: Fix inconsistent build task attempt logging  https://review.openstack.org/46673813:59
*** salv-orlando has joined #openstack-kolla14:03
*** goldyfruit has joined #openstack-kolla14:07
*** salv-orlando has quit IRC14:08
*** goldyfruit has quit IRC14:09
*** emccormick has joined #openstack-kolla14:10
*** skramaja has quit IRC14:18
*** eanylin has joined #openstack-kolla14:18
*** stefan_bo has joined #openstack-kolla14:20
*** stefan_bo has quit IRC14:25
sdakeJeffrey4l what do you make of this http://logs.openstack.org/31/464031/34/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/d9d8a58/console.html#_2017-05-21_04_52_44_91549714:28
Jeffrey4lsdake, seems like a ansible bug https://github.com/ansible/ansible/issues/1228214:32
*** david-lyle has quit IRC14:34
*** david-lyle has joined #openstack-kolla14:35
rhalliseyany cores mind looking at: https://review.openstack.org/#/c/465145/14:35
*** ddyer has quit IRC14:36
sdakesbezverk is there a way to print all secrets in kubernetes?14:37
sdakesbezverk (the contents of the secrets)14:37
sbezverksdake: but then they would not be secrets any more ;)14:38
sdakesbezverk i could swear i found a website on google that explained how to do so14:38
sbezverkyou just need to do reverse of base6414:39
sdakerhallisey enjoy14:39
sdakesbezverk in fact this code had base64 in it14:39
sdakesbezverk thanks for jogging my memory14:39
rhalliseysdake, cheers14:39
sbezverksdake: again 1 liner in bash ;)14:40
sdakesbezverk i am more interested in password rotation14:40
sdakewhich is not a one liner in any language :)14:40
*** ddyer has joined #openstack-kolla14:40
sbezverksdake: kfox1111 has programmed token rotation for fernet, check his code..14:41
sdakesbezverk yup i know fernet rotates and i have had a brief look at his code14:41
sdakesbezverk in the case of full password rotation, all the configmaps must be modified14:41
sdakesbezverk any way work for the future :)14:42
sdakeone of my new servers is flakey14:42
* sdake growls14:43
*** korzen has joined #openstack-kolla14:43
sdakesbezverk do you use uefi in your servers?14:44
sbezverksdake: nope, do not see any reason14:44
openstackgerritSurya Prakash (spsurya) proposed openstack/kolla-kubernetes master: Fixed all Warning and Errors in existing docs  https://review.openstack.org/46347614:44
sbezverkjust because it is there is not good enough reason for me..14:45
*** schwicht has quit IRC14:48
sdakesbezverk fair - just curious :) thanks14:48
sdakeuefi is pretty nice, especially secure boot (when it works)14:49
sdakei find in general secure boot gets in the way of good dev, although if I was going to prod, I'd use secure boot14:49
*** david-lyle has quit IRC14:52
*** iniazi_ has joined #openstack-kolla14:52
*** salv-orlando has joined #openstack-kolla14:52
*** powerd has joined #openstack-kolla14:54
*** iniazi has quit IRC14:55
*** lucasxu has joined #openstack-kolla14:56
*** eaguilar has joined #openstack-kolla14:56
*** salv-orlando has quit IRC14:57
inc0good morning15:02
inc0Jeffrey4l: arround?15:02
kbaegisinc0: Morning!15:03
Jeffrey4linc0, morning15:03
inc0hey, I won't be able to make it for Wed meeting probably15:03
inc0would you be kind enough to run it?15:03
Jeffrey4lok.15:03
inc0thank you15:03
Jeffrey4lmy pleasure ;)15:03
inc0ok, off to gates I go, I'm furious because I'm really close to get it working, but still small issues appear here and there15:06
*** powerd has quit IRC15:09
*** bmaltais has joined #openstack-kolla15:13
bmaltaisGood morning. I have a question regarding kolla-ansible15:13
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate  https://review.openstack.org/46600715:14
bmaltaisI built a 2 node test cluster and I am trying to figure out how to do a "shutdown/restart"15:14
inc0bmaltais: of everything?15:14
bmaltaisWe have yearly power shutdown and I try to simulate what would happen when we do that15:14
bmaltaisI found kolla-ansible stop -i multinode15:15
inc0we were working on it, not sure what's state od it15:15
bmaltaisBut I can't find how to issue a "start" equivalent to cleanly start things up15:15
inc0deploy will start things up15:15
inc0try rerunning depliy15:15
inc0deploy15:15
bmaltaisOK. So just do a deploy on the whole multinode cluster to bring it back up.15:16
inc0that's the plan15:16
bmaltaisI will give that a try. Thanks inc0:15:16
inc0one thing tho15:16
inc0mariadb is notorious in terms of full shutdown15:16
inc0so you might need to get this one up manually15:16
bmaltaisinc0: This is what I see if I manually docker start processes on the nodes15:17
bmaltaismariadb get in a restart loop15:17
inc0http://galeracluster.com/documentation-webpages/restartingcluster.html15:17
bmaltaisI hope deploy will fix that15:17
inc0it won't15:17
inc0it's safety built into mariadb itself15:17
inc0or rather, galera15:17
inc0since galera is multi master15:18
inc0one of nodes will have latest state15:18
inc0issue is, we don't know which one15:18
inc0and if you start wrong one, that may end with data corruption15:18
inc0so you need to follow procedure I linked to bring up mariadb manually15:18
bmaltaisI see... so this is a fairly significant issue15:19
inc0in reality it's just chaning safe_to_bootstrap from 0 to 1 on single node15:19
bmaltaisI will try that15:19
inc0finding correct node is more problematic15:19
inc0bmaltais: if you can gracefully shutdown cluster it will make your life easier15:19
bmaltaisinc0: By gracefully you mean using the kolla-ansible stop command or something else?15:20
inc0on top of that one issue, just starting up docker containers should work15:20
inc0no, manually put down one mariadb container at the time15:20
inc0I don't think our stop command is that clever15:20
inc0would be nice addition tho;)15:21
matrohonegonzalez, you told me that you were testing upgrades to 4.0.3, but the current tag for ocata is 4.0.2, am I wrong?15:21
inc0bmaltais: you can find files used in this guide I linked in /var/lib/docker/volumes/mariadb/_data15:21
bmaltaisinc0: OK... will try that to see how much work it is15:21
SamYapleinc0: you have to change safe_to_bootstrap AND update the gcomm:// list to empty15:21
*** eaguilar has quit IRC15:22
SamYaplesafe_to_bootstrap is a safety feature added on top of the old way15:22
inc0yeah, what he said:)15:22
egonzalezmatrohon, yep, images built for gates are 4.0.3, are the ones I used15:22
inc0SamYaple: what's up with galera doing this automatically?15:22
inc0you mentioned it one day15:22
SamYapleinc0: it can do it automatically in a number of situations, but you have to know those situations15:23
SamYaplethe logic can be baked into startup scripts if those startup scripts can query the other nodes15:23
SamYapleif they are stand alone the logic can be backed into a 100% idempotent ansible playbook15:23
SamYaplethere are only 2 sceanrios where galera doesnt know how to recover15:23
SamYapleand those involve real data loss15:24
*** vhosakot has joined #openstack-kolla15:24
matrohonegonzalez, I'm suprised, since the 4.0.2 has not even been released15:25
*** jmccarthy has left #openstack-kolla15:25
inc0matrohon: it hasn't been pushed to dockerhub15:26
inc0dockerhub is not really our release;)15:26
inc0we're working on resolving this now15:26
matrohoninc0, but there is no 4.0.2 tag on git15:26
egonzalezmatrohon, sorry was 4.0.2, 4.0.1 is released but not pushed to DH15:26
matrohonegonzalez, ack15:26
inc0ahh, that's true, well if you build today it will be 4.0.215:26
egonzalez4.0.2 is tagged internally by pbr15:26
inc0until we tag 4.0.2 in git then it's 4.0.315:27
egonzalezrelease + 115:27
matrohonok thanks15:27
egonzalezinc0, may be the db stuff related with https://bugs.launchpad.net/kolla-ansible/+bug/169250715:27
openstackLaunchpad bug 1692507 in kolla-ansible "mariadb fail to upgrade ocata->master" [Undecided,New]15:27
*** eaguilar has joined #openstack-kolla15:27
bmaltaisinc0: Given containers... how do you gracefully shutdown a container running mariadb? I guess docker stop <mariadb id> is not it...15:28
egonzalezjust tested upgrade from ocata -=> master and mariadb keeps restarting15:28
inc0bmaltais: it does...to a point15:28
egonzaleznewton->ocata works fine15:28
inc0it will timeout after 10s (I think that's the default)15:28
inc0and then force kill15:28
*** ducttape_ has quit IRC15:29
*** ducttape_ has joined #openstack-kolla15:30
inc0docker stop -t 30 mariadb will work better15:30
inc0or whatever t you need15:30
SamYapleinc0: i dont think kolla has mariadb setup to cleanly exist, like ever15:30
SamYapleyou have to issue the shutdown a different way than just TERM15:30
SamYaple10 seconds, 30 seconds isnt going to matter much15:31
SamYapleit shutsdown cleanly in about 2 seconds on my active cluster when issued with mysqladmin15:31
inc0well I think dump init will relay sigkill to mariadb15:31
SamYaplesigkill will just kill it15:31
*** myoung_ is now known as myoung15:31
inc0ahh, yay15:31
SamYapleit wont respect term though15:31
SamYaplewell i think it *does*, but the proper way to do it is with mysqladmin or the like15:32
inc0so docker exec -it mariadb mysqladmin stop?15:32
SamYapleand have the process kill itself15:32
SamYapleshutdown, but thats the jist yea15:32
inc0thanks SamYaple15:32
inc0that's good knowledge15:32
SamYaplei enable socket based auth so it doesnt need a password either15:32
inc0brb15:33
vhosakotbmaltais: is there a mariadb/mysql command to graceflly shutdown mariedb if mariadb wasn't running in a container?15:33
*** powerd has joined #openstack-kolla15:34
*** lpetrut has quit IRC15:35
*** salv-orlando has joined #openstack-kolla15:36
vhosakotmasuberu: here are the steps to remove the ceph_mon_config container if ceph's volume or config.json is corrupted --> https://github.com/openstack/kolla-ansible/blob/master/doc/ceph-guide.rst#deploy-fails-with-fetching-ceph-keyrings--no-json-object-could-be-decoded15:38
openstackgerritSurya Prakash (spsurya) proposed openstack/kolla-kubernetes master: Fixed all Warning and Errors in existing docs  https://review.openstack.org/46347615:40
vhosakotmasuberu: I'd check if this task passed and if ceph-mon.config.json exists on your ceph-mon node -->  https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/ceph/tasks/config.yml#L12-L1915:41
*** srwilkers has quit IRC15:44
vhosakotsdake: I just use deprecation_warnings=False in my ansible.cfg is deprecation warning cause blockers :)15:46
bmaltaisinc0: Running "docker exec -it mariadb mysqladmin stop" produce error: 'Access denied for user 'mysql'@'localhost' (using password: NO)'15:47
vhosakotsdake: ^^^ about the warning in http://logs.openstack.org/31/464031/34/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/d9d8a58/console.html#_2017-05-21_04_52_44_915497.15:47
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements  https://review.openstack.org/46525815:48
sdakevhosakot thanks15:48
vhosakotsdake: np15:48
bmaltaisinc0: Is there a default password for mariadb?15:48
vhosakotbmaltais: is it database_password in etc/kolla/passwords.yml on yout deploy node  -->  https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/mariadb/tasks/register.yml#L915:51
*** chenn has quit IRC15:52
*** manheim_ has quit IRC15:57
*** egonzalez has quit IRC15:58
*** pcaruana has quit IRC15:59
sdakebmaltais the mariadb password is in the passwords.yml file15:59
*** goldyfruit has joined #openstack-kolla16:01
*** AzazeL_ has joined #openstack-kolla16:01
vhosakotkbaegis: the task https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/tasks/precheck.yml#L29-L30 assumes the target node has systemd/systemctl.  If not, it is expected to fail if the command "systemctl" does not exist :)16:01
openstackgerritMerged openstack/kolla-kubernetes master: Fixed all Warning and Errors in existing docs  https://review.openstack.org/46347616:02
*** david-lyle has joined #openstack-kolla16:03
bmaltaissdake: That worked... but running "mysqladmin -u root -psomepassword shutdown" work but the alwaysrestart on the container start it back up ;-)16:03
*** lucasxu has quit IRC16:04
vhosakotbmaltais: did you get somepassword from database_password in etc/kolla/passwords.yml on your deploy node?16:04
bmaltaisSo perhaps running "docker exec -it mariadb mysqladmin -u root -psomepass shutdown ; docker stop mariadb" would work... and it look like it does.16:05
*** matrohon has quit IRC16:06
bmaltaisvhosakot: Yes, that was the one16:06
vhosakotbmaltais: ah cool16:06
bmaltaisNow that both mariadb container are "cleanly" showdown I will attempt a restart... starting with the last node that ran mariadb and see if things will come back clean16:08
*** chas has quit IRC16:09
vhosakotbmaltais: are you using sysvinit instead of systemd in Ubuntu?16:11
bmaltaisNo, running on centos716:11
vhosakotbmaltais: ah ok16:11
bmaltaisUnfortunatly evendoing a shutdown mariadb is still restarting ;-(16:12
bmaltaisSo clearly need more work than this ;-)16:12
kbaegisvhosakot: For those of us that are supremely unhappy with systemd, it would be nice if it could at least fallback to a behavior with sysvinit that doesn't completely break precheck16:14
kbaegisvhosakot: The documentation purports to support non-systemd setups (upstart/sysv)16:14
vhosakotkbaegis: I see this is the only task that needs systemctl which fails when systemctl is not available.  It is a precheck task, if systemd does not exist, that task's failure could be ignored and the opertor could check docker's hralth manually using the eqvuivalent sysvinit command :)  -->   https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/tasks/precheck.yml#L29-L3016:16
*** lucasxu has joined #openstack-kolla16:18
vhosakotkbaegis: agreed, doc has stuff about sysvinit but the roles/tasks assume systemctl.16:19
*** tvignaud is now known as tvignaud_afk16:19
vhosakotI remember seeing a systemd vs sysvinit cheatsheet to convert one to another for health checks and restarts.16:20
*** korzen has quit IRC16:20
*** ntpttr__ is now known as ntpttr16:21
*** srwilkers has joined #openstack-kolla16:21
*** goldyfruit has quit IRC16:23
*** powerd has quit IRC16:23
kbaegisvhosakot: Question: if the behavior can be gotten from a lower level function (ala docker), wouldn't you want it to check that rather than doing it through systemd?16:27
kbaegisI actually don't know if docker-api or docker-cli expose the necessary information, but if it does it would be nice16:29
*** korzen has joined #openstack-kolla16:29
kbaegisAt a more basic level, I don't understand what MountFlags=shared does, nor does anyone on #docker apparently16:31
kbaegisOr maybe my ignorance is simply contagious and I'm asking the wrong questions :)16:33
kfox1111morning.16:34
vhosakotkbaegis: right, don't know what gives the required info about MountFlags other than "systemctl show docker".  I'll check if there is a docker CLI to provide the info about MountFlags.  It is a task to check if MountFlags is set to shared on certain neutron containers.16:34
vhosakotmorning kfox111116:34
sbezverkkfox1111: good morning16:34
kfox1111sbezverk: yeah. working on it.16:35
*** eaguilar has quit IRC16:35
vhosakotkbaegis: will keep you posted.16:35
kfox1111am dealing with a failing laptop today. :/16:35
kbaegisvhosakot: Thank you!16:35
sbezverkkfox1111: got bored on sunday and clean up a bit more v4 gate jobs, hopefully soon we will see more green than red..16:35
vhosakotnp16:35
kfox1111sbezverk: cool. :)16:35
kfox1111shoot... another fire. :/16:36
*** korzen has quit IRC16:36
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: v4 gate.  https://review.openstack.org/45484116:39
*** salv-orlando has quit IRC16:40
*** jaosorior is now known as jaosorior_away16:45
*** pbourke has quit IRC16:46
rwellumsbezverk: starting today I start to see this when deploying - this is on a happily running cluster with canal installed https://www.irccloud.com/pastebin/qeapLkXr/16:47
*** unicell has quit IRC16:52
*** Serlex has quit IRC16:53
inc0back, sorry16:56
inc0look at number of gates here:)16:57
inc0https://review.openstack.org/#/c/466007/5516:58
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate  https://review.openstack.org/46600716:59
*** salv-orlando has joined #openstack-kolla17:01
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multi-scenerio gates  https://review.openstack.org/46681517:04
*** salv-orlando has quit IRC17:05
bmaltaisOK, I think I figured out how to "properly?" shutdown a kolla-ansible cluster and then start it back up properly17:10
bmaltaisNot automated right now and involve multiple manual steps but it worked17:11
inc0nice bmaltais17:11
vhosakotbmaltais: cool, can you submit a ops_best_practices.rst please ;)17:11
inc0anything besides mariadb was manual?17:11
vhosakot^^^17:12
inc0yeah, vhosakot is right, would you mind writing short doc about it?17:12
bmaltaisEssentially onew need to do "docker exec -it mariadb mysqladmin -u root -pckZoS17Pkg4LfMZJQIhKoJvML4P6KnOhm4lk3znW shutdown ; docker stop mariadb" on each controller nodes17:12
inc0I'll be happy to guide you through process17:12
bmaltaisThen do a "kolla-ansible stop -i multinode"17:12
bmaltaisvhosakot: Sure. Not sure if it is "best" practice but it did work ;-)17:13
inc0bmaltais: so what we wanted to add after summit17:13
bmaltaisNExt you start back the nodes and do a "docker stop mariadb" on each controller nodes17:13
inc0is set of quick howtos for different things17:13
inc0stopping cluster would be great one to kick this off17:14
vhosakotbmaltais: ah, you used "kolla-ansible stop" as well..  we have  https://github.com/openstack/kolla-ansible/blob/master/tools/kolla-ansible#L50  as well and it worked when I lasted tested it is Mitaka.17:14
bmaltaisLast you do a "kolla-ansible mariadb_recovery -i multinode"17:14
bmaltaisAnd done17:14
*** lucasxu has quit IRC17:14
inc0vhosakot: mariadb needs to be manually gracefully shitdown17:14
inc0if you want to avoid problems later17:14
vhosakotinc0: shitdown lol.. ah ok17:14
inc0there you go17:15
inc0unless you shit this down17:15
inc0it's going to shit you up17:15
bmaltaisinc0: That's if you forget the gracefull shutdown step ;-)17:15
vhosakotbmaltais: glad it is working... would be awesome if you please doc it with steps how to stop/shutdown and turn back on a kolla cluster.17:15
bmaltaisvhosakot: WHere is the ops_best_practices.rst file? Don't see it in doc17:17
inc0bmaltais: let's create it17:17
inc0you'll going to be first one:)17:18
*** unicell has joined #openstack-kolla17:18
bmaltaisOK. I will try to add it. I am new to gerrit/openstack documentation.17:18
*** jrich523 has quit IRC17:19
inc0so, just create this file and add link to it here: https://github.com/openstack/kolla-ansible/blob/master/doc/index.rst#overview17:19
inc0github doesn't render rst too well17:19
inc0bmaltais: do you need help with gerrit itself?17:20
vhosakotbmaltais: it doesn't exist yet :)17:20
vhosakotbmaltais: I can help you add one :)  I'm planning to add one to for both single-node and multi-node.17:20
inc0vhosakot: forger single node imho17:21
inc0let's focus on all things multinode17:21
inc0as it's really not different and much more useful17:21
vhosakotright, ops don't use singlenode anyway, good point17:21
inc0also graceful shutdown of cluster while talking about single node is...well...17:21
inc0they call it cluster for a reason;)17:22
vhosakotright ;)17:22
vhosakotbrb 2 mins, rebooting.17:22
bmaltaisI used https://review.openstack.org/gitweb?p=openstack/kolla-ansible.git;a=tree;f=doc;h=ad2f262c1b240b43d743e3080d01af8c120c41b8;hb=HEAD in the past17:23
bmaltaisBut can't see how to add fiolesvia this interface17:23
inc0bmaltais: let me link you guide17:24
inc0I suggest reading this one https://docs.openstack.org/infra/manual/developers.html17:24
*** vhosakot_ has joined #openstack-kolla17:25
bmaltaisOK17:25
inc0don't hesitate to ask if you have any questions17:25
bmaltaisIt is sort of frustrating when the process is in the way of creativity ;-)17:25
inc0it's not that bad17:25
bmaltaisBut I understand the need for proper change control and tracking17:26
inc0and some process is required if you have north of 200 contributors;)17:26
bmaltaisTHe hard part is the 1st setup.17:26
inc0yeah gerrit has its quirks17:26
*** vhosakot has quit IRC17:27
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multi-scenerio gates  https://review.openstack.org/46681517:31
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate  https://review.openstack.org/46600717:31
vhosakot_back17:33
*** rpittau_ has joined #openstack-kolla17:42
*** kbaegis has quit IRC17:46
*** tonanhngo has joined #openstack-kolla17:48
*** rpittau_ has quit IRC17:49
*** kbaegis has joined #openstack-kolla17:49
*** sambetts is now known as sambetts|afk17:59
inc0guys we have issue at hand, kolla gates seems to be broken18:02
kbaegisHave you tried wiggling the latch?18:03
kbaegis:)18:03
inc0yeah, I also tried surgical kick18:03
inc0it seems like aodh became uninstallable18:04
kbaegisWell, where there's smoke there's Aodh18:06
*** jemcevoy has joined #openstack-kolla18:08
*** chas has joined #openstack-kolla18:09
vhosakot_inc0: is it always the first Dockerfile?  have you changed it to zaodh> ;)18:10
jemcevoyAre there any tools for pruning the elasticsearch indices Ours have grown to 36GB and are on the root partition.18:10
*** lucasxu has joined #openstack-kolla18:12
*** chas has quit IRC18:14
*** itlinux has joined #openstack-kolla18:21
bmaltaisinc0: vhosakot: Thank's, figured out gerrit and now working on my 1st commit ;-)18:24
vhosakot_bmaltais: nice +1 :)18:24
inc0bmaltais: thank you!18:24
bmaltaisShould have something ready soon for validation18:24
sbezverksbezverk: why do not you check actual labels/taints on the node you try to remove it from?18:25
vhosakot_jemcevoy: good to have a separate parition for logs as node might be unusable if /root is full.18:25
sbezverkrwellum: ^^^18:25
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate  https://review.openstack.org/46600718:29
*** chas has joined #openstack-kolla18:31
jemcevoyvhosakot_ Well it is a little late... Already in production...  And I gave the majority of the boot drives to the LVM thin pool 'cuz I thought all the docker runtime stuff would go to the thin pool... MISTAKE...18:33
vhosakot_jemcevoy: yeah, docker's runtime increases usage in /var/lib/docker in root partition I've seen.18:34
vhosakot_especially the logging container, which is supposed to grow its volume in size fast due to all the logging happening :)18:35
*** chas has quit IRC18:35
vhosakot_in staging, we just truncate logs once the logging volume reaches a threshold disk space.18:37
*** harlowja has joined #openstack-kolla18:37
jemcevoyvhosakot_ : Could I move the /var/lib/docker/volumes/kolla_logs/ and elasticsearch into a large ceph volumes on each controller?18:38
*** masuberu has quit IRC18:39
rwellumWhich comment sbezverk ? the separate partition of the label/taint?18:39
vhosakot_jemcevoy: hmm, haven't tried it... the docker volumes must reside locally on the docker host I think, instead of attaching to a remove volume like ceph over the network.18:40
sbezverkrwellum: label/taint18:40
*** dave-mccowan has quit IRC18:41
*** lucasxu has quit IRC18:41
jemcevoyI'll need to do some lab experiments next week... :-)  vhosakot_18:41
*** itlinux has quit IRC18:41
vhosakot_+1 ;)18:42
vhosakot_never heard of remote docker volume over the wire tbh18:42
*** itlinux has joined #openstack-kolla18:42
*** ArminderSingh has joined #openstack-kolla18:43
*** lucasxu has joined #openstack-kolla18:43
*** harlowja has quit IRC18:43
vhosakot_jemcevoy: looks like someone did it using NFS server+client -->  http://container42.com/2014/03/29/docker-quicktip-4-remote-volumes/18:44
*** zhuzeyu has quit IRC18:48
*** itlinux has quit IRC18:48
*** zhuzeyu has joined #openstack-kolla18:48
*** manheim_ has joined #openstack-kolla18:51
*** manheim_ has quit IRC18:55
*** Mannheim has joined #openstack-kolla18:57
*** lucasxu has quit IRC18:58
openstackgerritBernard Maltais proposed openstack/kolla-ansible master: Add instruction on shutting down and restarting complete kolla-ansible environment.  https://review.openstack.org/46687618:59
*** powerd has joined #openstack-kolla19:02
*** zhubingbing has quit IRC19:02
*** zhubingbing has joined #openstack-kolla19:02
*** lucasxu has joined #openstack-kolla19:03
*** feisky has quit IRC19:07
*** blallau has joined #openstack-kolla19:09
*** goldyfruit has joined #openstack-kolla19:11
*** goldyfruit has quit IRC19:11
*** chas has joined #openstack-kolla19:12
*** iniazi has joined #openstack-kolla19:13
*** iniazi_ has quit IRC19:16
*** chas has quit IRC19:17
*** salv-orlando has joined #openstack-kolla19:18
*** salv-orlando has quit IRC19:28
*** chas has joined #openstack-kolla19:33
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate  https://review.openstack.org/46600719:38
*** chas has quit IRC19:38
*** egonzalez has joined #openstack-kolla19:44
*** spsurya has quit IRC19:44
*** eaguilar has joined #openstack-kolla19:44
openstackgerritOpenStack Proposal Bot proposed openstack/kolla stable/newton: Updated from global requirements  https://review.openstack.org/45330919:46
openstackgerritBernard Maltais proposed openstack/kolla-ansible master: Added pointer to new file in index.rst  https://review.openstack.org/46688119:50
*** chas has joined #openstack-kolla19:54
vhosakot_inc0: I see you removed tools/deploy_aio.sh in https://review.openstack.org/#/c/466007/.  won't it break all the gates?  :)   is the idea to remove tools/deploy_aio.sh and use the new workflow in gate you have in https://review.openstack.org/#/c/466007/ ?19:55
vhosakot_inc0: instead of removing tools/deploy_aio.sh, why not add something like deploy_multinode.sh?19:56
*** salv-orlando has joined #openstack-kolla19:56
openstackgerritBernard Maltais proposed openstack/kolla-ansible master: Fixing small typos.  https://review.openstack.org/46688119:57
openstackgerritOpenStack Proposal Bot proposed openstack/kolla stable/ocata: Updated from global requirements  https://review.openstack.org/46480719:57
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-ansible stable/ocata: Updated from global requirements  https://review.openstack.org/45287019:57
*** chas has quit IRC19:59
egonzalezplease, review to fix source builds https://review.openstack.org/#/c/466639/20:08
*** powerd has quit IRC20:08
openstackgerritBernard Maltais proposed openstack/kolla-ansible master: Fixing small typos.  https://review.openstack.org/46688120:10
bmaltaisI think I made a faux pas in gerrit as this is my 1st experience...20:11
bmaltaisI commited a change over a change instead of doing a --amend20:12
bmaltaisNow some are asking why I am not making patch on original commit ;-(20:12
bmaltaisHow do I fix this error?20:12
*** chas has joined #openstack-kolla20:15
egonzalezbmaltais  what I use to do when  screw up  git things is remove local branch  and fetch gerrit change, then work on it again with commit --amend ,  i do  not complicate  too much with git commands unless i  have  so  much  work on the wrong  change20:17
*** chas has quit IRC20:19
*** stefan_bo has joined #openstack-kolla20:22
*** stefan_bo has quit IRC20:27
vhosakot_bmaltais: yes, without --amend, "git review" will generate multiple reviews in gerrit, one review for each commit20:29
vhosakot_bmaltais: there is way to undo the commits and move back to HEAD using "git reset"20:30
*** chas has joined #openstack-kolla20:36
*** ducttape_ has quit IRC20:39
*** salv-orlando has quit IRC20:39
*** ducttape_ has joined #openstack-kolla20:40
rwellumsorry sbezverk caught up in long meeting. I'm trying to untaint a new cluster. Default behaviour for an AIO.20:40
*** salv-orlando has joined #openstack-kolla20:40
*** chas has quit IRC20:41
*** blallau has quit IRC20:42
*** Pavo has quit IRC20:43
*** salv-orlando has quit IRC20:44
*** ddyer has quit IRC20:49
*** eaguilar has quit IRC20:51
inc0vhosakot_: idea is that multinode and aio aren't that different20:53
inc0I'm collapsing everything to single, ansible-driven, workflow20:53
*** ddyer has joined #openstack-kolla20:53
inc0and I got it working already, issue I'm struggling right now is managing failures and downloading logs from everywhere20:53
*** eaguilar has joined #openstack-kolla20:54
openstackgerritBernard Maltais proposed openstack/kolla-ansible master: Add instruction on shutting down and restarting complete kolla-ansible environment.  https://review.openstack.org/46688120:55
vhosakot_inc0: cool, so will deploy_aoi.sh be removed?20:55
vhosakot_aio*20:55
inc0vhosakot_: there is no reason to keep it20:56
vhosakot_inc0: well, gate uses it daily :)  so, the gate will use the new apporach using your ansible playbooks to deploy in gate instead of deploy_aio.sh?20:56
inc0yup20:57
*** chas has joined #openstack-kolla20:57
vhosakot_inc0: cool, got it, thanks!20:57
inc0lots of that stuff, like deploy_aio or setup_node are leftovers from early Kolla20:57
inc0now we have things like boostrap-servers that can take care of most of it20:58
openstackgerritBernard Maltais proposed openstack/kolla-ansible master: Add instruction on shutting down and restarting complete kolla-ansible environment.  https://review.openstack.org/46688120:58
inc0bmaltais: soo, one thing about gerrit:)20:58
inc0one commit == one change in gerrit20:59
inc0so for example https://review.openstack.org/#/c/466876/ this is one commit and https://review.openstack.org/#/c/466881/ this is another set21:00
inc0look at Change-Id, it's a bit messed up;)21:00
*** ddyer has quit IRC21:00
inc0what you want to do is fix the feedback from review21:00
inc0and do git commit --amend21:00
inc0--amend flag means "edit last commit instead of creating new one"21:00
*** chas has quit IRC21:02
*** ddyer has joined #openstack-kolla21:03
*** bmace__ has quit IRC21:04
*** bmace__ has joined #openstack-kolla21:04
*** dave-mccowan has joined #openstack-kolla21:05
openstackgerritOpenStack Proposal Bot proposed openstack/kolla stable/newton: Updated from global requirements  https://review.openstack.org/45330921:06
egonzalezinc0 can you review this change  which  fix source build gates https://review.openstack.org/#/c/466639/21:08
*** ducttape_ has quit IRC21:08
inc0done, I was looking at it myself21:09
*** ducttape_ has joined #openstack-kolla21:09
egonzalezthanks21:09
*** dasTor_ has joined #openstack-kolla21:10
*** dasTor has quit IRC21:11
*** chas has joined #openstack-kolla21:18
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate  https://review.openstack.org/46600721:22
*** chas has quit IRC21:22
*** rhallisey has quit IRC21:22
*** fooliouno has joined #openstack-kolla21:24
*** jheroux has quit IRC21:30
*** ddyer has quit IRC21:30
*** ddyer has joined #openstack-kolla21:34
*** ddyer has quit IRC21:34
*** Pavo has joined #openstack-kolla21:36
*** ddyer has joined #openstack-kolla21:38
*** chas has joined #openstack-kolla21:39
*** chas has quit IRC21:43
*** eaguilar has quit IRC21:56
*** lucasxu has quit IRC21:58
*** chas has joined #openstack-kolla21:59
openstackgerritMerged openstack/kolla master: Fix aodh-base image  https://review.openstack.org/46663922:02
vhosakot_inc0: great work, deploy_aio.sh is used for kolla gate as well, not just for kolla-ansible gate  -->  http://logs.openstack.org/39/466639/1/check/gate-kolla-dsvm-deploy-centos-binary-centos-7-nv/b363a9e/console.html#_2017-05-22_20_24_16_43460722:04
vhosakot_inc0: so, the kolla gate must also use the new approach in your review instead of deploy_aio.sh, right?22:04
*** chas has quit IRC22:04
*** egonzalez has quit IRC22:06
*** Mannheim has quit IRC22:12
*** ducttape_ has quit IRC22:12
*** ducttape_ has joined #openstack-kolla22:12
*** itlinux has joined #openstack-kolla22:13
inc0vhosakot_: I think kolla gate should work, it uses kolla-ansible's gate code22:16
vhosakot_inc0: ah cool, good both kolla and kolla-ansible use the new approach with ansible playbooks, thanks for the info.22:16
inc0that's the plan:)22:17
vhosakot_inc0: plan is all that is needed ;)22:17
*** manheim_ has joined #openstack-kolla22:20
*** chas has joined #openstack-kolla22:20
*** Pavo has quit IRC22:20
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate  https://review.openstack.org/46600722:21
*** manheim__ has joined #openstack-kolla22:23
*** fooliouno has quit IRC22:25
*** chas has quit IRC22:25
*** manheim_ has quit IRC22:26
openstackgerritBernard Maltais proposed openstack/kolla-ansible master: Add instruction on shutting down and restarting complete kolla-ansible environment.  https://review.openstack.org/46688122:37
*** manheim__ has quit IRC22:40
*** chas has joined #openstack-kolla22:41
*** itlinux has quit IRC22:42
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements  https://review.openstack.org/46439322:42
*** chas has quit IRC22:46
*** celo_r has joined #openstack-kolla22:47
*** powerd has joined #openstack-kolla22:48
*** ducttape_ has quit IRC22:49
*** ducttape_ has joined #openstack-kolla22:50
*** ducttap__ has joined #openstack-kolla22:54
*** ducttape_ has quit IRC22:54
*** itlinux has joined #openstack-kolla23:01
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate  https://review.openstack.org/46600723:02
*** chas has joined #openstack-kolla23:02
*** itlinux has quit IRC23:05
*** chas has quit IRC23:06
*** ducttap__ has quit IRC23:08
*** powerd has quit IRC23:10
*** chas has joined #openstack-kolla23:23
*** chas has quit IRC23:27
*** stefan_bo has joined #openstack-kolla23:32
*** srwilkers has quit IRC23:34
*** rwsu has quit IRC23:34
*** rwsu has joined #openstack-kolla23:41
*** salv-orlando has joined #openstack-kolla23:41
*** spsurya has joined #openstack-kolla23:41
spsuryamorning23:42
*** chas has joined #openstack-kolla23:44
*** chas has quit IRC23:48

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!