| *** sputnik13 has quit IRC | 00:04 | |
| *** dkalleg has quit IRC | 01:05 | |
| *** abitha has quit IRC | 01:08 | |
| *** abitha has joined #openstack-cue | 03:49 | |
| *** abitha has quit IRC | 04:12 | |
| *** miqui has joined #openstack-cue | 13:16 | |
| *** sputnik13 has joined #openstack-cue | 16:03 | |
| *** sputnik13 has quit IRC | 16:45 | |
| *** sputnik13 has joined #openstack-cue | 16:52 | |
| *** sputnik13 has quit IRC | 16:53 | |
| *** sputnik13 has joined #openstack-cue | 16:53 | |
| *** abitha has joined #openstack-cue | 17:11 | |
| *** dkalleg has joined #openstack-cue | 17:25 | |
| *** sputnik13 has quit IRC | 17:36 | |
| *** sputnik13 has joined #openstack-cue | 17:42 | |
| *** esmute has joined #openstack-cue | 18:24 | |
| esmute | vipul, sputnik13: Out of 10 times, integration tests failed 2 times. Both cases the cluster failed to get active and both cases was due to rabbitmq crashing with the same error. Here is the log: http://paste.openstack.org/show/296001/ | 18:26 |
|---|---|---|
| esmute | Something about setting this config: {rabbit_disk_monitor,start_link,[{mem_relative,1.0}]}} | 18:27 |
| esmute | I looked online and saw http://rabbitmq.1065348.n5.nabble.com/disk-free-limit-mem-relative-1-0-td36187.html | 18:27 |
| esmute | Looks like this is an issue for Rabbitmq 3.2.x | 18:27 |
| openstackgerrit | Min Pae proposed stackforge/cue: Fix revert on ERROR/Exception https://review.openstack.org/190081 | 18:27 |
| esmute | Here is the logs for the run: http://logs.openstack.org/73/187273/8/check/gate-cue-integration-dsvm-rabbitmq/a59fee6/logs/devstack-gate-post_test_hook.txt.gz | 18:27 |
| sputnik13 | vipul https://review.openstack.org/#/c/190081/ is ready for review and merge | 18:28 |
| sputnik13 | that's the patch to fix the revert logic | 18:28 |
| sputnik13 | taskflow 0.11.0 got released a little while ago | 18:28 |
| sputnik13 | ah nuts I clobbered my own change during rebase | 18:29 |
| openstackgerrit | Min Pae proposed stackforge/cue: Fix revert on ERROR/Exception https://review.openstack.org/190081 | 18:30 |
| sputnik13 | ok, now it's read | 18:30 |
| sputnik13 | s/read/ready | 18:30 |
| esmute | vipul, sputnik13: i suggest to make this gate voting since the failures are not that frequent and it is due to a rabbitmq bug | 18:32 |
| esmute | which can be fixed if we installed rabbitmq 3.3 | 18:32 |
| sputnik13 | esmute: well, I would kind of agree except we said let's run it a minimum of 20 times | 18:32 |
| sputnik13 | let's finish the remaining 10 or more runs, and then see | 18:33 |
| esmute | sputnik13: Sure... 10 more run to go | 18:33 |
| esmute | i was just giving you guys an update :p | 18:33 |
| sputnik13 | but in general I agree the error you point out has nothing to do with the tempest test itself | 18:33 |
| sputnik13 | and we're probably good with making this voting | 18:34 |
| sputnik13 | but at the same time, we also said let's not set ourselves up for failure by turning on a job we know fails | 18:34 |
| sputnik13 | perhaps we should update the image build process to use the latest version of rabbit, or some pinned version before we make the tempest check voting | 18:35 |
| sputnik13 | so that we at least address this bug | 18:35 |
| vipul | have we tried modifying the rabbitmq config to not set this? | 18:35 |
| esmute | it is set by default | 18:36 |
| esmute | http://www.rabbitmq.com/configure.html | 18:36 |
| esmute | disk_free_limit | 18:36 |
| esmute | perhaps we can set it not to be mem relative | 18:37 |
| sputnik13 | why is it even mem relative in the first place | 18:37 |
| sputnik13 | is it because of swap? | 18:37 |
| sputnik13 | swap is usually its own partition | 18:38 |
| sputnik13 | so that makes no sense | 18:38 |
| vipul | maybe it has to do with durable queues.. | 18:38 |
| esmute | ill put a patch to set this config to an integer number instead | 18:38 |
| sputnik13 | maybe, but I think the real answer is we don't know | 18:38 |
| esmute | and see if that fixes the issues | 18:39 |
| vipul | https://github.com/stackforge/cue/blob/master/cue/templates/install_rabbit.sh.tmpl#L37 | 18:39 |
| esmute | how do we set rabbitmq conf values? | 18:39 |
| vipul | so we are setting this | 18:39 |
| vipul | why not just get rid of the config | 18:39 |
| sputnik13 | we got that from the legacy product I think | 18:39 |
| esmute | ahh you are right... the default is 50MB. http://www.rabbitmq.com/configure.html | 18:40 |
| vipul | yep | 18:40 |
| sputnik13 | I agree let's just remove it for now | 18:40 |
| esmute | yeah.. i didnt know that was being explicitly set | 18:40 |
| sputnik13 | when we dive in deeper with tuning rabbit we'll be able to make better sense of these configs | 18:40 |
| esmute | ill remove it | 18:40 |
| openstackgerrit | Steve Leon proposed stackforge/cue: Removing disk_free_limit from rabbitmq conf https://review.openstack.org/192356 | 18:47 |
| esmute | vipul, sputnik13: ^ | 18:48 |
| vipul | esmute: let's recheck that patch | 18:49 |
| vipul | meaning use that patch to gauge stability | 18:50 |
| *** sputnik13 has quit IRC | 18:52 | |
| esmute | ok | 18:53 |
| *** dkalleg has quit IRC | 19:06 | |
| *** harlowja has quit IRC | 19:25 | |
| *** miqui has quit IRC | 19:33 | |
| *** dkalleg has joined #openstack-cue | 19:58 | |
| openstackgerrit | Abitha Palaniappan proposed stackforge/cue: Fix to return broker values for 'broker add' command https://review.openstack.org/189945 | 20:08 |
| *** harlowja has joined #openstack-cue | 21:13 | |
| openstackgerrit | Abitha Palaniappan proposed stackforge/cue: Fix to return broker values for 'broker add' command https://review.openstack.org/189945 | 21:17 |
| *** davideagnello has quit IRC | 21:21 | |
| *** davideagnello has joined #openstack-cue | 21:21 | |
| *** harlowja has quit IRC | 22:00 | |
| *** harlowja has joined #openstack-cue | 22:02 | |
| *** openstackgerrit has quit IRC | 22:38 | |
| *** openstackgerrit has joined #openstack-cue | 22:39 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!