Friday, 2016-04-08

*** david-lyle has quit IRC00:02
*** david-lyle has joined #openstack-operators00:03
*** SimonChung has quit IRC00:07
*** ducttape_ has joined #openstack-operators00:15
*** shakamunyi has joined #openstack-operators00:18
*** shakamunyi has quit IRC00:23
*** ducttape_ has quit IRC00:34
*** cdelatte has quit IRC00:53
*** ducttape_ has joined #openstack-operators01:05
*** apoorvad has quit IRC01:17
*** shakamunyi has joined #openstack-operators01:18
*** ducttape_ has quit IRC01:22
*** shakamunyi has quit IRC01:23
*** mriedem has quit IRC01:31
*** VW has joined #openstack-operators02:08
*** VW has quit IRC02:15
*** VW has joined #openstack-operators02:15
*** cburgess has quit IRC02:16
*** shakamunyi has joined #openstack-operators02:19
*** cburgess has joined #openstack-operators02:19
*** B_Smith_ has quit IRC02:22
*** shakamunyi has quit IRC02:24
*** B_Smith_ has joined #openstack-operators02:25
*** ducttape_ has joined #openstack-operators02:37
*** SimonChung has joined #openstack-operators02:39
*** SimonChung1 has joined #openstack-operators02:40
*** ducttape_ has quit IRC02:41
*** SimonChung has quit IRC02:43
*** Marga_ has quit IRC02:53
*** beddari has quit IRC03:09
*** shakamunyi has joined #openstack-operators03:18
*** beddari has joined #openstack-operators03:20
*** shakamunyi has quit IRC03:23
*** Marga_ has joined #openstack-operators03:27
*** Marga_ has quit IRC03:31
*** Marga_ has joined #openstack-operators03:45
*** shakamunyi has joined #openstack-operators04:18
*** shakamunyi has quit IRC04:22
*** ducttape_ has joined #openstack-operators04:37
*** shakamunyi has joined #openstack-operators04:40
*** ducttape_ has quit IRC04:41
*** shakamunyi has quit IRC04:45
*** rcernin has joined #openstack-operators04:54
*** VW has quit IRC05:00
*** shakamunyi has joined #openstack-operators05:18
*** shakamunyi has quit IRC05:23
*** markvoelker has quit IRC05:30
*** bvandenh has joined #openstack-operators05:31
*** admin0 has joined #openstack-operators05:45
*** saneax_AFK is now known as saneax05:55
*** admin0 has quit IRC05:56
*** eddima has joined #openstack-operators06:11
*** simon-AS559 has joined #openstack-operators06:14
*** shakamunyi has joined #openstack-operators06:18
*** shakamunyi has quit IRC06:23
*** markvoelker has joined #openstack-operators06:30
*** markvoelker has quit IRC06:36
*** ducttape_ has joined #openstack-operators06:38
*** ducttape_ has quit IRC06:42
*** admin0 has joined #openstack-operators07:06
*** bvandenh has quit IRC07:07
*** zeih has joined #openstack-operators07:17
*** shakamunyi has joined #openstack-operators07:18
*** bvandenh has joined #openstack-operators07:22
*** shakamunyi has quit IRC07:23
*** haukebruno has joined #openstack-operators07:46
*** pcaruana has joined #openstack-operators07:49
*** pilgrimstack has joined #openstack-operators07:55
*** PerfectChaos has quit IRC07:58
*** PerfectChaos has joined #openstack-operators08:01
*** zeih has quit IRC08:03
*** tojuvone_ has joined #openstack-operators08:05
*** tojuvone has quit IRC08:06
*** admin0 has quit IRC08:11
*** PerfectChaos has quit IRC08:12
*** PerfectChaos has joined #openstack-operators08:15
*** admin0 has joined #openstack-operators08:16
*** PerfectChaos has quit IRC08:19
*** PerfectChaos has joined #openstack-operators08:20
*** derekh has joined #openstack-operators08:28
*** markvoelker has joined #openstack-operators08:31
*** openstackgerrit has quit IRC08:33
*** openstackgerrit has joined #openstack-operators08:33
*** markvoelker has quit IRC08:36
*** ducttape_ has joined #openstack-operators08:38
*** ducttape_ has quit IRC08:43
*** saneax is now known as saneax_AFK09:00
*** harlowja has quit IRC09:06
*** tojuvone_ is now known as tojuvone09:19
*** flaviosr has quit IRC09:30
*** subscope has joined #openstack-operators09:38
*** subscope has quit IRC09:44
*** subscope has joined #openstack-operators09:45
*** simon-AS559 has quit IRC09:47
*** fwdit has joined #openstack-operators09:56
*** simon-AS559 has joined #openstack-operators09:59
*** simon-AS559 has quit IRC10:05
*** subscope has quit IRC10:10
*** beddari has quit IRC10:12
*** subscope has joined #openstack-operators10:15
*** beddari has joined #openstack-operators10:19
*** subscope has quit IRC10:20
*** markvoelker has joined #openstack-operators10:32
*** markvoelker has quit IRC10:39
*** ducttape_ has joined #openstack-operators10:39
*** ducttape_ has quit IRC10:43
*** simon-AS559 has joined #openstack-operators11:07
*** simon-AS559 has quit IRC11:09
*** simon-AS559 has joined #openstack-operators11:10
*** simon-AS559 has quit IRC11:15
*** simon-AS559 has joined #openstack-operators11:15
*** saneax_AFK is now known as saneax11:17
*** saneax is now known as saneax_AFK11:18
*** shakamunyi has joined #openstack-operators11:18
*** saneax_AFK is now known as saneax11:18
*** shakamunyi has quit IRC11:23
*** piet_ has joined #openstack-operators11:30
*** saneax is now known as saneax_AFK11:30
*** subscope has joined #openstack-operators11:38
*** cdelatte has joined #openstack-operators11:48
*** subscope has quit IRC11:50
*** Marga_ has quit IRC11:54
*** Marga_ has joined #openstack-operators11:54
*** haukebruno has quit IRC11:55
*** clickboom has joined #openstack-operators11:55
*** derekh has quit IRC11:58
*** zeih has joined #openstack-operators11:59
*** subscope has joined #openstack-operators12:03
*** shakamunyi has joined #openstack-operators12:18
*** markvoelker has joined #openstack-operators12:21
*** shakamunyi has quit IRC12:23
zioprotohello all. I have weird problem in my staging cluster running Kilo. 'nova service-list' shows all nova-compute services are down. however the nova-compute service is running on the hypervisors. Rabbitmq looks fine. I am running out of ideas. Logs of nova-conductor and nova-compute dont show anything useful12:24
*** ducttape_ has joined #openstack-operators12:25
*** Marga_ has quit IRC12:26
*** busterswt has joined #openstack-operators12:30
*** VW has joined #openstack-operators12:35
*** VW has quit IRC12:35
*** fwdit has quit IRC12:35
*** VW has joined #openstack-operators12:36
*** ducttape_ has quit IRC12:38
*** alaski is now known as lascii12:39
*** derekh has joined #openstack-operators12:43
*** eddima has quit IRC12:46
*** eddima has joined #openstack-operators12:46
*** piet_ has quit IRC12:47
*** VW has quit IRC13:09
*** piet has joined #openstack-operators13:12
zioprotohello. I have a problem with nova-conductor not updating the service table in the nova database. I am running Kilo with latest ubuntu packages. I have already verbose=True and debug=True but I cant find the problem. Any hint to troubleshoot ?13:14
*** dronshaw has joined #openstack-operators13:14
toddnnizioproto: does instances start normally?13:14
*** clickboom has quit IRC13:15
zioprototoddnni: of course not, because service compute is down on all hypervisors13:16
zioprotobasically nova-compute is down because the entry 'updated_at' does not get updated13:16
zioprotoQuestion: how can I troubleshoot the communication between nova-compute on the compute node and nova-conductor on the controller node ? I captured network traffic and I see that they exchange data over rabbitmq. But there is anything special I have to check ?13:17
toddnniyou said that rabbitmq is okay, does it mean that rabbitmq list_connections is ok?13:18
toddnnido you have clustered rabbitmq setup?13:19
zioprotoI have a single rabbitmq server13:20
zioprotoha_queues=False13:20
toddnnithen I suspect that connections are blocked, because that is the only problem we have had with single rabbitmq installation13:20
zioprotohow can I check on my rabbitmq server if some connections are blocked ?13:22
*** mriedem has joined #openstack-operators13:22
toddnnirabbitmqctl list_connections13:22
toddnniand if they are you are running out of memory or out of disk space13:22
zioprotoI get a list in 'running' state13:22
toddnniok, they are fine then13:22
zioprotoall connections are marked as running, I have 89 of them at the moment13:23
admin0zioproto: login to your rabbitmq web and see if you have unack messages .. if you have, then you can find out which process is doign that13:23
*** ducttape_ has joined #openstack-operators13:23
*** clickboom has joined #openstack-operators13:24
zioprotoadmin0: on the first page, overview, unacked = 013:24
zioprotoadmin0: I checked the last eight hours13:25
admin0ok .. what does the log say ? what about tcpdump ? does rabbitmq shows sockets towards compute ?13:25
zioprotoyes I sniffed with tcpdump and I saved the pcap to look at it with wireshark13:27
zioprotowhen I restart nova-compute on a compute node, I see that it connects to the rabbitmq server and messages are exchanged13:27
toddnnizioproto: I don't have any more Ideas. We have had problems with kilo and clustered setup because of too old python-kombu package, but in that case there werer errors in nova-compute log13:27
zioprotoI am checking port tcp 5672 only, right ?13:28
admin0what about other operations ? like neutron agent-list ?13:30
*** subscope has quit IRC13:30
zioprotoadmin0: cinder-volume is also down. Also there I could find why13:31
zioprotoneutron agent-list all agents are up and running13:31
*** VW has joined #openstack-operators13:32
zioprotoadmin0: neutron agent-list shows all agents on hypervisor up and running. I have cinder-volume down, and also in this case I cannot understand why :(13:32
*** dims is now known as dimsum__13:41
zioprotoother stuff that I tried already: make sure ntp is running and date/time is consistent.13:42
*** dansmith is now known as superdan13:42
*** piet has quit IRC13:42
*** flaviosr has joined #openstack-operators13:50
*** VW has quit IRC13:55
*** esker has quit IRC13:56
*** subscope has joined #openstack-operators13:56
*** saneax_AFK is now known as saneax14:02
*** ToMiles has joined #openstack-operators14:07
toddnniziproto: when you restarted nova-compute, did it start to work? an did you already tried to restart rabbitmq?14:07
*** ToMiles has quit IRC14:07
zioprototoddnni: I restarted all. I even rebooted completely the servers. The controller and the compute node. I am close to giving up and having a look at it again monday14:09
*** kstev has joined #openstack-operators14:10
toddnniYou have quite a tricky problem there. Have a nice weekend then14:11
admin0toddnni: with verbose and debug if there are issues, it will get reported ..14:12
admin0you said you have all but cannot see the error14:12
zioprotoadmin0: yes, I am running the latest ubuntu packages14:15
zioprotofor Kilo14:15
zioprotofrom the Ubuntu Cloud Archive14:15
*** zeih has quit IRC14:15
*** zeih has joined #openstack-operators14:16
admin0zioproto: things won’ tbreak down all of a sudden .. so in one compute node, maybe stop nova-compute, then backup the logs, clear it out and start the process again .. to see what the logs14:16
zioprotowe upgraded to Kilo two weeks ago. Yesterday this problem started on our staging environment. I am lucky because I can make all the tests I want to fix it. It is 1 controller 1 netowkr node and two compute nodes14:16
zioprotoadmin0: yes monday I will wipe one of the compute nodes and install it from scratch to see if I can reproduce the problem14:17
admin0don’t wipe ..  what if this node had 100 customers14:17
admin0try finding it right there without wiping14:17
*** VW has joined #openstack-operators14:18
zioprotosure, but I have done this today over and over again with no result.14:18
zioprotois there any other nova component involved that I did not check ? or it is al between nova-compute and nova-conductor ?14:19
zioprotoI mean to publish the nova-compute service14:20
*** zeih has quit IRC14:21
yankcrimezioproto: all comms between nova-compute and nova-conductor is rpc over amqp, i.e via rabbitmq14:26
yankcrimeit sounds like nova-compute isn't starting properly or completely for some reason14:26
yankcrimeif you've enabled debug logging then there should be something in nova-compute's logs to go on14:27
yankcrimei have see nova-compute hang indefinitely on startup when there's been a problem with libvirtd14:27
*** piet has joined #openstack-operators14:28
zioprotoyankcrime: thanks for the hint, I will look into that direction. Should I check nova-compute with strace ?14:29
yankcrimedo you see any output at all in nova-compute's logs after you've started it up with debug?14:30
*** ducttape_ has quit IRC14:31
zioprotoyankcrime: give me a sec I am preparing a pastebin14:32
zioprotoyankcrime: http://paste.openstack.org/show/493478/14:34
*** subscope has quit IRC14:34
yankcrimeand nothing after that zioproto?14:34
yankcrimewith debug logging enabled it spews loads of stuff constantly to nova-compute.log14:36
zioprotoyankcrime: sorry paste web site did not work, look here http://pastebin.com/mHfjK3cJ14:36
zioprotoyankcrime: no after ~1300 lines it hangs14:37
zioprotoyankcrime: the pastebin is all the output I get from the log14:37
yankcrimezioproto: does 'virsh list' work on that compute node?14:39
zioprotoyes. I have only 1 instance in shutoff at the moment14:40
zioprotoI can do virsh list and I get the output14:40
yankcrimestill feels like something's up with libvirtd14:43
zioprotoyankcrime: we are close to the solution, I have giving virsh commands and stuff acts funny ! :)14:43
yankcrimeanything in /var/log/libvirt/libvirtd.log ?14:43
yankcrimeyeah14:43
yankcrimei would try restarting libvirtd14:44
*** ducttape_ has joined #openstack-operators14:44
yankcrimeif you're on ubuntu then `restart libvirt-bin` should do the trick14:44
*** piet has quit IRC14:46
zioprotoI found the problem. rbd backend14:46
zioprotovirStorageFileBackendForTypeInternal:1229 : internal error: missing storage backend for network files using rbd protocol14:46
zioprotoso that is why all the hypervisors had the same problem at the same time14:46
admin0\o/14:47
admin0cake for us zioproto14:47
yankcrimeahh, heh14:47
zioprotothe only bad luck is that nova-compute just freeze and nothing comes ever up in the log files !14:47
yankcrimeyeah14:47
zioprotoyou'll have to wait Barcelona for the beer :) I cant make it to come to Austin :(14:47
yankcrimethe only time i've seen that like i say is when there's a problem with libvirtd, it'll just hang indefinitely waiting for some response14:47
yankcrimeso that follows in your case as well i guess14:48
zioprotothanks to all for the help !14:49
*** clickboom has quit IRC14:53
*** esker has joined #openstack-operators14:57
*** esker has quit IRC14:57
*** esker has joined #openstack-operators14:58
*** snecklifter has quit IRC15:02
*** subscope has joined #openstack-operators15:04
*** mdorman has joined #openstack-operators15:06
*** eddima has quit IRC15:16
*** snecklifter has joined #openstack-operators15:19
*** SimonChung1 has quit IRC15:19
*** ducttape_ has quit IRC15:21
*** lhcheng has quit IRC15:25
*** simon-AS559 has quit IRC15:26
*** piet has joined #openstack-operators15:29
*** dronshaw has quit IRC15:30
*** dronshaw has joined #openstack-operators15:31
*** admin0 has quit IRC15:34
*** saneax is now known as saneax_AFK15:37
*** ducttape_ has joined #openstack-operators15:42
*** saneax_AFK is now known as saneax15:49
*** snecklifter has quit IRC15:52
*** ducttape_ has quit IRC15:53
*** regXboi has joined #openstack-operators16:00
*** piet has quit IRC16:05
*** dronshaw has quit IRC16:07
*** SimonChung has joined #openstack-operators16:12
*** dminer has joined #openstack-operators16:13
klindgren^^ thats interesting - will keep that in mind as we start doing rbd backed cinder volumes16:15
*** shakamunyi has joined #openstack-operators16:18
*** mriedem is now known as mriedem_lunch16:20
*** apoorvad has joined #openstack-operators16:22
*** shakamunyi has quit IRC16:23
*** VW has quit IRC16:23
*** dronshaw has joined #openstack-operators16:24
*** _ducttape_ has joined #openstack-operators16:26
*** simon-AS5591 has joined #openstack-operators16:27
*** derekh has quit IRC16:33
*** subscope has quit IRC16:35
*** esker has quit IRC16:38
*** Marga_ has joined #openstack-operators16:43
*** VW has joined #openstack-operators16:44
*** harlowja has joined #openstack-operators16:46
*** pcaruana has quit IRC16:46
*** _ducttape_ has quit IRC16:49
*** rcernin has quit IRC16:55
*** SimonChung1 has joined #openstack-operators16:59
*** SimonChung has quit IRC16:59
*** esker has joined #openstack-operators17:01
*** dronshaw has quit IRC17:05
*** pilgrimstack has quit IRC17:11
*** VW has quit IRC17:27
*** haukebruno has joined #openstack-operators17:29
*** nikhil has quit IRC17:36
*** fawadkhaliq has joined #openstack-operators17:39
*** piet has joined #openstack-operators17:46
*** ducttape_ has joined #openstack-operators17:47
*** mriedem has joined #openstack-operators17:49
*** mriedem_lunch has quit IRC17:49
*** pilgrimstack has joined #openstack-operators17:55
*** Marga_ has quit IRC17:57
*** fawadkhaliq has quit IRC18:03
*** fawadkhaliq has joined #openstack-operators18:03
*** kstev has quit IRC18:04
*** nikhil has joined #openstack-operators18:13
*** pilgrimstack has quit IRC18:15
*** ducttape_ has quit IRC18:18
*** Marga_ has joined #openstack-operators18:26
*** simon-AS5591 has quit IRC18:44
*** piet has quit IRC18:59
*** VW has joined #openstack-operators19:05
wasmumanyone successfully used the new centos-release-openstack-mitaka.rpm from the centos extras repo?  I'm not seeing it just a sanity check.19:10
dmsimardwasmum: it hasn't yet been released19:14
dmsimardWe're planning to release it early next week.19:14
dmsimardThere are instructions on how to test ahead of time if you'd like: https://www.redhat.com/archives/rdo-list/2016-April/msg00048.html19:15
*** piet has joined #openstack-operators19:15
*** kstev has joined #openstack-operators19:22
*** nikhil has quit IRC19:50
*** nikhil has joined #openstack-operators19:51
*** kfjohnson has quit IRC19:55
*** zeus has quit IRC19:58
*** zeus has joined #openstack-operators20:03
*** zeus is now known as Guest8001920:03
*** kfjohnson has joined #openstack-operators20:07
*** fawadkhaliq has quit IRC20:11
*** fawadkhaliq has joined #openstack-operators20:11
*** ducttape_ has joined #openstack-operators20:18
*** clickboom has joined #openstack-operators20:20
*** ducttape_ has quit IRC20:23
*** simon-AS559 has joined #openstack-operators20:26
*** simon-AS5591 has joined #openstack-operators20:27
*** simon-AS559 has quit IRC20:31
*** nikhil has quit IRC20:32
*** simon-AS5591 has quit IRC20:33
wasmumdmsimard: docs newer than the repo - well thats a first ;)20:33
wasmumthanks for the inof20:33
wasmum**info20:33
dmsimardwasmum: It's a lot of work by a lot of people. We'll make sure to shout it everywhere we can when we cut the release :)20:33
wasmumfor sure.  forever grateful for the hard work by everyone!20:34
*** clickboom has quit IRC20:36
*** admin0 has joined #openstack-operators20:39
*** piet has quit IRC20:51
*** haukebruno has quit IRC21:01
*** regXboi has quit IRC21:02
*** cdelatte has quit IRC21:07
*** piet has joined #openstack-operators21:10
*** SimonChung1 has quit IRC21:27
*** SimonChung has joined #openstack-operators21:27
*** SimonChung has quit IRC21:27
*** SimonChung has joined #openstack-operators21:28
*** SimonChung has quit IRC21:29
*** mdorman has quit IRC21:30
*** SimonChung has joined #openstack-operators21:31
*** cfarquhar_ has quit IRC21:33
*** cfarquhar has joined #openstack-operators21:35
*** cfarquhar has quit IRC21:35
*** cfarquhar has joined #openstack-operators21:35
*** VW has quit IRC21:35
*** mriedem has quit IRC21:35
*** ducttape_ has joined #openstack-operators21:42
*** piet has quit IRC21:45
*** piet has joined #openstack-operators21:46
*** kstev has quit IRC21:47
*** nikhil has joined #openstack-operators21:47
*** kstev has joined #openstack-operators21:49
*** kstev has quit IRC21:55
*** ducttape_ has quit IRC21:59
*** piet has quit IRC22:00
*** lascii is now known as alaski22:02
*** piet has joined #openstack-operators22:09
*** lhcheng has joined #openstack-operators22:14
*** lhcheng has quit IRC22:15
*** markvoelker has quit IRC22:16
*** busterswt has quit IRC22:16
*** lhcheng has joined #openstack-operators22:18
*** shakamunyi has joined #openstack-operators22:18
*** shakamunyi has quit IRC22:22
*** Marga_ has quit IRC22:23
*** fawadkhaliq has quit IRC22:25
*** fawadkhaliq has joined #openstack-operators22:25
*** lhcheng has quit IRC22:28
*** Marga__ has joined #openstack-operators22:28
*** VW has joined #openstack-operators22:31
*** Marga__ has quit IRC22:33
*** admin0 has quit IRC22:33
*** Guest80019 is now known as zeus22:34
*** zeus has joined #openstack-operators22:34
*** VW has quit IRC22:36
*** admin0 has joined #openstack-operators22:39
*** dminer has quit IRC22:41
*** apoorvad has quit IRC22:42
*** SimonChung1 has joined #openstack-operators22:52
*** SimonChung has quit IRC22:52
*** SimonChung has joined #openstack-operators22:53
*** SimonChung1 has quit IRC22:53
*** Marga_ has joined #openstack-operators22:56
*** piet has quit IRC23:00
*** markvoelker has joined #openstack-operators23:16
*** shakamunyi has joined #openstack-operators23:18
*** markvoelker has quit IRC23:21
*** shakamunyi has quit IRC23:23
*** Marga_ has quit IRC23:25
*** Marga_ has joined #openstack-operators23:26
*** cburgess has quit IRC23:30
*** cburgess has joined #openstack-operators23:32
*** admin0 has quit IRC23:35
*** lhcheng has joined #openstack-operators23:40
*** SimonChung has quit IRC23:46
*** fawadkhaliq has quit IRC23:48
*** fawadkhaliq has joined #openstack-operators23:48
*** markvoelker has joined #openstack-operators23:52
*** krobzaur_ has joined #openstack-operators23:56
*** piet has joined #openstack-operators23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!