Thursday, 2017-05-04

*** jamesdenton has quit IRC00:00
*** jamesdenton has joined #openstack-operators00:17
*** manheim has joined #openstack-operators00:25
*** jamesdenton has quit IRC00:28
*** VW has joined #openstack-operators00:28
*** manheim has quit IRC00:29
*** gyee has quit IRC00:29
*** rebase_ has quit IRC00:53
*** rebase_ has joined #openstack-operators00:56
*** bsilverman_13 has quit IRC01:03
*** bsilverman_ has joined #openstack-operators01:03
*** rebase_ has quit IRC01:16
*** rebase_ has joined #openstack-operators01:16
*** Apoorva__ has quit IRC01:21
*** masber has quit IRC01:56
*** marst has joined #openstack-operators02:01
*** rebase_ has quit IRC02:01
*** mriedem has quit IRC02:03
*** rebase_ has joined #openstack-operators02:04
*** VW has quit IRC02:04
*** VW has joined #openstack-operators02:05
*** VW has quit IRC02:09
*** rebase_ has quit IRC02:19
*** yankcrime has quit IRC02:40
*** markvoelker has quit IRC02:40
*** fragatina has joined #openstack-operators02:51
*** masber has joined #openstack-operators02:52
*** fragatina has quit IRC02:53
*** fragatina has joined #openstack-operators02:53
*** fragatin_ has quit IRC02:54
*** fragatin_ has joined #openstack-operators03:01
*** shash____ has joined #openstack-operators03:02
*** Long_yanG has joined #openstack-operators03:03
*** fragatina has quit IRC03:04
*** fragatin_ has quit IRC03:05
*** cemason has joined #openstack-operators03:06
*** shasha_t- has joined #openstack-operators03:06
*** cemason has quit IRC03:10
*** bauruine_ has joined #openstack-operators03:10
*** tdasilva- has quit IRC03:11
*** blairo has quit IRC03:11
*** shasha___ has quit IRC03:11
*** LongyanG has quit IRC03:11
*** dalees has quit IRC03:11
*** bauruine has quit IRC03:11
*** mdavidson has quit IRC03:11
*** cemason has joined #openstack-operators03:11
*** trungnv has quit IRC03:18
*** dalees has joined #openstack-operators03:18
*** mdavidson has joined #openstack-operators03:18
*** trungnv has joined #openstack-operators03:18
*** tdasilva has joined #openstack-operators03:23
*** masber has quit IRC03:24
*** blair has joined #openstack-operators03:28
*** markvoelker has joined #openstack-operators03:44
*** shash____ has quit IRC04:08
*** shasha_t_ has joined #openstack-operators04:09
*** arnewiebalck_ has joined #openstack-operators04:09
*** masber has joined #openstack-operators04:11
*** fragatina has joined #openstack-operators04:39
*** fragatina has quit IRC04:40
*** fragatina has joined #openstack-operators04:40
*** simon-AS559 has joined #openstack-operators04:44
*** slaweq has quit IRC04:46
*** zenirc369 has joined #openstack-operators04:48
*** simon-AS5591 has joined #openstack-operators05:02
*** simon-AS559 has quit IRC05:02
*** marst has quit IRC05:11
*** simon-AS5591 has quit IRC05:12
*** simon-AS559 has joined #openstack-operators05:18
*** trungnv has quit IRC05:38
*** Dinesh_Bhor has quit IRC05:41
*** Dinesh_Bhor has joined #openstack-operators05:41
*** Dinesh_Bhor has quit IRC05:43
*** shasha_t_ has quit IRC05:44
*** yprokule has joined #openstack-operators05:47
*** Dinesh_Bhor has joined #openstack-operators05:48
*** trungnv has joined #openstack-operators05:48
*** trungnv has quit IRC05:51
*** zenirc369 has quit IRC05:56
*** trungnv has joined #openstack-operators06:01
*** Oku_OS-away is now known as Oku_OS06:28
*** simon-AS5591 has joined #openstack-operators06:35
*** simon-AS559 has quit IRC06:35
*** simon-AS5591 has quit IRC06:43
*** rmart04_ has joined #openstack-operators06:44
*** pcaruana has joined #openstack-operators06:45
*** slaweq has joined #openstack-operators06:47
*** slaweq has quit IRC06:52
*** simon-AS559 has joined #openstack-operators06:57
*** zenirc369 has joined #openstack-operators07:00
*** simon-AS5591 has joined #openstack-operators07:05
*** Caterpillar has joined #openstack-operators07:07
*** simon-AS559 has quit IRC07:08
*** manheim has joined #openstack-operators07:19
*** tesseract has joined #openstack-operators07:29
*** shasha_t_ has joined #openstack-operators07:44
*** aojea has joined #openstack-operators07:48
*** shasha_t_ has quit IRC07:49
*** rmart04_ has quit IRC07:49
*** racedo has joined #openstack-operators07:52
*** yprokule_ has joined #openstack-operators07:55
*** simon-AS5591 has quit IRC07:57
*** yprokule has quit IRC07:57
*** yprokule_ is now known as yprokule07:57
*** simon-AS559 has joined #openstack-operators08:01
*** simon-AS559 has quit IRC08:04
*** arnewiebalck_ has quit IRC08:05
*** simon-AS559 has joined #openstack-operators08:07
*** yprokule_ has joined #openstack-operators08:10
*** yprokule has quit IRC08:12
*** yprokule_ is now known as yprokule08:12
*** rarcea has joined #openstack-operators08:14
*** rmart04 has joined #openstack-operators08:15
*** rarcea has quit IRC08:17
*** paramite has joined #openstack-operators08:21
*** derekh has joined #openstack-operators08:36
*** simon-AS5591 has joined #openstack-operators08:44
*** simon-AS559 has quit IRC08:44
*** _nick has joined #openstack-operators08:44
*** Tyrantelf has quit IRC08:45
*** racedo has quit IRC08:45
*** Tyrantelf has joined #openstack-operators08:45
*** _nick is now known as yankcrime08:45
*** slaweq has joined #openstack-operators08:48
*** slaweq has quit IRC08:53
*** arnewiebalck_ has joined #openstack-operators09:13
*** markvoelker has quit IRC09:25
*** rmart04 has quit IRC09:29
*** j^2 has quit IRC09:40
*** j^2 has joined #openstack-operators09:41
*** markvoelker has joined #openstack-operators10:26
*** markvoelker has quit IRC10:33
*** arnewiebalck_ has quit IRC10:42
*** mrhillsman has quit IRC10:48
*** mrhillsman has joined #openstack-operators10:49
*** slaweq has joined #openstack-operators10:49
*** slaweq has quit IRC10:53
*** paramite has quit IRC11:21
*** catintheroof has joined #openstack-operators11:47
*** catintheroof has quit IRC11:51
*** simon-AS559 has joined #openstack-operators12:17
*** simon-AS5592 has joined #openstack-operators12:19
*** simon-AS5591 has quit IRC12:19
*** simon-AS559 has quit IRC12:21
*** liverpooler has joined #openstack-operators12:22
*** slaweq has joined #openstack-operators12:50
*** slaweq has quit IRC12:54
*** paramite has joined #openstack-operators12:59
*** zenirc369 has quit IRC13:02
*** dminer has joined #openstack-operators13:05
*** jamesdenton has joined #openstack-operators13:13
*** bvandenh has joined #openstack-operators13:20
*** BoogPowell_ has joined #openstack-operators13:22
*** markvoelker has joined #openstack-operators13:23
*** mriedem has joined #openstack-operators13:24
*** mriedem has left #openstack-operators13:25
*** mriedem has joined #openstack-operators13:25
*** chlong has joined #openstack-operators13:32
BoogPowell_Seeking any helpl possible!  I have Openstack Newton on Ubuntu 16.04 with 3 controllers, 16 compute, and 16 storage nodes. I'm running linuxbridge in Neutron and everything has been working since last August. Monday I started getting VirtualInterfaceCreateException: Virtual Interface creation failed13:34
BoogPowell_I tried the vif timeout params and it only masks the issue.13:35
BoogPowell_Has anyone seen this before and know what the true fix is?13:35
BoogPowell_nc shows all ports responding from all servers. All services show Enabled/UP from the command line and via Horizon.13:36
BoogPowell_and curl to the endpoints is successful for all services13:36
BoogPowell_I know this is a major deal, but can someone at least tell me the proper startup sequence for Openstack services? I can't find it anywhere in the doc and I know neutron needs to come up in a certain order, although the systemd scripts don't specify anything outside of keystone and MQ.13:44
*** arnewiebalck_ has joined #openstack-operators13:51
*** chlong has quit IRC14:01
*** VW_ has joined #openstack-operators14:01
*** chlong has joined #openstack-operators14:14
*** arnewiebalck_ has quit IRC14:14
BoogPowell_Maybe this will make it easier: I ran rabbitmqctl list_queues and the dhcp_agent.neutron* queues are climbing. What needs to be restarted?14:19
yankcrimeBoogPowell_: sounds like some of your agents haven't started14:19
BoogPowell_They're all running14:19
BoogPowell_and have been restarted14:19
yankcrimeBoogPowell_: do 'openstack network agent list' and see if anything's marked as dead14:19
*** VW_ has quit IRC14:19
yankcrimeand if there are any down, check out the logs from the agents themselves14:19
BoogPowell_Everything is marked Enabled/Up14:19
*** arnewiebalck_ has joined #openstack-operators14:19
*** VW has joined #openstack-operators14:20
*** catintheroof has joined #openstack-operators14:20
yankcrimethere must be a reason as to why your dhcp agents aren't handling queued messages - i'd start looking at their logs first of all14:20
yankcrimeand also neutron-server's logs14:20
BoogPowell_DHCP is probably the effect of another issue, namely this: VirtualInterfaceCreateException: Virtual Interface creation failed14:22
*** catinthe_ has joined #openstack-operators14:23
BoogPowell_I tried the vif_plugging_is_fatal and vif_pluggin_timeout settings, but they seem only to mask the real issue - which is what has me baffled.14:23
yankcrimethat's a pretty common error and can come about for all kinds of reasons14:24
yankcrimedoes the instance actually get scheduled to a compute node?14:24
*** catintheroof has quit IRC14:24
BoogPowell_I think so, but will have to check on that.14:27
BoogPowell_It should show in nova-scheduler.log, right?14:28
BoogPowell_Sorry, I'm going on 50+ hours and I'm brain dead. :-)14:28
*** markvoelker has quit IRC14:31
*** manheim has quit IRC14:33
*** slaweq has joined #openstack-operators14:51
*** tongli has joined #openstack-operators14:52
*** markvoelker has joined #openstack-operators14:53
yankcrimeif it ends up on a compute node and you're seeing that error then look in the nova compute agent logs on your hypervisor14:54
yankcrimethere'll almost certainly be an exception thrown14:54
yankcrimeif it's not making it to a compute node then it's something a bit more fundamental - look in neutron-server's log as you launch an instance as well as nova-api and nova-scheduler's logs14:55
*** slaweq has quit IRC14:55
yankcrimeand yeah, openstack server show - if you've the admin role assigned - should show the hypervisor that instance has been scheduled to14:55
BoogPowell_VirtualInterfaceCreateException: Virtual Interface creation failed14:56
yankcrimei assume you don't have anything like elasticsearch / logstash / kibana installed?14:57
yankcrimeyou might need to bump up logging levels to get more information but i'd expect to see an exception being thrown somewhere14:57
BoogPowell_Failed to allocate the network(s), not rescheduling.14:57
yankcrimetbh elk or something like it is a pre-requisite for operating openstack, there's too many moving parts and too many log files to hunt through14:57
yankcrimereally you need to be able to just search of instance id or request id and have a way of viewing all associated log entries and failure levels14:58
BoogPowell_Thanks.14:58
yankcrimebump up nova's logging and neutron's logging and then try again14:59
BoogPowell_It'll take an act of congress but I will do that as soon as they let me. Thanks for your assistance! I really appreciate your efforts.14:59
yankcrimegood luck15:00
BoogPowell_ It is NOT getting scheduled to a compute node! OS-EXT-SRV-ATTR:hypervisor_hostname  | None15:03
*** cemason has quit IRC15:07
*** shasha_t_ has joined #openstack-operators15:07
*** cemason has joined #openstack-operators15:07
*** Oku_OS is now known as Oku_OS-away15:11
*** shasha_t_ has quit IRC15:11
*** zenirc369 has joined #openstack-operators15:16
*** arnewiebalck_ has quit IRC15:20
*** arnewiebalck_ has joined #openstack-operators15:27
*** yprokule has quit IRC15:29
*** marst has joined #openstack-operators15:31
*** simon-AS559 has joined #openstack-operators15:34
*** simon-AS559 has quit IRC15:35
*** chyka has joined #openstack-operators15:35
*** chlong has quit IRC15:36
*** simon-AS5592 has quit IRC15:38
mriedemdoes anyone know if paypal or ebay are using cellsv1?15:39
*** VW has quit IRC15:43
mriedemor if paypal or ebay ops actually hang out here?15:46
*** gyee has joined #openstack-operators15:50
klindgrenWe hired a manager from paypal.  They used cells v115:53
klindgrenmainly for ironic/non-ironic split.  But they had a few clouds that were nova-cells for compute due to size.15:54
*** Apoorva has joined #openstack-operators15:54
*** dtrainor has quit IRC15:55
*** slaweq has joined #openstack-operators16:00
*** simon-AS559 has joined #openstack-operators16:02
*** simon-AS5591 has joined #openstack-operators16:03
*** vinsh has quit IRC16:05
*** simon-AS559 has quit IRC16:07
*** mrhillsman has quit IRC16:35
*** j^2 has quit IRC16:35
*** pcaruana has quit IRC16:40
*** j^2 has joined #openstack-operators16:44
*** shasha_t_ has joined #openstack-operators16:44
*** mrhillsman has joined #openstack-operators16:46
klindgrenCan someone explain to me on how to use the openstack client with an auth token.  Because the documentation is pure shit and their are no examples.16:48
klindgren$ openstack --os-token <token> --os-auth-type token --os-endpoint <nova-endpoint> --os-auth-url <auth url> server list16:52
klindgrenMissing value auth-url required for auth plugin token16:52
*** simon-AS5591 has quit IRC16:53
*** fragatina has quit IRC16:54
*** derekh has quit IRC17:01
*** hieulq_ has joined #openstack-operators17:05
*** paramite has quit IRC17:06
*** aojea has quit IRC17:10
*** aojea has joined #openstack-operators17:11
*** fragatina has joined #openstack-operators17:13
*** aojea has quit IRC17:15
*** cemason has quit IRC17:19
BoogPowell_@klindgren - do you have an adminrc file...a file containging your OS* variables?17:22
klindgrenI am not attempting to do that.  I got an auth token, and with just using CLI flags trying to use that auto token in openstack client.17:23
klindgrenbascially I am scripting a bunch of stuff.  So instead of doing auth a bunch of times.  I jsut want to auth once and then re-use the token in the later commands17:26
*** tesseract has quit IRC17:30
*** hieulq_ has quit IRC17:37
*** hieulq_ has joined #openstack-operators17:39
*** basilAB has quit IRC17:45
*** gyee has quit IRC17:45
*** vaishali has quit IRC17:46
*** fragatina has quit IRC17:54
*** fragatina has joined #openstack-operators17:54
*** Apoorva has quit IRC17:56
klindgrenso I partially figured it out. --os-token --os-url are needed.  But openstack client blows up on a server show.  Because it attempts to use the os-url for nova in the glance client to get the information about the image thats used17:57
klindgrenand blows up on parsing the v2/tenant_id on the nova api endpoint17:57
*** rebase has joined #openstack-operators18:02
*** VW has joined #openstack-operators18:04
*** slaweq has quit IRC18:17
*** slaweq has joined #openstack-operators18:18
*** harlowja has quit IRC18:19
*** efried has joined #openstack-operators18:20
mriedemklindgren: mdorman: re: the service catalog thread in nova, are you guys using effectively 'internal' or 'admin' endpoint interfaces for inter-service communication, i.e. nova to cinder/glance/neutron?18:21
mriedemthe spec is proposing that if an interface is not configured specifically, nova would iterate over the 3 looking for one that works18:22
mriedemwondering if we start with admin or internal18:22
mriedemhttps://review.openstack.org/#/c/461481/18:22
klindgrencurrently for us - they are all the same.  Planning on redoing it so public is ots own setup of api's18:22
mriedemksa defaults to 'public' if you don't specify an interface18:22
mriedemok18:23
klindgrenI would prefer to default to internal -> admin -> public18:23
mriedemok that's what mordred had proposed too18:23
mriedemefried: ^ there is your answer18:23
efriedbeautiful, thanks y'all.18:23
*** slaweq has quit IRC18:23
*** vaishali has joined #openstack-operators18:32
*** basilAB has joined #openstack-operators18:33
*** simon-AS559 has joined #openstack-operators18:39
*** zenirc369 has quit IRC18:46
*** Apoorva has joined #openstack-operators18:50
*** manheim_ has joined #openstack-operators18:54
*** simon-AS559 has quit IRC18:55
*** manheim_ has quit IRC19:10
*** rebase has quit IRC19:25
*** rebase has joined #openstack-operators19:26
*** aojea has joined #openstack-operators19:27
*** hieulq_ has quit IRC19:28
*** VW has quit IRC19:36
*** shasha_t_ has quit IRC19:39
*** hieulq_ has joined #openstack-operators19:41
*** harlowja has joined #openstack-operators19:48
*** aojea has quit IRC19:50
*** aojea has joined #openstack-operators19:50
*** aojea_ has joined #openstack-operators19:52
*** aojea has quit IRC19:53
*** hieulq_ has quit IRC19:55
*** Apoorva_ has joined #openstack-operators20:05
*** simon-AS559 has joined #openstack-operators20:08
*** Apoorva has quit IRC20:09
*** liverpooler has quit IRC20:09
*** aojea_ has quit IRC20:19
*** slaweq has joined #openstack-operators20:22
*** simon-AS559 has quit IRC20:31
*** VW has joined #openstack-operators20:34
*** shasha_t_ has joined #openstack-operators20:35
*** shasha___ has joined #openstack-operators20:41
*** cnf has quit IRC20:41
*** shasha_t_ has quit IRC20:41
*** verdurin_ is now known as verdurin20:42
*** cnf has joined #openstack-operators20:42
*** simon-AS559 has joined #openstack-operators21:04
*** Apoorva_ has quit IRC21:04
*** Apoorva has joined #openstack-operators21:05
klindgrenanybody know what channel supports openstack-client?21:08
klindgrenits bascially worthless as hell with a token21:08
*** BoogPowell_ has quit IRC21:09
efriedklindgren I think #openstack-sdks21:19
efriedoh, looks like you found it ;-)21:19
klindgrenyea asking my question - because this appears to be full of fail21:19
klindgrenbut thanks21:20
*** marst has quit IRC21:23
*** rebase has quit IRC21:25
*** rebase has joined #openstack-operators21:27
*** marst has joined #openstack-operators21:30
*** dminer has quit IRC21:36
*** Caterpillar has quit IRC21:39
*** manheim has joined #openstack-operators21:43
*** simon-AS559 has quit IRC21:48
*** jbadiapa has quit IRC22:01
*** Apoorva_ has joined #openstack-operators22:05
*** jamesdenton has quit IRC22:08
*** Apoorva has quit IRC22:09
*** jbadiapa has joined #openstack-operators22:15
*** slaweq has quit IRC22:22
*** manheim has quit IRC22:51
*** craigs_ has quit IRC22:52
*** Apoorva has joined #openstack-operators22:56
*** Apoorva_ has quit IRC22:56
*** catinthe_ has quit IRC23:08
*** rebase has quit IRC23:15
*** chyka has quit IRC23:37
*** chyka has joined #openstack-operators23:38
*** rebase has joined #openstack-operators23:41
*** chyka has quit IRC23:43
*** Apoorva has quit IRC23:52
*** rebase has quit IRC23:54
*** Apoorva has joined #openstack-operators23:54
*** rebase has joined #openstack-operators23:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!