Friday, 2015-03-13

mgagneso ~2m before connection lost is detected and a new one is established right?00:00
*** j05h has quit IRC00:00
*** vinsh has joined #openstack-operators00:00
*** markvoelker has quit IRC00:03
mgagneon a side note, I'm testing aptly, not too bad at all00:04
klindgrenstill testing that one.... :-)00:04
klindgrenwhat I am noticing in gerneal00:04
klindgrenis that if I get a rabbitmq error00:04
*** vinsh has quit IRC00:04
klindgrenI always see a message following it on the nova compute side that says <something> failed retrying in 1 second00:05
klindgrenthat message that follows is usually timestamped within 5ms or less00:05
mgagnehaha 5ms < 1s :P00:06
klindgrenso going to turn down the brokenness of haproxy to test rabbitmq failover00:06
klindgrenwell is like error trace00:06
mgagnelogs lying to us? how could this be00:07
*** markvoelker has joined #openstack-operators00:08
klindgren5ms later messag saying error, retyring in 1 second00:08
klindgrenand then we get no errors following that00:08
klindgrenso that a vast improvement over tracing forever :-)00:08
klindgrenuntil you restart00:09
mgagne=)00:10
*** reed has joined #openstack-operators00:14
*** Marga_ has quit IRC00:15
*** alop has quit IRC00:16
*** vmtrooper has joined #openstack-operators00:17
mgagneklindgren: btw, I'm investigating the idea of dropping the concept of a long-live repository à-la UCA (which I'm currently using) for specialized repositories built and published by my projects.00:17
mgagneIn fact, I believe dvorak shared a similar idea earlier this week.00:17
mgagneI came to the conclusion that trying to synchronize and version a monolithic repository where multiple projects can publish to is too challenging and prone to race conditions.00:18
klindgrenyes it is00:19
mgagneklindgren: my goal is to have a similar setup to upstream openstack where multiple projects/commits can be done and tested at the same time but with binary packages.00:19
*** mdorman has quit IRC00:19
mgagneklindgren: because I'm more or less heading toward using venv within .deb. I'm not ready for the container madness.00:20
*** vmtrooper has quit IRC00:22
mgagneklindgren: what I would like to do is write about the different packaging and distribution methods with pros/cons. I don't think we will ever be able to come up with ONE magic packaging solution. but if we can educate people on the different methods, they will then be able to see and choose what fits their needs.00:24
klindgrenyea00:25
klindgrenI would like to move to venv's as well00:26
klindgrenor more package wrapped venv's00:26
klindgrenI am also not against doing lxc containers on that as well00:26
mgagneyea, looks like the saner direction to take for now00:26
*** Marga_ has joined #openstack-operators00:27
*** j05h has joined #openstack-operators00:35
klindgrenhttp://lists.openstack.org/pipermail/openstack-dev/2015-March/059034.html00:35
klindgren:-/00:35
harlowja lol00:39
harlowja*i mean sorry, not funny00:40
mgagnejesus00:40
*** jaypipes has quit IRC00:45
*** radez is now known as radez_g0n300:51
*** j05h has quit IRC00:52
klindgrenis it sad that I am not even surprised/phased if thats what it really ends up doing?00:56
mgagneI'm not surprised00:58
*** j05h has joined #openstack-operators00:59
mgagneI'm not sure if the original author is actually using this feature in production or not00:59
mgagneand I'm not sure why it wasn't caught earlier01:00
mgagneI mean, someone should have seen that their instances disappeared each times they "archived" the tables01:01
klindgrenso... side note heartbeating - the patch seemed to do nothing to fix nova/cells01:01
mgagneklindgren: how is it broken?01:02
klindgrenrestarted haproxy and got errors every 2 minutes in the api cell trying to send a rabbit message from the "cells" component of nova01:02
mgagneklindgren: isn't nova-cells using oslo.messaging?01:02
klindgrendeletes seem to have left vm's running in child cells01:03
harlowjaklindgren https://github.com/openstack/nova/blob/master/nova/compute/api.py#L157901:03
harlowjaperhaps that commit in the _delete function is useful...01:03
klindgrenmgagne,  Failed to publish message to topic 'cells': [Errno 1] _ssl.c:1302: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry01:03
harlowjathat probably won't phase u either though, lol01:04
klindgrenis what I would see every 2 minutes01:04
klindgrenfrom oslo_messaging._drivers_impl_rabbit01:04
harlowja'cause a cast to the child cell' might be why, afaik those messages are just thrown out there, which probably == lost01:05
klindgrenoslo_messaging._drivers.impl_rabbit01:05
*** j05h has quit IRC01:15
*** j05h has joined #openstack-operators01:23
*** j05h has quit IRC01:38
*** Marga_ has quit IRC01:53
*** Marga_ has joined #openstack-operators02:03
*** sbfox has joined #openstack-operators02:04
*** vmtrooper has joined #openstack-operators02:06
*** harlowja is now known as harlowja_away02:07
*** vmtrooper has quit IRC02:11
*** j05h has joined #openstack-operators02:18
*** j05h1 has joined #openstack-operators02:18
*** j05h has quit IRC02:22
*** nugget_ has joined #openstack-operators02:28
*** reed has quit IRC02:33
*** sbfox has quit IRC02:33
*** devlaps has quit IRC02:43
*** mpopow_ has quit IRC02:43
*** jaypipes has joined #openstack-operators03:18
*** sbfox has joined #openstack-operators03:21
*** vmtrooper has joined #openstack-operators03:55
*** vmtrooper has quit IRC03:59
*** mpopow_ has joined #openstack-operators04:02
*** mpopow__ has joined #openstack-operators04:03
*** mpopow_ has quit IRC04:07
*** mpopow has joined #openstack-operators04:13
*** mpopow_ has joined #openstack-operators04:13
*** cdelatte has quit IRC04:14
*** mpopow__ has quit IRC04:15
*** Marga_ has quit IRC04:16
*** mpopow has quit IRC04:17
*** delattec has joined #openstack-operators04:21
*** dboik_ has quit IRC04:34
*** Marga_ has joined #openstack-operators04:37
*** sbfox has quit IRC04:49
*** mpopow_ has quit IRC05:00
*** sbfox has joined #openstack-operators05:28
*** SimonChung has joined #openstack-operators05:35
*** delattec has quit IRC05:36
*** SimonChung1 has quit IRC05:37
*** cdelatte has joined #openstack-operators05:37
*** vmtrooper has joined #openstack-operators05:43
*** vmtrooper has quit IRC05:48
*** sbfox has quit IRC05:48
*** sbfox has joined #openstack-operators05:58
*** blair has quit IRC06:03
*** blair has joined #openstack-operators06:04
*** markvoelker has quit IRC06:11
*** markvoelker has joined #openstack-operators06:12
*** markvoelker has quit IRC06:16
*** blair has quit IRC06:18
*** michchap has quit IRC06:38
*** markvoelker has joined #openstack-operators06:42
*** markvoelker has quit IRC06:47
*** michchap has joined #openstack-operators06:58
*** sbfox has quit IRC07:24
*** vmtrooper has joined #openstack-operators07:32
*** vmtrooper has quit IRC07:37
*** markvoelker has joined #openstack-operators07:43
*** markvoelker has quit IRC07:48
*** matrohon has joined #openstack-operators08:21
*** blair has joined #openstack-operators08:35
*** blair has quit IRC08:40
*** markvoelker has joined #openstack-operators08:44
*** markvoelker has quit IRC08:48
*** blair has joined #openstack-operators08:57
*** bvandenh_call_tr has joined #openstack-operators09:00
*** Miouge_ has joined #openstack-operators09:06
*** Miouge has quit IRC09:08
*** Miouge_ is now known as Miouge09:08
*** vmtrooper has joined #openstack-operators09:21
*** vmtrooper has quit IRC09:26
*** markvoelker has joined #openstack-operators09:45
*** markvoelker has quit IRC09:49
*** derekh has joined #openstack-operators09:57
*** cdelatte has quit IRC10:24
*** Miouge has quit IRC10:41
*** markvoelker has joined #openstack-operators10:46
*** markvoelker has quit IRC10:50
*** blair has quit IRC10:52
*** blair has joined #openstack-operators10:52
*** harlowja_away has quit IRC10:59
*** vmtrooper has joined #openstack-operators11:10
*** puck has quit IRC11:12
*** vinsh has joined #openstack-operators11:12
*** blair has quit IRC11:13
*** ggillies_ has quit IRC11:15
*** vmtrooper has quit IRC11:15
*** puck has joined #openstack-operators11:15
*** vinsh has quit IRC11:32
*** Miouge has joined #openstack-operators11:34
*** markvoelker has joined #openstack-operators11:46
*** markvoelker has quit IRC11:51
*** turnerg has joined #openstack-operators11:54
*** pboros has joined #openstack-operators11:54
*** markvoelker has joined #openstack-operators12:07
*** blair has joined #openstack-operators12:14
*** blair has quit IRC12:16
*** blair has joined #openstack-operators12:16
*** vinsh has joined #openstack-operators12:17
*** blair has quit IRC12:21
*** vinsh has quit IRC12:22
*** radez_g0n3 is now known as radez12:22
*** Marga_ has quit IRC12:24
*** ferest has joined #openstack-operators12:32
*** vinsh has joined #openstack-operators12:37
*** vmtrooper has joined #openstack-operators12:59
*** Miouge_ has joined #openstack-operators13:03
*** Miouge has quit IRC13:04
*** Miouge_ is now known as Miouge13:04
*** vmtrooper has quit IRC13:04
*** radez is now known as radez_g0n313:10
*** markvoelker has quit IRC13:16
*** markvoelker has joined #openstack-operators13:17
*** blair has joined #openstack-operators13:17
*** dboik has joined #openstack-operators13:21
*** blair has quit IRC13:22
*** Marga_ has joined #openstack-operators13:23
*** j05h1 has quit IRC13:33
*** j05h has joined #openstack-operators13:47
*** dboik has quit IRC13:50
*** dboik has joined #openstack-operators13:50
*** Miouge has quit IRC13:51
*** Miouge has joined #openstack-operators13:53
*** ferest has quit IRC13:53
*** Miouge has quit IRC14:02
*** radez_g0n3 is now known as radez14:02
*** blinky42 has quit IRC14:02
*** jaypipes is now known as leakypipes14:07
*** Piet has quit IRC14:11
*** Miouge has joined #openstack-operators14:13
*** dboik has quit IRC14:15
*** csoukup has joined #openstack-operators14:16
*** dboik has joined #openstack-operators14:16
*** blair has joined #openstack-operators14:18
*** dboik has quit IRC14:21
*** dboik has joined #openstack-operators14:21
*** blair has quit IRC14:23
*** Piet has joined #openstack-operators14:42
*** alaski is now known as lascii14:48
*** vmtrooper has joined #openstack-operators14:48
*** vmtrooper has quit IRC14:53
*** subscope has quit IRC14:53
*** mdorman has joined #openstack-operators15:01
*** radez is now known as radez_g0n315:02
*** j05h has quit IRC15:03
*** radez_g0n3 is now known as radez15:05
*** turnerg has quit IRC15:10
*** Marga_ has quit IRC15:10
*** blair has joined #openstack-operators15:19
*** blair has quit IRC15:21
*** blair has joined #openstack-operators15:21
*** blair has quit IRC15:27
*** esker has joined #openstack-operators15:31
*** j05h has joined #openstack-operators15:32
*** SimonChung has quit IRC15:33
*** david-lyle_afk is now known as david-lyle15:33
*** DaSchab has joined #openstack-operators15:35
*** reed has joined #openstack-operators15:40
*** Miouge has quit IRC15:40
*** dboik_ has joined #openstack-operators15:43
*** dboik has quit IRC15:43
*** mpopow has joined #openstack-operators15:44
*** esker has quit IRC15:45
*** bvandenh_call_tr has quit IRC15:46
*** resker has joined #openstack-operators15:47
*** Miouge has joined #openstack-operators15:50
*** DaSchab has quit IRC15:51
*** logan2 has joined #openstack-operators15:54
*** alop has joined #openstack-operators15:56
*** Miouge has quit IRC15:59
*** Miouge has joined #openstack-operators16:02
*** Miouge has quit IRC16:04
*** Piet has quit IRC16:05
*** mpopow has quit IRC16:10
*** blair has joined #openstack-operators16:22
*** mpopow has joined #openstack-operators16:22
*** blair has quit IRC16:27
*** SimonChung has joined #openstack-operators16:28
*** sbfox has joined #openstack-operators16:30
*** j05h has quit IRC16:31
*** sbfox has quit IRC16:31
*** sbfox has joined #openstack-operators16:32
*** j05h has joined #openstack-operators16:35
*** dboik_ has quit IRC16:36
*** dboik has joined #openstack-operators16:37
*** vmtrooper has joined #openstack-operators16:37
*** resker has quit IRC16:37
*** vmtrooper has quit IRC16:42
*** mpopow has quit IRC16:44
*** Marga_ has joined #openstack-operators16:46
*** turnerg has joined #openstack-operators16:48
*** matrohon has quit IRC16:49
*** harlowja has joined #openstack-operators16:57
*** sbfox has quit IRC17:12
*** Miouge has joined #openstack-operators17:15
*** Piet has joined #openstack-operators17:19
klindgrenIN testing the rabbitmq stuff I noticed that I managed to get into a state where vm's are deleted from the api cell, but the rpc call blew up and never actually removed stuff in the child cells.  Does anyone have a script to attempt to true up the compute cells to match the api cell in this event?17:21
*** SimonChung1 has joined #openstack-operators17:22
*** SimonChung has quit IRC17:22
*** blair has joined #openstack-operators17:23
harlowjaklindgren i wonder if the nova guys are addressing this in the new cells redesign, might be worth a ML post/email?17:27
harlowjai'm sure others would like to know :-P17:27
*** blair has quit IRC17:28
klindgrenI assume that you are talking about to the dev mailing list?17:29
harlowjaya17:37
harlowjaor operators, or both?17:37
mdormani would bet rax and/or nectar probably have some tooling around cleaning stuff up17:37
*** SimonChung1 has quit IRC17:40
*** SimonChung has joined #openstack-operators17:40
*** Marga_ has quit IRC17:40
*** Marga_ has joined #openstack-operators17:41
*** derekh has quit IRC17:45
*** Marga_ has quit IRC17:46
*** signed8bit has joined #openstack-operators17:48
*** turnerg has quit IRC17:55
*** turnerg has joined #openstack-operators17:57
*** esker has joined #openstack-operators18:04
*** alop has quit IRC18:11
mgagneharlowja: my understanding is that cells v2 will make the api cell a router. If the api cell is now aware of where each instances are located, I see no reason for cells to continue using broadcast instead of unicast message to manage instances.18:13
harlowjamgagne quite possibly, i'm not so sure on the details of cells v218:13
mgagneharlowja: me neither but my understanding is that api cell won't manage instance info in its own database anymore, it's just gonna route RPC calls to the appropriate cell bus and/or access the compute cell database directly.18:14
harlowjapossibly, routing around stuff though i sort think makes inconsistencies/syncs even more problematic18:15
mgagnethe idea is you avoid data discrepancy by not replicating anymore and accessing them directly18:15
mgagneyep18:15
harlowjaat that point why isn't cells v2 just a load-balancer, lol18:17
harlowjai heard they route things around to, lol18:18
harlowja*and/or something completly external to nova18:18
*** sbfox has joined #openstack-operators18:21
*** blair has joined #openstack-operators18:24
mgagneharlowja: some concepts need to be global such as AZ, flavors, aggregates, etc. so you still need some state in the api cell.18:25
*** vmtrooper has joined #openstack-operators18:26
*** turnerg has quit IRC18:27
harlowjasounds like that could also seperate service (likes call it the AZ-flavor-aggregate service) that each nova 'cluster/cell/whatever' calls into to get such info, then u just put loadbalancers/other in front of them18:27
harlowjaand then each nova cluster/cell just gets that information from 'AZ-flavor-aggregate service' and they don't need top level state-thing18:27
harlowjabut to each there own, i just comment here, lol18:29
*** blair has quit IRC18:29
*** Marga_ has joined #openstack-operators18:29
*** vmtrooper has quit IRC18:31
harlowjathen u can scale out  'AZ-flavor-aggregate service'  and nova clusters as u wish (and independently...)18:31
harlowjaand joe-blow service can also use info in 'AZ-flavor-aggregate service'  if it wants18:32
*** esker has quit IRC18:35
*** esker has joined #openstack-operators18:38
mgagneyet an other service to maintain =)18:43
harlowjameh, service oriented architectures have alot of services18:44
harlowja*see name, lol18:44
harlowjahttp://siliconangle.com/furrier/2011/10/12/google-engineer-accidently-shares-his-internal-memo-about-google-platform/ lol18:46
harlowjafunny to read over that :-P18:46
mgagneyep, I also read one about Amazon and how a dictator forced all departments to communicate through API, making it easy later to release AWS which they already used internally18:48
*** esker has quit IRC18:52
mgagneoh, it's the same article lol18:53
mgagneI thought it was a new one18:53
mgagneharlowja: thanks for that one, will share with our team again :P18:54
harlowja:-P18:54
harlowja'6) Anyone who doesn’t do this will be fired.18:54
harlowja7) Thank you; have a nice day!'18:54
harlowjalol18:54
mgagnehehe18:57
mgagneI like this Jeff Bezos =)18:57
harlowjaha18:57
mgagnethe part about his mandate18:57
mgagneIt just makes sense. Even if people don't agree yet, they often agree later when situations and circumstances change18:59
harlowja:-P19:00
harlowjaidk, amazon turned out ok, whether that email helped out or not, idk19:00
harlowjahard to correlate19:00
harlowjabut SOA shouldn't scare people (and more services, meh, it at least worked out for amazon and others)19:00
mgagneI don't care much about Amazon but I do care about the general idea19:01
harlowjasure19:01
mgagnedev: why can't service X accesses service Y's database? why can't you open the port for me?19:03
mgagnenetadmin: because you shouldn't, it's wrong19:03
harlowjaya19:03
mgagneand now you have devs ranting about our netadmin because they do not understand why they can't access the damn database19:04
harlowjaya, send them that link then, lol19:05
klindgrenman - lots of new changes in oslo.config19:05
klindgrenlooks like everything is getting moved to its own config section - vs's stashing everything in "default"19:05
*** mpopow has joined #openstack-operators19:06
*** vinsh has quit IRC19:11
*** vinsh has joined #openstack-operators19:12
*** vinsh has quit IRC19:16
*** blair has joined #openstack-operators19:25
*** esker has joined #openstack-operators19:27
*** esker has quit IRC19:28
*** esker has joined #openstack-operators19:28
*** vinsh has joined #openstack-operators19:29
*** blair has quit IRC19:30
*** esker has quit IRC19:37
*** esker has joined #openstack-operators19:38
*** radez is now known as radez_g0n319:40
*** esker has quit IRC19:41
*** j05h has quit IRC19:41
*** j05h has joined #openstack-operators19:43
*** esker has joined #openstack-operators19:44
*** esker has joined #openstack-operators19:44
*** j05h has quit IRC19:49
*** j05h has joined #openstack-operators19:49
*** alop has joined #openstack-operators19:49
*** esker has quit IRC19:51
mgagneklindgren: again?20:00
mgagneklindgren: what kind of changes? which sections? for what?20:00
klindgrenmgagne, due checking out the rabbitmq heartbeat stuff20:01
*** vinsh has quit IRC20:01
klindgrenit looks like all the rabbitmq config options that were in DEFAULT sections are getting moved to its own oslo_messaging_rabbitmq section20:02
*** esker has joined #openstack-operators20:04
*** esker has quit IRC20:08
*** mpopow has quit IRC20:09
*** esker has joined #openstack-operators20:10
*** esker has quit IRC20:11
*** SimonChung has quit IRC20:14
*** vmtrooper has joined #openstack-operators20:15
*** flaviosr has joined #openstack-operators20:16
*** vmtrooper has quit IRC20:20
*** sbfox has quit IRC20:22
*** blair has joined #openstack-operators20:27
*** blair has quit IRC20:32
*** j05h has quit IRC20:42
*** bvandenh_call_tr has joined #openstack-operators20:42
*** harlowja has quit IRC21:00
*** esker has joined #openstack-operators21:00
*** bvandenh_call_tr has quit IRC21:01
*** harlowja has joined #openstack-operators21:06
*** esker has quit IRC21:10
*** SimonChung has joined #openstack-operators21:14
*** esker has joined #openstack-operators21:15
*** csoukup has quit IRC21:15
*** Piet has quit IRC21:17
*** j05h has joined #openstack-operators21:24
*** j05h1 has joined #openstack-operators21:27
*** blair has joined #openstack-operators21:28
*** j05h has quit IRC21:28
*** flaviosr has quit IRC21:30
*** blair has quit IRC21:33
*** SimonChung has quit IRC21:33
*** SimonChung has joined #openstack-operators21:33
*** SimonChung1 has joined #openstack-operators21:34
*** SimonChung has quit IRC21:34
*** lascii is now known as alaski21:35
*** pboros has quit IRC21:45
*** dboik has quit IRC21:46
*** mpopow has joined #openstack-operators21:58
*** Miouge has quit IRC22:00
*** Piet has joined #openstack-operators22:02
*** vmtrooper has joined #openstack-operators22:04
*** harlowja has quit IRC22:04
*** harlowja has joined #openstack-operators22:06
*** harlowja has quit IRC22:06
*** harlowja has joined #openstack-operators22:06
*** vmtrooper has quit IRC22:08
*** csoukup has joined #openstack-operators22:15
*** j05h1 has quit IRC22:20
*** dvorak is now known as clayton22:28
*** clayton is now known as dvorak22:28
*** blair has joined #openstack-operators22:29
*** blair has quit IRC22:34
*** blair has joined #openstack-operators22:45
*** esker has joined #openstack-operators22:46
*** cdelatte has joined #openstack-operators22:49
*** mpopow has quit IRC22:52
*** mpopow has joined #openstack-operators22:53
*** cdelatte has quit IRC22:53
*** Miouge has joined #openstack-operators22:55
*** markvoelker has quit IRC22:56
*** mpopow has quit IRC22:57
*** j05h has joined #openstack-operators22:58
*** j05h has joined #openstack-operators22:58
*** mpopow has joined #openstack-operators23:06
*** Miouge has quit IRC23:24
*** j05h has quit IRC23:28
*** david-lyle is now known as david-lyle_afk23:29
*** mdorman has quit IRC23:32
*** esker has quit IRC23:40
*** csoukup has quit IRC23:41
*** alop has quit IRC23:50
*** vmtrooper has joined #openstack-operators23:53
*** markvoelker has joined #openstack-operators23:57
*** vmtrooper has quit IRC23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!