Wednesday, 2014-12-17

jlkWould be nice if various upstream projects would agree on foo=bar  vs foo = bar  in config file standards.00:00
*** SimonChung1 has joined #openstack-operators00:02
*** SimonChung has quit IRC00:02
*** Marga_ has joined #openstack-operators00:15
*** Marga_ has quit IRC00:15
*** Marga_ has joined #openstack-operators00:15
*** signed8bit is now known as signed8bit_ZZZzz00:20
*** alop has quit IRC00:21
*** emagana has quit IRC00:36
*** Piet has quit IRC00:40
*** emagana has joined #openstack-operators01:36
*** emagana has quit IRC01:48
*** jaypipes has quit IRC01:52
*** SimonChung1 has quit IRC02:01
*** erkules_ has joined #openstack-operators02:05
*** erkules has quit IRC02:07
*** blair has quit IRC02:28
*** emagana has joined #openstack-operators02:43
*** emagana has quit IRC02:48
*** Marga_ has quit IRC02:52
*** Marga_ has joined #openstack-operators02:53
*** blair has joined #openstack-operators02:55
*** yapeng has quit IRC03:00
*** mdorman_ has joined #openstack-operators03:11
*** mdorman has quit IRC03:12
*** mdorman_ is now known as mdorman03:12
*** VW_ has joined #openstack-operators03:12
*** mdorman has quit IRC03:14
*** emagana has joined #openstack-operators03:24
*** SimonChung has joined #openstack-operators03:24
*** harlowja is now known as harlowja_away03:25
*** SimonChung1 has joined #openstack-operators03:26
*** blair has quit IRC03:26
*** blair has joined #openstack-operators03:26
*** SimonChung has quit IRC03:29
*** VW_ has quit IRC03:40
*** VW_ has joined #openstack-operators03:43
*** VW_ has quit IRC03:47
*** blairo has joined #openstack-operators03:57
*** blair has quit IRC03:57
*** VW_ has joined #openstack-operators03:58
*** VW_ has quit IRC04:03
*** VW_ has joined #openstack-operators04:04
*** esker has joined #openstack-operators04:11
*** klindgren_ has quit IRC04:13
*** blair has joined #openstack-operators04:19
*** blairo has quit IRC04:22
*** blair has quit IRC04:31
*** blairo has joined #openstack-operators04:37
*** emagana has quit IRC04:40
*** sanjayu has joined #openstack-operators04:49
*** sanjayu has quit IRC04:56
*** sanjayu has joined #openstack-operators04:56
*** esker has quit IRC04:58
*** Marga_ has quit IRC05:01
*** blairo has quit IRC05:06
*** blair has joined #openstack-operators05:07
*** VW_ has quit IRC05:15
*** VW_ has joined #openstack-operators05:18
*** blair has quit IRC05:30
*** VW_ has quit IRC05:32
*** emagana has joined #openstack-operators05:51
*** emagana has quit IRC05:55
*** bvandenh has joined #openstack-operators06:28
*** avozza is now known as zz_avozza06:32
*** bvandenh has quit IRC06:36
*** reed has quit IRC06:44
*** VW_ has joined #openstack-operators07:15
*** pcaruana has joined #openstack-operators07:16
*** VW_ has quit IRC07:19
*** belmoreira has joined #openstack-operators07:21
*** bvandenh has joined #openstack-operators07:52
*** erkules_ is now known as erkules08:11
*** pboros has joined #openstack-operators08:52
*** blairo has joined #openstack-operators08:54
*** blairo has quit IRC09:02
*** blair has joined #openstack-operators09:15
*** derekh has joined #openstack-operators09:16
*** blairo has joined #openstack-operators09:20
*** zz_avozza is now known as avozza09:23
*** blair has quit IRC09:23
*** avozza is now known as zz_avozza09:59
*** chojny has joined #openstack-operators10:07
*** zz_avozza is now known as avozza10:08
*** chojny has quit IRC10:20
*** chojny has joined #openstack-operators10:28
*** racedo_ has joined #openstack-operators10:36
*** racedo_ has quit IRC10:49
*** chojny has quit IRC11:19
*** chojny has joined #openstack-operators11:35
*** VW_ has joined #openstack-operators11:53
*** VW__ has joined #openstack-operators11:54
*** jemangs has quit IRC11:56
*** VW_ has quit IRC11:57
*** VW__ has quit IRC12:03
*** VW_ has joined #openstack-operators12:04
*** pasquier-s has quit IRC12:05
*** dmsimard_away is now known as dmsimard12:06
*** blairo has quit IRC12:06
*** VW_ has quit IRC12:30
*** VW_ has joined #openstack-operators12:30
*** jemangs has joined #openstack-operators12:42
*** pasquier-s has joined #openstack-operators12:46
*** pasquier-s has quit IRC12:58
*** VW_ has quit IRC13:02
*** VW_ has joined #openstack-operators13:03
*** VW_ has quit IRC13:12
*** VW_ has joined #openstack-operators13:13
*** belmoreira has quit IRC13:13
*** pasquier-s has joined #openstack-operators13:15
*** pasquier-s has quit IRC13:16
*** pasquier-s has joined #openstack-operators13:21
*** pasquier-s has quit IRC13:32
*** pasquier-s has joined #openstack-operators13:37
*** pasquier-s has quit IRC13:39
*** pasquier-s has joined #openstack-operators13:43
*** jaypipes has joined #openstack-operators13:52
*** esker has joined #openstack-operators14:03
*** dmsimard is now known as dmsimard_away14:04
*** DaSchab has joined #openstack-operators14:06
*** signed8bit_ZZZzz is now known as signed8bit14:09
*** pasquier-s has quit IRC14:09
*** pasquier-s has joined #openstack-operators14:09
*** emagana has joined #openstack-operators14:09
*** radez_g0n3 is now known as radez14:11
*** pasquier-s has quit IRC14:14
*** pasquier-s has joined #openstack-operators14:14
*** pasquier-s has quit IRC14:14
*** pasquier-s has joined #openstack-operators14:14
*** pasquier-s has quit IRC14:14
*** pasquier-s has joined #openstack-operators14:15
*** esker has quit IRC14:18
*** esker has joined #openstack-operators14:19
*** chojny has quit IRC14:22
*** sanjayu has quit IRC14:26
*** VW_ has quit IRC14:29
*** VW_ has joined #openstack-operators14:29
*** resker has joined #openstack-operators14:30
*** esker has quit IRC14:30
*** resker has quit IRC14:33
*** pasquier-s has quit IRC14:39
*** pasquier-s has joined #openstack-operators14:43
*** pasquier-s has quit IRC14:44
*** pasquier-s has joined #openstack-operators14:49
*** pasquier-s has quit IRC14:49
*** pcaruana has quit IRC14:51
*** pasquier-s has joined #openstack-operators14:51
*** pasquier-s has quit IRC14:52
*** IPPhoneGuy has joined #openstack-operators14:54
*** esker has joined #openstack-operators14:57
*** resker has joined #openstack-operators14:58
*** signed8bit has quit IRC15:00
*** pasquier-s has joined #openstack-operators15:01
*** esker has quit IRC15:01
*** DaSchab has quit IRC15:02
*** VW_ has quit IRC15:07
*** VW_ has joined #openstack-operators15:13
*** pasquier-s has quit IRC15:14
*** pasquier-s has joined #openstack-operators15:15
*** chojny has joined #openstack-operators15:21
*** signed8bit has joined #openstack-operators15:25
*** pasquier-s has quit IRC15:26
*** VW_ has quit IRC15:30
*** Piet has joined #openstack-operators15:33
*** VW_ has joined #openstack-operators15:33
*** pasquier-s has joined #openstack-operators15:35
*** pasquier-s has quit IRC15:36
*** pasquier-s has joined #openstack-operators15:40
*** belmoreira has joined #openstack-operators15:41
*** pasquier-s has quit IRC15:49
*** mdorman has joined #openstack-operators15:52
*** belmoreira has quit IRC15:55
*** reed has joined #openstack-operators15:59
*** pasquier-s has joined #openstack-operators16:02
*** chojny has quit IRC16:02
*** pasquier-s has quit IRC16:08
PietIf you haven't already, please complete the following the online card sort to improve the information architecture in Horizon.  http://ows.io/os/0v46l86716:13
Piet**Participants will be automatically added to a drawing for one of three HP 7 G2 tablets.**16:13
*** pasquier-s has joined #openstack-operators16:15
*** klindgren has joined #openstack-operators16:20
*** pasquier-s has quit IRC16:29
*** pasquier-s has joined #openstack-operators16:39
*** alop has joined #openstack-operators16:47
*** chojny has joined #openstack-operators16:47
*** chojny has quit IRC16:47
*** pasquier-s has quit IRC16:58
*** Marga_ has joined #openstack-operators17:01
*** harlowja_away is now known as harlowja17:01
*** Piet has quit IRC17:04
*** pasquier-s has joined #openstack-operators17:07
*** emagana has quit IRC17:14
*** Marga_ has quit IRC17:15
*** Marga_ has joined #openstack-operators17:19
*** emagana has joined #openstack-operators17:36
*** bvandenh has quit IRC17:41
*** avozza is now known as zz_avozza17:42
mfischops meeting starting  in 2 mins in #openstack-operators17:58
mfischderp sorry17:58
mfischin #openstack-meeting-417:58
klindgrenhehehe17:59
*** gfa has joined #openstack-operators17:59
*** derekh has quit IRC18:00
harlowjameeting room 4?18:02
harlowjaholy crap there are that many meetings rooms18:02
jlksuch openstack, very projects, much contention18:02
*** resker has quit IRC18:08
*** Marga_ has quit IRC18:13
*** Marga_ has joined #openstack-operators18:14
*** Marga_ has quit IRC18:49
*** Marga_ has joined #openstack-operators18:49
*** vinsh has joined #openstack-operators18:52
*** praneshp has joined #openstack-operators18:53
xavpaiceso was there an outcome on the discussions re a stackforge of tools for operators?18:53
xavpaiceor is that generally what stackforge is anyway?18:54
*** esker has joined #openstack-operators18:54
mdormani think the outcome was essentially just to table it for later18:54
alopwell that was fun18:55
mdormanwhen we originally set up the osops repo, people didn’t want to do stackforge because we just wanted a quick and easy place to put stuff18:55
mdormanwithout having to have all teh gating and other infrastructure that stackforge requires18:55
xavpaicethat makes sense18:55
mdormani don’t remember the full argument/reasons, but that’s how it happened the way it did18:56
xavpaiceand is kinda why we leave stuff on our github too18:56
mdormanyeah18:56
xavpaicebtw, we're re-writing a bunch of those collectd plugins, there's some serious inefficiencies in them18:56
*** GeoffArnold has joined #openstack-operators18:57
mdormanyeah i have a few ideas in mind on those, too18:59
mdormanjust haven’t gotten around to it18:59
*** GeoffArnold has quit IRC19:01
*** Marga_ has quit IRC19:01
*** Marga_ has joined #openstack-operators19:02
xavpaicethanks for the pull request - we're using that option too, and is what triggered us looking at them more clearly19:02
xavpaiceI've re-done the cinder plugin, but one of our devs pointed out something more helpful so I'll have another bash at it before calling it done19:03
* xavpaice gtr19:03
mdormancool19:03
*** VW_ has quit IRC19:25
*** VW_ has joined #openstack-operators19:25
*** fifieldt_ has quit IRC19:30
klindgrenwho was it that sent out the email yeaterday about logs in openstack - that had a spec and the like19:38
klindgrenI have an example of logs that it would be nice if they were more consistent across projects19:39
klindgrenbut I cant seem to find that email19:39
klindgren?19:39
mfischklindgren: want me to forward it?19:43
mfischRocheelle Grober19:43
mfischhas anyone seen where after a live-migration qemu doesn't stop on the orginal host?19:43
mfischI now have 2 qemus running19:44
klindgrenmfisch, please - I did some work to have logstash grab all of the api responses/timing stuff19:44
klindgrenand to parse the differences between the wsgi logs between all the projects is crazy19:44
klindgrenmfisch, I haven't done a live migrate - but is it asking you to accept the migrated vm?19:45
mfischyou dont need that with live19:45
mfischI didnt do the migrate though let me sanity check19:46
*** fifieldt_ has joined #openstack-operators19:46
mfischwe've created a new nova command19:46
mfischnova reproduce19:46
klindgrendoes that go through a different code path?  IF so - care on sharing :-)19:48
mfischall I know is that you dont have to accept it or whatever like a resize or cold migrate19:48
mfischbut its not supposed to leave the old one19:48
klindgrenkk19:48
mfischnow I'm not sure which one is really responding to pings19:48
mfischyou need shared storage to do this19:49
klindgrenwha wha wha...19:49
mfischmost vendors sales guys say YES IT WORKS GREAT19:49
mfischthe tech guys not so much19:49
klindgrenyea personal prefrence... not to run root volumes off shared storage19:50
klindgrenDue to past experience19:50
*** reed has quit IRC20:04
*** reed has joined #openstack-operators20:21
*** Piet has joined #openstack-operators20:28
*** blair has joined #openstack-operators21:00
*** pasquier-s has quit IRC21:04
*** pasquier-s has joined #openstack-operators21:11
*** Marga_ has quit IRC21:21
*** pasquier-s has quit IRC21:25
*** VW__ has joined #openstack-operators21:26
*** VW_ has quit IRC21:26
*** VW__ has quit IRC21:30
*** Rockyg has joined #openstack-operators21:32
jlkI don't suppose anybody is around that has done the libvirt/qemu upgrade on ubuntu?21:38
jlkmy stuff isn't coming back right, even a simple virsh21:38
klindgrenI haven't done it - but I saw some rumblings about it a while back?21:39
jlkyeah I've been following the thread21:39
jlkI'm running into something they weren't though.21:39
klindgrenI can't help - sorry - I am RHEL based :-/21:40
jlkI wish I were21:40
*** blair has quit IRC21:43
klindgrenhonestly - I kinda wish we were ubuntu based on the compute nodes21:44
*** blair has joined #openstack-operators21:44
klindgrenmainly due to newer kernels and versions of libvirt/kvm/qemu21:45
jlkwhat the hell21:48
*** blair has quit IRC21:48
jlkbackign rev, then forwarding rev again worked.21:48
*** blair has joined #openstack-operators21:49
*** blairo has joined #openstack-operators21:53
*** blair has quit IRC21:56
*** pasquier-s has joined #openstack-operators22:01
*** blairo has quit IRC22:05
*** Marga_ has joined #openstack-operators22:08
*** pasquier-s has quit IRC22:10
*** esker has quit IRC22:14
*** pasquier-s has joined #openstack-operators22:18
*** blairo has joined #openstack-operators22:21
*** jaypipes has quit IRC22:26
*** zz_avozza is now known as avozza22:28
*** pasquier-s has quit IRC22:40
*** blair has joined #openstack-operators22:48
*** b1air has joined #openstack-operators22:49
*** blairo has quit IRC22:51
*** pboros has quit IRC22:52
*** blair has quit IRC22:53
*** Rockyg has quit IRC23:03
*** Rockyg has joined #openstack-operators23:03
RockygHey!23:08
*** praneshp has quit IRC23:09
klindgrenhi - so logging23:09
RockygA question for operators:  there is a config option for having logs output in syslog format.  It only works if the message generator uses oslo.log, but it supposedly works.  anyone using it, aware of it, etc?23:10
klindgrenwas parsing the wsgi stuff and noticed that the wsgi logs between all the services is pretty different23:10
RockygYeah.  I'm looking at your email.  Pretty ugly.23:11
klindgrenI know a few people are doing logs -> syslog23:11
RockygDo you need to put that in every projects' config file, or is it a set once globally?23:11
klindgrenits in every project23:12
RockygUGLY23:12
RockygOK.  good to know.23:12
RockygSo, theoretically, if it were set in each of the projects you use, you'd get a consistent format (but only outside the payload, I'd expect)23:13
RockygIs wsgi logging a separate logging library/process from the project(s)?  Do you know?  I'm just digging in on this, so I've lots to learn ridiculously fast.23:14
klindgrenI honestly dont know23:15
RockygBut, I know how bad it is to have inconsistent logs and know that is the case right now for OpenStack.23:15
Rockygthat's ok.  the logging working group has both devs and ops in it.  the devs will know.23:16
klindgrenafaik this issue is more payload logging being different23:16
RockygYeah.  That was what I thought and the meetings at the summit reinforced that idea.  So, we have at least two problems.  message format around the payload and the payloads themselves.23:17
klindgrenoutside of possibly the name of the module being logged23:17
xavpaiceit's the multiline stuff that messes with syslog23:17
RockygOh, yeah.  that would.23:17
RockygI'm getting the log rationalization etherpad back up so I can add these questions to it.  Need to collect up all the questions to make the devs work hard to explain why logging is so messed up23:20
*** blairo has joined #openstack-operators23:20
Rockyghttps://etherpad.openstack.org/p/Log-Rationalization23:21
*** signed8bit has quit IRC23:21
RockygI'll transcribe this stuff into docs, agenda items, etc.23:22
*** b1air has quit IRC23:22
RockygThere is one cross project spec out on log guidelines, and we need to get more operator input on it, but there are a couple more that are out there and not yet identified as crossproject.23:24
RockygThat set is to get requestIDs passed in the log messages so that a glance info log message would have the nova requestID for matching23:25
RockygWe also plan to put a cros project spec out for error codes23:26
Rockygreqid is in payload, but error code is outside23:27
RockygI think mid to longer term is how we get the payloads more standard.  We need to come up with a proposal that the devs can stomach, and get them to enforce it.  Then we can get the old messages fixed during refoactors, bug fixes, etc.23:28
Rockygklindgren: of all the log messages you sent me, is there one that has the info needed but is better/easier to read/parse/script for?23:30
RockygOh, I see your suggestion at the bottom.  Great!23:30
klindgrensorry - got a call - catching up23:30
klindgrenyea - the suggestion at the bottom seems to be fairly close to what most projects were doing23:31
klindgrenso other people can see what we are talking about: https://gist.github.com/krislindgren/586d289e477f02db09d923:32
RockygCool.  this is something we could likely make standard fairly quickly.  Especially if it can use oslo.log to do the wsgi logging.23:33
RockygThere are development  projects that are hitting the same issues.  so this is getting a little more traction.23:35
*** praneshp has joined #openstack-operators23:35
*** pasquier-s has joined #openstack-operators23:35
RockygAnother question I have:  if the log format for a project is set to syslog (or systemd format), do the wsgi log message contents get rearranged?23:37
klindgrenthats a good question - I do not know23:37
*** praneshp_ has joined #openstack-operators23:38
RockygPart of my problem is that there are so many areas where logging can be improved, that I'm having a hard time focusing on high value, low hanging fruit23:38
klindgrenIf someone is using syslog can you pull some example logs from from services of the wsgi stuff and see if its rearanged - the examples I gave above are from icehouse23:38
*** avozza is now known as zz_avozza23:39
*** praneshp has quit IRC23:39
*** praneshp_ is now known as praneshp23:39
RockygThe other thing I notice about the timestamps is that the nested one is a totally different format from the main one.23:40
RockygOn the keystone message, the eventlet.wsgi.server, could you file a bug about it saying eventlet rather than keystone or keystone.eventlet?  Tag it with "log" or "logging" and subcribe rockyg23:42
RockygI want to wade through the bug and identify the log bugs.  I want as many bugs recorded as possible against bad/unclear/useless/wrong log outputs.23:43
RockygI figured out what the [-] means.  It means that either there is no request id, or more likely, request id has not been implemented there yet.23:46
klindgrenright but what about: [1377447c-beb0-4916-8fda-62062ec1c00f 02d775v09ngA fc4d336616d944ee86d04766d099fa3f - - -]23:48
klindgren[reqid user tenant/project ? ? ?]23:49
RockygYeah.  gotta track that one down.  the reqid one was easy because I've been reading the specs/bps/patches23:49
Rockygnova, cinder, glance have them for nova logs23:50
Rockyghttp://specs.openstack.org/openstack/nova-specs/specs/juno/approved/log-request-id-mappings.html23:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!