| *** rarcea_ has quit IRC | 00:01 | |
| *** lhcheng has quit IRC | 00:15 | |
| *** ducttape_ has quit IRC | 00:15 | |
| *** mdorman has quit IRC | 00:17 | |
| *** zhangjn has joined #openstack-operators | 00:30 | |
| *** Marga_ has joined #openstack-operators | 00:35 | |
| *** david-lyle has joined #openstack-operators | 00:36 | |
| *** david-lyle_ has quit IRC | 00:36 | |
| *** VW_ has joined #openstack-operators | 00:52 | |
| *** VW has quit IRC | 00:54 | |
| *** VW_ has quit IRC | 00:54 | |
| *** VW has joined #openstack-operators | 00:54 | |
| *** signed8bit has quit IRC | 00:55 | |
| *** signed8bit has joined #openstack-operators | 00:57 | |
| *** signed8bit has quit IRC | 00:58 | |
| *** VW has quit IRC | 00:58 | |
| *** VW has joined #openstack-operators | 01:00 | |
| *** gfa has joined #openstack-operators | 01:01 | |
| *** piet has quit IRC | 01:02 | |
| *** piet has joined #openstack-operators | 01:04 | |
| *** eil397 has quit IRC | 01:05 | |
| *** VW has quit IRC | 01:05 | |
| *** VW has joined #openstack-operators | 01:08 | |
| *** VW has quit IRC | 01:13 | |
| *** armax has quit IRC | 01:14 | |
| *** chlong has joined #openstack-operators | 01:21 | |
| *** eil397 has joined #openstack-operators | 01:24 | |
| *** markvoelker has quit IRC | 01:29 | |
| *** eil397 has quit IRC | 01:33 | |
| *** VW has joined #openstack-operators | 01:46 | |
| *** piet has quit IRC | 01:49 | |
| *** VW has quit IRC | 01:52 | |
| *** piet has joined #openstack-operators | 01:57 | |
| *** dims has quit IRC | 01:59 | |
| *** lhcheng has joined #openstack-operators | 02:04 | |
| *** lhcheng has quit IRC | 02:09 | |
| *** zhangjn has quit IRC | 02:11 | |
| *** zhangjn has joined #openstack-operators | 02:16 | |
| *** zhangjn has quit IRC | 02:17 | |
| *** piet has quit IRC | 02:17 | |
| *** zhangjn has joined #openstack-operators | 02:19 | |
| *** zhangjn has quit IRC | 02:24 | |
| *** markvoelker has joined #openstack-operators | 02:29 | |
| *** VW has joined #openstack-operators | 02:30 | |
| *** zhangjn has joined #openstack-operators | 02:30 | |
| *** VW has quit IRC | 02:34 | |
| *** markvoelker has quit IRC | 02:34 | |
| *** zhangjn has quit IRC | 02:38 | |
| *** VW has joined #openstack-operators | 02:45 | |
| *** zhangjn has joined #openstack-operators | 02:52 | |
| *** VW has quit IRC | 02:53 | |
| *** zhangjn has quit IRC | 02:54 | |
| *** csoukup has joined #openstack-operators | 02:55 | |
| *** zhangjn has joined #openstack-operators | 02:58 | |
| *** zhangjn has quit IRC | 03:03 | |
| *** zhangjn has joined #openstack-operators | 03:04 | |
| *** zhangjn has quit IRC | 03:06 | |
| *** zhangjn has joined #openstack-operators | 03:07 | |
| *** zhangjn has quit IRC | 03:10 | |
| *** mriedem_away has left #openstack-operators | 03:11 | |
| *** ducttape_ has joined #openstack-operators | 03:16 | |
| *** zhangjn has joined #openstack-operators | 03:19 | |
| *** ducttape_ has quit IRC | 03:22 | |
| *** VW has joined #openstack-operators | 03:23 | |
| *** VW has quit IRC | 03:24 | |
| *** VW has joined #openstack-operators | 03:24 | |
| *** dims has joined #openstack-operators | 03:45 | |
| *** markvoelker has joined #openstack-operators | 03:46 | |
| *** markvoelker has quit IRC | 03:50 | |
| *** zhangjn has quit IRC | 03:56 | |
| *** armax has joined #openstack-operators | 04:00 | |
| *** ducttape_ has joined #openstack-operators | 04:04 | |
| *** armax has quit IRC | 04:07 | |
| *** armax has joined #openstack-operators | 04:20 | |
| *** dims has quit IRC | 04:20 | |
| *** Guest25637 is now known as saneax_AFK | 04:26 | |
| *** zhangjn has joined #openstack-operators | 04:30 | |
| *** zhangjn has quit IRC | 04:52 | |
| *** piet has joined #openstack-operators | 05:11 | |
| *** ducttape_ has quit IRC | 05:13 | |
| *** saneax_AFK is now known as saneax | 05:26 | |
| *** saneax is now known as Guest46104 | 05:27 | |
| *** piet has quit IRC | 05:30 | |
| *** zhangjn has joined #openstack-operators | 05:32 | |
| *** maishsk has quit IRC | 05:35 | |
| *** maishsk has joined #openstack-operators | 05:36 | |
| *** Marga_ has quit IRC | 05:39 | |
| *** Marga_ has joined #openstack-operators | 05:40 | |
| *** markvoelker has joined #openstack-operators | 05:46 | |
| *** markvoelker has quit IRC | 05:51 | |
| *** VW_ has joined #openstack-operators | 06:01 | |
| *** VW has quit IRC | 06:01 | |
| *** VW_ has quit IRC | 06:23 | |
| *** csoukup has quit IRC | 06:30 | |
| *** Marga_ has quit IRC | 06:32 | |
| *** Marga_ has joined #openstack-operators | 06:33 | |
| *** signed8bit has joined #openstack-operators | 06:42 | |
| *** harshs has joined #openstack-operators | 06:45 | |
| *** markvoelker has joined #openstack-operators | 06:47 | |
| *** markvoelker has quit IRC | 06:52 | |
| *** simon-AS559 has joined #openstack-operators | 06:55 | |
| *** armax has quit IRC | 06:57 | |
| *** harshs_ has joined #openstack-operators | 07:01 | |
| *** harshs has quit IRC | 07:01 | |
| *** harshs_ is now known as harshs | 07:01 | |
| *** rcernin has joined #openstack-operators | 07:01 | |
| *** rcernin_ has joined #openstack-operators | 07:12 | |
| *** rcernin has quit IRC | 07:13 | |
| *** signed8bit has quit IRC | 07:19 | |
| *** rcernin_ has quit IRC | 07:21 | |
| *** VW has joined #openstack-operators | 07:22 | |
| *** rcernin has joined #openstack-operators | 07:22 | |
| *** VW has quit IRC | 07:27 | |
| *** maishsk has joined #openstack-operators | 07:27 | |
| *** sgordon has quit IRC | 07:30 | |
| *** elo has joined #openstack-operators | 07:32 | |
| *** sgordon has joined #openstack-operators | 07:33 | |
| *** elo has quit IRC | 07:33 | |
| *** elo has joined #openstack-operators | 07:40 | |
| *** sgordon` has joined #openstack-operators | 07:41 | |
| *** sgordon has quit IRC | 07:44 | |
| *** sgordon` has quit IRC | 07:48 | |
| *** sgordon has joined #openstack-operators | 07:53 | |
| *** maishsk has quit IRC | 08:00 | |
| *** maishsk has joined #openstack-operators | 08:04 | |
| *** matrohon has joined #openstack-operators | 08:15 | |
| *** chlong has quit IRC | 08:17 | |
| *** harshs has quit IRC | 08:29 | |
| *** lhcheng has joined #openstack-operators | 08:40 | |
| *** piet has joined #openstack-operators | 08:41 | |
| *** markvoelker has joined #openstack-operators | 08:48 | |
| *** maishsk has quit IRC | 08:48 | |
| *** itsuugo has joined #openstack-operators | 08:50 | |
| *** rcernin has quit IRC | 08:50 | |
| *** egonzalez has joined #openstack-operators | 08:52 | |
| *** markvoelker has quit IRC | 08:53 | |
| *** piet has quit IRC | 08:54 | |
| *** dims has joined #openstack-operators | 08:55 | |
| *** maishsk has joined #openstack-operators | 08:58 | |
| *** Guest46104 is now known as saneax_AFK | 09:00 | |
| *** maishsk has quit IRC | 09:03 | |
| *** rcernin has joined #openstack-operators | 09:08 | |
| *** maishsk has joined #openstack-operators | 09:08 | |
| *** dims has quit IRC | 09:13 | |
| *** piet has joined #openstack-operators | 09:18 | |
| *** lhcheng has quit IRC | 09:21 | |
| *** VW has joined #openstack-operators | 09:25 | |
| *** csoukup has joined #openstack-operators | 09:27 | |
| *** VW has quit IRC | 09:29 | |
| *** csoukup has quit IRC | 09:31 | |
| *** piet has quit IRC | 09:34 | |
| *** pilgrimstack has quit IRC | 09:35 | |
| *** lhcheng has joined #openstack-operators | 09:37 | |
| *** vsedelnik has joined #openstack-operators | 09:41 | |
| *** pilgrimstack has joined #openstack-operators | 09:47 | |
| *** admin0 has joined #openstack-operators | 09:54 | |
| *** zhangjn has quit IRC | 10:27 | |
| *** zhangjn has joined #openstack-operators | 10:34 | |
| *** zhangjn has quit IRC | 10:38 | |
| *** zhangjn has joined #openstack-operators | 10:40 | |
| *** zhangjn has quit IRC | 10:43 | |
| *** zhangjn has joined #openstack-operators | 10:48 | |
| *** markvoelker has joined #openstack-operators | 10:49 | |
| *** zhangjn has quit IRC | 10:49 | |
| *** admin0 has quit IRC | 10:50 | |
| *** zhangjn has joined #openstack-operators | 10:50 | |
| *** markvoelker has quit IRC | 10:53 | |
| *** zhangjn has quit IRC | 10:55 | |
| *** vsedelnik has quit IRC | 10:58 | |
| *** admin0 has joined #openstack-operators | 10:59 | |
| *** vsedelnik has joined #openstack-operators | 11:00 | |
| *** vsedelnik has quit IRC | 11:01 | |
| *** lhcheng_ has joined #openstack-operators | 11:06 | |
| *** lhcheng has quit IRC | 11:06 | |
| *** electrofelix has joined #openstack-operators | 11:08 | |
| *** admin0 has quit IRC | 11:12 | |
| *** vsedelnik has joined #openstack-operators | 11:16 | |
| *** VW has joined #openstack-operators | 11:27 | |
| *** simon-AS559 has quit IRC | 11:29 | |
| *** simon-AS559 has joined #openstack-operators | 11:29 | |
| *** VW has quit IRC | 11:32 | |
| *** elo has quit IRC | 11:36 | |
| *** admin0 has joined #openstack-operators | 12:13 | |
| *** markvoelker has joined #openstack-operators | 12:35 | |
| *** lhcheng_ has quit IRC | 12:37 | |
| *** markvoelker has quit IRC | 12:39 | |
| *** rcernin has quit IRC | 12:58 | |
| *** ducttape_ has joined #openstack-operators | 13:06 | |
| *** markvoelker has joined #openstack-operators | 13:09 | |
| *** rarcea_ has joined #openstack-operators | 13:10 | |
| *** chlong has joined #openstack-operators | 13:10 | |
| *** rcernin has joined #openstack-operators | 13:13 | |
| *** lhcheng has joined #openstack-operators | 13:14 | |
| *** rarcea_ has quit IRC | 13:15 | |
| *** rarcea has joined #openstack-operators | 13:16 | |
| *** rarcea has quit IRC | 13:17 | |
| *** ducttape_ has quit IRC | 13:18 | |
| *** VW has joined #openstack-operators | 13:30 | |
| *** VW has quit IRC | 13:34 | |
| *** matrohon has quit IRC | 13:36 | |
| *** admin0 has quit IRC | 13:38 | |
| *** dslev_ has quit IRC | 13:38 | |
| *** piet has joined #openstack-operators | 13:44 | |
| *** admin0 has joined #openstack-operators | 13:46 | |
| *** delatte has joined #openstack-operators | 13:48 | |
| *** dslev has joined #openstack-operators | 13:49 | |
| *** VW has joined #openstack-operators | 13:49 | |
| *** delattec has quit IRC | 13:52 | |
| *** lhcheng has quit IRC | 13:52 | |
| *** VW has quit IRC | 13:55 | |
| *** piet has quit IRC | 13:55 | |
| *** piet has joined #openstack-operators | 13:56 | |
| *** rcernin has quit IRC | 14:06 | |
| *** k_stev has joined #openstack-operators | 14:08 | |
| *** delattec has joined #openstack-operators | 14:09 | |
| *** mdavidson has quit IRC | 14:11 | |
| *** VW has joined #openstack-operators | 14:12 | |
| *** delatte has quit IRC | 14:13 | |
| *** mriedem has joined #openstack-operators | 14:16 | |
| *** zhangjn has joined #openstack-operators | 14:24 | |
| *** admin0 has quit IRC | 14:27 | |
| *** signed8bit has joined #openstack-operators | 14:29 | |
| *** zhangjn has quit IRC | 14:36 | |
| *** regXboi has joined #openstack-operators | 14:38 | |
| *** kencjohnston has joined #openstack-operators | 14:38 | |
| *** kencjohnston has quit IRC | 14:38 | |
| *** kencjohnston has joined #openstack-operators | 14:39 | |
| *** laron has joined #openstack-operators | 14:40 | |
| *** admin0 has joined #openstack-operators | 14:46 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 14:53 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 14:58 | |
| *** derekh has joined #openstack-operators | 15:01 | |
| *** ferest has joined #openstack-operators | 15:07 | |
| *** ferest has quit IRC | 15:08 | |
| *** fubi has joined #openstack-operators | 15:08 | |
| *** ferest has joined #openstack-operators | 15:08 | |
| *** ducttape_ has joined #openstack-operators | 15:08 | |
| *** csoukup has joined #openstack-operators | 15:12 | |
| *** ferest has quit IRC | 15:13 | |
| *** Marga_ has quit IRC | 15:23 | |
| *** Marga_ has joined #openstack-operators | 15:23 | |
| *** mdorman has joined #openstack-operators | 15:25 | |
| *** egonzalez has quit IRC | 15:30 | |
| *** lhcheng has joined #openstack-operators | 15:37 | |
| *** vsedelnik has quit IRC | 15:44 | |
| *** vsedelni_ has joined #openstack-operators | 15:44 | |
| *** piet has quit IRC | 15:45 | |
| *** piet has joined #openstack-operators | 15:45 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 15:49 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 15:49 | |
| *** egonzalez has joined #openstack-operators | 15:55 | |
| *** maishsk has quit IRC | 15:59 | |
| *** k_stev has quit IRC | 16:00 | |
| *** k_stev has joined #openstack-operators | 16:01 | |
| *** armax has joined #openstack-operators | 16:09 | |
| *** VW has quit IRC | 16:20 | |
| *** ducttape_ has quit IRC | 16:21 | |
| *** ducttape_ has joined #openstack-operators | 16:21 | |
| *** VW has joined #openstack-operators | 16:30 | |
| *** vsedelni_ has quit IRC | 16:31 | |
| *** VW has quit IRC | 16:31 | |
| *** vsedelnik has joined #openstack-operators | 16:32 | |
| *** VW has joined #openstack-operators | 16:32 | |
| *** maishsk has joined #openstack-operators | 16:35 | |
| *** maishsk has quit IRC | 16:43 | |
| *** admin0 has quit IRC | 16:43 | |
| *** alejandrito has joined #openstack-operators | 16:44 | |
| *** esker has joined #openstack-operators | 16:48 | |
| *** admin0 has joined #openstack-operators | 16:50 | |
| *** admin0 has quit IRC | 16:53 | |
| *** signed8bit has quit IRC | 16:55 | |
| openstackgerrit | Christian Berendt proposed openstack/osops-tools-generic: Add script get_floating_pools.sh to neutron directory https://review.openstack.org/258631 | 16:55 |
|---|---|---|
| *** vsedelnik has quit IRC | 16:55 | |
| openstackgerrit | JJ Asghar proposed openstack/osops-tools-generic: Cleanup errors in shell scripts identified by bashate https://review.openstack.org/229031 | 16:56 |
| *** vsedelnik has joined #openstack-operators | 16:57 | |
| *** vsedelnik has quit IRC | 16:57 | |
| *** k_stev has quit IRC | 16:58 | |
| *** harshs has joined #openstack-operators | 16:59 | |
| *** k_stev1 has joined #openstack-operators | 17:00 | |
| *** vsedelnik has joined #openstack-operators | 17:01 | |
| *** signed8bit has joined #openstack-operators | 17:02 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 17:02 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 17:02 | |
| *** vsedelnik has quit IRC | 17:02 | |
| *** elo has joined #openstack-operators | 17:10 | |
| *** lhcheng has quit IRC | 17:21 | |
| *** gyee has joined #openstack-operators | 17:26 | |
| *** rockyg has joined #openstack-operators | 17:27 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 17:28 | |
| *** lhcheng has joined #openstack-operators | 17:28 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 17:29 | |
| *** lhcheng has quit IRC | 17:33 | |
| *** egonzalez has quit IRC | 17:35 | |
| *** alop has joined #openstack-operators | 17:37 | |
| *** mriedem is now known as mriedem_lunch | 17:49 | |
| *** logan- has quit IRC | 17:51 | |
| *** harshs has quit IRC | 17:52 | |
| *** gsteinmuller has joined #openstack-operators | 17:54 | |
| *** VW has quit IRC | 17:54 | |
| *** priteau has joined #openstack-operators | 17:56 | |
| *** mriedem_lunch is now known as mriedem | 18:03 | |
| *** eil397 has joined #openstack-operators | 18:06 | |
| *** derekh has quit IRC | 18:06 | |
| *** VW has joined #openstack-operators | 18:13 | |
| *** VW has quit IRC | 18:17 | |
| *** VW has joined #openstack-operators | 18:17 | |
| *** electrofelix has quit IRC | 18:23 | |
| *** simon-AS559 has quit IRC | 18:32 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 18:34 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 18:36 | |
| *** rarcea has joined #openstack-operators | 18:38 | |
| *** vsedelnik has joined #openstack-operators | 18:48 | |
| *** david-lyle_ has joined #openstack-operators | 18:49 | |
| *** david-lyle has quit IRC | 18:50 | |
| *** david-lyle_ is now known as david-lyle | 18:54 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 18:57 | |
| *** ducttape_ has quit IRC | 18:59 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 19:01 | |
| *** simon-AS5592 has joined #openstack-operators | 19:03 | |
| *** simon-AS5592 has quit IRC | 19:03 | |
| *** simon-AS5592 has joined #openstack-operators | 19:03 | |
| *** ducttape_ has joined #openstack-operators | 19:06 | |
| *** kencjohnston has quit IRC | 19:12 | |
| *** signed8bit has quit IRC | 19:14 | |
| *** kencjohnston has joined #openstack-operators | 19:16 | |
| *** david-lyle has quit IRC | 19:17 | |
| *** david-lyle has joined #openstack-operators | 19:20 | |
| *** VW_ has joined #openstack-operators | 19:24 | |
| *** VW has quit IRC | 19:24 | |
| *** david-lyle_ has joined #openstack-operators | 19:24 | |
| *** david-lyle has quit IRC | 19:24 | |
| *** dims has joined #openstack-operators | 19:27 | |
| *** VW_ has quit IRC | 19:28 | |
| *** VW has joined #openstack-operators | 19:29 | |
| *** david-lyle_ has quit IRC | 19:30 | |
| *** ed_b has left #openstack-operators | 19:36 | |
| *** admin0 has joined #openstack-operators | 19:39 | |
| *** signed8bit has joined #openstack-operators | 19:41 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 19:48 | |
| *** david-lyle has joined #openstack-operators | 19:48 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 19:52 | |
| *** kencjohnston has quit IRC | 19:54 | |
| *** piet has quit IRC | 19:57 | |
| *** dslev has quit IRC | 19:58 | |
| *** laron has quit IRC | 20:04 | |
| *** laron has joined #openstack-operators | 20:05 | |
| *** laron has quit IRC | 20:11 | |
| *** piet has joined #openstack-operators | 20:11 | |
| *** laron has joined #openstack-operators | 20:12 | |
| *** dims has quit IRC | 20:18 | |
| *** dims has joined #openstack-operators | 20:19 | |
| rockyg | rbradfor, hey! another ten for me? company politics | 20:20 |
| *** elo has quit IRC | 20:28 | |
| *** elo has joined #openstack-operators | 20:28 | |
| *** vsedelni_ has joined #openstack-operators | 20:29 | |
| *** vsedelnik has quit IRC | 20:29 | |
| rockyg | rbradfor, ping! | 20:30 |
| rbradfor | rockyg, pong | 20:33 |
| rockyg | Hey! Great to have you on logging! | 20:34 |
| rbradfor | thanks. | 20:34 |
| rockyg | Lemme give you some history.... | 20:34 |
| rbradfor | I hope I can help cover the void between operators and developers. | 20:34 |
| rbradfor | well, been working in IT 25+ years. | 20:35 |
| rockyg | I'm not a coder any more, but I come from a QA background and know what it takes to support a SAAS system | 20:35 |
| rockyg | Kewl! then you not only have a clue, but mad skills. | 20:35 |
| rbradfor | started out highly technical, (e.g. Unix Kernel core dump analysis), had a programming backgound | 20:36 |
| rockyg | Part of the problem with the logs is as you've seen, each project is a product to itself in many ways and don't talk to each other | 20:36 |
| rbradfor | worked into architecting large systems, focus on Ingres/Oracle and in past 15 years MySQL. | 20:36 |
| rockyg | Plus, most of the devs have never run any system larger than the devstack on their laptops | 20:36 |
| rockyg | Man, you could be killer on the db side of this, too. | 20:37 |
| *** ed_b has joined #openstack-operators | 20:37 | |
| rockyg | But, I am glad you're on logging right now. | 20:37 |
| rbradfor | Then from almost 20 years ago to my most recent job booked managing operations, first enterprise level, last large new web app side | 20:37 |
| *** mrdoug has joined #openstack-operators | 20:38 | |
| rockyg | The perfect skillset | 20:38 |
| rbradfor | inbetween extensive DB skills, but have always missed being able to code regularly, so this year decided that's what I wnated to focus on. | 20:38 |
| rbradfor | and so now I'm full-time in Openstack development | 20:38 |
| rbradfor | which I hope gives me a benefit, I'm not just a dev with no production experience | 20:39 |
| rbradfor | and I really do not like (e.g. hate) inconsistencies, and inefficiencies. | 20:39 |
| rbradfor | and so back to your comment | 20:39 |
| *** admin0 has quit IRC | 20:39 | |
| rbradfor | Openstack is very developer centric projects, it's not one cohesive product | 20:39 |
| rockyg | I'm a systems engineer that got into coding through modelling (EE background), then into SW process and QA when I didn't have the compiler writing background I needed for a massively parallel system I was hired to develop on | 20:40 |
| rbradfor | I've landed in Oslo, I want to focus on logging initially for this cycle | 20:40 |
| rockyg | Right. But, Oslo is bringing them all closer together. Except for Swift. | 20:40 |
| rockyg | Oslo is the perfect place for you. | 20:40 |
| rbradfor | rockyg, I feel we would overlap in experiences during our lifetimes, based on just this cursory discussion | 20:40 |
| rockyg | can I ask what company is providing your paycheck? | 20:41 |
| rbradfor | HPE | 20:41 |
| rockyg | Ah. Good. They understand the ops side needs. Which I suspect is one reason they are getting out of public cloud. | 20:41 |
| rockyg | But, that's just my assumptions. | 20:42 |
| rbradfor | I'm in the upstream team, rather than my primary focus on Helion | 20:42 |
| rbradfor | that's a long story, and I'm sure even my knowledge is but a small portion of [I assume] a larger decision for financial and customer needs | 20:42 |
| rockyg | OK. So you've noticed two things that I've been tracking. No consistency in formats across projects. No global config | 20:43 |
| rbradfor | yes, those two things I found out when I installed my own cloud at the start of the year | 20:43 |
| rbradfor | hence I clearly saw many projects, not one product | 20:43 |
| rbradfor | I've spent a good amount of time discecting the code, and docs, and non-production examples. | 20:44 |
| rockyg | Ops would love both to be there. My take on the easiest way to get this is to get it into Oslo, then have interns do the grunt work of changes all oveer | 20:44 |
| rbradfor | yes, even oslo.log needs work to be consistent and flexible for projects to build off of | 20:45 |
| rockyg | Yeah. I had to go look at the oslo config code and ask questions to realize the lack of global config :P | 20:45 |
| rockyg | Well, oslo.log is better than it had been. | 20:46 |
| rockyg | It's closer at least. | 20:46 |
| rbradfor | 'n' config files across 'x' /etc project locations alone leads to many mismatches just in groups, let alone heaps of duplication | 20:46 |
| rockyg | Lemme give you a quick recap of what is supposed to be happening in this cycle. | 20:46 |
| rbradfor | that would be great to hear, | 20:46 |
| rbradfor | have only read etherpadds | 20:46 |
| rockyg | Well, sorta. So, two firm things came out of Tokyo: dynamic reconfig on mutable config options will get coded this cycle, with focus on logging as the POC | 20:47 |
| rockyg | 2: we managed to get agreement on how to propagate the requestIDs out to the logs | 20:48 |
| rockyg | Unfortunately, I haven't been able to write up that spec. But it's pretty much using the keyed list field in syslog format to provide: | 20:49 |
| rockyg | Originating request; current; next (or, if it's easier, origin; last; current) | 20:49 |
| rockyg | Which allows to even track forked instances | 20:49 |
| rbradfor | ok | 20:50 |
| rockyg | It took almost the whole time to explain to devs why ops needed this. | 20:50 |
| rockyg | Devs move forward and didn't see how backtracking required different info from the system | 20:50 |
| rbradfor | it's a tree stack trace for [parallel] threads. | 20:50 |
| rockyg | Yup. | 20:51 |
| rbradfor | actually more a branch, you would need multiple branches to build the tree | 20:51 |
| rockyg | Pretty simple, but gets us halfway to being able to fully qualify where a log message happened. | 20:51 |
| rbradfor | ok | 20:51 |
| rbradfor | so, any notes on how you will hold that state for a log message. | 20:52 |
| rockyg | The second part needed is Error Code. There's a spec out that needs a bit of work, but the devs don't want to put that much change into the system. | 20:52 |
| rbradfor | yep I saw that, again I complete get it | 20:52 |
| rockyg | Gotta bunch of etherpads I can pass your way. | 20:52 |
| rbradfor | completely get it | 20:52 |
| rbradfor | so a question for you then | 20:53 |
| rockyg | So, the other thing is that devs don't want to entertain error codes until Ops provide a bunch of bugs that get the levels of the messages more consistent and correct | 20:53 |
| rockyg | Go for it. | 20:54 |
| rbradfor | when your done | 20:54 |
| rockyg | It gets lost on them, if we are changing log levels on specific messages, we may as well be adding codes in at the same time. | 20:54 |
| *** lhcheng has joined #openstack-operators | 20:55 | |
| *** ducttape_ has quit IRC | 20:55 | |
| rockyg | We counted somewhere close to 3k places error codes might be needed. But changing oslo.log to hav th field in the header would mean any time someone is near a message generator, they could insert the code. | 20:56 |
| rockyg | Error code that is. | 20:56 |
| rockyg | Anyway, lots of log stuff we can still work on. | 20:57 |
| rbradfor | whats your take on error codes, do you want to start with CRITICAL/ERROR across projects, or go thru an entire project first. What does an operator want first visibility on parsing logs. | 20:57 |
| *** lhcheng_ has joined #openstack-operators | 20:57 | |
| rbradfor | right, so you want to add the error code to log.record, as an additional argument | 20:57 |
| rockyg | Ops wants ERROR and CRITICAL | 20:57 |
| rockyg | Right. And if it's not in the call, it just gets a place holder (-) | 20:57 |
| rbradfor | makes sense | 20:58 |
| rockyg | Oh, and Ops want placeholders for all the fields so parsing is easier. | 20:58 |
| rbradfor | I find that the varying formats of log strings can make it harder to parse the error code also | 20:58 |
| *** ducttape_ has joined #openstack-operators | 20:58 | |
| rbradfor | unless it's positioned before message | 20:58 |
| rockyg | The ugliest, most unfriendly error message right now is "no valid host" | 20:58 |
| rbradfor | what's the source of that? | 20:59 |
| rockyg | It is nova, and it's generated by the conductor. | 20:59 |
| rockyg | But, the error condition that caused the conductor error is not propagated up | 21:00 |
| rbradfor | so having the request heirarchy just useful info then | 21:00 |
| *** lhcheng has quit IRC | 21:00 | |
| rockyg | Oh, yeah. | 21:00 |
| rockyg | that error means anything from flavor not available to host not available to network not available to..... | 21:01 |
| * rbradfor helps if I -i my grep to find them in the code | 21:01 | |
| rockyg | On one of the etherpads, I have a bunch of the ops hotbutton issues wrt logging. | 21:01 |
| rbradfor | I'd like to get ops take on that so I can be of value to a wider audience first | 21:02 |
| rbradfor | so, let me get back to my question, because (without production access) I see this as something I hope is high on that list | 21:03 |
| rockyg | Great. There will be an ops midcycle in London in mid Feb. But, I can provide you with all the etherpads from the sessions at both midcycles and summits | 21:03 |
| rockyg | Yes. | 21:04 |
| rockyg | ? | 21:05 |
| rbradfor | The logging_context_format_string alone for projects is very inconsistent (just using devstack gets me 4 different formats for 4 projects). The combination of variances of logging elements (user, user_id, user_name and user_identity all referring to a user.). The inability to know what's valid, and not to throw an error on an inconsistent parameter across projects must be so difficult (matching a point about tutorials,examples,docs etc) | 21:05 |
| rockyg | Right. Yes. | 21:05 |
| rbradfor | combined with the multiple code paths to support this, | 21:05 |
| rockyg | Also yes. | 21:05 |
| rbradfor | so at least the user_identity format is a small portion of this https://review.openstack.org/#/c/218139/ | 21:06 |
| rbradfor | and the app agnostic parameters a pre-requisite to navigating towards consistency https://etherpad.openstack.org/p/liberty-app-agnostic-logging-parameters | 21:07 |
| rockyg | I so wanted to slap someone around for that, but didn't have the developer/ops ammo | 21:07 |
| rbradfor | combined of course with 5 other things, of which we have talked about just a few. | 21:07 |
| rockyg | Also, yes | 21:07 |
| *** signed8bit is now known as signed8bit_ZZZzz | 21:07 | |
| rbradfor | ok, so I get from what I've read, and from my experience I can understand your pain when you share it. Helps to know I'm starting on close to the same page | 21:08 |
| rockyg | Oh, very much so. | 21:08 |
| rbradfor | ok, so I'm sure its difficult to prioritize one thing, because that are all equally important. | 21:09 |
| rbradfor | so it some down to two other factors. | 21:09 |
| *** regXboi has quit IRC | 21:09 | |
| rbradfor | what's the least dev pain to develop, and what's the easiest wins for cross project willingness to commit | 21:09 |
| rockyg | Yes. | 21:09 |
| rbradfor | I am happy to do the work, but it's the later of those two which as a new person I have no easy way to understand | 21:10 |
| *** signed8bit_ZZZzz is now known as signed8bit | 21:10 | |
| rockyg | And one last thing: I keep my eyes open for projects doing their own thing with logging. | 21:10 |
| rbradfor | just standardizing the use of oslo.log in two projects (matching documentation) has drawn some not friendly feedback | 21:10 |
| rbradfor | oslo.log setup of configuration that is | 21:11 |
| rockyg | There are some specs in flight right now that I need a dev who understands to make comments on them. | 21:11 |
| rbradfor | happy to read over. | 21:11 |
| rockyg | I can help with some of that. As Ops, we can file bugs. Then it's either fi or reject, but we get infra in on the comments, too. | 21:11 |
| rockyg | One of the things I discussed with Docs at Tokyo was a section on Log config in the config guide, separate from the project log configs | 21:12 |
| rbradfor | re other things, I first started in oslo.context, and one of the first things I noticed was other projects not using it (as desired), this makes any oslo.log context work difficult. | 21:12 |
| rbradfor | https://etherpad.openstack.org/p/Usage-oslo_context.context.RequestContext | 21:13 |
| *** __nick has left #openstack-operators | 21:13 | |
| rockyg | Yeah. Head out to the release channel (not sure the full name) and talk to Doug Hellmann about that. He's got the info you need on how to make it happen. | 21:13 |
| rbradfor | I work with Doug | 21:14 |
| rbradfor | this was just part of my own poking around before I landed in oslo | 21:14 |
| rbradfor | Having also a very strong QA background, one of the first things I did was fix all the code coverage in oslo projects | 21:15 |
| rbradfor | anyway we digress | 21:15 |
| rockyg | So, There is another file to look at that may provide more insight on where to start. It's an infr file. Lemme find it | 21:16 |
| rockyg | it | 21:18 |
| rockyg | s an .erb file.... | 21:18 |
| rbradfor | I've found at least on reference of logging configs -- http://docs.openstack.org/liberty/config-reference/content/list-of-compute-config-options.html#config_table_nova_logging (as it's nova specific) I see your point about a product level section | 21:19 |
| *** esker has quit IRC | 21:20 | |
| rbradfor | rockyg, brb | 21:21 |
| rockyg | Yeah. About a third of the project sections have a log config discussion. And, as you saw in devstack, /etc/nova, /etc/keystone, /etc/cinder etc | 21:23 |
| rockyg | Each has their own config file and a section for logging | 21:23 |
| *** Marga__ has joined #openstack-operators | 21:25 | |
| *** Marga__ has quit IRC | 21:26 | |
| rockyg | Haven't found the .erb file, but here's one that gives you an idea of what infra does: http://git.openstack.org/cgit/openstack-infra/logstash-filters/tree/filters/indexer.conf | 21:27 |
| *** Marga__ has joined #openstack-operators | 21:27 | |
| *** rcernin has joined #openstack-operators | 21:28 | |
| rbradfor | ok, I take this is a config for what you see of the consolidate logstash lines (I looked at this recently, wanted to know how to see the raw openstack log messages) | 21:28 |
| *** Marga_ has quit IRC | 21:29 | |
| rockyg | This looks the same: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/logstash/indexer.conf.erb | 21:29 |
| rockyg | The raw logs in a CI run are all in the zip file for the run. Have you seen one of those yet? | 21:30 |
| rbradfor | I've not delved into the raw zuul logs, but I know where to find them. | 21:31 |
| rbradfor | sort of related (applicable to originating request ids), is there any comments on uuids as request-ids. They seem heavy weight, and have no effective heirarchy built in. | 21:32 |
| rockyg | So, there are about thirty log files (or more) for each CI run. they get zipped up and there is an infra server that let's you get to them | 21:32 |
| rockyg | So, the uuid stuff is sort of being debated right now. | 21:33 |
| rbradfor | I figured that's had a few turns around the table | 21:33 |
| rockyg | The co authors of the cross project spec on them is the place to go | 21:33 |
| rockyg | The issue is that many devs thing uuid needs to be 32 bits. Most ops would be happy with something smaller plus a timeframe, which would keep log message smaller and still give good enough uniqueness | 21:34 |
| rbradfor | concur | 21:35 |
| rockyg | And, the oslo library call for uuid lets you pick the size. | 21:35 |
| rbradfor | uuid is a lazy approach at random uniquess | 21:35 |
| rbradfor | I was not are of that? | 21:35 |
| rockyg | Definitely agree there. Yup. It generates uuid and I think you can go as low as uuid(4) | 21:36 |
| rbradfor | I was thinking of context -- http://git.openstack.org/cgit/openstack/oslo.context/tree/oslo_context/context.py#n32 | 21:36 |
| rbradfor | there maybe a project specific uuid shortner | 21:36 |
| rockyg | that might be what I remember. Yeah. The projects can use oslo, or python package for uuid gen. Would be nice to have them all use the same thing. | 21:37 |
| *** ed_b has left #openstack-operators | 21:38 | |
| rockyg | The size of the requestID can still be debated, but the window is closing. I | 21:38 |
| rockyg | 'd reach out to the two and see what they are willinhg to propose or listen to | 21:39 |
| *** mrdoug has left #openstack-operators | 21:39 | |
| rockyg | Here's the spec: http://specs.openstack.org/openstack/openstack-specs/specs/return-request-id.html | 21:39 |
| rbradfor | well, I don't want to bite off to much for my first full cycle in development, hope to see something make it | 21:40 |
| rbradfor | I'd seen that, I though this was more for http responses then logs. | 21:40 |
| rockyg | Well, the spec I need to write over the holidays is the one that takes those and puts them into log messages. | 21:41 |
| rbradfor | but I see get_previous_request_id() which is at least neede | 21:41 |
| rockyg | One issue which I haven't commentd on in the use of string or unicode. For log, it needs to be ascii | 21:41 |
| rbradfor | I've seen some discussion in oslo on this, especially as it regards backward compatibility. | 21:42 |
| rockyg | If it's in the header. But, I would think a translate would be ok? Or just get them to do ascii? | 21:42 |
| rbradfor | also that get_request_id() function I pointed to changed a few months back. I remember following along. | 21:42 |
| *** ed_b has joined #openstack-operators | 21:42 | |
| rockyg | Oh. Ok. | 21:42 |
| *** admin0 has joined #openstack-operators | 21:42 | |
| rockyg | Where are you based? | 21:43 |
| *** admin0 has quit IRC | 21:43 | |
| rbradfor | Just moved from NYC to CT, so EST | 21:43 |
| rbradfor | and you? | 21:43 |
| rockyg | Ah. Thanks. California. Bay area | 21:43 |
| rbradfor | in my consulting days I basically worked there and lived in NYC. | 21:44 |
| rockyg | So, request id stuff is just comments/steering/reviewing. The coauthors are doing the coding. Just need a plausible developer to do some influencing. Your timing is impeccable | 21:44 |
| *** maishsk has joined #openstack-operators | 21:45 | |
| rockyg | Boy, the two priciest places. | 21:45 |
| rbradfor | try flying across country every week | 21:45 |
| rockyg | Ouch. | 21:46 |
| rbradfor | when you get to know the hosts/hostess and you find out you clock more domestic miles then they do, it's time to change | 21:46 |
| rockyg | Maybe when young. Not now. | 21:46 |
| *** rarcea_ has joined #openstack-operators | 21:46 | |
| rbradfor | yep, way too old now. | 21:46 |
| rbradfor | in the past 4-5 years I've moved mostly to international. I speak at a lot of conferences. | 21:46 |
| rbradfor | more consolidate however. helps if you can do 3-4-5 events in one trip | 21:47 |
| rockyg | Ah. As QA, I seldom went anywhere. Now, at least six times a year, likely more. | 21:47 |
| *** vsedelni_ has quit IRC | 21:47 | |
| rbradfor | ok, well I'd appreciate your direction on specific review/specs etc, there is a lot of info, and as you are likely a great point person until I learn to finetune my searching capabilities | 21:48 |
| rockyg | Although one of the worst travels was PA in January where I would work on trucks, verifying mobile compute HW/SW from about 8pm until 2 or 3am. | 21:48 |
| rockyg | In cold garages. | 21:48 |
| rockyg | A week of that. :P | 21:49 |
| rockyg | I'll send you all the links to the etherpads and a few others when I get to the office. I think getting the oslo side cleaned up first will go a long way. | 21:49 |
| *** rarcea has quit IRC | 21:49 | |
| rockyg | And, the ops midcycle is a good place to start working on getting the bugs files. Also, Ops care about: nova, cinder, keystone, then neutron. | 21:50 |
| rockyg | There is a spec I'll point you at for neutron where the seem to be designing a new logger for specific network log messages | 21:51 |
| rbradfor | good to know the care priorities | 21:51 |
| rockyg | I need dev comments on that and some behind the scenes/or IRC qeustions to make sure they know how to use oslo | 21:52 |
| rockyg | Yeah. More later. I'll send that stuff off in probably about an hour. | 21:52 |
| rockyg | Great to meet you! | 21:52 |
| rbradfor | rockyg, np, thanks for the discussion | 21:52 |
| rbradfor | likewise, I've got a number of etherpads I've read, so it was good to get your ops perspective, and priorities. | 21:53 |
| rbradfor | not that I need more work to do, but on the side I'd like to hear more of operators experiences of MySQL problems at scale, I was a top performance consultant in a not to recent past. Not that it's my focus now, but I've got it all still rolling around | 21:54 |
| *** priteau has quit IRC | 21:58 | |
| *** harlowja has quit IRC | 21:59 | |
| *** priteau has joined #openstack-operators | 22:02 | |
| rockyg | I can pass on some of that, too. But, just connecting with Mike Bayer is a good start. | 22:03 |
| *** harlowja has joined #openstack-operators | 22:04 | |
| rbradfor | good to know | 22:11 |
| *** laron has quit IRC | 22:18 | |
| *** priteau has quit IRC | 22:22 | |
| *** elo has quit IRC | 22:26 | |
| *** elo has joined #openstack-operators | 22:27 | |
| *** simon-AS5592 has quit IRC | 22:30 | |
| *** k_stev1 has quit IRC | 22:31 | |
| *** priteau has joined #openstack-operators | 22:31 | |
| *** piet has quit IRC | 22:34 | |
| openstackgerrit | JJ Asghar proposed openstack/osops-tools-generic: Cleanup errors in shell scripts identified by bashate https://review.openstack.org/229031 | 22:35 |
| *** piet has joined #openstack-operators | 22:39 | |
| mdorman | ok on our new fancy gerrit, how can i get the git fetch link for a particular patchset, like used to display below the files section? | 22:47 |
| *** vsedelnik has joined #openstack-operators | 22:48 | |
| *** rockyg has quit IRC | 22:48 | |
| mdorman | git pull i should say | 22:52 |
| *** vsedelnik has quit IRC | 22:53 | |
| mdorman | TIL: git review -d <review #> | 22:53 |
| mdorman | \o/ | 22:53 |
| jamespd | heh. | 22:54 |
| openstackgerrit | Mike Dorman proposed openstack/osops-tools-generic: Cleanup errors in shell scripts identified by bashate https://review.openstack.org/229031 | 22:55 |
| *** piet has quit IRC | 22:58 | |
| openstackgerrit | Mike Dorman proposed openstack/osops-tools-generic: Cleanup errors in shell scripts identified by bashate https://review.openstack.org/229031 | 22:58 |
| *** harlowja has quit IRC | 23:01 | |
| *** dims has quit IRC | 23:01 | |
| j^2 | mdorman: i'll never get that damn patch in | 23:02 |
| j^2 | ah nice | 23:02 |
| j^2 | i should learn to look at gerrit before commenting | 23:03 |
| *** csoukup has quit IRC | 23:04 | |
| *** zhangjn has joined #openstack-operators | 23:04 | |
| *** alejandrito has quit IRC | 23:08 | |
| *** priteau has quit IRC | 23:11 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 23:18 | |
| *** VW has quit IRC | 23:19 | |
| *** zhangjn has quit IRC | 23:22 | |
| *** mriedem has quit IRC | 23:22 | |
| *** dims has joined #openstack-operators | 23:24 | |
| *** rarcea_ has quit IRC | 23:25 | |
| *** sdake_ has joined #openstack-operators | 23:30 | |
| *** sdake_ has quit IRC | 23:32 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 23:34 | |
| *** vsedelnik has joined #openstack-operators | 23:49 | |
| *** vsedelnik has quit IRC | 23:55 | |
| *** derekh has joined #openstack-operators | 23:56 | |
| *** derekh has quit IRC | 23:56 | |
| *** chlong has quit IRC | 23:57 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 23:57 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!