| *** chhavi has joined #openstack-powervm | 00:07 | |
| *** chhavi has quit IRC | 00:11 | |
| *** thorst has joined #openstack-powervm | 00:11 | |
| *** thorst has quit IRC | 00:15 | |
| *** thorst has joined #openstack-powervm | 00:57 | |
| *** thorst has quit IRC | 01:02 | |
| *** esberglu has quit IRC | 01:13 | |
| *** thorst has joined #openstack-powervm | 01:30 | |
| *** thorst has quit IRC | 01:35 | |
| *** esberglu has joined #openstack-powervm | 01:51 | |
| *** esberglu has quit IRC | 01:55 | |
| *** thorst has joined #openstack-powervm | 02:01 | |
| *** thorst has quit IRC | 02:07 | |
| *** AlexeyAbashkin has joined #openstack-powervm | 02:12 | |
| *** thorst has joined #openstack-powervm | 02:15 | |
| *** thorst has quit IRC | 02:16 | |
| *** AlexeyAbashkin has quit IRC | 02:16 | |
| *** zerick_ has joined #openstack-powervm | 02:19 | |
| *** zerick has quit IRC | 02:22 | |
| *** thorst has joined #openstack-powervm | 02:44 | |
| *** thorst has quit IRC | 02:44 | |
| *** thorst has joined #openstack-powervm | 03:25 | |
| *** thorst has quit IRC | 03:29 | |
| *** thorst has joined #openstack-powervm | 04:04 | |
| *** thorst has quit IRC | 04:08 | |
| *** thorst has joined #openstack-powervm | 04:44 | |
| *** thorst has quit IRC | 04:49 | |
| *** thorst has joined #openstack-powervm | 05:20 | |
| *** esberglu has joined #openstack-powervm | 05:23 | |
| *** thorst has quit IRC | 05:25 | |
| *** esberglu has quit IRC | 05:27 | |
| *** chhavi has joined #openstack-powervm | 05:35 | |
| *** thorst has joined #openstack-powervm | 05:54 | |
| *** thorst has quit IRC | 06:00 | |
| *** esberglu has joined #openstack-powervm | 06:03 | |
| *** esberglu has quit IRC | 06:08 | |
| *** thorst has joined #openstack-powervm | 06:29 | |
| *** thorst has quit IRC | 06:38 | |
| *** thorst has joined #openstack-powervm | 07:09 | |
| *** thorst has quit IRC | 07:13 | |
| *** thorst has joined #openstack-powervm | 07:47 | |
| *** thorst has quit IRC | 07:52 | |
| *** AlexeyAbashkin has joined #openstack-powervm | 08:15 | |
| *** thorst has joined #openstack-powervm | 08:25 | |
| *** thorst has quit IRC | 08:29 | |
| *** esberglu has joined #openstack-powervm | 08:48 | |
| *** thorst has joined #openstack-powervm | 08:57 | |
| *** thorst has quit IRC | 09:01 | |
| *** thorst has joined #openstack-powervm | 09:29 | |
| *** thorst has quit IRC | 09:34 | |
| *** thorst has joined #openstack-powervm | 10:07 | |
| *** thorst has quit IRC | 10:11 | |
| *** thorst has joined #openstack-powervm | 10:44 | |
| *** thorst has quit IRC | 10:48 | |
| *** chas has joined #openstack-powervm | 10:59 | |
| *** chas has quit IRC | 10:59 | |
| *** thorst has joined #openstack-powervm | 11:17 | |
| *** thorst has quit IRC | 11:22 | |
| *** esberglu has quit IRC | 11:49 | |
| *** esberglu has joined #openstack-powervm | 11:49 | |
| *** thorst has joined #openstack-powervm | 11:57 | |
| *** thorst has quit IRC | 12:01 | |
| *** thorst has joined #openstack-powervm | 12:11 | |
| *** catmando has joined #openstack-powervm | 12:48 | |
| catmando | hey all | 12:48 |
|---|---|---|
| catmando | i have a question regarding cloud-init | 12:48 |
| catmando | specifically, i am reading https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Wc1c29d23e0fd_4346_b509_f1c00a2099f0/page/Troubleshooting%20when%20a%20VM%20using%20cloud-init%20does%20not%20ping%20or%20boot | 12:49 |
| catmando | not that my cloud-init instance does not boot, but because I am trying to understand how config drive works on powervc | 12:50 |
| catmando | the section that contains "On a VM you can mount the CD drive..." confuses me: what CD drive? | 12:50 |
| efried | catmando CD stands for "Compact Disc". It is an ancient technology whereby you would place a small, flat, plastic donut into a tray or slot in a physical computer, and an actual laser would read data from it. | 12:52 |
| catmando | @efried :| | 12:53 |
| catmando | what I meant was, why is that in the documentation? | 12:53 |
| efried | Technically the drivers on the guest that are reading the data off of the config drive are the same ones from decades ago that read a physical CD drive. The config drive is a virtualized CD. | 12:54 |
| efried | catmando But you're right that we probably shouldn't be calling it "CD" in the documentation, ever. | 12:54 |
| efried | thorst mdrabe Do you know who owns that documentation? | 12:54 |
| efried | esberglu Are you actually there? I did some work on log coloring yesterday and I'm eager to try it out. | 12:55 |
| catmando | wait, so the config drive is automatically set up on the guest when cloud-init runs? | 12:55 |
| efried | catmando Other way around. The config drive is set up automatically (assuming you have the appropriate conf options set, which you do if you're using PowerVC) and cloud-init is run based off of the data on the config drive. | 12:56 |
| catmando | the logs contain many lines of the type: "[ 257.767867] cloud-init[2234]: 2017-12-01 12:40:59,341 - url_helper.py[WARNING]: Calling 'http://172.16.8.254/latest/meta-data/instance-id..." | 12:56 |
| catmando | from my (so far basic) understanding of cloud-init, that is the config drive service it's trying to find | 12:56 |
| efried | Is that a log on the guest or the host? | 12:56 |
| catmando | guest, cloud-init.log | 12:57 |
| efried | I'm not an expert there, I'm afraid. mdrabe may know more. | 12:57 |
| catmando | and i'm trying to understand what, if anything i'm missing. the guests are coming up fine, but that timeout is enourmous and it adds 2-3 minutes to initial boot | 12:57 |
| catmando | . /s/enourmous/enormous | 12:58 |
| efried | catmando Is that a real IP? | 12:59 |
| efried | Is there anything behind it? | 13:00 |
| catmando | that ip is not real, since there's nothing in the docs telling me what service i would need to run there or how to set it up | 13:02 |
| catmando | efried ^ | 13:02 |
| efried | catmando We're going to need to hook you up with someone who actually understands cloud-init, cause I don't. mdrabe may know more, or know who to ask. | 13:05 |
| catmando | efried, sure, i'll hang about, see if anyone chips in | 13:05 |
| efried | catmando I assume there's some way to specialize what gets run in there. | 13:06 |
| catmando | as would i, but there's nothing that i can see in the docs | 13:06 |
| *** efried is now known as fried_rice | 13:22 | |
| *** svenkat has joined #openstack-powervm | 13:22 | |
| fried_rice | svenkat Greetings. Do you know anything about cloud-init? | 13:25 |
| svenkat | Good morning! yes. I can help. If not, i will get answers from my team… i have set up cloudinit before. | 13:27 |
| fried_rice | Great. catmando ^ care to repeat your question for svenkat? | 13:28 |
| fried_rice | svenkat You can see the previous discussion on eavesdrop: http://eavesdrop.openstack.org/irclogs/%23openstack-powervm/%23openstack-powervm.2017-12-01.log.html | 13:29 |
| catmando | svenkat: cloud-init defines datastores, one of which is config drive | 13:29 |
| fried_rice | svenkat Including /me being snotty. | 13:29 |
| svenkat | let me look. yes it defines config drive… | 13:29 |
| catmando | my cloud-init guests are coming up fine, but i don't understand why they're looking (for 2-3 minutes) for a config drive service at address .254 on the network they come up on | 13:30 |
| catmando | nor do i understand the docs that refer to a cd drive and how that (virtual) drive plays into it | 13:30 |
| fried_rice | catmando We ought to fix the docs, but you can just mentally map CD=>config | 13:30 |
| catmando | sure | 13:31 |
| fried_rice | (svenkat, if you know who owns those docs, perhaps we can get 'em fixed?) | 13:31 |
| svenkat | Ist his AIX, ubuntu, rhel, sles? | 13:31 |
| svenkat | sure. I will work with ID team.. good point. | 13:31 |
| catmando | in this initial case, ubuntu 16.04 | 13:31 |
| svenkat | ok… thanks, i will ping our ID team | 13:32 |
| catmando | but | 13:33 |
| catmando | i still don't understand what i'm supposed to do | 13:33 |
| catmando | do i want / need to set up a config drive service at that IP? how do I do that? I understand https://docs.openstack.org/project-install-guide/baremetal/draft/configdrive.html | 13:34 |
| *** svenkat_ has joined #openstack-powervm | 13:35 | |
| *** svenkat has quit IRC | 13:36 | |
| *** svenkat_ is now known as svenkat | 13:36 | |
| catmando | but is that implemented in powervm/vc? there's nothing in the documentation that I can find about it, but I may be looking in the wrong place... | 13:37 |
| svenkat | in your cloud.cfg, you should have an entry like : datasource_list: ['ConfigDrive'] | 13:38 |
| svenkat | do you see it? | 13:38 |
| catmando | indeed | 13:38 |
| *** apearson has joined #openstack-powervm | 13:40 | |
| svenkat | ok… i guess our troubleshooting wiki needs update. let me get it updated. I will work with ID team. | 13:40 |
| *** apearson has quit IRC | 13:44 | |
| *** svenkat has quit IRC | 13:46 | |
| *** svenkat has joined #openstack-powervm | 13:48 | |
| -openstackstatus- NOTICE: gerrit has been restarted to get it back to its normal speed. | 13:52 | |
| *** svenkat_ has joined #openstack-powervm | 13:52 | |
| *** svenkat has quit IRC | 13:53 | |
| *** svenkat_ is now known as svenkat | 13:53 | |
| *** AlexeyAbashkin has quit IRC | 14:02 | |
| *** esberglu has quit IRC | 14:10 | |
| *** esberglu has joined #openstack-powervm | 14:10 | |
| *** catmando has quit IRC | 14:18 | |
| mdrabe | fried_rice: burgerk is the cloud-init guy if that's what you're looking for | 14:45 |
| fried_rice | mdrabe Good to know, thanks. Next time we see catmando... | 14:45 |
| svenkat | i emailed burgerk and Eric, I copied you… | 14:53 |
| fried_rice | oh, yeah, saw that, thanks svenkat | 15:04 |
| fried_rice | esberglu ? | 15:13 |
| esberglu | fried_rice: whats up? | 15:44 |
| *** thorst has quit IRC | 15:45 | |
| fried_rice | esberglu I want to get at that log server and fiddle with the htmlification bits. | 15:45 |
| fried_rice | esberglu First of all, is os-loganalyze up to date there? | 15:45 |
| esberglu | fried_rice: Should be. I think it's in /opt/os-loganalyze | 15:46 |
| esberglu | I can ping you creds | 15:46 |
| fried_rice | esberglu Okay, I have a patch for it. Then we can turn log coloring back on in journalctl. | 15:47 |
| fried_rice | Meh, you were one patch behind. | 15:54 |
| esberglu | fried_rice: Link to the patch? | 15:59 |
| fried_rice | Just posted it: https://review.openstack.org/#/c/524658/ | 15:59 |
| fried_rice | Two different options in there. They produce slightly different results, and I haven't dug into why yet. | 15:59 |
| fried_rice | esberglu Okay, the patch is installed on the log server. Do we need to restart things (like apache) to pick it up? First thing I want to do is see if/how it changes what the regular logs look like. | 16:04 |
| esberglu | fried_rice: You pip installed from /opt/os-loganalyze? Or you just have the patch there | 16:06 |
| fried_rice | esberglu I put the patch there. Assumed it was already pip installed or whatever. Not so? | 16:08 |
| fried_rice | esberglu Oh, no, looks like it's picking up out of the lib | 16:09 |
| esberglu | fried_rice: I'm looking at how that works right now, we just run a puppet manifest that sets up all the logserver stuff | 16:09 |
| fried_rice | Let me pip install the local os-loganalyze. Might just work. | 16:09 |
| esberglu | fried_rice: If not I can probably just rerun the manifest | 16:10 |
| esberglu | As long as that doesn't pull down master over your change | 16:10 |
| fried_rice | esberglu eeek, I'm now getting perm errors... | 16:10 |
| fried_rice | http://184.172.12.213/16/524316/1/check/nova-out-of-tree-pvm/f0d1b76/logs/n-cpu.txt.gz | 16:10 |
| esberglu | fried_rice: hmm | 16:11 |
| fried_rice | I'm sure it's apache being safe. | 16:12 |
| fried_rice | esberglu I can hack the patch into the regular location. | 16:12 |
| fried_rice | esberglu Dangit, I don't know how to un-pip-install this directory. | 16:13 |
| * fried_rice broke the log server. | 16:14 | |
| *** thorst has joined #openstack-powervm | 16:15 | |
| esberglu | fried_rice: We can always just pip uninstall it and rerun the playbook if things get too messed up | 16:18 |
| esberglu | fried_rice: You're trying to switch back to using the version in the lib? | 16:18 |
| fried_rice | esberglu Yeah. Which seems to be gone now. | 16:18 |
| *** thorst has quit IRC | 16:20 | |
| esberglu | fried_rice: /usr/local/lib/python2.7/dist-packages/os-loganalyze/ | 16:20 |
| esberglu | Its there? | 16:20 |
| fried_rice | I just put it there. | 16:20 |
| esberglu | oh | 16:20 |
| fried_rice | But I copied the git repo | 16:20 |
| fried_rice | So it's wrong. | 16:20 |
| fried_rice | esberglu Where are the apache configs? | 16:22 |
| fried_rice | Nyaha, I got it. | 16:22 |
| esberglu | /etc/apache2/sites-enabled/ | 16:22 |
| fried_rice | esberglu Okay, this is weird; it's showing up, but it's showing the old style content, unchanged. | 16:23 |
| fried_rice | The changes I made should at least have altered the background colors and stuff. | 16:23 |
| *** thorst has joined #openstack-powervm | 16:24 | |
| fried_rice | Maybe restart apached... | 16:24 |
| fried_rice | esberglu How do we do ^ | 16:24 |
| esberglu | sudo systemctl restart apache2.service | 16:24 |
| fried_rice | ... | 16:25 |
| esberglu | fried_rice: Heh it's all showing up as 1 line now | 16:25 |
| fried_rice | yeah. | 16:25 |
| esberglu | I'm digging the black background though | 16:25 |
| fried_rice | bolz, I'm gonna have to restart apached for every delta. | 16:27 |
| *** thorst has quit IRC | 16:28 | |
| fried_rice | esberglu Better now. I need to tweak the default colors. | 16:29 |
| esberglu | fried_rice: Sweet. And yeah the info lines are tough to see | 16:31 |
| fried_rice | esberglu Okay, that's nicer. | 16:33 |
| fried_rice | Now | 16:33 |
| fried_rice | Can we turn log colors back on for one run? | 16:33 |
| esberglu | fried_rice: Nope. I could put a run through manually though | 16:34 |
| fried_rice | esberglu Whatever's easiest. I want you to see what it looks like, and then we can decide what to do f'real. | 16:34 |
| fried_rice | esberglu Actually also want to make sure it renders right. I had some trouble with newlines (as you can tell) | 16:35 |
| esberglu | fried_rice: Yeah I can push a manual through, I'll ping you when its done | 16:35 |
| fried_rice | esberglu Thanks! | 16:35 |
| *** thorst has joined #openstack-powervm | 16:41 | |
| *** thorst has quit IRC | 16:50 | |
| *** AlexeyAbashkin has joined #openstack-powervm | 17:13 | |
| fried_rice | esberglu JFYI, the reaction from -nova and -infra is such that this change is not likely to get beyond our CI. | 17:13 |
| fried_rice | Which is fine by me | 17:13 |
| *** fried_rice is now known as fried_rolls | 17:14 | |
| *** AlexeyAbashkin has quit IRC | 17:17 | |
| *** thorst has joined #openstack-powervm | 17:21 | |
| esberglu | fried_rolls: Yeah that's cool with me. Once we know it's working I'll figure out the best way to carry it in the deployment playbooks | 17:21 |
| *** thorst has quit IRC | 17:26 | |
| *** thorst has joined #openstack-powervm | 17:28 | |
| *** thorst has quit IRC | 17:33 | |
| *** thorst has joined #openstack-powervm | 17:34 | |
| *** AlexeyAbashkin has joined #openstack-powervm | 17:37 | |
| *** AlexeyAbashkin has quit IRC | 17:41 | |
| esberglu | fried_rolls: http://184.172.12.213/manual/ansi_logs/ | 17:57 |
| esberglu | Most of the files aren't displaying anything | 17:57 |
| *** chhavi has quit IRC | 18:10 | |
| *** AlexeyAbashkin has joined #openstack-powervm | 20:11 | |
| *** AlexeyAbashkin has quit IRC | 20:15 | |
| *** thorst has quit IRC | 20:19 | |
| *** fried_rolls is now known as fried_rice | 20:24 | |
| fried_rice | esberglu Is there an apache log where exceptions from os-loganalyze might show up? | 20:24 |
| esberglu | fried_rice: /var/log/apache2/... | 20:27 |
| esberglu | logs.powervm.ibm.com_error.log maybe? | 20:29 |
| esberglu | Oh you must be looking at it already | 20:29 |
| fried_rice | yup, looking. | 20:29 |
| fried_rice | Not very helpful. | 20:30 |
| fried_rice | was hoping for a stack trace. | 20:30 |
| fried_rice | Okay, let me do this: I'm just gonna patch it to remove the color codes entirely for now, and play with it more when I have some time. | 20:31 |
| esberglu | fried_rice: Fine by me | 20:31 |
| fried_rice | esberglu I think something else is wrong here - those files really are empty. | 20:34 |
| fried_rice | esberglu Anyway, I patched it up and restarted apache2. So if we can get some real logs in there... | 20:35 |
| esberglu | fried_rice: No color at all now | 20:42 |
| esberglu | fried_rice: nvm back to the way it was before all of this | 20:42 |
| fried_rice | esberglu link? | 20:42 |
| esberglu | http://184.172.12.213/manual/htmlify_logs/logs/n-cpu.txt.gz | 20:42 |
| fried_rice | Wait, what? Now there's no color. | 20:43 |
| fried_rice | Did I miss something? | 20:43 |
| fried_rice | no color until partway down, then all red from then on. | 20:44 |
| fried_rice | That's effed up. | 20:44 |
| fried_rice | at least the escape sequences are gone. | 20:45 |
| fried_rice | wtf | 20:45 |
| fried_rice | ah, I found a bug from earlier. | 20:46 |
| *** thorst has joined #openstack-powervm | 20:50 | |
| *** thorst has quit IRC | 20:55 | |
| fried_rice | I also know wtf is happening with the new shit. The regexes are getting applied before the color codes are coming off, so they're not matching properly. | 21:01 |
| fried_rice | esberglu There, quick and dirty and simple. Now we can re-enable ansi colors in the CI runs. | 21:05 |
| fried_rice | run git diff from /usr/local/lib/python2.7/dist-packages/os-loganalyze | 21:06 |
| *** AlexeyAbashkin has joined #openstack-powervm | 21:11 | |
| esberglu | fried_rice: I still don't get color codes when I download the files | 21:13 |
| fried_rice | I'm not getting anything now. | 21:13 |
| esberglu | fried_rice: When you wget using that URL you get the modified version | 21:13 |
| esberglu | fried_rice: Try now | 21:14 |
| fried_rice | esberglu modified as in html? | 21:14 |
| fried_rice | esberglu Oh. It must be running the filter even when it's not running the htmlview. That's a pain. | 21:15 |
| *** AlexeyAbashkin has quit IRC | 21:15 | |
| esberglu | fried_rice: yep | 21:15 |
| fried_rice | esberglu Sigh. I can still make this work, but it's going to be more annoying. | 21:16 |
| fried_rice | esberglu I'll hack at it some more once I fix this here bug... | 21:16 |
| esberglu | fried_rice: Sounds good | 21:20 |
| *** thorst has joined #openstack-powervm | 21:20 | |
| *** thorst has quit IRC | 21:25 | |
| fried_rice | esberglu THERE. | 21:29 |
| *** thorst has joined #openstack-powervm | 21:31 | |
| *** thorst has quit IRC | 21:36 | |
| fried_rice | esberglu Oh. There was already an attempt to strip control characters. | 21:45 |
| fried_rice | It just wasn't working very well, because the regex was too restrictive. | 21:46 |
| fried_rice | which is actually probably the culprit of my mismatchings | 21:46 |
| esberglu | fried_rice: Where's that at? | 21:46 |
| fried_rice | filter.py line 126 | 21:46 |
| fried_rice | 'cept actually that's switched off. | 21:47 |
| fried_rice | And I don't see anywhere it's switched on. | 21:47 |
| *** thorst has joined #openstack-powervm | 21:49 | |
| *** thorst has quit IRC | 22:03 | |
| *** svenkat has quit IRC | 22:16 | |
| fried_rice | esberglu I implemented themes :) | 22:36 |
| *** thorst has joined #openstack-powervm | 22:40 | |
| *** thorst has quit IRC | 22:45 | |
| esberglu | fried_rice: That's sweet | 22:55 |
| fried_rice | The feedback I got from the nova devs was "I use solarized" and "Colors are a distraction - get rid of them". | 22:55 |
| *** fried_rice is now known as efried | 22:59 | |
| esberglu | Heh | 23:05 |
| esberglu | I wholeheartedly disagree | 23:05 |
| *** thorst has joined #openstack-powervm | 23:14 | |
| *** thorst has quit IRC | 23:19 | |
| *** thorst has joined #openstack-powervm | 23:45 | |
| *** thorst has quit IRC | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!