| *** edmondsw has joined #openstack-powervm | 00:28 | |
| *** edmondsw has quit IRC | 00:33 | |
| *** edmondsw has joined #openstack-powervm | 02:16 | |
| *** edmondsw has quit IRC | 02:21 | |
| *** k0da has joined #openstack-powervm | 04:53 | |
| *** edmondsw has joined #openstack-powervm | 05:53 | |
| *** k0da has quit IRC | 05:57 | |
| *** edmondsw has quit IRC | 05:58 | |
| *** Guest1988 has joined #openstack-powervm | 07:11 | |
| *** Guest1988 has quit IRC | 07:17 | |
| *** k0da has joined #openstack-powervm | 07:35 | |
| *** edmondsw has joined #openstack-powervm | 07:42 | |
| *** edmondsw has quit IRC | 07:47 | |
| *** edmondsw has joined #openstack-powervm | 12:07 | |
| *** esberglu has joined #openstack-powervm | 13:31 | |
| esberglu | edmondsw: efried: Production CI is down. It's hitting similar jenkins IOErrors that staging was hitting | 13:36 |
|---|---|---|
| esberglu | It ended up being an issue with the nic type on the staging mgmt server | 13:36 |
| esberglu | I'm checking with thorst to see if the same issue could be hitting prod | 13:37 |
| efried | ack | 13:37 |
| *** tjakobs has joined #openstack-powervm | 13:37 | |
| *** tjakobs has quit IRC | 13:53 | |
| edmondsw | ack | 14:16 |
| *** tjakobs has joined #openstack-powervm | 14:20 | |
| *** k0da has quit IRC | 16:02 | |
| esberglu | efried: edmondsw: Update on prod CI. Was not the nic type issue hit on staging | 17:24 |
| esberglu | Looks like it might be related to running glance under uwsgi | 17:24 |
| efried | thought we fixed that | 17:24 |
| edmondsw | yeah | 17:24 |
| edmondsw | rather, I thought we had a workaround for that, because I don't think the community changes merged | 17:24 |
| esberglu | efried: edmondsw: We ran into issues back when we were upgrading to pike instead of queens | 17:25 |
| esberglu | We had to manually patch in changes because they never got merged to pike | 17:25 |
| esberglu | But those fixes are in queens | 17:25 |
| esberglu | However, uwsgi is still not technically supported by glance for queens | 17:25 |
| edmondsw | why didn't we hit this in staging? | 17:26 |
| edmondsw | and can you use httpd instead of uwsgi just for glance? | 17:26 |
| edmondsw | or what is the recommendation | 17:26 |
| esberglu | edmondsw: https://review.openstack.org/#/c/545483/ | 17:27 |
| esberglu | That's the patch to do so. I'm patching into the env and redeploying right now to test | 17:27 |
| edmondsw | we should make a stink with the glance guys that they need to merge these patches... | 17:28 |
| edmondsw | they don't want folks to use uwsgi, but they won't merge the patch to allow folks not to?!? | 17:28 |
| esberglu | edmondsw: Yep | 17:28 |
| edmondsw | bonkers | 17:28 |
| esberglu | edmondsw: I'm thinking that since on staging I upgraded to pike, then from pike to queens | 17:29 |
| esberglu | The issue we're hitting might have been missed | 17:30 |
| edmondsw | esberglu I dug up the patches I was thinking of, and I think they should both be there for queens | 17:40 |
| edmondsw | https://review.openstack.org/#/q/Iab2e2848877fa1497008d18c05b0154892941589 | 17:41 |
| edmondsw | https://review.openstack.org/#/q/a3488d5f0067b570974c14572c58bbf8cedf7ed2 | 17:41 |
| edmondsw | ignore that second one... meant this: | 17:41 |
| edmondsw | https://review.openstack.org/#/q/Idf6b4b891ba31cccbeb53d373b40fce5380cea64 | 17:41 |
| edmondsw | esberglu they merged for queens, but not pike | 17:42 |
| edmondsw | so maybe there was a problem with those? Or we did something to workaround the issue for pike, and that causes a problem with queens? | 17:43 |
| esberglu | edmondsw: Yep. That's why I was surprised to hit this on production | 17:44 |
| esberglu | But the error message is a timeout from glance calling uwsgi_response_write_body_do() | 17:44 |
| esberglu | And since it's recommended NOT to use uwsgi on queens, I'm giving that a try | 17:47 |
| *** k0da has joined #openstack-powervm | 17:50 | |
| esberglu | Might be some sort of scaling issue as well. I'm not super confident that this switch will solve it, but it's the only idea I have so far | 17:50 |
| *** k0da has quit IRC | 18:12 | |
| *** k0da has joined #openstack-powervm | 18:23 | |
| *** k0da has quit IRC | 19:11 | |
| *** k0da has joined #openstack-powervm | 19:56 | |
| esberglu | edmondsw: Looks like that patch to disable uwsgi for glance is gonna work | 20:16 |
| esberglu | At least it got past the previous point of failure | 20:16 |
| edmondsw | esberglu good | 20:17 |
| esberglu | edmondsw: CI should be up within the hour if no other issues | 20:18 |
| esberglu | I don't think it was blocking anything though | 20:18 |
| esberglu | efried: Prod logserver is now running with https://review.openstack.org/#/c/563947/ | 20:37 |
| esberglu | I linked an example in the comments | 20:37 |
| efried | esberglu: Cool man. Did you pick up PS11? | 20:38 |
| esberglu | efried: Yeah | 20:38 |
| efried | esberglu: Oh, yeah, this isn't working at all the way it oughtta. Are we stripping the color codes maybe? | 20:39 |
| efried | esberglu: You completely replaced the os-loganalyze we had patched in there before? | 20:39 |
| esberglu | efried: Yep. This is os_loganalyze master with only the above patch | 20:39 |
| efried | thanks dude. | 20:39 |
| esberglu | efried: np | 20:40 |
| esberglu | efried: I'm just gonna leave it alone for now. If it takes a while to get this moving I can revert back to what we've been using | 20:40 |
| efried | esberglu: can you PM me the deets so I can go replace it as Matthieu comes up with more patch sets? (So we don't have to bug you with it) | 20:40 |
| efried | esberglu: Hum, this seems also not to make a difference when I remove colorize=true | 20:42 |
| esberglu | efried: Could also be how we are outputting the journalctl logs to files | 20:44 |
| efried | esberglu: When I download the file, it doesn't have the ANSI codes in it. | 20:45 |
| esberglu | efried: https://github.com/powervm/powervm-ci/blob/master/scripts/process_logs.sh#L40 | 20:45 |
| efried | esberglu: -a should be preserving the color codes. | 20:45 |
| efried | esberglu: I suspect os-loganalyze is stripping them. | 20:45 |
| efried | esberglu: Where are the actual source files on the log server? | 20:46 |
| esberglu | efried: /srv/static/logs/ | 20:46 |
| esberglu | Then whatever path is in the URL | 20:46 |
| esberglu | efried: From the commit message "ANSI escape codes are removed for readability" | 20:47 |
| esberglu | I that applies to downloading as well | 20:47 |
| esberglu | *I think that | 20:47 |
| efried | esberglu: Yeah, the source still has 'em in there. | 20:47 |
| esberglu | efried: I believe your changes also stripped the ANSI codes, but only when displaying | 20:48 |
| efried | esberglu: That's correct. But that was for the other colorizer. We hadn't played with my ANSI one in a while. | 20:48 |
| efried | I'll leave this alone until Matthieu gets a chance to look at it; then maybe I'll mess with mine again. | 20:49 |
| esberglu | efried: You aware of what patches were applied before I switched it today? | 20:50 |
| esberglu | https://review.openstack.org/#/c/524731/ | 20:50 |
| esberglu | and https://review.openstack.org/#/c/524744/ | 20:50 |
| efried | esberglu: Ah, okay. Thanks. | 20:51 |
| *** edmondsw has quit IRC | 21:07 | |
| *** edmondsw_ has joined #openstack-powervm | 21:11 | |
| esberglu | efried: edmondsw_: CI is up and running on queens | 21:15 |
| *** k0da has quit IRC | 21:15 | |
| efried | cool | 21:15 |
| *** edmondsw_ has quit IRC | 21:15 | |
| *** esberglu has quit IRC | 21:22 | |
| *** k0da has joined #openstack-powervm | 21:28 | |
| *** k0da has quit IRC | 21:30 | |
| *** tjakobs has quit IRC | 22:13 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!