| *** strigazi has quit IRC | 00:01 | |
| *** vabada has quit IRC | 00:01 | |
| *** strigazi has joined #openstack-infra | 00:02 | |
| *** vabada has joined #openstack-infra | 00:02 | |
| ianw | clarkb: do you have thoughts on https://review.openstack.org/#/c/605583/ . we could do that, or i could put in a simpler "if fedora { write only ipv4 nameservers }" with a reference to the bug | 00:03 |
|---|---|---|
| *** bobh has quit IRC | 00:04 | |
| *** eernst has quit IRC | 00:06 | |
| clarkb | my only concern with scoping it to fedora is other newer distros likely have the issue too? and we may not remember to exclude them as well | 00:06 |
| openstackgerrit | Goutham Pacha Ravi proposed openstack-infra/project-config master: [manila-ui] Don't run python35 tests until Rocky https://review.openstack.org/605893 | 00:07 |
| openstackgerrit | Merged openstack-infra/zuul master: Don't report non-live items in stats https://review.openstack.org/605540 | 00:09 |
| ianw | clarkb: yeah, maybe the larger change is best, i'll unwip it for comments. i have built with it, but my attempt to upload it to rax failed with an OverLimit Retry... (HTTP 413) whatever that means | 00:09 |
| *** eernst has joined #openstack-infra | 00:10 | |
| openstackgerrit | Goutham Pacha Ravi proposed openstack-infra/project-config master: [manila-ui] Don't run python35 tests until Rocky https://review.openstack.org/605893 | 00:13 |
| *** eernst has quit IRC | 00:14 | |
| *** eernst has joined #openstack-infra | 00:17 | |
| *** sthussey has quit IRC | 00:18 | |
| *** yamamoto has joined #openstack-infra | 00:21 | |
| *** eernst has quit IRC | 00:23 | |
| *** eernst has joined #openstack-infra | 00:24 | |
| *** hamzy has joined #openstack-infra | 00:25 | |
| *** eernst has joined #openstack-infra | 00:27 | |
| *** jamesmcarthur has quit IRC | 00:28 | |
| *** jamesmcarthur has joined #openstack-infra | 00:29 | |
| *** eernst has quit IRC | 00:31 | |
| *** bobh has joined #openstack-infra | 00:33 | |
| *** jamesmcarthur has quit IRC | 00:34 | |
| *** anteaya has quit IRC | 00:34 | |
| *** longkb has joined #openstack-infra | 00:34 | |
| *** rlandy has quit IRC | 00:36 | |
| *** gyee has quit IRC | 00:51 | |
| *** jamesmcarthur has joined #openstack-infra | 00:55 | |
| *** jamesmcarthur has quit IRC | 00:59 | |
| *** smarcet has joined #openstack-infra | 01:03 | |
| *** diablo_rojo has quit IRC | 01:05 | |
| *** bobh has quit IRC | 01:10 | |
| *** shardy has quit IRC | 01:10 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Normalise more of the API stats calls https://review.openstack.org/605898 | 01:12 |
| *** felipemonteiro has joined #openstack-infra | 01:15 | |
| *** shardy has joined #openstack-infra | 01:18 | |
| *** dpawlik has joined #openstack-infra | 01:21 | |
| *** harlowja has quit IRC | 01:24 | |
| *** dpawlik has quit IRC | 01:26 | |
| *** jamesmcarthur has joined #openstack-infra | 01:28 | |
| *** bobh has joined #openstack-infra | 01:31 | |
| *** smarcet has quit IRC | 01:39 | |
| *** bobh has quit IRC | 01:42 | |
| *** smarcet has joined #openstack-infra | 01:45 | |
| *** zzzeek has quit IRC | 01:48 | |
| *** zzzeek has joined #openstack-infra | 01:49 | |
| *** bobh has joined #openstack-infra | 01:49 | |
| *** bobh has quit IRC | 01:53 | |
| *** mrsoul has quit IRC | 01:55 | |
| *** jamesdenton has joined #openstack-infra | 02:07 | |
| *** yamamoto has quit IRC | 02:16 | |
| *** ykarel has joined #openstack-infra | 02:18 | |
| *** hongbin has joined #openstack-infra | 02:20 | |
| *** smarcet has quit IRC | 02:24 | |
| *** stakeda has joined #openstack-infra | 02:26 | |
| *** ykarel has quit IRC | 02:32 | |
| *** smarcet has joined #openstack-infra | 02:34 | |
| *** smarcet has quit IRC | 02:43 | |
| *** smarcet has joined #openstack-infra | 02:45 | |
| *** imacdonn has quit IRC | 02:51 | |
| *** imacdonn has joined #openstack-infra | 02:51 | |
| *** felipemonteiro has quit IRC | 03:00 | |
| *** roman_g has quit IRC | 03:04 | |
| *** ykarel has joined #openstack-infra | 03:13 | |
| *** dpawlik has joined #openstack-infra | 03:22 | |
| *** yamamoto has joined #openstack-infra | 03:24 | |
| *** eernst has joined #openstack-infra | 03:26 | |
| *** dpawlik has quit IRC | 03:27 | |
| *** psachin has joined #openstack-infra | 03:29 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Initial port of install-docker role https://review.openstack.org/605585 | 03:33 |
| *** hongbin has quit IRC | 03:38 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Grafana: set zuul node requests yaxis min https://review.openstack.org/605886 | 03:45 |
| *** graphene has quit IRC | 03:50 | |
| *** yamamoto has quit IRC | 03:50 | |
| *** graphene has joined #openstack-infra | 03:51 | |
| *** graphene has quit IRC | 03:55 | |
| *** graphene has joined #openstack-infra | 03:56 | |
| *** njohnston has quit IRC | 04:02 | |
| *** graphene has quit IRC | 04:20 | |
| *** graphene has joined #openstack-infra | 04:21 | |
| *** graphene has quit IRC | 04:28 | |
| *** graphene has joined #openstack-infra | 04:29 | |
| *** yamamoto has joined #openstack-infra | 04:41 | |
| *** toabctl has quit IRC | 04:48 | |
| AJaeger | config-core, please put https://review.openstack.org/598323 and https://review.openstack.org/605893 on your review queue | 04:54 |
| *** jamesmcarthur has quit IRC | 04:57 | |
| *** jamesmcarthur has joined #openstack-infra | 05:01 | |
| *** jamesmcarthur has quit IRC | 05:06 | |
| *** graphene has quit IRC | 05:08 | |
| *** ramishra has joined #openstack-infra | 05:09 | |
| *** graphene has joined #openstack-infra | 05:09 | |
| *** udesale has joined #openstack-infra | 05:09 | |
| *** ykarel_ has joined #openstack-infra | 05:11 | |
| *** ykarel has quit IRC | 05:14 | |
| *** ykarel__ has joined #openstack-infra | 05:15 | |
| *** ykarel_ has quit IRC | 05:18 | |
| *** ykarel_ has joined #openstack-infra | 05:20 | |
| *** yamamoto has quit IRC | 05:21 | |
| *** ykarel__ has quit IRC | 05:23 | |
| *** dpawlik has joined #openstack-infra | 05:23 | |
| *** ykarel__ has joined #openstack-infra | 05:24 | |
| *** ykarel_ has quit IRC | 05:27 | |
| *** dpawlik has quit IRC | 05:29 | |
| *** ykarel_ has joined #openstack-infra | 05:29 | |
| *** yamamoto has joined #openstack-infra | 05:29 | |
| *** ykarel__ has quit IRC | 05:32 | |
| *** quiquell|off is now known as quiquell | 05:41 | |
| *** chkumar|off is now known as chandankumar | 05:46 | |
| *** e0ne has joined #openstack-infra | 05:52 | |
| openstackgerrit | Merged openstack-infra/project-config master: Grafana: set zuul node requests yaxis min https://review.openstack.org/605886 | 05:57 |
| *** gfidente has joined #openstack-infra | 06:07 | |
| quiquell | Good morning all | 06:07 |
| quiquell | AJaeger: I think we have a review at wrong queue | 06:09 |
| quiquell | AJaeger: https://review.openstack.org/#/c/594511/ | 06:09 |
| *** pcaruana has joined #openstack-infra | 06:11 | |
| *** smarcet has quit IRC | 06:21 | |
| *** verdurin has quit IRC | 06:23 | |
| *** e0ne has quit IRC | 06:24 | |
| *** yamamoto has quit IRC | 06:27 | |
| *** mtreinish has joined #openstack-infra | 06:27 | |
| *** dpawlik has joined #openstack-infra | 06:27 | |
| *** eernst has quit IRC | 06:28 | |
| *** jamesmcarthur has joined #openstack-infra | 06:29 | |
| chandankumar | ianw: Hello | 06:31 |
| chandankumar | ianw: it appears that Zuul queue is very long 92 hours in post anything we can do to minimize it or is it expected? | 06:31 |
| *** dpawlik has quit IRC | 06:32 | |
| *** dpawlik has joined #openstack-infra | 06:32 | |
| *** jamesmcarthur has quit IRC | 06:34 | |
| *** mrsoul has joined #openstack-infra | 06:38 | |
| *** bhavikdbavishi has joined #openstack-infra | 06:40 | |
| AJaeger | quiquell: what do you mean? | 06:43 |
| AJaeger | chandankumar: http://lists.openstack.org/pipermail/openstack-dev/2018-September/134867.html | 06:44 |
| *** jtomasek has joined #openstack-infra | 06:44 | |
| jaosorior | chandankumar: well, we (tripleo) are taking most of the resources. And our timeout issues are still present. So... fixing our timeout issues (which are better than last week), will help in this. | 06:46 |
| AJaeger | quiquell: you need to rebase 594511, it's not current, see the orange dot beside parent | 06:47 |
| AJaeger | jaosorior: 594511 is tripleo, quiquell asked about it, see above as FYI ^ | 06:48 |
| *** e0ne has joined #openstack-infra | 06:51 | |
| *** icey has quit IRC | 06:54 | |
| quiquell | AJaeger: thanks | 06:59 |
| chandankumar | AJaeger: Thanks ! | 07:00 |
| *** florianf|afk has quit IRC | 07:01 | |
| *** shardy has quit IRC | 07:01 | |
| *** shardy has joined #openstack-infra | 07:02 | |
| *** quiquell is now known as quiquell|brb | 07:04 | |
| *** yamamoto has joined #openstack-infra | 07:04 | |
| egonzalez | hi, ask.openstack.org is down | 07:06 |
| *** jamesmcarthur has joined #openstack-infra | 07:07 | |
| *** icey has joined #openstack-infra | 07:07 | |
| dpawlik | egonzalez: ask here :D | 07:09 |
| xinliang | ianw: ping | 07:09 |
| *** jamesmcarthur has quit IRC | 07:11 | |
| *** ginopc has joined #openstack-infra | 07:11 | |
| *** florianf has joined #openstack-infra | 07:12 | |
| *** rcernin has quit IRC | 07:12 | |
| AJaeger | xinliang: best leave a message so that he can read it once he comes back - or somebody else might be able to help. | 07:14 |
| xinliang | ianw: The kolla-debian-building-arm job can build, but it get stuck because of no disk space | 07:15 |
| xinliang | http://logs.openstack.org/59/557659/24/experimental/kolla-build-debian-source-arm64/0c15810/job-output.txt.gz#_2018-09-27_15_15_53_159444 | 07:15 |
| AJaeger | infra-root, ask.openstack.org is down ;( | 07:15 |
| *** ssbarnea|bkp has quit IRC | 07:16 | |
| xinliang | AJaeger: thanks, leave a message:) | 07:17 |
| AJaeger | xinliang: we have 80 GB, see https://docs.openstack.org/infra/manual/testing.html - if you hit that, you need to rework your job | 07:18 |
| *** aojea has joined #openstack-infra | 07:18 | |
| AJaeger | xinliang, I wonder what partition is used, you might want to add some strategic "df" commands for debugging... | 07:19 |
| xinliang | AJaeger: Yes, flavor is enough. we found there is a issue of resize root before. Not sure if it has been fixed yet | 07:22 |
| xinliang | will check with "df" | 07:22 |
| *** jamesmcarthur has joined #openstack-infra | 07:29 | |
| *** dpawlik has quit IRC | 07:32 | |
| *** dpawlik has joined #openstack-infra | 07:34 | |
| *** jamesmcarthur has quit IRC | 07:34 | |
| *** dpawlik has quit IRC | 07:34 | |
| *** dpawlik has joined #openstack-infra | 07:34 | |
| *** quiquell|brb is now known as quiquell | 07:35 | |
| *** dpawlik has quit IRC | 07:36 | |
| *** shu-mutow has joined #openstack-infra | 07:36 | |
| *** markvoelker has quit IRC | 07:36 | |
| *** markvoelker has joined #openstack-infra | 07:37 | |
| *** markvoelker has quit IRC | 07:42 | |
| *** hashar has joined #openstack-infra | 07:43 | |
| *** jpena|off is now known as jpena | 07:43 | |
| *** longkb has quit IRC | 07:44 | |
| *** tosky has joined #openstack-infra | 07:50 | |
| *** quiquell is now known as quiquell|brb | 07:55 | |
| *** longkb has joined #openstack-infra | 07:55 | |
| *** jamesmcarthur has joined #openstack-infra | 07:56 | |
| *** alexchadin has joined #openstack-infra | 07:57 | |
| *** jpich has joined #openstack-infra | 07:58 | |
| *** jamesmcarthur has quit IRC | 08:01 | |
| *** alexchadin has quit IRC | 08:01 | |
| *** rossella_s has joined #openstack-infra | 08:01 | |
| *** bauzas is now known as PapaOurs | 08:03 | |
| *** quiquell|brb is now known as quiquell | 08:05 | |
| *** rossella_s has quit IRC | 08:09 | |
| *** rossella_s has joined #openstack-infra | 08:10 | |
| *** alexchadin has joined #openstack-infra | 08:11 | |
| *** dpawlik has joined #openstack-infra | 08:12 | |
| *** alexchadin has quit IRC | 08:16 | |
| *** alexchadin has joined #openstack-infra | 08:18 | |
| xinliang | growroot still not working for arm64 node, ianw, AJaeger | 08:24 |
| xinliang | http://logs.openstack.org/59/557659/25/experimental/kolla-build-debian-source-arm64/ff00a12/job-output.txt.gz#_2018-09-28_07_57_15_539528 | 08:24 |
| AJaeger | xinliang: thanks for investigating - I cannot help, hope others can | 08:25 |
| *** alexchadin has quit IRC | 08:25 | |
| xinliang | AJaeger: that's fine. | 08:26 |
| *** stephenfin is now known as finucannot | 08:27 | |
| xinliang | This patch: https://review.openstack.org/#/c/578265/ merged and should be fix this issue. | 08:27 |
| xinliang | posted by ianw | 08:28 |
| *** alexchadin has joined #openstack-infra | 08:28 | |
| *** ykarel__ has joined #openstack-infra | 08:28 | |
| AJaeger | ok, then let's wait for ianw - might need to wait until Monday... | 08:28 |
| xinliang | ok | 08:28 |
| *** ykarel_ has quit IRC | 08:31 | |
| *** derekh has joined #openstack-infra | 08:37 | |
| *** markvoelker has joined #openstack-infra | 08:37 | |
| *** olivierb has joined #openstack-infra | 08:38 | |
| frickler | ask.o.o seems fine for me now, maybe the usual morning outage took a bit longer? | 08:49 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Normalise more of the API stats calls https://review.openstack.org/605898 | 08:49 |
| AJaeger | frickler, ianw , please put https://review.openstack.org/598323 and https://review.openstack.org/605893 on your review queue. | 08:50 |
| ianw | xinliang: weird, i can't ssh into the debian arm64 node we have. the others are like 100 days old :/ i'm going to kill them all and cycle them, see what happens | 08:51 |
| AJaeger | yeah, number of periodic jobs and post jobs is slowly going down - still LARGE backlog | 08:51 |
| xinliang | ianw: i see no ready arm64 nodes on london cloud. they are building | 08:54 |
| *** roman_g has joined #openstack-infra | 08:54 | |
| xinliang | ianw: i notice that when job post to run there is no ready node for it. node building just in time | 08:55 |
| ianw | right, i just killed the old ones so min-nodes is kicking in for them | 08:55 |
| *** vivsoni_ has quit IRC | 08:56 | |
| *** ykarel__ is now known as ykarel | 08:59 | |
| *** markvoelker has quit IRC | 08:59 | |
| openstackgerrit | Merged openstack-infra/project-config master: [manila-ui] Don't run python35 tests until Rocky https://review.openstack.org/605893 | 09:01 |
| *** tosky has quit IRC | 09:02 | |
| ianw | xinliang: i can't ssh into the debian one ... which suggests to me the same thing we were seeing before with the config drive not being mounted and the ssh keys not being rolled out | 09:02 |
| ianw | for a xenial host "/dev/sda3 75G 8.8G 63G 13% /" | 09:03 |
| ianw | which looks right | 09:03 |
| *** tosky has joined #openstack-infra | 09:03 | |
| xinliang | ianw: but i can see sr0 device from the log http://logs.openstack.org/59/557659/25/experimental/kolla-build-debian-source-arm64/ff00a12/zuul-info/host-info.primary.yaml | 09:04 |
| *** ykarel is now known as ykarel|lunch | 09:04 | |
| openstackgerrit | brandon zhao proposed openstack/ansible-role-cloud-launcher master: use include_tasks instead of include https://review.openstack.org/606011 | 09:07 |
| xinliang | ianw: there is the node booting log: https://uk.linaro.cloud/project/instances/6d62f976-3025-4569-9243-6b35d2651ef4/console | 09:08 |
| xinliang | not sure if it helps | 09:08 |
| ianw | xinliang: see anything from glean in there? i don't have the login details handy | 09:09 |
| AJaeger | ianw, then https://docs.openstack.org/infra/manual/testing.html - is wrong, it states 80 GB. I didn't realize first taht this is ARM - do we need to update the doc? | 09:10 |
| xinliang | ianw: paste one: http://paste.openstack.org/show/731075/ | 09:10 |
| *** alexchadin has quit IRC | 09:10 | |
| *** rossella_s has quit IRC | 09:11 | |
| xinliang | ianw: sorry just one log is ubuntu node's. | 09:11 |
| xinliang | ianw: this one is debian's , it has glean things: http://paste.openstack.org/show/731076/ | 09:13 |
| ianw | yeah the debian one in in cn-1 | 09:13 |
| *** rossella_s has joined #openstack-infra | 09:13 | |
| xinliang | ianw: so currently, we can't specific london node to run job? | 09:15 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: update api-sig meeting times https://review.openstack.org/605808 | 09:15 |
| xinliang | specify | 09:15 |
| ianw | no, it will balance between them | 09:16 |
| ianw | linaro has all the right keys in the cloud | 09:16 |
| ianw | xinliang: whatever debian this image is, it's not the most recent one | 09:20 |
| ianw | 3 17.8MB 14.8GB 14.8GB ext4 "root" | 09:20 |
| ianw | it's got the quotes | 09:20 |
| xinliang | if so growpart will not work, right? | 09:23 |
| ianw | dib on nb03 is 2.16.0 | 09:23 |
| ianw | looks like puppet is failing on it | 09:24 |
| *** pbourke has quit IRC | 09:24 | |
| *** alexchadin has joined #openstack-infra | 09:25 | |
| *** pbourke has joined #openstack-infra | 09:25 | |
| xinliang | ianw: there might be a problem. I mean using node on cn cloud. nodes on the cloud are not working due to networking issue | 09:25 |
| ianw | no, it seems like puppet is not running on nb03, so it hasn't been updated to the lastest dib. so the images it has built are out of date i guess | 09:26 |
| ianw | i'm trying a manual puppet run to see what's up there | 09:26 |
| ianw | Could not get latest version: undefined method `[]' for nil:NilClass ? | 09:26 |
| cmurphy | ianw: hi do you want debugging help? | 09:28 |
| ianw | cmurphy: maybe :) i'll see if i can get some sort of sensible error with kick.sh on nb03 | 09:29 |
| cmurphy | o7 | 09:30 |
| ianw | one of hte problems with the dib puppet was that pip installing in on arm took a long time due to it building everything under the sun, and it was timing out. i'm pretty sure i merged a fix for that though | 09:31 |
| *** armax has quit IRC | 09:31 | |
| ianw | ok, so this is the problem http://paste.openstack.org/show/731083/ | 09:34 |
| ianw | comes from http://git.openstack.org/cgit/openstack-infra/puppet-diskimage_builder/tree/manifests/init.pp#n79 | 09:35 |
| ianw | which looks pretty straight forward to me :/ | 09:35 |
| cmurphy | must be a bug in the openstack_pip provider | 09:35 |
| ianw | yeah, that's what i'm thinking, has there been updates to that lately? | 09:36 |
| cmurphy | not since last year | 09:36 |
| ianw | root@nb03:~# pip --version | 09:37 |
| ianw | pip 18.0 | 09:37 |
| ianw | was that recently released or something? | 09:37 |
| ianw | not really, july | 09:37 |
| cmurphy | it's coming from either http://git.openstack.org/cgit/openstack-infra/puppet-pip/tree/lib/puppet/provider/package/openstack_pip.rb#n23 or http://git.openstack.org/cgit/openstack-infra/puppet-pip/tree/lib/puppet/provider/package/openstack_pip.rb#n28 so either the output of `pip list --outdated` or `pip show diskimage-builder` is unexpected | 09:39 |
| ianw | http://paste.openstack.org/show/731088/ is the output, looks about right to me for show | 09:40 |
| ianw | http://paste.openstack.org/show/731089/ looks about right too | 09:41 |
| *** toabctl has joined #openstack-infra | 09:42 | |
| cmurphy | it's looking for 'Latest: ' but the header is 'Latest' | 09:43 |
| ianw | ahhh, yes, the output is quite different | 09:45 |
| ianw | it's expecting something that looks like "cryptography (1.2.3) - Latest: 2.3.1 [wheel]" | 09:46 |
| cmurphy | that's annoying | 09:47 |
| *** ykarel|lunch is now known as ykarel | 09:48 | |
| ianw | cmurphy: it's pip 18 i guess :/ | 09:48 |
| ianw | maybe we're just pinned on other servers and haven't noticed | 09:49 |
| ianw | for the immediate issue, i've install pip 9.0.1 on nb03 and that should get it going ... | 09:50 |
| ianw | cmurphy: is it ok if i throw handling later pip in openstack_pip on your plate? i'm on pto for a bit | 09:51 |
| cmurphy | ianw: on it | 09:51 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-pip master: Fix openstack_pip provider for pip 18 https://review.openstack.org/606021 | 09:52 |
| cmurphy | ianw: something like that maybe ^ | 09:52 |
| *** jtomasek has quit IRC | 09:56 | |
| *** markvoelker has joined #openstack-infra | 09:57 | |
| ianw | cmurphy: ++ ! | 09:57 |
| openstackgerrit | Merged openstack-infra/zuul master: replace dict.update by a dict merge in zuul_return https://review.openstack.org/602054 | 09:57 |
| ianw | xinliang: so .. nb03 now has dib 2.17.0 ... so i'll trigger some fresh arm64 builds | 09:58 |
| ianw | talk about yak shaving! | 09:58 |
| *** longkb has quit IRC | 09:58 | |
| cmurphy | heh | 09:59 |
| *** scroll is now known as hfjvjffju | 09:59 | |
| *** alexchadin has quit IRC | 10:05 | |
| xinliang | ianw: thanks, will try the new nodes | 10:11 |
| *** jamesmcarthur has joined #openstack-infra | 10:12 | |
| ianw | xinliang: cool, you can see the status @ http://nl01.openstack.org/dib-image-list and http://nl01.openstack.org/image-list | 10:14 |
| *** xinliang has quit IRC | 10:16 | |
| *** e0ne has quit IRC | 10:16 | |
| *** jamesmcarthur has quit IRC | 10:16 | |
| *** markvoelker has quit IRC | 10:18 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-sphinx master: Add attr-overview directive https://review.openstack.org/604980 | 10:18 |
| *** alexchadin has joined #openstack-infra | 10:22 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Use zuul-sphinx for configuration layout https://review.openstack.org/604274 | 10:23 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Add overview of config options https://review.openstack.org/604984 | 10:23 |
| *** e0ne has joined #openstack-infra | 10:25 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [WIP] Provision graphite01.o.o via docker container https://review.openstack.org/606028 | 10:27 |
| *** alexchadin has quit IRC | 10:29 | |
| *** yamamoto has quit IRC | 10:31 | |
| *** yamamoto has joined #openstack-infra | 10:32 | |
| *** yamamoto has quit IRC | 10:38 | |
| *** bhavikdbavishi has quit IRC | 10:38 | |
| *** alexchadin has joined #openstack-infra | 10:39 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove release-openstack-server and publish-xstatic templates https://review.openstack.org/531830 | 10:39 |
| *** stakeda has quit IRC | 10:43 | |
| *** olivierb has quit IRC | 10:47 | |
| *** AJaeger has quit IRC | 10:56 | |
| *** njohnston has joined #openstack-infra | 10:56 | |
| *** dtantsur|afk is now known as dtantsur | 10:57 | |
| *** njohnston has left #openstack-infra | 10:58 | |
| *** rfolco has quit IRC | 11:07 | |
| *** jpena is now known as jpena|lunch | 11:07 | |
| *** AJaeger has joined #openstack-infra | 11:07 | |
| *** ssbarnea|bkp has joined #openstack-infra | 11:08 | |
| *** alexchadin has quit IRC | 11:08 | |
| *** xinliang has joined #openstack-infra | 11:09 | |
| xinliang | ianw: root resize still not working : http://logs.openstack.org/59/557659/25/experimental/kolla-build-debian-source-arm64/f1088a3/job-output.txt.gz#_2018-09-28_10_53_05_991827 | 11:09 |
| *** rossella_s has quit IRC | 11:21 | |
| *** rossella_s has joined #openstack-infra | 11:22 | |
| *** rfolco has joined #openstack-infra | 11:26 | |
| *** yamamoto has joined #openstack-infra | 11:26 | |
| *** rossella_s has quit IRC | 11:26 | |
| *** rossella_s has joined #openstack-infra | 11:27 | |
| ianw | xinliang: the new images haven't finished uploading | 11:31 |
| ianw | http://nl01.openstack.org/image-list | 11:31 |
| *** udesale has quit IRC | 11:32 | |
| *** rossella_s has quit IRC | 11:33 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-pip master: Fix openstack_pip provider for pip 18 https://review.openstack.org/606021 | 11:34 |
| *** rossella_s has joined #openstack-infra | 11:37 | |
| *** dpawlik has quit IRC | 11:37 | |
| *** agopi|brb is now known as agopi | 11:40 | |
| *** ssbarnea|bkp has quit IRC | 11:40 | |
| *** panda|off is now known as panda | 11:42 | |
| *** shu-mutow has quit IRC | 11:43 | |
| *** yolanda has joined #openstack-infra | 11:44 | |
| *** rossella_s has quit IRC | 11:53 | |
| *** mrsoul has quit IRC | 11:54 | |
| *** rossella_s has joined #openstack-infra | 11:55 | |
| *** Bhujay has joined #openstack-infra | 11:58 | |
| *** EmilienM is now known as EvilienM | 12:00 | |
| *** jpena|lunch is now known as jpena | 12:05 | |
| *** rossella_s has quit IRC | 12:09 | |
| *** rossella_s has joined #openstack-infra | 12:10 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.openstack.org/576907 | 12:10 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: CLI: add create-web-token command https://review.openstack.org/605386 | 12:10 |
| *** rossella_s has quit IRC | 12:14 | |
| *** rossella_s has joined #openstack-infra | 12:15 | |
| *** rh-jelabarre has joined #openstack-infra | 12:15 | |
| *** yamamoto has quit IRC | 12:16 | |
| *** rfolco has quit IRC | 12:18 | |
| *** weshay is now known as weshay_ruck | 12:23 | |
| *** rlandy has joined #openstack-infra | 12:24 | |
| openstackgerrit | neilsun proposed openstack-infra/zuul master: Add type check for zuul conf https://review.openstack.org/591917 | 12:25 |
| *** ykarel_ has joined #openstack-infra | 12:28 | |
| *** boden has joined #openstack-infra | 12:29 | |
| *** mriedem has joined #openstack-infra | 12:29 | |
| *** ykarel has quit IRC | 12:31 | |
| *** ykarel_ is now known as ykarel | 12:31 | |
| *** nicolasbock_ has joined #openstack-infra | 12:32 | |
| *** agopi is now known as agopi|brb | 12:34 | |
| *** jcoufal has joined #openstack-infra | 12:34 | |
| *** agopi|brb has quit IRC | 12:39 | |
| *** tpsilva has joined #openstack-infra | 12:40 | |
| *** graphene has quit IRC | 12:42 | |
| *** trown|outtypewww is now known as trown | 12:43 | |
| *** graphene has joined #openstack-infra | 12:44 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Temporarily bump up capacity by 50 VMs https://review.openstack.org/606058 | 12:44 |
| *** rossella_s has quit IRC | 12:45 | |
| *** jamesmcarthur has joined #openstack-infra | 12:45 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Revert "Temporarily bump up capacity by 50 VMs" https://review.openstack.org/606059 | 12:45 |
| mnaser | infra-root: ^ can someone promote this to top of the check queue and gate afterwards? | 12:45 |
| *** rossella_s has joined #openstack-infra | 12:47 | |
| AJaeger | mnaser: the first one I hope ;) | 12:47 |
| mnaser | AJaeger: aha, yes | 12:47 |
| AJaeger | thanks a lot, mnaser ! | 12:47 |
| AJaeger | fungi, frickler, are you around to help ? ^ | 12:48 |
| mnaser | or maybe we can drag dmsimard back out :P | 12:48 |
| mnaser | considering east coast | 12:48 |
| cmurphy | awesome mnaser | 12:48 |
| mnaser | :) | 12:49 |
| AJaeger | config-core, a long but mechanic change to update release jobs, please review https://review.openstack.org/598323 . Could we give dhellmann a +2A, please? | 12:49 |
| AJaeger | pabelanger: are you around? | 12:49 |
| dmsimard | I'm here | 12:49 |
| AJaeger | dmsimard: know how to promote a change? | 12:49 |
| AJaeger | https://review.openstack.org/#/c/606058 | 12:50 |
| openstackgerrit | neilsun proposed openstack-infra/zuul master: Add type check for zuul conf https://review.openstack.org/591917 | 12:51 |
| dmsimard | AJaeger: there's docs for it so I suppose https://zuul-ci.org/docs/zuul/admin/client.html#promote | 12:51 |
| *** hashar is now known as hasharAway | 12:51 | |
| mnaser | seems straight forward | 12:52 |
| mnaser | zuul promote --tenant openstack --pipeline check --changes 606058,1 | 12:52 |
| AJaeger | mnaser: currently situation is getting slowly under control, the backlog on nodes is not really growing so far http://grafana.openstack.org/d/T6vSHcSik/zuul-status | 12:52 |
| *** rossella_s has quit IRC | 12:52 | |
| AJaeger | Zuul even got through 40 periodic jobs and some post jobs... | 12:53 |
| dmsimard | mnaser: we need to enqueue in gate first | 12:53 |
| mnaser | oh yes | 12:54 |
| mnaser | dmsimard: or rather check i think you mean there | 12:54 |
| *** rossella_s has joined #openstack-infra | 12:54 | |
| dmsimard | it's in gate and promoted | 12:54 |
| mnaser | also | 12:54 |
| mnaser | i really really really think we should look into smaller instance sizes for smaller jobs | 12:55 |
| mnaser | it'll help use our resources much more efficently | 12:55 |
| AJaeger | dmsimard: thanks! | 12:55 |
| dmsimard | #status log (dmsimard) enqueued https://review.openstack.org/606058 to gate and promoted it to increase nodepool capacity | 12:55 |
| mnaser | dmsimard: thank you so much :> | 12:55 |
| *** kgiusti has joined #openstack-infra | 12:55 | |
| openstackstatus | dmsimard: finished logging | 12:55 |
| *** bhavikdbavishi has joined #openstack-infra | 12:55 | |
| dmsimard | mnaser: yes, in fact we can do that now | 12:55 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/zuul-sphinx master: Raise an error if a file in zuul.d is empty https://review.openstack.org/606062 | 12:56 |
| dmsimard | mnaser: nodepool is quota aware so instead of using max-servers we can use max-ram/mac-cores and use different flavors for different jobs | 12:56 |
| mnaser | i think that'd be really beneficial.. for example running doc jobs on a 1 core / 1g instance | 12:56 |
| mnaser | that way.. 8 doc jobs can run at once instead for example | 12:56 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/zuul-sphinx master: Raise an error if a file in zuul.d is empty https://review.openstack.org/606062 | 12:58 |
| fungi | mnaser: we would just enqueue directly to the gate and then no need to promote as the project-config queue is basically empty anyway | 12:59 |
| mnaser | fungi: gotcha | 12:59 |
| fungi | dmsimard: were you handling that, or should i? | 12:59 |
| AJaeger | mnaser, fungi , could either of you review https://review.openstack.org/598323 , please? I know it's long - but really mechanical... | 13:00 |
| mnaser | fungi: dmsimard already did :) | 13:00 |
| AJaeger | fungi: it's done | 13:00 |
| fungi | thanks dmsimard and mnaser! | 13:00 |
| dmsimard | fungi: I did it and I did a status log | 13:00 |
| *** jaosorior has quit IRC | 13:00 | |
| mnaser | AJaeger: i was confused why `release-openstack-server` jobs were replaced by `publish-to-pypi-python3 | 13:00 |
| dmsimard | fungi: I am wondering if we should really keep that backlog of >300 periodic jobs | 13:00 |
| mnaser | its a change in behaviour.. but the commit message didnt explain why or what | 13:00 |
| fungi | mnaser: the release team is working on publishing server projects to pypi now | 13:01 |
| mnaser | ok, the commit message seemed to imply that it was just moving to a python3 release job, but let me see | 13:01 |
| fungi | release-openstack-server was basically like publish-to-pypi-python3 except it skipped the actual pypi upload | 13:01 |
| fungi | dhellmann: ^ can you confirm? | 13:01 |
| mnaser | yeah i figured, but dont we need to make sure all these projects have pre-configured stuff in pypi? | 13:01 |
| mnaser | so the job doesnt fail? | 13:01 |
| mnaser | the acls and allowing openstackci to upload to them | 13:02 |
| fungi | pretty sure they're going around registering the missing ones | 13:02 |
| AJaeger | mnaser: we discussed on IRC yesterday, the release team will take care of that | 13:02 |
| fungi | with a few exceptions (keystone, magnum, congress) which need coordination with previous registrants on pypi to possibly allow us to take over those names | 13:02 |
| mnaser | cool, in that case, its ok wtih me | 13:02 |
| AJaeger | mnaser: there was an email as well ot openstack-dev (I agree, the commit message could be more verbose) | 13:03 |
| AJaeger | thanks, mnaser | 13:03 |
| mnaser | yeah let's get it rolling, release team is accessible enough to talk to should there be any issues | 13:04 |
| tosky | uhm, couldn't you have make release-openstack-server derive from publish-to-pypi-python3, instead of replacing all jobs? | 13:04 |
| tosky | made* | 13:04 |
| *** dpawlik has joined #openstack-infra | 13:05 | |
| AJaeger | tosky: sure, we could have just changed the template - but then have two templates that do the same... | 13:05 |
| tosky | ok | 13:05 |
| fungi | deduplication of jobs | 13:06 |
| fungi | or rather of templates in this case | 13:06 |
| *** rfolco has joined #openstack-infra | 13:06 | |
| openstackgerrit | Merged openstack-infra/project-config master: Temporarily bump up capacity by 50 VMs https://review.openstack.org/606058 | 13:06 |
| mnaser | um | 13:07 |
| mnaser | do the current doc job builds for other languages? | 13:08 |
| fungi | um? | 13:08 |
| *** dpawlik has quit IRC | 13:08 | |
| mnaser | i'm seeing the OSA docs getting translated to german (yay!!!) but i dont see a link in our docs to show those translations | 13:08 |
| fungi | if the pofiles are in the repo they should... | 13:08 |
| AJaeger | mnaser: we started wit hfirst repos - eumel8 has started. I think OSA is one of the three guinea pigs ;) | 13:08 |
| mnaser | well thats why i was wondering what we have to tweak | 13:08 |
| *** dpawlik has joined #openstack-infra | 13:08 | |
| mnaser | https://review.openstack.org/#/c/605990/1 | 13:08 |
| AJaeger | mnaser: best talk with dhellmann and eumel8 | 13:09 |
| mordred | yeah - we discussed translated docs at the ptg (exciting) | 13:09 |
| AJaeger | mnaser: I think the building is not done - just pushing to translation server and back. | 13:09 |
| mnaser | AJaeger: ahhh okay | 13:09 |
| mnaser | oh | 13:09 |
| mnaser | there's a `build-tox-manuals-checklang` job | 13:09 |
| mnaser | which we dont run | 13:09 |
| mnaser | eumel8: whenever you're around, let me know what it takes to make it possible for the docs to be seen in HTML :) | 13:10 |
| AJaeger | mnaser: that is only for openstack-manuals and friends, don't use it | 13:11 |
| mnaser | fine i'll make my own then if i cant use yours >:( | 13:11 |
| mnaser | :p | 13:11 |
| AJaeger | mnaser: idea is to enhance docs tox environment - so you use openstack-tox-docs ;) | 13:12 |
| AJaeger | mnaser: I'm happy to converge in the end to a common job for this if that's the way forward... | 13:12 |
| AJaeger | mnaser: right now checklang has some "strange" requirements | 13:13 |
| *** psachin has quit IRC | 13:13 | |
| eumel8 | mnaser: dunno. There were some discussions during PTG which I didn't attend. Best to ask dhellmann or ianychoi. My first shot was wrong: https://review.openstack.org/#/c/604568/ Now I don't know how to proceed. | 13:13 |
| *** agopi|brb has joined #openstack-infra | 13:14 | |
| *** agopi|brb is now known as agopi|afk | 13:14 | |
| AJaeger | mnaser: http://lists.openstack.org/pipermail/openstack-dev/2018-September/134609.html | 13:14 |
| openstackgerrit | Merged openstack-infra/project-config master: switch all official python projects to python3 publishing job https://review.openstack.org/598323 | 13:16 |
| *** felipemonteiro has joined #openstack-infra | 13:18 | |
| AJaeger | mnaser: btw. feel free to get back your 50 nodes anytime - and self approve https://review.openstack.org/#/c/606059/ ... | 13:18 |
| *** zul has joined #openstack-infra | 13:18 | |
| mnaser | AJaeger: yep, that's the plan | 13:19 |
| mnaser | thank you | 13:19 |
| mnaser | i guess we'll have to wait till bridge kicks a nodepool run | 13:20 |
| *** ramishra has quit IRC | 13:22 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove release-openstack-server and publish-xstatic templates https://review.openstack.org/531830 | 13:22 |
| AJaeger | mnaser: followup cleanup ^ | 13:24 |
| AJaeger | mnaser: ignore, needs more work ;( | 13:24 |
| mnaser | fungi: sorry to bother you but could you get puppet running on nodepool? i jus want to make sure everything starts cleanly so i get back to other stuff | 13:24 |
| dhellmann | eumel8 : mordred was going to with with you and ianychoi on updating the underlying job. If you work on a script that does the translation build for an existing HTML build created by "tox -e docs" I think that would be a good next step. | 13:24 |
| *** efried is now known as fried_rice | 13:25 | |
| *** derekh has quit IRC | 13:26 | |
| *** derekh has joined #openstack-infra | 13:26 | |
| fungi | mnaser: i can kick the launchers manually, sure | 13:27 |
| fungi | just a sec | 13:27 |
| eumel8 | dhellmann: ok, then I have to catch mordred and ianychoi, thx. | 13:28 |
| mnaser | fungi: thank you :) | 13:33 |
| *** brokencycle has joined #openstack-infra | 13:33 | |
| brokencycle | Hi! I am unhappy with the website, www.openstack.org | 13:33 |
| mnaser | brokencycle: whats up? | 13:33 |
| dhellmann | eumel8 : your existing script is probably a good start, it just needs to take into account the right starting assumptions | 13:33 |
| mnaser | if there's something in specific i can help point to the right folks to help address any issues | 13:34 |
| *** david-lyle has joined #openstack-infra | 13:34 | |
| brokencycle | mnaser: In particular, it seems to be impossible to open some parts of the website with a right click in a new tab. | 13:34 |
| mnaser | brokencycle: are you talking about the summit schedule pages? | 13:34 |
| AJaeger | fungi, didn't clarkb disable manually OVH? I see the graph growing far too much for nodepool... | 13:35 |
| brokencycle | Eg. when I look at the project list, https://www.openstack.org/software/project-navigator/deployment-tools, if I right-click on any of them, the new tab just opens as 'www.openstack.org'. If I left-click on the same link, I get to the actual project. | 13:35 |
| brokencycle | But I want to right click and open the project's page in a new tab, not the existing one. | 13:36 |
| mnaser | brokencycle: you are right, looking at hte html code, i see <a href> but without a link there | 13:36 |
| mnaser | brokencycle: could you pm me your email and i can start an email thread with someone that can help you? | 13:37 |
| *** dklyle has quit IRC | 13:37 | |
| *** bhavikdbavishi has quit IRC | 13:39 | |
| mnaser | brokencycle: voila, fired off an email, foundation staff are awesome when it comes to this so i expect things to clear up soon | 13:39 |
| mnaser | thanks for letting us know | 13:39 |
| AJaeger | fungi, did you see my comment above? | 13:41 |
| eumel8 | dhellmann: from my understanding you want everything to build into the repo. But that requires to have the similar script in each repo with translation. I tried to centralize it, in the wrong way. Second solution would be to bring this script into the repo like in openstack-manuals, so tox -edocs builds the whole documentation | 13:43 |
| *** zzzeek has quit IRC | 13:43 | |
| mnaser | does anyone know how we can make pbr generate version for project via cli? | 13:44 |
| mnaser | openstack ansible maintains a hard coded variable for the version we're at right now | 13:44 |
| mnaser | we'd like to use pbr instead, we can run a local lookup somehow | 13:44 |
| mnaser | pbr info or pbr sha hasnt helped too much | 13:44 |
| *** zzzeek has joined #openstack-infra | 13:45 | |
| dmsimard | mnaser: https://github.com/openstack/ara/blob/master/ara/__init__.py | 13:45 |
| mnaser | ah so we might need to run a python script | 13:45 |
| dmsimard | mnaser: it probably wouldn't be too hard to do a one liner ? | 13:45 |
| mnaser | i mean probably cleaner to get a small python script probably.. i think | 13:45 |
| mnaser | that way we run it with our own python virtualenv | 13:45 |
| mnaser | ah damn | 13:47 |
| mnaser | but we don't actually install the package locally | 13:47 |
| *** zzzeek has quit IRC | 13:48 | |
| mnaser | oh we do nevermind | 13:48 |
| dmsimard | mandre: /opt/venv/bin/python -c 'import pbr.version; print(pbr.version.VersionInfo("foo").version_string())' ? | 13:48 |
| dmsimard | er, mnaser ^ | 13:49 |
| mnaser | lemme try that | 13:49 |
| *** zzzeek has joined #openstack-infra | 13:49 | |
| mnaser | /opt/ansible-runtime/bin/python -c 'import pbr.version; print(pbr.version.VersionInfo("openstack-ansible").release_string())' | 13:49 |
| mnaser | works perfectly | 13:50 |
| dmsimard | \o/ | 13:50 |
| fungi | AJaeger: clarkb increased max-servers in bhs1 to something like 20 yesterday and it seemed to be mostly holding but going above that ended up with more port leak/pileup | 13:51 |
| *** agopi|afk is now known as agopi | 13:53 | |
| dhellmann | eumel8 : the job can check out the doc tools repo so we don't have to have a copy of the script everywhere | 13:54 |
| dhellmann | eumel8 : or we can put the script in the repo where the job is defined | 13:54 |
| dhellmann | mnaser : "python setup.py --version" | 13:54 |
| mnaser | dhellmann: wow that was an obviosu one | 13:55 |
| mnaser | lol | 13:55 |
| *** yamamoto has joined #openstack-infra | 13:55 | |
| AJaeger | fungi: looking at grafana: All is fine again | 13:55 |
| dhellmann | mnaser :-) | 13:55 |
| *** eernst has joined #openstack-infra | 13:59 | |
| *** felipemonteiro has quit IRC | 14:03 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: remove job settings for ironic repositories https://review.openstack.org/592472 | 14:06 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Simplify vexxhost nodepool configuration https://review.openstack.org/605469 | 14:08 |
| pabelanger | AJaeger: mnaser: dmsimard: fungi: ^ should reduce some copypasta for vexxhost for nodepool | 14:09 |
| *** rossella_s has quit IRC | 14:10 | |
| AJaeger | mnaser: I'd like you to +2 first ;) Happy to +2 afterwards... | 14:12 |
| mnaser | pabelanger: AJaeger added a comment.. i'd like us to bring ca-ymq-1 to queens before we bring bfv back | 14:13 |
| eumel8 | dhellmann: That confused me because I was in the docs tool repo and AJaeger mentioned it's the wrong place ;) | 14:13 |
| pabelanger | mnaser: Oh, I didn't see that was a different | 14:13 |
| pabelanger | yah, that won't work then | 14:13 |
| *** rossella_s has joined #openstack-infra | 14:14 | |
| AJaeger | eumel8, dhellmann, we discussed putting this in a role in ansible. But yes, script can life anywhere... | 14:14 |
| mnaser | infra-root: i suspect some form of quota is being hit at sjc1 .. any nodepool logs to help me bump up the right thing? | 14:16 |
| AJaeger | eumel8: maybe I read too much in dhellmann's email... | 14:16 |
| *** jamesmcarthur has quit IRC | 14:18 | |
| *** jamesmcarthur has joined #openstack-infra | 14:18 | |
| *** bnemec is now known as beekneemech | 14:20 | |
| eumel8 | AJaeger, dhellmann: my focus was also to build docs locally. With an Ansible Role you need to download first and setup Ansible for testing the docs. Thats looks complicated to me | 14:21 |
| AJaeger | dhellmann: ironic is done, isn't it? | 14:22 |
| corvus | mnaser: openstack.exceptions.SDKException: Error in creating the server: Build of instance ef7fbe70-d2da-433c-8368-234de8a20db3 aborted: VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed gigabytes quota. Requested 80G, quota is 5120G and 5120G has been consumed. | 14:22 |
| dhellmann | AJaeger , eumel8 : we want the job to run the script so we don't have to update tox.ini. I don't really mind where we put the script, as long as it is in a place where we can update it if we have to. | 14:24 |
| dhellmann | AJaeger : I'm still catching up; dealing with TC election stuff this morning | 14:24 |
| dhellmann | eumel8 : if you write the script so the ansible role can call it, that should give us the best of both worlds. | 14:25 |
| AJaeger | dhellmann: and I would put the script in openstack-zuul-jobs then | 14:25 |
| dhellmann | AJaeger : sounds good to me | 14:25 |
| AJaeger | dhellmann: just run your goal tools script after pushing ironic for some +2As;) | 14:26 |
| *** edmondsw_ has joined #openstack-infra | 14:26 | |
| eumel8 | dhellmann: okay, will think about it, thx | 14:27 |
| *** edmondsw has quit IRC | 14:29 | |
| *** edmondsw_ is now known as edmondsw | 14:29 | |
| *** agopi is now known as agopi|afk | 14:29 | |
| *** roman_g has quit IRC | 14:30 | |
| *** quiquell is now known as quiquell|off | 14:31 | |
| *** bobh has joined #openstack-infra | 14:32 | |
| *** electrofelix has quit IRC | 14:34 | |
| mnaser | corvus: thanks, let me do some math | 14:36 |
| mnaser | corvus: disk quota bumped to 6144 which should put us in a good place | 14:36 |
| *** jamesmcarthur has quit IRC | 14:38 | |
| *** jamesmcarthur has joined #openstack-infra | 14:38 | |
| *** jamesmcarthur has quit IRC | 14:41 | |
| *** jamesmcarthur has joined #openstack-infra | 14:42 | |
| mnaser | cool i think we're good | 14:44 |
| mnaser | i see ~142 in use | 14:44 |
| *** felipemonteiro has joined #openstack-infra | 14:44 | |
| *** armax has joined #openstack-infra | 14:45 | |
| *** gfidente is now known as gfidenteN00b | 14:46 | |
| *** jamesmcarthur has quit IRC | 14:46 | |
| openstackgerrit | Merged openstack-infra/system-config master: Only replicate openstack namespaces to github https://review.openstack.org/605486 | 14:52 |
| AJaeger | zuul experts, I'm confused https://review.openstack.org/#/c/593884/ runs openstack-tox-py35 but we disabled that with change https://review.openstack.org/605893 . Is that change not rolled out? Or anything wrong with it? | 14:53 |
| *** jamesmcarthur has joined #openstack-infra | 14:55 | |
| AJaeger | corvus, mordred, any ideas? ^ | 14:56 |
| *** e0ne has quit IRC | 14:58 | |
| *** Bhujay has quit IRC | 14:58 | |
| *** jamesmcarthur has quit IRC | 14:59 | |
| *** jamesmcarthur has joined #openstack-infra | 15:00 | |
| AJaeger | tbarron just asked the same question in #zuul - we can discuss there as well | 15:01 |
| tbarron | AJaeger: ty :) | 15:01 |
| *** e0ne has joined #openstack-infra | 15:02 | |
| AJaeger | tbarron: I'm still puzzled ;) | 15:03 |
| corvus | tbarron, AJaeger: 2018-09-28 14:30:30,576 DEBUG zuul.layout: Pipeline variant <Job openstack-tox-py35 branches: None source: openstack-infra/openstack-zuul-jobs/zuul.d/project-templates.yaml@master#515> matched <Change 0x7f183b99f | 15:03 |
| corvus | 6a0 openstack/manila-ui 593884,2> | 15:03 |
| *** smarcet has joined #openstack-infra | 15:03 | |
| corvus | tbarron, AJaeger: that points to this: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n515 | 15:04 |
| AJaeger | corvus: so, the "branches: ^(?!stable/(ocata|pike|queens)).*$" get ignored? | 15:05 |
| corvus | that project does use that project-template | 15:05 |
| corvus | AJaeger: for that invocation, not for the other | 15:05 |
| AJaeger | corvus: so, we need to remove the template as well here? | 15:05 |
| corvus | AJaeger: yes | 15:05 |
| AJaeger | corvus: ah, did that semantic change in the last months? I might have missed that... | 15:05 |
| *** sthussey has joined #openstack-infra | 15:05 | |
| corvus | AJaeger, tbarron: you can add the project-template in-repo to just the branches it should apply to | 15:06 |
| AJaeger | tbarron: know what to do? I'm happy to +2 a change to remove the template compltely | 15:06 |
| AJaeger | corvus: that's what they will do as part of python3-first migration. | 15:06 |
| AJaeger | tbarron: I think corvus is right, you converted master etc already | 15:06 |
| AJaeger | tbarron: so, just remove from project-config the rest and you get what you want... | 15:06 |
| AJaeger | tbarron: that allows you to finish conversion | 15:07 |
| *** dave-mccowan has joined #openstack-infra | 15:07 | |
| AJaeger | tbarron: still with us? | 15:07 |
| tbarron | AJaeger: just trying to follow :) | 15:08 |
| corvus | AJaeger: (also, btw, you can find the same debug information in the "_inheritance_path" variable here: http://logs.openstack.org/84/593884/2/check/openstack-tox-py35/d7efc50/zuul-info/inventory.yaml) | 15:08 |
| AJaeger | corvus: ah, thanks! | 15:09 |
| AJaeger | tbarron: ok, waiting for a change by manila team and will guide you through it... | 15:10 |
| AJaeger | tbarron: change for project-config | 15:10 |
| * tbarron is fetching fresh project-config | 15:12 | |
| *** dave-mccowan has quit IRC | 15:13 | |
| tbarron | AJaeger: should I be looking at zuul.d/projects.yaml? openstack-manila-ui project? | 15:16 |
| AJaeger | tbarron: yes, similar to https://review.openstack.org/#/c/605893/ | 15:16 |
| AJaeger | tbarron: just remove everything py35 related from manila-ui ;) | 15:16 |
| *** rossella_s has quit IRC | 15:16 | |
| AJaeger | tbarron: the delete key is your key to success for that change ;) | 15:17 |
| *** rossella_s has joined #openstack-infra | 15:19 | |
| openstackgerrit | Tom Barron proposed openstack-infra/project-config master: Remove py3 jobs for manila-ui project https://review.openstack.org/606114 | 15:20 |
| tbarron | AJaeger: ^^ | 15:20 |
| *** bhavikdbavishi has joined #openstack-infra | 15:23 | |
| *** ginopc has quit IRC | 15:24 | |
| *** yamamoto has quit IRC | 15:25 | |
| AJaeger | tbarron: one line too much - otherwise ok | 15:25 |
| openstackgerrit | Tom Barron proposed openstack-infra/project-config master: Remove py3 jobs for manila-ui project https://review.openstack.org/606114 | 15:28 |
| AJaeger | tbarron: LGTM, +2 - any other config-core to +2A ^, please? That allows manila team to finish python3-first imports... | 15:28 |
| tbarron | AJaeger: ty! | 15:29 |
| AJaeger | tbarron: due to backlog, this will take at least two hours until we have it tested... | 15:29 |
| tbarron | AJaeger: kk, gouthamr will be getting up in Seattle by then :) | 15:29 |
| AJaeger | ;) | 15:30 |
| tbarron | though it looks like he was working quite late last night | 15:30 |
| AJaeger | then he deserves his rest.. | 15:32 |
| *** ykarel is now known as ykarel|away | 15:35 | |
| *** lbragstad is now known as elbragstad | 15:38 | |
| *** zul has quit IRC | 15:41 | |
| fungi | clarkb: did you see the reply from amorin? looks like we should be okay to crank bhs1 back up to max again | 15:42 |
| fungi | i'll prep the change | 15:42 |
| clarkb | fungi: I havent yet, still booting my day ++ to getting things rolling | 15:45 |
| mnaser | fungi: i think it happened indirectly | 15:45 |
| mnaser | when you kicked off nodepool | 15:45 |
| fungi | oh, maybe | 15:46 |
| *** adriancz has quit IRC | 15:46 | |
| fungi | also my internet at the house here is out, so i may not be pushing any changes for a bit anyway | 15:46 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move openstackdocstheme api-ref job in-tree https://review.openstack.org/604610 | 15:47 |
| clarkb | fungi: maybe manually set it to 80 again on nl04? | 15:51 |
| clarkb | that seemed to trigger thr behavior quickly last time | 15:51 |
| *** yamamoto has joined #openstack-infra | 15:51 | |
| mrhillsman | how does one trigger periodic pipeline? i tried but maybe i am doing it wrong | 15:52 |
| mrhillsman | also using github | 15:53 |
| mrhillsman | zuul enqueue-ref --tenant openlab --trigger github --pipeline cloud-provider-openstack-acceptance-test-e2e-conformance-stable-branch-v1.12 --project kubernetes/cloud-provider-openstack --ref refs/heads/master | 15:53 |
| mrhillsman | i think the --ref is wrong maybe? | 15:53 |
| clarkb | the trigger is periodic not github I think | 15:54 |
| fungi | clarkb: i ran kick.sh against nl* a little while ago, so as mnaser noted i may have inadvertently cranked it back up to max anyway | 15:54 |
| clarkb | fungi: ah ok does that ignore the emergency file? | 15:54 |
| mrhillsman | ah ok | 15:57 |
| mrhillsman | trigger is timer | 15:57 |
| fungi | clarkb: http://grafana.openstack.org/d/BhcSH5Iiz/nodepool-ovh?orgId=1&var-region=ovh-bhs1 would suggest so | 15:58 |
| clarkb | ok we'll have to watch ti then. I will remove nl04 from the emrgency file now | 16:01 |
| fungi | thanks | 16:02 |
| fungi | i also will need to perform some local surgery to tether the machine from which i ssh to openstack servers if this internet outage persists, so logging into things in general is a bit of a pain at the moment | 16:02 |
| dhellmann | I see the post queue is continuing to grow. Where did we come down on the decision about queue priorities yesterday? | 16:12 |
| *** graphene has quit IRC | 16:12 | |
| clarkb | dhellmann: I think you mostly convinced corvus that it could be changed at this point (since we don't run coverage jobs (or at least many coverage jobs) there anymore) | 16:12 |
| clarkb | dhellmann: are you itnerested in pushing the change up to update the priority on post or do you want one of us to do it? | 16:13 |
| dhellmann | if I propose a patch, does landing it trigger the change or does zuul need to restart? | 16:13 |
| dhellmann | I'm happy to do it if we think it's a good idea | 16:13 |
| *** graphene has joined #openstack-infra | 16:13 | |
| clarkb | dhellmann: I want to say landing it is sufficient since that is part of the reloadable config. What I don't know is if existing node requests will have their priorities updated | 16:13 |
| clarkb | Shrews: ^ may know off the top of his head | 16:14 |
| dhellmann | what are valid values for "precedent"? high and low? or high, medium, and low? | 16:14 |
| *** fried_rice is now known as fried_rolls | 16:15 | |
| dhellmann | I see a promote pipeline in there now; is anything using that? | 16:15 |
| AJaeger | clarkb: we run coverage only on stable branches - but I tried to update master everywhere and will continue so... | 16:15 |
| clarkb | https://zuul-ci.org/docs/zuul/user/config.html#attr-pipeline.precedence high normal low | 16:15 |
| dhellmann | clarkb : thanks | 16:15 |
| clarkb | dhellmann: oddly only the infra CD jobs use promote that I know of. Its sort of a WIP | 16:15 |
| clarkb | AJaeger: yup I think it was your work refactoring that that helped | 16:15 |
| dhellmann | ok, I'll leave promote alone for now | 16:16 |
| AJaeger | let me reprhase: clarkb: we run coverage *in post* only on stable branches - but I tried to update master everywhere to move cover to check and will continue so... | 16:16 |
| AJaeger | clarkb: ianw and my work... | 16:16 |
| clarkb | AJaeger: gotcha, still much fewer coverage jobs running in post then | 16:16 |
| AJaeger | clarkb: yes, much fewer. I considered it not worth updating stable branches for this. | 16:17 |
| *** aojea has quit IRC | 16:17 | |
| *** dpawlik has quit IRC | 16:18 | |
| *** manjeets has joined #openstack-infra | 16:19 | |
| clarkb | thank you AJaeger ianw and mnaser for those project-config reviews | 16:20 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: make pipeline precedence progressively higher https://review.openstack.org/606129 | 16:20 |
| dhellmann | clarkb : let me know what you think of that ^ | 16:21 |
| Shrews | clarkb: priorities of existing requests will not change | 16:21 |
| dhellmann | ah, well, that's a shame | 16:21 |
| *** mriedem is now known as mriedem_lunch | 16:21 | |
| AJaeger | dhellmann, clarkb that makes periodic and check the same priority - right now periodic is low and check is normal. Is that kind of change ok? | 16:23 |
| dhellmann | hmm | 16:24 |
| dhellmann | it seems we have only 3 available levels but 4 desired levels | 16:24 |
| *** gyee has joined #openstack-infra | 16:24 | |
| AJaeger | yep | 16:24 |
| clarkb | ya that is an unfortunate gearman protocol exposure | 16:25 |
| dhellmann | ah | 16:25 |
| dhellmann | too bad it's not just an integer I guess | 16:25 |
| clarkb | we may be able to workaround it now that we use zk for the node requests, but would require changes | 16:25 |
| clarkb | (but the three values are a holdover from gearman for sure) | 16:25 |
| dhellmann | I think having periodic and check both set to low is probably ok | 16:26 |
| dhellmann | the point is to stop later parts of the process from being hung up if jobs keep entering the earlier part | 16:26 |
| Shrews | yeah, for zk requests, it's just a number. we can have as many as we want in reality | 16:26 |
| clarkb | infra-root https://review.openstack.org/#/c/605583/1 is a fix for the dns problems we've had on fedora during infra jobs (other jobs use the normal base job and should be fine) | 16:28 |
| clarkb | ianw found a bug in unbound in the process too | 16:28 |
| *** zul has joined #openstack-infra | 16:28 | |
| *** ykarel_ has joined #openstack-infra | 16:29 | |
| *** ykarel|away has quit IRC | 16:30 | |
| *** ykarel__ has joined #openstack-infra | 16:31 | |
| *** yamamoto has quit IRC | 16:31 | |
| *** ykarel_ has quit IRC | 16:31 | |
| clarkb | dhellmann: I think we should set the precedence on the third party check pipeline at the end of that file too. Otherwise lgtm | 16:33 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: make pipeline precedence progressively higher https://review.openstack.org/606129 | 16:33 |
| clarkb | (I left commentson the change) | 16:33 |
| dhellmann | clarkb : I just saw your comment and did update the precedence | 16:33 |
| clarkb | perfect | 16:33 |
| * dhellmann goes for food | 16:35 | |
| clarkb | fungi: reading the ovh bhs1 graph I don't think the cloud is entirely happy, but at the same time it isn't spiralling out of control | 16:36 |
| clarkb | fungi: likely other growing pains on the new version that happen to be less fatal to nodepool | 16:36 |
| fungi | yeah, looks like boot and delete are taking a while maybe | 16:36 |
| *** roman_g has joined #openstack-infra | 16:37 | |
| clarkb | hopefully nodepool's usage of that cloud region is constructive feedback for ovh :) | 16:38 |
| *** ianychoi has quit IRC | 16:39 | |
| clarkb | jungleboyj: is http://logs.openstack.org/22/600722/1/gate/legacy-tempest-dsvm-neutron-full/fe7320a/job-output.txt.gz#_2018-09-28_15_52_01_683952 a known issue with cinder/tempest? | 16:39 |
| AJaeger | dhellmann: ok to approve https://review.openstack.org/592472 now? Ironic should be ready... | 16:41 |
| clarkb | mriedem_lunch: melwitt in http://logs.openstack.org/06/604906/5/gate/openstack-tox-py35/c633c4e/ nova uses MoxStubout which is apparently deprecated and creates a very large log file. Might be something worth cleaning up to make it easier to read the logs when things fail | 16:43 |
| AJaeger | clarkb: could you +2A https://review.openstack.org/606114 - to help manila to finish python3-first, please? I expect it takes some more time to pass tests... | 16:43 |
| *** ianychoi has joined #openstack-infra | 16:44 | |
| clarkb | AJaeger: done | 16:44 |
| *** Dobroslaw has quit IRC | 16:45 | |
| *** dtantsur is now known as dtantsur|afk | 16:45 | |
| clarkb | mriedem_lunch: melwitt as for why that job failed it appears that the nova os profiler test never returned control back to the calling process after completing its test run | 16:45 |
| clarkb | then the job timed out | 16:46 |
| *** smarcet has quit IRC | 16:46 | |
| *** rossella_s has quit IRC | 16:51 | |
| *** jpich has quit IRC | 16:52 | |
| *** mdbooth has joined #openstack-infra | 16:53 | |
| mdbooth | Hello, looking at https://docs.openstack.org/infra/elastic-recheck/readme.html#adding-bug-signatures but I don't see which repo that stuff is in | 16:53 |
| clarkb | mdbooth: it is in the elastic-recheck repo itself | 16:54 |
| clarkb | openstack-infra/elastic-recheck | 16:54 |
| *** priteau has joined #openstack-infra | 16:54 | |
| mdbooth | clarkb: I don't know? I'll check it out and look, thanks. | 16:54 |
| clarkb | mdbooth: there is a link to them in that document | 16:54 |
| clarkb | the git.openstack.org link | 16:54 |
| *** rossella_s has joined #openstack-infra | 16:54 | |
| mdbooth | clarkb: I'm aware from PTG trivia night that infra has more repos than anybody else :P | 16:54 |
| mordred | mdbooth: the question is - does infra have more repos than the rest of openstack combined? | 16:55 |
| *** derekh has quit IRC | 16:55 | |
| clarkb | mriedem_lunch: following up on the yum install bug, the vast majority of the hits on that are from tripleo and in those tripleo cases it fails due to dns resolution failures to the infra mirror. Checking all but one of our mirrors have a dns ttl of an hour (I will fix the one with a 5 minute ttl), but this affects all the cloud regions so don't expect that is the cause | 16:55 |
| mdbooth | mordred: I wouldn't be surprised :) | 16:55 |
| clarkb | mriedem_lunch: I expect that something in the jobs themselves is causing problems for dns | 16:56 |
| clarkb | mriedem_lunch: those jobs don't fail 100% of the time because yum will try other mirrors if the first one doesn't resolve | 16:56 |
| jungleboyj | clarkb: That one doesn't look familar to me. | 16:57 |
| jungleboyj | smcginnis: ^^ | 16:57 |
| clarkb | jungleboyj: smcginnis ok I think http://status.openstack.org/elastic-recheck/gate.html#1794143 is the bug for that, we are just behind on indexing so the more recent occurences haven't shown up there | 16:57 |
| *** jpena is now known as jpena|off | 16:57 | |
| clarkb | (took me a while to dig that up) | 16:58 |
| *** ykarel__ has quit IRC | 17:00 | |
| clarkb | http://status.openstack.org/elastic-recheck/gate.html#1793370 causes job failures, but if zuul is doing its job properly all of those failures should be retried (because network connectivity losses like that should trigger a retry) | 17:00 |
| clarkb | I wonder if we can easily tell if zuul is retrying those jobs | 17:00 |
| mordred | clarkb: zuul isnt' going to retry those, as those are job-content failures in post jobs | 17:02 |
| clarkb | mordred: not all of them are post, some of them are copying ssh keys in pre | 17:04 |
| clarkb | (actually it seemd a lot of them were because if networking to the instance is flaky we hit it early rather than late) | 17:04 |
| clarkb | mordred: the title is too specific | 17:04 |
| clarkb | mordred: http://logs.openstack.org/98/603498/3/gate/openstack-tox-pep8/84b0c29/job-output.txt#_2018-09-27_21_44_42_572812 is an example. It actually fails early because netowrking doesn't work. Then we also fail trying to collect the logs in post | 17:05 |
| clarkb | that job should be restarted right? | 17:06 |
| clarkb | http://logs.openstack.org/50/603050/3/gate/openstack-tox-py36/f29117b/job-output.txt#_2018-09-26_08_47_23_846898 same with that one | 17:06 |
| *** bobh has quit IRC | 17:07 | |
| clarkb | http://logs.openstack.org/12/602112/3/gate/openstack-tox-py36/9612f0c/job-output.txt#_2018-09-26_08_47_22_578970 and so on | 17:07 |
| jungleboyj | clarkb: That elastic recheck bug looks different as that is on a retype, not an extend. | 17:08 |
| mordred | clarkb: yah - if pre failed we shoudl totally re-try it | 17:08 |
| clarkb | jungleboyj: ah ok | 17:08 |
| *** psachin has joined #openstack-infra | 17:08 | |
| mordred | clarkb: I wonder if we can detect that we're in a post job that's associated with a job that failed in pre and send an additional *something* to elasticsearch? | 17:09 |
| clarkb | mordred: ok thanks for confirming my understanding of that. I had deprioritzed debugging that problem because it looks like its across all the providers and we should retry in many of the cases | 17:09 |
| clarkb | mordred: maybe a zuul indiciation of whether or not the failure is fatal? | 17:09 |
| *** gfidenteN00b has quit IRC | 17:10 | |
| mordred | ++ | 17:10 |
| mordred | clarkb: because collecting info on things that failed in pre is still useful - but also being able to filter out those failures since zuul handles them as 'expected' types of failures we can retry on | 17:10 |
| jungleboyj | There are issues around actions like volume extension that can be teased out depending on the load on the system. I am guessing this is one of this edge cases where it takes longer than expected. | 17:10 |
| melwitt | clarkb: thanks for the heads up. will look at that | 17:11 |
| *** trown is now known as trown|lunch | 17:11 | |
| clarkb | mordred: yup exactly. | 17:12 |
| clarkb | mordred: checking the two py36 job failures against the changes they ran against there are no reported py36 failures to those changes. I think that maens the job retries are working as expected | 17:13 |
| clarkb | mordred: I expect too that in the old system the vast majority of these failures were weeded out by our ready script | 17:13 |
| clarkb | mordred: but now we've shifted that into zuul jobs themselves | 17:13 |
| *** bobh has joined #openstack-infra | 17:13 | |
| mordred | yah | 17:13 |
| *** mdbooth has quit IRC | 17:15 | |
| clarkb | mwhahaha: looking at logstash for http://status.openstack.org/elastic-recheck/gate.html#1708704 it seems that the yum install of dstat in the overcloud has this issue quite a bit. I think that points at something in the job around installing dstat that makes dns flaky? It is weird that that one package install (when all the other package installs are happening) is a problem | 17:17 |
| mwhahaha | clarkb: maybe we're pulling it from a different repo? not sure. i don't think that's the cause but rather a sympton | 17:18 |
| clarkb | mwhahaha: also we don't seem to collect the dstat logs in the overcloud, the logs are collected from the undercloud though | 17:18 |
| *** bobh has quit IRC | 17:21 | |
| mwhahaha | clarkb: so i pulled up the log stash and a job where that happened was actually a successfull job, we didn't fail on dstat | 17:22 |
| mwhahaha | http://logs.openstack.org/20/603220/1/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/43093ac/job-output.txt | 17:22 |
| clarkb | jungleboyj: http://logs.openstack.org/44/598244/2/gate/openstack-tox-lower-constraints/b21ebac/job-output.txt.gz#_2018-09-28_17_03_18_365816 that is the bug I linked? | 17:22 |
| clarkb | mwhahaha: correct, it is failing over to some other mirror | 17:23 |
| mrhillsman | in the gate pipeline i see "Queue: integrated", how do you set the value there; i.e. i want "Queue: arbitrary" | 17:24 |
| *** jtomasek has joined #openstack-infra | 17:24 | |
| clarkb | mwhahaha: mostly pointing it out because the behavior is odd and it is happenign a lot | 17:24 |
| mwhahaha | clarkb: so it's actually failing post deployment, that's really weird | 17:24 |
| mwhahaha | weshay_ruck: -^ fyi | 17:24 |
| jungleboyj | clarkb: No, that is a different one but we were made aware of that one yesterday. | 17:25 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use bionic for openstack-manuals publishing https://review.openstack.org/606147 | 17:25 |
| *** anteaya has joined #openstack-infra | 17:25 | |
| clarkb | mrhillsman: https://zuul-ci.org/docs/zuul/user/config.html#attr-project.%3Cpipeline%3E.queue | 17:25 |
| jungleboyj | That one is being looked into. | 17:25 |
| mrhillsman | ty sir | 17:25 |
| mwhahaha | i wonder if unbound is getting stomped on by a dnsmasq or something | 17:25 |
| AJaeger | config-core, could I get a +2A on https://review.openstack.org/606147 to fix openstakc-manuals publishing, please? | 17:25 |
| *** yamamoto has joined #openstack-infra | 17:26 | |
| *** rossella_s has quit IRC | 17:27 | |
| *** harlowja has joined #openstack-infra | 17:27 | |
| mwhahaha | clarkb: i figured it out, it's designate | 17:28 |
| clarkb | mwhahaha: neat | 17:28 |
| mwhahaha | clarkb: it's conflicting with unbound on scenario003 | 17:28 |
| mwhahaha | so post-deployment, dns no longer works | 17:28 |
| mwhahaha | beekneemech, weshay_ruck -^ | 17:28 |
| mwhahaha | let me file a bug | 17:28 |
| *** rossella_s has joined #openstack-infra | 17:30 | |
| *** roman_g has quit IRC | 17:31 | |
| *** eernst has quit IRC | 17:32 | |
| beekneemech | mwhahaha: Isn't unbound running on the undercloud though? | 17:34 |
| mwhahaha | beekneemech: multinode, it's run on both. and the default resolve.conf points to 127.0.0.1 | 17:34 |
| beekneemech | Or does it run on both? | 17:34 |
| beekneemech | Ah. :-/ | 17:34 |
| mwhahaha | beekneemech: https://bugs.launchpad.net/tripleo/+bug/1795043 | 17:34 |
| openstack | Launchpad bug 1795043 in tripleo "designate's named is conflicting with unbound in CI scenario003" [High,Triaged] | 17:34 |
| mwhahaha | not completely sure why, but it's pretty consistent on scenario003 according to logstash | 17:34 |
| clarkb | mriedem_lunch: for the pip no packages found issue, the two most recent occurrences of that were ara trying to install a package that didn't support the local version of python. Previous to that we had the broken mirrors in limestone and gra1 both of which should be fixed | 17:34 |
| clarkb | mriedem_lunch: we should keep tracking that but I think its a non issue for the last ~4 days | 17:35 |
| mwhahaha | speaking of pip, did the version of ansible recently get updated on the images? | 17:35 |
| mwhahaha | we've noticed something is pip installing ansible 2.6.4 | 17:35 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix node leak on job removal https://review.openstack.org/605527 | 17:35 |
| mwhahaha | which has broken some of our stable jobs | 17:35 |
| * weshay_ruck reading through it | 17:36 | |
| clarkb | mwhahaha: devstack-gate installs its own ansible in a virtualenv which was updated. I don't think other things are expected to use that ansible install | 17:37 |
| clarkb | it is a devstack gate implementation detail and not a contract with everyone else | 17:37 |
| mwhahaha | clarkb: yea we're seeing it on the actual host itself | 17:37 |
| mwhahaha | starting as of 2 days ago | 17:37 |
| clarkb | mwhahaha: I don't think we install ansible on the test nodes themselves out side of the jobs | 17:37 |
| mwhahaha | somethign is, not sure what though because we use packages or a venv | 17:38 |
| *** smarcet has joined #openstack-infra | 17:38 | |
| mwhahaha | http://logs.openstack.org/24/567224/109/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/8b476ee/logs/undercloud/var/log/extra/pip.txt.gz vs http://logs.openstack.org/24/567224/111/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/448effe/logs/undercloud/var/log/extra/pip.txt.gz | 17:38 |
| *** eernst has joined #openstack-infra | 17:39 | |
| clarkb | I don't think it is infra | 17:39 |
| mwhahaha | but it should be 2.4.4.0 from the package http://logs.openstack.org/24/567224/111/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/448effe/logs/undercloud/var/log/extra/rpm-list.txt.gz | 17:39 |
| mwhahaha | k i'll continue to poke at it | 17:39 |
| fungi | could it be the old devstack-gate logic which uses ansible to orchestrate the overlay networking for multi-node configurations (or something oooq lifted from d-g a while back)? | 17:40 |
| clarkb | fungi: that is in a dedicated d-g virtualenv in /tmp | 17:40 |
| fungi | ahh, nm | 17:40 |
| clarkb | mwhahaha: could it possibly be ara? | 17:40 |
| clarkb | I don't think ara deps on asnible but maybe that changed? | 17:40 |
| clarkb | fungi: we intentionally did it that way for this reason. People run ansible in the jobs and ansible is not really consistent over minor version releases for compat | 17:41 |
| mwhahaha | yea not sure, yet. our logs point to older versions being pip installed | 17:41 |
| clarkb | ansible does not show up in our dib elements so doubt it is coming from that | 17:41 |
| logan- | yes newer ara versions have an ansible version requirement, if you dont pin ara it will pull in newer ansible. broke a few jobs where i had it unpinned last week | 17:42 |
| mwhahaha | k i'll keep trying to disect the logs then | 17:42 |
| fungi | thanks logan-! | 17:43 |
| clarkb | dmsimard: ^ you are probably interested in this | 17:43 |
| dmsimard | I am dmsimard and I might be interested in this | 17:43 |
| *** e0ne has quit IRC | 17:44 | |
| dmsimard | logan-: newer ansible ? the pin is currently >=2.4.5 which is the lowest version not currently EOL :p | 17:45 |
| dmsimard | https://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html | 17:45 |
| dmsimard | mwhahaha: ^ | 17:46 |
| dmsimard | clarkb: ara 0.x does depend on ansible because it leverages ansible to configure itself (ansible.cfg etc) | 17:47 |
| mwhahaha | K that's probably the issue | 17:47 |
| clarkb | depending on how you use pip >=2.4.5 can pull in 2.6 say if you already have 2.5 installed | 17:48 |
| *** jamesmcarthur has quit IRC | 17:48 | |
| mwhahaha | weshay_ruck: we probably need to pin ara in quickstart | 17:50 |
| *** agopi|afk is now known as agopi | 17:51 | |
| *** harlowja has quit IRC | 17:51 | |
| *** mriedem_lunch has quit IRC | 17:52 | |
| *** diablo_rojo has joined #openstack-infra | 17:52 | |
| *** harlowja has joined #openstack-infra | 17:52 | |
| *** auristor has quit IRC | 17:54 | |
| *** mriedem has joined #openstack-infra | 17:54 | |
| weshay_ruck | mwhahaha, ara==0.15.0 | 17:55 |
| dmsimard | 0.15.0 is kind of old | 17:56 |
| *** manjeets has quit IRC | 17:56 | |
| *** david-lyle has quit IRC | 17:56 | |
| dmsimard | May 3rd | 17:56 |
| dmsimard | 0.16.1 was released 24 days ago | 17:56 |
| dmsimard | this is the pin for 0.15.0: https://github.com/openstack/ara/blob/41427039de3b9ed1859bb3afdc1f8629e6c72a7a/requirements.txt#L4 | 17:57 |
| *** e0ne has joined #openstack-infra | 18:02 | |
| *** TheJulia is now known as needssleep | 18:02 | |
| *** auristor has joined #openstack-infra | 18:02 | |
| *** e0ne has quit IRC | 18:05 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: ensure the twine check command runs in the correct directory https://review.openstack.org/606152 | 18:05 |
| dhellmann | config-core: I think ^^ fixes an issue with the new packaging check job, as exhibited in the failure on http://logs.openstack.org/24/591624/2/gate/test-release-openstack-python3/3cf3d56/ara-report/result/3e1b200d-0ec8-4b96-9882-ece83edbe0e8/ | 18:06 |
| clarkb | looking | 18:06 |
| *** jcoufal has quit IRC | 18:08 | |
| clarkb | dhellmann: I'm not sure if zuul_work_dir is valid in that context, but gave a path that is based on zuul inventory vars that should work | 18:11 |
| clarkb | (we use this alternate path in the post.yaml playbook) | 18:11 |
| dhellmann | clarkb : hmm, ok. I copied that out of the other playbook but maybe it was in a role or something | 18:12 |
| dhellmann | I see lots of other uses of that variable in | 18:13 |
| dhellmann | http://codesearch.openstack.org/?q=zuul_work_dir&i=nope&files=&repos= | 18:13 |
| *** smarcet has quit IRC | 18:13 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove py3 jobs for manila-ui project https://review.openstack.org/606114 | 18:13 |
| clarkb | I think they define it in their defaults file let me check | 18:13 |
| dhellmann | since I'm still learning, when you say "valid in that context" what is different about that context than any of the others? | 18:13 |
| dhellmann | ah | 18:13 |
| clarkb | http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/bindep/defaults/main.yaml like that | 18:13 |
| clarkb | looks like that value might be better than the one I gave though as it is probably rooted | 18:14 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: ensure the twine check command runs in the correct directory https://review.openstack.org/606152 | 18:14 |
| clarkb | dhellmann: the ansible vars are global so if one of the roles in the check.yaml playbook defines zuul_work_dir it will be valid, but if they don't it won't be valid | 18:14 |
| dhellmann | yay for namespaces | 18:14 |
| clarkb | dhellmann: but I think it is bad to rely on side effects like that | 18:14 |
| dhellmann | ok, I've updated it to use zuul.project.src_dir | 18:14 |
| dhellmann | yeah, I don't want to have it suddenly fail if something else is changed | 18:15 |
| *** anteaya has quit IRC | 18:15 | |
| dhellmann | I saw zuul in the name and assumed it was being defined by zuul. TIL | 18:15 |
| clarkb | dhellmann: the zuul.foo vars should be defined by zuul in the inventory and are safe to use anywhere in the job | 18:16 |
| dhellmann | yeah, but not zuul_foo | 18:17 |
| clarkb | thinking about this removing aliases like http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/bindep/defaults/main.yaml#n5 might be a good idea | 18:17 |
| clarkb | it really isn't any shorter to type or harder to understand but will be more consistent | 18:17 |
| dhellmann | yeah | 18:17 |
| clarkb | (I think some of the motivation there is to make these roles useable outside of zuul, but not sure how realistic that is) | 18:18 |
| *** smarcet has joined #openstack-infra | 18:18 | |
| *** hasharAway is now known as hasharRlyAwy | 18:22 | |
| *** manjeets has joined #openstack-infra | 18:25 | |
| *** e0ne has joined #openstack-infra | 18:25 | |
| *** bobh has joined #openstack-infra | 18:26 | |
| *** yamamoto has quit IRC | 18:28 | |
| *** bobh has quit IRC | 18:31 | |
| AJaeger | dhellmann: I'll approve the ironic python3-first change now... | 18:33 |
| dhellmann | AJaeger : ack | 18:33 |
| dhellmann | and thank you | 18:33 |
| clarkb | dhellmann: AJaeger I see you have debugged the stackviz issues with python3 in the past. Seems like we are still hitting that, is that a known issue? | 18:34 |
| dhellmann | stackviz doesn't ring any bells | 18:34 |
| clarkb | ok thanks I'm working on a reproducer locally | 18:35 |
| AJaeger | clarkb: I have? Sorry, forgotten ;( | 18:36 |
| clarkb | based on git logs it looks like you all added python3.6 testing? its ok I think I have a minimal ish reproduction | 18:36 |
| *** trown|lunch is now known as trown | 18:37 | |
| AJaeger | clarkb: yes, but debugging was only fixing bindep.txt - and then it worked by magic ;) | 18:37 |
| *** bobh has joined #openstack-infra | 18:38 | |
| clarkb | http://logs.openstack.org/71/605271/1/check/tempest-full-py3/cb623b6/job-output.txt#_2018-09-28_04_00_03_179985 is what I am looking at and appears to be some interaction between shutil.copyfileobj and input that isn't utf8 | 18:38 |
| clarkb | ya sys.stdin has an ecoding that is platform dependentm utf8 in this case, but we are trying to copy the data into another buffer and that triggers a fault because the input isn't utf8 | 18:40 |
| clarkb | I can trigger the bug by reading from sys.stdin as well | 18:40 |
| *** rossella_s has quit IRC | 18:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add api-ref job to ironic-inspector https://review.openstack.org/599890 | 18:41 |
| *** david-lyle has joined #openstack-infra | 18:41 | |
| *** rossella_s has joined #openstack-infra | 18:42 | |
| *** david-lyle is now known as dklyle | 18:42 | |
| *** bobh has quit IRC | 18:42 | |
| *** smarcet has quit IRC | 18:43 | |
| *** mriedem has quit IRC | 18:44 | |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for ironic repositories https://review.openstack.org/592472 | 18:44 |
| *** felipemonteiro has quit IRC | 18:47 | |
| *** e0ne has quit IRC | 18:50 | |
| *** rossella_s has quit IRC | 18:50 | |
| *** smarcet has joined #openstack-infra | 18:55 | |
| *** rossella_s has joined #openstack-infra | 18:57 | |
| *** anteaya has joined #openstack-infra | 18:59 | |
| mnaser | is logstash having issues | 19:02 |
| mnaser | or is it just really behind | 19:02 |
| *** rossella_s has quit IRC | 19:02 | |
| *** rossella_s has joined #openstack-infra | 19:03 | |
| AJaeger | mnaser: 86k jobs behind according to http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1 | 19:03 |
| mnaser | AJaeger: is that..normal? :p | 19:03 |
| AJaeger | mnaser: this week nothing is normal ;( | 19:03 |
| AJaeger | mnaser: I don't think it's normal but don't check this often... | 19:04 |
| *** smarcet has quit IRC | 19:04 | |
| clarkb | it isn't normal, its due to the large number of jobs we are runnign | 19:07 |
| clarkb | the gate resets cause that because we run a bunch of jobs then start them all over again | 19:07 |
| *** bobh has joined #openstack-infra | 19:09 | |
| mwhahaha | dmsimard: is ara packaged in rpm form yet? | 19:09 |
| *** elbragstad has quit IRC | 19:10 | |
| *** elbragstad has joined #openstack-infra | 19:10 | |
| AJaeger | corvus: do you want to review dhellmann's change of priorities for our pipelines? https://review.openstack.org/#/c/606129/ | 19:13 |
| *** florianf is now known as florianf|afk | 19:14 | |
| *** bobh has quit IRC | 19:23 | |
| *** bobh has joined #openstack-infra | 19:23 | |
| clarkb | dhellmann: AJaeger fwiw there is no python side test suite for stackviz | 19:24 |
| *** graphene has quit IRC | 19:24 | |
| clarkb | I'm adding a couple tests and in the process have found other bugs with python3, but this fix should fix our usage of it I think | 19:24 |
| AJaeger | ok | 19:24 |
| dhellmann | clarkb : those patches are part of the goal work this cycle. | 19:25 |
| pabelanger | dhellmann: AJaeger: clarkb: not related to current topic, but with pipelines, anything in the release / pre-release / release-post pipelines is managed releases team, right? | 19:26 |
| dhellmann | pabelanger : those jobs are triggered by tags, and some tags are added after we review release things, but some are also pushed directly by unofficial teams or teams that aren't part of "The OpenStack Release" (tm) | 19:27 |
| AJaeger | pabelanger: no - unofficial repos can self-tag | 19:27 |
| AJaeger | pabelanger: why? | 19:28 |
| pabelanger | dhellmann: AJaeger: okay, let me ask differently, is the release team using the tags pipeline? | 19:28 |
| dhellmann | pabelanger : I think the release notes build jobs run there | 19:29 |
| pabelanger | AJaeger: nothing specific to openstack, trying to work out pipelines for github based zuul. | 19:29 |
| *** e0ne has joined #openstack-infra | 19:29 | |
| pabelanger | and couldn't remember what went into tags vs release pipelines | 19:29 |
| fungi | pabelanger: you can check the regexes. all three of tag, pre-release and release are triggered from tags | 19:30 |
| dhellmann | release and pre-release used to run separate jobs. I think now we have them configured to run the same job for python but I don't know about other languages | 19:30 |
| clarkb | pabelanger: tags is any tag. release is semver pbr appropriate tags | 19:30 |
| pabelanger | Yah, that's what I was seeing. Guess releases is mostly semver things | 19:31 |
| mnaser | hey uh | 19:31 |
| mnaser | im not late to the party but | 19:31 |
| dhellmann | pre-release is alpha, beta, rc | 19:31 |
| mnaser | packethost is borked right | 19:31 |
| fungi | actually we never did trim release down to just pbr-appropriate version patterns because it's used for non-python projects and things like xstatic packages which may have additional version components | 19:31 |
| mnaser | 0 in use, 0 ready, 95 deleting? 28 building | 19:31 |
| dhellmann | release is a version number without alpha, beta, or rc | 19:31 |
| pabelanger | mnaser: yah, for a while. clarkb they tried working on it at PTG | 19:31 |
| fungi | mnaser: packethost has been basically broken since before the ptg | 19:32 |
| mnaser | oh since the PTG | 19:32 |
| mnaser | ouch | 19:32 |
| mnaser | um | 19:32 |
| mnaser | what if i ask nicely for root access | 19:32 |
| mnaser | on the machines/platform | 19:32 |
| pabelanger | dhellmann: okay, cool. That helps | 19:32 |
| fungi | studarus has root access to them, i think? | 19:32 |
| dmsimard | mwhahaha: It's packaged in fedora, for CentOS the only packages there are were made by tristanC for software factory | 19:32 |
| dmsimard | mwhahaha: it probably wouldn't be too hard to pull his package in RDO if you wanted | 19:33 |
| dmsimard | I don't have the bandwidth to do the legwork though | 19:33 |
| mnaser | fungi: i mean if you/infra-core agrees to it, i'd like to have my hand at fixing whats wrong.. | 19:33 |
| mnaser | so maybe if you want to email him about it | 19:33 |
| mwhahaha | dmsimard: k i'll try and round up someone | 19:33 |
| mnaser | 100 nodes would be nice | 19:33 |
| dhellmann | pabelanger : as I said, the jobs that ran for those types of version numbers used to be different. That change we just merged to update to the new python packaging job may make that less important now, but I haven't reviewed the jobs for other types of artifacts lately | 19:34 |
| *** e0ne has quit IRC | 19:34 | |
| pabelanger | dhellmann: understood | 19:34 |
| fungi | mnaser: i don't think we have access to the machines | 19:35 |
| fungi | studarus might | 19:35 |
| mnaser | fungi: he does. right, but i'm guessing a request from infra-root to let me rather than me emailing asking for it might be more reasonable :) | 19:35 |
| fungi | ahh, i see | 19:35 |
| pabelanger | mnaser: I mean, I'd +2 a patch to disable it, to help reduce launcher errors in grafana for nodepool. But that is just me :) | 19:36 |
| clarkb | dhellmann: AJaeger mtreinish https://review.openstack.org/606184 | 19:36 |
| mnaser | i'd love to fix it and i think i could get it done with the right access | 19:36 |
| fungi | mnaser: he might go for that, worth asking i suppose. let's see what clarkb thinks when he has a moment | 19:36 |
| clarkb | oh looks like there is already a change for that | 19:37 |
| clarkb | this will teahc me to check for open changes before writing a change, but now I feel like I can review the other change | 19:37 |
| clarkb | https://review.openstack.org/#/c/555388/3 is the other change and it doesn't pass tests, I have rechecked it to get logs to figure out why | 19:39 |
| clarkb | my fix also fixes the file in put case | 19:39 |
| clarkb | mnaser: fungi: I'm fine with it, but I am not sure how much access studarus has either | 19:39 |
| mnaser | clarkb: ill write up an email | 19:39 |
| clarkb | that was one of the things I found out at the PTG, he is an openstack admin but not necesarily root on the control plane? something like that | 19:39 |
| clarkb | anyway the qa team should really get on top of that or we should consider removing stackviz from our jobs | 19:40 |
| *** hasharRlyAwy is now known as hasharAway | 19:42 | |
| AJaeger | config-core, could you put the following changes on your review queue, please? https://review.openstack.org/605583 https://review.openstack.org/604610 https://review.openstack.org/606147 https://review.openstack.org/605128 https://review.openstack.org/604889 | 19:43 |
| timothyb89 | clarkb: I think your patch is better, though removing stackviz would be a good option if nobody is using it | 19:49 |
| clarkb | timothyb89: mostly I mention that option beacuse I realized how old your chagne is without it getting much attention from the team that should be responsible | 19:49 |
| *** mriedem has joined #openstack-infra | 19:50 | |
| clarkb | timothyb89: I think fixing it is a fine option too if the QA team gets a fix in (I'm fine with either patch, if yours goes in first I will rebase mine to add the file provider fix too) | 19:50 |
| clarkb | looks like core membership is actually pretty minimal there, should we add the rest of the qa team to it? | 19:50 |
| * AJaeger thanks clarkb for reviewing and waves good night | 19:51 | |
| timothyb89 | clarkb: yours seems to supercede it in all ways that matters so I'm happy to abandon | 19:51 |
| timothyb89 | clarkb: but stackviz has been essentially unmaintained for > 1 year, if nobody's benefitting from it removal would be pragmatic | 19:52 |
| clarkb | timothyb89: I would've updated your change if I had noticed it but wasn't until I saw the conflicts with that I noticed :( sorry | 19:52 |
| *** Emine has quit IRC | 19:52 | |
| timothyb89 | clarkb: it's all good, mainly I don't want my old intern project to be a continual time sink for you guys :) | 19:52 |
| clarkb | eh I think it is useful when it works (which is the python2 jobs currently) | 19:53 |
| clarkb | these are python3 transition pains | 19:53 |
| clarkb | to be expected | 19:53 |
| clarkb | the time based graph that shows test overlap and resource usage is actually quite useful imo | 19:54 |
| *** mdbooth has joined #openstack-infra | 19:54 | |
| clarkb | timothyb89: just earlier today jungleboyj mentioned that a bug with cinder seemed to be resource contention related and stackviz shows us the info to figure that out | 19:54 |
| timothyb89 | clarkb: huh, well, glad to hear it's still in use :) | 19:55 |
| *** Emine has joined #openstack-infra | 19:55 | |
| openstackgerrit | Merged openstack-infra/project-config master: Adding openstack/octavia-lib project https://review.openstack.org/604889 | 20:01 |
| mriedem | clarkb: have you seen this one yet? http://logs.openstack.org/28/605828/1/check/neutron-grenade-multinode/40ddb0f/logs/grenade.sh.txt.gz#_2018-09-28_00_17_54_198 | 20:03 |
| *** smarcet has joined #openstack-infra | 20:04 | |
| clarkb | mriedem: I have not | 20:06 |
| *** openstackgerrit has quit IRC | 20:07 | |
| *** rossella_s has quit IRC | 20:08 | |
| *** rossella_s has joined #openstack-infra | 20:09 | |
| *** mdbooth has quit IRC | 20:10 | |
| *** psachin has quit IRC | 20:10 | |
| mriedem | guh logs/undercloud/var/log/extra/logstash.txt | 20:10 |
| mriedem | there is that giant single indexed file | 20:10 |
| mriedem | clarkb: is ^ killing e-s? | 20:11 |
| clarkb | mriedem: probably, I'd have to go grep logs though | 20:11 |
| clarkb | mriedem: I can do that after lunch | 20:11 |
| *** bhavikdbavishi has quit IRC | 20:12 | |
| mriedem | apparently this shows up a lot but it's mostly not causing failures | 20:12 |
| mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Exception%20in%20thread%5C%22%20AND%20message%3A%5C%22(most%20likely%20raised%20during%20interpreter%20shutdown)%5C%22%20AND%20NOT%20tags%3A%5C%22logs%2Fundercloud%2Fvar%2Flog%2Fextra%2Flogstash.txt%5C%22&from=7d | 20:12 |
| clarkb | mriedem: just sent a followup on the zuul queue backlog thread too | 20:12 |
| mriedem | clarkb: thanks | 20:12 |
| *** smarcet has quit IRC | 20:14 | |
| mriedem | prometheanfire: have any paramiko releases / upper-constraints been approved lately? | 20:14 |
| mriedem | actually nvm, | 20:15 |
| mriedem | i think this is just the job getting killed from being slow | 20:15 |
| mriedem | http://logs.openstack.org/28/605828/1/check/neutron-grenade-multinode/40ddb0f/job-output.txt.gz#_2018-09-28_02_21_33_475740 | 20:16 |
| clarkb | mriedem: http://logs.openstack.org/70/605070/1/check/kuryr-kubernetes-tempest-daemon-containerized-octavia-py36/d6c9fbd/controller/logs/screen-kubelet.txt.gz has killed at least one log worker. It is 32MB large after compression | 20:16 |
| mriedem | yup http://status.openstack.org/elastic-recheck/#1686542 | 20:17 |
| mriedem | cripes almighty | 20:17 |
| clarkb | we should get kuryr to clean that up | 20:17 |
| clarkb | mriedem: http://logs.openstack.org/66/582466/34/check/tripleo-ci-centos-7-containers-multinode-queens/b42e065/job-output.txt.gz?level=INFO killed another worker | 20:18 |
| clarkb | (they OOM) | 20:18 |
| clarkb | as did http://logs.openstack.org/59/603059/6/check/monasca-tempest-python-cassandra/5b6697c/logs/screen-monasca-persister.txt.gz 18MB large after compression | 20:19 |
| clarkb | should have monsasca clean that up too | 20:19 |
| *** smarcet has joined #openstack-infra | 20:19 | |
| mwhahaha | what killed the worker? the job output? | 20:19 |
| clarkb | mwhahaha: yes, basically the job output being too large and we OOM | 20:20 |
| mriedem | dmellado: see re http://logs.openstack.org/70/605070/1/check/kuryr-kubernetes-tempest-daemon-containerized-octavia-py36/d6c9fbd/controller/logs/screen-kubelet.txt.gz | 20:20 |
| mwhahaha | also that's a log from 9 weeks ago? | 20:20 |
| clarkb | mwhahaha: ya that is when the worker died, I'm just going through worker by worker and finding what crashed them | 20:20 |
| mwhahaha | clarkb: you sure that's the cause or a side effect? | 20:21 |
| clarkb | mwhahaha: pretty sure it is teh cause. We get these multiple hundred of megabyte log files that cause the prpcessors to OOM and the worker crashes | 20:21 |
| clarkb | OOMkiller gets invoked on them | 20:21 |
| mwhahaha | clarkb: yea but job-output.txt from tripleo is not that big | 20:21 |
| clarkb | mwhahaha: ya in this case with it being an old log that isn't on the log servers anymore we probably can't know for sure | 20:22 |
| mwhahaha | the tripleo job-output is typically ~100k | 20:22 |
| clarkb | mwhahaha: we can likely ignore that one for now since its in ahard to debug state | 20:22 |
| mriedem | i pinged witek over in -monasca | 20:23 |
| mwhahaha | k do let me know if you wander across any larger ones. we do try and keep those down | 20:23 |
| mriedem | so these logs aren't using the oslo log format so rather than just index INFO logs we're indexing everything? | 20:23 |
| mriedem | is the registration to index these logs at all still in system-config? | 20:24 |
| clarkb | mwhahaha: will do | 20:24 |
| clarkb | mriedem: yes we are indexing everything in that case | 20:24 |
| prometheanfire | mriedem: :D | 20:24 |
| clarkb | mriedem: no it moved with zuulv3 let me find it | 20:24 |
| clarkb | mriedem: project-config/playbooks/base/post-logs.yaml is what calls the submit-logstash-jobs role in the roles dir of that repo | 20:25 |
| *** yamamoto has joined #openstack-infra | 20:26 | |
| *** rossella_s has quit IRC | 20:26 | |
| *** rlandy is now known as rlandy|brb | 20:27 | |
| *** rossella_s has joined #openstack-infra | 20:29 | |
| clarkb | http://logs.openstack.org/36/528336/12/check/neutron-tempest-dvr-ha-multinode-full/f897fea/logs/screen-q-svc.txt.gz some really largel neutron files | 20:29 |
| mriedem | i don't know how to turn this off for these jobs | 20:30 |
| mriedem | that neutron HA one is a 3-node job | 20:30 |
| clarkb | I think the config is a regex we could negative lookahead on them? | 20:31 |
| mriedem | i couldn't find the definition of submit-logstash-jobs via codesearch.o.o | 20:31 |
| clarkb | mriedem: project-config/roles/ | 20:32 |
| clarkb | looks like the config is in defaults/main.yaml | 20:32 |
| mriedem | ok i see it | 20:33 |
| mriedem | http://git.openstack.org/cgit/openstack-infra/project-config/tree/roles/submit-logstash-jobs/defaults/main.yaml#n79 yeah? | 20:34 |
| mriedem | so kubelet and monasca-persister | 20:34 |
| mriedem | on it | 20:34 |
| clarkb | ya that file | 20:34 |
| *** openstackgerrit has joined #openstack-infra | 20:35 | |
| openstackgerrit | Merged openstack-infra/project-config master: ensure the twine check command runs in the correct directory https://review.openstack.org/606152 | 20:35 |
| mriedem | https://storyboard.openstack.org/#!/story/2003911 | 20:36 |
| clarkb | http://logs.openstack.org/76/597876/1/check/networking-ovn-tempest-dsvm-ovs-release/976ea81/logs/screen-ovn-northd.txt.gz is 16MB large and caused a crash | 20:37 |
| *** anteaya has quit IRC | 20:37 | |
| *** agopi is now known as agopi|brb | 20:37 | |
| clarkb | found another monasca persister crash too | 20:37 |
| clarkb | http://logs.openstack.org/99/602599/1/gate/mistral-rally-task/9c9a906/controller/logs/screen-mistral-engine.txt.gz 33MB | 20:38 |
| mriedem | https://bugs.launchpad.net/kuryr-kubernetes/+bug/1795067 | 20:39 |
| openstack | Launchpad bug 1795067 in kuryr-kubernetes "screen-kubelet.txt is causing logstash index OOM errors" [Undecided,New] | 20:39 |
| mriedem | that's mistral | 20:39 |
| clarkb | yup and ovn | 20:39 |
| clarkb | (sorry I'm just sort of throwing them out as I got through and investigate | 20:39 |
| mriedem | https://bugs.launchpad.net/mistral/+bug/1795068 | 20:41 |
| openstack | Launchpad bug 1795068 in Mistral "screen-mistral-engine.txt size is causing logstash index OOM" [Undecided,New] | 20:41 |
| *** agopi|brb has quit IRC | 20:41 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config master: Blacklist logstash indexing of some very large screen logs https://review.openstack.org/606197 | 20:45 |
| clarkb | that ovn one and the mistral one are pretty common | 20:46 |
| clarkb | as is the monasca persister | 20:46 |
| mriedem | oh i'll update with the ovn one | 20:46 |
| mriedem | it used to be that you had to opt into logstash indexing.... | 20:47 |
| *** kgiusti has left #openstack-infra | 20:47 | |
| mriedem | now everyone gets it for free? | 20:47 |
| mriedem | i mean, by default? | 20:47 |
| mriedem | that seems pretty reckless when there are projects that don't know how their CI is setup | 20:48 |
| *** rlandy|brb is now known as rlandy | 20:49 | |
| clarkb | mriedem: we alweays indexed all jobs | 20:51 |
| clarkb | what has changed is the ruleset for finding logfiles | 20:51 |
| clarkb | so its a bit more greedy now particularly with screen-* log files | 20:51 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config master: Blacklist logstash indexing of some very large screen logs https://review.openstack.org/606197 | 20:52 |
| mriedem | well hopefully ^ helps | 20:53 |
| clarkb | mriedem: +2 thanks. any other config-core willing to review ^ | 20:53 |
| mnaser | clarkb: mriedem +W | 20:54 |
| *** bobh has quit IRC | 20:54 | |
| boden | is there a kosher way to run another projects python UTs in the gate? for example: https://review.openstack.org/#/c/605861 | 20:54 |
| clarkb | boden: look at the requirements repo, they run unittests of a variety of projects as part of checking new dependencies | 20:55 |
| clarkb | boden: it does so by invoking the tox in the target repo not the tox in the test with repo if that makes sense | 20:56 |
| boden | clarkb: will it require playbooks? | 20:56 |
| clarkb | boden: https://git.openstack.org/cgit/openstack/requirements/tree/.zuul.d/cross-jobs.yaml | 20:56 |
| *** panda has quit IRC | 20:57 | |
| *** panda has joined #openstack-infra | 20:58 | |
| boden | clarkb: ack thanks | 20:58 |
| *** shardy has quit IRC | 21:00 | |
| *** PapaOurs is now known as bauzas | 21:05 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Add zuul user to bridge.openstack.org https://review.openstack.org/604925 | 21:11 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Manage user ssh keys from urls https://review.openstack.org/604932 | 21:11 |
| clarkb | having CI of the infra things is really handy | 21:12 |
| *** yamamoto has quit IRC | 21:15 | |
| *** rossella_s has quit IRC | 21:17 | |
| *** rfolco has quit IRC | 21:18 | |
| *** rossella_s has joined #openstack-infra | 21:20 | |
| *** hasharAway has quit IRC | 21:22 | |
| *** agopi|brb has joined #openstack-infra | 21:28 | |
| *** slaweq has quit IRC | 21:28 | |
| *** bobh has joined #openstack-infra | 21:30 | |
| *** rossella_s has quit IRC | 21:31 | |
| *** rossella_s has joined #openstack-infra | 21:31 | |
| clarkb | ze07 seems to have leaked build dirs (we aren't cleaning those up on start I guess?) due to its having crashed and needing a reboot | 21:35 |
| clarkb | I am going to clean out the older directories by hand to avoid running out of disk there | 21:35 |
| *** mriedem has quit IRC | 21:37 | |
| *** boden has quit IRC | 21:37 | |
| *** priteau has quit IRC | 21:42 | |
| *** rossella_s has quit IRC | 21:49 | |
| clarkb | mnaser: mriedem the logstash queue is trending in the downward direction at the rate of ~3k jobs per hour | 21:51 |
| clarkb | we should catch up over the weekend | 21:51 |
| clarkb | infra-root I offered to restart the apache server on the etherpad server to clear out any stale connections prior to ansiblefest as they will be using our etherpad server I guess. | 21:51 |
| clarkb | I am going to do that now | 21:52 |
| fungi | thanks clarkb | 21:52 |
| clarkb | and done | 21:52 |
| clarkb | now to fix mirror.dfw.rax.openstack.org dns | 21:53 |
| fungi | we've had recent-ish trouble tickets about two or three zuul executors whose host hypervisor servers underwent emergency reboots, so ze07 may not be the only one | 21:54 |
| clarkb | mirror.dfw.rax.openstack.org should have a CNAME record with ttl of 3600 now instead of 300 | 21:56 |
| clarkb | the A and AAAA records were fine | 21:57 |
| clarkb | fungi: the zuul status grafana graphs don't show others as having the same issue (ze02 does too but for other reasons, it is an old one with bigger git repos iirc) | 21:57 |
| fungi | ahh | 21:58 |
| *** smarcet has joined #openstack-infra | 22:00 | |
| *** bobh has quit IRC | 22:03 | |
| clarkb | finding a day that works for this opendev discussion is difficult. Maybe I will try to draft an email instead | 22:17 |
| clarkb | silly ansiblefest travel | 22:17 |
| *** EvilienM is now known as EmilienM | 22:20 | |
| *** fried_rolls is now known as efried | 22:20 | |
| *** panda is now known as panda|off | 22:26 | |
| *** rlandy has quit IRC | 22:27 | |
| *** jamesmcarthur has joined #openstack-infra | 22:36 | |
| *** smarcet has quit IRC | 22:37 | |
| *** eernst has quit IRC | 22:40 | |
| *** smarcet has joined #openstack-infra | 22:40 | |
| *** jamesmcarthur has quit IRC | 22:40 | |
| *** tosky has quit IRC | 22:42 | |
| *** ijw has joined #openstack-infra | 22:48 | |
| *** elbragstad has quit IRC | 22:48 | |
| *** tpsilva has quit IRC | 22:52 | |
| *** pbourke has quit IRC | 22:56 | |
| *** pbourke has joined #openstack-infra | 22:56 | |
| *** pbourke has quit IRC | 22:59 | |
| *** pbourke has joined #openstack-infra | 23:00 | |
| *** felipemonteiro has joined #openstack-infra | 23:00 | |
| *** elbragstad has joined #openstack-infra | 23:01 | |
| melwitt | clarkb: opened https://bugs.launchpad.net/nova/+bug/1795086 FYI. didn't find how it could be happening yet | 23:03 |
| openstack | Launchpad bug 1795086 in OpenStack Compute (nova) "nova.tests.unit.test_profiler.TestProfiler.test_all_public_methods_are_traced sometimes does not return" [Low,Confirmed] | 23:03 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Manage user ssh keys from urls https://review.openstack.org/604932 | 23:05 |
| clarkb | melwitt: thanks | 23:05 |
| *** brokencycle has quit IRC | 23:07 | |
| *** elbragstad has quit IRC | 23:11 | |
| johnsom | Hi there, can someone bootstrap the octavia-lib-core group in gerrit with the octavia-core group? https://review.openstack.org/#/admin/groups/1951,members Thank you! | 23:12 |
| clarkb | johnsom: done | 23:14 |
| johnsom | Thanks! | 23:14 |
| *** yamamoto has joined #openstack-infra | 23:15 | |
| fungi | clarkb: 604932 worries me for reasons i can't quite put my finger on | 23:19 |
| fungi | how often are those public keys likely to change? | 23:20 |
| clarkb | fungi: I don't think we've decided at this point, but corvus was quite interested in having that behavior. I can be convinced either way | 23:23 |
| clarkb | there are definitely upsides and drawbacks to both approaches | 23:23 |
| clarkb | in particular not needing to rotate those keys ourselves is nice but could inadverdently add keys we don't want | 23:23 |
| clarkb | fungi: that change is a result of feedback corvus had on the parent change | 23:24 |
| fungi | it's probably no less secure than proposing and reviewing the retrieved keys to the config repository, i'm just trying to find good arguments to help convince myself that's the case | 23:25 |
| clarkb | fungi: https://review.openstack.org/#/c/604925/5/playbooks/bridge.yaml note the current setup dynamically adds the keys we just can't ssh as that user from anywhere | 23:26 |
| *** felipemonteiro has quit IRC | 23:26 | |
| clarkb | fungi: in my change to get CD working with a new user I dropped that in the first change for simplicit,y then followed up with the second change which does the dynamic thing | 23:26 |
| *** sthussey has quit IRC | 23:27 | |
| fungi | ahh, yes. so it's not a regression, however the interim state might have been a more secure (if less convenient) choice | 23:29 |
| clarkb | I'm happy to only merge the first change and manage that a bit more directly if we aren't comfortable with the second change | 23:29 |
| clarkb | should get corvus' input though | 23:29 |
| fungi | we're basically trusting the https cert on zuul either way, but putting the certs in git means more people an attacker might theoretically need to mitm. also a much shorter window of opportunity | 23:31 |
| *** gyee has quit IRC | 23:33 | |
| *** smarcet has quit IRC | 23:33 | |
| fungi | s/putting the certs/putting the public keys/ | 23:34 |
| *** jamesmcarthur has joined #openstack-infra | 23:34 | |
| *** mdbooth has joined #openstack-infra | 23:36 | |
| *** jamesmcarthur has quit IRC | 23:39 | |
| *** mdbooth has quit IRC | 23:42 | |
| johnsom | Zuul Ansible question. If you set host-vars and/or group-vars does that mean the "vars:" block is totally ignored? This appears to be what I am seeing. | 23:44 |
| johnsom | The way I read the docs (which very well could be wrong) is that the "vars:" block from the parent would still be honored, but applied to both, and could be overridden by "host-vars" and "group-vars". | 23:45 |
| johnsom | both being both nodes in a two node nodeset | 23:46 |
| *** mdbooth has joined #openstack-infra | 23:46 | |
| clarkb | I think the ansible variable precendence takes effect | 23:47 |
| clarkb | Im not sure what that is for host and group and zuul vars | 23:47 |
| johnsom | I am trying to do a "native" two node tempest gate with devstack-tempest as a parent, but all of the "vars:" from the parent, DATABASE_PASSWORD etc. are not showing up in the local_conf.txt. | 23:49 |
| fungi | by "the docs" you mean https://zuul-ci.org/docs/zuul/user/config.html#attr-job.vars i guess? | 23:49 |
| johnsom | Correct | 23:50 |
| johnsom | I was hoping I was doing something wrong and I don't have to duplicate all of these settings. | 23:50 |
| fungi | i thought they were all merged with precedence simply being used to determine which value wins when there are conflicts, but i could be misremembering | 23:51 |
| johnsom | https://review.openstack.org/#/c/605163/ if you want to have a look. (though sorry for asking late on a Friday) | 23:51 |
| clarkb | johnsom: there is no controller group but you set group vars for that group | 23:53 |
| johnsom | Maybe the variable override only goes one layer deep, so the host-vars with "devstack_localrc" completely replaces the "var:" devstack_localrc | 23:53 |
| clarkb | I also dont see a subnode grouo | 23:54 |
| clarkb | *group | 23:54 |
| johnsom | subnode is line 19, but yes, I see that controller is wrong. | 23:54 |
| johnsom | I don't think that is this issue however, since the localrc stuff is all in the host-vars. | 23:55 |
| clarkb | what does the resulting inventory look like? | 23:59 |
| clarkb | that is usually a good place to start when figuring this stuff out | 23:59 |
| *** yamamoto has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!