| *** Swami has quit IRC | 00:02 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Turn down pkg-map and hook copy tracing output https://review.openstack.org/611468 | 00:04 |
|---|---|---|
| *** ram5391 has joined #openstack-infra | 00:15 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: simplify python3.6 selection on gentoo https://review.openstack.org/608576 | 00:21 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: simplify overlay logic for Gentoo https://review.openstack.org/608577 | 00:21 |
| *** sthussey has quit IRC | 00:25 | |
| *** diablo_rojo has quit IRC | 00:39 | |
| fungi | ianw: tonyb: mordred: i think we should start relying on the ensure-tox role so we can drop one more nonstandard/language-specific tool from our images | 00:49 |
| *** longkb has joined #openstack-infra | 00:51 | |
| tonyb | fungi: I'd be fine with that, the DIB element and ensure-tox both ultimately install from pip so it's close to a noop for me | 00:51 |
| tonyb | fungi: using ensure-tox introduces one more place we're pull from the network but I doubt in reality that's cause any new failures | 00:52 |
| ianw | fungi: yeah, seems fine and likely only like that due to historical baggage. the only thing would be to make sure we have the old "install package first, then pip upgrade over it" so we don't have people installing the package after and messing things up | 00:52 |
| tonyb | ianw: then we run into a gray area around if the ensure-tox is generic or openstack specific? perhaps I'm overthinkingn it | 00:55 |
| fungi | generic | 00:57 |
| tonyb | fungi: OK. | 00:58 |
| *** bobh has quit IRC | 01:00 | |
| *** carl_cai has joined #openstack-infra | 01:00 | |
| *** markvoelker has joined #openstack-infra | 01:05 | |
| *** markvoelker has quit IRC | 01:09 | |
| openstackgerrit | Merged openstack/diskimage-builder master: fix tox python3 overrides https://review.openstack.org/579748 | 01:14 |
| *** imacdonn has quit IRC | 01:22 | |
| *** imacdonn has joined #openstack-infra | 01:22 | |
| *** Emine has quit IRC | 01:22 | |
| *** Emine has joined #openstack-infra | 01:23 | |
| *** mrsoul has joined #openstack-infra | 01:23 | |
| *** tpsilva has quit IRC | 01:40 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Temporary fix for race in quick start job https://review.openstack.org/611476 | 01:58 |
| *** felipemonteiro has quit IRC | 02:16 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: ubuntu: Add options to ignore mirror components and use insecure repos https://review.openstack.org/610430 | 02:23 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Set EPEL mirror during openstack-ci-mirrors https://review.openstack.org/609169 | 02:23 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Provide for an EPEL mirror during build https://review.openstack.org/608834 | 02:23 |
| *** Emine has quit IRC | 02:23 | |
| *** lujinluo has joined #openstack-infra | 02:28 | |
| *** hongbin has joined #openstack-infra | 02:29 | |
| *** lujinluo has quit IRC | 02:30 | |
| *** lujinluo has joined #openstack-infra | 02:30 | |
| *** psachin has joined #openstack-infra | 02:53 | |
| *** xarses_ has joined #openstack-infra | 02:53 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: ubuntu: Add options to ignore mirror components and use insecure repos https://review.openstack.org/610430 | 02:54 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Set EPEL mirror during openstack-ci-mirrors https://review.openstack.org/609169 | 02:54 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Provide for an EPEL mirror during build https://review.openstack.org/608834 | 02:54 |
| *** xarses has quit IRC | 02:54 | |
| *** dave-mccowan has quit IRC | 02:56 | |
| *** rcernin has quit IRC | 03:02 | |
| *** bnemec has joined #openstack-infra | 03:05 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix DIB ubuntu-minimal running on bionic (18.04) https://review.openstack.org/604478 | 03:08 |
| *** bnemec has quit IRC | 03:10 | |
| *** rcernin has joined #openstack-infra | 03:28 | |
| *** ramishra has joined #openstack-infra | 03:35 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:36 | |
| *** owalsh_ has joined #openstack-infra | 03:39 | |
| *** owalsh has quit IRC | 03:43 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 03:50 | |
| *** bhavikdbavishi has quit IRC | 03:52 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:52 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Turn down pkg-map and hook copy tracing output https://review.openstack.org/611468 | 03:56 |
| *** hongbin has quit IRC | 03:57 | |
| *** lujinluo has quit IRC | 04:07 | |
| openstackgerrit | Merged openstack-infra/zuul master: Temporary fix for race in quick start job https://review.openstack.org/611476 | 04:20 |
| openstackgerrit | Merged openstack-infra/zuul master: Fedora docker-compose fixes for selinux https://review.openstack.org/611417 | 04:23 |
| AJaeger | config-core, please review: https://review.openstack.org/610386 https://review.openstack.org/611114 https://review.openstack.org/611181 https://review.openstack.org/609901 https://review.openstack.org/610628 | 04:25 |
| openstackgerrit | Merged openstack-infra/zuul master: Use zuul/nodepool-launcher container for docker-compose https://review.openstack.org/611442 | 04:29 |
| *** ykarel has joined #openstack-infra | 04:33 | |
| openstackgerrit | Merged openstack/diskimage-builder master: ubuntu: Add options to ignore mirror components and use insecure repos https://review.openstack.org/610430 | 04:34 |
| *** yamamoto has quit IRC | 04:34 | |
| *** yamamoto has joined #openstack-infra | 04:34 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Set EPEL mirror during openstack-ci-mirrors https://review.openstack.org/609169 | 04:40 |
| *** ram5391 has quit IRC | 04:44 | |
| *** udesale has joined #openstack-infra | 04:45 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add support for Fedora 28, remove EOL Fedora 26 https://review.openstack.org/566337 | 04:58 |
| *** ccamacho has quit IRC | 05:03 | |
| *** kjackal has joined #openstack-infra | 05:04 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add mysql to quick-start https://review.openstack.org/610697 | 05:10 |
| openstackgerrit | Merged openstack-infra/zuul-base-jobs master: Correct zuul-jobs path https://review.openstack.org/599607 | 05:13 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix periodic job display in builds page https://review.openstack.org/611352 | 05:15 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add support for Fedora 28, remove EOL Fedora 26 https://review.openstack.org/566337 | 05:22 |
| *** bhavikdbavishi has quit IRC | 05:23 | |
| *** felipemonteiro has joined #openstack-infra | 05:24 | |
| *** bhavikdbavishi has joined #openstack-infra | 05:25 | |
| *** lbragstad_503 has quit IRC | 05:27 | |
| *** lbragstad_503 has joined #openstack-infra | 05:27 | |
| *** lujinluo has joined #openstack-infra | 05:33 | |
| *** lujinluo has quit IRC | 05:38 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Extract pep8 messages for inline comments https://review.openstack.org/589634 | 05:39 |
| *** rlandy|bbl is now known as rlandy | 05:39 | |
| *** quiquell has joined #openstack-infra | 05:45 | |
| *** lujinluo has joined #openstack-infra | 05:47 | |
| openstackgerrit | Merged openstack/diskimage-builder master: enable caching for gentoo builds https://review.openstack.org/604268 | 05:51 |
| openstackgerrit | Merged openstack/diskimage-builder master: simplify python3.6 selection on gentoo https://review.openstack.org/608576 | 05:51 |
| openstackgerrit | Merged openstack/diskimage-builder master: simplify overlay logic for Gentoo https://review.openstack.org/608577 | 05:51 |
| *** lujinluo has quit IRC | 05:52 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Limit ensure-python to Debian/Ubuntu use https://review.openstack.org/610948 | 06:04 |
| AJaeger | frickler, could you please review: https://review.openstack.org/610386 https://review.openstack.org/611114 https://review.openstack.org/611181 https://review.openstack.org/609901 https://review.openstack.org/610628 - thanks! | 06:04 |
| *** felipemonteiro has quit IRC | 06:07 | |
| *** carl_cai has quit IRC | 06:10 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/611509 | 06:10 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix RST formatting https://review.openstack.org/610198 | 06:11 |
| *** hashar has joined #openstack-infra | 06:15 | |
| *** lujinluo has joined #openstack-infra | 06:15 | |
| *** jtomasek has joined #openstack-infra | 06:19 | |
| *** longkb has quit IRC | 06:22 | |
| *** longkb has joined #openstack-infra | 06:23 | |
| *** roman_g has joined #openstack-infra | 06:25 | |
| *** noama has joined #openstack-infra | 06:29 | |
| *** lujinluo has quit IRC | 06:32 | |
| *** lujinluo has joined #openstack-infra | 06:36 | |
| *** Emine has joined #openstack-infra | 06:37 | |
| *** e0ne has joined #openstack-infra | 06:38 | |
| *** Emine has quit IRC | 06:42 | |
| *** lujinluo has quit IRC | 06:44 | |
| *** elod has quit IRC | 06:46 | |
| *** elod has joined #openstack-infra | 06:47 | |
| *** janki has joined #openstack-infra | 06:51 | |
| *** ginopc has joined #openstack-infra | 06:54 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/611509 | 06:58 |
| dirk | hwoarang: frickler : can you confirm that the unbound issue with dnssec and opensuse leap 15 is fixed? The update was released yesterday | 07:02 |
| *** rcernin has quit IRC | 07:07 | |
| *** lpetrut has joined #openstack-infra | 07:09 | |
| frickler | dirk: it was still failing earlier this morning, but that might be because our mirrors will need to pull the update and then maybe another round of image builds has to happen. I'll take a closer look later | 07:10 |
| dirk | frickler: thanks. Can you point me to the log? If we have a capturr of the rpm version installed I can tell you whether updates were included | 07:12 |
| *** ykarel has quit IRC | 07:13 | |
| *** alexchadin has joined #openstack-infra | 07:14 | |
| frickler | dirk: http://logs.openstack.org/23/610323/4/check/devstack-platform-opensuse-150/4d76cc0/ | 07:15 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add support for Fedora 28, remove EOL Fedora 26 https://review.openstack.org/566337 | 07:16 |
| dirk | frickler: yep, missing in there. Release should be 2.3.1 | 07:17 |
| *** alexchadin has quit IRC | 07:19 | |
| *** janki has quit IRC | 07:19 | |
| *** persia has quit IRC | 07:19 | |
| dirk | Can't find the mirror location in the logs | 07:19 |
| *** owalsh_ is now known as owalsh | 07:20 | |
| *** persia has joined #openstack-infra | 07:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update governance-uc docs publishing https://review.openstack.org/611114 | 07:21 |
| *** roman_g has quit IRC | 07:22 | |
| *** shardy has joined #openstack-infra | 07:23 | |
| openstackgerrit | Merged openstack-infra/project-config master: Retire project Anchor - step 4 https://review.openstack.org/611181 | 07:25 |
| *** aojea has joined #openstack-infra | 07:25 | |
| *** ykarel has joined #openstack-infra | 07:26 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add gabbi-tempest unofficial project https://review.openstack.org/610628 | 07:27 |
| *** tosky has joined #openstack-infra | 07:31 | |
| *** roman_g has joined #openstack-infra | 07:35 | |
| *** openstackgerrit has quit IRC | 07:35 | |
| *** apetrich has quit IRC | 07:39 | |
| hwoarang | dirk: checking now | 07:50 |
| *** dpawlik has quit IRC | 07:55 | |
| *** dtantsur|afk is now known as dtantsur | 07:56 | |
| *** apetrich has joined #openstack-infra | 08:00 | |
| *** jpich has joined #openstack-infra | 08:00 | |
| *** e0ne has quit IRC | 08:02 | |
| hwoarang | dirk: doesnt look it's fixed yet. i am guessing we need to trigger new dib builds | 08:02 |
| *** kopecmartin|off is now known as kopecmartin | 08:02 | |
| *** slaweq has joined #openstack-infra | 08:02 | |
| *** openstackgerrit has joined #openstack-infra | 08:05 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/elastic-recheck master: Add queries for 2 neutron fullstack test bugs https://review.openstack.org/611529 | 08:05 |
| *** roman_g has quit IRC | 08:11 | |
| *** e0ne has joined #openstack-infra | 08:13 | |
| *** dpawlik has joined #openstack-infra | 08:13 | |
| *** shardy has quit IRC | 08:18 | |
| *** olivierb has joined #openstack-infra | 08:20 | |
| *** shardy has joined #openstack-infra | 08:26 | |
| *** ykarel_ has joined #openstack-infra | 08:27 | |
| *** ykarel has quit IRC | 08:30 | |
| *** ykarel_ is now known as ykarel | 08:35 | |
| *** jesusaur has quit IRC | 08:38 | |
| *** derekh has joined #openstack-infra | 08:39 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add support for Fedora 28, remove EOL Fedora 26 https://review.openstack.org/566337 | 08:43 |
| *** carl_cai has joined #openstack-infra | 08:47 | |
| ianw | slaweq: any updates on the gra1 issues? | 08:52 |
| slaweq | ianw: I don't have any | 08:55 |
| slaweq | but let me ask guys from OVH | 08:55 |
| slaweq | ianw: yesterday they told me that it is some issue with IP allocations in Neutron but I don't know if they fixed it somehow or not yet | 08:56 |
| slaweq | I asked guy from OVH to join this channel | 08:57 |
| dpawlik | slaweq: we are still working on it | 08:57 |
| dpawlik | ianw: ^^ | 08:57 |
| amorin | infra is in better shape now | 08:57 |
| ianw | slaweq dpawlik : ok, thanks. i had to disable gra1 -- see notes in https://storyboard.openstack.org/#!/story/2004090 | 08:58 |
| amorin | neutron is able to handle the traffic | 08:58 |
| amorin | did you try today again? | 08:58 |
| *** sshnaidm_ has joined #openstack-infra | 08:59 | |
| ianw | amorin: i did not, but i could manually enable it for a bit if there's some likelyhood it will work | 08:59 |
| amorin | ianw: yes please | 09:00 |
| amorin | let me know the result | 09:00 |
| *** dpawlik has quit IRC | 09:02 | |
| ianw | ok, i've just given it one server, let's see if that boots | 09:02 |
| amorin | ianw: give me ID I will also check internaly | 09:03 |
| *** dpawlik has joined #openstack-infra | 09:04 | |
| ianw | amorin: b1156bab-3336-4535-b501-04591d8fc804 is building | 09:04 |
| amorin | ok | 09:04 |
| amorin | neutron timed out again... | 09:05 |
| amorin | shit | 09:05 |
| *** roman_g has joined #openstack-infra | 09:05 | |
| *** maciejjozefczyk has joined #openstack-infra | 09:06 | |
| ianw | amorin: you saw that from your side? | 09:09 |
| amorin | ianw: yup, I found why | 09:09 |
| amorin | wait a minute and we will try again | 09:10 |
| amorin | nova is trying again on another host | 09:10 |
| ianw | amorin: ahh, ok, that might explain "openstack.exceptions.SDKException: Error in creating the server: No valid host was found. There are not enough hosts available." which i was seeing last time? it kept going around hosts until it ran out, and neutron never responded? | 09:11 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: Revert "Extract pep8 messages for inline comments" https://review.openstack.org/611549 | 09:11 |
| amorin | ianw: exactly | 09:11 |
| amorin | AFAIR the timeout is 10 minutes | 09:12 |
| amorin | so it will loop over host during that time | 09:12 |
| amorin | and fail at the end | 09:12 |
| amorin | it's currently trying on a third one | 09:13 |
| amorin | it's weird that it does not work for your tenant | 09:13 |
| amorin | maybe there is something else, I am digging | 09:14 |
| ianw | just special i guess :) | 09:15 |
| amorin | did you spawn a second one? | 09:16 |
| ianw | umm, let me check ... nodepool is in control atm | 09:17 |
| amorin | yes and the second is active | 09:18 |
| ianw | 2018-10-18 09:13:19,327 INFO nodepool.DeletedNodeWorker: Deleting deleting instance b1156bab-3336-4535-b501-04591d8fc804 from ovh-gra1 | 09:18 |
| amorin | first one is looping in error | 09:18 |
| amorin | ok | 09:18 |
| ianw | yeah so it thinks it has deleted it and has started a new one | 09:18 |
| *** sshnaidm_ has quit IRC | 09:18 | |
| amorin | so weird, because the first one was supposed to boot on the same host, it got an error because instance info cache was not found | 09:19 |
| amorin | can you try boot again, maybe 10 times? | 09:20 |
| ianw | openstack.exceptions.ResourceTimeout: Timeout waiting for the server to come up. | 09:20 |
| ianw | yep so on that first one, nodepool timed out waiting for it (rather than say, it getting back an error) | 09:21 |
| ianw | i can up the max servers and it will try creating more | 09:21 |
| amorin | ok | 09:21 |
| ianw | alright, up to 10 ... should start taking more nodes now | 09:22 |
| ianw | few are coming up 3bd77a5c-c5d7-493d-972f-9ff5868fa035 50406766-c495-48ad-b458-0c42eeb85cf9 | 09:23 |
| ianw | http://paste.openstack.org/show/732413/ <- that's the 10 currently building | 09:24 |
| amorin | yup | 09:24 |
| amorin | so far so good | 09:24 |
| ianw | amorin: cool! did you do something? | 09:26 |
| amorin | I stopped a script on our side that was overloading neutron | 09:26 |
| amorin | the weird thing is that neutron was not overloaded at all | 09:27 |
| amorin | sounds like the DB was | 09:27 |
| amorin | this is related to your tenant, because it has a lot of history lines | 09:27 |
| amorin | neutron is taking ages to list ports for example | 09:27 |
| amorin | we have two things to do: spwan a bigger DB with bigger balls | 09:28 |
| amorin | and also clean your tenant so the DB lookup will be quicker | 09:28 |
| amorin | and also, refactor our side that is overloding neutron | 09:29 |
| amorin | I am currently working on that | 09:29 |
| ianw | amorin: sound like a plan! do you think this is similar for BHS1? we have a lot of errors launching nodes there too http://grafana.openstack.org/d/BhcSH5Iiz/nodepool-ovh?orgId=1&var-region=ovh-bhs1 | 09:30 |
| *** jesusaur has joined #openstack-infra | 09:31 | |
| amorin | yup, I think BHS1 and SBG1 and under same load | 09:31 |
| ianw | amorin: cool, what should i do with gra1 quota? revert it back to 80, keep it lower? | 09:32 |
| ianw | or i can go back to having it turned off | 09:33 |
| amorin | maybe you can keep it to a low value (I dont know if 80 is low) for today, to test the infra | 09:34 |
| amorin | and if it's correct | 09:35 |
| amorin | then open all tomorrow? | 09:35 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Revert "Disable ovh-gra1" https://review.openstack.org/611552 | 09:38 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Restore full OVH-GRA1 quota https://review.openstack.org/611553 | 09:38 |
| ianw | amorin: ^ that should do it | 09:38 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Revert "Disable ovh-gra1" https://review.openstack.org/611552 | 09:41 |
| *** betherly has joined #openstack-infra | 09:43 | |
| *** roman_g has quit IRC | 09:47 | |
| *** priteau has joined #openstack-infra | 09:52 | |
| *** bhavikdbavishi has quit IRC | 09:56 | |
| *** roman_g has joined #openstack-infra | 09:57 | |
| *** jbadiapa has quit IRC | 09:58 | |
| *** fresta has joined #openstack-infra | 10:02 | |
| *** fresta_ has quit IRC | 10:05 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] testing trusty image https://review.openstack.org/611559 | 10:05 |
| *** sshnaidm has joined #openstack-infra | 10:06 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add more information to build page https://review.openstack.org/610138 | 10:08 |
| *** yamamoto has quit IRC | 10:11 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] testing trusty image https://review.openstack.org/611559 | 10:11 |
| *** yamamoto has joined #openstack-infra | 10:12 | |
| *** sshnaidm has quit IRC | 10:12 | |
| *** yamamoto has quit IRC | 10:16 | |
| *** yamamoto has joined #openstack-infra | 10:16 | |
| *** dave-mccowan has joined #openstack-infra | 10:18 | |
| *** ifat_afek has joined #openstack-infra | 10:21 | |
| *** gfidente has joined #openstack-infra | 10:22 | |
| *** aojea has quit IRC | 10:32 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Support node caching in the nodeIterator https://review.openstack.org/604648 | 10:38 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Use node cache in quota calculations https://review.openstack.org/604649 | 10:38 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Cache iterations over ready nodes https://review.openstack.org/604650 | 10:38 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Use cache when counting poolnodes https://review.openstack.org/604651 | 10:38 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Use cache when deleting oldest unused nodes https://review.openstack.org/604652 | 10:38 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Cache node iterations in cleanup workers https://review.openstack.org/604691 | 10:38 |
| *** ykarel is now known as ykarelunch | 10:40 | |
| *** aojea has joined #openstack-infra | 10:45 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fix DIB_DISTRIBUTION_MIRROR_UBUNTU_IGNORE regex typo https://review.openstack.org/611559 | 10:50 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Support node caching in the nodeIterator https://review.openstack.org/604648 | 10:52 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Use node cache in quota calculations https://review.openstack.org/604649 | 10:52 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Cache iterations over ready nodes https://review.openstack.org/604650 | 10:52 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Use cache when counting poolnodes https://review.openstack.org/604651 | 10:52 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Use cache when deleting oldest unused nodes https://review.openstack.org/604652 | 10:52 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Cache node iterations in cleanup workers https://review.openstack.org/604691 | 10:52 |
| *** aojea has quit IRC | 10:52 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:54 | |
| *** sshnaidm has joined #openstack-infra | 10:58 | |
| *** kjackal has quit IRC | 10:59 | |
| *** kjackal has joined #openstack-infra | 11:02 | |
| *** hashar has quit IRC | 11:02 | |
| *** udesale has quit IRC | 11:04 | |
| *** roman_g has quit IRC | 11:11 | |
| ianw | infra-root: something really werid is going on with zuul's commenting. e.g. it has not commented on https://review.openstack.org/#/c/611552/ but the logs have http://paste.openstack.org/show/732416/ | 11:12 |
| ianw | "2018-10-18 10:49:32,738 DEBUG zuul.GerritConnection: Received: 400 file 2018-10-18 10 not found in revision 611552,2" ? | 11:13 |
| *** roman_g has joined #openstack-infra | 11:13 | |
| ianw | mordred / corvus : ^ i'm wondering if this is related to comments ... ? | 11:14 |
| *** jbadiapa has joined #openstack-infra | 11:14 | |
| *** jangutter has joined #openstack-infra | 11:16 | |
| AJaeger | tobiash: ^ | 11:17 |
| Miouge | Anyone from Packet Host hanging out here? | 11:19 |
| *** longkb has quit IRC | 11:19 | |
| *** yamamoto has quit IRC | 11:20 | |
| *** yamamoto has joined #openstack-infra | 11:21 | |
| pabelanger | Miouge: John S sometimes done, but doesn't look online right now | 11:22 |
| pabelanger | Miouge: you can try asking question here if related to openstack-infra running jobs, or try #packet (I think) | 11:23 |
| Miouge | #packet is empty so I’ll shout him an email, thanks pabelanger | 11:24 |
| *** yamamoto has quit IRC | 11:25 | |
| pabelanger | Miouge: sorry, #packethost | 11:26 |
| *** icey has quit IRC | 11:30 | |
| *** icey has joined #openstack-infra | 11:34 | |
| panda | is this ok to be merged ? https://review.openstack.org/602377 | 11:35 |
| panda | we're starting to have to maintain the branch in some way, and it's not worth to use reviews for that branch | 11:37 |
| *** electrofelix has joined #openstack-infra | 11:38 | |
| *** panda is now known as panda|lunch | 11:39 | |
| pabelanger | panda|lunch: I know in the past we (infra-root) has worked in the past to help merge projects together, I don't think we did ACL changes to the projects however. AJaeger do you happen to remember the process?^ | 11:41 |
| *** ykarelunch is now known as ykarel | 11:42 | |
| pabelanger | giving users mergePush permissions to a project might not be safe long term | 11:42 |
| *** owalsh has quit IRC | 11:43 | |
| AJaeger | pabelanger: I do not - since I never had those permissions. | 11:44 |
| *** e0ne has quit IRC | 11:50 | |
| *** sshnaidm_ has joined #openstack-infra | 11:50 | |
| *** sshnaidm has quit IRC | 11:53 | |
| *** quiquell is now known as quiquell|lunch | 11:53 | |
| *** owalsh has joined #openstack-infra | 11:59 | |
| *** yamamoto has joined #openstack-infra | 12:03 | |
| *** carl_cai has quit IRC | 12:06 | |
| *** ykarel has quit IRC | 12:07 | |
| *** ykarel has joined #openstack-infra | 12:07 | |
| jangutter | This change depends on a change that failed to merge. | 12:09 |
| frickler | dirk: I tested upgrading to the new unbound package on a freshly held node now and it did not resolve the issue. | 12:09 |
| jangutter | Noobie question here: Zuul said that for https://review.openstack.org/#/c/610916 ^ | 12:09 |
| jangutter | (sorry, ctrl-V didn't work like I thought it would) | 12:09 |
| pabelanger | jangutter: the previous patch in the stack failed, and because they were submitted together, zuul wasn't able to rebase that one and merge | 12:10 |
| pabelanger | so you need to recheck the whole stack to ensure zuul is able to land them all together | 12:11 |
| jangutter | pabelanger: that's what I thought too, but I thought there would be a job that failed? | 12:11 |
| jangutter | pabelanger: the only other os-vif patch I could find left the queue with no message. | 12:12 |
| frickler | dirk: it works after I stopped unbound, removed the broken root.key file manually, run the unbound-anchor command to generate a fresh, correct version, start unbound again | 12:12 |
| frickler | dirk: in a couple of hours we will have an image that contains the new unbound version from the start, maybe that will work better | 12:13 |
| *** owalsh_ has joined #openstack-infra | 12:13 | |
| pabelanger | jangutter: Yah, i think ianw posted a link about zuul not reporting messages for some reason above, maybe it is related | 12:13 |
| *** panda|lunch is now known as panda | 12:13 | |
| pabelanger | infra-root: think we might have a 2nd instances of zuul not reporting comments^ | 12:13 |
| pabelanger | I am not in a position to debug ATM | 12:14 |
| jangutter | https://review.openstack.org/#/q/project:openstack/os-vif+label:Workflow%252B1+is:open | 12:14 |
| jangutter | (I'm the guilty party here) | 12:14 |
| *** ansmith has joined #openstack-infra | 12:15 | |
| *** owalsh has quit IRC | 12:15 | |
| jangutter | pabelanger: and it's really weird, since they don't need merge (the set starts at master tip) | 12:16 |
| *** owalsh_ has quit IRC | 12:16 | |
| jangutter | pabelanger: anyway, thanks for making me less paranoid. | 12:16 |
| *** owalsh has joined #openstack-infra | 12:16 | |
| *** yamamoto has quit IRC | 12:20 | |
| *** yamamoto has joined #openstack-infra | 12:20 | |
| *** apetrich has quit IRC | 12:21 | |
| *** eharney has joined #openstack-infra | 12:27 | |
| *** ykarel_ has joined #openstack-infra | 12:27 | |
| *** ykarel has quit IRC | 12:30 | |
| *** dims has quit IRC | 12:30 | |
| *** trown|outtypewww is now known as trown | 12:30 | |
| *** owalsh has quit IRC | 12:31 | |
| frickler | infra-root: the broken sqlalchemy wheel seem back from the dead once more. could someone remove it and maybe also walk me through how I could do this myself? | 12:31 |
| jangutter | general newbie question.... is 'reverify' still a thing? | 12:32 |
| frickler | jangutter: nope, only "recheck" | 12:32 |
| *** dims has joined #openstack-infra | 12:33 | |
| *** apetrich has joined #openstack-infra | 12:33 | |
| *** jesusaur has quit IRC | 12:33 | |
| *** lpetrut has quit IRC | 12:33 | |
| jangutter | frickler: thanks.... need to wait for ooooold blogs to bitrot before I'm going to stop typing it. | 12:33 |
| *** rlandy has joined #openstack-infra | 12:36 | |
| *** mriedem has joined #openstack-infra | 12:38 | |
| *** kgiusti has joined #openstack-infra | 12:41 | |
| tobiash | AJaeger, ianw: maybe we should land the revert as it seems to have a bigger impact than we thought? | 12:41 |
| tobiash | (https://review.openstack.org/611549) | 12:42 |
| *** quiquell|lunch is now known as quiquell | 12:45 | |
| *** owalsh has joined #openstack-infra | 12:46 | |
| *** kaiokmo has joined #openstack-infra | 12:49 | |
| Shrews | tobiash: that seems sensible to me, unless mordred is around to help fix it forward | 12:49 |
| *** psachin has quit IRC | 12:50 | |
| tobiash | Shrews: ++ | 12:50 |
| AJaeger | tobiash, Shrews , yeah, let's merge the revert | 12:50 |
| AJaeger | tobiash: we can always revert back later ;) | 12:50 |
| Shrews | i just +2'd it | 12:50 |
| AJaeger | Shrews: please +A | 12:50 |
| AJaeger | Shrews: I just +2A | 12:51 |
| AJaeger | sorry, didn't recognize directly it's in zuul-jobs where I have permissions... | 12:51 |
| Shrews | np | 12:51 |
| Shrews | the question is, will that bug affect that merging? | 12:53 |
| *** ifat_afek has quit IRC | 12:53 | |
| Shrews | let's hope not | 12:53 |
| AJaeger | at least the change passed in testing ;) Let's see.. | 12:53 |
| AJaeger | infra-root, can you delete sqlalchemy-upgrade again, please? | 12:55 |
| AJaeger | infra-root, ianw proposes https://review.openstack.org/611444 as fix | 12:55 |
| AJaeger | I mean SQLAlchemy-Utils | 12:56 |
| *** bobh has joined #openstack-infra | 13:01 | |
| fungi | AJaeger: sure, but interesting that it always seems to hit that race | 13:02 |
| fungi | frickler: https://docs.openstack.org/infra/system-config/afs.html#deleting-files | 13:03 |
| fungi | though the other sections of that document may also need to be read for greater context | 13:03 |
| AJaeger | fungi: yes, very interesting. Hope that 611444 fixes it... | 13:03 |
| openstackgerrit | Gonéri Le Bouder proposed openstack-infra/zuul master: encrypt_secret: support OpenSSL 1.1.1 https://review.openstack.org/611414 | 13:03 |
| *** quiquell has quit IRC | 13:05 | |
| AJaeger | fungi, FYI, there was a report on openstack-dev mailing list about SQLAlchemy-Utils that I answered and thus asked here | 13:05 |
| *** quiquell has joined #openstack-infra | 13:05 | |
| fungi | #status log manually deleted corrupt /afs/.openstack.org/mirror/wheel/ubuntu-16.04-x86_64/s/sqlalchemy-utils/SQLAlchemy_Utils-0.33.6-py2.py3-none-any.whl and released mirror.wheel.xenialx64 volume | 13:05 |
| openstackstatus | fungi: finished logging | 13:05 |
| fungi | thanks | 13:05 |
| fungi | reviewing 611444 now | 13:06 |
| fungi | and approved | 13:07 |
| *** e0ne has joined #openstack-infra | 13:07 | |
| fungi | i'm sort of in and out today. replacement appliances are being delivered from a couple different places | 13:07 |
| fungi | infra-root: i expect to be offline most of tomorrow and saturday to go car shopping on the mainland, but will be around again by saturday afternoon/evening | 13:08 |
| AJaeger | fungi, hope you have it soon nice again! | 13:08 |
| fungi | thanks! it's coming along nicely. will all be even better than it was before the flood | 13:09 |
| AJaeger | then let's hope that the next flood does not force you to upgrade ;) | 13:11 |
| *** jrist has quit IRC | 13:11 | |
| fungi | indeed | 13:11 |
| *** weshay is now known as weshay_meeting | 13:12 | |
| *** jrist has joined #openstack-infra | 13:13 | |
| *** adriancz has quit IRC | 13:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: wheel-mirror: serialise copies to AFS https://review.openstack.org/611444 | 13:22 |
| AJaeger | amorin, infra-root, do we have again a problem in OVH GRA1? See http://grafana.openstack.org/d/BhcSH5Iiz/nodepool-ovh?orgId=1 , we have more errors and are not at capacity (only 6 out of 20 nodes) | 13:27 |
| amorin | AJaeger: checking | 13:27 |
| AJaeger | thanks, amorin ! | 13:27 |
| *** felipemonteiro has joined #openstack-infra | 13:28 | |
| mordred | Shrews, frickler, tobiash: weird | 13:28 |
| mordred | we have an ansible lint skip tag on that task | 13:28 |
| mordred | oh - the problem is that openstack-zuul-jobs doesn't add zuul to the roles path | 13:29 |
| AJaeger | mordred: yeah. | 13:30 |
| AJaeger | mordred: but more serious is https://review.openstack.org/#/c/611553/ , see backscroll | 13:30 |
| AJaeger | mordred: I mean ianw 's comment about 611552 | 13:31 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Revert "Extract pep8 messages for inline comments" https://review.openstack.org/611549 | 13:31 |
| tobiash | mordred: looks like it is telling zuul to post messages to non-existent files (which is bad) and this breaks zuul reporting (which is really bad) | 13:31 |
| mordred | tobiash: ah! yes, I agree - that is definitely bad | 13:32 |
| *** lbragstad_503 is now known as lbragstad | 13:32 | |
| mordred | reverting was definitey the right choice | 13:32 |
| tobiash | mordred: I guess the first is introduces with that patch in that case (maybe bug in tox output parsing) | 13:32 |
| tobiash | mordred: the second one is probably a bug that is currently in zuul and nobody ran into it yet | 13:33 |
| mordred | ++ | 13:33 |
| mordred | what a fun morning! | 13:33 |
| Shrews | ikr? | 13:33 |
| frickler | fungi: on which host would I run that? and what about the credential setup step? do I need to do https://docs.openstack.org/infra/system-config/afs.html#adding-a-superuser or are there generic infra-root credentials somewhere? | 13:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add zuul to tox linters env https://review.openstack.org/611607 | 13:36 |
| mordred | there's the simple one | 13:36 |
| corvus | tobiash: i'm not sure the second one is a bug in zuul -- gerrit will refuse to accept comments on invalid files. we might be able to protect against it by comparing to the files list and filtering out, but i'm not sure i like that. i don't like to hide errors. | 13:38 |
| fungi | frickler: i run the deletion commands from my workstation | 13:38 |
| corvus | tobiash, mordred: maybe we could filter out non-existing files but add a warning message to the report. | 13:39 |
| mordred | oh good, it's a corvus | 13:39 |
| mordred | corvus: ++ | 13:39 |
| tobiash | corvus: ++ | 13:39 |
| fungi | frickler: the initial account setup can be done on one of the afs servers via -local auth | 13:39 |
| amorin | AJaeger: neutron on our side is facing some overload | 13:39 |
| amorin | that's why | 13:39 |
| amorin | we are checking why | 13:39 |
| mordred | corvus: your comment from yesterday that maybe we should have saved the zuul_return file so we could look at it seems to still apply here :) | 13:39 |
| corvus | mordred: yeah. where should we do that? base job? | 13:40 |
| corvus | or in the tox job? | 13:40 |
| mordred | corvus: dunno. maybe for now we could to it in the tox job | 13:40 |
| *** lujinluo has joined #openstack-infra | 13:41 | |
| corvus | mordred: i can take a look at doing the zuul warning fix if you want to do that. | 13:41 |
| corvus | mordred: there was a clue from the earlier errors though -- it looked like it parsed a timestamp as a file name | 13:41 |
| mordred | oh joy | 13:41 |
| mordred | corvus: kk. we could also make saving the files a feature of zuul_return? | 13:42 |
| fungi | frickler: https://docs.openstack.org/infra/system-config/afs.html#administration mentions the -localauth option | 13:42 |
| corvus | mordred: 11:13 < ianw> "2018-10-18 10:49:32,738 DEBUG zuul.GerritConnection: Received: 400 file 2018-10-18 10 not found in revision 611552,2" ? | 13:42 |
| mordred | corvus: yah | 13:42 |
| corvus | mordred: yeah, zuul_return feature could work | 13:43 |
| *** bnemec has joined #openstack-infra | 13:43 | |
| corvus | gotta go catch a plane now | 13:44 |
| mordred | corvus: have fun! | 13:44 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Initialize label statistics to zero https://review.openstack.org/610993 | 13:45 |
| amorin | AJaeger: your tenant is having a lot of ports | 13:47 |
| *** plestang has joined #openstack-infra | 13:47 | |
| amorin | seems that you are over quota | 13:47 |
| amorin | somehow related to the issues on the infra | 13:48 |
| amorin | should I clean the ports? | 13:48 |
| AJaeger | amorin: yes, would be nice. | 13:49 |
| AJaeger | infra-root, so, we're back to the port leakage ^ ;( | 13:49 |
| mordred | \o/ | 13:50 |
| * mordred should have slept in this morning | 13:50 | |
| Shrews | So is it worth continuing trying to add port cleanup to nodepool (thus hiding the problem, but not fixing it), or wait for the provider to fix the port leak problem? | 13:51 |
| Shrews | i'd hate to add something to nodepool that was temporary and hide actual problems | 13:52 |
| Shrews | s/hide/hid/ | 13:52 |
| fungi | amorin: yes, we've been trying to delete stale-looking ports in a loop. i gather if nova times out talking to neutron then it doesn't follow through removing the port at instance deletion which results in a leak | 13:52 |
| *** lujinluo has quit IRC | 13:52 | |
| *** lujinluo has joined #openstack-infra | 13:53 | |
| frickler | fungi: o.k., so I have successfully run the "addprinc" commands on kdc01, now I would run "sudo pts createuser frickler.admin -id 8 -localauth" on afs01.dfw.o.o, correct? | 13:53 |
| AJaeger | mordred: do we need the ozj change for project-config as well? | 13:55 |
| fungi | frickler: that should do it, yes | 13:55 |
| fungi | frickler: assuming we only have admin uids up to 7 so far | 13:56 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Fix zuul_work_dir default for build-reno-releasenotes https://review.openstack.org/611616 | 13:57 |
| *** efried_pto is now known as efried | 14:06 | |
| *** slaweq has quit IRC | 14:06 | |
| mordred | Shrews: well - we already have the floating ip leak cleanup ... and if this leak is the result of an existing openstack bug, then I could imagine someone else running in to it | 14:07 |
| *** slaweq has joined #openstack-infra | 14:08 | |
| fungi | we've already run into it in more than one provider over the years | 14:08 |
| mordred | AJaeger: probably so - although now I'm looking at things again and i'm not 100% sure how this is all working - so I'm looking a little more | 14:09 |
| *** apetrich has quit IRC | 14:11 | |
| *** hamzy has quit IRC | 14:15 | |
| aspiers | corvus/anyone: when reviewing diffs in gertty, is there not a way to jump between comments? | 14:16 |
| aspiers | other than scrolling line by line, I mean | 14:16 |
| openstackgerrit | Merged openstack-infra/nodepool master: Run zuul-quick-start job https://review.openstack.org/610159 | 14:16 |
| fungi | aspiers: none i'm aware of, though that sounds like it could be a neat feature | 14:17 |
| aspiers | hrm, that's surprising | 14:17 |
| fungi | also jumping up/down by file would be neat | 14:17 |
| aspiers | exactly | 14:17 |
| aspiers | TBH I wonder - is it really possible to efficiently review without these? | 14:18 |
| aspiers | I guess it must be for some | 14:18 |
| fungi | i use pgup/pgdn | 14:18 |
| aspiers | ah | 14:18 |
| fungi | so scroll by a full screen | 14:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add zuul to test-requirements for linting https://review.openstack.org/611607 | 14:18 |
| mordred | AJaeger: ^^ updated that patch to be better | 14:18 |
| aspiers | well, my tmux already stole pgup/pgdown | 14:19 |
| fungi | aspiers: a jump to top/bottom could be nice too, in similar vein | 14:19 |
| aspiers | fungi: yes definitely | 14:19 |
| fungi | i run gertty under tmux and it doesn't grab my pgup/pgdn events | 14:19 |
| mordred | I run weechat under tmux and it doesn't grab my pgup/pgdn events | 14:20 |
| fungi | same | 14:20 |
| *** apetrich has joined #openstack-infra | 14:20 | |
| aspiers | sure, that's because I've bound pgup/pgdown and you haven't ;-) | 14:20 |
| fungi | also mutt, wyrd... | 14:20 |
| fungi | aspiers: oh! got it ;) | 14:20 |
| aspiers | https://github.com/aspiers/screenrc/blob/master/.tmux.d/prompt-navigation | 14:21 |
| aspiers | https://github.com/aspiers/screenrc/tree/master/.tmux.d/sequences | 14:21 |
| aspiers | that's one of the most helpful shell hacks I ever set up ... | 14:21 |
| fungi | neat | 14:22 |
| fungi | so you can scroll forward/back in the terminal buffer jumping between previous shell commands? | 14:22 |
| aspiers | exactly | 14:22 |
| fungi | you could probably apply that on your shell windows only and exec gertty from tmux directly? | 14:22 |
| aspiers | yeah | 14:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Update ANSIBLE_LIBRARY to use envsitepackagesdir https://review.openstack.org/611622 | 14:23 |
| fungi | also, looks like that's ctrl+pgup and ctrl+pgdn? | 14:23 |
| aspiers | oh yeah sorry, pgup/pgdown just do what they sound like they do ;-) | 14:24 |
| aspiers | using tmux copy buffer | 14:24 |
| fungi | and when you're not in a tmux copy buffer can't you use raw pgup/pgdn in console applications? | 14:24 |
| fungi | under tmux | 14:25 |
| aspiers | I have them auto-switching straight into the copy mode | 14:25 |
| aspiers | so I can instantly scroll back | 14:25 |
| aspiers | also very useful | 14:25 |
| aspiers | I was looking at gertty because my Gerrit web UI is constantly jumping to a different place in the diff even when I don't ask it to | 14:27 |
| aspiers | it's a really weird glitch | 14:27 |
| aspiers | started a few months back | 14:27 |
| aspiers | sounds exactly like this bug https://phabricator.wikimedia.org/T159919 | 14:27 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-periodic-tempest-dsvm-oslo-latest-full-master https://review.openstack.org/610386 | 14:28 |
| aspiers | am trying rendering set to slow instead of fast | 14:28 |
| mordred | aspiers: yah - I hit that same issue sometimes | 14:30 |
| *** ccamacho has joined #openstack-infra | 14:30 | |
| aspiers | getting the impression the workaround works so far | 14:31 |
| *** sambetts|afk is now known as sambetts | 14:31 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Disable ovh-gra1" https://review.openstack.org/611552 | 14:34 |
| fungi | aspiers: ahh, i use ctrl+b,pgup instead | 14:36 |
| amorin | AJaeger: fungi I am purging the ports on your tenant | 14:37 |
| aspiers | fungi: yeah, I use it so often that I decided to avoid the extra keystroke to switch mode | 14:37 |
| amorin | for gra1 | 14:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add nodepool to project-config-nodepool required-projects https://review.openstack.org/611629 | 14:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Update ansible library location to envsitepackagesdir https://review.openstack.org/611630 | 14:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Stop installing nodepool in all the test envs https://review.openstack.org/611631 | 14:39 |
| mordred | AJaeger: ok. I may have just fallen down a rabbit hole there :) | 14:39 |
| AJaeger | ;) | 14:40 |
| AJaeger | thanks, mordred | 14:40 |
| mordred | project-config-core: https://review.openstack.org/#/q/topic:zuul-jobs-linters <-- may be work a look | 14:40 |
| fungi | thanks amorin! | 14:41 |
| *** hamzy has joined #openstack-infra | 14:42 | |
| *** felipemonteiro has quit IRC | 14:42 | |
| AJaeger | mordred: two +2s, one -1 - fourth needs more time... | 14:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Stop installing nodepool in all the test envs https://review.openstack.org/611631 | 14:46 |
| mordred | AJaeger: yah - definitely want to see the linters jobs pass ... | 14:46 |
| *** ykarel_ is now known as ykarel|away | 14:49 | |
| *** carl_cai has joined #openstack-infra | 14:49 | |
| *** egonzalez has quit IRC | 14:55 | |
| *** Swami has joined #openstack-infra | 14:56 | |
| *** egonzalez has joined #openstack-infra | 14:56 | |
| openstackgerrit | Tobias Henkel proposed openstack/diskimage-builder master: Fix creating block device in block-device phase https://review.openstack.org/611644 | 14:57 |
| *** admcleod is now known as admcleod_away | 15:02 | |
| *** kjackal has quit IRC | 15:04 | |
| *** kjackal has joined #openstack-infra | 15:04 | |
| *** dklyle has joined #openstack-infra | 15:06 | |
| *** sthussey has joined #openstack-infra | 15:07 | |
| *** lujinluo has quit IRC | 15:07 | |
| *** e0ne has quit IRC | 15:12 | |
| *** betherly has quit IRC | 15:13 | |
| *** fuentess_ has joined #openstack-infra | 15:14 | |
| clarkb | I have removed nb04 from the emergency file at ianw's request since https://review.openstack.org/#/c/611552 has merged | 15:15 |
| *** eharney has quit IRC | 15:16 | |
| clarkb | thank you amorin dpawlik and ianw for working through that! | 15:16 |
| clarkb | I'll keep an eye on things while I grab breakfast to make sure it is still happy | 15:17 |
| amorin | this is still not working properly on our side | 15:17 |
| amorin | fyi, neutron is not being able to handle the requests | 15:17 |
| amorin | it timeouts | 15:17 |
| amorin | we are checking why | 15:17 |
| clarkb | amorin: oh, should I keep the system disabled? | 15:18 |
| clarkb | oh I see in the graphs the behavior reverted | 15:18 |
| *** e0ne has joined #openstack-infra | 15:18 | |
| amorin | as you want | 15:18 |
| amorin | ianw enable it few hours ago | 15:18 |
| amorin | he enable for 20 spawns | 15:18 |
| amorin | it was working quite correctly but started to fail again an hour ago | 15:19 |
| *** lujinluo has joined #openstack-infra | 15:19 | |
| amorin | our team is on it at OVH | 15:19 |
| clarkb | amorin: would it help you if we changed that 20 value to something else? | 15:19 |
| amorin | I dont think so | 15:19 |
| *** jesusaur has joined #openstack-infra | 15:20 | |
| clarkb | ok I'll leave it as is then, and thanks again for helping with this! | 15:20 |
| *** yamamoto has quit IRC | 15:21 | |
| *** yamamoto has joined #openstack-infra | 15:22 | |
| *** jamesmcarthur has joined #openstack-infra | 15:24 | |
| *** lujinluo has quit IRC | 15:24 | |
| *** yamamoto has quit IRC | 15:26 | |
| *** devananda has joined #openstack-infra | 15:30 | |
| *** ccamacho has quit IRC | 15:36 | |
| *** agopi has quit IRC | 15:38 | |
| *** ccamacho has joined #openstack-infra | 15:38 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add zuul to test-requirements for linting https://review.openstack.org/611607 | 15:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add nodepool to project-config-nodepool required-projects https://review.openstack.org/611629 | 15:40 |
| *** hamzy has quit IRC | 15:42 | |
| *** hamzy has joined #openstack-infra | 15:43 | |
| *** lujinluo has joined #openstack-infra | 15:43 | |
| *** gyee has joined #openstack-infra | 15:43 | |
| clarkb | mordred: this might be unpopular but I don't think ansible-lint adds much value. The only times it ever points out a problem its wrong because the git module can't do 95% of what git can do and I need to fork a git process | 15:47 |
| *** ccamacho has quit IRC | 15:47 | |
| mordred | clarkb: I think I remember it catching a real problem once? | 15:47 |
| clarkb | it would be nice if there was a way to run it in catch functional issues vs catch someones opinion issues | 15:48 |
| clarkb | like pyflakes catching missing imports or invalid variable names, that type of linting is great | 15:49 |
| slaweq | clarkb: hi | 15:50 |
| clarkb | slaweq: hello | 15:50 |
| slaweq | clarkb: can You take a look at https://review.openstack.org/#/c/611529/1 if You will have some time? | 15:50 |
| slaweq | clarkb: it's adding fullstack issues to elastic-recheck as You asked on Tuesday | 15:51 |
| clarkb | slaweq: yes I'll take a look | 15:51 |
| slaweq | thx | 15:51 |
| slaweq | it's first time I am doing something like that so I don't know if that is fine as I did it :) | 15:51 |
| *** lujinluo has quit IRC | 15:51 | |
| clarkb | slaweq: at first glance it looks right. I tend to check it by using kibana to query with the same query in the e-r yaml file | 15:52 |
| slaweq | clarkb: ok, thx | 15:52 |
| *** lujinluo has joined #openstack-infra | 15:55 | |
| *** dtantsur is now known as dtantsur|afk | 16:00 | |
| *** eharney has joined #openstack-infra | 16:00 | |
| clarkb | mordred: fwiw any idea why the meme is that if there is a module you must do that rather than shell out? | 16:00 |
| *** e0ne has quit IRC | 16:00 | |
| *** ykarel|away has quit IRC | 16:02 | |
| amorin | clarkb: AJaeger jungleboyj ianw so far, status about GRA1 on OVH: we are still having issues, the team is still checking what is wrong on neutron side | 16:07 |
| amorin | I cleaned your ports but it's filling very quicly | 16:07 |
| *** olivierb has quit IRC | 16:08 | |
| *** lujinluo has quit IRC | 16:08 | |
| *** mriedem is now known as mriedem_lunch | 16:10 | |
| clarkb | amorin: ya I think nodepool is trying to boot instances in a loop and as they fail they are leaking the ports | 16:10 |
| openstackgerrit | Merged openstack-infra/nodepool master: Initialize label statistics to zero https://review.openstack.org/610993 | 16:10 |
| clarkb | amorin: I can tell nodepool to stop if it would help you | 16:10 |
| amorin | clarkb: ok, maybe that could help until the issue is fixed | 16:11 |
| *** yamamoto has joined #openstack-infra | 16:11 | |
| clarkb | amorin: ok I'll do that for gra1 now | 16:11 |
| clarkb | I think in bhs1 its more under control and we have ianw's workaround working there? | 16:11 |
| amorin | I am going to be afk until tomorrow (european time) | 16:11 |
| amorin | clarkb: ok | 16:11 |
| clarkb | alright gra1 should be disabled now (though there may be some thread that have to catch up on the new config so probably won't be immediate) | 16:11 |
| *** sshnaidm_ is now known as sshnaidm | 16:14 | |
| *** jamesmcarthur has quit IRC | 16:15 | |
| *** dklyle has quit IRC | 16:16 | |
| *** dklyle has joined #openstack-infra | 16:16 | |
| *** atarakt has left #openstack-infra | 16:17 | |
| *** nhicher has joined #openstack-infra | 16:18 | |
| AJaeger | amorin: thanks | 16:20 |
| *** Swami has quit IRC | 16:21 | |
| *** roman_g has quit IRC | 16:21 | |
| *** cdent has joined #openstack-infra | 16:24 | |
| fungi | i'm heading out to run errands and grab lunch but should be back soon | 16:25 |
| *** agopi has joined #openstack-infra | 16:26 | |
| cdent | can someone please do me the favor of adding me (cdent@anticdent.org / chdent ) to gabbi-tempest-core in gerrit? | 16:26 |
| cdent | thanks, and thanks for getting that imported to git.o.o | 16:26 |
| *** manjeets has joined #openstack-infra | 16:28 | |
| *** jamesmcarthur has joined #openstack-infra | 16:30 | |
| *** jpich has quit IRC | 16:30 | |
| *** lujinluo has joined #openstack-infra | 16:31 | |
| *** quiquell is now known as quiquell|off | 16:32 | |
| mordred | clarkb: nope, no clue | 16:35 |
| *** lujinluo has quit IRC | 16:36 | |
| *** dklyle has quit IRC | 16:37 | |
| *** lujinluo has joined #openstack-infra | 16:40 | |
| *** lujinluo has quit IRC | 16:40 | |
| *** shardy has quit IRC | 16:43 | |
| openstackgerrit | Merged openstack-infra/puppet-log_processor master: Add support for running a standalone geard https://review.openstack.org/529874 | 16:44 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add queries for 2 neutron fullstack test bugs https://review.openstack.org/611529 | 16:45 |
| clarkb | infra-root I'm going to approve the change that needed ^ this will convert existing logstash.o.o t running standalone geard then I'm going to look at booting a xenial logstash.o.o | 16:45 |
| clarkb | actually I'm going to put logstash.o.o in the emergency list and boot a new xenial server with the standalone geard. Then I don't have to migrate the system on the trusty server | 16:48 |
| AJaeger | config-core, gmann, could you review https://review.openstack.org/610688 , please? changes legacy-tempest-dsvm-neutron-full to not run on pike | 16:48 |
| jangutter | I'm seeing a zuul job that finished... but it didn't trigger the comment on gerrit? | 16:49 |
| AJaeger | jangutter: when did thta happen? | 16:49 |
| jangutter | http://zuul.openstack.org/build/4c652e627d03464788dc1ffeedbd1272 | 16:49 |
| jangutter | end time: 2018-10-18T13:42:24 | 16:49 |
| cdent | AJaeger: can you make me core in gabbi-tempest-core? | 16:49 |
| clarkb | cdent: I can do that | 16:49 |
| AJaeger | jangutter: we reverted a change that broke the reporting, so it should be fine now - please recheck | 16:49 |
| cdent | (by which I mean "able") | 16:50 |
| cdent | cool, thanks clarkb | 16:50 |
| AJaeger | cdent: I'm not able ... | 16:50 |
| jangutter | AJaeger, thanks! | 16:50 |
| clarkb | cdent: done | 16:50 |
| cdent | clarkb: thank you so much | 16:50 |
| clarkb | cdent: no problem | 16:51 |
| *** derekh has quit IRC | 16:54 | |
| *** yamamoto has quit IRC | 16:56 | |
| clarkb | I also need to finish up the etherpad server upgrades by deleting the old servers. I haven't seen anything indicating a problem with new servers. Has anyone seen behavior from etherpad that would indicate new regressions? | 16:56 |
| *** yamamoto has joined #openstack-infra | 16:56 | |
| *** sambetts is now known as sambetts|afk | 17:07 | |
| *** noama has quit IRC | 17:11 | |
| *** ramishra has quit IRC | 17:11 | |
| mordred | clarkb: I have not | 17:19 |
| openstackgerrit | Merged openstack-infra/system-config master: Run standalone geard for log processing https://review.openstack.org/529875 | 17:30 |
| *** panda is now known as panda|off | 17:30 | |
| *** dklyle has joined #openstack-infra | 17:30 | |
| *** dkehn has quit IRC | 17:30 | |
| clarkb | once bridge has ^ installed on it I will be booting a logstash01.openstack.org replacement server | 17:32 |
| *** yamamoto has quit IRC | 17:33 | |
| clarkb | kibana relies on logstash.o.o being able to talk to the elasticsearch servers, but everything else like subunit and log processing should switch over cleanly once I update DNS | 17:33 |
| clarkb | I'll have to kick the es servers to update their firewall rules and I might have to get the logstash and subunit works to reconnect? | 17:33 |
| clarkb | basically there may be some short bumps while we transition | 17:33 |
| *** dkehn has joined #openstack-infra | 17:37 | |
| *** carl_cai has quit IRC | 17:39 | |
| *** dklyle has quit IRC | 17:42 | |
| *** diablo_rojo has joined #openstack-infra | 17:44 | |
| *** hamzy has quit IRC | 17:44 | |
| *** betherly has joined #openstack-infra | 17:45 | |
| *** hamzy has joined #openstack-infra | 17:45 | |
| odyssey4me | hey folks - it seems like https://review.openstack.org/#/c/611419/ and https://review.openstack.org/#/c/611438/ started into gate, and then disappeared... any ideas what happened, and do we just recheck? | 17:47 |
| *** electrofelix has quit IRC | 17:48 | |
| *** betherly has quit IRC | 17:49 | |
| clarkb | odyssey4me: there was apparently a bug on how zuul commented back to gerrit? I don't have all the details as I've not fully caught up on scrollback | 17:49 |
| clarkb | odyssey4me: but at least for other changes the suggested was to rerun the tests as commenting is fixed | 17:49 |
| clarkb | odyssey4me: my hunch is it ran through the gate and failed to comment back due to this issue | 17:50 |
| *** dklyle has joined #openstack-infra | 17:50 | |
| odyssey4me | clarkb: alrighty - recheck it is! | 17:50 |
| *** mriedem_lunch is now known as mriedem | 17:51 | |
| *** priteau has quit IRC | 17:52 | |
| *** dklyle has quit IRC | 18:02 | |
| *** gfidente is now known as gfidente|afk | 18:04 | |
| *** betherly has joined #openstack-infra | 18:05 | |
| clarkb | mordred: interesting launch node behavior, we don't seem to apply our group filtering to the host names at athat point? | 18:08 |
| clarkb | mordred: its fine I have to kick.sh for puppet to run anyway which should fix that, but noticing this might be something we want to improve later? | 18:08 |
| mordred | clarkb: hrm. that is interesting | 18:09 |
| *** jamesmcarthur has quit IRC | 18:09 | |
| mordred | odyssey4me: it was all my fault | 18:09 |
| clarkb | mordred: noticed it because the firewall rules for group logstash were not applied to logstash01.openstack.org. Our groups filter should place logstash01.openstack.org into the logstash group | 18:09 |
| *** betherly has quit IRC | 18:09 | |
| odyssey4me | mordred: so we just get to blame you and go home then? | 18:09 |
| clarkb | mordred: I'm kick.shing it now to get puppet to run and will double check that the firewall rules apply at this point as expected | 18:10 |
| *** apetrich has quit IRC | 18:10 | |
| clarkb | mordred: ya kick.sh applied the rules as expected | 18:11 |
| mordred | odyssey4me: yup! | 18:11 |
| *** diablo_rojo has quit IRC | 18:12 | |
| mordred | clarkb: oh - you knw what - I think launch_node does a thing where it skips the openstack dynamic inventory because of how caching used to work | 18:12 |
| mordred | clarkb: or, really, how it still does - but - we should have it include the group inventory config settings | 18:13 |
| clarkb | mordred: I think taht would be most intuitive | 18:13 |
| mordred | so tha it'll ovelay those groups on the stub inventory it makes | 18:13 |
| *** diablo_rojo has joined #openstack-infra | 18:13 | |
| clarkb | or we could make it clear launch node is super barebones then always run kick.sh after | 18:13 |
| odyssey4me | mordred: oh man, that's great - cya! :) | 18:13 |
| mordred | clarkb: might make it easier anyway - the main thing you want launch_node to do is run the base playbook so that ssh keys are set up properly - and if it can't delete the node | 18:14 |
| *** yamamoto has joined #openstack-infra | 18:14 | |
| mordred | clarkb: but we don't actually need it to do additional things | 18:14 |
| clarkb | ya | 18:16 |
| clarkb | I've run into a really weird puppet error. Oct 18 18:13:25 logstash01 puppet-user[5481]: Could not get latest version: undefined method `[]' for nil:NilClass | 18:17 |
| clarkb | everything else seemed to mostly work ok? | 18:17 |
| *** lbragstad has quit IRC | 18:17 | |
| clarkb | I do want to fix that issue before I switch over otherwise we'll have an always failing puppet run I think | 18:17 |
| clarkb | Oct 18 18:13:25 logstash01 puppet-user[5481]: (/Stage[main]/Log_processor/Package[statsd]/ensure) change from 2.1.2 to latest failed: Could not get latest version: undefined method `[]' for nil:NilClass | 18:18 |
| clarkb | ok so maybe it was just a pip issue? I'll rekick to see if it succeeds and if so probably rebuild the instance anyway just to be sure everything is happy | 18:19 |
| *** dklyle has joined #openstack-infra | 18:20 | |
| clarkb | it happened again. I wonder if it is actually a conflict between explicit geard install and whatever else is using statsd | 18:23 |
| *** apetrich has joined #openstack-infra | 18:24 | |
| *** betherly has joined #openstack-infra | 18:24 | |
| clarkb | mordred: we use provider => openstack_pip for those installs. Any idea if that is still necessary? I wonder if we just want regular pip install? | 18:25 |
| *** diablo_rojo has quit IRC | 18:26 | |
| clarkb | the old trusty host has latest statsd installed, this is weird | 18:27 |
| *** scas has quit IRC | 18:28 | |
| *** dklyle has quit IRC | 18:29 | |
| *** betherly has quit IRC | 18:29 | |
| clarkb | 'os-performance-tools 0.1.0 has requirement statsd<3.0,>=1.0.0, but you'll have statsd 3.3.0 which is incompatible.' | 18:31 |
| clarkb | ok where does os-performance-tools come from | 18:32 |
| *** weshay_meeting is now known as weshay | 18:33 | |
| AJaeger | config-core, could you review https://review.openstack.org/610688 , please? changes legacy-tempest-dsvm-neutron-full to not run on pike | 18:33 |
| clarkb | subunit2sql installs os-performance-tools | 18:34 |
| clarkb | AJaeger: done | 18:34 |
| AJaeger | thanks, clarkb | 18:35 |
| *** jamesmcarthur has joined #openstack-infra | 18:37 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-log_processor master: Install statsd version 2.1.2 https://review.openstack.org/611695 | 18:40 |
| clarkb | infra-root ^ I think that will cleanup the logstash boot issue. if I can get reviews on that soonish that would be great as I'd like to keep trying to replace logstash.o.o today. Thanks! | 18:40 |
| * fungi is back from errands and catching up | 18:40 | |
| AJaeger | config-core, smcginnis has a change up to remove x-vrif-minus-2 which is not used from gerritbot, want to +2A? https://review.openstack.org/609728 | 18:42 |
| *** jamesmcarthur has quit IRC | 18:43 | |
| *** lifeless has joined #openstack-infra | 18:43 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Don't run legacy-tempest-dsvm-neutron-full on pike https://review.openstack.org/610688 | 18:43 |
| *** diablo_rojo has joined #openstack-infra | 18:44 | |
| *** betherly has joined #openstack-infra | 18:44 | |
| clarkb | AJaeger: ya I can take a look but first I am going to make some ramen for lunch. Hopefully that chases this cold away | 18:44 |
| *** lbragstad has joined #openstack-infra | 18:48 | |
| *** betherly has quit IRC | 18:49 | |
| *** panda|off has quit IRC | 18:59 | |
| *** panda has joined #openstack-infra | 19:00 | |
| fungi | you went salmon fishing and all you caught was a cold? | 19:01 |
| fungi | that makes for a *really* bad fishing trip | 19:01 |
| *** hamzy has quit IRC | 19:07 | |
| *** dklyle has joined #openstack-infra | 19:07 | |
| *** hamzy has joined #openstack-infra | 19:08 | |
| *** jamesmcarthur has joined #openstack-infra | 19:12 | |
| clarkb | fungi: we got 4 crabs too but ya the fishing (and crabbing) was slow :( | 19:13 |
| clarkb | also I left thr house yesterday eith the cold was actually good to be outside brcause weather was perfect | 19:13 |
| fungi | ahh, well yeah it's good to get out to the big blue room once in a while | 19:14 |
| *** dklyle has quit IRC | 19:14 | |
| *** jamesmcarthur has quit IRC | 19:16 | |
| *** bobh has quit IRC | 19:17 | |
| *** lbragstad has quit IRC | 19:21 | |
| *** bhavikdbavishi has quit IRC | 19:22 | |
| *** lujinluo has joined #openstack-infra | 19:22 | |
| mordred | the big blue room is terrifying | 19:22 |
| *** lbragstad has joined #openstack-infra | 19:24 | |
| fungi | i've had to interact with people there | 19:29 |
| fungi | can confirm | 19:29 |
| *** lujinluo has quit IRC | 19:30 | |
| *** lujinluo has joined #openstack-infra | 19:31 | |
| *** tpsilva has joined #openstack-infra | 19:33 | |
| *** diablo_rojo has quit IRC | 19:35 | |
| anteaya | I have a theory about why people are afraid of nature | 19:37 |
| anteaya | those who experience this and want to appreciate nature again might enjoy spending time with tiger eye stone or smokey quartz | 19:38 |
| anteaya | clarkb: glad you are feeling good about your cold | 19:38 |
| *** hamzy has quit IRC | 19:41 | |
| *** betherly has joined #openstack-infra | 19:44 | |
| *** apetrich has quit IRC | 19:44 | |
| eharney | i'm curious about the pylint comments zuul left all over this review: https://review.openstack.org/#/c/606346/ -- is this a feature? a funny bug? | 19:46 |
| *** lujinluo has quit IRC | 19:46 | |
| *** lujinluo has joined #openstack-infra | 19:47 | |
| *** devananda has quit IRC | 19:47 | |
| clarkb | eharney: I think it is part of a new feature to start doing inline comments for pep8 tox output? | 19:47 |
| clarkb | mordred: ^ has been far more involved in it than I | 19:47 |
| eharney | clarkb: sounds right, i hadn't heard of that | 19:48 |
| openstackgerrit | Merged openstack-infra/storyboard master: Revert "skip some alembic migrations for sqlite" https://review.openstack.org/611446 | 19:48 |
| eharney | our pylint job is, obviously, not in great shape for this :) | 19:48 |
| *** betherly has quit IRC | 19:48 | |
| imacdonn | turn it off! turn it off! ;) | 19:48 |
| eharney | is there a per-job option for it? | 19:49 |
| clarkb | I don't know. All of this seems to have changed while I was on a boat on a river without a computer yesterday :) mordred and possibly fungi will know | 19:50 |
| clarkb | infra-root 85140e9f-9759-4c8b-aca1-bd92ad1cb6b3 is the old trusty etherpad-dev server which I plan to delete shortly | 19:51 |
| mordred | eharney, imacdonn: it's not on any more - there were a couple of issues with it - but yes, when we turn it back on, we'll definitely add an option to allow it to be silenced | 19:52 |
| *** e0ne has joined #openstack-infra | 19:52 | |
| eharney | sounds good | 19:52 |
| imacdonn | agreed | 19:53 |
| mordred | although thanks for the example there - it shows another issue we should take in to account - ansi color codes | 19:53 |
| clarkb | that said if pylint is being ignored and having comments generated by it is too noisy. Why wouldn't you just stop using pylint? | 19:53 |
| eharney | well, we had a job previously that would only show pylint errors that were introduced in the new patch | 19:53 |
| eharney | this was deemed complicated and it was changed to do something else... i'm not sure what the theory is there now tbh | 19:54 |
| clarkb | mordred: fungi https://review.openstack.org/#/c/611695/1 should fix the problem I had with new logsatsh server. If you can review that it would be great | 19:55 |
| *** gfidente|afk is now known as gfidente | 19:55 | |
| clarkb | and now to review smcginnis_vaca gerritbot change | 19:56 |
| *** lujinluo has quit IRC | 20:00 | |
| *** ssbarnea_ has joined #openstack-infra | 20:02 | |
| *** eharney has quit IRC | 20:03 | |
| clarkb | #status log Old Trusty etherpad-dev server (85140e9f-9759-4c8b-aca1-bd92ad1cb6b3) deleted now that new Xenial etherpad-dev01 server has been running for a few days without apparent issue | 20:03 |
| openstackstatus | clarkb: finished logging | 20:03 |
| *** e0ne has quit IRC | 20:05 | |
| ianw | amorin clarkb : so i realised i didn't restart the port clearer on GRA1, which might explain why it stopped working after a few hours. it's clearing out about 400 ports now, so i'll re-enable some quota after that and see what happens | 20:05 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: WIP: Cleanup down ports https://review.openstack.org/609829 | 20:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove x-vrif-minus-2 from gerritbot notifications https://review.openstack.org/609728 | 20:10 |
| *** hamzy has joined #openstack-infra | 20:10 | |
| *** cdent has left #openstack-infra | 20:10 | |
| dmsimard | Gerrit 2.16RC0 is out | 20:11 |
| dmsimard | https://www.gerritcodereview.com/2.16.html | 20:12 |
| *** bgmccollum has quit IRC | 20:15 | |
| fungi | just in time for us to finish deciding how to upgrade to 2.15 | 20:16 |
| *** bgmccollum has joined #openstack-infra | 20:19 | |
| clarkb | the set of notes there looks pretty sane chances are we could upgrade to 2.16 fairly easily once on 2.15 | 20:20 |
| clarkb | all that to say the focus should still be 2.15 because thats the huge leap as far as upgrade cost | 20:20 |
| fungi | and also who knows what new major bugs lurk beneath the surface of any shiny new gerrit release | 20:21 |
| fungi | we've had to roll back gerrit upgrades before | 20:21 |
| *** gfidente has quit IRC | 20:21 | |
| mordred | yah | 20:21 |
| clarkb | ya I try not to think about that too much | 20:21 |
| mordred | I think it'll be good to upgrade to 2.15 and then go ahead and plan for a 2.16 soon after - it would be nice to be closer to our friends upstream | 20:22 |
| *** agopi has quit IRC | 20:24 | |
| *** agopi has joined #openstack-infra | 20:24 | |
| clarkb | apparently gerrit lets you delete changes starting in 2.14 | 20:24 |
| anteaya | clarkb: do you mean completely remove all history? | 20:27 |
| clarkb | anteaya: I don't know, just saw it as a one line feature list for 2.14 | 20:27 |
| clarkb | my guess is yes it will delete the git commits and review history | 20:28 |
| mordred | I imagine that will be one of the things we don't let people do | 20:28 |
| clarkb | ya | 20:28 |
| *** bgmccollum has quit IRC | 20:28 | |
| anteaya | oh good | 20:29 |
| anteaya | good == one of the things not enabled for all users | 20:29 |
| *** bgmccollum has joined #openstack-infra | 20:30 | |
| fungi | one of the many, many features we don't enable for non-admins | 20:30 |
| anteaya | awesome | 20:30 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add zuul to test-requirements for linting https://review.openstack.org/611607 | 20:35 |
| *** openstackgerrit has quit IRC | 20:36 | |
| *** kgiusti has left #openstack-infra | 20:36 | |
| clarkb | mordred: https://review.openstack.org/#/c/611695/1 has a +1 from zuul now. Care to review it so I can spin up another logstash replacement? | 20:36 |
| mordred | clarkb: +A | 20:38 |
| clarkb | tyty | 20:39 |
| *** pcaruana has quit IRC | 20:44 | |
| *** carl_cai has joined #openstack-infra | 20:44 | |
| *** betherly has joined #openstack-infra | 20:49 | |
| *** weshay is now known as weshay_pto | 20:52 | |
| *** betherly has quit IRC | 20:54 | |
| *** openstackgerrit has joined #openstack-infra | 21:04 | |
| openstackgerrit | Merged openstack-infra/puppet-log_processor master: Install statsd version 2.1.2 https://review.openstack.org/611695 | 21:04 |
| *** panda has quit IRC | 21:08 | |
| *** panda has joined #openstack-infra | 21:12 | |
| *** diablo_rojo has joined #openstack-infra | 21:12 | |
| *** trown is now known as trown|outtypewww | 21:13 | |
| *** bnemec is now known as bnemec-bbl | 21:20 | |
| *** agopi_ has joined #openstack-infra | 21:26 | |
| *** rlandy is now known as rlandy|bbl | 21:27 | |
| *** agopi has quit IRC | 21:28 | |
| *** quite has quit IRC | 21:32 | |
| *** dklyle has joined #openstack-infra | 21:36 | |
| *** diablo_rojo has quit IRC | 21:42 | |
| *** quite has joined #openstack-infra | 21:44 | |
| *** diablo_rojo has joined #openstack-infra | 21:44 | |
| *** dklyle has quit IRC | 21:45 | |
| *** agopi_ is now known as agopi | 21:47 | |
| *** ssbarnea_ has quit IRC | 21:53 | |
| *** jamesmcarthur has joined #openstack-infra | 21:55 | |
| clarkb | ok puppet worked with ^ in place. geard and apache are running. I'm going to update DNS, then kick.sh elasticsearch0*.openstack.org so that they pick up new iptable rules. And also restart the log workers so that they connect to the new geard | 21:58 |
| clarkb | ianw: it looks like gra1 is working again with the port cleaner running | 21:59 |
| *** jamesmcarthur has quit IRC | 21:59 | |
| ianw | clarkb: yep, i might unemergency it and let it go back up to 20 for the rest of the day | 22:02 |
| ianw | ok, done, will keep an eye over the next few hours | 22:03 |
| clarkb | hrm geard only listens on ipv4 by default? | 22:08 |
| *** scas has joined #openstack-infra | 22:09 | |
| *** mriedem has quit IRC | 22:12 | |
| mordred | clarkb: that seems ungood | 22:15 |
| clarkb | I've hacked in a fix, you can configure it to listen on :: | 22:19 |
| clarkb | I will push that fix as soon as I figure out why elasticsaerch firewalls are not updating and why logstash-worker01 cannot talk to logstash01 | 22:20 |
| ianw | clarkb: that sounds similar to statsd | 22:20 |
| clarkb | ianw: ya other logstash workers can talk to it | 22:21 |
| ianw | i didn't quite realise that "::" works basically everywhere on linux because it dual stacks. i don't think it works on windows though | 22:21 |
| clarkb | oh maybe its another thing | 22:21 |
| clarkb | beacuse logsatsh-worker01 cannot ping logstash01 but logstash-worker02 can | 22:22 |
| clarkb | I may ignore that for now and focus on elasticsearch since then we'll have a completely working system if slightly degraded | 22:22 |
| ianw | hrm, i have dejavu about hosts with weird random ipv6 pings | 22:22 |
| ianw | that was ... zuul executors IIRC? had to end up removing the interface and re-adding it, or maybe deleting the whole host | 22:23 |
| ianw | don't you just love cloud sdn | 22:23 |
| ianw | a packet goes in ... goes through willy wonkas packet mangling factory and maybe comes out the other side :) | 22:24 |
| clarkb | 01, 10, 14, and 18 logstash workers cannot ping logstash01 | 22:24 |
| clarkb | ianw: removing the interface from the cloud perspective or in linux within the instance? | 22:25 |
| ianw | clarkb: removing from the cloud perspective. but by changing the ip, i think in the end it was easier to just rebuild the whole thing | 22:26 |
| ianw | fungi and i had tcpdumps etc showing the ipv6 packets going out one side and never making it to the other | 22:26 |
| fungi | there is a kernel option which if set would require you to separately listen on :: and 0.0.0.0 | 22:27 |
| clarkb | ianw: in this case ipv6 does work generally 16 of the 20 logstash workers can talk to logstash01 | 22:28 |
| fungi | years ago there was a lengthy debate on which behavior should be the default for linux | 22:28 |
| clarkb | ianw: was that what you saw before or did it not work at all? | 22:28 |
| ianw | http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-02-07.log.html#t2018-02-07T02:13:10 | 22:28 |
| clarkb | this is going to be painful if I have to rebuild the host and do the dns firewall dance again :( | 22:29 |
| ianw | that was when we last discussed it. there was nothing systematic about it iirc, that we could see anyway. but some suspicion it was cross-cell communication or some such | 22:29 |
| clarkb | I wonder if in this case beacuse we have 16/20 workable we can get rax to look at it? | 22:30 |
| ianw | of course this may not be the same thing ... it just sounds very similar ... | 22:30 |
| clarkb | might be a good thing for everyone involved to fix the problem | 22:30 |
| clarkb | ianw: ya this does seem simlar | 22:30 |
| fungi | there is still a trouble ticket open | 22:30 |
| clarkb | ok I think elasticsaerch connectivity is working now. My hunch for why that broke is that the dns resolution happens on bridge.o.o and not the remote (es0X) so when I checked with dig on es02 I was checking the wrong thing | 22:33 |
| *** rcernin has joined #openstack-infra | 22:33 | |
| clarkb | rerunning kick.sh against elasticsearch0* seems to have updated firewalls as expected | 22:33 |
| clarkb | I'm going to finish up restarting all the logstash workers now, then push the fix for geard not listening on :: | 22:33 |
| clarkb | then we can decide if we need to do another rebuild :/ | 22:33 |
| clarkb | but I don't want another rebuild until those fixes are in | 22:34 |
| *** pbourke has quit IRC | 22:35 | |
| *** pbourke has joined #openstack-infra | 22:36 | |
| *** bobh has joined #openstack-infra | 22:37 | |
| *** bobh has quit IRC | 22:41 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-log_processor master: Force geard to listen on :: https://review.openstack.org/611727 | 22:48 |
| clarkb | that is the fix for the ipv6 thing | 22:48 |
| clarkb | I think at this point everything is working except for the 4 logstash-workers that cannot talk to logstash01 | 22:48 |
| clarkb | I don't mind doing another rebuild tomorrow if we get ^ in | 22:48 |
| clarkb | now that is curious we seem to have 80 workers running. I wonder if our connections are falling back to ipv4 automatically just need time to timeout? | 22:49 |
| clarkb | if so cool I can probably live with that and not rebuild? | 22:49 |
| *** rpioso is now known as rpioso|afk | 22:50 | |
| *** betherly has joined #openstack-infra | 22:51 | |
| *** diablo_rojo has quit IRC | 22:52 | |
| clarkb | I've double checked all the ze hosts can talk to logstash01 over ipv6. So just those 4 worker nodes which I confirmed did fall back to ipv4 | 22:53 |
| clarkb | given that I think a rebuild is less urgent, but the fix above should get in so I can remove logstash01 from the emergency file | 22:53 |
| clarkb | I probably won't attempt to do any more server upgrades this week and instead focus on cleaning things up (old servers etc) for the three I did do | 22:56 |
| *** betherly has quit IRC | 22:56 | |
| *** carl_cai has quit IRC | 22:59 | |
| *** tosky has quit IRC | 23:02 | |
| ianw | ++ thanks! | 23:03 |
| *** tpsilva has quit IRC | 23:03 | |
| *** diablo_rojo has joined #openstack-infra | 23:03 | |
| clarkb | ianw: on https://review.openstack.org/#/c/611644/1 (a dib fix from tobiash) I left a comment but +2'd the change. I'll let you decide if you want the comment to be addressed before merging | 23:04 |
| ianw | clarkb: heh was just writing up comments :) | 23:05 |
| clarkb | ianw: and re https://review.openstack.org/#/c/611075/ I figure I can just abandon that if we don't corrupt the wheels now that your fix is in | 23:07 |
| clarkb | I'll wait for tomorrow to confirm that your fix has made things happier | 23:08 |
| clarkb | if we do continue to corrupt I think we should figure out hwo to get some version of that change in while we continue to debug the underlying cause | 23:08 |
| *** agopi is now known as agopi|brb | 23:08 | |
| ianw | clarkb: yeah, if that's not it, i'll keep digging and agree we can work around. but i strongly suspect ... we trigger both copies at basically the same second, and they're both going to be copying very similar things | 23:08 |
| ianw | so it may not be surprising on days where the requirements haven't changed, we end up hitting the same race | 23:09 |
| ianw | i can find no references to bugs with pip creating invalid zips (except some odd things with very old pre-1980 files, which i checked for), and we have no errors ... so it does seem like something underneath that | 23:10 |
| clarkb | ya my other hunch was maybe something about that particular setup.py but its pretty sane | 23:11 |
| clarkb | I definitely think it was likely in how we write the files and/or publish them | 23:11 |
| *** betherly has joined #openstack-infra | 23:12 | |
| *** agopi|brb has quit IRC | 23:13 | |
| *** betherly has quit IRC | 23:16 | |
| *** rlandy|bbl is now known as rlandy | 23:19 | |
| *** agopi|brb has joined #openstack-infra | 23:26 | |
| clarkb | ianw: similar situation with https://review.openstack.org/#/c/611200/2 I'll let you decide if you want to fix those comments or keep it as is. They are minor nits overall. | 23:26 |
| clarkb | er similar situation as with the dib change | 23:26 |
| kmalloc | fwiw, i want to say i'm looking forward to polygerrit... the current UI is starting to do even stranger things than before :P [This is not meant to make anyone feel rushed, just a voice of yay! when we get there] | 23:32 |
| *** betherly has joined #openstack-infra | 23:32 | |
| clarkb | kmalloc: we havent changed anything in months with current gerrit so may be good to understand the issues in the interim | 23:36 |
| *** jamesmcarthur has joined #openstack-infra | 23:36 | |
| *** betherly has quit IRC | 23:37 | |
| *** diablo_rojo has quit IRC | 23:37 | |
| kmalloc | clarkb: i think it's just chrome being stupid/advancing | 23:38 |
| kmalloc | but the UI is bouncing around a lot now when used (both linux/windows) | 23:39 |
| *** jamesmcarthur has quit IRC | 23:39 | |
| kmalloc | and losing focus randomly in boxes/comments when typing, meaning things like "a" and other hot-keys are starting to use their "bound" actions in the ui | 23:39 |
| *** jamesmcarthur has joined #openstack-infra | 23:39 | |
| kmalloc | i even had an example today where the whole ui started flickering anytime I tried to add a comment | 23:39 |
| kmalloc | had to hard-refresh it to correct the problem. | 23:40 |
| kmalloc | it is all the same kinds of weirdness i've seen since gerrit went to the terrible javascript UI on mobile, but the issues are bleeding to desktop now. | 23:40 |
| kmalloc | and i really can't pin it on anything but the browsers and other such are moving forward and gerrit's ui has been left to bitrot (not our fault) | 23:41 |
| clarkb | kmalloc: supposedly changing your settings to the slow mode helps | 23:41 |
| kmalloc | hm. where does one fine the "slow" mode? | 23:42 |
| clarkb | kmalloc: open a diff and click the settings gear in the top right. change render from fast to slow | 23:43 |
| kmalloc | ah | 23:43 |
| kmalloc | i don't see that option | 23:44 |
| openstackgerrit | Merged openstack/diskimage-builder master: Add support for Fedora 28, remove EOL Fedora 26 https://review.openstack.org/566337 | 23:44 |
| kmalloc | found it. | 23:44 |
| kmalloc | now... if only it would let me click it :P | 23:45 |
| kmalloc | clarkb: yeah i can't click on it, looks "disabled" | 23:46 |
| kmalloc | as in locked to fast | 23:46 |
| kmalloc | aha there we go | 23:46 |
| clarkb | huh aspiers reported it worked just today | 23:46 |
| clarkb | I havent tried changing it because firefox seems to work fine | 23:46 |
| kmalloc | something was being loaded from somewhere external | 23:46 |
| kmalloc | and was being viewed as a tracker | 23:46 |
| kmalloc | so my browser blocked it | 23:47 |
| clarkb | uh we shouldnt load anything external on our gerrit server | 23:47 |
| kmalloc | whelp, i disabled the external loader blocker and i could click it :P | 23:47 |
| kmalloc | i'll dig in and figure out why | 23:47 |
| clarkb | I wonder if that help menu comes from google? | 23:48 |
| fungi | that would be weird (and unfortunate) | 23:48 |
| *** felipemonteiro has joined #openstack-infra | 23:48 | |
| kmalloc | nope, it seems like i have to hard refresh the settings page each time or i can't toggle a number of things | 23:49 |
| kmalloc | it wasn't the external loader | 23:49 |
| kmalloc | the blocker just forced a refresh of the page | 23:49 |
| fungi | i use privacy badger, ddg privacy essentials and have dnt set in firefox. seeing if i can reproduce | 23:49 |
| kmalloc | this is consistent with the janky-ness of the javascript ui | 23:49 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add support for enabling the ARA callback plugin in install-ansible https://review.openstack.org/611228 | 23:49 |
| *** bnemec has joined #openstack-infra | 23:50 | |
| fungi | yeah, i already had it set to slow but am able to toggle it | 23:51 |
| *** bnemec-bbl has quit IRC | 23:52 | |
| dmsimard | I'm not sure I 100% understand the new system-config ansible bridge things | 23:54 |
| kmalloc | fungi: firefox seems to be less janky with it | 23:54 |
| kmalloc | fungi: which is surprising because ... well never mind it isn't that surprising | 23:54 |
| kmalloc | was going to say something something google developer things | 23:55 |
| kmalloc | then realized what i was going to say | 23:55 |
| dmsimard | I'd like to enable ara on bridge.o.o (backed by a trove instance) and have WIP patches https://review.openstack.org/#/q/topic:ara-on-bridge | 23:55 |
| *** jamesmcarthur has quit IRC | 23:55 | |
| clarkb | dmsimard: I dont think we'll want new trove instances, but mordred probably has opinions | 23:56 |
| dmsimard | oh, it doesn't need to be trove | 23:56 |
| dmsimard | sqlite is just not very good with concurrency | 23:56 |
| dmsimard | it works well on executors but each job runs on their own database | 23:56 |
| dmsimard | so anyway the thing I'm not sure I understand | 23:57 |
| dmsimard | I have an ara_install parameter that I use to toggle on/off installation (defaults to false) https://review.openstack.org/#/c/611228/3/playbooks/roles/install-ansible/defaults/main.yaml | 23:58 |
| dmsimard | I set it true in the playbook https://review.openstack.org/#/c/611228/3/playbooks/zuul/run-base.yaml | 23:58 |
| *** jamesmcarthur has joined #openstack-infra | 23:58 | |
| dmsimard | but the task that checks on that condition ended up getting skipped | 23:59 |
| dmsimard | so I'm not sure if it's an issue of the var not making it there or something | 23:59 |
| *** gyee has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!