| *** ahosam has quit IRC | 00:01 | |
| fungi | i want to know where to find the git2bzr migration utility | 00:02 |
|---|---|---|
| *** nicolasbock has quit IRC | 00:03 | |
| ianw | fungi: why did debuntu split venv out into a non-default package? | 00:06 |
| fungi | aiui, the messy way its dependencies have to be handled | 00:07 |
| fungi | venv needs to be able to install pip, which in turn has quite a few other non-stdlib deps | 00:07 |
| fungi | and this all needs to work without any network connection | 00:08 |
| fungi | so it has a dependency on a package which has a bunch of wheel files for all of that | 00:08 |
| fungi | basically, venv is part of stdlib, but relies on python libs outside the stdlib, via the ensurepip shim | 00:09 |
| fungi | so it's the (currently) one area where stdlib isn't self-contained, aside from linked c libs | 00:10 |
| ianw | that ... seems fair enough | 00:12 |
| fungi | i don't entirely agree with the choice, and think they could have been a little more pragmatic about it, but debian is where extreme paranoia shines | 00:12 |
| *** sgw has joined #openstack-infra | 00:17 | |
| *** migawa|lunch is now known as migawa|AFK | 00:19 | |
| *** jamespage has quit IRC | 00:26 | |
| *** jamespage has joined #openstack-infra | 00:27 | |
| *** tetsuro has quit IRC | 00:33 | |
| *** tetsuro_ has joined #openstack-infra | 00:33 | |
| *** gyee has quit IRC | 00:50 | |
| *** sgw has quit IRC | 00:52 | |
| *** armax has quit IRC | 01:07 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [dnm] ensure venv available, install glean with it https://review.opendev.org/707513 | 01:09 |
| *** auristor has quit IRC | 01:10 | |
| *** auristor has joined #openstack-infra | 01:16 | |
| *** dangtrinhnt has joined #openstack-infra | 01:20 | |
| *** sshnaidm is now known as sshnaidm|afk | 01:20 | |
| *** armax has joined #openstack-infra | 01:22 | |
| *** jamesmcarthur has joined #openstack-infra | 01:22 | |
| *** Adri2000 has joined #openstack-infra | 01:25 | |
| *** jamesmcarthur has quit IRC | 01:34 | |
| *** jamesmcarthur has joined #openstack-infra | 01:41 | |
| *** yamamoto has joined #openstack-infra | 02:04 | |
| *** jamesmcarthur has quit IRC | 02:05 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [dnm] ensure venv available, install glean with it https://review.opendev.org/707513 | 02:08 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] pip-and-virtualenv : home everything in environments https://review.opendev.org/707517 | 02:08 |
| *** jamesmcarthur has joined #openstack-infra | 02:11 | |
| *** xinranwang has joined #openstack-infra | 02:21 | |
| *** dave-mccowan has joined #openstack-infra | 02:23 | |
| openstackgerrit | Merged openstack/devstack-gate master: Cap virtualenv to a version < 20 https://review.opendev.org/707441 | 02:31 |
| *** roman_g has quit IRC | 02:34 | |
| prometheanfire | woo | 02:34 |
| prometheanfire | recheck the world? | 02:34 |
| *** lbragstad has quit IRC | 02:37 | |
| kevinz | ianw: now I can add the resources to nodepool linaro-us? | 02:39 |
| kevinz | ianw: actually now the devstack works fine on arm64 | 02:40 |
| ianw | kevinz: it should already be live, as it were | 02:40 |
| *** jamesmcarthur has quit IRC | 02:40 | |
| prometheanfire | well, done did it | 02:41 |
| kevinz | ianw: OK | 02:42 |
| kevinz | I will try | 02:42 |
| *** jamesmcarthur has joined #openstack-infra | 02:43 | |
| *** jamesmcarthur has quit IRC | 02:48 | |
| ianw | fungi: what do you understand pip install --root=/ would try to do? | 02:52 |
| *** zxiiro has quit IRC | 02:54 | |
| prometheanfire | --root <dir> Install everything relative to this alternate root directory. | 02:54 |
| ianw | oohhh, i see what it's doing | 02:55 |
| ianw | if it runs from a a virtualenv /home/ianw/env/pip ... then --root=/tmp/test actually installs everything in "/tmp/test/home/ianw/env/" | 02:55 |
| ianw | i.e. it also appends the path of the virtualenv it's in | 02:56 |
| ianw | ... sigh | 02:56 |
| prometheanfire | eww | 02:56 |
| ianw | i don't think that's right but can't tell | 02:56 |
| ianw | that basically explodes my idea that we could run pip out of a virtualenv but keep the current semantics of global install | 02:57 |
| *** jamesmcarthur has joined #openstack-infra | 02:57 | |
| *** dangtrinhnt has quit IRC | 02:58 | |
| prometheanfire | I don't see why you can't run pip out of a venv | 02:58 |
| prometheanfire | there may be another option | 02:59 |
| prometheanfire | --prefix <dir> Installation prefix where lib, bin and other top-level folders are placed | 02:59 |
| prometheanfire | well, just run `pip install --help` | 02:59 |
| prometheanfire | instead of me copypastaing | 02:59 |
| ianw | yeah, that doesn't work either | 03:03 |
| *** goldyfruit has quit IRC | 03:05 | |
| ianw | actually, maybe it's the combo that doesn't wokr ... --prefix=/usr might do what we want | 03:05 |
| *** migawa|AFK is now known as migawa | 03:05 | |
| *** goldyfruit has joined #openstack-infra | 03:05 | |
| ianw | "want" ... nobody wants this but "maintain enough backwards compatibility to work" | 03:05 |
| ianw | it puts binaries in /usr/bin ... not quite the same thing | 03:07 |
| prometheanfire | /usr/local/bin | 03:07 |
| ianw | hrm, maybe it works. the binary in /usr/bin still references the venv python ... but maybe that's ok | 03:09 |
| *** dangtrinhnt has joined #openstack-infra | 03:10 | |
| *** dangtrinhnt has quit IRC | 03:12 | |
| *** apetrich has quit IRC | 03:13 | |
| *** dangtrinhnt has joined #openstack-infra | 03:13 | |
| prometheanfire | well, depends on your end goals, where do you want things linked? | 03:14 |
| prometheanfire | wonder if there's a env-var that helps here (or is set for the venv python) | 03:15 |
| *** ramishra has joined #openstack-infra | 03:15 | |
| prometheanfire | hmm, hitting retry_limit, fun | 03:16 |
| *** dangtrinhnt has quit IRC | 03:19 | |
| ianw | umm, that's not great | 03:19 |
| prometheanfire | http://zuul.openstack.org/status on 707520 | 03:20 |
| ianw | looking at one -> http://paste.openstack.org/show/789490/ | 03:24 |
| ianw | trying to see if this is a pattern, there's certainly a lot ATM | 03:24 |
| prometheanfire | ya | 03:24 |
| *** ykarel|away is now known as ykarel | 03:25 | |
| ianw | # cat executor-debug.log | grep _restoreRepoState | wc -l | 03:25 |
| ianw | 506 | 03:25 |
| ianw | if that is not a red herring, it's coming up a bit | 03:25 |
| *** jamesmcarthur has quit IRC | 03:25 | |
| *** ramishra has quit IRC | 03:25 | |
| *** jamesmcarthur has joined #openstack-infra | 03:27 | |
| *** rlandy|bbl is now known as rlandy | 03:29 | |
| *** dangtrinhnt has joined #openstack-infra | 03:30 | |
| ianw | it's worryingly deep in ... magic | 03:34 |
| ianw | but maybe it's ze12 only? | 03:34 |
| weshay|ruck | you guys seeing jobs go to retry-limit on the swap playbook? | 03:52 |
| weshay|ruck | 2020-02-13 03:16:50.057885 | secondary | included: /var/lib/zuul/builds/d08994440dfb4a4f9cb9f0d057f34160/untrusted/project_3/opendev.org/openstack/openstack-zuul-jobs/roles/configure-swap/tasks/ephemeral.yaml | 03:53 |
| weshay|ruck | "TASK [configure-swap : Copy old /opt]" | 03:53 |
| weshay|ruck | http://zuul.openstack.org/stream/b4c69331300c408d9c8838ad5ba942f2?logfile=console.log | 03:54 |
| *** jamesmcarthur has quit IRC | 03:55 | |
| ianw | weshay|ruck: i'm not sure it's there ... i think i've convinced myself it's ze12 that's gone bad | 03:55 |
| *** ramishra has joined #openstack-infra | 03:55 | |
| *** jamesmcarthur has joined #openstack-infra | 03:56 | |
| ianw | [Thu Feb 13 01:48:25 2020] Out of memory: Kill process 9053 (zuul-executor) score 8 or sacrifice child | 03:57 |
| ianw | i'm going to restart it as the first measure, then i will write it up | 03:57 |
| weshay|ruck | http://cacti.openstack.org/cacti/graph.php?action=properties&local_graph_id=66231&rra_id=5&view_type=&graph_start=1567326473&graph_end=1567340873 | 03:57 |
| weshay|ruck | that's ze12? | 03:57 |
| weshay|ruck | ianw, thanks | 03:57 |
| weshay|ruck | ianw, you ever see sunil? | 03:58 |
| ianw | weshay|ruck: no, but i have recently moved to melbourne | 03:58 |
| weshay|ruck | ah.. well.. thanks for the assist :) | 03:59 |
| *** ykarel is now known as ykarel|afk | 03:59 | |
| *** jamesmcarthur has quit IRC | 04:01 | |
| *** rlandy has quit IRC | 04:03 | |
| *** tetsuro_ has quit IRC | 04:03 | |
| *** tetsuro has joined #openstack-infra | 04:04 | |
| *** udesale has joined #openstack-infra | 04:07 | |
| ianw | ok, i am quite convinced it was isolated to ze12 at this point | 04:14 |
| ianw | https://storyboard.openstack.org/#!/story/2007290 | 04:14 |
| ianw | i will reboot it just for good measure since it was in an unpredictable state | 04:15 |
| *** migawa is now known as migawa|lunch|AFK | 04:18 | |
| *** ykarel|afk is now known as ykarel | 04:23 | |
| johnsom | I'm seeing jobs bomb out with RETRY_LIMIT. It seems this was discussed above, but it's not clear to me if the issue is resolved and I can recheck or if I should wait. | 04:24 |
| johnsom | Well, the queue is light with some things passing, so I will error on the side of recheck. | 04:26 |
| *** jamesmcarthur has joined #openstack-infra | 04:26 | |
| prometheanfire | johnsom: sup | 04:30 |
| *** rosmaita has quit IRC | 04:36 | |
| *** igordc has joined #openstack-infra | 04:37 | |
| johnsom | prometheanfire: Hi there neighbor. I was just checking if the retry limit is was resolved or not. | 04:42 |
| prometheanfire | I think so, since ianw rebooted the machine | 04:44 |
| prometheanfire | but he hasn't said anything, so who knows :P | 04:44 |
| * prometheanfire has a bunch of rechecks waiting too | 04:44 | |
| *** igordc has quit IRC | 04:48 | |
| *** igordc has joined #openstack-infra | 04:48 | |
| *** igordc has quit IRC | 04:48 | |
| *** rosmaita has joined #openstack-infra | 04:49 | |
| johnsom | Ok, thanks to Ian for kicking whatever needed a boot | 04:49 |
| johnsom | ping me if there is something I can help with. I should be around for another hour or so. | 04:51 |
| *** migawa|lunch|AFK is now known as migawa|lunch | 04:52 | |
| ianw | sorry, just doing school run | 04:56 |
| ianw | there have been no more retry_limit's report afaics from the zuul logs since the reboot, so i'll call that one fixed | 04:58 |
| ianw | #status log rebooted ze12 due to it hitting an OOM situation (see https://storyboard.openstack.org/#!/story/2007290). please recheck any changes with RETRY_LIMIT failed jobs | 04:59 |
| openstackstatus | ianw: finished logging | 04:59 |
| prometheanfire | ok, gonna recheck | 05:02 |
| johnsom | Yep, my recheck is looking good | 05:02 |
| *** dangtrinhnt has quit IRC | 05:09 | |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #openstack-infra | 05:34 | |
| *** sgw has joined #openstack-infra | 05:53 | |
| *** migawa|lunch is now known as migawa|AFK | 06:24 | |
| *** migawa|AFK is now known as migawa | 06:25 | |
| openstackgerrit | yatin proposed zuul/zuul-jobs master: Ensure indirect deps are updated too https://review.opendev.org/707569 | 06:32 |
| *** ccamacho has quit IRC | 06:39 | |
| openstackgerrit | Merged zuul/zuul master: Don't report enqueue of non-live item https://review.opendev.org/707479 | 06:50 |
| *** tetsuro has quit IRC | 06:51 | |
| *** lmiccini has joined #openstack-infra | 06:51 | |
| *** tetsuro_ has joined #openstack-infra | 06:51 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Ensure correct cleanup on repo update and reset https://review.opendev.org/701531 | 07:02 |
| *** cshen has joined #openstack-infra | 07:25 | |
| *** ociuhandu has joined #openstack-infra | 07:30 | |
| *** ykarel is now known as ykarel|lunch | 07:31 | |
| *** AJaeger has quit IRC | 07:35 | |
| *** ociuhandu has quit IRC | 07:35 | |
| *** AJaeger has joined #openstack-infra | 07:38 | |
| *** ramishra has quit IRC | 07:45 | |
| *** pgaxatte has joined #openstack-infra | 07:49 | |
| *** jpena|off is now known as jpena | 07:51 | |
| *** imacdonn has quit IRC | 07:54 | |
| *** imacdonn has joined #openstack-infra | 07:54 | |
| *** ramishra has joined #openstack-infra | 08:02 | |
| *** dchen has quit IRC | 08:03 | |
| *** slaweq has joined #openstack-infra | 08:10 | |
| *** ccamacho has joined #openstack-infra | 08:13 | |
| *** hashar has joined #openstack-infra | 08:14 | |
| *** ralonsoh has joined #openstack-infra | 08:18 | |
| *** ysastri has joined #openstack-infra | 08:18 | |
| *** tosky has joined #openstack-infra | 08:19 | |
| *** tesseract has joined #openstack-infra | 08:20 | |
| *** kopecmartin has quit IRC | 08:21 | |
| *** pkopec has joined #openstack-infra | 08:26 | |
| *** ykarel|lunch is now known as ykarel | 08:28 | |
| *** tkajinam has quit IRC | 08:29 | |
| *** amoralej|off is now known as amoralej | 08:29 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Make most test cases work on MacOS https://review.opendev.org/707585 | 08:41 |
| *** kopecmartin has joined #openstack-infra | 08:45 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Reduce gearman logging in tests https://review.opendev.org/707587 | 08:52 |
| *** tetsuro_ has quit IRC | 09:03 | |
| *** lucasagomes has joined #openstack-infra | 09:07 | |
| *** tetsuro has joined #openstack-infra | 09:07 | |
| dulek | Hi! I requested a VM to get freezed yesterday on https://review.opendev.org/#/c/706529/. Can you add my key to it: https://github.com/dulek.keys ? | 09:09 |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report retried builds in a build set via mqtt. https://review.opendev.org/632727 | 09:11 |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report retried builds via sql reporter. https://review.opendev.org/633501 | 09:11 |
| ianw | dulek: try root@158.69.72.63 | 09:16 |
| frickler | ianw: seems we did that in parallel ;) | 09:16 |
| dulek | ianw: Works, thanks! | 09:16 |
| ianw | frickler: twice as good :) | 09:16 |
| *** derekh has joined #openstack-infra | 09:29 | |
| *** ociuhandu has joined #openstack-infra | 09:37 | |
| *** ociuhandu has quit IRC | 09:40 | |
| *** apetrich has joined #openstack-infra | 09:43 | |
| openstackgerrit | Elod Illes proposed openstack/devstack-gate master: Do not gzip files under logs in job results https://review.opendev.org/705255 | 09:48 |
| openstackgerrit | Elod Illes proposed openstack/devstack-gate master: Do not gzip files under logs in job results https://review.opendev.org/705255 | 09:51 |
| *** ociuhandu has joined #openstack-infra | 09:53 | |
| *** ociuhandu has quit IRC | 10:01 | |
| *** ociuhandu has joined #openstack-infra | 10:01 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Make most test cases work on MacOS https://review.opendev.org/707585 | 10:02 |
| *** ociuhandu has quit IRC | 10:02 | |
| *** ociuhandu has joined #openstack-infra | 10:02 | |
| *** yamamoto has quit IRC | 10:04 | |
| *** ociuhandu has quit IRC | 10:05 | |
| *** dtantsur|afk is now known as dtantsur | 10:05 | |
| *** ociuhandu has joined #openstack-infra | 10:05 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report retried builds via sql reporter. https://review.opendev.org/633501 | 10:07 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: afs-release: don't use paramiko https://review.opendev.org/707601 | 10:09 |
| *** AJaeger has quit IRC | 10:10 | |
| *** gfidente|afk is now known as gfidente | 10:11 | |
| *** AJaeger has joined #openstack-infra | 10:11 | |
| *** ociuhandu has quit IRC | 10:13 | |
| *** yamamoto has joined #openstack-infra | 10:16 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Add support for tracing using OpenTelemetry https://review.opendev.org/705170 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Enable tracing of trigger events https://review.opendev.org/706289 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace Github webhook trigger events https://review.opendev.org/706290 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace Gerrit trigger events https://review.opendev.org/706291 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace timer trigger events https://review.opendev.org/707606 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace git trigger nevents https://review.opendev.org/707607 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace zuul trigger events https://review.opendev.org/707608 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace Pagure webhook trigger events https://review.opendev.org/707609 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace queue items https://review.opendev.org/707610 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace merge jobs (client) https://review.opendev.org/707611 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace merge jobs (server) https://review.opendev.org/707612 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: DNM test https://review.opendev.org/707613 | 10:26 |
| openstackgerrit | Merged openstack/cookiecutter master: [ussuri][goal] Drop python 2.7 support and testing https://review.opendev.org/705691 | 10:26 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add load-branch to tenant configuration https://review.opendev.org/705664 | 10:31 |
| *** Lucas_Gray has joined #openstack-infra | 10:39 | |
| *** gshippey has joined #openstack-infra | 10:44 | |
| openstackgerrit | Thierry Carrez proposed opendev/yaml2ical master: H803 hacking have been deprecated https://review.opendev.org/413564 | 10:48 |
| *** udesale has quit IRC | 11:02 | |
| *** AJaeger has quit IRC | 11:04 | |
| *** hashar has quit IRC | 11:05 | |
| openstackgerrit | Merged opendev/yaml2ical master: H803 hacking have been deprecated https://review.opendev.org/413564 | 11:07 |
| *** AJaeger has joined #openstack-infra | 11:09 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Add project, change in RequirementError message https://review.opendev.org/707620 | 11:17 |
| *** dtantsur is now known as dtantsur|brb | 11:22 | |
| *** sshnaidm|afk is now known as sshnaidm|brb | 11:25 | |
| *** hamzy_ is now known as hamzy | 11:26 | |
| *** jpena is now known as jpena|lunch | 11:56 | |
| *** sshnaidm|brb is now known as sshnaidm | 11:56 | |
| *** hwoarang has quit IRC | 11:58 | |
| *** ociuhandu has joined #openstack-infra | 11:59 | |
| *** amoralej is now known as amoralej|lunch | 12:00 | |
| *** ociuhandu has quit IRC | 12:05 | |
| *** nicolasbock has joined #openstack-infra | 12:07 | |
| *** ysastri has quit IRC | 12:12 | |
| *** rh-jelabarre has joined #openstack-infra | 12:17 | |
| *** ykarel is now known as ykarel|afk | 12:19 | |
| *** ociuhandu has joined #openstack-infra | 12:22 | |
| openstackgerrit | Daniel Bengtsson proposed openstack/cookiecutter master: Update the minversion parameter. https://review.opendev.org/707397 | 12:22 |
| *** derekh has quit IRC | 12:23 | |
| *** ccamacho has quit IRC | 12:25 | |
| *** rfolco has joined #openstack-infra | 12:25 | |
| *** ociuhandu has quit IRC | 12:28 | |
| noonedeadpunk | o/ folks, trying to understand why I get "No package python-virtualenv available." on centos7 image? https://bb4e8592658ccf6acb57-65cf29ccd095bbc5984e6f01063fc964.ssl.cf5.rackcdn.com/707212/7/check/openstack-ansible-deploy-aio_metal-centos-7/27a8f7d/job-output.txt | 12:34 |
| *** ociuhandu has joined #openstack-infra | 12:35 | |
| noonedeadpunk | on blank centos this package is present from base/7/x86_64 | 12:35 |
| *** hwoarang has joined #openstack-infra | 12:40 | |
| *** tetsuro_ has joined #openstack-infra | 12:40 | |
| *** ociuhandu has quit IRC | 12:40 | |
| *** tetsuro has quit IRC | 12:44 | |
| *** tetsuro_ has quit IRC | 12:45 | |
| *** hwoarang has quit IRC | 12:46 | |
| *** hwoarang has joined #openstack-infra | 12:54 | |
| *** ykarel|afk is now known as ykarel | 12:56 | |
| *** rlandy has joined #openstack-infra | 13:00 | |
| *** adriant has quit IRC | 13:04 | |
| *** adriant has joined #openstack-infra | 13:04 | |
| *** smarcet has joined #openstack-infra | 13:11 | |
| *** jamesmcarthur has quit IRC | 13:12 | |
| *** jamesmcarthur has joined #openstack-infra | 13:12 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Improve error reporting when pr merge fails https://review.opendev.org/707637 | 13:12 |
| *** ykarel is now known as ykarel|afk | 13:18 | |
| *** ccamacho has joined #openstack-infra | 13:19 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Run ensure-tox on all platforms https://review.opendev.org/707238 | 13:19 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Make ensure-tox pass cross-platform https://review.opendev.org/707439 | 13:19 |
| *** slaweq has quit IRC | 13:21 | |
| *** rh-jelabarre has quit IRC | 13:23 | |
| *** njohnston has joined #openstack-infra | 13:24 | |
| *** eharney has quit IRC | 13:24 | |
| *** slaweq has joined #openstack-infra | 13:25 | |
| njohnston | Hi! I am seeing some weird failures in neutron grenade jobs, where it looks like some of the foundation files are missing. This started a couple of days ago. Does this ring any bells with you folks? | 13:25 |
| njohnston | example https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6e3/706514/1/gate/neutron-grenade-multinode/6e3aa74/job-output.txt | 13:25 |
| frickler | njohnston: that should get fixed by https://review.opendev.org/707490 | 13:26 |
| *** eharney has joined #openstack-infra | 13:26 | |
| njohnston | frickler: Thanks very much! | 13:26 |
| *** ociuhandu has joined #openstack-infra | 13:27 | |
| frickler | infra-root: there's also new breakage in stable devstack jobs on xenial due to stackviz wanting py>=3.6 now, hopefully gmann can fix this one way or the other soon | 13:27 |
| * frickler will be afk for about 2h now | 13:28 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add foreground option https://review.opendev.org/635649 | 13:29 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Deprecate -d switch for running in foreground https://review.opendev.org/705185 | 13:29 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Don't enforce foreground with -d switch https://review.opendev.org/705189 | 13:29 |
| *** jpena|lunch is now known as jpena | 13:29 | |
| *** amoralej|lunch is now known as amoralej | 13:32 | |
| *** jamesmcarthur has quit IRC | 13:38 | |
| *** ociuhandu has quit IRC | 13:39 | |
| *** Nizars has joined #openstack-infra | 13:39 | |
| *** ociuhandu has joined #openstack-infra | 13:39 | |
| *** Goneri has quit IRC | 13:40 | |
| *** Lucas_Gray has quit IRC | 13:41 | |
| *** Lucas_Gray has joined #openstack-infra | 13:46 | |
| *** icarusfactor has quit IRC | 13:49 | |
| openstackgerrit | Merged openstack/devstack-gate master: Just use the pip from the virtualenv https://review.opendev.org/707490 | 13:53 |
| *** Lucas_Gray has quit IRC | 13:54 | |
| *** Lucas_Gray has joined #openstack-infra | 13:59 | |
| *** ociuhandu has quit IRC | 13:59 | |
| mordred | frickler: isn't stackviz one of our things? | 14:10 |
| *** lbragstad has joined #openstack-infra | 14:11 | |
| *** lbragstad has quit IRC | 14:11 | |
| *** lbragstad has joined #openstack-infra | 14:12 | |
| fungi | noonedeadpunk: i think you need epel for python-virtualenv on centos-7, maybe see if it's not enabled? | 14:13 |
| noonedeadpunk | fungi: it's in base I think http://paste.openstack.org/show/789526/ but anyway I guess I found the reason... | 14:14 |
| noonedeadpunk | Like we're running the same script twice in CI without any need for that... | 14:14 |
| fungi | mordred: if by "our" you mean an openstack qa project, yep | 14:14 |
| * noonedeadpunk should check locale before pasting stuff... | 14:15 | |
| mordred | fungi: yeah - that's what I meant | 14:16 |
| mordred | the issue with stable branches and 3.6 is that some run on nodes with only 3.5 yeah? | 14:18 |
| fungi | as part of the process of dropping support for python 2.7 this cycle, projects have been encouraged to declare they depend on python 3.6 and later | 14:18 |
| fungi | but only in master | 14:19 |
| mordred | yeah - which is great for most things | 14:19 |
| fungi | though some tools are run from master on stable branch jobs on xenial, so... *boom* | 14:19 |
| mordred | yah | 14:19 |
| mordred | remote: https://review.opendev.org/707646 Support python 3.5 for stable branches | 14:19 |
| mordred | there's a stackviz patch | 14:19 |
| mordred | btw - I cannot wait for our 3.6 overlords and with them f'' format strings | 14:20 |
| *** Wryhder has joined #openstack-infra | 14:20 | |
| *** Lucas_Gray has quit IRC | 14:20 | |
| *** Wryhder is now known as Lucas_Gray | 14:21 | |
| *** yamamoto has quit IRC | 14:23 | |
| *** dtantsur|brb is now known as dtantsur | 14:24 | |
| *** udesale has joined #openstack-infra | 14:29 | |
| gmann | mordred: ok. Tempest also dropped py3.5 because of its deps and we pinned Tempest for rocky which has xenial and stein onwards its bionic distro so we run the master tools on py3.6. if stackviz is ok on py3.5 from its deps then yes we can keep supporting. | 14:30 |
| mordred | gmann: it looks like it should be ok I think | 14:31 |
| *** Goneri has joined #openstack-infra | 14:31 | |
| mordred | gmann: but - you know - either way works for me, mostly just trying to be mildly helpful :) | 14:31 |
| gmann | mordred: first try to keep supporting py3.5 for stable branch was via installing py3.6 on xenial via ppa but that is not so reliable solution so i had to choose the older version of tooling to use for py3.5 env | 14:32 |
| gmann | mordred: i can check it on xenial node and if all good there, we can merge. | 14:32 |
| mordred | gmann: sweet. because pinning to an earlier stackviz seems ... hard :) | 14:37 |
| *** ociuhandu has joined #openstack-infra | 14:38 | |
| gmann | yeah, I will check other tooling also (we do not use them on gate though) and keep py3.5 if possible | 14:38 |
| gmann | and that clearly brought up lack of testing on those tooling. we need to add stable job on their gate to know the issues before we break | 14:40 |
| mordred | ++ | 14:41 |
| *** yamamoto has joined #openstack-infra | 14:42 | |
| *** ociuhandu has quit IRC | 14:45 | |
| *** smarcet has quit IRC | 14:54 | |
| *** dtantsur is now known as dtantsur|afk | 14:58 | |
| *** rh-jelabarre has joined #openstack-infra | 14:59 | |
| *** ykarel|afk is now known as ykarel | 15:02 | |
| *** Lucas_Gray has quit IRC | 15:04 | |
| *** ysastri has joined #openstack-infra | 15:04 | |
| *** smarcet has joined #openstack-infra | 15:04 | |
| *** Lucas_Gray has joined #openstack-infra | 15:05 | |
| *** smarcet has quit IRC | 15:09 | |
| *** rlandy is now known as rlandy|brb | 15:10 | |
| openstackgerrit | Monty Taylor proposed openstack/diskimage-builder master: Add simple-pip element https://review.opendev.org/707499 | 15:13 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use LE certs for Apache https://review.opendev.org/705690 | 15:15 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update letsencrypt_gid documentation https://review.opendev.org/705878 | 15:15 |
| *** smarcet has joined #openstack-infra | 15:16 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add curl to python-builder image https://review.opendev.org/707668 | 15:19 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Move oc download to before src copy https://review.opendev.org/707250 | 15:20 |
| *** xinranwang has quit IRC | 15:21 | |
| dulek | infra-root: Hey, 158.69.72.63 can be scrapped, I successfully finished debugging on that frozen VM. | 15:22 |
| noonedeadpunk | fungi: actually no, I didn't find the issue why python-virtualenv is not available from base repos on centos... | 15:23 |
| *** rlandy|brb is now known as rlandy | 15:26 | |
| *** derekh has joined #openstack-infra | 15:27 | |
| noonedeadpunk | Like I'm pretty sure that even without epel it should be available as it's present http://mirror.centos.org/centos/7/os/x86_64/Packages/ | 15:28 |
| *** smarcet has quit IRC | 15:28 | |
| *** cshen has quit IRC | 15:31 | |
| weshay|ruck | anyone feel like a force push?? https://review.opendev.org/#/c/707330/ | 15:31 |
| frickler | dulek: deleting that node now, thanks for the feedback | 15:31 |
| dulek | No problem, thanks! | 15:32 |
| openstackgerrit | Ghanshyam Mann proposed openstack/os-performance-tools master: Support back python 3.5 https://review.opendev.org/707671 | 15:40 |
| *** Lucas_Gray has quit IRC | 15:45 | |
| fungi | noonedeadpunk: yeah, i see it in our ci mirrors too: http://mirror.dfw.rax.openstack.org//centos/7/os/x86_64/Packages/python-virtualenv-15.1.0-2.el7.noarch.rpm | 15:46 |
| noonedeadpunk | hm.... weird... | 15:46 |
| *** ysastri has quit IRC | 15:46 | |
| fungi | but `yum install python-virtualenv` on centos-7 gives an error? | 15:47 |
| noonedeadpunk | yeah http://zuul.openstack.org/stream/a29f108abba746c2abee9fcfc1a260e7?logfile=console.log | 15:47 |
| noonedeadpunk | http://paste.openstack.org/show/789533/ | 15:48 |
| mordred | that's not the package being missing | 15:50 |
| mordred | that's the command being missing | 15:50 |
| mordred | oh - wait - I see the line | 15:51 |
| mordred | WEIRD | 15:51 |
| *** Nizars has quit IRC | 15:51 | |
| *** priteau has joined #openstack-infra | 15:51 | |
| frickler | do we blacklist that pkg? | 15:52 |
| *** michael-beaver has joined #openstack-infra | 15:53 | |
| frickler | https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip#L199-L209 | 15:53 |
| frickler | seems we do | 15:54 |
| frickler | noonedeadpunk: mordred: ^ | 15:54 |
| mordred | ah yes - this is the whole big long conversation from yesterday | 15:54 |
| mordred | the system is currently designed such that one should not attempt to install virtualenv via packages | 15:55 |
| noonedeadpunk | and this is _only_ for centos.... | 15:56 |
| noonedeadpunk | yeah, I see... | 15:57 |
| *** Lucas_Gray has joined #openstack-infra | 15:57 | |
| mordred | we're working on fixing that - but it's SUPER complicated - so it'll likely be a few days | 15:57 |
| mordred | at the very least | 15:57 |
| mordred | in the mean time - a workaround might be to do a check for a pre-existing virtualenv and if it exists don't try to install it | 15:58 |
| mordred | if you're otherwise blocked | 15:58 |
| *** zxiiro has joined #openstack-infra | 15:58 | |
| mordred | I still don't know why you're getting that command not found though | 15:58 |
| jrosser | i've been trying to fix the issue where the osa code mis-parses the output from virtualenv --version to work around this | 15:58 |
| jrosser | and im getting strangeness on centos too which i have to say i don't understand | 15:58 |
| mordred | yeah. sorry for the heartburn on this one ... I think we at least have ALL of the use cases now :) | 15:59 |
| *** artom has joined #openstack-infra | 15:59 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/hacking master: [ussuri][goal] Drop python 2.7 support and testing https://review.opendev.org/705514 | 15:59 |
| jrosser | no worries - i was just looking to see if there was a quick fix from our side | 15:59 |
| openstackgerrit | Ghanshyam Mann proposed openstack/hacking master: [ussuri][goal] Drop python 2.7 support and testing https://review.opendev.org/705514 | 16:00 |
| *** eharney has quit IRC | 16:02 | |
| fungi | wow, right, i totally forgot we blocked those packages in yum's config | 16:02 |
| * fungi sighs | 16:03 | |
| fungi | i really think we should err on the side of uncomplicating all this and letting folks shoot themselves in the foot if they want to | 16:03 |
| mordred | yeah. I agree | 16:04 |
| openstackgerrit | Ghanshyam Mann proposed openstack/os-testr master: [ussuri][goal] Drop python 2.7 support and testing https://review.opendev.org/705522 | 16:05 |
| mordred | although I think we're all still advocating for slightly different approaches to uncomplicate things :) | 16:05 |
| mordred | ultimately one of them will work though | 16:05 |
| *** jpena is now known as jpena|off | 16:06 | |
| fungi | yeah, i have faith we'll zero in on a plan we can all tolerate | 16:06 |
| mordred | ++ | 16:06 |
| mordred | fungi: https://review.opendev.org/#/c/707499/ has NODE_FAILURE from Zuul ARM64 - is that expected/ | 16:06 |
| *** smarcet has joined #openstack-infra | 16:07 | |
| fungi | mordred: probably? i think we've seen issues getting nodes in those providers from time to time, so for some definition of "expected" (it's why we don't gate on those) | 16:08 |
| mordred | kk. just making sure it wasn't something that should induce panic | 16:10 |
| *** sreejithp has joined #openstack-infra | 16:13 | |
| *** udesale_ has joined #openstack-infra | 16:14 | |
| *** udesale has quit IRC | 16:14 | |
| *** ociuhandu has joined #openstack-infra | 16:16 | |
| openstackgerrit | Thierry Carrez proposed openstack/project-config master: Disable direct releases in js-openstack-lib ACL https://review.opendev.org/707682 | 16:21 |
| *** tosky has quit IRC | 16:21 | |
| mordred | fungi: do you remember off the top of your head where the code is that symlinks certs with the LE names to the snakeoil certs for when we're running tests? | 16:24 |
| mordred | fungi: oh - I tihnk I found it - it's the selfsign case in driver.sh | 16:25 |
| fungi | yeah, i don't recall but playbooks/roles/letsencrypt-acme-sh-install/files/driver.sh does look likely | 16:26 |
| mordred | https://zuul.opendev.org/t/openstack/build/a64d1e63eeb9450089d66a680e4c8ce0/log/review01.opendev.org/syslog.txt#1503 | 16:27 |
| mordred | is saying cert doesn't exist | 16:27 |
| fungi | Generate a fake CA key...Create fake CA root certificate...et cetera | 16:27 |
| mordred | but https://review.opendev.org/#/c/705690/14/playbooks/host_vars/review01.opendev.org.yaml ... | 16:27 |
| *** lmiccini has quit IRC | 16:28 | |
| fungi | shouldn't it all be in /etc/letsencrypt-certs/review.opendev.org/review.openstack.org.cer ? | 16:28 |
| fungi | just as subject altnames? | 16:28 |
| fungi | er, i mean in /etc/letsencrypt-certs/review.opendev.org/review.opendev.org.cer | 16:28 |
| fungi | we ought to grab all of /etc/apache2/sites-enabled/*.conf and /etc/letsencrypt-certs/*/* | 16:29 |
| mordred | fungi: oh - is that what it is? | 16:30 |
| fungi | i'll see what we're winding up with on zuul.o.o | 16:30 |
| mordred | that would make sense though | 16:31 |
| fungi | htm, we haven't moved the zuul.openstack.org cert into letsencrypt yet, it's still puppeted in, while zuul.opendev.org is le | 16:33 |
| fungi | so that's probably a bad example | 16:33 |
| mordred | fungi: I think I've convinced myself that's what makes sense though | 16:35 |
| mordred | because review-dev01 and review-dev both exist from before and there's only one cert | 16:36 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use LE certs for Apache https://review.opendev.org/705690 | 16:36 |
| fungi | right, as long as le lets us do domain validation for certs from different second-level domains as subject altnames in the same cert | 16:36 |
| fungi | i'm just looking for an example to confirm | 16:36 |
| mordred | fungi: if that's right, then that ^^ should go green and we might actually maybe be just maybe close to you know, being close to something | 16:36 |
| fungi | but it lets us do it for third-level subdomains so i don't see why it wouldn't | 16:37 |
| *** ociuhandu has quit IRC | 16:37 | |
| *** ykarel is now known as ykarel|away | 16:38 | |
| *** udesale_ has quit IRC | 16:40 | |
| *** jamesmcarthur has joined #openstack-infra | 16:41 | |
| *** ociuhandu has joined #openstack-infra | 16:42 | |
| *** gyee has joined #openstack-infra | 16:43 | |
| *** tosky has joined #openstack-infra | 16:43 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Centralize merge handling https://review.opendev.org/707692 | 16:55 |
| *** tesseract has quit IRC | 16:59 | |
| *** pgaxatte has quit IRC | 16:59 | |
| *** jamesmcarthur has quit IRC | 17:02 | |
| *** lucasagomes has quit IRC | 17:06 | |
| *** jamesmcarthur has joined #openstack-infra | 17:06 | |
| *** jamesmcarthur has quit IRC | 17:09 | |
| *** jamesmcarthur has joined #openstack-infra | 17:15 | |
| mordred | fungi: you were right! | 17:20 |
| mordred | fungi: https://review.opendev.org/#/c/705690/ is now green | 17:20 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Add project, change in RequirementError message https://review.opendev.org/707620 | 17:21 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update letsencrypt_gid documentation https://review.opendev.org/705878 | 17:21 |
| mordred | and https://review.opendev.org/#/c/705878/ should be too | 17:21 |
| openstackgerrit | Monty Taylor proposed openstack/diskimage-builder master: Add simple-pip element https://review.opendev.org/707499 | 17:25 |
| *** dave-mccowan has quit IRC | 17:26 | |
| *** jpena|off is now known as jpena | 17:27 | |
| fungi | okay, here's a strange gerrit behavior... | 17:30 |
| fungi | gerrit (silently) refused to merge https://review.opendev.org/679278 and marked it as being in merge conflict, even though zuul had no trouble with it | 17:31 |
| fungi | i tried the remote rebase (via gertty ^B) which did not produce a new patchset but removed the merge conflict flag in gerrit | 17:31 |
| fungi | maybe the rebase api call rechecks mergeability without necessarily producing a new patch set? | 17:32 |
| *** evrardjp has quit IRC | 17:34 | |
| *** evrardjp has joined #openstack-infra | 17:34 | |
| dansmith | has someone asked about "Unable to freeze job graph" spam already? | 17:36 |
| fungi | on what change? | 17:40 |
| dansmith | a bunch of nova ones, lemme get one | 17:40 |
| dansmith | https://review.opendev.org/#/c/697940/ | 17:40 |
| dansmith | fungi: second to last curretly | 17:40 |
| mordred | I got one on a dib change | 17:41 |
| mordred | https://review.opendev.org/#/c/707499/ | 17:41 |
| mordred | comments from Aditi Pai Dukle | 17:41 |
| fungi | oh, not from the zuul user | 17:42 |
| dansmith | I'm guessing some new IBM research bot | 17:42 |
| fungi | yeah, looks like someone has misconfigured a ci system connected with their account | 17:42 |
| mordred | that's a zuul v3 message - so it sounds like someone is attempting to run a zuul v3 pointed at openstack repos but it isn't configured quite correctly | 17:43 |
| fungi | https://review.opendev.org/#/q/reviewer:%22Aditi+Pai+Dukle%22 | 17:43 |
| fungi | looks like a human used it to recheck a change in september | 17:44 |
| mordred | we might want to disable their account and send them a friendly message? | 17:44 |
| fungi | and then starting at 10:32 utc today it began to leave that error message | 17:45 |
| *** ociuhandu_ has joined #openstack-infra | 17:45 | |
| fungi | mostly on nova and cinder changes though | 17:46 |
| fungi | but also that one dib change mordred noted | 17:46 |
| *** jamesmcarthur has quit IRC | 17:47 | |
| *** neiljerram has joined #openstack-infra | 17:47 | |
| *** ociuhandu has quit IRC | 17:48 | |
| *** eharney has joined #openstack-infra | 17:48 | |
| *** jamesmcarthur has joined #openstack-infra | 17:50 | |
| *** ociuhandu_ has quit IRC | 17:50 | |
| *** derekh has quit IRC | 17:59 | |
| *** gfidente is now known as gfidente|afk | 17:59 | |
| *** gfidente|afk is now known as gfidente | 18:03 | |
| *** Lucas_Gray has quit IRC | 18:07 | |
| *** armax has quit IRC | 18:11 | |
| *** amoralej is now known as amoralej|off | 18:12 | |
| *** gfidente is now known as gfidente|afk | 18:14 | |
| smcginnis | I've noticed a bunch of them. | 18:16 |
| sgw | Morning Infra team, I seem to be having an issue with adding someone to as a core reviewer in gerrit (review.opendev.org). Interestingly, I can remove a person. I am a member of the group myself. I have logged off, restarted my browser, tried different browsers and even different machines! | 18:20 |
| clarkb | sgw: whoch group? | 18:28 |
| clarkb | possible the group isnt self owned. The other issue may be using non unique id for new member | 18:28 |
| *** igordc has joined #openstack-infra | 18:28 | |
| sgw | any of the starlingx-*-core groups that I am a member of. I just tried starlingx-integ-core | 18:28 |
| *** igordc has quit IRC | 18:28 | |
| *** priteau has quit IRC | 18:29 | |
| clarkb | sgw is it the same individual in all case | 18:30 |
| sgw | clarkb: I am tring to add one person, but I think I tried with ildiko last week and it failed also, let me try you | 18:31 |
| *** jpena is now known as jpena|off | 18:31 | |
| clarkb | that group is self owned so that aspect should be fine | 18:31 |
| sgw | BTW: this is the error: | 18:32 |
| sgw | Code Review - Error | 18:32 |
| sgw | The page you requested was not found, or you do not have permission to view this page. | 18:32 |
| sgw | clarkb: what's your name in review.opendev.org? | 18:32 |
| clarkb | I just added myself | 18:33 |
| AJaeger | sgw: https://review.opendev.org/#/admin/groups/1970,info shows that it's self-owned | 18:33 |
| clarkb | can you confirm you can remove me then you can add me again | 18:33 |
| sgw | clarkb: removed OK, but add give the error above | 18:34 |
| sgw | Audit log shows your add and my removal | 18:34 |
| clarkb | ok, being able to remove implies that you are logged in properly | 18:34 |
| AJaeger | clarkb: let me try adding you to one of the groups that I have access to - to see whether it's a general problem. | 18:35 |
| sgw | I don't think it is since another person in that group was able to add someone. | 18:35 |
| clarkb | AJaeger: config-core? | 18:35 |
| sgw | Not that group per see | 18:36 |
| AJaeger | clarkb: https://review.opendev.org/#/admin/groups/30,members - openstack-doc-core | 18:36 |
| AJaeger | clarkb: so, that worked fine | 18:36 |
| AJaeger | So, it's not a problem with your account or with gerrit in general | 18:36 |
| clarkb | AJaeger: cam you test remove as well? | 18:36 |
| AJaeger | clarkb: removed you again ;) | 18:37 |
| AJaeger | all fine | 18:37 |
| AJaeger | sgw: must be your lucky day ;) | 18:37 |
| clarkb | sgw: are ylu running browser extensions that might conflict with this? | 18:37 |
| fungi | if the user you're trying to add has an e-mail address which belongs to more than one account, gerrit can exhibit an odd error and reject the addition (same for adding them as a requested reviewer on a change) | 18:37 |
| *** jamesmcarthur has quit IRC | 18:37 | |
| clarkb | fungi: it happened to my account though which shouldnt be kn that state | 18:38 |
| sgw | no idea, like I said, tried multiple machines / OSes, incognito page even | 18:38 |
| AJaeger | fungi: and I could add clarkb ... | 18:38 |
| clarkb | sgw: ok incognito should rule out extensions | 18:38 |
| *** jamesmcarthur has joined #openstack-infra | 18:38 | |
| clarkb | sgw: can you try the ssh interface, let me find docs | 18:38 |
| clarkb | sgw https://review.opendev.org/Documentation/cmd-set-members.html | 18:39 |
| clarkb | if that fails then it isnt web/browser specific | 18:40 |
| sgw | Looking | 18:40 |
| clarkb | and maybe wewill get a better error | 18:40 |
| AJaeger | clarkb: want to remove Anita and Sean from project-config-core? https://review.opendev.org/#/admin/groups/459,members | 18:40 |
| clarkb | AJaeger: I can. I've caught the illness that my kids had so I'll put it on todo list | 18:41 |
| AJaeger | clarkb: no urgency - get well soon! | 18:42 |
| sgw | So I use slin14 for Shuicheng: user/Lin Shuicheng (slin14) | 18:42 |
| mordred | clarkb: have you considered not getting sick instead? | 18:43 |
| *** jamesmcarthur has quit IRC | 18:44 | |
| mordred | (feel better) | 18:44 |
| sgw | Ok, really my lucky day, the ssh command worked! | 18:44 |
| AJaeger | \o/ | 18:44 |
| fungi | i bet he's, at this very moment, fondly reflecting on what it's like to not be sick | 18:44 |
| sgw | oh clarkb sorry, feel better. | 18:45 |
| clarkb | ok so probably web specific | 18:46 |
| clarkb | maybe you can retry with web dev tools up and see whoch url is 404ing or erroring? | 18:46 |
| clarkb | fungi: pretty much | 18:46 |
| sgw | hmm, honestly web dev is not my strong suit, like non-existent! I can try and see what my google-fu does for that. | 18:49 |
| clarkb | sgw: under browser options should be something like dev tools. then that pulls up a bunch of stuff. Look for a network tab and open it. then redo the operation and look for red/yellow entries | 18:52 |
| clarkb | (this is largely from memory and varies by browser so may not be exact) | 18:52 |
| *** ralonsoh has quit IRC | 18:52 | |
| fungi | looks like configuration error comments from that ibm user's account were continuing as recently as 40 minutes ago... should we assume it's abated or go ahead and disable the account? | 18:52 |
| sgw | looks like I am getting a 404 on user%3Aslin14 | 18:57 |
| sgw | There are more details when I click on that line, not sure what you want from the details | 18:58 |
| fungi | also i've confirmed the account belonging to that username does not share an e-mail address with any other accounts, looks like a perfectly normal account | 19:02 |
| *** ramishra has quit IRC | 19:02 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Report robot comments to gerrit https://review.opendev.org/707708 | 19:02 |
| *** armax has joined #openstack-infra | 19:02 | |
| sgw | I was able to add that account vs ssh to other core groups | 19:03 |
| fungi | yep, definitely pointing to something odd browser-side | 19:03 |
| sgw | I double checked with clarkb's account it also failes in the same way | 19:08 |
| *** smarcet has quit IRC | 19:13 | |
| *** smarcet has joined #openstack-infra | 19:21 | |
| *** rkukura has quit IRC | 19:31 | |
| *** yamamoto has quit IRC | 19:32 | |
| gmann | clarkb: when this gets updated. it still does not have latest stackviz with py3.5 fix - https://tarballs.openstack.org/stackviz/dist/ | 19:38 |
| gmann | or should we use stackviz-master.tar.gz instead of dist/stackviz-latest.tar.gz | 19:38 |
| *** smarcet has quit IRC | 19:40 | |
| gmann | its periodic job - periodic-package-stackviz-element | 19:41 |
| *** rkukura has joined #openstack-infra | 19:43 | |
| *** pkopec_ has joined #openstack-infra | 19:47 | |
| *** pkopec has quit IRC | 19:48 | |
| frickler | infra-root: I'm not sure we have seen this virtualenv failure mode yet? https://zuul.opendev.org/t/openstack/build/e3358f9df9844522b8034497a1ab7185/log/job-output.txt#5998 | 19:51 |
| mordred | frickler: yeah - that's old-six-with-new-virtualenv | 19:52 |
| frickler | gmann: we might be able to trigger that job manually now if you don't want to wait until tomorrow morning | 19:52 |
| gmann | frickler: yeah that can be good. meanwhile I am disabling failing job for stackviz issues. | 19:53 |
| frickler | mordred: yeah, but in the variant that the virtualenv is running under py2 even if all we want it a py3 venv? to we want to switch this to a "python3 -m venv" call instead? | 19:53 |
| frickler | s/to/do/ | 19:53 |
| mordred | frickler: looking | 19:54 |
| gmann | frickler: can you trigger that job? | 19:54 |
| mordred | frickler: yeah, that would be best. 'virtualenv' is currently *always* going to be python2-virtualenv because otherwise things get really squirrely with tox | 19:55 |
| frickler | gmann: I'll try to now | 19:55 |
| gmann | frickler: thanks | 19:55 |
| mordred | if you want virtualenv itself to be run under python3 - python3 -m venv or python3 -m virtualenv would be bets | 19:55 |
| mordred | best | 19:55 |
| mordred | frickler: keep in mind that venv may not always be present, as it's a separate package on ubuntu and I don't know that we've confirmed that we're always installing it yet | 19:56 |
| mordred | but you should, as of today, always be able to count on python3 -m virtualenv working (the other *should* work - but there might be one more layer of debugging) | 19:56 |
| *** lbragsta_ has joined #openstack-infra | 20:00 | |
| frickler | gmann: the job is running now | 20:05 |
| gmann | frickler: ok | 20:05 |
| frickler | mordred: o.k., I'll do some tests with that tomorrow, thx | 20:07 |
| *** yamamoto has joined #openstack-infra | 20:12 | |
| *** factor has joined #openstack-infra | 20:18 | |
| noonedeadpunk | o/ Returning back to our discussion of preinstalled venv - I think we got it working that way https://review.opendev.org/#/c/707212/18/zuul.d/playbooks/pre-gate-cleanup.yml - the only thing to adjust here is to extend pip package list with packages from https://review.opendev.org/#/c/707442/7/nodepool/elements/simple-pip/bin/cleanup-pip.sh | 20:18 |
| noonedeadpunk | just wanted to show what we've ended up with so that we could place this fix and once https://review.opendev.org/#/c/707442 will be ready we will replace cleanupwith running a command. | 20:20 |
| noonedeadpunk | And to make sure that we're not doing things even worse | 20:20 |
| fungi | that sounds fairly consistent with what i think we discussed so far | 20:23 |
| *** yamamoto has quit IRC | 20:23 | |
| fungi | thanks for the details! | 20:24 |
| fungi | (and the effective poc) | 20:24 |
| mordred | noonedeadpunk: oh awesome! yeah - I think that looks great | 20:28 |
| mordred | and yes - thanks for the POC of that | 20:28 |
| noonedeadpunk | ok, nice, thanks for your help as well! | 20:29 |
| mordred | fungi: the main thing I don't like about that cleanup script is the hardcoded list of packages ... I wish there was a good way to say "hey pip, what packages would you also install if you installed X" so we could capture that in a file that we used for the cleanup ... | 20:29 |
| mordred | but I think what I'm wishing for is for pip to be dpkg | 20:30 |
| fungi | could we generaet a pip freeze diff? | 20:30 |
| fungi | er, generate | 20:30 |
| mordred | oh - that's a great idea | 20:31 |
| mordred | I'll play with that | 20:31 |
| fungi | #status log used mailman's `rmlist` tool to retire the abandoned openstack-sos mailing list on lists.openstack.org following confirmation from the list owner | 20:44 |
| openstackstatus | fungi: finished logging | 20:44 |
| fungi | discussion for that was http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2020-01-16.log.html#t2020-01-16T14:15:14-2 i'm just somewhat backlogged ;) | 20:47 |
| ianw | infra-root: could i get a couple of eyes on https://review.opendev.org/#/c/707601/ to drop paramiko from the afs-release.py script, as bionic 2.0.0 doesn't have support for ed25519 keys :/ | 20:47 |
| fungi | yeesh | 20:48 |
| ianw | note i have done a manual run (because releases are failing) -> http://paste.openstack.org/show/789506/ | 20:49 |
| mordred | fungi: don't even need a diff - pip freeze | cut -f1 -d= after the initial installs gets the list | 20:51 |
| fungi | fair, it's a diff against the empty set, i suppose | 20:51 |
| fungi | an alternative is to do like i added in the wheel builder job and parse the pip log | 20:52 |
| mordred | ianw: I love the "It's always something ... " | 20:52 |
| mordred | fungi: that seems hard | 20:52 |
| fungi | yeah, likely overkill for this case | 20:52 |
| fungi | in that particular case it was preferable because i wanted to specifically identify which wheels were getting downloaded by pip vs built locally | 20:52 |
| fungi | and there was no simpler source for that info | 20:53 |
| weshay|ruck | hey o | 20:56 |
| weshay|ruck | hate asking this.. feels dirty https://review.opendev.org/#/c/707330/ | 20:57 |
| weshay|ruck | force push? | 20:57 |
| * weshay|ruck runs | 20:57 | |
| weshay|ruck | if you can | 20:57 |
| openstackgerrit | Monty Taylor proposed openstack/diskimage-builder master: Add get-pip element https://review.opendev.org/707499 | 20:57 |
| mordred | weshay|ruck: it looks like it's in the gate and the check jobs passed - why force push? | 20:58 |
| weshay|ruck | mordred, one job hiccup'd on tempest | 20:58 |
| weshay|ruck | https://8c0ba05a47a984d5626d-0d251d010a8cbb7a6dcf7a9d0b0f93e9.ssl.cf1.rackcdn.com/707330/5/gate/tripleo-ci-centos-7-standalone/e1d5572/logs/undercloud/var/log/tempest/stestr_results.html | 20:58 |
| EmilienM | sounds like we get hit by random timeouts which has nothing to do with the patch :/ | 20:59 |
| weshay|ruck | aye | 20:59 |
| mordred | and it looks like this is digging out of the virtualenv mess yeah? | 21:00 |
| EmilienM | correct | 21:00 |
| weshay|ruck | we come here only w/ the really fun stuff | 21:01 |
| EmilienM | :D | 21:01 |
| weshay|ruck | this also unlocks centos-8 containers .. first step in getting off centos-7 | 21:01 |
| mordred | merry christmas | 21:02 |
| * EmilienM sends hugs | 21:02 | |
| weshay|ruck | woot.. and a jolly good night! | 21:02 |
| weshay|ruck | thank you! | 21:02 |
| mordred | fungi: latest copy of 707499 has the pip freeze idea | 21:09 |
| fungi | awesome | 21:10 |
| *** sreejithp has quit IRC | 21:10 | |
| *** Lucas_Gray has joined #openstack-infra | 21:11 | |
| mordred | fungi: luckily we test dib patches with nodepool, so it's actually decently testable | 21:11 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] pip-and-virtualenv : home everything in environments https://review.opendev.org/707517 | 21:11 |
| ianw | yep, i'm still thinking through but didn't have much luck | 21:12 |
| ianw | fungi: i could *sort* of make your suggestions with --user / --root work | 21:13 |
| ianw | fungi: it would install the libraries in the system locations | 21:14 |
| *** neiljerram has quit IRC | 21:14 | |
| ianw | fungi: however, if it installed a bin/ file, that would have a #!/path/to/venv/python header | 21:14 |
| ianw | i could not seem to make --system-site-packages work as i hoped it would | 21:15 |
| openstackgerrit | Merged opendev/system-config master: afs-release: don't use paramiko https://review.opendev.org/707601 | 21:15 |
| fungi | #status log temporarily deactivated gerrit account 30112 for "Aditi Pai Dukle" due to comment spam http://lists.openstack.org/pipermail/openstack-infra/2020-February/006600.html | 21:21 |
| openstackstatus | fungi: finished logging | 21:21 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [dnm] ensure venv available, install glean with it https://review.opendev.org/707513 | 21:23 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] pip-and-virtualenv : home everything in environments https://review.opendev.org/707517 | 21:23 |
| *** pkopec_ has quit IRC | 21:23 | |
| mordred | ianw: I hate it when a tool seems so close to being able to be more flexible but just isn't | 21:23 |
| *** pkopec has joined #openstack-infra | 21:23 | |
| *** xek has quit IRC | 21:25 | |
| *** pkopec has quit IRC | 21:26 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Adjust io-level logging in gerrit/git drivers https://review.opendev.org/707728 | 21:28 |
| *** lbragsta_ has quit IRC | 21:29 | |
| mordred | ianw: I had this idea a while back that since rpm has a --root option it should be easy-ish to use it to be able to install stuff without permissions into ~/.local kind of like pip --user ... | 21:29 |
| mordred | NOPE | 21:29 |
| mordred | not even a little bit :) | 21:29 |
| *** dpawlik is now known as dpawlik_off | 21:30 | |
| fungi | okay, headed out to grab early dinner, back in a couple hours | 21:32 |
| *** slaweq has quit IRC | 22:01 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Gerrit checks: trigger new patchset behavior https://review.opendev.org/707729 | 22:01 |
| *** diablo_rojo has quit IRC | 22:04 | |
| *** fresta_ has quit IRC | 22:08 | |
| ianw | fungi: oooohhhh ... i see what's going on | 22:08 |
| ianw | you can install the latest pip in a virtualenv and use "pip install --prefix /usr/local" and it will, almost, work ... execpt | 22:09 |
| ianw | it will install everything into site-packages (so like /usr/local/lib/python2.7/site-packages) | 22:10 |
| smcginnis | Looks like we got an odd release-openstack-python failure: | 22:10 |
| smcginnis | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_957/146c4f450f903bcd3ae3bf158dd3c30c8015ec26/release/release-openstack-python/9572c35/job-output.txt | 22:10 |
| smcginnis | I've seen a few of these WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! failures lately. | 22:10 |
| smcginnis | With other patches, This is the first time on a release patch. | 22:10 |
| ianw | fungi: note, *not* dist-packages ... because the virtualenv hasn't had all the magic patching for that | 22:10 |
| *** rh-jelabarre has quit IRC | 22:10 | |
| smcginnis | Looks like it may have uploaded the tarball (can't really tell for sure) but skipped all the package publishing. | 22:11 |
| smcginnis | Anyone around that can reenqueue that job for us? | 22:11 |
| *** slaweq has joined #openstack-infra | 22:11 | |
| ianw | smcginnis: hrm, interesting, are they all on the same provider? my first suspicion would be ip reuse | 22:11 |
| ianw | smcginnis: i can look at rerunning that, just a minute | 22:11 |
| smcginnis | I have not noticed which provider, but I can start paying attention to that. | 22:12 |
| smcginnis | I just wrote it off on past ones I saw as a network fluke and moved on. | 22:12 |
| ianw | ubuntu-bionic-rax-ord-0014570426 ... rax seems less likely to have this happen, i don't remember ever seeing that before there | 22:13 |
| cmurphy | still seeing stackviz issues, it looked like the tarball was updated with https://review.opendev.org/707646 though? or do we have to wait for new images to be built with that cached? | 22:15 |
| *** slaweq has quit IRC | 22:16 | |
| *** eharney has quit IRC | 22:18 | |
| *** sshnaidm is now known as sshnaidm|afk | 22:21 | |
| smcginnis | ianw: I believe this would be the command: | 22:21 |
| smcginnis | zuul enqueue-ref --tenant=openstack --trigger=gerrit --pipeline=release --project=openstack/python-brick-cinderclient-ext --ref=refs/tags/1.0.0 --newrev=146c4f450f903bcd3ae3bf158dd3c30c8015ec26 | 22:21 |
| ianw | smcginnis: yep :) should be running | 22:22 |
| ianw | i wonder how much "[ubuntu-bionic] Waiting on logger" has to do with it | 22:24 |
| smcginnis | I hadn't seen that one before. | 22:25 |
| openstackgerrit | Merged zuul/zuul master: Report robot comments to gerrit https://review.opendev.org/707708 | 22:28 |
| *** gshippey has quit IRC | 22:29 | |
| openstackgerrit | Kendall Nelson proposed openstack/cookiecutter master: Add include for CONTRIBUTING.rst https://review.opendev.org/707735 | 22:41 |
| *** diablo_rojo has joined #openstack-infra | 22:41 | |
| *** Goneri has quit IRC | 22:51 | |
| *** tkajinam has joined #openstack-infra | 23:04 | |
| fungi | mordred: we got a ticket from rackspace about a bad hypervisor host impacting the instance named "mttest" | 23:10 |
| fungi | if you're still using that, you may want to double check it | 23:10 |
| fungi | if you're *not* still using that, it might be a good reminder to delete it | 23:10 |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** paladox|UKInEU has quit IRC | 23:14 | |
| *** slaweq has quit IRC | 23:16 | |
| diablo_rojo | clarkb, I'm going to vote -1 on your OpenDev governance patch since you haven't gotten the update to also move out the infra-manuals yet. I'll make mention of the decision in the meeting. | 23:17 |
| *** paladox has joined #openstack-infra | 23:17 | |
| clarkb | diablo_rojo: ok I'll try to get a new ps up this evening | 23:17 |
| diablo_rojo | clarkb, no rush (especially if you're feeling crappy). I just wanted to make sure people are aware. | 23:18 |
| *** Lucas_Gray has quit IRC | 23:26 | |
| diablo_rojo | And give you a heads up. | 23:26 |
| *** armax has quit IRC | 23:26 | |
| *** Lucas_Gray has joined #openstack-infra | 23:28 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [dnm] ensure venv available, install glean with it https://review.opendev.org/707513 | 23:29 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] pip-and-virtualenv : home everything in environments https://review.opendev.org/707517 | 23:29 |
| *** rlandy is now known as rlandy|bbl | 23:30 | |
| ianw | mordred/fungi: my current thinking, which changes every second, is to basically ditch the idea we can home the latest versions of pip & virtualenv in their own virtualenvs and retain the same semantics | 23:31 |
| *** dchen has joined #openstack-infra | 23:32 | |
| *** lbragstad_ has joined #openstack-infra | 23:32 | |
| fungi | you *can* do it for virtualenv. pip is... harder | 23:32 |
| ianw | yeah; bifurcating the problem further doesn't seem too useful a path to follow, though | 23:33 |
| fungi | and virtualenv'd pip will work for pip install --user but not really for sudo pip install | 23:33 |
| *** lbragstad has quit IRC | 23:33 | |
| ianw | fungi: right, see my point on dist- v site-packages above when pip tries to break out of it's virtualenv | 23:34 |
| ianw | things like fiddling site.py files in virtualenvs seems even *more* madness | 23:34 |
| fungi | i'd like to say let's treat `sudo pip install ...` as an outlier and force folks who need that to go to the extra steps to get a system-wide pip installation in place, but we probably have way too much in our job base right now assuming it can sudo pip install something with no extra work | 23:34 |
| ianw | what i can maybe see is if we can settle on using venv for installing util things in dib (glean, bindep, etc) and just symlinking them to /usr/local/bin, we could ship "bare" images and then do all the fixups in ansible | 23:36 |
| *** dchen has quit IRC | 23:40 | |
| *** dchen has joined #openstack-infra | 23:42 | |
| *** lbragstad__ has joined #openstack-infra | 23:42 | |
| *** lbragstad_ has quit IRC | 23:44 | |
| *** lbragstad_ has joined #openstack-infra | 23:49 | |
| *** lbragstad__ has quit IRC | 23:51 | |
| *** lbragstad__ has joined #openstack-infra | 23:56 | |
| *** lbragstad_ has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!