| *** weifan has joined #openstack-infra | 00:04 | |
| *** michael-beaver has quit IRC | 00:05 | |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Replace item with hard-coded value https://review.opendev.org/692246 | 00:06 |
|---|---|---|
| mordred | fungi, clarkb, ianw, Shrews: ^^ buildset registry is still broken after Shrews patch from earlier - that should fix it | 00:06 |
| *** gyee has quit IRC | 00:07 | |
| mordred | AJaeger: ^^ | 00:07 |
| mordred | also - if y'all have a second to look at https://review.opendev.org/#/c/691777 and https://review.opendev.org/#/c/692003 I'd appreciate it | 00:07 |
| mordred | I'll land those in just a bit | 00:07 |
| *** weifan has quit IRC | 00:09 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** slaweq has quit IRC | 00:15 | |
| *** xarses has joined #openstack-infra | 00:16 | |
| *** xarses has quit IRC | 00:16 | |
| *** xarses has joined #openstack-infra | 00:17 | |
| *** persia has quit IRC | 00:18 | |
| *** rajinir has quit IRC | 00:18 | |
| *** gsantomaggio has quit IRC | 00:18 | |
| *** mgoddard has quit IRC | 00:22 | |
| *** rajinir has joined #openstack-infra | 00:24 | |
| *** persia has joined #openstack-infra | 00:24 | |
| *** gsantomaggio has joined #openstack-infra | 00:24 | |
| *** markvoelker has joined #openstack-infra | 00:30 | |
| *** jamesmcarthur has joined #openstack-infra | 00:40 | |
| openstackgerrit | Merged opendev/base-jobs master: Replace item with hard-coded value https://review.opendev.org/692246 | 00:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Plumb through secure.config contents https://review.opendev.org/691800 | 00:56 |
| *** jamesmcarthur has quit IRC | 01:02 | |
| *** ociuhandu has joined #openstack-infra | 01:08 | |
| *** markvoelker has quit IRC | 01:09 | |
| *** ociuhandu has quit IRC | 01:22 | |
| *** rh-jelabarre has quit IRC | 01:22 | |
| Shrews | mordred: oops. thx for fixing my fix | 01:34 |
| mordred | Shrews: np! | 01:40 |
| *** ociuhandu has joined #openstack-infra | 01:51 | |
| *** ociuhandu has quit IRC | 01:57 | |
| *** ricolin has joined #openstack-infra | 02:05 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Plumb through secure.config contents https://review.opendev.org/691800 | 02:06 |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** slaweq has quit IRC | 02:16 | |
| *** jtomasek has quit IRC | 02:16 | |
| *** jklare has joined #openstack-infra | 02:17 | |
| *** diablo_rojo has quit IRC | 02:18 | |
| *** jerryz has quit IRC | 02:20 | |
| ianw | pabelanger: https://launchpad.net/~openstack-ci-core/+archive/ubuntu/vhd-util/+packages ... i updated the patch and it's rebuilt for bionic | 02:24 |
| *** ykarel|away has joined #openstack-infra | 02:32 | |
| *** diablo_rojo has joined #openstack-infra | 02:33 | |
| *** ykarel|away has quit IRC | 02:38 | |
| *** tkajinam has quit IRC | 03:04 | |
| *** diablo_rojo has quit IRC | 03:05 | |
| *** anteaya has left #openstack-infra | 03:05 | |
| *** markvoelker has joined #openstack-infra | 03:10 | |
| *** markvoelker has quit IRC | 03:16 | |
| *** ociuhandu has joined #openstack-infra | 03:24 | |
| *** ociuhandu has quit IRC | 03:29 | |
| *** psachin has joined #openstack-infra | 03:33 | |
| *** ociuhandu has joined #openstack-infra | 03:45 | |
| *** ociuhandu has quit IRC | 03:50 | |
| *** igordc has joined #openstack-infra | 04:09 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Plumb through secure.config contents https://review.opendev.org/691800 | 04:13 |
| *** tkajinam has joined #openstack-infra | 04:14 | |
| *** slaweq has quit IRC | 04:16 | |
| *** ykarel|away has joined #openstack-infra | 04:27 | |
| *** surpatil has joined #openstack-infra | 04:27 | |
| *** dchen has quit IRC | 04:30 | |
| *** ykarel|away is now known as ykarel | 04:34 | |
| *** soniya29 has joined #openstack-infra | 04:37 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Stub out dry run in the uploader https://review.opendev.org/592929 | 04:44 |
| ianw | jhesketh: ^ thanks for review, sorry for missing that first time | 04:45 |
| *** Buggys has joined #openstack-infra | 04:59 | |
| *** ociuhandu has joined #openstack-infra | 05:01 | |
| *** soniya29 has quit IRC | 05:02 | |
| *** ociuhandu has quit IRC | 05:06 | |
| *** udesale has joined #openstack-infra | 05:12 | |
| *** yamamoto has joined #openstack-infra | 05:14 | |
| *** dchen has joined #openstack-infra | 05:21 | |
| *** rakhmerov has quit IRC | 05:24 | |
| *** raukadah is now known as chandankumar | 05:27 | |
| *** ociuhandu has joined #openstack-infra | 05:30 | |
| *** yamamoto has quit IRC | 05:35 | |
| *** ociuhandu has quit IRC | 05:35 | |
| *** jtomasek has joined #openstack-infra | 05:36 | |
| *** yamamoto has joined #openstack-infra | 05:37 | |
| *** ociuhandu has joined #openstack-infra | 05:41 | |
| *** kjackal has joined #openstack-infra | 05:41 | |
| *** ociuhandu has quit IRC | 05:45 | |
| *** dave-mccowan has quit IRC | 05:51 | |
| *** gagehugo has quit IRC | 06:02 | |
| *** dpawlik has joined #openstack-infra | 06:05 | |
| *** gagehugo has joined #openstack-infra | 06:10 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** slaweq has quit IRC | 06:15 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: upload-logs-swift: test bulk download script change https://review.opendev.org/689930 | 06:26 |
| openstackgerrit | Merged zuul/zuul-jobs master: upload-logs-swift: Stub out dry run in the uploader https://review.opendev.org/592929 | 06:27 |
| openstackgerrit | Merged zuul/zuul-jobs master: upload-logs-swift: test bulk download script change https://review.opendev.org/689930 | 06:36 |
| *** janki has joined #openstack-infra | 06:37 | |
| *** icarusfactor has quit IRC | 06:38 | |
| *** ccamacho has quit IRC | 06:39 | |
| *** tesseract has joined #openstack-infra | 06:45 | |
| *** kjackal has quit IRC | 06:50 | |
| *** dpawlik has quit IRC | 06:57 | |
| *** dpawlik has joined #openstack-infra | 06:58 | |
| *** dpawlik has quit IRC | 07:07 | |
| *** markvoelker has joined #openstack-infra | 07:13 | |
| *** markvoelker has quit IRC | 07:17 | |
| *** dpawlik has joined #openstack-infra | 07:20 | |
| *** iurygregory has joined #openstack-infra | 07:22 | |
| *** surpatil has quit IRC | 07:31 | |
| *** igordc has quit IRC | 07:34 | |
| *** dpawlik has quit IRC | 07:35 | |
| *** brtknr has quit IRC | 07:42 | |
| *** brtknr has joined #openstack-infra | 07:46 | |
| *** kjackal has joined #openstack-infra | 07:46 | |
| *** yamamoto has quit IRC | 07:48 | |
| *** pcaruana has joined #openstack-infra | 07:50 | |
| *** yamamoto has joined #openstack-infra | 07:51 | |
| *** lpetrut has joined #openstack-infra | 07:51 | |
| *** trident has quit IRC | 07:53 | |
| *** auristor has quit IRC | 07:53 | |
| *** slaweq has joined #openstack-infra | 07:55 | |
| *** pgaxatte has joined #openstack-infra | 07:55 | |
| *** yamamoto has quit IRC | 07:55 | |
| *** dpawlik has joined #openstack-infra | 07:56 | |
| *** auristor has joined #openstack-infra | 07:57 | |
| *** slaweq has quit IRC | 07:57 | |
| *** trident has joined #openstack-infra | 08:00 | |
| *** kopecmartin|off is now known as kopecmartin | 08:01 | |
| *** ykarel is now known as ykarel|lunch | 08:02 | |
| *** amoralej|off is now known as amoralej | 08:04 | |
| *** yamamoto has joined #openstack-infra | 08:04 | |
| *** slaweq has joined #openstack-infra | 08:06 | |
| AJaeger | ianw: the commit message talks about a job for base-test - is that up? | 08:12 |
| *** gfidente has joined #openstack-infra | 08:27 | |
| *** yamamoto has quit IRC | 08:36 | |
| *** jpena|off is now known as jpena | 08:36 | |
| *** zbr has quit IRC | 08:37 | |
| *** dchen has quit IRC | 08:39 | |
| *** tkajinam has quit IRC | 08:39 | |
| *** ralonsoh has joined #openstack-infra | 08:44 | |
| *** surpatil has joined #openstack-infra | 08:51 | |
| *** ccamacho has joined #openstack-infra | 08:51 | |
| *** rpittau|afk is now known as rpittau | 08:52 | |
| *** yamamoto has joined #openstack-infra | 08:55 | |
| *** pkopec has quit IRC | 09:02 | |
| *** lucasagomes has joined #openstack-infra | 09:03 | |
| *** pkopec has joined #openstack-infra | 09:07 | |
| *** ccamacho has quit IRC | 09:09 | |
| *** pgaxatte has quit IRC | 09:11 | |
| *** zbr has joined #openstack-infra | 09:11 | |
| *** hashar has joined #openstack-infra | 09:12 | |
| *** pgaxatte has joined #openstack-infra | 09:13 | |
| *** lucasagomes has quit IRC | 09:14 | |
| *** lucasagomes has joined #openstack-infra | 09:14 | |
| *** ociuhandu has joined #openstack-infra | 09:15 | |
| *** mgoddard has joined #openstack-infra | 09:15 | |
| *** yamamoto has quit IRC | 09:17 | |
| *** licanwei has joined #openstack-infra | 09:18 | |
| openstackgerrit | Harald Jensås proposed openstack/diskimage-builder master: WIP: Add IPv6 support in dhcp-all-interfaces https://review.opendev.org/692110 | 09:19 |
| *** aspiers has joined #openstack-infra | 09:21 | |
| *** yamamoto has joined #openstack-infra | 09:24 | |
| *** yamamoto has quit IRC | 09:24 | |
| openstackgerrit | Harald Jensås proposed openstack/diskimage-builder master: WIP: Add IPv6 support in dhcp-all-interfaces https://review.opendev.org/692110 | 09:26 |
| *** ykarel|lunch is now known as ykarel | 09:29 | |
| *** priteau has joined #openstack-infra | 09:33 | |
| *** trident has quit IRC | 09:42 | |
| *** rascasoft has joined #openstack-infra | 09:44 | |
| *** rascasoft has quit IRC | 09:44 | |
| *** rascasoft has joined #openstack-infra | 09:45 | |
| *** xek_ has joined #openstack-infra | 09:45 | |
| *** surpatil has quit IRC | 09:48 | |
| *** trident has joined #openstack-infra | 09:49 | |
| *** ociuhandu has quit IRC | 10:19 | |
| *** dtantsur|afk is now known as dtantsur | 10:20 | |
| licanwei | hi~ | 10:37 |
| licanwei | Does anyone have the right of OpenStack Infra (hudson-openstack)? | 10:37 |
| AJaeger | licanwei: on which system and why? | 10:38 |
| licanwei | Watcher launchpad Maintainer was mistakenly changed to openstack infra | 10:38 |
| licanwei | Can you help to change back? thanks | 10:39 |
| licanwei | https://launchpad.net/watcher | 10:39 |
| licanwei | pls change to watcher drivers | 10:39 |
| AJaeger | licanwei: No, I don't have those permissions, maybe some other infra-root later on (note some are travelling today). | 10:41 |
| licanwei | AJaeger: ok, thanks | 10:42 |
| *** rpittau is now known as rpittau|bbl | 11:00 | |
| *** dchen has joined #openstack-infra | 11:07 | |
| *** surpatil has joined #openstack-infra | 11:08 | |
| *** markvoelker has joined #openstack-infra | 11:15 | |
| *** tkajinam has joined #openstack-infra | 11:17 | |
| *** markvoelker has quit IRC | 11:20 | |
| *** rh-jelabarre has joined #openstack-infra | 11:25 | |
| *** pgaxatte has quit IRC | 11:26 | |
| *** ccamacho has joined #openstack-infra | 11:30 | |
| *** rh-jelabarre has quit IRC | 11:32 | |
| *** rh-jelabarre has joined #openstack-infra | 11:33 | |
| *** goldyfruit___ has joined #openstack-infra | 11:39 | |
| *** goldyfruit___ has quit IRC | 11:41 | |
| *** dchen has quit IRC | 11:45 | |
| *** arxcruz is now known as arxcruz|lunch | 12:01 | |
| *** rfolco has joined #openstack-infra | 12:11 | |
| *** FlorianFa has joined #openstack-infra | 12:13 | |
| *** markvoelker has joined #openstack-infra | 12:13 | |
| *** udesale has quit IRC | 12:14 | |
| *** jpena is now known as jpena|lunch | 12:14 | |
| *** udesale has joined #openstack-infra | 12:14 | |
| *** rlandy has joined #openstack-infra | 12:17 | |
| *** jamesmcarthur has joined #openstack-infra | 12:24 | |
| *** dave-mccowan has joined #openstack-infra | 12:29 | |
| AJaeger | clarkb: looking at https://review.opendev.org/#/q/status:open+topic:use-opensuse-15, I do not see that the stable/stein changes for bifrost, nova, glance will pass tests - these need IMHO a force merge. And then the rest can merge after recheck... Your two changes for devstack and tempest should be duplicates of dirk's. I suggest to abandon yours. | 12:29 |
| *** jamesmcarthur has quit IRC | 12:35 | |
| *** jamesmcarthur has joined #openstack-infra | 12:41 | |
| *** ykarel is now known as ykarel|afk | 12:48 | |
| *** Goneri has joined #openstack-infra | 13:01 | |
| *** jamesmcarthur has quit IRC | 13:04 | |
| *** jpena|lunch is now known as jpena | 13:05 | |
| *** ykarel|afk is now known as ykarel | 13:06 | |
| *** tkajinam has quit IRC | 13:06 | |
| *** jamesmcarthur has joined #openstack-infra | 13:07 | |
| *** arxcruz|lunch is now known as arxcruz | 13:09 | |
| *** yamamoto has joined #openstack-infra | 13:11 | |
| *** mriedem has joined #openstack-infra | 13:13 | |
| *** yamamoto has quit IRC | 13:14 | |
| *** rpittau|bbl is now known as rpittau | 13:16 | |
| *** panda is now known as panda|pto | 13:16 | |
| *** pgaxatte has joined #openstack-infra | 13:17 | |
| *** sshnaidm is now known as sshnaidm|rover | 13:23 | |
| *** dtantsur is now known as dtantsur|brb | 13:28 | |
| *** goldyfruit has joined #openstack-infra | 13:28 | |
| *** janki has quit IRC | 13:28 | |
| *** jaosorior has joined #openstack-infra | 13:30 | |
| *** psachin has quit IRC | 13:35 | |
| *** jamesmcarthur has quit IRC | 13:40 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for nova functional test fail bug 1850682 https://review.opendev.org/692203 | 13:42 |
| openstack | bug 1850682 in OpenStack Compute (nova) "functional tests in rocky randomly fail with "Build of instance was re-scheduled: Cannot modify readonly field uuid"" [Undecided,Confirmed] https://launchpad.net/bugs/1850682 | 13:42 |
| *** dklyle has quit IRC | 14:00 | |
| *** dklyle has joined #openstack-infra | 14:01 | |
| *** jcoufal has joined #openstack-infra | 14:07 | |
| *** amoralej is now known as amoralej|lunch | 14:12 | |
| fungi | licanwei: i'll take a look now | 14:13 |
| fungi | this could take a few minutes. i'll need to dig up credentials to log into lp as that user | 14:14 |
| *** pgaxatte has quit IRC | 14:15 | |
| *** dpawlik has quit IRC | 14:16 | |
| *** ociuhandu has joined #openstack-infra | 14:17 | |
| fungi | licanwei: similarly, can you please change the owner of the https://launchpad.net/~watcher-drivers group to OpenStack Administrators (openstack-admins)? | 14:18 |
| *** pkopec has quit IRC | 14:19 | |
| fungi | #status log corrected the https://launchpad.net/watcher owner from hudson-openstack to watcher-drivers at licanwei's request | 14:20 |
| openstackstatus | fungi: finished logging | 14:20 |
| *** ociuhandu has quit IRC | 14:22 | |
| *** rkukura has quit IRC | 14:25 | |
| fungi | #status log manually updated the project description at https://storyboard.openstack.org/#!/project/openstack/whereto to indicate it uses launchpad | 14:25 |
| openstackstatus | fungi: finished logging | 14:25 |
| *** rkukura has joined #openstack-infra | 14:26 | |
| *** rkukura has quit IRC | 14:26 | |
| *** ykarel is now known as ykarel|away | 14:27 | |
| *** pgaxatte has joined #openstack-infra | 14:28 | |
| *** goldyfruit has quit IRC | 14:29 | |
| *** goldyfruit has joined #openstack-infra | 14:32 | |
| *** mattw4 has joined #openstack-infra | 14:38 | |
| *** dtantsur|brb is now known as dtantsur | 14:40 | |
| openstackgerrit | Julia Kreger proposed openstack/diskimage-builder master: Enable IPv6 DHCPv6 https://review.opendev.org/690777 | 14:41 |
| *** lpetrut has quit IRC | 14:43 | |
| *** rh-jelabarre has quit IRC | 14:45 | |
| *** yamamoto has joined #openstack-infra | 14:47 | |
| *** amoralej|lunch is now known as amoralej | 14:49 | |
| *** yamamoto has quit IRC | 14:51 | |
| *** yamamoto has joined #openstack-infra | 14:51 | |
| *** yamamoto has quit IRC | 14:59 | |
| *** lastmikoi has left #openstack-infra | 15:00 | |
| *** mattw4 has quit IRC | 15:03 | |
| *** yamamoto has joined #openstack-infra | 15:04 | |
| *** surpatil has quit IRC | 15:08 | |
| openstackgerrit | Thierry Carrez proposed openstack/ptgbot master: Add update instructions to etherpads page https://review.opendev.org/692393 | 15:18 |
| *** sreejithp has joined #openstack-infra | 15:19 | |
| *** lucasagomes has quit IRC | 15:21 | |
| *** lucasagomes has joined #openstack-infra | 15:22 | |
| efried | clarkb, fungi, corvus: Sorry to bug y'all here, but you're all together... | 15:22 |
| efried | I noticed y'all are osc cores; would you mind approving https://review.opendev.org/#/c/691980/ to unblock it? | 15:22 |
| efried | mordred was +2 before a trivial pep8 fix fwiw | 15:23 |
| *** armax has quit IRC | 15:25 | |
| *** rlandy is now known as rlandy|ruck | 15:32 | |
| *** goldyfruit_ has joined #openstack-infra | 15:32 | |
| *** michael-beaver has joined #openstack-infra | 15:34 | |
| *** goldyfruit has quit IRC | 15:35 | |
| efried | Shrews: ^ | 15:37 |
| Shrews | efried: is that blocking something important? i'm hesitant to approve something in a code base i've never looked at | 15:39 |
| * Shrews reads review history... | 15:40 | |
| efried | Shrews: It's blocking anything from merging in python-openstackclient. Importance of things waiting to merge is debatable I suppose, but there's at least one thing holding up the release of python-novaclient 16.0.0 -- mriedem? | 15:41 |
| efried | Shrews: The changes are test-only and trivially py2->3 fwiw, not really requiring understanding of osc. | 15:41 |
| efried | but I just saw you on the core list, didn't realize you weren't really into the codebase. | 15:42 |
| *** jtomasek has quit IRC | 15:44 | |
| Shrews | gimme a few to go through it | 15:45 |
| *** jtomasek has joined #openstack-infra | 15:46 | |
| Shrews | efried: oh, actually, i don't have the approval powers there | 15:46 |
| clarkb | efried: I'm not sure I grok the file mode changes. Some of them are for reads only like line 111 here https://review.opendev.org/#/c/691980/3/openstackclient/tests/functional/compute/v2/test_keypair.py | 15:47 |
| efried | oh, whoops. I must have been looking at the wrong core list :( | 15:47 |
| efried | clarkb: Yeah, it was for the sake of minimal change from the default of 'w+b' | 15:47 |
| efried | that could stand to be called out | 15:47 |
| efried | ...but it looks like you're not core there anyway, my bad. | 15:48 |
| clarkb | efried: why nit open it r only? | 15:48 |
| efried | yup, could have probably, but that would be an additional unrelated change. | 15:48 |
| efried | really, I was trying to do the minimum possible to get the tests green. | 15:49 |
| efried | Anyway, sorry to bother y'all, not sure which core list I was looking at that had all of you in it. I need mordred dtroyer amotoki. | 15:52 |
| Shrews | you were probably looking at openstacksdk-core | 15:53 |
| *** rh-jelabarre has joined #openstack-infra | 15:54 | |
| Shrews | infra has a much more vested interest in that :) | 15:54 |
| *** FlorianFa has quit IRC | 15:54 | |
| efried | ah | 15:55 |
| *** mattw4 has joined #openstack-infra | 15:55 | |
| *** lpetrut has joined #openstack-infra | 15:59 | |
| Shrews | corvus, mordred: the buildset registry jobs insist on finding new ways to be broken: https://zuul.opendev.org/t/zuul/build/69a7699ece2748f485442d4ebd863d86/log/job-output.txt#420 | 15:59 |
| *** lpetrut has quit IRC | 16:00 | |
| *** lpetrut has joined #openstack-infra | 16:00 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul-jobs master: Add requests to list of python packages to install https://review.opendev.org/692400 | 16:08 |
| *** kopecmartin is now known as kopecmartin|off | 16:10 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: IGNORE: Testing buildset registry job https://review.opendev.org/692401 | 16:11 |
| *** kjackal has quit IRC | 16:14 | |
| AJaeger | Shrews: I think you want https://packages.ubuntu.com/bionic/python3-requests, don't you? | 16:16 |
| AJaeger | Shrews: https://packages.ubuntu.com/bionic/requests does not exist | 16:17 |
| Shrews | AJaeger: maybe? that's why i put up that IGNORE review | 16:17 |
| * Shrews tests in venv | 16:17 | |
| AJaeger | Won't help - you install *DEB* packages here, not pypi ones | 16:17 |
| Shrews | oh! haha | 16:18 |
| AJaeger | So, rename to python-requests and python3-requests | 16:18 |
| Shrews | AJaeger: yup. thx | 16:18 |
| AJaeger | and try again ;) | 16:18 |
| Shrews | i'm still unclear why this is just now happening | 16:18 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul-jobs master: Add requests to list of python packages to install https://review.opendev.org/692400 | 16:19 |
| AJaeger | did we install it previously but not anymore? | 16:19 |
| *** hashar has quit IRC | 16:20 | |
| Shrews | the package list was altered a bit in mid Oct | 16:20 |
| Shrews | https://review.opendev.org/#/c/689238/10/roles/run-buildset-registry/tasks/main.yaml | 16:21 |
| AJaeger | Shrews: that merged two days ago - but was part of the buildset registry jobs changes.... | 16:22 |
| Shrews | AJaeger: oh, so it did. i was looking at the date in 'git log' | 16:23 |
| Shrews | it's been broken for the past couple of days, so we probably just haven't gotten this far to see it yet | 16:24 |
| AJaeger | maybe ;) That shows that we didn't test the change fully :( | 16:24 |
| Shrews | i'm not certain my test of the change will actually have a test that uses that role. i saw the failure during the zuul-upload-image gate job, so will have to poke around a bit | 16:27 |
| Shrews | hoping the zuul-build-image check job uses it | 16:28 |
| AJaeger | Looks like 692401 is still getting the same failure - is the job not installing the package list? | 16:28 |
| *** weshay|rover is now known as weshay | 16:30 | |
| *** pgaxatte has quit IRC | 16:31 | |
| Shrews | i see a new error in 400: "msg": "[Errno 2] No such file or directory: 'docker': 'docker'" | 16:31 |
| *** lmiccini has quit IRC | 16:31 | |
| AJaeger | That's strange ; | 16:32 |
| Shrews | this is all cattywhampus | 16:32 |
| *** tesseract has quit IRC | 16:33 | |
| * AJaeger needs the full log to see what got installed - have to wait a bit... | 16:33 | |
| *** kjackal has joined #openstack-infra | 16:35 | |
| *** kjackal has quit IRC | 16:36 | |
| *** kjackal has joined #openstack-infra | 16:36 | |
| *** efried has quit IRC | 16:37 | |
| *** markvoelker has quit IRC | 16:40 | |
| *** markvoelker has joined #openstack-infra | 16:43 | |
| *** lucasagomes has quit IRC | 16:44 | |
| AJaeger | https://review.opendev.org/692400 is failing directly, no need for a depends-on at least ;) | 16:44 |
| Shrews | https://zuul.opendev.org/t/zuul/build/4183fcad79e7410395a487ba71451e5a/log/job-output.txt#720 | 16:44 |
| *** dtantsur is now known as dtantsur|afk | 16:45 | |
| Shrews | but that's ignored, looks like? | 16:45 |
| AJaeger | https://zuul.opendev.org/t/zuul/build/a6a58b1f1f37423f86a3f0751d963f79/console#1/0/28/ubuntu-bionic shows that python-requests gets installed | 16:45 |
| Shrews | yet the error still shows: https://zuul.opendev.org/t/zuul/build/4183fcad79e7410395a487ba71451e5a/log/job-output.txt#830 | 16:46 |
| Shrews | we must be missing something | 16:46 |
| AJaeger | Yep ;( | 16:47 |
| AJaeger | Are we installing locally instead of remote (or vice versa)? | 16:47 |
| Shrews | i'm thinking yeah. | 16:47 |
| Shrews | it's using image: zuul/zuul-registry:latest | 16:48 |
| Shrews | so i guess we need to find where that's created | 16:48 |
| *** lpetrut has quit IRC | 16:49 | |
| *** prometheanfire has quit IRC | 16:50 | |
| *** gyee has joined #openstack-infra | 16:50 | |
| *** prometheanfire has joined #openstack-infra | 16:51 | |
| AJaeger | sorry, won't be able to help further - hope you find the problem, Shrews ! | 16:52 |
| Shrews | np. only affecting nodepool and zuul i think, and most of those devs are flying around the world | 16:53 |
| * Shrews lunches | 16:54 | |
| *** trident has quit IRC | 16:55 | |
| *** trident has joined #openstack-infra | 17:01 | |
| * fungi hits the road. see some of you in shanghai next week! | 17:02 | |
| AJaeger | safe travels, fungi ! | 17:03 |
| *** ricolin has quit IRC | 17:03 | |
| fungi | thanks! | 17:05 |
| *** igordc has joined #openstack-infra | 17:05 | |
| *** yamamoto has quit IRC | 17:08 | |
| *** jamesmcarthur has joined #openstack-infra | 17:11 | |
| *** ykarel|away has quit IRC | 17:11 | |
| *** udesale has quit IRC | 17:11 | |
| *** armax has joined #openstack-infra | 17:12 | |
| *** jamesmcarthur has quit IRC | 17:12 | |
| *** jamesmcarthur has joined #openstack-infra | 17:12 | |
| *** rpittau is now known as rpittau|afk | 17:14 | |
| *** weifan has joined #openstack-infra | 17:16 | |
| *** gfidente is now known as gfidente|afk | 17:18 | |
| *** weifan has quit IRC | 17:18 | |
| *** chandankumar is now known as alaadin | 17:22 | |
| *** alaadin is now known as raukadah | 17:23 | |
| *** jamesmcarthur has quit IRC | 17:28 | |
| *** jamesmcarthur has joined #openstack-infra | 17:28 | |
| *** jamesmcarthur has quit IRC | 17:31 | |
| *** jpena is now known as jpena|off | 17:34 | |
| *** jamesmcarthur has joined #openstack-infra | 17:35 | |
| *** jamesmcarthur has quit IRC | 17:36 | |
| *** armax has quit IRC | 17:36 | |
| *** harlowja has quit IRC | 17:44 | |
| *** yamamoto has joined #openstack-infra | 17:48 | |
| *** yamamoto has quit IRC | 17:59 | |
| *** jaosorior has quit IRC | 18:04 | |
| *** jcoufal has quit IRC | 18:06 | |
| *** jerryz has joined #openstack-infra | 18:08 | |
| *** ociuhandu has joined #openstack-infra | 18:19 | |
| *** amoralej is now known as amoralej|off | 18:19 | |
| *** armax has joined #openstack-infra | 18:22 | |
| *** ociuhandu has quit IRC | 18:23 | |
| *** priteau has quit IRC | 18:28 | |
| *** yamamoto has joined #openstack-infra | 18:34 | |
| *** sshnaidm|rover is now known as sshnaidm|afk | 18:41 | |
| *** ralonsoh has quit IRC | 18:47 | |
| *** paladox has quit IRC | 18:52 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul-jobs master: Add requests to list of python packages to install https://review.opendev.org/692400 | 18:54 |
| *** paladox has joined #openstack-infra | 18:55 | |
| *** rascasoft has quit IRC | 18:55 | |
| *** rascasoft has joined #openstack-infra | 18:55 | |
| *** rkukura has joined #openstack-infra | 19:14 | |
| *** dave-mccowan has quit IRC | 19:16 | |
| *** kjackal has quit IRC | 19:17 | |
| *** mriedem is now known as mriedem_afk | 19:21 | |
| AJaeger | Shrews: why did you abandon this? What did you find out? | 19:25 |
| Shrews | AJaeger: because even though we are installing requests, the docker_container Ansible module is still failing. I suspect the issue to be something else. | 19:28 |
| AJaeger | I see... | 19:30 |
| Shrews | I wonder if docker_container is using the wrong python version? | 19:30 |
| Shrews | nah, that can't be it | 19:32 |
| *** dave-mccowan has joined #openstack-infra | 19:32 | |
| *** yamamoto has quit IRC | 19:34 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul-jobs master: Add requests to list of python packages to install https://review.opendev.org/692400 | 19:36 |
| Shrews | testing something | 19:36 |
| *** sshnaidm|afk is now known as sshnaidm|off | 19:37 | |
| Shrews | ya know... if ansible_python_interpreter is 'auto', that could break it, yeah? | 19:38 |
| Shrews | according to https://zuul.opendev.org/t/zuul/build/4183fcad79e7410395a487ba71451e5a/log/zuul-info/inventory.yaml, i believe that's what it is | 19:39 |
| *** lastmikoi has joined #openstack-infra | 19:43 | |
| AJaeger | Shrews: https://zuul.opendev.org/t/zuul/build/a6a58b1f1f37423f86a3f0751d963f79/log/zuul-info/inventory.yaml#7 | 19:47 |
| AJaeger | it's auto | 19:47 |
| AJaeger | Argh, yeah - sorry, copied same file basically | 19:47 |
| AJaeger | yes, that could explain that we install the wrong stuff | 19:48 |
| Shrews | yeah | 19:48 |
| Shrews | attempting to figure out the way to determine the version when using auto | 19:48 |
| Shrews | probably in an ansible fact | 19:48 |
| AJaeger | In case of auto and having python2, it would install python3 packages ;( | 19:48 |
| AJaeger | no, the other way round... | 19:49 |
| AJaeger | python3 not in interpreter -> would be true for auto | 19:49 |
| AJaeger | why did it work before? What did we change that it breaks now? | 19:49 |
| Shrews | there is an ansible_python fact and an ansible_python_version fact | 19:50 |
| Shrews | AJaeger: i don't know | 19:51 |
| AJaeger | neither do I ;( | 19:51 |
| *** diablo_rojo has joined #openstack-infra | 19:51 | |
| Shrews | the registry stuff has been moving so fast, keeping up is very difficult | 19:52 |
| AJaeger | the registry stuff is the last thing merged into zuul-jobs | 19:54 |
| Shrews | http://paste.openstack.org/show/785694/ | 19:54 |
| *** eharney has quit IRC | 19:54 | |
| Shrews | i think we should pull the info from there ^^ | 19:55 |
| Shrews | that's when i set ansible_python_interpreter=auto in my hosts file | 19:55 |
| AJaeger | codesearch showed me https://opendev.org/recordsansible/ara/src/branch/master/roles/ara_api/tasks/pre-requirements.yaml#L19 | 19:56 |
| Shrews | if i set it to python3, it changes appropriately | 19:56 |
| AJaeger | Also, we do already https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/validate-host/tasks/main.yaml#L37 | 19:56 |
| AJaeger | those are the only two current uses | 19:56 |
| Shrews | ooh, that last one is promising | 19:57 |
| AJaeger | hope you figure it out - I'm calling it a day... | 19:57 |
| Shrews | i wish i could find docs on how to use that version() filter | 20:03 |
| *** markvoelker has quit IRC | 20:06 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul-jobs master: Add requests to list of python packages to install https://review.opendev.org/692400 | 20:07 |
| Shrews | if that works, i bet we don't need to install requests since it should be a dependency of the docker install | 20:13 |
| Shrews | w00t! passed the failing task | 20:16 |
| *** jamesdenton has quit IRC | 20:17 | |
| *** jamesden_ has joined #openstack-infra | 20:17 | |
| Shrews | zuul-jobs-test-registry now passes, but zuul-jobs-test-registry-buildset-registry is still failing for some reason | 20:20 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul-jobs master: run-buildset-registry: Fix python version check https://review.opendev.org/692400 | 20:21 |
| Shrews | for some reason, zuul-jobs-test-registry-buildset-registry still fails with that error, but it fixes the other job | 20:26 |
| Shrews | i'm super confused | 20:26 |
| * Shrews realizes he is mostly talking to himself, but recording his thoughts for folks who get bored when they get to China | 20:27 | |
| Shrews | fungi: i think we may have some zuul-finger daemons killed again since i'm not getting streaming logs sometimes | 20:28 |
| *** harlowja has joined #openstack-infra | 20:29 | |
| *** diablo_rojo has quit IRC | 20:31 | |
| Shrews | corvus: about to call it a day, but you might want to check out 692400. It fixes the zuul-jobs-test-registry job but not the zuul-jobs-test-registry-buildset-registry job and I'm unclear why | 20:36 |
| *** goldyfruit_ has quit IRC | 20:40 | |
| *** mriedem_afk is now known as mriedem | 20:44 | |
| *** eharney has joined #openstack-infra | 20:50 | |
| *** rkukura has quit IRC | 20:51 | |
| *** Goneri has quit IRC | 20:58 | |
| *** eharney has quit IRC | 20:59 | |
| *** xek_ has quit IRC | 21:04 | |
| *** ociuhandu has joined #openstack-infra | 21:08 | |
| *** rlandy|ruck has quit IRC | 21:09 | |
| *** ociuhandu has quit IRC | 21:13 | |
| *** rlandy has joined #openstack-infra | 21:15 | |
| *** rlandy is now known as rlandy|ruck | 21:15 | |
| clarkb | Looks like we are still at zero rax nodes | 21:25 |
| *** rfolco is now known as rfolco|afk | 21:29 | |
| *** pcaruana has quit IRC | 21:31 | |
| ianw | clarkb: i haven't checked that ticket, will now (just have to get login) | 21:35 |
| *** mriedem has quit IRC | 21:36 | |
| clarkb | thanks. I'm packing for trip and trying to take it easy as I don't get home until more than a week from now | 21:36 |
| ianw | there is a response but not on turning back quota | 21:37 |
| auristor | ianw: fedora 31 was released on Tuesday. | 21:41 |
| auristor | dnf install krb5-workstation kafs-client | 21:41 |
| auristor | systemctl start afs.mount | 21:41 |
| auristor | systemctl enable afs.mount | 21:41 |
| auristor | ls /afs/openstack.org/ | 21:41 |
| ianw | auristor: excellent! i haven't got f31 building in our infra yet, but will do soon | 21:43 |
| *** kjackal has joined #openstack-infra | 21:46 | |
| *** jtomasek has quit IRC | 21:47 | |
| ianw | clarkb: updated with a request to restore quota | 21:48 |
| fungi | Shrews: yeah, when i looked yesterday there were a couple of executors (9 and 12 i think?) which had sacrificed their finger servers to oom events since the last full system restart a week ago | 21:56 |
| *** sreejithp has quit IRC | 21:56 | |
| * fungi just reached his hotel for the night, may be able to take a look in a bit | 21:56 | |
| clarkb | I've just about finished putting clothes in a bag | 21:57 |
| clarkb | ianw: thanks! | 21:57 |
| *** zbr has quit IRC | 21:58 | |
| *** zbr has joined #openstack-infra | 21:59 | |
| *** mattw4 has quit IRC | 22:06 | |
| *** mattw4 has joined #openstack-infra | 22:06 | |
| *** markvoelker has joined #openstack-infra | 22:07 | |
| openstackgerrit | Ian Wienand proposed openstack/openstack-zuul-jobs master: [dnm] using base-test https://review.opendev.org/692457 | 22:10 |
| *** markvoelker has quit IRC | 22:11 | |
| openstackgerrit | Ian Wienand proposed openstack/openstack-zuul-jobs master: [dnm] using base-test https://review.opendev.org/692457 | 22:13 |
| *** kjackal has quit IRC | 22:20 | |
| fungi | hrm, nope, not going to be looking into zuul executors this evening after all | 22:20 |
| fungi | just realized that i powered down my bastion at home planning to boot it back up in an orderly fashion before leaving the house, but then forgot to actually do so | 22:21 |
| fungi | i can get christine to power it on later, but she's out with friends for hallowe'en at the moment | 22:22 |
| ianw | fungi: i can in trade for a +2 in https://review.opendev.org/#/c/689932/? :) | 22:22 |
| fungi | ianw: i can do code review, yep ;) | 22:22 |
| *** lpetrut has joined #openstack-infra | 22:23 | |
| fungi | doing my part to support curl|bash | 22:26 |
| fungi | i seriously hope folks are reviewing the content of that script before each and every time they run it | 22:27 |
| fungi | since it's put there by possibly random code reviews | 22:27 |
| fungi | seems like it would be really easy to sneak a backdoor in via one of those | 22:28 |
| ianw | it hasn't worked yet :) but i think it might. the bar is probably slightly higher than random code reviews due to it's nature of being part of base jobs | 22:28 |
| fungi | yeah, i wonder what happens if you have add a separate post playbook to fetch the same file... likely gets overwritten by the one there | 22:29 |
| clarkb | basepost runs last | 22:30 |
| clarkb | the playbook order is nesting doll like | 22:30 |
| fungi | seems like it would make more sense to just write a reusable client script which acts on the log manifest though | 22:30 |
| fungi | what's the benefit of storing the script itself in the log tree? | 22:31 |
| ianw | the theory is, espeically with devstack, finding a problem really requires grepping across all the logs | 22:31 |
| ianw | s/really/often/ | 22:31 |
| ianw | so if we have an easy way for people to grab things to a local fs, hopefully it lowers the bar to debugging issues | 22:32 |
| fungi | right, but why an infinite number of copies of a script embedded with the logs rather than a reusable tool which can pull the logs listed in the log manifest? | 22:32 |
| openstackgerrit | Merged opendev/base-jobs master: Use test-upload-logs-swift in base-test https://review.opendev.org/689932 | 22:33 |
| ianw | that is an option, or just wget --recursive ... my theory at least is to find the thing that has the lowest barrier to entry | 22:33 |
| fungi | seems like the latter would at least not further the curl|bash anti-pattern | 22:33 |
| fungi | "it's okay to point this specific gun at your foot because we're pretty sure it's not loaded" | 22:34 |
| ianw | yes, i mean i wouldn't die on this hill, you're right. but also, people run devstack and trust that it's not going to send their private keys to remote servers, although it could | 22:35 |
| fungi | i'm all for the experiment, but i do think encouraging folks to run scripts they download from a site which contains mostly untrusted files sets a bad precedent, and zuul does also publish a full list of the files it uploaded as well as the base url for them, so it would only be a few lines to stitch those together in a locally-installable tool | 22:37 |
| fungi | so with just the link to the build result page you could quite easily generate a full list of exactly what should be downloaded without having to rely on wget's recursive features | 22:38 |
| *** dchen has joined #openstack-infra | 22:39 | |
| ianw | i think the trust level for this particular file is high, given how it gets updated and generated. but i agree it's hard to communicate "you can probably trust this file, but not the other files" | 22:41 |
| *** armax has quit IRC | 22:41 | |
| *** threestrands has joined #openstack-infra | 22:42 | |
| fungi | granted, i had similar concerns with the old reproduce.sh script too | 22:42 |
| fungi | though hopefully folks were only ever running that on throwaway systems | 22:42 |
| *** sreejithp has joined #openstack-infra | 22:43 | |
| ianw | port 7900 right? | 22:43 |
| ianw | for finger daemon | 22:43 |
| fungi | yeah | 22:43 |
| *** slaweq has quit IRC | 22:44 | |
| fungi | netstat -lnt should show it on :: if the process is running | 22:44 |
| fungi | it was missing from 09 and 12 last i checked | 22:44 |
| ianw | same, that's where i see them missing | 22:47 |
| ianw | i can restart | 22:47 |
| *** sreejithp has quit IRC | 22:48 | |
| *** ociuhandu has joined #openstack-infra | 22:48 | |
| fungi | thanks! | 22:53 |
| *** ociuhandu has quit IRC | 22:53 | |
| ianw | hrm, starting the service on ze09 doesn't seem to want to ... start the service | 22:54 |
| *** mattw4 has quit IRC | 22:55 | |
| *** mattw4 has joined #openstack-infra | 22:55 | |
| fungi | after stopping it presumably | 22:56 |
| clarkb | ianw: the stoop command returns before the service stops | 22:56 |
| ianw | it has gone, the pid file doesn't seem to be there | 22:56 |
| fungi | yeah, you have to wait for it to actually wind down | 22:57 |
| fungi | hrm | 22:57 |
| *** tkajinam has joined #openstack-infra | 23:00 | |
| ianw | maybe a config error? http://paste.openstack.org/show/785700/ | 23:03 |
| *** mattw4 has quit IRC | 23:04 | |
| *** mattw4 has joined #openstack-infra | 23:04 | |
| *** igordc has quit IRC | 23:10 | |
| ianw | Exception: connection "gerrit" | 23:12 |
| ianw | somewhere in the gerrit config section? did any of this change? | 23:13 |
| ianw | configparser.InterpolationSyntaxError: '%' must be followed by '%' or '(', found: '%Y-%m-%dT%T%z ' | 23:16 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Plumb through secure.config contents https://review.opendev.org/691800 | 23:19 |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 23:19 | |
| *** mattw4 has quit IRC | 23:21 | |
| mordred | ianw: uh | 23:22 |
| ianw | '%Y-%m-%dT%T%z ' ... the only place this comes from is HISTTIMEFORMAT? ... ???? | 23:22 |
| mordred | ianw: yeah - I don't see anything like that in config files itself | 23:24 |
| ianw | i ran it twice in a row, and got two different errors! http://paste.openstack.org/show/785704/ | 23:24 |
| mordred | oh awesome | 23:25 |
| mordred | ianw: we did merge f2229705f303ad71d37e98323e3f43031e4e0197 reasonably recently | 23:26 |
| ianw | this has to be it, it's getting stuff from the environment somehow | 23:27 |
| ianw | File "/usr/lib/python3.5/configparser.py", line 406, in _interpolate_some | 23:27 |
| ianw | raise Exception(rest) | 23:27 |
| ianw | Exception: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/puppetlabs/bin | 23:27 |
| mordred | yeah - and somehow the os.environ it's passing in is causing the parsing to go sideways | 23:27 |
| ianw | any % in os.environ won't be escaped | 23:27 |
| ianw | ^^ that was me trying to see what configparser thought it was parsing | 23:27 |
| ianw | i think it needs rawconfigparser | 23:28 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Revert "Add the process environment to zuul.conf parser" https://review.opendev.org/692463 | 23:28 |
| mordred | ianw: there's a bare revert - but if rawconfigparser would fix it - I think doing that would also be a good idea | 23:29 |
| mordred | ianw: yeah - definitely think it's from os.environ: 'HISTTIMEFORMAT': '%Y-%m-%dT%T%z ' | 23:29 |
| ianw | ok, just changing to RawConfigParser, the error goes away (live editing on ze09) | 23:30 |
| mordred | ianw: yah - but... RawConfigParser doesn't do interpolation | 23:30 |
| mordred | which removes the point of that patch | 23:31 |
| *** yamamoto has joined #openstack-infra | 23:31 | |
| ianw | hrm, ok ... "service zuul-executor start" on ze09 still doesn't work with RawConfigParser anyway, although now it starts from command line | 23:32 |
| mordred | ianw: what about removing os.environ from the constructor but using normal configparser? | 23:33 |
| ianw | nup, that doesn't either :/ | 23:33 |
| ianw | i've managed to strace systemd trying to start the daemon before, can't quite remember how i did it | 23:34 |
| mordred | blerg. so there might be two issues? | 23:34 |
| ianw | it was this, a long time ago https://review.opendev.org/#/c/547889/ | 23:35 |
| *** yamamoto has quit IRC | 23:36 | |
| sgw | mordred: I have a question about PBR and tagging. I am aware that pbr behaves differently for rpm and deb in order to handle ordering. I am trying to understand how to have a correct tag in the starlingx repos. I saw in nova there is mixed tags (date style and semvar), we have that in starlingx (see the starlingx/fault repo for example). when I run setup.py -q rpm_version on nova I get 20.0.9999.dev198 which seems reasonable | 23:37 |
| sgw | given a 20.0.0.0rc2 tag. But in fault I get 1.9999.9999.rc1.dev5 or 2018.9.9999.rc2.dev157 (depending on which version of pbr is installed! | 23:37 |
| mordred | ianw: so - we need a similar patch for zuul - we need to land that one for nodepool and the similar one for zuul - then we can debug? | 23:37 |
| ianw | mordred: i dunno ... i'd have to context switch back into that | 23:37 |
| mordred | sgw: repo is starlingx/fault ? | 23:37 |
| sgw | yes, I was checking the python-fmclient/fmclient setup | 23:38 |
| mordred | kk. let me poke at them real quick | 23:39 |
| sgw | thanks, | 23:39 |
| *** SpamapS has quit IRC | 23:43 | |
| Shrews | oh nice. the scrollback is now more than just me talking to myself and AJaeger :) | 23:44 |
| *** armax has joined #openstack-infra | 23:47 | |
| *** ociuhandu has joined #openstack-infra | 23:47 | |
| mordred | Shrews: I enjoy you talking to yourself though | 23:48 |
| Shrews | mordred: i thought it was just me | 23:48 |
| *** ociuhandu has quit IRC | 23:52 | |
| ianw | mordred: ok, i give up. i straced systemd pid 1, and it doesn't even seem to try and start zuul-executor | 23:52 |
| ianw | ok, i just moved the "zuul-executor" binary out of /usr/local/bin and "service zuul-executor start" still returns 0! | 23:55 |
| mordred | ianw: maybe we need to do a systemctl stop zuul-executor? | 23:56 |
| ianw | so it can't even be tyrying to start it | 23:56 |
| mordred | ianw: hrm. systemctl status zuul-executor shows an issue with stop | 23:57 |
| Shrews | try /etc/init.d/zuul-executor | 23:58 |
| ianw | mordred: ok, you're onto something | 23:58 |
| ianw | after a stop, i can start it, and got an strace of systemd trying to start it | 23:59 |
| mordred | sgw: so ... I might not be smart enough any more to know the answer to this ... I *think* it has something to do with the 20.0.0 tag on stable/train (in nova) being merged back into master - or maybe something to do with datetimes? in any case ... I think smcginnis or ttx or AJaeger or stephenfin might know the answer here | 23:59 |
| mordred | sgw: the things I looked at and tried seemed fine - but there's some subtle issue that I believe we worked through openstack-side | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!