| *** slaweq has quit IRC | 00:02 | |
| efried_pto | infra-root, please promote https://review.opendev.org/#/c/683009/ directly to gate to make 80% of spurious nova failures go away for the time being so we can merge our feature code before RC1, kthx | 00:08 |
|---|---|---|
| *** mriedem is now known as mriedem_afk | 00:09 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: Fix gerrit errors from production https://review.opendev.org/683006 | 00:10 |
| mriedem_afk | clarkb: fyi we decided to do a skip patch https://review.opendev.org/#/c/683009/ to get past http://status.openstack.org/elastic-recheck/#1823251 until RC1 | 00:12 |
| mriedem_afk | should help the gate | 00:12 |
| clarkb | and no concern that stuff landing in rc1 might break in db migrations? | 00:13 |
| *** jamesmcarthur has quit IRC | 00:14 | |
| *** slaweq has joined #openstack-infra | 00:18 | |
| *** mattw4 has quit IRC | 00:21 | |
| *** gyee has quit IRC | 00:22 | |
| *** slaweq has quit IRC | 00:23 | |
| *** markvoelker has joined #openstack-infra | 00:25 | |
| *** mpjetta has quit IRC | 00:27 | |
| *** markvoelker has quit IRC | 00:29 | |
| *** whoami-rajat has joined #openstack-infra | 00:29 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul master: Support nodes setting 'auto' python-path https://review.opendev.org/682275 | 00:29 |
| clarkb | change has been enueued. Not promoting because there are no passing nova changes ahead of it in the gate and this only affects nova unittests | 00:29 |
| *** ociuhandu has joined #openstack-infra | 00:40 | |
| *** n-saito10 has joined #openstack-infra | 00:41 | |
| *** n-saito10 has left #openstack-infra | 00:41 | |
| *** Goneri has quit IRC | 00:45 | |
| *** ociuhandu has quit IRC | 00:45 | |
| *** dave-mccowan has quit IRC | 00:47 | |
| *** dave-mccowan has joined #openstack-infra | 00:52 | |
| *** jamesmcarthur has joined #openstack-infra | 00:58 | |
| *** markvoelker has joined #openstack-infra | 01:00 | |
| *** jamesmcarthur has quit IRC | 01:10 | |
| *** jamesmcarthur has joined #openstack-infra | 01:10 | |
| *** jamesmcarthur has quit IRC | 01:15 | |
| *** mriedem_afk has quit IRC | 01:21 | |
| *** rkukura has quit IRC | 01:24 | |
| *** yamamoto has joined #openstack-infra | 01:30 | |
| johnsom | clarkb FYI, DNS failures over IPv6: https://zuul.opendev.org/t/openstack/build/d57d301d843b4b84a8408ff2cdee81ef/log/controller/logs/unbound_log.txt.gz | 01:40 |
| *** jamesmcarthur has joined #openstack-infra | 01:49 | |
| *** jamesmcarthur has quit IRC | 01:58 | |
| *** harlowja has joined #openstack-infra | 01:59 | |
| *** rlandy|bbl is now known as rlandy | 02:01 | |
| *** rlandy has quit IRC | 02:01 | |
| *** apetrich has quit IRC | 02:10 | |
| *** ricolin has joined #openstack-infra | 02:22 | |
| *** jamesmcarthur has joined #openstack-infra | 02:28 | |
| *** yamamoto has quit IRC | 02:33 | |
| *** roman_g has quit IRC | 02:35 | |
| *** whoami-rajat has quit IRC | 02:40 | |
| *** markvoelker has quit IRC | 02:47 | |
| *** xinranwang has joined #openstack-infra | 02:59 | |
| *** yamamoto has joined #openstack-infra | 03:05 | |
| *** ykarel|away has joined #openstack-infra | 03:11 | |
| *** weshay has quit IRC | 03:27 | |
| *** psachin has joined #openstack-infra | 03:34 | |
| *** jamesmcarthur has quit IRC | 03:38 | |
| *** jamesmcarthur has joined #openstack-infra | 03:40 | |
| *** jamesmcarthur has quit IRC | 03:44 | |
| *** udesale has joined #openstack-infra | 04:04 | |
| *** jamesmcarthur has joined #openstack-infra | 04:09 | |
| *** rh-jelabarre has quit IRC | 04:14 | |
| *** jamesmcarthur has quit IRC | 04:16 | |
| *** signed8bit has quit IRC | 04:17 | |
| openstackgerrit | Adam Harwell proposed openstack/project-config master: Identify voting bots https://review.opendev.org/683028 | 04:29 |
| *** ykarel|away has quit IRC | 04:30 | |
| *** signed8bit has joined #openstack-infra | 04:30 | |
| *** pcaruana has joined #openstack-infra | 04:42 | |
| *** ramishra has joined #openstack-infra | 04:43 | |
| *** xarses has quit IRC | 04:46 | |
| *** xarses has joined #openstack-infra | 04:47 | |
| openstackgerrit | Merged openstack/infra-manual master: Document "clean check" requirement https://review.opendev.org/682933 | 04:50 |
| *** jamesmcarthur has joined #openstack-infra | 04:51 | |
| *** dchen has quit IRC | 04:54 | |
| *** ykarel|away has joined #openstack-infra | 04:54 | |
| *** jamesmcarthur has quit IRC | 04:56 | |
| *** rcernin_ has joined #openstack-infra | 04:56 | |
| *** dchen has joined #openstack-infra | 04:57 | |
| *** ykarel|away is now known as ykarel | 04:57 | |
| *** rcernin has quit IRC | 04:59 | |
| *** jtomasek has joined #openstack-infra | 05:02 | |
| *** signed8bit has quit IRC | 05:03 | |
| *** aaronsheffield has quit IRC | 05:04 | |
| *** dchen has quit IRC | 05:11 | |
| *** dchen has joined #openstack-infra | 05:13 | |
| *** jamesmcarthur has joined #openstack-infra | 05:17 | |
| *** signed8bit has joined #openstack-infra | 05:18 | |
| *** janki has joined #openstack-infra | 05:21 | |
| *** jamesmcarthur has quit IRC | 05:22 | |
| rm_work | AJaeger: wait what does "sort properly" mean? | 05:37 |
| openstackgerrit | Adam Harwell proposed openstack/project-config master: Identify voting bots https://review.opendev.org/683028 | 05:39 |
| rm_work | trying my only guess but it doesn't make sense to me | 05:39 |
| *** soniya29 has joined #openstack-infra | 05:54 | |
| *** xinranwang has quit IRC | 06:08 | |
| AJaeger | rm_work: https://zuul.opendev.org/t/openstack/build/091d3fa9f77046b28871f4548c90ebb5/log/job-output.txt#479 is the error, you need to update the tool we use for checking | 06:09 |
| rm_work | oh | 06:10 |
| rm_work | ok so the order in the file doesn't matter? | 06:10 |
| rm_work | or this is in addition? | 06:10 |
| AJaeger | in addition ;) | 06:10 |
| rm_work | lol k | 06:10 |
| openstackgerrit | Adam Harwell proposed openstack/project-config master: Identify voting bots https://review.opendev.org/683028 | 06:11 |
| rm_work | what do you think? :D | 06:11 |
| rm_work | the bots are getting frustrating for us, now that they cause so much noise in a field we intended to use for filtering <_< | 06:12 |
| rm_work | before they were just funny | 06:12 |
| *** kopecmartin|off is now known as kopecmartin | 06:18 | |
| *** prometheanfire has quit IRC | 06:23 | |
| *** prometheanfire has joined #openstack-infra | 06:24 | |
| *** markvoelker has joined #openstack-infra | 06:27 | |
| AJaeger | rm_work: I saw these bots mis-vote and was laughing when I saw it. | 06:27 |
| AJaeger | rm_work: but I also saw real contributors (cores in other projects) get confused and click on the wrong line. | 06:28 |
| *** ralonsoh has joined #openstack-infra | 06:28 | |
| rm_work | yes i've seen that too, but normally quickly fixed | 06:28 |
| rm_work | i've even done it once | 06:28 |
| rm_work | but real people realize quickly | 06:28 |
| AJaeger | rm_work: let's give the TC a heads-up since this points out a community problem. | 06:28 |
| rm_work | yeah, seems like a fair idea | 06:28 |
| AJaeger | evrardjp, mnaser, ttx, heads up on https://review.opendev.org/683028 , in case you want to share with the rest of the TC or have any opinion on it. | 06:29 |
| rm_work | jroll still TC too? he should be up soon, lol | 06:30 |
| AJaeger | yes, he is. | 06:30 |
| AJaeger | Let's see what the rest of the team thinks as well... | 06:31 |
| AJaeger | so, I'll wait a few days before I vote. | 06:31 |
| *** markvoelker has quit IRC | 06:31 | |
| rm_work | kk | 06:40 |
| *** jaosorior has joined #openstack-infra | 06:43 | |
| *** ociuhandu has joined #openstack-infra | 06:46 | |
| *** pgaxatte has joined #openstack-infra | 06:48 | |
| *** rcernin_ has quit IRC | 06:48 | |
| *** takamatsu has quit IRC | 06:50 | |
| *** ociuhandu has quit IRC | 06:51 | |
| *** psachin has quit IRC | 06:53 | |
| *** gfidente has joined #openstack-infra | 06:54 | |
| *** udesale has quit IRC | 06:57 | |
| *** udesale has joined #openstack-infra | 06:58 | |
| *** rpittau|afk is now known as rpittau | 07:05 | |
| *** xek has joined #openstack-infra | 07:05 | |
| *** slaweq has joined #openstack-infra | 07:08 | |
| *** armax has joined #openstack-infra | 07:11 | |
| *** apetrich has joined #openstack-infra | 07:15 | |
| *** rcosnita has joined #openstack-infra | 07:16 | |
| *** rcosnita has quit IRC | 07:16 | |
| *** e0ne has joined #openstack-infra | 07:17 | |
| *** kjackal has joined #openstack-infra | 07:18 | |
| *** udesale has quit IRC | 07:18 | |
| *** jamesmcarthur has joined #openstack-infra | 07:18 | |
| *** udesale has joined #openstack-infra | 07:19 | |
| *** slaweq has quit IRC | 07:20 | |
| *** xenos76 has joined #openstack-infra | 07:22 | |
| *** jamesmcarthur has quit IRC | 07:23 | |
| *** tosky has joined #openstack-infra | 07:23 | |
| *** tesseract has joined #openstack-infra | 07:26 | |
| *** jpena|off is now known as jpena | 07:33 | |
| *** slaweq has joined #openstack-infra | 07:33 | |
| *** armax has quit IRC | 07:37 | |
| *** xek has quit IRC | 07:38 | |
| *** ykarel is now known as ykarel|lunch | 07:38 | |
| *** xek has joined #openstack-infra | 07:40 | |
| *** roman_g has joined #openstack-infra | 07:42 | |
| *** takamatsu has joined #openstack-infra | 07:44 | |
| *** jaosorior has quit IRC | 07:51 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Fix upstream_translation_update https://review.opendev.org/683046 | 07:51 |
| AJaeger | config-core, please review to fix the upstream_translation jobs ^ | 07:52 |
| *** kaiokmo has quit IRC | 07:57 | |
| *** zbr is now known as zbr|ruck | 07:57 | |
| *** iurygregory has joined #openstack-infra | 08:00 | |
| *** kaiokmo has joined #openstack-infra | 08:02 | |
| *** soniya29 has quit IRC | 08:04 | |
| *** ildikov has joined #openstack-infra | 08:10 | |
| ildikov | Hi infra team | 08:11 |
| ildikov | we have a merged review in StarlingX that didn't get rendered to the docs page: https://review.opendev.org/#/c/681850 | 08:12 |
| ildikov | can someone help me looking into where it got stuck in the post-processing phase? | 08:12 |
| *** yamamoto has quit IRC | 08:13 | |
| AJaeger | ildikov: I'll check in a bit... | 08:14 |
| *** dchen has quit IRC | 08:18 | |
| ttx | AJaeger: I commented | 08:19 |
| *** elod has quit IRC | 08:19 | |
| *** tkajinam has quit IRC | 08:19 | |
| ildikov | AJaeger: thank you | 08:19 |
| ttx | will bring it up to the TC too | 08:20 |
| *** arxcruz|ruck is now known as arxcruz|rover | 08:20 | |
| *** ykarel|lunch is now known as ykarel | 08:21 | |
| *** lennyb has quit IRC | 08:21 | |
| *** elod has joined #openstack-infra | 08:24 | |
| *** soniya29 has joined #openstack-infra | 08:26 | |
| frickler | AJaeger: ildikov: seems the new docs get published to /afs/openstack.org/project/starlingx.io/www/docs/ , but content is served from /afs/openstack.org/project/starlingx.io/www | 08:27 |
| *** markvoelker has joined #openstack-infra | 08:28 | |
| frickler | here's the new content https://docs.starlingx.io/docs/developer_resources/navigate_source_code.html | 08:28 |
| *** derekh has joined #openstack-infra | 08:30 | |
| *** zzzeek has quit IRC | 08:30 | |
| ildikov | frickler: hmm, there might be something with the build job? | 08:30 |
| ildikov | as the review touches an already existing page only | 08:31 |
| *** zzzeek has joined #openstack-infra | 08:31 | |
| ildikov | I'll check a little later | 08:31 |
| *** dtantsur|afk is now known as dtantsur | 08:32 | |
| *** markvoelker has quit IRC | 08:33 | |
| *** lpetrut has joined #openstack-infra | 08:33 | |
| *** pgaxatte has quit IRC | 08:34 | |
| *** lennyb has joined #openstack-infra | 08:34 | |
| frickler | ildikov: yes, that's an issue either with the build or promote job, but I'll defer that to AJaeger who know the details of those better than me | 08:35 |
| *** kaiokmo has quit IRC | 08:36 | |
| *** kaiokmo has joined #openstack-infra | 08:37 | |
| AJaeger | frickler: Oh, thanks for debugging that - now I know what to fix | 08:37 |
| *** lpetrut has quit IRC | 08:39 | |
| *** pgaxatte has joined #openstack-infra | 08:43 | |
| *** ociuhandu has joined #openstack-infra | 08:45 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add promote job for starlingx/docs https://review.opendev.org/683059 | 08:45 |
| *** ociuhandu has quit IRC | 08:46 | |
| *** takamatsu_ has joined #openstack-infra | 08:46 | |
| *** takamatsu has quit IRC | 08:46 | |
| AJaeger | ildikov, frickler, part 1 of the fix ^ | 08:46 |
| *** xenos76 has quit IRC | 08:46 | |
| AJaeger | part 2 is https://review.opendev.org/683061 | 08:47 |
| *** ociuhandu has joined #openstack-infra | 08:47 | |
| AJaeger | frickler, could you +2 683046 , please? | 08:48 |
| *** ociuhandu has quit IRC | 08:53 | |
| *** ociuhandu has joined #openstack-infra | 08:53 | |
| *** ociuhandu has quit IRC | 08:54 | |
| *** ociuhandu has joined #openstack-infra | 08:54 | |
| *** lpetrut has joined #openstack-infra | 09:05 | |
| *** yamamoto has joined #openstack-infra | 09:06 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Fix upstream_translation_update https://review.opendev.org/683046 | 09:06 |
| *** gfidente has quit IRC | 09:07 | |
| *** xenos76 has joined #openstack-infra | 09:08 | |
| *** gfidente has joined #openstack-infra | 09:10 | |
| *** iurygregory has quit IRC | 09:13 | |
| *** iurygregory has joined #openstack-infra | 09:14 | |
| *** takamatsu_ has quit IRC | 09:19 | |
| *** slaweq has quit IRC | 09:25 | |
| *** slaweq has joined #openstack-infra | 09:27 | |
| *** yamamoto has quit IRC | 09:32 | |
| *** takamatsu has joined #openstack-infra | 09:36 | |
| *** yamamoto has joined #openstack-infra | 09:38 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul master: Add a more conversational overview to README.rst https://review.opendev.org/683085 | 09:45 |
| *** ociuhandu has quit IRC | 09:46 | |
| *** ociuhandu has joined #openstack-infra | 09:47 | |
| *** ociuhandu has quit IRC | 09:52 | |
| *** janki has quit IRC | 09:57 | |
| *** takamatsu has quit IRC | 10:00 | |
| *** udesale has quit IRC | 10:01 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Add reference pipelines file for Github driver https://review.opendev.org/672712 | 10:04 |
| *** takamatsu has joined #openstack-infra | 10:06 | |
| *** yamamoto has quit IRC | 10:10 | |
| *** yamamoto has joined #openstack-infra | 10:12 | |
| *** ociuhandu has joined #openstack-infra | 10:15 | |
| *** rcernin_ has joined #openstack-infra | 10:15 | |
| *** ykarel is now known as ykarel|afk | 10:27 | |
| *** markvoelker has joined #openstack-infra | 10:29 | |
| *** kjackal has quit IRC | 10:31 | |
| *** udesale has joined #openstack-infra | 10:32 | |
| *** markvoelker has quit IRC | 10:34 | |
| *** trident has quit IRC | 10:35 | |
| *** rcernin_ has quit IRC | 10:36 | |
| *** udesale has quit IRC | 10:37 | |
| *** udesale has joined #openstack-infra | 10:37 | |
| ildikov | AJaeger: thank you! | 10:41 |
| ildikov | AJaeger: I assume the changes were needed as it's a different path than the original templates? | 10:42 |
| *** e0ne has quit IRC | 10:42 | |
| ildikov | I just wonder how it worked until now as they have docs published | 10:43 |
| ildikov | sorry for disappearing so long, I'm at a conference and had some duties to take care of | 10:43 |
| AJaeger | ildikov: correct, those are different paths | 10:44 |
| AJaeger | ildikov: I recently changed the publishing job to promote jobs and missed that the docs repo publishes to "/" instead of "/docs" :( | 10:44 |
| *** trident has joined #openstack-infra | 10:48 | |
| ildikov | AJaeger: ah ok, no worries | 10:51 |
| ildikov | AJaeger: I just wanted to understand if we missed something or keep a closer eye or items, etc | 10:52 |
| ildikov | AJaeger: thanks for the quick fix | 10:52 |
| *** pcaruana has quit IRC | 10:54 | |
| *** rpittau is now known as rpittau|afk | 10:56 | |
| *** takamatsu has quit IRC | 11:00 | |
| *** udesale has quit IRC | 11:01 | |
| *** jpena is now known as jpena|lunch | 11:04 | |
| dtantsur | folks, I remember asking, but I apparently forgot the answer: is it a known problem? http://paste.openstack.org/show/777692/ | 11:06 |
| dtantsur | it seems to affect the ironic CI | 11:06 |
| *** takamatsu has joined #openstack-infra | 11:06 | |
| frickler | dtantsur: to me this looks like neutron may be messing up IPv6 connectivity to the test node, causing zuul to lose the connection. does this happen 100% or only sometimes, possibly related to certain cloud providers? | 11:11 |
| dtantsur | frickler: only sometimes | 11:11 |
| dtantsur | this one was https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_650/682647/3/check/ironic-standalone-ipa-src/65091a8/job-output.txt | 11:12 |
| *** yamamoto has quit IRC | 11:13 | |
| *** pcaruana has joined #openstack-infra | 11:17 | |
| *** yamamoto has joined #openstack-infra | 11:17 | |
| frickler | dtantsur: so this is fortnebula, one of the providers where we access nodes via v6, so that would match my hypothesis. you may want to crosscheck this with other failures that you see. it will be difficult to debug this, though. best probably would be to try to reproduce with a local instance where you can verify v6 connectivity during the run | 11:17 |
| *** e0ne has joined #openstack-infra | 11:21 | |
| *** takamatsu has quit IRC | 11:22 | |
| *** n-saito has joined #openstack-infra | 11:24 | |
| *** n-saito has left #openstack-infra | 11:25 | |
| *** gfidente has quit IRC | 11:28 | |
| *** takamatsu has joined #openstack-infra | 11:28 | |
| *** lucasagomes has joined #openstack-infra | 11:33 | |
| jroll | rm_work: fun :) | 11:33 |
| *** yamamoto has quit IRC | 11:42 | |
| *** gfidente has joined #openstack-infra | 11:43 | |
| *** yamamoto has joined #openstack-infra | 11:46 | |
| *** rh-jelabarre has joined #openstack-infra | 11:54 | |
| *** yamamoto has quit IRC | 11:57 | |
| *** yamamoto has joined #openstack-infra | 12:00 | |
| *** ykarel|afk is now known as ykarel | 12:02 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add promote job for starlingx/docs https://review.opendev.org/683059 | 12:04 |
| *** rfolco has joined #openstack-infra | 12:04 | |
| AJaeger | frickler: thanks for review ^ | 12:05 |
| *** ricolin has quit IRC | 12:06 | |
| *** Goneri has joined #openstack-infra | 12:07 | |
| *** goldyfruit_ has quit IRC | 12:11 | |
| *** markvoelker has joined #openstack-infra | 12:13 | |
| *** jamesmcarthur has joined #openstack-infra | 12:18 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove bazel version hack https://review.opendev.org/682469 | 12:18 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Install jeepyb into the gerrit images https://review.opendev.org/682601 | 12:18 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Expose volumes for the gerrit hooks and static dirs https://review.opendev.org/678875 | 12:18 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run a gerrit container (test only) https://review.opendev.org/630406 | 12:18 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Fix gerrit java invocation https://review.opendev.org/683124 | 12:18 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add admin users to docker group https://review.opendev.org/683125 | 12:19 |
| *** takamatsu has quit IRC | 12:21 | |
| *** jpena|lunch is now known as jpena | 12:23 | |
| *** aaronsheffield has joined #openstack-infra | 12:25 | |
| *** takamatsu has joined #openstack-infra | 12:27 | |
| *** ociuhandu has quit IRC | 12:27 | |
| *** lbragstad has quit IRC | 12:29 | |
| *** lbragstad has joined #openstack-infra | 12:30 | |
| *** jamesmcarthur has quit IRC | 12:30 | |
| *** jamesmcarthur has joined #openstack-infra | 12:30 | |
| *** eandersson has quit IRC | 12:31 | |
| mordred | infra-root: I'm manually doing things on review-dev | 12:31 |
| *** rlandy has joined #openstack-infra | 12:31 | |
| *** yamamoto has quit IRC | 12:33 | |
| mordred | (I have put it into the emergency file - but still, thought I'd let you know) | 12:34 |
| fungi | thanks for the heads-up | 12:34 |
| mordred | also - review-dev is currently successfully running gerrit 2.13 from our docker container of 2.13 | 12:35 |
| *** jamesmcarthur has quit IRC | 12:36 | |
| *** ociuhandu has joined #openstack-infra | 12:36 | |
| *** priteau has joined #openstack-infra | 12:36 | |
| *** takamatsu has quit IRC | 12:38 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run a gerrit container (test only) https://review.opendev.org/630406 | 12:38 |
| fungi | clarkb: thanks for the reminder on the reprepro problem, i had gotten as far as wondering if it was writing the new package indices to a different/wrong path since it's not for weeks so much as touched the ones our nodes hit even though it's been updating the packages in the pool | 12:38 |
| *** ociuhandu has quit IRC | 12:40 | |
| *** ociuhandu has joined #openstack-infra | 12:41 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add admin users to docker group https://review.opendev.org/683125 | 12:41 |
| mordred | fungi: uhm. https://review.opendev.org/#/c/682601/4 <-- zuul seems real unhappy - am I missing something there? | 12:43 |
| mordred | or did I just get unlucky? | 12:43 |
| dtantsur | frickler: is it possible to stop running ironic jobs on fortnebula? it seems to have become a real problem. | 12:44 |
| dtantsur | (I wonder why neutron doesn't suffer from it) | 12:45 |
| dtantsur | although.. we see so many other random failures that it may not even help | 12:46 |
| *** markvoelker has quit IRC | 12:46 | |
| *** fdegir has quit IRC | 12:47 | |
| *** fdegir has joined #openstack-infra | 12:48 | |
| *** lpetrut has quit IRC | 12:48 | |
| fungi | mordred: i haven't looked closely yet, but maybe we need to add those projects to the openhdev tenant? | 12:50 |
| fungi | a lot of the opendev namespace projects are still i the openstack tenant in zuul | 12:50 |
| mordred | fungi: I *think* I just got unlucky and zuul was in the middle of a reconfig or something - I rechecked and they seem to be better | 12:51 |
| mordred | fungi: (the patches worked in previous iterations and are in the openstack tenant) | 12:51 |
| fungi | ahh, okay | 12:51 |
| mordred | oh! nope, nevermind. I lied - I rechecked and still unhappy | 12:51 |
| mordred | fungi: is opendev/system-config in the opendev tenant now? | 12:52 |
| mordred | OH - I get what's happening | 12:53 |
| mordred | system-config is in both, and it's importing jobs into the opendev tenant - but jeepyb isn't in the opendev tenant, so the job import is unhappy - while in the openstack tenant everything is there so things are happy | 12:54 |
| *** ociuhandu has quit IRC | 12:55 | |
| *** ociuhandu has joined #openstack-infra | 12:55 | |
| *** zzzeek has quit IRC | 12:56 | |
| sshnaidm | is there a way to see gzipped site like that? https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_de8/682527/2/check/tripleo-ci-centos-7-containers-multinode/de865c2/logs/ara_oooq_oc/ | 12:57 |
| sshnaidm | I remember it worked once | 12:58 |
| *** zzzeek has joined #openstack-infra | 12:58 | |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Add repos to opendev tenant for system-config jobs https://review.opendev.org/683146 | 12:59 |
| *** ociuhandu has quit IRC | 13:00 | |
| *** ociuhandu has joined #openstack-infra | 13:01 | |
| donnyd | dtantsur: What iss are you having with FN | 13:02 |
| mordred | fungi, AJaeger: ^^ that's related to https://review.opendev.org/#/c/682755/ which recently landed which added system-config to the opendev tenant but didn't quite pull in enough things for required-projects to be happy | 13:03 |
| dtantsur | donnyd: it's just a guess, but we're seeing http://paste.openstack.org/show/777692/ | 13:04 |
| *** whoami-rajat has joined #openstack-infra | 13:04 | |
| *** slaweq has quit IRC | 13:07 | |
| donnyd | So looking through the logs using node_provider:"fortnebula-regionone" AND filename:job-output.txt AND message:"Upload logs to swift" AND build_status:"FAILURE" | 13:08 |
| donnyd | http://logstash.openstack.org/#/dashboard/file/logstash.json | 13:08 |
| donnyd | And then looking at node_provider:"fortnebula-regionone" AND filename:job-output.txt AND message:"Upload logs to swift" AND build_status:"SUCCESS" | 13:11 |
| donnyd | You can see there are some jobs that pass and some that don't | 13:11 |
| donnyd | Could be because of ipv6 | 13:11 |
| donnyd | but I am happy to take a deeper look into your issue | 13:11 |
| *** ociuhandu has quit IRC | 13:12 | |
| donnyd | could it be that you are setting def route's in the job for ipv6? | 13:12 |
| *** eharney has joined #openstack-infra | 13:12 | |
| donnyd | It would seem that when you create the public side of your networking, its overridding the real ipv6 routes. In other providers you don't have to worry about it because they come inbound on v4. FN is the opposite | 13:13 |
| *** dtantsur is now known as dtantsur|bbl | 13:14 | |
| donnyd | But I see the same job run fine other times | 13:15 |
| donnyd | 2019-09-19 12:31:41.094258 | controller | +++ lib/neutron_plugins/services/l3:_neutron_create_public_subnet_v6:330 : oscwrap --os-cloud devstack-admin --os-region RegionOne subnet create --ip-version 6 --gateway 2001:db8::2 --network 13b55f13-0625-434f-8c7e-20dd23e08d87 --subnet-range 2001:db8::/64 --no-dhcp ipv6-public-subnet | 13:15 |
| donnyd | https://4c15fff3fe75ce2ac3df-05cf8d45f98e4972de234be1a80e25da.ssl.cf1.rackcdn.com/674496/5/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/50c2591/job-output.txt | 13:16 |
| donnyd | dtantsur|bbl: ^^^^ | 13:16 |
| *** takamatsu has joined #openstack-infra | 13:16 | |
| *** yamamoto has joined #openstack-infra | 13:16 | |
| *** goldyfruit_ has joined #openstack-infra | 13:16 | |
| donnyd | also could you share what job that paste run in? | 13:16 |
| *** armax has joined #openstack-infra | 13:17 | |
| *** yamamoto_ has joined #openstack-infra | 13:17 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Install jeepyb into the gerrit images https://review.opendev.org/682601 | 13:17 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Expose volumes for the gerrit hooks and static dirs https://review.opendev.org/678875 | 13:17 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run a gerrit container (test only) https://review.opendev.org/630406 | 13:17 |
| mordred | fungi: added the depends-on | 13:17 |
| *** jaosorior has joined #openstack-infra | 13:18 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove unused stx docs publish job https://review.opendev.org/683154 | 13:21 |
| *** yamamoto has quit IRC | 13:21 | |
| *** udesale has joined #openstack-infra | 13:23 | |
| *** udesale has quit IRC | 13:24 | |
| AJaeger | mordred: could you +2A https://review.opendev.org/683059 , please? | 13:24 |
| *** yamamoto_ has quit IRC | 13:24 | |
| *** udesale has joined #openstack-infra | 13:25 | |
| pabelanger | so, in fedora images, do we do anything to /tmp to make it larger then 512MB? | 13:27 |
| *** pcaruana has quit IRC | 13:28 | |
| mordred | pabelanger: jeez, why would it be 512M? | 13:29 |
| *** derekh has quit IRC | 13:29 | |
| pabelanger | unknown right now | 13:29 |
| pabelanger | this is zuul.a.c, wanted to check with opendev to see | 13:29 |
| pabelanger | but pip install is like taking up almost 500mb in some cases | 13:29 |
| pabelanger | which, seems to use /tmp for storage | 13:29 |
| *** mriedem has joined #openstack-infra | 13:30 | |
| pabelanger | mordred: Oh, default is 1/2 the size of ram, which these instances are 1GB | 13:32 |
| openstackgerrit | Merged openstack/project-config master: Add repos to opendev tenant for system-config jobs https://review.opendev.org/683146 | 13:32 |
| *** jamesmcarthur has joined #openstack-infra | 13:36 | |
| *** jamesmcarthur has quit IRC | 13:36 | |
| frickler | donnyd: what ipv6 addr mode do you use for FN? RA or dhcpv6? | 13:36 |
| *** jamesmcarthur has joined #openstack-infra | 13:36 | |
| openstackgerrit | Merged openstack/project-config master: Add promote job for starlingx/docs https://review.opendev.org/683059 | 13:38 |
| *** ociuhandu has joined #openstack-infra | 13:38 | |
| frickler | I think the critical code is this one, so it seems it should affect neutron jobs, too. will take the discussion to #-neutron | 13:38 |
| frickler | https://opendev.org/openstack/devstack/src/branch/master/lib/neutron_plugins/services/l3#L398-L411 | 13:38 |
| donnyd | frickler: I use DHCPv6-stateless -Address configuration using RA and optional information using DHCPv6. | 13:39 |
| donnyd | Same as how limestone is setup | 13:39 |
| *** ociuhandu has quit IRC | 13:40 | |
| *** ociuhandu has joined #openstack-infra | 13:41 | |
| frickler | donnyd: o.k., I'm going to repeat my test with that. it certainly breaks with dhcpv6-stateful, but likely with any configuration | 13:42 |
| mordred | pabelanger: ah - the old "mount a tmpfs on /tmp" trick | 13:43 |
| *** yamamoto has joined #openstack-infra | 13:43 | |
| donnyd | frickler: I can also increase the timeouts on my edge FW to allow them to stay open a little longer | 13:43 |
| donnyd | If the issue is only FN and in IPv6... then it could possibly be that the state is being dropped | 13:44 |
| *** gfidente has quit IRC | 13:45 | |
| pabelanger | mordred: systemctl mask tmp.mount | 13:45 |
| pabelanger | should be myfix | 13:45 |
| pabelanger | but yah | 13:45 |
| donnyd | We have been making adjustments to the edge FW because we were previsouly using it for NAT to the mirror because of a retransmit issue. (dropped packets). This was isolated down to an MTU issue which has since been solved | 13:46 |
| frickler | donnyd: one thing that may help would be sending RAs more often. is that configurable on your edge router? or is it an openstack router? | 13:47 |
| *** ociuhandu has quit IRC | 13:47 | |
| donnyd | So I can make thing a little more conservative on the edge (where ssh from zuul comes inbound), because we don't have to worry about expiring connections fast enough to keep up with the NAT churn anymore | 13:47 |
| donnyd | Well the RA's are all done on Openstack | 13:47 |
| donnyd | Not sure if that value is configurable.. but if it is I am happy to change it | 13:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Expose volumes for the gerrit hooks, tmp static dirs https://review.opendev.org/678875 | 13:53 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run a gerrit container (test only) https://review.opendev.org/630406 | 13:53 |
| *** armax has quit IRC | 13:54 | |
| *** pcaruana has joined #openstack-infra | 13:54 | |
| *** zhangfei has joined #openstack-infra | 13:56 | |
| donnyd | We moved the outbound NAT for v4 connections to an Openstack router a little while back | 13:56 |
| *** efried_pto is now known as efried | 13:56 | |
| fungi | mordred: whack-a-mole time... 682601 still needs more, but at least the zuul errors are different now | 13:58 |
| *** derekh has joined #openstack-infra | 13:58 | |
| mordred | fungi: yay! | 13:58 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Expose volumes for the gerrit hooks, static and tmp dirs https://review.opendev.org/678875 | 13:58 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run a gerrit container (test only) https://review.opendev.org/630406 | 13:58 |
| *** kjackal has joined #openstack-infra | 14:00 | |
| *** tesseract has quit IRC | 14:00 | |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Add additional repos need for system-config https://review.opendev.org/683167 | 14:01 |
| mordred | fungi, AJaeger: whackamole! | 14:01 |
| AJaeger | ;( | 14:03 |
| *** soniya29 has quit IRC | 14:04 | |
| *** ociuhandu has joined #openstack-infra | 14:06 | |
| *** ociuhandu has quit IRC | 14:06 | |
| *** ociuhandu has joined #openstack-infra | 14:07 | |
| *** ociuhandu has quit IRC | 14:12 | |
| *** ykarel is now known as ykarel|afk | 14:13 | |
| *** goldyfruit___ has joined #openstack-infra | 14:15 | |
| *** takamatsu has quit IRC | 14:16 | |
| *** goldyfruit_ has quit IRC | 14:18 | |
| TheJulia | mordred: I _think_ two of our 3rd party CIs are v3 based. One I believe is presently offline due to silly human issues | 14:18 |
| *** ykarel|afk is now known as ykarel|away | 14:22 | |
| *** gfidente has joined #openstack-infra | 14:23 | |
| mordred | TheJulia: silly humans | 14:24 |
| *** xenos76 has quit IRC | 14:25 | |
| *** e0ne has quit IRC | 14:28 | |
| fungi | TheJulia: other third-party ci operators might appreciate input/guidance from your ci operators on what was involved in switching from zuul v2 to v3 (assuming they weren't green-field) | 14:30 |
| TheJulia | I know one was greenfielded, the other I'm unsure off-hand | 14:31 |
| TheJulia | definitely worth raising though | 14:31 |
| fungi | my only experience in moving from v2 to v3 is opendev, and involved an experimental transitional v2.5 release along with a very lengthy period of eliminating non-essential jenkins plugins, and also doing it while v3 was still somewhat under development, so probably not as applicable to the third-party ci crowd | 14:32 |
| openstackgerrit | Merged openstack/project-config master: Add additional repos need for system-config https://review.opendev.org/683167 | 14:32 |
| *** xenos76 has joined #openstack-infra | 14:33 | |
| *** pcaruana has quit IRC | 14:33 | |
| *** ociuhandu has joined #openstack-infra | 14:34 | |
| *** michael-beaver has joined #openstack-infra | 14:35 | |
| *** mmedvede has quit IRC | 14:36 | |
| *** ociuhandu has quit IRC | 14:36 | |
| openstackgerrit | Jeremy Freudberg proposed opendev/irc-meetings master: Reduce frequency of Sahara team meeting https://review.opendev.org/683171 | 14:36 |
| *** ociuhandu has joined #openstack-infra | 14:37 | |
| mordred | fungi: also - we used the semi-functional migration script I assaulted the world with - which we do not recommend anyone else do | 14:38 |
| fungi | yeah, mechanical translation of jeepyb configuration into zuul v3 "legacy" jobs | 14:38 |
| *** ociuhandu has quit IRC | 14:39 | |
| *** ociuhandu has joined #openstack-infra | 14:39 | |
| fungi | the path we took was influenced both by being tightly coupled to zuul's development process, and also having to deal with migrating something like 10k jobs | 14:40 |
| pabelanger | FWIW: zuul-migrate works well on single use jobs. Multi-node, you need to rewrite from scratch | 14:40 |
| pabelanger | s/use/node | 14:41 |
| fungi | but also, for a small deployment with few users and a handful of jobs, setting up a new zuul deployment and rewriting those jobs is likely easier | 14:41 |
| *** ykarel|away has quit IRC | 14:41 | |
| fungi | especially if they're based off jobs maintained in the openstack job set which have already mostly been rewritten | 14:41 |
| *** ociuhandu has quit IRC | 14:45 | |
| *** iurygregory has quit IRC | 14:46 | |
| *** iurygregory has joined #openstack-infra | 14:47 | |
| *** sthussey has joined #openstack-infra | 14:49 | |
| *** dtantsur|bbl is now known as dtantsur | 14:54 | |
| dtantsur | donnyd: one of examples https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_650/682647/3/check/ironic-standalone-ipa-src/65091a8/job-output.txt | 14:55 |
| dtantsur | thanks for looking into it | 14:55 |
| donnyd | I am thinking frickler figured it out https://review.opendev.org/#/c/683166/ | 14:56 |
| *** gyee has joined #openstack-infra | 14:56 | |
| *** tosky has quit IRC | 14:57 | |
| *** efried is now known as efried_pto | 14:57 | |
| *** tosky has joined #openstack-infra | 14:58 | |
| *** yamamoto has quit IRC | 14:59 | |
| clarkb | johnsom: ^ fyi that likely explains dns problems too | 14:59 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: WIP: Do not overwrite image upload zk data on delete https://review.opendev.org/681857 | 14:59 |
| fungi | corvus: oh, hey! one of the "openstOck users list" spams just showed up in the foundation ml moderation queue, so i can no longer claim not to have seen one ;) | 15:00 |
| johnsom | clarkb Cool! | 15:00 |
| *** ociuhandu has joined #openstack-infra | 15:02 | |
| *** e0ne has joined #openstack-infra | 15:02 | |
| *** ociuhandu has quit IRC | 15:02 | |
| clarkb | frickler: thank you for tracking that down. I had been pushing towards that for a while now because the network was alays going down right around when neutron was starting up according to the job logs that johnsom had but no on was able to figure it out | 15:02 |
| *** ociuhandu has joined #openstack-infra | 15:03 | |
| *** slaweq has joined #openstack-infra | 15:03 | |
| clarkb | slaweq: ^ fyi https://review.opendev.org/#/c/683166/ appears to maybe be the fix for networking in tempest jobs on ipv6 only clouds. Thi was something I had asked you to debug previously with octavia having dns issues right around when neutron was started | 15:04 |
| *** yamamoto has joined #openstack-infra | 15:05 | |
| frickler | clarkb: the trick seems to have been that the "bad" sysctl didn't even make it into the log, because it is probably buffered, so the logs ends a handful of lines early | 15:05 |
| mordred | clarkb, corvus: is this: https://zuul.opendev.org/t/openstack/build/b1fdbfd2e79c499eb9b7ca66a5cf0fb0 a symptom of the zero-byte in swift registry issue | 15:05 |
| clarkb | mordred: yes that sha256sum is for an empty file | 15:06 |
| clarkb | (thats the trick for identifying the issue, the sha256sum is the same for a no byte file) | 15:07 |
| corvus | mordred: yes; it's easy to confirm because it says it received the sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 which is the sha256sum of the empty string (you can check that with "echo -n ''|sha256sum") | 15:07 |
| mordred | ah - cool | 15:07 |
| mordred | so - when we have confirmed that - is there a process to remediate? | 15:07 |
| corvus | mordred: does this change depend on another with an image build, or is it the head of the queue? | 15:08 |
| mordred | it depends on another with an image build | 15:08 |
| mordred | shoudl we just land that image? | 15:08 |
| corvus | mordred: yes, this change will no longer work until the change ahead is out of the picture in some way; so we need to land the change ahead, or push up a new patchset for it or something. | 15:09 |
| mordred | kk. well - https://review.opendev.org/#/c/683124/1 is the change ahead of it - and it's good to go | 15:10 |
| *** yamamoto has quit IRC | 15:10 | |
| corvus | mordred: why run daemon? | 15:10 |
| mordred | that's the subcommand that gerrit.sh would be running, I think? is there a better subcommand? | 15:10 |
| clarkb | mordred: does gerrit have a run in foreground option? typically that works better with docker due to grabbing stdout and stderror | 15:11 |
| mordred | looking | 15:11 |
| fungi | i suspect this gets a bit wacky due to the onion layers of processes in jvm in container | 15:13 |
| mordred | clarkb: no - daemon is the command | 15:13 |
| mordred | http://paste.openstack.org/show/777703/ | 15:14 |
| fungi | the jar/war is already sort of a container image, and the jvm is already a sort of container | 15:14 |
| clarkb | mordred: I agree daemon appears to be the thing. corvus ^ | 15:14 |
| mordred | clarkb: fwiw - running the java command with daemon works happily | 15:14 |
| clarkb | fungi: the jvm should look like nay other process to docker though. I don't expect it will cause problems | 15:15 |
| corvus | clarkb, mordred: yes, even if you give gerrit.sh an arg of "run" it calls | 15:15 |
| corvus | RUN_ARGS="-jar $GERRIT_WAR daemon -d $GERRIT_SITE" | 15:15 |
| mordred | yah | 15:15 |
| clarkb | I didn't approve in order to give corvus a chance to review | 15:16 |
| corvus | +3 | 15:16 |
| mordred | cool | 15:16 |
| mordred | don't know if y | 15:16 |
| corvus | i suspect that this will still run it in the foreground | 15:16 |
| *** derekh has quit IRC | 15:16 | |
| mordred | y'all caught in scrollback - but review-dev is now running from the container image (albeit with an overridden run line) | 15:16 |
| corvus | gerrit.sh sticks a "&" after it | 15:16 |
| clarkb | corvus: aha | 15:16 |
| corvus | mordred: \o/ | 15:17 |
| mordred | yeah - I thnik "daemon" here is not "daemonize" but "start the daemon that listens on the network" | 15:17 |
| corvus | words are hard | 15:17 |
| fungi | clarkb: yeah, just noting that i'm not sure if it's possible to expose the stddin/stdout/stderr of the actual gerrit process(es) through the jvm layer, which means maybe it's less important whether the jvm's stdin/stdout/stderr are disassociated and the wrapper backgrounded since it's not going to be following a real "containery" paradigm anyway? | 15:18 |
| mordred | corvus: I learned a few things in the process - like we can't install docker-compose from apt packages because we have installed jeepyb with pip | 15:18 |
| *** derekh has joined #openstack-infra | 15:18 | |
| mordred | I think we can get to a place where we do not need to install jeepyb globally on the gerrit servers - but that'll take a minute to unwind | 15:18 |
| clarkb | fungi: it should expose the std* of the jvm process (which is the only linux process gerrit is running in) | 15:18 |
| clarkb | mordred: note you can install docker-compose from pypi | 15:19 |
| fungi | clarkb: sure, but that doesn't get us gerrit logging to stdout/stderr though right? | 15:19 |
| mordred | clarkb: yes. that is the solution in this case | 15:19 |
| clarkb | fungi: its been a while since I ran gerrit in the foreground. I seem to recall it does write some stuff to stdout/stderr but maybe not with a proper config | 15:19 |
| mordred | installing from apt produces some really excellent openssl library mismatch errors, because python grabs some bits from /usr/local and some bits from /usr and they are not happy with each other | 15:20 |
| fungi | that's a good point though, it does at least write its logfiles outside the jvm storage, so it's presumably exposed somehow | 15:20 |
| *** ricolin has joined #openstack-infra | 15:21 | |
| mordred | yeah - there is an option to log to the console - so presumably we could use that and then we'd have logs in the docker logs location ... | 15:22 |
| mordred | but given we're already trained to look for logs in /home/gerrit2/review_site/logs - I'm not sure that's a big win at the moment | 15:22 |
| corvus | mordred: did i see a patch to install jeepyb in the container? did that land? is it both in and out of the container? or is it only out and you bind-mount stuff in? | 15:24 |
| mordred | corvus: yes - you did. and it's both in and out of the container at the moment | 15:24 |
| corvus | cool, gtk | 15:24 |
| mordred | it's needed in the container because of the hook scripts | 15:24 |
| mordred | but it's needed outside of the container because of manage-projects | 15:25 |
| fungi | if it's https://review.opendev.org/682601 it has not merged yet | 15:25 |
| fungi | it's past complaining about missing repos though | 15:25 |
| mordred | we could, of course, rework our usage of jeepyb outside of the container to just use the container to execut jeepyb scripts | 15:25 |
| fungi | and on to actual job failures | 15:25 |
| openstackgerrit | Kerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters https://review.opendev.org/683183 | 15:25 |
| mordred | but I think we should step our way to that :) | 15:25 |
| mordred | I also learned that we need the gerrit tmp dir to exist in the container - because loading plugins expands them :) | 15:28 |
| zbr|ruck | anyone aware of limestone mirror issues? (pip): https://d3f814c2cb98e7325b60-dc7b8fac10688d132d58a9050206c3ae.ssl.cf2.rackcdn.com/682385/1/gate/tripleo-ci-centos-7-undercloud-containers/3f5c9c4/job-output.txt | 15:30 |
| mnaser | mirror.regionone.limestone.openstack.org looks like it sfine | 15:33 |
| mnaser | but maybe network issues | 15:33 |
| mnaser | cc logan- | 15:33 |
| clarkb | zbr|ruck: mnaser logan- the mirror itself is up (if you hit / its happy) the /pypi/simple path is indeed timing out and that is a caching proxy path to pypi proper | 15:35 |
| clarkb | wget pypi.org seems to be failing from that host | 15:35 |
| clarkb | Connecting to pypi.org (pypi.org)|2a04:4e42:400::223|:80 and it is just sitting there | 15:35 |
| *** gfidente has quit IRC | 15:35 | |
| *** pgaxatte has quit IRC | 15:35 | |
| clarkb | so either a pypi outage or a network outage | 15:35 |
| *** kopecmartin is now known as kopecmartin|off | 15:35 | |
| clarkb | I am able to wget google.com over ipv6 fwiw | 15:36 |
| fungi | likely a problem with the local fastly cdn endpoint it's being directed to | 15:39 |
| openstackgerrit | wes hayutin proposed opendev/elastic-recheck master: add query for bindep pip install failures https://review.opendev.org/683188 | 15:39 |
| openstackgerrit | wes hayutin proposed opendev/elastic-recheck master: add query for bindep pip install failures https://review.opendev.org/683188 | 15:39 |
| clarkb | fungi: ya I'm able to wget from pypi via ovh ipv6 in canada | 15:39 |
| *** zhangfei has quit IRC | 15:39 | |
| clarkb | using the same IP in limestone fails | 15:39 |
| clarkb | likely is a region cdn or network problem | 15:40 |
| *** weshay_ has joined #openstack-infra | 15:40 | |
| slaweq | clarkb: thx, I will look at it later today | 15:43 |
| openstackgerrit | Merged openstack/project-config master: Add zuul-registry project https://review.opendev.org/682743 | 15:43 |
| *** ykarel|away has joined #openstack-infra | 15:46 | |
| fungi | clarkb: yeah, could be bgp trouble impacting specific routes/peering points | 15:47 |
| *** Goneri has quit IRC | 15:47 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Do not overwrite image upload ZK data on delete https://review.opendev.org/681857 | 15:53 |
| paladox | mordred: are you using gerrit.sh to start? | 15:53 |
| paladox | And using systemd with the script? | 15:54 |
| clarkb | paladox: on current production we are | 15:54 |
| clarkb | paladox: the docker images invoke the war directly as the docker image command | 15:54 |
| *** ykarel|away has quit IRC | 15:54 | |
| paladox | Ah! | 15:54 |
| paladox | Yeh, we migrated to using the war directly when we moved towards the scap deploy | 15:54 |
| mordred | paladox: yah. it doesn't seem to make much sense to run a script around the war inside of the container image | 15:56 |
| *** jklare has quit IRC | 15:56 | |
| paladox | Yup | 15:57 |
| mordred | or, at least, not in our context - I think it does for the general purpose images that are being made, since those don't assume any surrounding orchestration and the script can make sure the things ahve been done and whatnot | 15:57 |
| *** mgoddard has quit IRC | 15:58 | |
| zbr|ruck | weshay_: ^ | 15:58 |
| *** jklare has joined #openstack-infra | 15:59 | |
| logan- | clarkb zbr|ruck mnaser ack. looking into it. | 15:59 |
| donnyd | https://www.irccloud.com/pastebin/b6OP1R9s/ | 15:59 |
| donnyd | clarkb: I have no issue from here | 15:59 |
| *** mgoddard has joined #openstack-infra | 15:59 | |
| zbr|ruck | yeah, works-for-me too locally, but the builds are not happening on my localhost ;) | 16:00 |
| zbr|ruck | i remember seeing similar issue in the past, not sure if was limestone something else, but it was pypi, same kind of issue. | 16:01 |
| clarkb | pypi is hosted with fastly cdn and they have endpoints around the world including DFW which is where limestone is hosted. I presume that it is going to be specific to the routes or cdn hosts there | 16:01 |
| clarkb | https://status.python.org/ claims they are all operational though | 16:02 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Do not overwrite image upload ZK data on delete https://review.opendev.org/681857 | 16:03 |
| openstackgerrit | Merged opendev/system-config master: Fix gerrit java invocation https://review.opendev.org/683124 | 16:05 |
| corvus | mordred: ^ | 16:05 |
| *** openstackgerrit has quit IRC | 16:06 | |
| *** takamatsu has joined #openstack-infra | 16:06 | |
| mordred | \o/ | 16:06 |
| *** iurygregory has quit IRC | 16:08 | |
| *** kjackal has quit IRC | 16:09 | |
| *** jamesmcarthur has quit IRC | 16:12 | |
| *** e0ne has quit IRC | 16:14 | |
| clarkb | as a sanity check ipv4 does work to wget pypi from the limestone mirror | 16:14 |
| clarkb | assuming that ends up at the same cdn endpoint as v6 the cdn itself is probably fine and instead we have networking problems? | 16:15 |
| *** markvoelker has joined #openstack-infra | 16:15 | |
| *** mriedem is now known as mriedem_afk | 16:15 | |
| clarkb | rax's DFW mirror can hit pypi over ipv6 too | 16:17 |
| clarkb | (not sure if that helps isloate it at all) | 16:17 |
| *** ykarel|away has joined #openstack-infra | 16:18 | |
| logan- | what i'm seeing from our dfw smokeping towards the mirror this morning: https://i.imgur.com/BGjTgRz.jpg (ping) vs https://i.imgur.com/XsJTj6Y.jpg (http), so yes i agree, not external, it seems like something internal to the nodepool cluster's neutron env. i'm disabling the nodepool host aggregate temporarily so we at least don't spawn more jobs in there while debugging. | 16:18 |
| clarkb | logan-: fwiw google ipv6 worked from there so wasn't a global ipv6 outage | 16:18 |
| clarkb | (also a global ipv6 outage in the region would prevent nodepool and zuul from sshing in and no jobs would run there) | 16:19 |
| clarkb | of cource that probably means it will be more difficult to debug :) | 16:19 |
| *** kjackal has joined #openstack-infra | 16:20 | |
| *** jamesmcarthur has joined #openstack-infra | 16:21 | |
| logan- | yeah i'm curious to see if it will persist after the load is drained | 16:21 |
| logan- | its similar to some breakage i've seen in the past, and it usually does seem to persist regardless of cluster usage until i reboot the network nodes :/ | 16:22 |
| *** dtantsur is now known as dtantsur|afk | 16:22 | |
| *** weshay_ is now known as weshay | 16:23 | |
| fungi | logan-: also it can't reach us.archive.ubuntu.com (2001:67c:1562::16) | 16:24 |
| fungi | presumably the same issue | 16:25 |
| *** kjackal has quit IRC | 16:25 | |
| *** kjackal_v2 has joined #openstack-infra | 16:26 | |
| *** udesale has quit IRC | 16:27 | |
| *** udesale has joined #openstack-infra | 16:28 | |
| *** Goneri has joined #openstack-infra | 16:29 | |
| zbr|ruck | i guess we kinda need some kinda of active monitoring on networking for each region | 16:30 |
| logan- | fungi: thanks, confirmed on a test instance instance. www.google.com (2607:f8b0:4000:808::2004) works, but the ubuntu archive IP doesn't even reach our switch :( i'll do some tracing through the neutron vxlan bridges and see if i can figure out where it is dropping | 16:32 |
| donnyd | zbr|ruck: maybe checkout skydive | 16:33 |
| donnyd | its does these types of things pretty well | 16:33 |
| *** mattw4 has joined #openstack-infra | 16:34 | |
| fungi | zbr|ruck: it would likely have to be somewhat involved to catch problems like this one which only result in trouble reaching some destinations and not others | 16:34 |
| zbr|ruck | donnyd: looks cool, as long you are not proposing me to build one i am ok. i seen too much stuff done in-house ;) | 16:34 |
| logan- | fungi: out of curiosity, do you see any globally addressable ipv6 addresses on mirror that are not in 2607:ff68::/32? | 16:36 |
| donnyd | I have it functional at FN, and so long as you have access you can trace paths all the way through the SDN | 16:36 |
| logan- | fungi: somehow these two mysterious addresses have appeared on my test node http://paste.openstack.org/raw/777717/ | 16:36 |
| fungi | logan-: yep, http://paste.openstack.org/show/777718/ | 16:38 |
| fungi | they're all eui64 addresses | 16:39 |
| *** mattw4 has quit IRC | 16:39 | |
| *** mattw4 has joined #openstack-infra | 16:39 | |
| fungi | for different prefixes | 16:39 |
| fungi | do you have route announcements leaking? | 16:40 |
| fungi | between networks | 16:40 |
| logan- | i suspect those are coming from other nodepool instances | 16:40 |
| *** ociuhandu has quit IRC | 16:40 | |
| fungi | oh, ouch | 16:40 |
| *** dciabrin has quit IRC | 16:41 | |
| fungi | nodepool instances spilling route announcements into the shared layer 2 segment? | 16:41 |
| fungi | from jobs setting up v6 routers on nodes maybe | 16:41 |
| *** Goneri has quit IRC | 16:41 | |
| logan- | yep, that's my first guess. i think we could probably tie the gateway nexthops in ip -6 route to instances and confirm it. | 16:42 |
| *** jamesmcarthur has quit IRC | 16:42 | |
| *** jamesmcarthur has joined #openstack-infra | 16:43 | |
| *** igordc has joined #openstack-infra | 16:44 | |
| *** e0ne has joined #openstack-infra | 16:45 | |
| donnyd | oo I wonder if that effects FN as well | 16:48 |
| donnyd | Never really looked | 16:48 |
| fungi | logan-: no dice, only default has a via: http://paste.openstack.org/show/777720/ | 16:48 |
| logan- | i see 5: http://paste.openstack.org/raw/777722/ | 16:51 |
| logan- | on my test instance | 16:51 |
| logan- | identified 1 as one of the neutron router ports, workign to identify the remaining ones now | 16:51 |
| *** udesale has quit IRC | 16:52 | |
| donnyd | can it hurt to drop outbound RA's in the sec group? | 16:53 |
| clarkb | donnyd: you can't blacklist traffic in the security groups so that might be painful to keep an otherwise open rulsset | 16:53 |
| clarkb | (though that may be necessary to fix this) | 16:54 |
| donnyd | I would only worry that it would effect multi-node jobs | 16:54 |
| clarkb | multinode jobs should carry all that traffic on a test specific overlay (in part because providers drop random traffic necessary to test that and also becuse some cloud don't have ipv6) | 16:55 |
| donnyd | yea that is true | 16:55 |
| logan- | no hits in neutron.ports.mac_address sql column.. but it looks like ports are hard deleted from that table afaict. i wonder if there's historical port information somewhere. | 16:57 |
| *** openstackgerrit has joined #openstack-infra | 16:59 | |
| openstackgerrit | Kerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters https://review.opendev.org/683205 | 16:59 |
| *** kjackal_v2 has quit IRC | 17:00 | |
| *** pcaruana has joined #openstack-infra | 17:01 | |
| *** markvoelker has quit IRC | 17:01 | |
| *** Goneri has joined #openstack-infra | 17:01 | |
| fungi | logan-: if we're not currently running jobs there, we could reboot the mirror and see if those routes reappear. also see if it suddenly starts being able to reach things it previously couldn't (though that seems unlikely) | 17:03 |
| clarkb | fungi: I like that idea | 17:04 |
| logan- | since you're not seeing the routes, only the unroutable addresses, maybe you could just ip -6 addr del them | 17:04 |
| *** markvoelker has joined #openstack-infra | 17:04 | |
| fungi | sure, that's worth a try, just a sec | 17:04 |
| *** zbr|ruck is now known as zbr | 17:07 | |
| fungi | RTNETLINK answers: Cannot assign requested address | 17:08 |
| logan- | mysqldump --opt --all-databases | grep -i 'FA:16:3E:16:9A:9A' yields no results. no hits in logstash either. the host facts that zuul collects for each node in zuul-info/host-info*.yml of each job stores the mac address, but it looks like that is not indexed in logstash and not searchable | 17:08 |
| fungi | that's confusing | 17:08 |
| fungi | i'm not trying to assign any address, only delete | 17:08 |
| fungi | that's when trying to `sudo ip ad del 2003::f816:3eff:febd:b93 dev ens3` | 17:09 |
| logan- | i think you'll need the cidr on the end to scope it properly | 17:09 |
| fungi | ahh | 17:10 |
| logan- | going to try searching through systemd journals for hits on these macs | 17:10 |
| fungi | also worth skimming http://codesearch.openstack.org/?q=2003%3A%3A | 17:10 |
| fungi | seems we use 2003:: as a test network prefix in a few scenarios | 17:10 |
| logan- | ++ good idea | 17:11 |
| fungi | logan-: you're right, adding the /64 solved it | 17:11 |
| fungi | surprising that it wants the mask when deleting, i would have thought non-linklocal addresses would be sufficiently unique that it wouldn't expect the same address with multiple masks | 17:12 |
| clarkb | FA:16:3E appears to be an unassigned prefix? | 17:12 |
| clarkb | in which case would be likely to come from not real interface managed by eg neutron? | 17:13 |
| *** Goneri has quit IRC | 17:14 | |
| *** pcaruana has quit IRC | 17:14 | |
| fungi | oh, you mean not in the oui list | 17:14 |
| clarkb | ya | 17:14 |
| *** ociuhandu has joined #openstack-infra | 17:16 | |
| *** ociuhandu has quit IRC | 17:16 | |
| clarkb | I can get pypi.org now | 17:17 |
| fungi | interesting | 17:17 |
| fungi | yeah, it's going to 2a04:4e42:400::223 | 17:17 |
| clarkb | what could be happening is that we originate the connection from the bad RA'd ip which then routes through the default route but has no route back to use | 17:17 |
| clarkb | s/use/us/ | 17:17 |
| clarkb | so the default route being set or not doesn't really matter | 17:18 |
| fungi | that's the same address it was getting directed to before, for the record | 17:18 |
| fungi | i concur, i think the server expected any of its three global scope addresses should be usable and was picking one to source the connection from | 17:19 |
| *** jcoufal has joined #openstack-infra | 17:21 | |
| *** ralonsoh has quit IRC | 17:21 | |
| logan- | rp_filter should be dropping it at the neutron router | 17:21 |
| *** gmann_afk is now known as gmann | 17:21 | |
| logan- | but yeah, i think it is originating traffic from those fake addresses | 17:21 |
| *** jpena is now known as jpena|off | 17:22 | |
| openstackgerrit | Merged opendev/system-config master: Remove bazel version hack https://review.opendev.org/682469 | 17:22 |
| logan- | and yep clarkb I think those are all virtual macs. I'm pretty sure it would be possible for our physical network to leak RA's into the nodepool network, as it is not bridged to anything physical. it is only connected via a neutron virtual router | 17:23 |
| clarkb | logan-: ya just good to be able to narrow it down | 17:23 |
| *** ramishra has quit IRC | 17:23 | |
| *** ociuhandu has joined #openstack-infra | 17:24 | |
| logan- | sorry i meant it should not be possible* | 17:25 |
| *** pcaruana has joined #openstack-infra | 17:25 | |
| clarkb | logan-: I understood | 17:25 |
| logan- | :) | 17:25 |
| clarkb | basically we see this from the virtual side and not real interfaces leaking through somehow | 17:25 |
| *** ociuhandu has quit IRC | 17:28 | |
| *** ricolin has quit IRC | 17:31 | |
| *** mriedem_afk is now known as mriedem | 17:33 | |
| *** ykarel|away has quit IRC | 17:34 | |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for test_boot_server_from_encrypted_volume_luks bug 1820007 https://review.opendev.org/683210 | 17:42 |
| openstack | bug 1820007 in os-brick "Failed to attach encrypted volumes after detach: volume device not found at /dev/disk/by-id" [Undecided,New] https://launchpad.net/bugs/1820007 | 17:42 |
| *** rkukura has joined #openstack-infra | 17:52 | |
| *** pcaruana has quit IRC | 17:55 | |
| AJaeger | config-core, please review these two changes to remove now obsolete publishing jobs: https://review.opendev.org/#/c/682972/ and https://review.opendev.org/683154 | 17:56 |
| AJaeger | config-core, and https://review.opendev.org/#/c/682544/ to finish retiring of os-acc | 17:57 |
| *** efried_pto has quit IRC | 18:01 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for test_boot_server_from_encrypted_volume_luks bug 1820007 https://review.opendev.org/683210 | 18:06 |
| openstack | bug 1820007 in os-brick "Failed to attach encrypted volumes after detach: volume device not found at /dev/disk/by-id" [Undecided,Confirmed] https://launchpad.net/bugs/1820007 | 18:06 |
| *** priteau has quit IRC | 18:07 | |
| *** Goneri has joined #openstack-infra | 18:12 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix gerrit errors from production https://review.opendev.org/683006 | 18:19 |
| *** igordc has quit IRC | 18:20 | |
| *** jamesmcarthur has quit IRC | 18:21 | |
| *** jamesmcarthur has joined #openstack-infra | 18:22 | |
| *** whoami-rajat has quit IRC | 18:25 | |
| *** michael-beaver has quit IRC | 18:25 | |
| *** priteau has joined #openstack-infra | 18:27 | |
| *** priteau has quit IRC | 18:29 | |
| *** JorgeFranco has joined #openstack-infra | 18:40 | |
| *** trident has quit IRC | 18:45 | |
| *** lucasagomes has quit IRC | 18:53 | |
| *** trident has joined #openstack-infra | 18:55 | |
| rm_work | hey can someone remind me, was Shade rolling into openstack-sdk / being replaced? | 18:56 |
| rm_work | were they kind of a merge? | 18:56 |
| clarkb | rm_work: shade is now part of openstack sdk | 18:57 |
| rm_work | kk | 18:57 |
| rm_work | so it's kinda the same thing | 18:57 |
| clarkb | rm_work: the sdk repo contains both low level and high (shade) level objects you can interact with | 18:57 |
| clarkb | they are in the same python package but you select whether or not you want one or the other based on objects that you use | 18:57 |
| clarkb | (or maybe its just function calls?) | 18:58 |
| fungi | and also has a direct-to-rest proxy as well for extra-low-level needs | 18:58 |
| fungi | you don't have to choose just one either | 18:58 |
| fungi | you can for example get a keystone session and then use that with some shade objects and some direct rest api calls | 18:59 |
| fungi | mix-n-match as needed | 18:59 |
| *** JorgeFranco has quit IRC | 18:59 | |
| *** trident has quit IRC | 19:01 | |
| *** JorgeFranco has joined #openstack-infra | 19:04 | |
| rm_work | thanks | 19:05 |
| rm_work | and neither is going away or necessarily recommended over the other | 19:06 |
| rm_work | no deprecations | 19:06 |
| rm_work | just use what makes sense | 19:06 |
| clarkb | that is how I understand it | 19:07 |
| *** Diabelko has quit IRC | 19:09 | |
| *** trident has joined #openstack-infra | 19:10 | |
| *** Diabelko has joined #openstack-infra | 19:15 | |
| *** kjackal has joined #openstack-infra | 19:21 | |
| rm_work | ok | 19:23 |
| rm_work | there's some confusion with the folks i'm talking with since they found in some doc where it says Shade is superseded by openstack-sdk | 19:24 |
| rm_work | so there's a lot of talk about "updating this code to support openstack-sdk instead of shade" | 19:24 |
| clarkb | rm_work: well that bit is true, its just that the spirit of shade (and most of its code) is now in openstacksdk | 19:25 |
| clarkb | so your update is to import the shade bits from sdk package instead of the shade package | 19:25 |
| rm_work | k | 19:25 |
| *** jcoufal has quit IRC | 19:26 | |
| AJaeger | config-core, please review these two changes to remove now obsolete publishing jobs: https://review.opendev.org/#/c/682972/ and https://review.opendev.org/683154; | 19:38 |
| clarkb | AJaeger: are the related secrets able to be cleaned up too or do we use them elsewhere? | 19:40 |
| openstackgerrit | Merged openstack/project-config master: Remove unused stx docs publish job https://review.opendev.org/683154 | 19:48 |
| *** yamamoto has joined #openstack-infra | 19:49 | |
| AJaeger | I think we use them elsewhere - let me double check | 19:49 |
| AJaeger | clarkb: yes, used still in many places | 19:51 |
| AJaeger | config-core, another +2a on https://review.opendev.org/682972 , please? | 19:51 |
| openstackgerrit | Matthew Booth proposed openstack/diskimage-builder master: Fix invalid element-dep in vm https://review.opendev.org/683221 | 19:51 |
| AJaeger | clarkb: thanks | 19:51 |
| *** pcaruana has joined #openstack-infra | 19:51 | |
| AJaeger | config-core, a few more ready changes if anybody has time: https://review.opendev.org/682544 https://review.opendev.org/567696 https://review.opendev.org/682954 | 19:52 |
| *** yamamoto has quit IRC | 19:53 | |
| *** jamesmcarthur has quit IRC | 20:01 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Remove parent from validate-zone-db https://review.opendev.org/683225 | 20:02 |
| *** jamesmcarthur has joined #openstack-infra | 20:04 | |
| *** kjackal has quit IRC | 20:06 | |
| *** jamesmcarthur has quit IRC | 20:20 | |
| *** jamesmcarthur has joined #openstack-infra | 20:21 | |
| *** jamesmcarthur has quit IRC | 20:22 | |
| *** larainema has joined #openstack-infra | 20:31 | |
| *** yamamoto has joined #openstack-infra | 20:31 | |
| *** pcaruana has quit IRC | 20:31 | |
| *** igordc has joined #openstack-infra | 20:33 | |
| *** TomStappaerts has joined #openstack-infra | 20:34 | |
| *** JorgeFranco has quit IRC | 20:35 | |
| *** JorgeFranco has joined #openstack-infra | 20:35 | |
| *** yamamoto has quit IRC | 20:36 | |
| *** sshnaidm is now known as sshnaidm|off | 20:42 | |
| *** eharney has quit IRC | 20:42 | |
| *** kjackal has joined #openstack-infra | 20:44 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Remove parent from validate-zone-db https://review.opendev.org/683225 | 20:47 |
| *** e0ne has quit IRC | 21:00 | |
| *** slaweq has quit IRC | 21:01 | |
| openstackgerrit | Colleen Murphy proposed openstack/project-config master: Use iptables for openSUSE https://review.opendev.org/683236 | 21:11 |
| *** e0ne has joined #openstack-infra | 21:12 | |
| *** markvoelker has quit IRC | 21:20 | |
| *** yamamoto has joined #openstack-infra | 21:20 | |
| *** benj_ has quit IRC | 21:31 | |
| *** benj has joined #openstack-infra | 21:31 | |
| *** benj is now known as Guest29634 | 21:32 | |
| *** yamamoto has quit IRC | 21:32 | |
| *** cloudnull has joined #openstack-infra | 21:34 | |
| *** yamamoto has joined #openstack-infra | 21:35 | |
| *** lbragstad has quit IRC | 21:37 | |
| *** lbragstad has joined #openstack-infra | 21:37 | |
| *** eharney has joined #openstack-infra | 21:39 | |
| *** yamamoto has quit IRC | 21:44 | |
| *** yamamoto has joined #openstack-infra | 21:44 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed opendev/system-config master: Add proxy for quay https://review.opendev.org/683240 | 21:46 |
| *** markvoelker has joined #openstack-infra | 21:48 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed opendev/system-config master: Add proxy for quay https://review.opendev.org/683240 | 21:49 |
| *** xek has quit IRC | 21:51 | |
| *** eharney has quit IRC | 21:52 | |
| *** mattw4 has quit IRC | 22:01 | |
| *** mattw4 has joined #openstack-infra | 22:02 | |
| *** kjackal has quit IRC | 22:08 | |
| *** weifan has joined #openstack-infra | 22:09 | |
| *** weifan has quit IRC | 22:09 | |
| *** TomStappaerts has quit IRC | 22:15 | |
| *** JorgeFranco has quit IRC | 22:16 | |
| *** e0ne has quit IRC | 22:24 | |
| *** rlandy is now known as rlandy|biab | 22:30 | |
| *** notmyname has quit IRC | 22:30 | |
| *** notmyname has joined #openstack-infra | 22:31 | |
| *** markvoelker has quit IRC | 22:32 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Don't use urllib3 1.25.4 https://review.opendev.org/683250 | 22:34 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix gerrit errors from production https://review.opendev.org/683006 | 22:35 |
| *** armax has joined #openstack-infra | 22:39 | |
| *** mriedem is now known as mriedem_away | 22:41 | |
| *** xenos76 has quit IRC | 22:43 | |
| *** igordc has quit IRC | 22:44 | |
| *** signed8b_ has joined #openstack-infra | 22:48 | |
| *** signed8bit has quit IRC | 22:50 | |
| *** Tengu has quit IRC | 22:51 | |
| *** yamamoto has quit IRC | 22:57 | |
| *** goldyfruit___ has quit IRC | 23:00 | |
| *** igordc has joined #openstack-infra | 23:01 | |
| *** armax has quit IRC | 23:01 | |
| *** tkajinam has joined #openstack-infra | 23:02 | |
| *** mattw4 has quit IRC | 23:03 | |
| *** yamamoto has joined #openstack-infra | 23:03 | |
| *** Tengu has joined #openstack-infra | 23:03 | |
| *** mattw4 has joined #openstack-infra | 23:03 | |
| *** yamamoto has quit IRC | 23:08 | |
| *** yamamoto has joined #openstack-infra | 23:08 | |
| *** yamamoto has quit IRC | 23:12 | |
| *** tosky has quit IRC | 23:12 | |
| *** yamamoto has joined #openstack-infra | 23:12 | |
| *** rlandy|biab is now known as rlandy|bbl | 23:18 | |
| *** kaiokmo has quit IRC | 23:23 | |
| *** rcernin has joined #openstack-infra | 23:25 | |
| *** yamamoto has quit IRC | 23:26 | |
| *** dchen has joined #openstack-infra | 23:27 | |
| *** yamamoto has joined #openstack-infra | 23:28 | |
| *** e0ne has joined #openstack-infra | 23:30 | |
| *** e0ne has quit IRC | 23:30 | |
| *** yamamoto has quit IRC | 23:31 | |
| *** yamamoto has joined #openstack-infra | 23:31 | |
| *** yamamoto has quit IRC | 23:39 | |
| *** aaronsheffield has quit IRC | 23:44 | |
| *** yamamoto has joined #openstack-infra | 23:46 | |
| *** sthussey has quit IRC | 23:46 | |
| *** yamamoto has quit IRC | 23:48 | |
| *** yamamoto has joined #openstack-infra | 23:49 | |
| *** e0ne has joined #openstack-infra | 23:55 | |
| *** e0ne has quit IRC | 23:55 | |
| *** signed8b_ has quit IRC | 23:56 | |
| *** signed8bit has joined #openstack-infra | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
