| clarkb | logan-: based on https://bugs.launchpad.net/juju/+bug/1764267 my hunch is that because we started mirroring bionic prior to release we pulled it in then when it was removed reprepro didn't remove it? | 00:00 |
|---|---|---|
| openstack | Launchpad bug 1764267 in juju 2.3 "python-software-properties not found on bionic" [High,Fix released] - Assigned to John A Meinel (jameinel) | 00:00 |
| *** gongysh has quit IRC | 00:02 | |
| *** eernst has quit IRC | 00:02 | |
| *** dklyle has quit IRC | 00:02 | |
| *** jamesmcarthur has quit IRC | 00:08 | |
| logan- | ah I see | 00:11 |
| *** jamesmcarthur has joined #openstack-infra | 00:11 | |
| *** jamesmcarthur has quit IRC | 00:13 | |
| *** jamesmcarthur has joined #openstack-infra | 00:14 | |
| *** felipemonteiro has joined #openstack-infra | 00:17 | |
| clarkb | down to 24 DOWN ports in packethost cloud | 00:17 |
| clarkb | those are all fairly recent, within the last day or so | 00:17 |
| clarkb | that is about half of the ports cleaned up | 00:17 |
| clarkb | going to try and cross reference them against nodepool logs to see if they crashed or otherwise failed to work properly | 00:18 |
| *** jamesmcarthur has quit IRC | 00:18 | |
| *** jamesmcarthur has joined #openstack-infra | 00:18 | |
| *** honza has joined #openstack-infra | 00:20 | |
| *** honza is now known as Guest70154 | 00:20 | |
| *** Guest70154 is now known as honza_ | 00:21 | |
| *** felipemonteiro has quit IRC | 00:24 | |
| *** markvoelker has quit IRC | 00:25 | |
| mordred | infra-root: don't restart nodepool at the moment - the bug in sdk related to vexxhost will make our vexxhost quota go away if we do | 00:26 |
| mordred | s/go away/be unusable/ | 00:26 |
| clarkb | mordred: can we update clouds.yaml to fix? | 00:26 |
| mordred | we can either fix temporarily by adding auth_type to clouds.yaml ... | 00:26 |
| mordred | or wait for sdk release when I wake up in the morning | 00:26 |
| mordred | (the patch to fix is going through the gate) | 00:26 |
| mordred | the fix in clouds.yaml is not dangerous - maybe I should just push that up for now? | 00:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Set vexxhost auth_type to password https://review.openstack.org/583800 | 00:28 |
| mordred | clarkb: ^^ | 00:28 |
| clarkb | ya we'd need an sdk release too | 00:29 |
| clarkb | so probably good idea for us to update our clouds.yaml directly in the interim | 00:29 |
| clarkb | on the leaked port front it affects more than one image (xenial and debian at least) and appears to happen after shade.exc.OpenStackCloudTimeout: Timeout waiting for the server to come up. | 00:29 |
| clarkb | there are many of those timeouts but only ~50 leaked ports | 00:30 |
| clarkb | so I don't think it fully explains the issue, but then after enough ports leak we also have errors due to port quita | 00:31 |
| clarkb | john_studarus: this means if you need to look at one of the two propblems from earlier I think the timeouts are more important | 00:31 |
| *** r-daneel has quit IRC | 00:32 | |
| *** felipemonteiro has joined #openstack-infra | 00:36 | |
| *** john_studarus has joined #openstack-infra | 00:39 | |
| john_studarus | clarkb port quota was down at 110 - bumped it up | 00:39 |
| *** Anticimex has quit IRC | 00:41 | |
| *** Anticimex has joined #openstack-infra | 00:46 | |
| *** jamesmcarthur has quit IRC | 00:47 | |
| *** john_studarus has quit IRC | 00:50 | |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Drop legacy-tempest-dsvm-neutron-full-ocata from Tempest https://review.openstack.org/583801 | 00:51 |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Drop legacy scenario multinode job from Tempest https://review.openstack.org/583802 | 00:53 |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Drop legacy-tempest-dsvm-neutron-full-ocata from Tempest https://review.openstack.org/583801 | 00:56 |
| *** alex_xu has quit IRC | 00:57 | |
| *** felipemonteiro has quit IRC | 00:57 | |
| *** dpawlik has joined #openstack-infra | 00:58 | |
| *** alex_xu has joined #openstack-infra | 01:02 | |
| *** dpawlik has quit IRC | 01:03 | |
| *** haleyb has quit IRC | 01:05 | |
| *** harlowja has quit IRC | 01:09 | |
| *** haleyb has joined #openstack-infra | 01:13 | |
| ianw | arrrgghh, it looks like the "id" returned by readthedocs "query the project api" is *not* the same id you need to pass to the webhook for the project to trigger it | 01:16 |
| *** mrsoul_ has joined #openstack-infra | 01:21 | |
| *** graphene has quit IRC | 01:22 | |
| *** graphene has joined #openstack-infra | 01:24 | |
| *** mschuppert has quit IRC | 01:24 | |
| *** dizquierdo has joined #openstack-infra | 01:25 | |
| *** hongbin has joined #openstack-infra | 01:34 | |
| *** dpawlik has joined #openstack-infra | 01:37 | |
| *** luzC has quit IRC | 01:37 | |
| *** dpawlik has quit IRC | 01:41 | |
| *** luzC has joined #openstack-infra | 01:41 | |
| *** agopi has joined #openstack-infra | 01:42 | |
| *** iyamahat__ has joined #openstack-infra | 01:43 | |
| *** evrardjp has quit IRC | 01:44 | |
| *** iyamahat_ has quit IRC | 01:45 | |
| *** yamahata has quit IRC | 01:46 | |
| *** iyamahat__ has quit IRC | 01:47 | |
| *** alex_xu has quit IRC | 01:47 | |
| *** iyamahat__ has joined #openstack-infra | 01:47 | |
| *** alex_xu has joined #openstack-infra | 01:48 | |
| *** evrardjp has joined #openstack-infra | 01:49 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Use readthedocs webhook to trigger build https://review.openstack.org/579434 | 01:49 |
| *** iyamahat__ has quit IRC | 01:51 | |
| *** bobh has joined #openstack-infra | 01:52 | |
| *** ramishra has joined #openstack-infra | 01:54 | |
| ianw | tristanC / pabelanger: weird comment from rdo testing on https://review.openstack.org/#/c/579434/? says it failed but all the jobs passed? | 01:57 |
| *** dizquierdo has quit IRC | 01:57 | |
| ianw | sorry, software factory CI | 01:58 |
| *** ramishra has quit IRC | 02:01 | |
| *** evrardjp has quit IRC | 02:02 | |
| tristanC | ianw: could it be because there is a syntax error? | 02:02 |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config master: Add openstack-dev/bashate to #openstack-qa channel https://review.openstack.org/583812 | 02:03 |
| gmann | ianw: ^^ | 02:03 |
| ianw | tristanC: yeah ... i guess i can't remove that job as part of this change. i wonder why this didn't trigger a syntax error before though | 02:04 |
| *** zhangfei has joined #openstack-infra | 02:05 | |
| *** evrardjp has joined #openstack-infra | 02:07 | |
| ianw | tristanC: about to upload a follow-on; want me to wait a bit? | 02:08 |
| tristanC | ianw: 1 minute please | 02:08 |
| tristanC | change is running here: https://softwarefactory-project.io/zuul/t/openstack.org/status.html | 02:09 |
| ianw | tristanC: i guess in this case, SF doesn't notice because it doesn't have the reference to the role that project-config does? | 02:11 |
| ianw | s/role/job | 02:11 |
| tristanC | hum, that's weird. maybe sf's zuul is having sympathy for openstack's zuul :) | 02:11 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-lint job https://review.openstack.org/532083 | 02:13 |
| *** SpamapS has quit IRC | 02:15 | |
| tristanC | ianw: ok, there is definitely something wrong, it also happens on 532083 | 02:16 |
| *** annp has joined #openstack-infra | 02:16 | |
| *** agopi has quit IRC | 02:17 | |
| openstackgerrit | Merged openstack-infra/zuul master: Remove redhat-rpm-config bindep https://review.openstack.org/581252 | 02:18 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix github setup in zuul from scratch https://review.openstack.org/581253 | 02:18 |
| *** graphene has quit IRC | 02:23 | |
| *** graphene has joined #openstack-infra | 02:24 | |
| *** mriedem has quit IRC | 02:25 | |
| *** markvoelker has joined #openstack-infra | 02:26 | |
| pabelanger | clarkb: wfm, but maybe also apply to RAM? Should be the same idea, percentage | 02:26 |
| *** bobh has quit IRC | 02:27 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Add role to trigger readthedocs via authenticated endpoint https://review.openstack.org/579434 | 02:28 |
| *** ramishra has joined #openstack-infra | 02:29 | |
| tristanC | ianw: hum, it seems like this is happening when no jobs are voting | 02:29 |
| *** bobh has joined #openstack-infra | 02:33 | |
| *** jamesmcarthur has joined #openstack-infra | 02:33 | |
| pabelanger | tristanC: ianw: I wonder if I21ea9b2d3b1711f0d55bbe3d626ac7dde5db2919 is the reason why | 02:33 |
| tristanC | ianw: pabelanger: it seems like https://review.openstack.org/578230 now makes build failed when all jobs are non voting | 02:33 |
| pabelanger | tristanC: yah, we might want to follow up with corvus tomorrow morning, that is something I didn't test | 02:34 |
| *** caphrim007 has quit IRC | 02:34 | |
| tristanC | i can add a noop job in the meantime | 02:35 |
| pabelanger | ++ | 02:36 |
| pabelanger | EOD now, but will follow up in morning | 02:36 |
| tristanC | pabelanger: thanks | 02:36 |
| *** psachin` has joined #openstack-infra | 02:36 | |
| *** bobh has quit IRC | 02:37 | |
| *** jamesmcarthur has quit IRC | 02:38 | |
| *** felipemonteiro has joined #openstack-infra | 02:40 | |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Drop legacy scenario multinode job from Tempest https://review.openstack.org/583802 | 02:42 |
| *** bobh has joined #openstack-infra | 02:44 | |
| *** dave-mcc_ has quit IRC | 02:45 | |
| *** ramishra has quit IRC | 02:46 | |
| *** bobh has quit IRC | 02:48 | |
| *** bobh has joined #openstack-infra | 02:49 | |
| *** aeng has joined #openstack-infra | 02:56 | |
| *** felipemonteiro has quit IRC | 02:59 | |
| *** markvoelker has quit IRC | 03:00 | |
| *** gongysh has joined #openstack-infra | 03:10 | |
| *** bobh has quit IRC | 03:24 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add trigger-readthedocs-webhook job https://review.openstack.org/583449 | 03:26 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Use webhook update for docs-on-readthedocs https://review.openstack.org/583834 | 03:28 |
| *** anteaya has quit IRC | 03:30 | |
| *** bobh has joined #openstack-infra | 03:31 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Remove trigger-readthedocs job https://review.openstack.org/583836 | 03:34 |
| ianw | mordred / corvus : ^ I think that completes the readthedocs update. i think the sticking point, if any, will be the webhook id mapping in the project-config playbook in https://review.openstack.org/583449 . i will point it out in the review, comments appreciated | 03:35 |
| *** bobh has quit IRC | 03:35 | |
| *** dpawlik has joined #openstack-infra | 03:38 | |
| *** dpawlik has quit IRC | 03:42 | |
| *** gongysh has quit IRC | 03:43 | |
| *** roman_g has quit IRC | 03:44 | |
| *** zhongjun__ has quit IRC | 03:44 | |
| *** bobh has joined #openstack-infra | 03:44 | |
| *** felipemonteiro has joined #openstack-infra | 03:47 | |
| *** bobh has quit IRC | 03:49 | |
| *** armax has quit IRC | 03:51 | |
| *** harlowja has joined #openstack-infra | 03:51 | |
| *** lpetrut has joined #openstack-infra | 03:52 | |
| *** sthussey has quit IRC | 03:57 | |
| *** markvoelker has joined #openstack-infra | 03:58 | |
| *** bobh has joined #openstack-infra | 04:00 | |
| *** udesale has joined #openstack-infra | 04:02 | |
| *** bobh_ has joined #openstack-infra | 04:03 | |
| *** bobh has quit IRC | 04:03 | |
| *** bobh_ has quit IRC | 04:07 | |
| *** links has joined #openstack-infra | 04:08 | |
| *** rh-jelabarre has quit IRC | 04:19 | |
| *** bobh has joined #openstack-infra | 04:20 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: github: log the project name when getBranch fail https://review.openstack.org/583841 | 04:21 |
| *** ccamacho has quit IRC | 04:21 | |
| *** harlowja has quit IRC | 04:22 | |
| *** zul has quit IRC | 04:22 | |
| *** bobh_ has joined #openstack-infra | 04:23 | |
| *** pcaruana has joined #openstack-infra | 04:23 | |
| *** bobh has quit IRC | 04:24 | |
| *** bobh_ has quit IRC | 04:27 | |
| *** ijw has joined #openstack-infra | 04:28 | |
| *** ramishra has joined #openstack-infra | 04:28 | |
| *** markvoelker has quit IRC | 04:31 | |
| *** ijw has quit IRC | 04:33 | |
| *** pcaruana has quit IRC | 04:34 | |
| *** bobh has joined #openstack-infra | 04:35 | |
| *** lpetrut has quit IRC | 04:36 | |
| *** felipemonteiro has quit IRC | 04:37 | |
| *** bobh has quit IRC | 04:39 | |
| *** bobh has joined #openstack-infra | 04:42 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config master: Remove legacy scenario multinode job for nova https://review.openstack.org/583849 | 04:43 |
| *** s-shiono has joined #openstack-infra | 04:44 | |
| *** felipemonteiro has joined #openstack-infra | 04:44 | |
| *** bobh has quit IRC | 04:47 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config master: Remove legacy scenario multinode job for nova https://review.openstack.org/583853 | 04:47 |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config master: Remove legacy scenario multinode job for cinder https://review.openstack.org/583853 | 04:48 |
| *** ykarel has joined #openstack-infra | 04:49 | |
| *** diablo_rojo has quit IRC | 04:51 | |
| *** bobh has joined #openstack-infra | 04:53 | |
| *** felipemonteiro has quit IRC | 04:55 | |
| *** bobh has quit IRC | 04:57 | |
| *** kjackal has joined #openstack-infra | 05:00 | |
| *** felipemonteiro has joined #openstack-infra | 05:01 | |
| *** mpjetta has quit IRC | 05:03 | |
| *** bobh has joined #openstack-infra | 05:03 | |
| *** bobh has quit IRC | 05:07 | |
| openstackgerrit | Merged openstack-infra/project-config master: Consolidate nodepool graphs https://review.openstack.org/577045 | 05:08 |
| *** Bhujay has joined #openstack-infra | 05:12 | |
| *** bobh has joined #openstack-infra | 05:13 | |
| *** pcichy has joined #openstack-infra | 05:13 | |
| *** bobh has quit IRC | 05:18 | |
| *** hongbin has quit IRC | 05:20 | |
| *** mpjetta has joined #openstack-infra | 05:21 | |
| *** ccamacho has joined #openstack-infra | 05:25 | |
| *** bobh has joined #openstack-infra | 05:25 | |
| *** quiquell|off is now known as quiquell | 05:26 | |
| *** markvoelker has joined #openstack-infra | 05:27 | |
| *** efried has quit IRC | 05:29 | |
| *** robcresswell has quit IRC | 05:29 | |
| *** fdegir has quit IRC | 05:29 | |
| *** ameade has quit IRC | 05:29 | |
| *** coreycb has quit IRC | 05:29 | |
| *** pas-ha has quit IRC | 05:29 | |
| *** jamespage has quit IRC | 05:29 | |
| *** kien-ha has quit IRC | 05:29 | |
| *** kien-ha has joined #openstack-infra | 05:29 | |
| *** bobh has quit IRC | 05:29 | |
| *** Dobroslaw_ has joined #openstack-infra | 05:29 | |
| *** robcresswell has joined #openstack-infra | 05:29 | |
| *** briancurtin_ has joined #openstack-infra | 05:29 | |
| *** ameade has joined #openstack-infra | 05:29 | |
| *** simondodsley_ has joined #openstack-infra | 05:29 | |
| *** mnasiadka_ has joined #openstack-infra | 05:29 | |
| *** auggy_ has joined #openstack-infra | 05:29 | |
| *** jpmaxman__ has joined #openstack-infra | 05:29 | |
| *** isviridov_away has quit IRC | 05:29 | |
| *** johnsom has quit IRC | 05:29 | |
| *** mnasiadka has quit IRC | 05:29 | |
| *** patricku has quit IRC | 05:29 | |
| *** NobodyCam has quit IRC | 05:29 | |
| *** beisner has quit IRC | 05:29 | |
| *** chrisyang_0660_ has joined #openstack-infra | 05:29 | |
| *** NobodyCam_ has joined #openstack-infra | 05:29 | |
| *** mnasiadka_ is now known as mnasiadka | 05:29 | |
| *** jamespage has joined #openstack-infra | 05:29 | |
| *** NobodyCam_ is now known as NobodyCam | 05:29 | |
| *** fdegir has joined #openstack-infra | 05:29 | |
| *** beisner_ has joined #openstack-infra | 05:29 | |
| *** DuncanT_ has joined #openstack-infra | 05:30 | |
| *** pas-ha has joined #openstack-infra | 05:30 | |
| *** LinuxJedi_ has joined #openstack-infra | 05:30 | |
| *** coreycb has joined #openstack-infra | 05:30 | |
| *** mgkwill_ has joined #openstack-infra | 05:30 | |
| *** davidlenwell_ has joined #openstack-infra | 05:30 | |
| *** Dmitrii-Sh_ has joined #openstack-infra | 05:30 | |
| *** mwhahaha_ has joined #openstack-infra | 05:30 | |
| *** serverascode_ has joined #openstack-infra | 05:30 | |
| *** dham1__ has joined #openstack-infra | 05:30 | |
| *** lxkong_ has joined #openstack-infra | 05:30 | |
| *** bauzas has quit IRC | 05:30 | |
| *** Dobroslaw has quit IRC | 05:30 | |
| *** serverascode has quit IRC | 05:30 | |
| *** LinuxJedi has quit IRC | 05:30 | |
| *** mwhahaha has quit IRC | 05:30 | |
| *** auggy has quit IRC | 05:30 | |
| *** jpmaxman_ has quit IRC | 05:30 | |
| *** dham1 has quit IRC | 05:30 | |
| *** Dmitrii-Sh has quit IRC | 05:30 | |
| *** chrisyang_0660 has quit IRC | 05:30 | |
| *** portdirect has quit IRC | 05:30 | |
| *** mgkwill has quit IRC | 05:30 | |
| *** Roamer` has quit IRC | 05:30 | |
| *** portdirect_ has joined #openstack-infra | 05:30 | |
| *** johnsom has joined #openstack-infra | 05:30 | |
| *** chrisyang_0660_ is now known as chrisyang_0660 | 05:30 | |
| *** Dobroslaw_ is now known as Dobroslaw | 05:30 | |
| *** auggy_ is now known as auggy | 05:30 | |
| *** jpmaxman__ is now known as jpmaxman_ | 05:30 | |
| *** LinuxJedi_ is now known as LinuxJedi | 05:30 | |
| *** mwhahaha_ is now known as mwhahaha | 05:30 | |
| *** portdirect_ is now known as portdirect | 05:30 | |
| *** dham1__ is now known as dham1 | 05:30 | |
| *** mgkwill_ is now known as mgkwill | 05:30 | |
| *** Dmitrii-Sh_ is now known as Dmitrii-Sh | 05:30 | |
| *** patricku has joined #openstack-infra | 05:30 | |
| *** greghaynes has quit IRC | 05:31 | |
| *** davidlenwell has quit IRC | 05:31 | |
| *** vaidy has quit IRC | 05:31 | |
| *** hamzy has quit IRC | 05:31 | |
| *** ihar has quit IRC | 05:31 | |
| *** lxkong has quit IRC | 05:31 | |
| *** aspiers has quit IRC | 05:31 | |
| *** simondodsley has quit IRC | 05:31 | |
| *** DuncanT has quit IRC | 05:31 | |
| *** briancurtin has quit IRC | 05:31 | |
| *** jmccrory has quit IRC | 05:31 | |
| *** jamielennox has quit IRC | 05:31 | |
| *** davidlenwell_ is now known as davidlenwell | 05:31 | |
| *** lxkong_ is now known as lxkong | 05:31 | |
| *** briancurtin_ is now known as briancurtin | 05:31 | |
| *** patricku has quit IRC | 05:31 | |
| *** patricku has joined #openstack-infra | 05:31 | |
| *** Roamer` has joined #openstack-infra | 05:32 | |
| *** ihar has joined #openstack-infra | 05:32 | |
| *** portdirect is now known as Guest72952 | 05:32 | |
| *** jmccrory has joined #openstack-infra | 05:35 | |
| *** bobh has joined #openstack-infra | 05:35 | |
| *** bauzas has joined #openstack-infra | 05:35 | |
| *** aspiers has joined #openstack-infra | 05:35 | |
| *** hamzy has joined #openstack-infra | 05:36 | |
| ianw | #status log grafana.o.o switched to new grafana02.o.o | 05:37 |
| openstackstatus | ianw: finished logging | 05:37 |
| ianw | anything old was cleared out, so it's all fresh from puppet. already a few versions behind, so will cleanup old stuff and we can probably directly bump | 05:38 |
| *** efried has joined #openstack-infra | 05:38 | |
| *** dpawlik has joined #openstack-infra | 05:38 | |
| *** bobh has quit IRC | 05:39 | |
| *** dpawlik has quit IRC | 05:43 | |
| *** greghaynes has joined #openstack-infra | 05:44 | |
| *** bobh has joined #openstack-infra | 05:45 | |
| *** threestrands has quit IRC | 05:49 | |
| *** bobh has quit IRC | 05:49 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add Zuul jobs for Airship specs repo https://review.openstack.org/581470 | 05:50 |
| *** pcichy has quit IRC | 05:50 | |
| *** cshastri has joined #openstack-infra | 05:50 | |
| *** LinuxJedi has quit IRC | 05:53 | |
| *** LinuxJedi has joined #openstack-infra | 05:53 | |
| *** janki has joined #openstack-infra | 05:56 | |
| *** felipemonteiro has quit IRC | 05:56 | |
| *** bobh has joined #openstack-infra | 05:57 | |
| *** End_ has joined #openstack-infra | 05:59 | |
| *** markvoelker has quit IRC | 06:01 | |
| *** bobh has quit IRC | 06:01 | |
| *** dpawlik has joined #openstack-infra | 06:02 | |
| *** aojea has joined #openstack-infra | 06:03 | |
| *** bobh has joined #openstack-infra | 06:03 | |
| *** hashar has joined #openstack-infra | 06:03 | |
| *** dpawlik has quit IRC | 06:05 | |
| *** dpawlik has joined #openstack-infra | 06:05 | |
| *** lpetrut has joined #openstack-infra | 06:05 | |
| *** bobh has quit IRC | 06:07 | |
| *** threestrands has joined #openstack-infra | 06:09 | |
| *** threestrands has quit IRC | 06:09 | |
| *** threestrands has joined #openstack-infra | 06:09 | |
| *** vaidy has joined #openstack-infra | 06:09 | |
| *** isviridov_away has joined #openstack-infra | 06:10 | |
| *** efried has quit IRC | 06:11 | |
| *** jamielennox has joined #openstack-infra | 06:11 | |
| *** efried has joined #openstack-infra | 06:11 | |
| *** bobh has joined #openstack-infra | 06:15 | |
| *** lpetrut has quit IRC | 06:16 | |
| *** lpetrut has joined #openstack-infra | 06:16 | |
| *** annp has quit IRC | 06:17 | |
| *** agopi has joined #openstack-infra | 06:17 | |
| *** bobh has quit IRC | 06:19 | |
| *** dhajare has joined #openstack-infra | 06:19 | |
| *** e0ne has joined #openstack-infra | 06:20 | |
| *** aeng has quit IRC | 06:21 | |
| *** udesale_ has joined #openstack-infra | 06:23 | |
| *** bobh has joined #openstack-infra | 06:25 | |
| *** udesale has quit IRC | 06:25 | |
| evrardjp | hello -- is it possible to have more complex acls for gerrit's label-Workflow than just -1..+1 group project-name-core ? Like require 2 +2 Code-Review before allowing +W except for PTL? | 06:26 |
| openstackgerrit | Merged openstack-infra/project-config master: Gate governance with project-config https://review.openstack.org/583654 | 06:28 |
| *** bobh has quit IRC | 06:29 | |
| *** vivsoni_ has quit IRC | 06:35 | |
| *** bobh has joined #openstack-infra | 06:36 | |
| *** threestrands has quit IRC | 06:36 | |
| *** pcaruana has joined #openstack-infra | 06:37 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add openstack-dev/bashate to #openstack-qa channel https://review.openstack.org/583812 | 06:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Drop legacy scenario multinode job from Tempest https://review.openstack.org/583802 | 06:39 |
| *** ramishra has quit IRC | 06:40 | |
| *** bobh has quit IRC | 06:40 | |
| *** ramishra has joined #openstack-infra | 06:41 | |
| *** pguimaraes has joined #openstack-infra | 06:50 | |
| *** udesale__ has joined #openstack-infra | 06:51 | |
| *** End_ has quit IRC | 06:52 | |
| *** bobh has joined #openstack-infra | 06:54 | |
| *** udesale_ has quit IRC | 06:54 | |
| *** kjackal has quit IRC | 06:56 | |
| *** pguimaraes has quit IRC | 06:56 | |
| *** dhajare has quit IRC | 06:58 | |
| *** markvoelker has joined #openstack-infra | 06:58 | |
| *** mrsoul_ is now known as mschuppert | 06:59 | |
| *** bobh has quit IRC | 06:59 | |
| *** peereb has joined #openstack-infra | 07:02 | |
| *** openstackgerrit has quit IRC | 07:04 | |
| *** dbecker has joined #openstack-infra | 07:09 | |
| *** ramishra has quit IRC | 07:09 | |
| *** ramishra has joined #openstack-infra | 07:11 | |
| *** openstackgerrit has joined #openstack-infra | 07:11 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/project-config master: Add tempest-scenario-job to Neutron failure rate dashboard https://review.openstack.org/583870 | 07:11 |
| *** shardy has joined #openstack-infra | 07:13 | |
| *** shardy has quit IRC | 07:13 | |
| *** shardy has joined #openstack-infra | 07:13 | |
| *** bobh has joined #openstack-infra | 07:20 | |
| *** gfidente has joined #openstack-infra | 07:21 | |
| *** gfidente has joined #openstack-infra | 07:21 | |
| *** bobh has quit IRC | 07:22 | |
| *** bobh has joined #openstack-infra | 07:23 | |
| *** pguimaraes has joined #openstack-infra | 07:23 | |
| *** kjackal has joined #openstack-infra | 07:24 | |
| *** ykarel is now known as ykarel|lunch | 07:27 | |
| *** janki has quit IRC | 07:29 | |
| *** rcernin has quit IRC | 07:29 | |
| *** zhangfei has quit IRC | 07:30 | |
| *** bobh has quit IRC | 07:31 | |
| *** markvoelker has quit IRC | 07:32 | |
| *** bobh has joined #openstack-infra | 07:35 | |
| *** tosky has joined #openstack-infra | 07:37 | |
| *** annp has joined #openstack-infra | 07:40 | |
| *** bobh has quit IRC | 07:40 | |
| *** lpetrut has quit IRC | 07:40 | |
| *** dhajare has joined #openstack-infra | 07:40 | |
| *** florianf has joined #openstack-infra | 07:42 | |
| *** zhangfei has joined #openstack-infra | 07:44 | |
| *** janki has joined #openstack-infra | 07:44 | |
| *** bobh has joined #openstack-infra | 07:54 | |
| *** mkoderer__ has joined #openstack-infra | 07:57 | |
| *** mkoderer__ is now known as mkoderer | 07:57 | |
| *** bobh has quit IRC | 07:58 | |
| *** pguimaraes has quit IRC | 07:59 | |
| *** pguimaraes has joined #openstack-infra | 08:00 | |
| slaweq | andreaf: hi | 08:02 |
| slaweq | andreaf: can You maybe help me with rewriting of multinode job to zuulv3 syntax? | 08:02 |
| slaweq | I'm trying to do it in https://review.openstack.org/#/c/578796/ | 08:03 |
| slaweq | but it looks that OpenStack services are not started on compute1 node and I don't know why | 08:03 |
| *** bobh has joined #openstack-infra | 08:05 | |
| *** dtantsur|afk is now known as dtantsur | 08:07 | |
| *** bobh has quit IRC | 08:09 | |
| *** alexchadin has joined #openstack-infra | 08:10 | |
| *** roman_g has joined #openstack-infra | 08:12 | |
| *** bobh has joined #openstack-infra | 08:18 | |
| *** bobh has quit IRC | 08:23 | |
| *** derekh has joined #openstack-infra | 08:23 | |
| *** dtantsur is now known as dtantsur|bbl | 08:24 | |
| *** stakeda has quit IRC | 08:24 | |
| *** bobh has joined #openstack-infra | 08:25 | |
| *** bobh has quit IRC | 08:29 | |
| *** markvoelker has joined #openstack-infra | 08:29 | |
| *** ykarel|lunch is now known as ykarel | 08:30 | |
| *** bobh has joined #openstack-infra | 08:38 | |
| *** tesseract has joined #openstack-infra | 08:41 | |
| *** bobh has quit IRC | 08:42 | |
| *** bobh has joined #openstack-infra | 08:48 | |
| *** viks_ has joined #openstack-infra | 08:50 | |
| *** electrofelix has joined #openstack-infra | 08:51 | |
| *** pbourke has quit IRC | 08:52 | |
| *** bobh has quit IRC | 08:53 | |
| *** pbourke has joined #openstack-infra | 08:53 | |
| *** udesale_ has joined #openstack-infra | 08:54 | |
| *** bobh has joined #openstack-infra | 08:56 | |
| *** udesale__ has quit IRC | 08:57 | |
| *** lpetrut has joined #openstack-infra | 08:57 | |
| *** bobh has quit IRC | 09:00 | |
| *** markvoelker has quit IRC | 09:01 | |
| *** ijw has joined #openstack-infra | 09:04 | |
| *** bobh has joined #openstack-infra | 09:08 | |
| *** panda|off is now known as panda | 09:08 | |
| *** ijw has quit IRC | 09:09 | |
| *** s-shiono has quit IRC | 09:12 | |
| *** bobh has quit IRC | 09:12 | |
| *** derekh has quit IRC | 09:14 | |
| *** bobh has joined #openstack-infra | 09:15 | |
| *** sambetts|afk is now known as sambetts | 09:15 | |
| *** dingyichen has quit IRC | 09:16 | |
| *** udesale__ has joined #openstack-infra | 09:17 | |
| *** derekh has joined #openstack-infra | 09:17 | |
| *** lpetrut has quit IRC | 09:18 | |
| *** lpetrut has joined #openstack-infra | 09:18 | |
| *** bobh has quit IRC | 09:20 | |
| *** udesale_ has quit IRC | 09:20 | |
| *** ijw has joined #openstack-infra | 09:27 | |
| *** bobh has joined #openstack-infra | 09:29 | |
| *** ijw has quit IRC | 09:32 | |
| *** bobh has quit IRC | 09:33 | |
| *** beisner_ is now known as beisner | 09:34 | |
| *** afazekas|pto is now known as afazekas | 09:39 | |
| *** bobh has joined #openstack-infra | 09:46 | |
| *** alexchad_ has joined #openstack-infra | 09:46 | |
| *** shardy has quit IRC | 09:46 | |
| *** alexchadin has quit IRC | 09:47 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add tenant yaml validation option to zuul client https://review.openstack.org/574265 | 09:50 |
| *** bobh has quit IRC | 09:51 | |
| *** bobh has joined #openstack-infra | 09:53 | |
| *** bobh has quit IRC | 09:57 | |
| *** markvoelker has joined #openstack-infra | 09:59 | |
| *** honza_ is now known as honza | 10:01 | |
| *** alex_xu has quit IRC | 10:03 | |
| *** ijw has joined #openstack-infra | 10:03 | |
| *** peereb has quit IRC | 10:06 | |
| *** bobh has joined #openstack-infra | 10:08 | |
| *** alex_xu has joined #openstack-infra | 10:08 | |
| *** ijw has quit IRC | 10:08 | |
| *** bobh has quit IRC | 10:12 | |
| *** bobh has joined #openstack-infra | 10:15 | |
| *** dhill_ has quit IRC | 10:17 | |
| *** v1a4 has joined #openstack-infra | 10:17 | |
| *** bobh has quit IRC | 10:19 | |
| *** Bhujay has quit IRC | 10:22 | |
| *** dpawlik has quit IRC | 10:24 | |
| *** dpawlik has joined #openstack-infra | 10:27 | |
| *** dpawlik has quit IRC | 10:27 | |
| *** ijw has joined #openstack-infra | 10:27 | |
| *** bobh has joined #openstack-infra | 10:30 | |
| *** alexchad_ has quit IRC | 10:30 | |
| *** alexchadin has joined #openstack-infra | 10:31 | |
| *** ijw has quit IRC | 10:31 | |
| *** Bhujay has joined #openstack-infra | 10:32 | |
| *** markvoelker has quit IRC | 10:32 | |
| *** jamesmcarthur has joined #openstack-infra | 10:34 | |
| *** dpawlik has joined #openstack-infra | 10:35 | |
| *** bobh has quit IRC | 10:35 | |
| *** alexchadin has quit IRC | 10:35 | |
| *** bobh has joined #openstack-infra | 10:36 | |
| *** mkoderer has quit IRC | 10:37 | |
| *** jamesmcarthur has quit IRC | 10:39 | |
| *** alexchadin has joined #openstack-infra | 10:40 | |
| *** bobh has quit IRC | 10:40 | |
| *** ramishra has quit IRC | 10:40 | |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: packaging: Remove support for pyN requirement files https://review.openstack.org/580417 | 10:42 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: builddoc: Remove support for Sphinx < 1.6 https://review.openstack.org/580416 | 10:42 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: tox: Re-add cover target https://review.openstack.org/583918 | 10:42 |
| *** bobh has joined #openstack-infra | 10:49 | |
| *** links has quit IRC | 10:49 | |
| *** boden has joined #openstack-infra | 10:52 | |
| *** bobh has quit IRC | 10:53 | |
| *** dhill_ has joined #openstack-infra | 10:58 | |
| *** kjackal has quit IRC | 10:59 | |
| *** bobh has joined #openstack-infra | 10:59 | |
| *** bobh has quit IRC | 11:03 | |
| *** rosmaita has joined #openstack-infra | 11:08 | |
| *** links has joined #openstack-infra | 11:12 | |
| *** bobh has joined #openstack-infra | 11:15 | |
| *** graphene has quit IRC | 11:19 | |
| *** bobh has quit IRC | 11:20 | |
| *** graphene has joined #openstack-infra | 11:21 | |
| *** quiquell is now known as quiquell|lunch | 11:21 | |
| *** zhangfei has quit IRC | 11:22 | |
| *** udesale__ has quit IRC | 11:28 | |
| *** bobh has joined #openstack-infra | 11:29 | |
| *** markvoelker has joined #openstack-infra | 11:30 | |
| *** agopi is now known as agopi|brb | 11:32 | |
| *** bobh has quit IRC | 11:33 | |
| *** bobh has joined #openstack-infra | 11:35 | |
| chkumar|ruck | tristanC: http://logs.openstack.org/24/567224/63/check/tripleo-ci-centos-7-undercloud-upgrades/bf44bda/job-output.txt.gz#_2018-07-19_10_22_49_165911 ssh error has returned again | 11:35 |
| *** alexchad_ has joined #openstack-infra | 11:36 | |
| *** rh-jelabarre has joined #openstack-infra | 11:37 | |
| *** alexchadin has quit IRC | 11:37 | |
| *** bobh has quit IRC | 11:39 | |
| tristanC | chkumar|ruck: not sure what's going on, this is openstack-infra zuul/nodepool | 11:43 |
| tristanC | perhaps there is still zookeeper issue, but i don't have access to those logs | 11:43 |
| *** bobh has joined #openstack-infra | 11:45 | |
| *** pcichy has joined #openstack-infra | 11:46 | |
| *** bobh has quit IRC | 11:49 | |
| *** shardy has joined #openstack-infra | 11:50 | |
| *** pguimaraes has quit IRC | 11:51 | |
| pabelanger | chkumar|ruck: tristanC: not zookeeper, you can look at http://grafana.openstack.org/d/rZtIH5Imz/nodepool for some clues | 11:56 |
| pabelanger | if was zookeeper, we'd see all nodes spike a DELETE | 11:57 |
| *** bobh has joined #openstack-infra | 11:57 | |
| pabelanger | in this case, I think is is just networking issue from zuul-executor (rackspace) to nodepool nodes | 11:57 |
| pabelanger | eg | 11:57 |
| pabelanger | 2018-07-19 11:31:05,817 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000823482 (state: used, allocated_to: 200-0005085012) | 11:57 |
| pabelanger | is a node in http://logs.openstack.org/38/583238/3/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/4ee7819/job-output.txt.gz | 11:57 |
| pabelanger | which nodepool deletes after the job has finished | 11:58 |
| pabelanger | meaning the SSH failures in the job-output seem unrelated to nodepool this time | 11:58 |
| *** kjackal has joined #openstack-infra | 11:58 | |
| pabelanger | tristanC: chkumar|ruck: I think the [primary] Waiting on logger are also indicators there is something going on with network, because zuul_stream is not working at that point too | 11:59 |
| *** dave-mccowan has joined #openstack-infra | 11:59 | |
| *** bobh has quit IRC | 12:01 | |
| *** dtantsur|bbl is now known as dtantsur | 12:02 | |
| *** markvoelker has quit IRC | 12:02 | |
| *** markvoelker has joined #openstack-infra | 12:02 | |
| *** agopi|brb has quit IRC | 12:02 | |
| *** trown|outtypewww is now known as trown | 12:03 | |
| *** alexchad_ has quit IRC | 12:04 | |
| *** bobh has joined #openstack-infra | 12:06 | |
| *** pguimaraes has joined #openstack-infra | 12:07 | |
| *** bobh has quit IRC | 12:10 | |
| *** bobh has joined #openstack-infra | 12:13 | |
| *** tpsilva has joined #openstack-infra | 12:15 | |
| *** bobh has quit IRC | 12:18 | |
| *** armax has joined #openstack-infra | 12:22 | |
| *** armax has quit IRC | 12:26 | |
| *** dpawlik has quit IRC | 12:27 | |
| dtroyer | Can I get a second look at https://review.openstack.org/#/c/579303/? (add zuul job for stx-docs repo) | 12:34 |
| *** ssbarnea1 has quit IRC | 12:36 | |
| *** neiloy has joined #openstack-infra | 12:38 | |
| chkumar|ruck | is https://logs.openstack.org/ working? | 12:39 |
| *** ssbarnea has joined #openstack-infra | 12:39 | |
| TheJulia | I'm wondering the same thing | 12:39 |
| *** v1a4 has quit IRC | 12:40 | |
| *** rlandy has joined #openstack-infra | 12:40 | |
| *** eharney has quit IRC | 12:40 | |
| *** Guest72952 is now known as portdirect | 12:41 | |
| *** edmondsw has joined #openstack-infra | 12:41 | |
| *** bobh has joined #openstack-infra | 12:42 | |
| *** alexchadin has joined #openstack-infra | 12:43 | |
| TheJulia | looks like redfish based raid creation didn't make it into proliantutils unless they are going to try for the last minute. | 12:45 |
| *** quiquell|lunch is now known as quiquell | 12:45 | |
| TheJulia | doh, wrong channel | 12:45 |
| *** bobh has quit IRC | 12:46 | |
| mnasiadka | About redfish - is that supported out of the box in Bifrost? Saw some TODO lines about that :-) | 12:50 |
| pabelanger | infra-root: chkumar|ruck: I too am not able to access logs.o.o | 12:51 |
| pabelanger | ping works, but no SSH again | 12:51 |
| mnasiadka | Opps, errchan - sorry | 12:52 |
| pabelanger | #status alert logs.openstack.org is offline, causing POST_FAILURE results from Zuul. Cause and resolution timeframe currently unknown. | 12:53 |
| openstackstatus | pabelanger: sending alert | 12:53 |
| *** mriedem has joined #openstack-infra | 12:53 | |
| *** bobh has joined #openstack-infra | 12:55 | |
| *** ramishra has joined #openstack-infra | 12:55 | |
| -openstackstatus- NOTICE: logs.openstack.org is offline, causing POST_FAILURE results from Zuul. Cause and resolution timeframe currently unknown. | 12:56 | |
| *** ChanServ changes topic to "logs.openstack.org is offline, causing POST_FAILURE results from Zuul. Cause and resolution timeframe currently unknown." | 12:56 | |
| *** sthussey has joined #openstack-infra | 12:59 | |
| openstackstatus | pabelanger: finished sending alert | 13:00 |
| *** bobh has quit IRC | 13:00 | |
| *** jamesmcarthur_ has joined #openstack-infra | 13:02 | |
| pabelanger | infra-root: I don't see any tickets in rackspace about volume maintenance, I'm leaning towards a reboot via UI to see if server comeback online, maybe even emergency mode first | 13:02 |
| *** jamesmcarthur has joined #openstack-infra | 13:03 | |
| *** alexchadin has quit IRC | 13:04 | |
| *** abhishekk has joined #openstack-infra | 13:05 | |
| pabelanger | okay, trying to enter rescue mode now | 13:07 |
| *** jpmaxman_ is now known as JpMaxMan | 13:09 | |
| *** jtomasek has joined #openstack-infra | 13:13 | |
| *** pcichy has quit IRC | 13:14 | |
| *** bobh has joined #openstack-infra | 13:15 | |
| *** pcichy has joined #openstack-infra | 13:15 | |
| *** agopi has joined #openstack-infra | 13:15 | |
| tosky | shouldn't the icon of the openstackinfra twitter account use the current OpenStack logo? | 13:15 |
| *** alexchadin has joined #openstack-infra | 13:16 | |
| *** pcichy has quit IRC | 13:17 | |
| pabelanger | infra-root: I don't see any kernel errors in syslog from static.o.o, was able to mount /dev/xvdb1 in rescue mode. I don't think we lost any data on log volumes | 13:17 |
| *** zul has joined #openstack-infra | 13:17 | |
| *** udesale has joined #openstack-infra | 13:18 | |
| *** bobh has quit IRC | 13:20 | |
| pabelanger | infra-root: So far, it looks like possible same issue corvus see on Friday. Unless somebody else would like to look, maybe we reboot and hope fsck isn't needed? | 13:23 |
| *** felipemonteiro has joined #openstack-infra | 13:26 | |
| pabelanger | okay, going to reboot and hope logs.o.o comes back online. Not much more I can look at in logs, and see nothing in kern.log or syslog | 13:27 |
| pabelanger | exiting rescue mode now | 13:28 |
| pabelanger | infra-root: I can SSH into logs.o.o again | 13:29 |
| *** eharney has joined #openstack-infra | 13:29 | |
| pabelanger | assuming we passed fsck, looking into ARA issue that happened on friday | 13:30 |
| cmurphy | \o/ | 13:30 |
| pabelanger | POST_FAILURE have stopped, and can confirm uploads working again | 13:34 |
| *** felipemonteiro has quit IRC | 13:35 | |
| pabelanger | #status ok logs.openstack.org is back on-line. Changes with "POST_FAILURE" job results should be rechecked. | 13:36 |
| openstackstatus | pabelanger: sending ok | 13:36 |
| mordred | pabelanger: yay! my morning of sleeping in seems to have worked out :) | 13:36 |
| pabelanger | +1 | 13:37 |
| pabelanger | yah, I don't know what happen, but maybe once corvus is online, we can validate if same ARA issue or not | 13:37 |
| pabelanger | could also be cloud thing, but didn't see anything in rackspace tickets | 13:38 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 13:39 | |
| -openstackstatus- NOTICE: logs.openstack.org is back on-line. Changes with "POST_FAILURE" job results should be rechecked. | 13:39 | |
| pabelanger | https://rackspace.service-now.com/system_status/?id=service_status&service=88da0af0db6cf200e93ff2e9af961995 about the only thing I see but looks limited to ORD | 13:39 |
| *** dpawlik has joined #openstack-infra | 13:40 | |
| *** e0ne has quit IRC | 13:42 | |
| openstackstatus | pabelanger: finished sending ok | 13:43 |
| *** dpawlik has quit IRC | 13:44 | |
| *** jamesmcarthur has quit IRC | 13:46 | |
| *** jamesmcarthur has joined #openstack-infra | 13:46 | |
| *** r-daneel has joined #openstack-infra | 13:47 | |
| *** annp_ has joined #openstack-infra | 13:48 | |
| *** ccamacho has quit IRC | 13:49 | |
| *** ccamacho1 has joined #openstack-infra | 13:49 | |
| *** pcichy has joined #openstack-infra | 13:50 | |
| *** jamesmcarthur has quit IRC | 13:50 | |
| *** dpawlik has joined #openstack-infra | 13:55 | |
| *** dpawlik has quit IRC | 13:55 | |
| *** dpawlik has joined #openstack-infra | 13:56 | |
| *** felipemonteiro has joined #openstack-infra | 13:58 | |
| *** gfidente has quit IRC | 14:00 | |
| *** gfidente has joined #openstack-infra | 14:01 | |
| *** gfidente has joined #openstack-infra | 14:01 | |
| *** r-daneel_ has joined #openstack-infra | 14:03 | |
| *** esarault has joined #openstack-infra | 14:05 | |
| *** r-daneel has quit IRC | 14:05 | |
| *** r-daneel_ is now known as r-daneel | 14:05 | |
| *** eernst has joined #openstack-infra | 14:06 | |
| *** jamesmcarthur has joined #openstack-infra | 14:06 | |
| *** cshastri has quit IRC | 14:12 | |
| *** felipemonteiro__ has joined #openstack-infra | 14:15 | |
| *** janki has quit IRC | 14:15 | |
| *** felipemonteiro has quit IRC | 14:16 | |
| *** janki has joined #openstack-infra | 14:16 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:16 | |
| *** armax has joined #openstack-infra | 14:19 | |
| *** felipemonteiro__ has quit IRC | 14:20 | |
| *** e0ne has joined #openstack-infra | 14:21 | |
| corvus | pabelanger: i don't know what happened friday (i only suspect ara because i saw a memory usage spike, and i was able to cause a single ara to use 3g of ram for a single request). i don't see a spike on the graphs this time, but there's a 5m window, so that's not conclusive. | 14:21 |
| *** hongbin_ has joined #openstack-infra | 14:22 | |
| pabelanger | corvus: okay, thanks for info. I too do not know what happen this morning | 14:22 |
| * cmurphy suspects gremlins | 14:22 | |
| corvus | gremlins is not on netflix :( | 14:23 |
| *** armax has quit IRC | 14:24 | |
| mordred | corvus: that's super sad! | 14:26 |
| mordred | corvus: although according to clarkb Jurrasic Park now is | 14:26 |
| corvus | it is! i checked last night. i watched the whole thing just to be sure. | 14:26 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-reviewday master: Don't create reviewday subdirectory https://review.openstack.org/582524 | 14:26 |
| mordred | corvus: and the whole thing is there? | 14:30 |
| corvus | mordred: yes, and it does indeed hold up well. i can confirm everything clarkb said. but don't take my word for it. | 14:31 |
| mordred | corvus: I'm skeptical. I believe I need to verify the claim | 14:32 |
| *** quiquell is now known as quiquell|off | 14:32 | |
| mordred | corvus: in fact, I'm pretty sure I need to verify that claim also while checking if it's still good at high volume | 14:32 |
| mordred | corvus: the real jurassic park used to be a really good way to show off a sound system - so I think that's a good way to check authenticity | 14:33 |
| mordred | corvus: https://review.openstack.org/#/c/583800/ <-- wanna push that in? | 14:35 |
| *** eernst has quit IRC | 14:38 | |
| *** dpawlik has quit IRC | 14:40 | |
| *** arxcruz has quit IRC | 14:41 | |
| *** dpawlik has joined #openstack-infra | 14:42 | |
| *** arxcruz has joined #openstack-infra | 14:44 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Add a dequeue command to zuul client https://review.openstack.org/95035 | 14:45 |
| *** r-daneel has quit IRC | 14:47 | |
| *** armax has joined #openstack-infra | 14:49 | |
| *** zhongjun__ has joined #openstack-infra | 14:49 | |
| *** armax has quit IRC | 14:53 | |
| *** janki has quit IRC | 14:55 | |
| *** r-daneel has joined #openstack-infra | 14:55 | |
| *** abhishekk has quit IRC | 14:59 | |
| auristor | clarkb: are you still looking for a fast way to test an empty afs volume? | 14:59 |
| *** armax has joined #openstack-infra | 15:00 | |
| *** annp_ has quit IRC | 15:01 | |
| *** dtantsur is now known as dtantsur|afk | 15:02 | |
| *** lbragstad has joined #openstack-infra | 15:03 | |
| *** esarault_ has joined #openstack-infra | 15:03 | |
| *** ykarel is now known as ykarel|away | 15:04 | |
| *** Bhujay has quit IRC | 15:05 | |
| openstackgerrit | Merged openstack-infra/system-config master: Set vexxhost auth_type to password https://review.openstack.org/583800 | 15:06 |
| *** e0ne has quit IRC | 15:06 | |
| *** esarault has quit IRC | 15:07 | |
| *** thrash has joined #openstack-infra | 15:09 | |
| thrash | Need some help figuring out why a Depends-On doesn't seem to be working. | 15:09 |
| *** graphene has quit IRC | 15:09 | |
| *** alexchadin has quit IRC | 15:09 | |
| pabelanger | thrash: which review are you looking at? | 15:10 |
| thrash | https://review.openstack.org/#/c/462056/ | 15:10 |
| *** alexchadin has joined #openstack-infra | 15:10 | |
| *** ccamacho1 has quit IRC | 15:10 | |
| thrash | pabelanger: it wouldn't be because it's a W-1 would it? | 15:10 |
| pabelanger | thrash: and where are you looking to see depends-on not working? | 15:10 |
| *** alexchadin has quit IRC | 15:10 | |
| pabelanger | no, w-1 should be okay | 15:11 |
| *** graphene has joined #openstack-infra | 15:11 | |
| *** alexchadin has joined #openstack-infra | 15:11 | |
| thrash | pabelanger: let me get a link to the log... | 15:11 |
| *** alexchadin has quit IRC | 15:11 | |
| thrash | pabelanger: http://logs.openstack.org/56/462056/17/check/mistral-devstack/f8ab99b/controller/logs/tempest_log.txt | 15:11 |
| *** alexchadin has joined #openstack-infra | 15:11 | |
| thrash | If you look for 'unexpected keyword argument' | 15:12 |
| thrash | pabelanger: That's what the depended-on patch provides... | 15:12 |
| *** alexchadin has quit IRC | 15:12 | |
| *** alexchadin has joined #openstack-infra | 15:12 | |
| *** pcaruana has quit IRC | 15:12 | |
| pabelanger | thrash: okay, zuul is setting up the proper dependency, you can see that by looking at the inventory file: http://logs.openstack.org/56/462056/17/check/mistral-devstack/f8ab99b/zuul-info/inventory.yaml Let me look at logs to see why it wasn't picked up | 15:12 |
| *** alexchadin has quit IRC | 15:13 | |
| thrash | pabelanger: that's what is maddening about it. :) | 15:13 |
| pabelanger | thrash: I can see mistral-lib getting installed via constraints: http://logs.openstack.org/56/462056/17/check/mistral-devstack/f8ab99b/controller/logs/devstacklog.txt.gz#_2018-07-18_18_40_23_513 | 15:15 |
| pabelanger | but doesn't seem to happy from local repo on dist | 15:15 |
| pabelanger | I'm guessing devstack isn't setup properly? Trying to see where job is defined | 15:17 |
| thrash | pabelanger: do we perhaps need to have mistral-lib as a proper devstack plugin? | 15:17 |
| pabelanger | not sure, since neutron-lib is working, we can use it to compare with | 15:19 |
| thrash | pabelanger: ack. I'll take a look there. | 15:19 |
| pabelanger | thrash: http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/neutron#n397 is what neutron-lib does, we need to make sure mistral does the same I believe | 15:20 |
| thrash | pabelanger: ack. I was wondering that... Thanks for the pointer. Let me go forth and... do. things. | 15:21 |
| pabelanger | thrash: yah, I think if you update: http://git.openstack.org/cgit/openstack/mistral/tree/devstack/plugin.sh#n116 or add a section to install_mistral_lib, you'll be on the right track | 15:22 |
| thrash | pabelanger: groovy | 15:22 |
| pabelanger | good luck! | 15:22 |
| *** dklyle has joined #openstack-infra | 15:23 | |
| *** sshnaidm|rover is now known as sshnaidm|afk | 15:24 | |
| *** alexchadin has joined #openstack-infra | 15:26 | |
| *** r-daneel has quit IRC | 15:31 | |
| *** njohnston_ has joined #openstack-infra | 15:33 | |
| *** njohnston_ has quit IRC | 15:33 | |
| clarkb | auristor: maybe? I think I've managed to convince myself that mirroring pypi is not really practical anymore given its growth rate | 15:39 |
| clarkb | mordred: I highly recommend verifying the Jurassic Park claims | 15:40 |
| auristor | my offer to openstack.org for a zero cost license to AuriStorFS still stands. AuriStorFS customers regularly replicate volumes between 50TB and 250TB. | 15:40 |
| auristor | anyway, my suggestion for testing empty afs volumes is to rename the old one out of the way and create a new volume with the original name. | 15:41 |
| *** ykarel|away has quit IRC | 15:41 | |
| clarkb | auristor: good to know, one thing I do want to do is see if anyone else is attemptign to address these pypi mirroring problems and what they have done (could learn something new) | 15:42 |
| clarkb | pabelanger: corvus re logs.o.o and ara we could switch back to generating the files again | 15:42 |
| *** alex_xu has quit IRC | 15:42 | |
| auristor | with regards to netflix. the availability of titles is by region. the licensed titles are not consistent across countries. | 15:43 |
| clarkb | auristor: ya, and they actively combat VPNs and services like HE's ipv6 tunnels (my excuse for not setting that up) | 15:45 |
| *** rpittau_ has joined #openstack-infra | 15:45 | |
| auristor | Jurassic Park is available in the U.S. via Netflix. Its also on Amazon Prime and Vudu. | 15:45 |
| *** udesale has quit IRC | 15:46 | |
| auristor | You can setup a HE tunnel and then blackhole the IPv6 subnets used by Netflix. Netflix will then use IPv4 | 15:46 |
| auristor | Same for Pandora | 15:46 |
| mordred | yah. that's what I did with my he tunnel | 15:46 |
| *** mnencia has quit IRC | 15:47 | |
| *** ramishra has quit IRC | 15:47 | |
| corvus | i have to run an errand, biab. | 15:47 |
| clarkb | corvus: when you get back let me know if you think we should consider ara precompiled html on current log server | 15:47 |
| *** alex_xu has joined #openstack-infra | 15:47 | |
| clarkb | I'm going to follow up on packethost port leakages now to see if it is getting worse and then if cmurphy is around we can try puppet future parser | 15:48 |
| *** rpittau has quit IRC | 15:48 | |
| rcarrillocruz | erm, do we use packet for nodepool ? | 15:49 |
| cmurphy | clarkb: i'm semi around but will be more around in about an hour | 15:49 |
| *** mnencia has joined #openstack-infra | 15:49 | |
| clarkb | rcarrillocruz: not directly, we have had a combo of platform9, packethost, and john studarus provide an openstack cloud running on top of packethost baremetal | 15:49 |
| auristor | Gremlins is available in the U.S. via Amazon Prime and Vudu | 15:49 |
| clarkb | cmurphy: an hour from now works for me | 15:49 |
| rcarrillocruz | ah ok, that makes sense, to my knowledge packet has its own api, is not openstack | 15:50 |
| clarkb | rcarrillocruz: correct | 15:50 |
| rcarrillocruz | cool, curious, i have some folks in my team playing with it for testing stuff | 15:50 |
| clarkb | we had ~24 ports yesterday that I thought we had leaked after I cleaned up obviously leaked ports. We now have 33 that appear to have been leaked. I'll go through them and see if I notice anything new | 15:50 |
| *** gfidente has quit IRC | 15:53 | |
| pabelanger | clarkb: another option is another server, but just for ara wsgi. However, I am guessing effort maybe better spend working on swift publisher | 15:56 |
| *** dpawlik has quit IRC | 15:56 | |
| *** gfidente has joined #openstack-infra | 15:57 | |
| *** gfidente has joined #openstack-infra | 15:57 | |
| *** gfidente^2nd has joined #openstack-infra | 15:57 | |
| *** felipemonteiro_ has quit IRC | 15:57 | |
| clarkb | pabelanger: ya switching to prebuilt ara on existing log server would be quick change | 15:58 |
| *** felipemonteiro_ has joined #openstack-infra | 15:58 | |
| pabelanger | clarkb: Yup, I'd be okay going back, if we think inodes are okay | 15:58 |
| *** Bhujay has joined #openstack-infra | 15:59 | |
| *** diablo_rojo has joined #openstack-infra | 16:03 | |
| *** gfidente^2nd has quit IRC | 16:05 | |
| *** alexchadin has quit IRC | 16:07 | |
| *** r-daneel has joined #openstack-infra | 16:07 | |
| *** diablo_rojo has quit IRC | 16:08 | |
| *** jamesmcarthur_ has quit IRC | 16:11 | |
| *** shardy has quit IRC | 16:11 | |
| *** sshnaidm|afk is now known as sshnaidm|rover | 16:11 | |
| *** felipemonteiro_ has quit IRC | 16:13 | |
| clarkb | I cleaned up any ports that persisted from yesterday. Got us down to 9 leaked ports | 16:19 |
| *** r-daneel_ has joined #openstack-infra | 16:19 | |
| clarkb | there is definitely a drop in boot errors. I'm assuming that is due to the increase in port quota so we aren't failing for that reason anymore | 16:20 |
| *** r-daneel has quit IRC | 16:20 | |
| *** r-daneel_ is now known as r-daneel | 16:20 | |
| *** janki has joined #openstack-infra | 16:20 | |
| *** agopi is now known as agopi|food | 16:21 | |
| clarkb | anyone know why port show -c id -c created_at works but port list -c id -c created_at does not? | 16:21 |
| clarkb | I guess the aggregate listing filters are either contrained in the info they can show due to apis or are buggy? | 16:22 |
| *** aojea has quit IRC | 16:22 | |
| clarkb | mordred: ^ that might be something you know about | 16:22 |
| *** agopi|food is now known as agopi | 16:25 | |
| *** ihar has quit IRC | 16:32 | |
| *** masayukig has quit IRC | 16:33 | |
| boden | hi. I'm looking for info on the proper process unoffical projects use to release. IIUC that publishing an unofficial release is done by "tagging a release" (https://docs.openstack.org/infra/manual/creators.html#tagging-a-release) and that in turn will trigger the necessary "publish" jobs (assuming they are in your pipeline) to handle creating the branch, publishing to pypi, etc... Is this the correct process? … I asked in | 16:33 |
| boden | openstack-requirements but was suggested to check here | 16:33 |
| clarkb | boden: yup that is correct | 16:33 |
| *** masayukig has joined #openstack-infra | 16:34 | |
| clarkb | assuming a python project you will need the jobs in the release pipeline that publish to pypi and your pypi project will need to have our user as an admin/owner/whateveritiscalled | 16:34 |
| pabelanger | infra-root: I posted http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-July/000508.html to zuul-discuss on Tuesday, but also think we could maybe try the refactor of base job in openstack-infra too. Happy to discuss more if people are interested, I can also just push up patches and ask for review too :) We've gone live with the change in rdoproject today, with no breakages to jobs | 16:34 |
| clarkb | then you push a signed git tag to gerrit, this triggers the publish job and things get published | 16:34 |
| *** gfidente is now known as gfidente|afk | 16:34 | |
| boden | clarkb is this all part of the publish-to-pypi-neutron template, or are there other templates that should be setup for my project? | 16:36 |
| corvus | clarkb: i think since we're already working on the logs-in-swift we give it one more chance before we switch ara generation | 16:37 |
| clarkb | boden: that runs release-openstack-python which appears to make tarballs, publish them to our tarballs server and to pypi. Should be what you need | 16:38 |
| clarkb | corvus: ok | 16:38 |
| *** dbecker has quit IRC | 16:39 | |
| boden | clarkb thanks… 2 more clarifications: (a) is there a way I can test this out, it clearly hasn’t been working for vmware-nsx since we haven’t published to pypi in yrs… (b) should we also be releasing our tempest repo, or is that not necessary? | 16:40 |
| clarkb | boden: you can make a pre release tag (like a rc or beta tag) it should go through all of the same process but pip won't installed those packages by default when they are on pypi | 16:41 |
| clarkb | boden: I'm not sure if the qa team is recommending you release your tempest repo or not | 16:41 |
| boden | clarkb OK many thanks! | 16:42 |
| *** agopi is now known as agopi|food | 16:44 | |
| *** SpamapS has joined #openstack-infra | 16:44 | |
| *** hashar is now known as hasharAway | 16:51 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Clarify Zuul Ram and HDD usage graphs https://review.openstack.org/583791 | 16:53 |
| *** dhajare has quit IRC | 16:56 | |
| *** tesseract has quit IRC | 16:56 | |
| *** thrash has left #openstack-infra | 16:57 | |
| *** owalsh is now known as owalsh_biab | 17:02 | |
| *** links has quit IRC | 17:03 | |
| clarkb | infra-root can cmurphy and I get a second set of eyeballs on https://review.openstack.org/#/c/582765/ mordred in particular probably groks this better than most | 17:04 |
| clarkb | tl;dr is adding review-dev to future parser group so that we can test puppet4 parser on review-dev | 17:04 |
| *** derekh has quit IRC | 17:04 | |
| cmurphy | clarkb: i'm here now | 17:06 |
| *** bdodd has quit IRC | 17:06 | |
| clarkb | cmurphy: were there other changes you wanted to get in before we update the group for review-dev.openstack.org? I don't see any depends on but wasn't sure if we need other fixes before we expect this to work | 17:08 |
| *** trown is now known as trown|lunch | 17:08 | |
| cmurphy | clarkb: i think that one is pretty independent, i didn't find any issues with the gerrit module or system-config that i would expect to be a problem | 17:08 |
| cmurphy | i think it shouldn't be affected by any of the templates in https://review.openstack.org/#/c/581993/ | 17:09 |
| clarkb | ya shouldn't be | 17:10 |
| cmurphy | and this one https://review.openstack.org/#/c/581596/ will only affect the puppet 4 upgrade not the parser change | 17:10 |
| clarkb | ok, lets see if we can get another infra-root to look at it really quickly just to make sure we havne't missed anything obvious with how ansible and puppet groupings work | 17:10 |
| cmurphy | ++ | 17:10 |
| pabelanger | +2, asked question about epel, but if you want to +A can | 17:12 |
| *** psachin` has quit IRC | 17:13 | |
| *** edmondsw has quit IRC | 17:13 | |
| cmurphy | oh wait i don't know why that's there | 17:14 |
| cmurphy | that might be an accidental git add | 17:14 |
| *** Bhujay has quit IRC | 17:14 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Use the future parser for review-dev.o.o https://review.openstack.org/582765 | 17:15 |
| *** jamesmcarthur has quit IRC | 17:16 | |
| cmurphy | pabelanger: that should have been part of https://review.openstack.org/#/c/580985 which does have a note about it, it was needed for exim | 17:17 |
| pabelanger | ah, cool | 17:17 |
| *** edmondsw has joined #openstack-infra | 17:18 | |
| *** eernst has joined #openstack-infra | 17:19 | |
| *** eernst has quit IRC | 17:32 | |
| clarkb | and thinking out loud here that shouldn't match review.openstack.org because it isn't regex and its longer string anyway | 17:33 |
| *** eernst has joined #openstack-infra | 17:33 | |
| clarkb | should be constrained to review-dev only | 17:33 |
| cmurphy | yeah i think it is safe | 17:33 |
| *** sambetts is now known as sambetts|afk | 17:33 | |
| cmurphy | it also purposely won't match any review-dev01.o.o etc | 17:34 |
| *** pguimaraes has quit IRC | 17:34 | |
| clarkb | oh review-dev.o.o is actually review-dev01 now | 17:35 |
| clarkb | so we might need to update to 01 | 17:35 |
| *** felipemonteiro has joined #openstack-infra | 17:35 | |
| cmurphy | oh hmm | 17:35 |
| *** r-daneel has quit IRC | 17:35 | |
| cmurphy | will ansible not know about it if we call it review-dev? | 17:36 |
| cmurphy | also are there more than one review-dev\d* ? | 17:36 |
| *** agopi|food is now known as agopi | 17:36 | |
| openstackgerrit | Rico Lin proposed openstack-infra/system-config master: Change Owner for TW User Group ML https://review.openstack.org/584035 | 17:36 |
| clarkb | the alias is in DNS only and ansible is looking at nova server listings so I don't think it will see review-dev.o.o | 17:36 |
| clarkb | cmurphy: I don't think there are any other review-dev0* hosts, just 01 | 17:37 |
| cmurphy | if there's only one review-dev\d* i'll just change it to match the node regex in site.pp | 17:37 |
| clarkb | I'll double check a listing from nova myself | 17:37 |
| cmurphy | oh, what does the ~ mean in groups.txt? i just noticed some groups use it and some don't | 17:39 |
| *** dpawlik has joined #openstack-infra | 17:39 | |
| clarkb | cmurphy: that is the prefix for doing a regex rather than just a glob (the default) I don't think you need ~ since you are doing an exact match | 17:39 |
| clarkb | cmurphy: only review-dev01.openstack.org is present in nova list | 17:40 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Use the future parser for review-dev.o.o https://review.openstack.org/582765 | 17:41 |
| cmurphy | clarkb: i just made it match the node in site.pp ^ | 17:41 |
| *** yamahata has joined #openstack-infra | 17:41 | |
| *** jamesmcarthur has joined #openstack-infra | 17:42 | |
| *** jamesmcarthur has quit IRC | 17:42 | |
| boden | clarkb for the “RC or beta” tag you mentioned earlier W/R/T testing the release process.. is that just something like ‘rocky-rc1’ or should I be using a different format? | 17:42 |
| *** jamesmcarthur has joined #openstack-infra | 17:42 | |
| clarkb | boden: it is a pep440 crossed with semver format tag | 17:42 |
| clarkb | boden: let me find the regex for you | 17:43 |
| clarkb | boden: https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/pipelines.yaml#n146 | 17:44 |
| *** felipemo_ has joined #openstack-infra | 17:47 | |
| boden | clarkb, so something like 2018.0rc1 should work IIUC (just quickly looked) | 17:47 |
| clarkb | boden: yes that looks right | 17:48 |
| *** jamesmcarthur has quit IRC | 17:50 | |
| *** jamesmcarthur has joined #openstack-infra | 17:50 | |
| clarkb | pabelanger: care to look at cmurphy's change again? | 17:53 |
| pabelanger | +2, approve when ready | 17:54 |
| *** electrofelix has quit IRC | 17:54 | |
| *** graphene has quit IRC | 17:54 | |
| clarkb | tyty | 17:54 |
| *** dpawlik has quit IRC | 17:56 | |
| *** graphene has joined #openstack-infra | 17:56 | |
| *** gfidente|afk has quit IRC | 17:56 | |
| clarkb | cmurphy: my only concern now is that ansible/puppet will somehow be confused by multiple groups (around hiera maybe), but I'm not sure there is any way to check that without trying first. mordred do you know off the top of your head? | 17:56 |
| *** dpawlik has joined #openstack-infra | 17:57 | |
| clarkb | cmurphy: specifically wondering if we try ot copy hiera data for all groups or if we just try one and are done. | 17:57 |
| *** sshnaidm|rover is now known as sshnaidm|off | 17:57 | |
| *** dpawlik has quit IRC | 17:58 | |
| cmurphy | clarkb: hrm i'm not totally sure | 17:59 |
| corvus | clarkb: if we put logs in swift, we will not have an index page of all builds for a buildset | 17:59 |
| *** jamesmcarthur has quit IRC | 17:59 | |
| clarkb | corvus: ah | 17:59 |
| corvus | clarkb: that is (one of) the reasons i don't think we should change the status url | 17:59 |
| corvus | clarkb: (another is that i'd like to maintain our independence of log configuration from pipeline configuration -- that's what let's us do things like switch log hosting systems on a whim :) | 18:00 |
| *** dpawlik has joined #openstack-infra | 18:00 | |
| *** felipemo_ has quit IRC | 18:01 | |
| cmurphy | clarkb: it looks like the ansible module takes multiple groups http://git.openstack.org/cgit/openstack-infra/ansible-role-puppet/tree/library/puppet_get_hiera_file_list#n54 | 18:01 |
| clarkb | cmurphy: cool in that case I think we can just go for it | 18:01 |
| clarkb | since hiera is really the only thing we are using groups for other than future parser | 18:02 |
| *** janki has quit IRC | 18:03 | |
| *** edmondsw has quit IRC | 18:03 | |
| clarkb | I've approved the cahnge | 18:04 |
| *** armaan has joined #openstack-infra | 18:07 | |
| clarkb | corvus: re swift indexes, swift can autogenerate them too, though they are fairly simple and may not have everything we need | 18:08 |
| *** imacdonn has joined #openstack-infra | 18:08 | |
| clarkb | corvus: you do have to create dummy objects for that path though /foo /foo/bar etc | 18:09 |
| clarkb | and set a metadata flag on them iirc | 18:09 |
| mordred | clarkb: https://review.openstack.org/#/c/584050/ | 18:09 |
| corvus | clarkb: yes. i've spent the past 2 hours trying to do that and failing. i'll get there. | 18:09 |
| clarkb | mordred: if you have a sec https://review.openstack.org/582765 could use more eyeballs as we dive into that. Mostly worried about any unexpected interactions with multiple groups between ansible and puppet | 18:10 |
| *** jamesmcarthur has joined #openstack-infra | 18:10 | |
| mordred | clarkb: I think it shold be fine ... the main thing is that any host can only have one puppet group - but can be in as many ansible groups as possible | 18:11 |
| clarkb | mordred: ya in this case we keep review-dev as the puppet side group, but are adding an additional group that affects how ansible invokes puppet but not puppet state | 18:11 |
| *** florianf has quit IRC | 18:12 | |
| *** trown|lunch is now known as trown | 18:13 | |
| *** r-daneel has joined #openstack-infra | 18:13 | |
| *** dhajare has joined #openstack-infra | 18:14 | |
| mordred | yah | 18:15 |
| clarkb | corvus: mordred for those traversable indexes the big issues people had (sdague mainly) was that you couldn't sort by size and date. I think proper zuul dashboard addresses the by date problem and size was mostly just a hint for "this file is big maybe it has errors in it" whcih isn't the best way to debug | 18:15 |
| clarkb | though swift will show you object size on a real object | 18:15 |
| mordred | yeah - so that should really be fine I think | 18:15 |
| clarkb | just hard to get a sense for dirs with contents above them | 18:15 |
| corvus | clarkb: well, the upload script i'm working from creates indexes | 18:16 |
| corvus | we can continue to do that, and make them look however we want. | 18:16 |
| corvus | there just won't be buildset indexes | 18:16 |
| *** panda is now known as panda|off | 18:20 | |
| imacdonn | Q: say there was a stable branch, and it needed two different fixes to make CI work (say one fix relating to pip 10, and another relating to sphinx requirements)... how would these two fixes get committed, since each depends on the other for the builds to work? | 18:21 |
| clarkb | imacdonn: the two common methods are to either squash the commits and live with that, or disable voting or tests entirely, make changes as necessary then reenable testing | 18:22 |
| imacdonn | hmm, wondering how squashing would be reflected in gerrit | 18:23 |
| cmurphy | imacdonn: we'd have to do it manually and push it to gerrit as one change | 18:24 |
| imacdonn | cmurphy: I wonder if it'd be easier to update your change to incorprate the pip fix ? | 18:24 |
| cmurphy | imacdonn: taking it back to -keystone | 18:27 |
| *** edmondsw has joined #openstack-infra | 18:31 | |
| *** jamesmcarthur has quit IRC | 18:31 | |
| *** jamesmcarthur has joined #openstack-infra | 18:35 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-reviewday master: Don't create reviewday subdirectory https://review.openstack.org/582524 | 18:39 |
| *** graphene has quit IRC | 18:39 | |
| *** akhilaki has joined #openstack-infra | 18:40 | |
| *** graphene has joined #openstack-infra | 18:41 | |
| *** bdodd has joined #openstack-infra | 18:48 | |
| clarkb | mordred: I left a comment on https://review.openstack.org/#/c/584050/1 I probably wouldnt' care except that shade apis tend to be used a lot | 18:48 |
| clarkb | I'm going to take a q uick break now because the future parser group change should go in soon and I want to be ready for that | 18:50 |
| *** esarault_ has quit IRC | 18:51 | |
| *** pcichy has quit IRC | 18:52 | |
| *** graphene has quit IRC | 18:52 | |
| *** graphene has joined #openstack-infra | 18:54 | |
| *** pcichy has joined #openstack-infra | 18:54 | |
| *** dpawlik has quit IRC | 18:56 | |
| openstackgerrit | Merged openstack-infra/system-config master: Use the future parser for review-dev.o.o https://review.openstack.org/582765 | 19:02 |
| cmurphy | clarkb: ^ | 19:02 |
| *** rpioso|afk is now known as rpioso | 19:02 | |
| *** hasharAway is now known as hashar | 19:03 | |
| clarkb | cmurphy: 1915UTC puppet run (in ~9 minutes) should run with that change | 19:06 |
| *** graphene has quit IRC | 19:11 | |
| *** graphene has joined #openstack-infra | 19:13 | |
| *** tosky has quit IRC | 19:16 | |
| *** armaan has quit IRC | 19:19 | |
| clarkb | cmurphy: it is puppeting git servers now, then gerrit then afs, then everything else including the dev gerrit server | 19:19 |
| clarkb | like about 10 minutes before we have results | 19:20 |
| *** Randall_ has joined #openstack-infra | 19:20 | |
| cmurphy | clarkb: o7 | 19:20 |
| clarkb | pabelanger: corvus https://review.openstack.org/#/c/583749/ is a followup cleanup post bandersnatch afs problems | 19:23 |
| *** zhipeng has joined #openstack-infra | 19:25 | |
| *** dbecker has joined #openstack-infra | 19:25 | |
| *** Nil_ has quit IRC | 19:27 | |
| *** Nil_ has joined #openstack-infra | 19:29 | |
| mordred | clarkb: is http://paste.openstack.org/show/726304 a better doc? | 19:34 |
| clarkb | mordred: ya I think that is a bit less magical | 19:36 |
| *** Nil_ has quit IRC | 19:36 | |
| clarkb | cmurphy: bah I had review-dev01 in emergency fiel to test a jeepyb thing :/ | 19:36 |
| clarkb | now I feel special | 19:36 |
| mordred | clarkb: :) | 19:37 |
| clarkb | I'll remove it so the next pass through actually runs puppet | 19:37 |
| *** Nil_ has joined #openstack-infra | 19:37 | |
| cmurphy | clarkb: hahaha | 19:38 |
| cmurphy | i was so pleased that the site still seemed to be up | 19:39 |
| *** lpetrut has quit IRC | 19:39 | |
| clarkb | it will restart at 1947 and then ~25 minutes later apply | 19:39 |
| clarkb | ugh my fault | 19:39 |
| *** eharney has quit IRC | 19:39 | |
| clarkb | mordred: https://review.openstack.org/#/c/566416/2 and its parent is what I was testing a while back fwiw | 19:40 |
| clarkb | mordred: I don't expect I'll page that back in this week but is somethign we should follow up on and get fixed | 19:40 |
| clarkb | I reverted jeepyb back to the way it was on review-dev | 19:42 |
| clarkb | will have to try and look at this next week | 19:42 |
| mordred | clarkb: ++ | 19:46 |
| *** zul has quit IRC | 19:46 | |
| *** e0ne has joined #openstack-infra | 19:50 | |
| pabelanger | clarkb: +3 | 19:52 |
| clarkb | pabelanger: on 583749? | 19:53 |
| pabelanger | yah | 19:53 |
| clarkb | thanks | 19:54 |
| openstackgerrit | Merged openstack-infra/project-config master: Clarify Zuul Ram and HDD usage graphs https://review.openstack.org/583791 | 19:56 |
| *** dhajare has quit IRC | 19:57 | |
| *** e0ne has quit IRC | 19:57 | |
| *** Randall_ has quit IRC | 19:59 | |
| *** myoung is now known as myoung|biab | 20:00 | |
| *** jamesmcarthur has quit IRC | 20:01 | |
| clarkb | cmurphy: http://paste.openstack.org/show/726306/ seems to have functioned | 20:04 |
| clarkb | cmurphy: service is still running. The only thing that looks out of place to me are the init script changes I am not sure why those happened | 20:04 |
| *** armax has quit IRC | 20:05 | |
| clarkb | cmurphy: I'm guessing the deprecation notices imply we ran it with future parser? | 20:05 |
| cmurphy | clarkb: yeah i didn't quite expect that | 20:05 |
| cmurphy | yes re deprecation notices | 20:05 |
| cmurphy | is parser = future in /etc/puppet/puppet.conf? | 20:05 |
| clarkb | cmurphy: parser = future is set in puppet.conf, the other entries are all foo=bar without the extra whitespace, not sure if that matters | 20:05 |
| cmurphy | heh it doesn't matter to puppet | 20:06 |
| cmurphy | yay success \o/ | 20:07 |
| clarkb | seems to be | 20:08 |
| cmurphy | i haven't figured out another safe candidate to switch over, would be good to get some more of the topic:puppet-4 patches merged | 20:08 |
| clarkb | ya I've reviewed the vast majority of them at this point, if we can get another infra-root to make a pass through I'm sure we can find a few more candidates. | 20:09 |
| clarkb | cmurphy: we will need to fix those deprecation notices before updating to puppet 4? | 20:10 |
| clarkb | or is that a fix before puppet 5 deal? | 20:10 |
| cmurphy | clarkb: before puppet 5 | 20:10 |
| clarkb | zuul memory use continues to look ok | 20:14 |
| mordred | clarkb: I redid those sdk patches, renamed the one method, added the better docs - and also added unittests | 20:14 |
| *** dbecker has quit IRC | 20:15 | |
| mriedem | can anyone explain why this didn't go into the gate queue https://review.openstack.org/#/c/582912/ ? | 20:16 |
| mriedem | it passed check | 20:16 |
| mriedem | i rebased it last night | 20:16 |
| mriedem | do i need to sacrifice a chicken? | 20:16 |
| clarkb | mriedem: the depends on is unmerged | 20:16 |
| mriedem | gdi | 20:16 |
| * mriedem looks for the shells | 20:16 | |
| *** gcb has joined #openstack-infra | 20:16 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Add a dequeue command to zuul client https://review.openstack.org/95035 | 20:19 |
| *** zhipeng has quit IRC | 20:21 | |
| *** dpawlik has joined #openstack-infra | 20:23 | |
| boden | is there something in devstack that needs to be updated with respect to devstack-gate “test-matrix” (https://github.com/openstack-infra/devstack-gate/blob/master/roles/test-matrix/library/test_matrix.py) to include ‘rocky’… seeing failures like 2018-07-19 20:05:45,804 ERROR: branch not allowed by features matrix: rocky when trying to land rocky branch related patches.. | 20:24 |
| boden | http://logs.openstack.org/78/584078/1/check/legacy-tempest-dsvm-neutron-src/c88f37e/job-output.txt.gz#_2018-07-19_20_05_45_658667 | 20:24 |
| clarkb | I have learned new things about pypi uploads. PySide2 just pushed to pypi (I noticed that when skimming recent packge updates in bandersnatch logs). Separately I ntoiced a blog post from them where they implied it was not possible for them to publish in the past. Follwing that thread I discover there were package size limits on pypi uploads that have either been lifted entirely or are much larger | 20:24 |
| clarkb | than before | 20:25 |
| clarkb | boden: yes, we'll need to add rocky to the features.yaml file | 20:25 |
| boden | clarkb; should I try do that, or better to have someone from infra? | 20:26 |
| clarkb | boden: should be a straightfoward change since we haven't added and removed default services, you can likely copy the queens entry | 20:26 |
| *** jamesmcarthur has joined #openstack-infra | 20:26 | |
| clarkb | let me check that ther eisn't already a change for this | 20:27 |
| boden | clarkb wouldn’t it just be adding rocky to https://github.com/openstack-infra/devstack-gate/blob/master/roles/test-matrix/files/features.yaml#L69 | 20:27 |
| boden | clarkb NM I see a few other hits | 20:27 |
| clarkb | boden: and to https://github.com/openstack-infra/devstack-gate/blob/master/roles/test-matrix/files/features.yaml#L4 | 20:27 |
| *** dpawlik has quit IRC | 20:27 | |
| boden | ya | 20:27 |
| clarkb | I don't see any existing changes | 20:27 |
| *** jamesmcarthur has quit IRC | 20:28 | |
| openstackgerrit | Merged openstack-infra/system-config master: Disable bandersnatch cron https://review.openstack.org/583749 | 20:28 |
| openstackgerrit | boden proposed openstack-infra/devstack-gate master: add rocky branch to features.yaml https://review.openstack.org/584100 | 20:31 |
| *** akhilaki_ has joined #openstack-infra | 20:31 | |
| boden | clarkb ^ | 20:31 |
| *** jamesmcarthur has joined #openstack-infra | 20:32 | |
| *** akhilaki has quit IRC | 20:33 | |
| clarkb | boden: as a side note the new devstack zuulv3 jobs largely avoid this by being branches in devstack with default service lists built in | 20:34 |
| boden | clarkb so this is a side effect of being a legacy job? | 20:35 |
| clarkb | sort of, its a side effect of needing and using devstack gate since its job was to figure things like this out (what projects to run on various branches), but we've largely removed the need for a special tool to do that and replaced it with more robust job configs | 20:36 |
| *** akhilaki has joined #openstack-infra | 20:37 | |
| openstackgerrit | Jill Rouleau proposed openstack-infra/project-config master: Add Tripleo Ansible repo https://review.openstack.org/583415 | 20:37 |
| boden | I will add a TODO on my list to see if I can migrate that legacy job | 20:37 |
| boden | which will likely result in me bugging this room more :) | 20:38 |
| *** akhilaki_ has quit IRC | 20:39 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: web: add /{tenant}/projects and /{tenant}/project/{project} routes https://review.openstack.org/550979 | 20:40 |
| clarkb | ok followingup with packethost we now appear to have leaked floating ips. I'm going to assume that when we messed with networking it just made some things unhappy and do my best to clean it up and see how we do in the future | 20:42 |
| clarkb | looking at floating ips now | 20:42 |
| pabelanger | clarkb: we still have clean-floating-ips in nodepool, use it downstrem in rdoproject still. Can help | 20:44 |
| clarkb | pabelanger: thanks, I think for first pass I'm going to look at all of them (there aren't too many like 15 that appear leaked) to see if I can make sense of any patterns | 20:46 |
| clarkb | so far most have been created back in june and updated early july (I think that was when networking changes were made?) | 20:47 |
| *** jamesmcarthur has quit IRC | 20:52 | |
| clarkb | the vast majority of these are from last month | 20:53 |
| clarkb | I wonder if they may be the source of leaked ports too | 20:54 |
| clarkb | in any case if I get both sides cleaned up hopefully that results in generally better reliabiltiy and fewer problems | 20:54 |
| *** akhilaki has quit IRC | 20:56 | |
| *** zhipeng has joined #openstack-infra | 20:57 | |
| *** jamesmcarthur has joined #openstack-infra | 20:57 | |
| *** dave-mccowan has quit IRC | 21:00 | |
| *** dave-mcc_ has joined #openstack-infra | 21:00 | |
| *** lifeless has quit IRC | 21:01 | |
| *** akhilaki has joined #openstack-infra | 21:02 | |
| *** dave-mccowan has joined #openstack-infra | 21:02 | |
| *** armaan has joined #openstack-infra | 21:02 | |
| *** zhipeng has quit IRC | 21:02 | |
| corvus | clarkb, mordred: https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/86bbbcfa8ad043109d2d7af530225c72/corvustest-1532033922/b4e3459/ is about 1800 files uploaded in about 30 seconds with... a shocking amount of concurrency (one thread per file). | 21:04 |
| corvus | 117MB | 21:04 |
| clarkb | I suppose since it is entirely io bound that having all the threads is actually a good thing | 21:04 |
| *** dave-mcc_ has quit IRC | 21:04 | |
| *** agopi has quit IRC | 21:05 | |
| corvus | that wasn't so much intentional as i had to comment out the thread-count-max from the earlier version of the script for structural reasons :) | 21:05 |
| corvus | i guess the question is, what do we want to do? | 21:05 |
| mordred | corvus: well ... | 21:05 |
| *** lifeless has joined #openstack-infra | 21:06 | |
| clarkb | we should probably enforce a reasnoable cap due to file ulimits | 21:06 |
| mordred | corvus: if we're running that on the executor, 1800 threads is probably a bad idea .. however, if we started uloading straight from the nodes using tempurl - then the max-out-the-threads part would have zero impact | 21:06 |
| clarkb | maybe 1k threads? | 21:06 |
| clarkb | oh ya running on shared system complicates the file descriptor problem | 21:07 |
| mordred | corvus: also - this has no bearing on the real question - but what are the [ ] on the left? | 21:07 |
| *** trown is now known as trown|outtypewww | 21:07 | |
| *** armax has joined #openstack-infra | 21:08 | |
| *** eharney has joined #openstack-infra | 21:08 | |
| corvus | https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/corvustest-1532034396/b4e3459/ same thing nearly identical performance | 21:08 |
| corvus | mordred: i think they may be missing icon files which i hacked out of the script and haven't fixed up yet | 21:09 |
| mordred | corvus: AH nod | 21:09 |
| corvus | those are zuul_swift_upload generated index files, not swift autogenerated | 21:09 |
| mordred | corvus: how hard would it be to compare performance at different levels of max threading? | 21:10 |
| corvus | pretty easy i reckon. i'll do some quick factors of 10. | 21:10 |
| mordred | corvus: cool. my biggest concern is thread-bombing the executors | 21:10 |
| *** slaweq has quit IRC | 21:10 | |
| clarkb | corvus: one thing I notice is that the htmlified logs don't have the header for selecting severity (maybe that is just work to be done). THe other thing we want to do is make sure the logstash job submission is only for the unprocessed files | 21:11 |
| clarkb | however, your examples are browseable and readable from here whcih is neat | 21:11 |
| corvus | mordred: i think the cpu will be ok (this will be in its own process so we don't have to worry about gil); the io and open files that clarkb mentioned are worrying | 21:11 |
| corvus | clarkb: yeah, apparently severity was implemented server side. so we're not going to get that without reworking osla. | 21:12 |
| corvus | (also, fwiw, whoever added severity broke command-line generation/testing; i have a local patch to fix that) | 21:12 |
| mordred | also - https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/86bbbcfa8ad043109d2d7af530225c72/corvustest-1532033922/b4e3459/controller/logs/devstacklog.txt.html | 21:13 |
| mordred | looks weird | 21:13 |
| mordred | but I'm sure that's all just osla fixes | 21:13 |
| *** armaan has quit IRC | 21:14 | |
| *** jamesmcarthur has quit IRC | 21:14 | |
| corvus | mordred: yeah, that looks like it should be an easy fix | 21:14 |
| mordred | corvus: also, on the list of things to fix ... https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/86bbbcfa8ad043109d2d7af530225c72/corvustest-1532033922/b4e3459/ara-report/static/images/logo-header.svg is showing up with content type text/html | 21:14 |
| mordred | so https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/86bbbcfa8ad043109d2d7af530225c72/corvustest-1532033922/b4e3459/ara-report/ has a weird header | 21:15 |
| mordred | BUT - I don't think that's a deal breaker or anything | 21:15 |
| mordred | the other images are fine - so it seems to just be that one | 21:15 |
| *** myoung|biab is now known as myoung | 21:16 | |
| corvus | mordred, clarkb: https://ethercalc.openstack.org/sa2u6xrz7vek | 21:17 |
| mordred | corvus: golly, even 10 is not bad | 21:19 |
| clarkb | we run ~100 jobs per executor at the same time, if we assume all are running uploads at once we can do ~64 threads/fds each | 21:19 |
| clarkb | 65k is the hard limit on fds right? | 21:19 |
| clarkb | woot no error launch attempts in packethost after my cleanups (been about 30 minutes) | 21:21 |
| corvus | clarkb: well the *system* limit is (much) larger than that; that might be the default hard ulimit? | 21:22 |
| clarkb | ya 1626874 is sysfs reported file-max here | 21:23 |
| corvus | heh, mine is 3264218 ? | 21:24 |
| corvus | my hard ulimit is 1048576 | 21:24 |
| corvus | mordred, clarkb: just got the result for 1 thread: 440s | 21:24 |
| mordred | corvus: that seems unpleasing | 21:25 |
| mordred | corvus: you going up between 10 and 100 by 10s now? | 21:25 |
| corvus | yep, at least till i hit 25s | 21:26 |
| corvus | ok so > 30 doesn't seem to make much of a difference | 21:26 |
| *** slaweq has joined #openstack-infra | 21:29 | |
| corvus | i say we choose between 10, 20, and 24 | 21:29 |
| clarkb | 24 seems to make a noticeable difference, I'd vote for that | 21:30 |
| clarkb | and 24 * 100 is only 2400 | 21:30 |
| *** owalsh_biab is now known as owalsh | 21:31 | |
| clarkb | mordred: shade.exc.OpenStackCloudException: Tried to delete floating ip 147.75.38.232 associated with server 54a17063-17c1-46ad-b84f-6409bb9b3003 but there was an error deleting it. Not deleting server. seeing a fair bit of that. But when I do a floating ip show on that ip it doesn't show up implying that it actually is getting deleted | 21:33 |
| clarkb | mordred: however, we don't have any logs for why or how the floating ip delete failed | 21:34 |
| clarkb | is there a way to get the inner exception from floating ip delete when deleting a server fails? | 21:34 |
| mordred | clarkb: there shouldn't be an inner exception | 21:34 |
| corvus | looks like the upload of that build to logs.o.o took 19 seconds | 21:35 |
| mordred | clarkb: it is possible that thereis more info from the remote error message - which I do believe we'll pick up once we restart nodepool on top of the openstacksdk patches (which picks up better logic for pulling error messages from remote errors) | 21:35 |
| corvus | of course, that was only 300 files instead of 1800 | 21:35 |
| clarkb | mordred: http://paste.openstack.org/show/726310/ is the extent of what we log now | 21:36 |
| *** zaneb has joined #openstack-infra | 21:37 | |
| *** akhilaki_ has joined #openstack-infra | 21:37 | |
| clarkb | mordred: it does seem to generally clean up after that (eg no leaked ports and no leaked fips) but I wonder if sometimes this is the source of our leaks | 21:37 |
| mordred | clarkb: ++ | 21:38 |
| clarkb | also worth noting that osc didn't error deleting any of the leaked fips when I did that | 21:39 |
| *** kjackal has quit IRC | 21:39 | |
| mordred | clarkb: we should track down what's up at some point | 21:39 |
| mordred | corvus: 24 does seem like a good starting point | 21:40 |
| zaneb | did Huang Zhiping really change his name to 'Andriy Shevchenko' in Gerrit, or is something strange going on here? https://review.openstack.org/#/q/owner:huang.zhiping%254099cloud.net+status:open | 21:42 |
| clarkb | mordred: reading through shade, I think _neutron_delete_floating_ip must be excepting OpenStackCloudResourceNotFound: then returning False which short circuits retrying and we raise in delete server | 21:44 |
| clarkb | mordred: which is odd because the check is if the resources is there | 21:44 |
| clarkb | zaneb: it certainly looks like that account has fullname set to andriy and email set to huang | 21:45 |
| clarkb | mordred: oh, actually it couldn't find the resource so returned false all the way up which is interpreted as not being deleted at all | 21:46 |
| clarkb | but if the resource isn't there then it was deleted right? | 21:46 |
| clarkb | mordred: I think it is possibly a race between Getting the state and trying to delete it again when retrying | 21:47 |
| clarkb | mordred: we delete, check if it is there (still there), we delete again, its gone return false, error | 21:47 |
| clarkb | mordred: we probably want to more directly handle failure vs its gone now | 21:47 |
| clarkb | zaneb: that is exactly what the database says is going on | 21:49 |
| *** armax has quit IRC | 21:51 | |
| *** hamzy has quit IRC | 21:53 | |
| *** hamzy has joined #openstack-infra | 21:53 | |
| *** hashar has quit IRC | 21:55 | |
| clarkb | zaneb: google says maybe it is a world cup themed update? I dunno but it appears to have been intentional | 21:56 |
| *** rcernin has joined #openstack-infra | 21:58 | |
| *** boden has quit IRC | 21:59 | |
| * clarkb takes a break. Back in a bit | 22:00 | |
| *** jtomasek has quit IRC | 22:00 | |
| *** zhipeng has joined #openstack-infra | 22:17 | |
| *** slaweq has quit IRC | 22:17 | |
| *** dklyle has quit IRC | 22:18 | |
| *** graphene has quit IRC | 22:18 | |
| *** graphene has joined #openstack-infra | 22:20 | |
| *** harlowja has joined #openstack-infra | 22:21 | |
| *** zhipeng has quit IRC | 22:21 | |
| *** neiloy has quit IRC | 22:27 | |
| *** graphene has quit IRC | 22:29 | |
| *** edmondsw has quit IRC | 22:30 | |
| *** graphene has joined #openstack-infra | 22:31 | |
| *** edmondsw has joined #openstack-infra | 22:31 | |
| *** jamesmcarthur has joined #openstack-infra | 22:32 | |
| *** eernst has quit IRC | 22:32 | |
| corvus | clarkb, mordred: a swift-generated directory listing: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/corvustest-1532039445/ | 22:33 |
| corvus | (having the directory marker there is not ideal) | 22:33 |
| *** eernst has joined #openstack-infra | 22:34 | |
| *** hongbin_ has quit IRC | 22:34 | |
| clarkb | corvus: I want to say there wasa workaround for that. Something like only create a test/ object not a test object too? | 22:35 |
| corvus | clarkb: i'll try that | 22:35 |
| corvus | clarkb: in *swift* we shouldn't create directory marker objects at all. ceph/radosgw appears to require them though. | 22:35 |
| *** edmondsw has quit IRC | 22:35 | |
| corvus | i'm hoping to find a way to make both happy until ceph/radosgw can be updated | 22:36 |
| corvus | here's the same with ceph: https://object-storage-ca-ymq-1.vexxhost.net/v1/86bbbcfa8ad043109d2d7af530225c72/corvustest-1532039661/ | 22:36 |
| corvus | clarkb: ah yep, directory markers with / don't show up as extra entries in auto-listings in swift: http://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/corvustest-1532039862/test/ | 22:38 |
| corvus | ...and that hoses ceph/rados: https://object-storage-ca-ymq-1.vexxhost.net/v1/86bbbcfa8ad043109d2d7af530225c72/corvustest-1532039953/test/ | 22:40 |
| clarkb | :/ | 22:40 |
| *** eharney has quit IRC | 22:40 | |
| *** graphene has quit IRC | 22:40 | |
| corvus | well, creating our own indexes is no big deal, we already have the code | 22:41 |
| corvus | plus we might want to nicify them | 22:41 |
| *** graphene has joined #openstack-infra | 22:42 | |
| *** jamesmcarthur has quit IRC | 22:43 | |
| clarkb | ya its just nice when things work well in consistent ways | 22:44 |
| *** imacdonn has quit IRC | 22:45 | |
| *** imacdonn has joined #openstack-infra | 22:45 | |
| *** rcernin_ has joined #openstack-infra | 22:47 | |
| *** rcernin has quit IRC | 22:49 | |
| *** isq has left #openstack-infra | 22:54 | |
| *** jamesmcarthur has joined #openstack-infra | 22:57 | |
| *** rlandy is now known as rlandy|bbl | 22:59 | |
| *** dklyle has joined #openstack-infra | 23:00 | |
| *** felipemonteiro has quit IRC | 23:00 | |
| *** zhipeng has joined #openstack-infra | 23:08 | |
| corvus | clarkb: http://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/corvustest-1532039862/test/2600-0.txt | 23:08 |
| corvus | clarkb: i zilb.compress()'d that before uploading it and set content-encoding to 'deflate' | 23:08 |
| corvus | clarkb: that should mean we're transferring and storing compressed data, basically transparently to the user | 23:09 |
| clarkb | nice | 23:09 |
| mordred | awesome | 23:09 |
| corvus | (it turns out requests doesn't do this automatically, we have to do it ourselves) | 23:09 |
| clarkb | also are we getting into the publishing business? | 23:09 |
| corvus | some of us have been for a while :) | 23:10 |
| corvus | actually, we use to have war and peace in one of the git repos, but i don't see it on codesearch | 23:11 |
| *** slaweq has joined #openstack-infra | 23:11 | |
| corvus | anyway, i think that's promising enough that i need to figure out a proper way to handle the compression, other than reading everything into memory | 23:11 |
| clarkb | corvus: I think osa may already ave a streaming recipe for gzip | 23:12 |
| clarkb | er osla | 23:12 |
| clarkb | zlib.compressobj() maybe | 23:13 |
| *** jamesmcarthur has quit IRC | 23:13 | |
| corvus | 'git grep gzip' inside of osla is one of the reasons i don't like tests in the package () | 23:13 |
| corvus | clarkb: ah yes, that would do it. it looked way too complicated for me to figure out, but if someone has already figured that out | 23:13 |
| corvus | oh, i don't see that in osla | 23:13 |
| *** dhill_ has quit IRC | 23:14 | |
| *** jamesmcarthur has joined #openstack-infra | 23:14 | |
| corvus | clarkb: i don't see any compression stuff in osla. it may have been relying on apache? | 23:14 |
| clarkb | oh that may be | 23:14 |
| corvus | maybe i'll just use zlib.compressobj with the defaults :) | 23:15 |
| *** eernst has quit IRC | 23:15 | |
| *** felipemonteiro has joined #openstack-infra | 23:15 | |
| clarkb | corvus: grep for fileinput in osla | 23:15 |
| clarkb | openhook=fileinput.hook_compressed | 23:15 |
| clarkb | you want the opposite I think /me checks if that exists | 23:15 |
| clarkb | doesn't seem to | 23:16 |
| *** slaweq has quit IRC | 23:16 | |
| *** dklyle has quit IRC | 23:17 | |
| *** zhipeng has quit IRC | 23:19 | |
| *** felipemonteiro has quit IRC | 23:20 | |
| *** felipemonteiro has joined #openstack-infra | 23:24 | |
| *** zhipeng has joined #openstack-infra | 23:25 | |
| *** felipemonteiro has quit IRC | 23:30 | |
| *** felipemonteiro has joined #openstack-infra | 23:31 | |
| *** jamesmcarthur has quit IRC | 23:35 | |
| *** felipemonteiro has quit IRC | 23:36 | |
| *** jamesmcarthur has joined #openstack-infra | 23:41 | |
| *** jamesmcarthur has quit IRC | 23:41 | |
| *** mriedem is now known as mriedem_away | 23:45 | |
| *** AhmadM has joined #openstack-infra | 23:47 | |
| *** sthussey has quit IRC | 23:49 | |
| *** rosmaita has quit IRC | 23:50 | |
| *** dklyle has joined #openstack-infra | 23:50 | |
| *** ijw has joined #openstack-infra | 23:50 | |
| *** rpioso is now known as rpioso|afk | 23:51 | |
| *** AhmadM_ has joined #openstack-infra | 23:55 | |
| *** tpsilva has quit IRC | 23:58 | |
| *** akhilaki__ has joined #openstack-infra | 23:59 | |
| *** dingyichen has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
