| *** jamesmcarthur has quit IRC | 00:00 | |
| *** jamesmcarthur has joined #openstack-infra | 00:00 | |
| fungi | it's much appreciated--thanks again | 00:10 |
|---|---|---|
| mordred | donnyd: so - I'm just the dumb cloud consumer dude. (or at least I play one on TV) - from my POV as a pure consumer, I have the use-cases of "I want to connect this VM to the internet and do not need to route the packets through a route/firewall" and "I have a VM that I don't want directly connected to the internet and want to to put it behind a router or a dmz or whatnot" | 00:15 |
| mordred | completely ignoring the underlying phsyical realities, I'd prefer it if I could pick those based on usecase ... but it seems like frequently the choice presented to the end-user is more about a manifestation of the topology the provider is running with | 00:16 |
| mordred | of course... | 00:16 |
| mordred | I also believe in understanding how the physical structure works ... so mostly I'm full of crap on most days :) | 00:17 |
| * mordred likes to demand unreasonable magic, and then tell other people that the unreasonable magic they demand is unreasonable | 00:17 | |
| mordred | fungi, corvus, clarkb: https://review.opendev.org/#/q/topic:gerrit-images+(status:open+OR+status:merged) is ready for all of your reviewing | 00:19 |
| *** jamesmcarthur has quit IRC | 00:43 | |
| *** jamesmcarthur has joined #openstack-infra | 00:44 | |
| *** jamesmcarthur has quit IRC | 00:49 | |
| *** jamesmcarthur has joined #openstack-infra | 00:53 | |
| *** armax has quit IRC | 00:55 | |
| *** jamesmcarthur has quit IRC | 01:15 | |
| *** igordc has quit IRC | 01:24 | |
| donnyd | mordred: | 01:30 |
| donnyd | mordred: So I am thinking that the way neutron RBAC's work, that is exactly what you get, however you have to pick an overlay. When I say overlay, I mean vlan or vxlan, or gre, etc. When you mix them you get stuck in a place where you can't have your cake and eat it too. | 01:32 |
| donnyd | So use case one is I just want a direct public ip address and use case 2 is I want provider to furnish a router and firewall for me. Using RBAC's both are possible on the same L2 network. It can appear external to one customer, and direct to another | 01:34 |
| *** jamesmcarthur has joined #openstack-infra | 01:43 | |
| *** bobh has quit IRC | 01:45 | |
| donnyd | clarkb: fungi if either of you when you get around tomorrow could update the dns records for the FN mirror, I would greatly appreciate it | 01:47 |
| donnyd | 2001:470:e045:2:f816:3eff:feb2:e3fb | 01:48 |
| donnyd | 108.44.198.37 | 01:48 |
| donnyd | mirror02.regionone.fortnebula.opendev.org | 01:48 |
| donnyd | Could we spin up multiple mirrors once the other hypervisors are online? Maybe just seeing if that could decrease build times... | 01:48 |
| donnyd | also it doesn't look like the mirror is actually responding on http | 01:52 |
| donnyd | https://usercontent.irccloud-cdn.com/file/QVE27XpI/Screenshot%20from%202019-07-28%2021-53-34.png | 01:54 |
| donnyd | Seems like it responds to icmp on both v4 and v6 | 01:54 |
| *** rfolco|rover has joined #openstack-infra | 02:08 | |
| clarkb | ansible needs to hit it and configure the httpp server | 02:12 |
| clarkb | that is probably waiting on an inventory update | 02:12 |
| *** bhavikdbavishi has joined #openstack-infra | 02:12 | |
| *** bobh has joined #openstack-infra | 02:13 | |
| *** rfolco|rover has quit IRC | 02:16 | |
| *** goldyfruit has joined #openstack-infra | 02:16 | |
| *** yamamoto has joined #openstack-infra | 02:18 | |
| *** bhavikdbavishi has quit IRC | 02:21 | |
| *** eharney has quit IRC | 02:23 | |
| *** goldyfruit has quit IRC | 02:24 | |
| *** markvoelker has joined #openstack-infra | 02:29 | |
| *** bobh has quit IRC | 02:38 | |
| *** bobh has joined #openstack-infra | 02:39 | |
| *** bobh has quit IRC | 02:39 | |
| *** bobh has joined #openstack-infra | 02:40 | |
| *** bobh has quit IRC | 02:45 | |
| *** rcernin has quit IRC | 02:45 | |
| *** rcernin has joined #openstack-infra | 02:47 | |
| *** bobh has joined #openstack-infra | 02:59 | |
| *** bobh has quit IRC | 03:00 | |
| *** bobh has joined #openstack-infra | 03:01 | |
| *** bobh has quit IRC | 03:01 | |
| *** bobh has joined #openstack-infra | 03:02 | |
| *** bobh has quit IRC | 03:03 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:06 | |
| *** ricolin_ has joined #openstack-infra | 03:06 | |
| *** yamamoto has quit IRC | 03:19 | |
| *** jamesmcarthur has quit IRC | 03:21 | |
| *** jamesmcarthur has joined #openstack-infra | 03:22 | |
| *** jamesmcarthur has quit IRC | 03:27 | |
| *** yamamoto has joined #openstack-infra | 03:29 | |
| *** psachin has joined #openstack-infra | 03:31 | |
| *** ykarel|away has quit IRC | 03:33 | |
| *** ykarel|away has joined #openstack-infra | 03:33 | |
| *** psachin has quit IRC | 03:34 | |
| *** psachin has joined #openstack-infra | 03:37 | |
| *** ykarel|away has quit IRC | 03:37 | |
| *** ykarel|away has joined #openstack-infra | 03:38 | |
| *** jamesmcarthur has joined #openstack-infra | 03:39 | |
| *** psachin has quit IRC | 03:55 | |
| *** jamesmcarthur has quit IRC | 03:58 | |
| *** jamesmcarthur has joined #openstack-infra | 03:59 | |
| *** udesale has joined #openstack-infra | 03:59 | |
| *** jamesmcarthur has quit IRC | 04:04 | |
| *** jamesmcarthur has joined #openstack-infra | 04:07 | |
| *** ykarel|away has quit IRC | 04:17 | |
| *** yamamoto has quit IRC | 04:29 | |
| *** ykarel|away has joined #openstack-infra | 04:37 | |
| *** ramishra has joined #openstack-infra | 04:38 | |
| *** ykarel|away is now known as ykarel | 04:38 | |
| *** jamesmcarthur has quit IRC | 04:40 | |
| *** yamamoto has joined #openstack-infra | 04:41 | |
| *** jamesmcarthur has joined #openstack-infra | 05:07 | |
| *** apetrich has joined #openstack-infra | 05:09 | |
| *** jamesmcarthur has quit IRC | 05:14 | |
| *** whoami-rajat has joined #openstack-infra | 05:22 | |
| *** jaosorior has joined #openstack-infra | 05:26 | |
| *** jamesmcarthur has joined #openstack-infra | 05:40 | |
| *** e0ne has joined #openstack-infra | 05:41 | |
| *** jamesmcarthur has quit IRC | 05:44 | |
| *** raukadah is now known as chandankumar | 05:46 | |
| *** AJaeger_ has quit IRC | 05:58 | |
| *** e0ne has quit IRC | 06:11 | |
| *** AJaeger has joined #openstack-infra | 06:15 | |
| *** kopecmartin|off is now known as kopecmartin | 06:15 | |
| *** jamesmcarthur has joined #openstack-infra | 06:20 | |
| *** jamesmcarthur has quit IRC | 06:25 | |
| *** dpawlik has joined #openstack-infra | 06:27 | |
| *** apetrich has quit IRC | 06:28 | |
| openstackgerrit | Merged openstack/diskimage-builder master: support alternate portage directories https://review.opendev.org/671530 | 06:29 |
| *** piotrowskim has joined #openstack-infra | 06:45 | |
| *** odicha has joined #openstack-infra | 06:47 | |
| *** odicha has quit IRC | 06:51 | |
| *** odicha has joined #openstack-infra | 06:51 | |
| *** slaweq has joined #openstack-infra | 06:58 | |
| *** janki has joined #openstack-infra | 07:02 | |
| *** tesseract has joined #openstack-infra | 07:08 | |
| *** iurygregory has joined #openstack-infra | 07:11 | |
| *** rcernin has quit IRC | 07:11 | |
| *** rpittau|afk is now known as rpittau | 07:11 | |
| *** tosky has joined #openstack-infra | 07:16 | |
| *** jamesmcarthur has joined #openstack-infra | 07:21 | |
| *** jamesmcarthur has quit IRC | 07:25 | |
| *** pgaxatte has joined #openstack-infra | 07:29 | |
| *** jpena|off is now known as jpena | 07:30 | |
| *** pcaruana has joined #openstack-infra | 07:31 | |
| *** ykarel is now known as ykarel|lunch | 07:33 | |
| *** jchhatbar has joined #openstack-infra | 07:34 | |
| *** janki has quit IRC | 07:38 | |
| *** arxcruz|off is now known as arxcruz | 07:38 | |
| *** lucasagomes has joined #openstack-infra | 07:41 | |
| *** apetrich has joined #openstack-infra | 07:45 | |
| *** jchhatbar has quit IRC | 07:46 | |
| *** jchhatbar has joined #openstack-infra | 07:50 | |
| *** lpetrut has joined #openstack-infra | 07:51 | |
| *** jamesmcarthur has joined #openstack-infra | 07:57 | |
| *** jaosorior has quit IRC | 07:59 | |
| *** iurygregory has quit IRC | 08:01 | |
| *** jamesmcarthur has quit IRC | 08:02 | |
| *** ralonsoh has joined #openstack-infra | 08:08 | |
| *** wpp has joined #openstack-infra | 08:19 | |
| *** dchen has quit IRC | 08:21 | |
| *** tkajinam has quit IRC | 08:30 | |
| *** iurygregory has joined #openstack-infra | 08:31 | |
| *** pkopec has joined #openstack-infra | 08:31 | |
| *** yolanda has joined #openstack-infra | 08:42 | |
| *** hwoarang__ has quit IRC | 08:43 | |
| *** hwoarang has joined #openstack-infra | 08:43 | |
| *** janki has joined #openstack-infra | 08:45 | |
| *** jchhatbar has quit IRC | 08:45 | |
| *** Goneri has joined #openstack-infra | 08:45 | |
| *** ykarel|lunch is now known as ykarel | 08:46 | |
| *** jaosorior has joined #openstack-infra | 08:46 | |
| *** panda has quit IRC | 08:47 | |
| *** panda has joined #openstack-infra | 08:48 | |
| *** derekh has joined #openstack-infra | 08:53 | |
| *** lpetrut has quit IRC | 08:58 | |
| *** dtantsur|afk is now known as dtantsur | 08:58 | |
| *** tesseract has quit IRC | 08:58 | |
| *** jamesmcarthur has joined #openstack-infra | 08:59 | |
| *** jamesmcarthur has quit IRC | 09:03 | |
| *** e0ne has joined #openstack-infra | 09:08 | |
| *** jchhatbar has joined #openstack-infra | 09:10 | |
| *** janki has quit IRC | 09:10 | |
| *** ccamacho has joined #openstack-infra | 09:13 | |
| *** trident has quit IRC | 09:16 | |
| *** trident has joined #openstack-infra | 09:17 | |
| *** tesseract has joined #openstack-infra | 09:20 | |
| *** yamamoto has quit IRC | 09:23 | |
| *** bhavikdbavishi has quit IRC | 09:23 | |
| *** altlogbot_1 has quit IRC | 09:24 | |
| *** yamamoto has joined #openstack-infra | 09:24 | |
| *** altlogbot_3 has joined #openstack-infra | 09:25 | |
| *** yamamoto has quit IRC | 09:27 | |
| *** ricolin_ has quit IRC | 09:29 | |
| *** yamamoto has joined #openstack-infra | 09:29 | |
| *** jamesmcarthur has joined #openstack-infra | 09:38 | |
| mordred | donnyd: ah yes - this makes total sense to me | 09:39 |
| *** yamamoto has quit IRC | 09:41 | |
| *** jamesmcarthur has quit IRC | 09:43 | |
| *** yamamoto has joined #openstack-infra | 09:49 | |
| *** Goneri has quit IRC | 09:49 | |
| *** apetrich has quit IRC | 09:50 | |
| *** Goneri has joined #openstack-infra | 10:04 | |
| *** Lucas_Gray has joined #openstack-infra | 10:21 | |
| *** jaosorior has quit IRC | 10:23 | |
| *** kaisers has quit IRC | 10:35 | |
| *** Wryhder has joined #openstack-infra | 10:35 | |
| *** Lucas_Gray has quit IRC | 10:35 | |
| *** Wryhder is now known as Lucas_Gray | 10:36 | |
| *** jamesmcarthur has joined #openstack-infra | 10:39 | |
| *** jamesmcarthur has quit IRC | 10:44 | |
| *** lpetrut has joined #openstack-infra | 11:07 | |
| *** adriancz has joined #openstack-infra | 11:08 | |
| *** jamesmcarthur has joined #openstack-infra | 11:11 | |
| *** Lucas_Gray has quit IRC | 11:15 | |
| *** jamesmcarthur has quit IRC | 11:16 | |
| *** smrcascao has quit IRC | 11:17 | |
| *** Lucas_Gray has joined #openstack-infra | 11:26 | |
| *** udesale has quit IRC | 11:26 | |
| *** jamesmcarthur has joined #openstack-infra | 11:27 | |
| *** Lucas_Gray has quit IRC | 11:28 | |
| *** Lucas_Gray has joined #openstack-infra | 11:33 | |
| *** jaosorior has joined #openstack-infra | 11:34 | |
| *** jamesmcarthur has quit IRC | 11:36 | |
| *** jamesmcarthur has joined #openstack-infra | 11:37 | |
| *** rfolco|rover has joined #openstack-infra | 11:37 | |
| *** rfolco|rover is now known as rfolco|ruck | 11:37 | |
| *** rfolco|ruck is now known as rfolco|rover | 11:39 | |
| *** n-saito has quit IRC | 11:41 | |
| *** derekh has quit IRC | 11:43 | |
| *** jlufr has joined #openstack-infra | 11:44 | |
| *** jpena is now known as jpena|lunch | 11:46 | |
| *** n-saito has joined #openstack-infra | 11:49 | |
| *** eharney has joined #openstack-infra | 12:00 | |
| *** rh-jelabarre has joined #openstack-infra | 12:02 | |
| *** jamesmcarthur has quit IRC | 12:05 | |
| donnyd | clarkb: makes sense | 12:11 |
| *** tdasilva has joined #openstack-infra | 12:11 | |
| *** tdasilva has quit IRC | 12:16 | |
| *** tdasilva has joined #openstack-infra | 12:16 | |
| *** jcoufal has joined #openstack-infra | 12:22 | |
| *** ysastri has joined #openstack-infra | 12:25 | |
| *** apetrich has joined #openstack-infra | 12:27 | |
| *** ricolin_ has joined #openstack-infra | 12:30 | |
| *** jpena|lunch is now known as jpena | 12:33 | |
| *** joeguo has quit IRC | 12:33 | |
| *** ykarel is now known as ykarel|afk | 12:34 | |
| *** n-saito has quit IRC | 12:40 | |
| *** yamamoto has quit IRC | 12:44 | |
| *** yamamoto has joined #openstack-infra | 12:45 | |
| *** rfolco|rover is now known as rfolco|ruck | 12:47 | |
| *** mriedem has joined #openstack-infra | 12:47 | |
| *** jamesmcarthur has joined #openstack-infra | 12:49 | |
| *** ykarel|afk has quit IRC | 12:49 | |
| *** ysastri has quit IRC | 12:54 | |
| *** ekultails has joined #openstack-infra | 13:01 | |
| fungi | yeah, i'll revise my inventory change to add it, just didn't want to do that until i knew what was happening with the ipv6 address | 13:04 |
| *** ramishra has quit IRC | 13:04 | |
| *** jroll has quit IRC | 13:07 | |
| *** jroll has joined #openstack-infra | 13:08 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Replace fortnebula mirror https://review.opendev.org/673189 | 13:09 |
| *** jchhatbar has quit IRC | 13:15 | |
| *** gfidente has joined #openstack-infra | 13:19 | |
| openstackgerrit | Jeremy Stanley proposed opendev/zone-opendev.org master: Replace fortnebula mirror records https://review.opendev.org/673289 | 13:20 |
| openstackgerrit | Jeremy Stanley proposed opendev/zone-opendev.org master: Update IP address for gitea05 https://review.opendev.org/673186 | 13:21 |
| *** ykarel|afk has joined #openstack-infra | 13:22 | |
| *** haleyb|away is now known as haleyb | 13:31 | |
| *** whoami-rajat has quit IRC | 13:42 | |
| *** dpawlik has quit IRC | 13:45 | |
| *** cmorpheus is now known as cmurphy | 13:46 | |
| *** dpawlik has joined #openstack-infra | 13:52 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: fix comments / spelling errors in gentoo element https://review.opendev.org/673295 | 13:55 |
| *** yamamoto has quit IRC | 13:57 | |
| *** dklyle has quit IRC | 14:00 | |
| *** dklyle has joined #openstack-infra | 14:00 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove openstackwatch https://review.opendev.org/505342 | 14:01 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Replace fortnebula mirror records https://review.opendev.org/673289 | 14:05 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Update IP address for gitea05 https://review.opendev.org/673186 | 14:07 |
| *** ykarel|afk is now known as ykarel | 14:08 | |
| *** ramishra has joined #openstack-infra | 14:09 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: use opendev image building system for zuul-operator test https://review.opendev.org/673020 | 14:15 |
| *** Lucas_Gray has quit IRC | 14:17 | |
| *** Lucas_Gray has joined #openstack-infra | 14:17 | |
| *** panda is now known as panda|brb | 14:19 | |
| fungi | mordred: frickler: since you approved the dns half of the gitea05 replacement, mind taking a look at the system-config part for the inventory addition? https://review.opendev.org/673188 | 14:19 |
| fungi | that way i can get that up and configured and we'll only have two more of those to rebuild | 14:20 |
| mordred | I approve of such things | 14:20 |
| *** michael-beaver has joined #openstack-infra | 14:23 | |
| frickler | fungi: +3 | 14:23 |
| fungi | thanks! | 14:23 |
| fungi | i'll get the data copied over to it as soon as ansible configures docker/gitea | 14:24 |
| *** smrcascao has joined #openstack-infra | 14:24 | |
| *** panda|brb is now known as panda | 14:25 | |
| donnyd | how hard would it be to get a more local git mirror at FN? | 14:26 |
| donnyd | or is that included in the mirror node too? | 14:27 |
| *** yamamoto has joined #openstack-infra | 14:27 | |
| clarkb | donnyd: each image we upload includes a copy of active git repos | 14:28 |
| clarkb | then zuul only has to push in the delta to run the jobs | 14:28 |
| fungi | yeah, local as it gets | 14:29 |
| *** smrcascao has quit IRC | 14:30 | |
| donnyd | Oh I c | 14:30 |
| fungi | so when we run a nova job, for example, nodepool boots a node which has (ideally) a <24-hour-old full clone of nova with all branches and tags and then just before the job starts zuul pushes in the missing refs | 14:30 |
| donnyd | yea that makes sense | 14:30 |
| donnyd | It also makes more sense at to why images are uploaded daily too | 14:31 |
| *** dpawlik has quit IRC | 14:31 | |
| fungi | yep, that and also because a lot of jobs upgrade the packages present on the node and so the fresher the image is the less time and network strain we put on package updates | 14:32 |
| *** smrcascao has joined #openstack-infra | 14:34 | |
| *** yamamoto has quit IRC | 14:38 | |
| *** yamamoto has joined #openstack-infra | 14:39 | |
| *** lpetrut has quit IRC | 14:39 | |
| *** smrcascao has quit IRC | 14:40 | |
| openstackgerrit | Paul Belanger proposed openstack/project-config master: Default zuul tenant to use ansible 2.8 for jobs https://review.opendev.org/673304 | 14:40 |
| *** electrofelix has joined #openstack-infra | 14:40 | |
| *** happyhemant has joined #openstack-infra | 14:42 | |
| *** yamamoto has quit IRC | 14:44 | |
| mnaser | jrosser, mordred, corvus, fungi: https://review.opendev.org/673109 just a follow up that we now have zuul clone things and i added a check to make sure that it fails if zuul hasn't cloned the role and we're in gate :) | 14:46 |
| openstackgerrit | Merged opendev/system-config master: Replace fortnebula mirror https://review.opendev.org/673189 | 14:47 |
| fungi | mnaser: awesome! | 14:48 |
| openstackgerrit | Merged opendev/system-config master: Add gitea05 replacement to inventory https://review.opendev.org/673188 | 14:49 |
| *** Lucas_Gray has quit IRC | 14:50 | |
| *** Lucas_Gray has joined #openstack-infra | 14:51 | |
| *** josephrsandoval has joined #openstack-infra | 15:01 | |
| *** bhavikdbavishi has joined #openstack-infra | 15:04 | |
| *** Lucas_Gray has quit IRC | 15:06 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 15:07 | |
| *** Lucas_Gray has joined #openstack-infra | 15:07 | |
| *** jlufr has quit IRC | 15:08 | |
| *** bhavikdbavishi has quit IRC | 15:09 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 15:09 | |
| *** odicha has quit IRC | 15:13 | |
| *** jamesmcarthur has quit IRC | 15:16 | |
| *** jamesmcarthur_ has joined #openstack-infra | 15:16 | |
| mordred | corvus: if you get a sec, https://review.opendev.org/#/c/673147/ is the fix for post_failures on image build failures (want to make sure that approach isn't violating a larger design or anything) | 15:18 |
| mordred | corvus: you can see it properly report failure and not post_failure on https://review.opendev.org/#/c/673142/2 (you have to look at the previous results to see the failure) | 15:19 |
| *** zul has quit IRC | 15:21 | |
| corvus | mordred: +3 thanks! | 15:22 |
| mordred | \o/ | 15:22 |
| *** udesale has joined #openstack-infra | 15:22 | |
| mordred | corvus: also - the image jobs themselves are ready to go | 15:23 |
| *** josephrsandoval has quit IRC | 15:23 | |
| *** Lucas_Gray has quit IRC | 15:26 | |
| *** e0ne has quit IRC | 15:30 | |
| *** pgaxatte has quit IRC | 15:32 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split out bazel builder and gerrit base image https://review.opendev.org/673142 | 15:36 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Separate the Dockerfiles for the bazel builds https://review.opendev.org/673143 | 15:36 |
| *** bhavikdbavishi has quit IRC | 15:37 | |
| corvus | mordred: is 142 still wip or is that final now? | 15:37 |
| mordred | corvus: yeah - that's final | 15:37 |
| openstackgerrit | Merged zuul/zuul-jobs master: Don't try to push images when the build failed https://review.opendev.org/673147 | 15:38 |
| *** gyee has joined #openstack-infra | 15:39 | |
| corvus | mordred: i really like those 2 changes | 15:40 |
| *** Lucas_Gray has joined #openstack-infra | 15:41 | |
| *** udesale has quit IRC | 15:41 | |
| *** udesale has joined #openstack-infra | 15:41 | |
| *** ykarel has quit IRC | 15:42 | |
| *** ykarel has joined #openstack-infra | 15:42 | |
| *** armax has joined #openstack-infra | 15:43 | |
| *** lucasagomes has quit IRC | 15:47 | |
| *** mattw4 has joined #openstack-infra | 15:50 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: use opendev image building system for zuul-operator test https://review.opendev.org/673020 | 15:51 |
| mordred | corvus: yay! | 15:52 |
| mordred | corvus: I thought you'd enjoy those | 15:52 |
| *** derekh has joined #openstack-infra | 15:54 | |
| *** gfidente has quit IRC | 15:57 | |
| *** rfarr has joined #openstack-infra | 15:57 | |
| *** rh-jelabarre has quit IRC | 15:58 | |
| *** ginopc has quit IRC | 16:01 | |
| *** rpittau is now known as rpittau|afk | 16:04 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Use human-readable names for artifact returns https://review.opendev.org/672382 | 16:06 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Remove download_artifact_name parameter https://review.opendev.org/673327 | 16:06 |
| *** mattw4 has quit IRC | 16:12 | |
| *** goldyfruit has joined #openstack-infra | 16:13 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Download-artifact: use the artifact type rather than name https://review.opendev.org/672557 | 16:14 |
| *** tdasilva has quit IRC | 16:14 | |
| smcginnis | Not an infra thing, but does anyone know if openstack.org in general is having DNS or routing issues? | 16:15 |
| *** tdasilva has joined #openstack-infra | 16:15 | |
| smcginnis | Having trouble getting the track chair tool to load and even www.openstack.org was slow to load (though it did eventually). | 16:15 |
| clarkb | jamesmcarthur_: ^ that is a question for you | 16:16 |
| jamesmcarthur_ | one moment... | 16:17 |
| jamesmcarthur_ | smcginnis: give it a try now | 16:18 |
| jamesmcarthur_ | the track chairs tool didn't get updated with the new expiration date | 16:19 |
| smcginnis | jamesmcarthur_: Loaded right up now. | 16:19 |
| jamesmcarthur_ | should be good to go now | 16:19 |
| smcginnis | jamesmcarthur_: Thanks! | 16:19 |
| jamesmcarthur_ | re: general DNS issues on the openstack.org site... something seems up in general on the interwebs today | 16:19 |
| jamesmcarthur_ | Slack, Google Docs have both had outages | 16:19 |
| smcginnis | That's what I was afraid was happening, but at least I can get to the tool now. | 16:20 |
| *** Lucas_Gray has quit IRC | 16:20 | |
| *** rh-jelabarre has joined #openstack-infra | 16:21 | |
| *** diablo_rojo has joined #openstack-infra | 16:23 | |
| clarkb | fungi: donnyd I've accepted the ssh hostkey for mirror02.regioneone.fortnebula.opendev.org on bridge so taht ansible can talk to it now | 16:23 |
| clarkb | hopefully we have an apache running there in about half an hour | 16:23 |
| donnyd | ok thanks clarkb | 16:23 |
| fungi | thanks clarkb! i got sidetracked by meetings there | 16:24 |
| *** Goneri has quit IRC | 16:24 | |
| fungi | gerrit replication to gitea05 is underway. once that's done i just need to git gc everything there and propose the change to bring it into service and take down gitea07 | 16:26 |
| fungi | we're getting close! | 16:26 |
| clarkb | fungi: should we do 07 and 08 together? | 16:26 |
| AJaeger | yeah! thanks, fungi ! | 16:26 |
| openstackgerrit | James E. Blair proposed zuul/zuul-operator master: DNM: test images https://review.opendev.org/673333 | 16:26 |
| clarkb | speed it up by a cycle? | 16:26 |
| fungi | clarkb: we can, sure | 16:26 |
| fungi | seems like they're not heavily loaded | 16:26 |
| clarkb | fungi: http://paste.openstack.org/show/755081/ is the current error on wiki-dev02. I don't understand that puppet error given the info pasted below it | 16:27 |
| clarkb | fungi: I think it helps that we've got better performing nodes now for the other 6 | 16:27 |
| fungi | also no signs of oom on any of the gitea servers since their most recent reboots to deal with the ceph incident on friday | 16:27 |
| fungi | clarkb: did the submodule removal change already merge? | 16:27 |
| *** tesseract has quit IRC | 16:28 | |
| clarkb | fungi: yes, this is the error after that | 16:28 |
| clarkb | fungi: that dir appears to be a normal git repo with that head and remotes | 16:28 |
| clarkb | not sure why puppet thinks it isn't a git repo | 16:28 |
| clarkb | I suppose we could try moving it aside and have puppet reclone it? but that doesn't help us undersatnd what the issue is | 16:29 |
| clarkb | I'm going to get breakfast but then I'll look more closely at the wiki situation | 16:30 |
| fungi | clarkb: look closer at that ls output in your paste | 16:31 |
| fungi | -rw-r--r-- 1 root root 31 Jul 9 19:13 .git | 16:31 |
| fungi | missing a d there | 16:31 |
| fungi | would expect drwxr-xr-x | 16:31 |
| fungi | or is that not how submodules actually work? | 16:32 |
| Shrews | . | 16:32 |
| fungi | … | 16:32 |
| clarkb | oh hrm | 16:35 |
| fungi | that's basically the same problem i was dealing with before | 16:36 |
| mordred | yeah - I mean - if there's a submodule there that's been initied, it'll look like a git repo | 16:36 |
| fungi | vcsrepo thinks it should clone there, but doesn't understand that mediawiki defines it as a submodule | 16:37 |
| clarkb | ya so that also needs to be removed frkm the list of directly managed repos | 16:37 |
| clarkb | I'll work on a chanage after eating | 16:37 |
| *** mattw4 has joined #openstack-infra | 16:40 | |
| *** jpena is now known as jpena|off | 16:42 | |
| *** dtantsur is now known as dtantsur|afk | 16:44 | |
| *** sthussey has joined #openstack-infra | 16:51 | |
| *** udesale has quit IRC | 16:57 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Swap gitea05 into service and bring down 07 and 08 https://review.opendev.org/673338 | 16:57 |
| fungi | git gc on 05 is still underway, but ought to wrap up sooner than that ^ will be able to merge | 16:58 |
| *** igordc has joined #openstack-infra | 16:59 | |
| *** derekh has quit IRC | 17:00 | |
| openstackgerrit | Clark Boylan proposed opendev/puppet-mediawiki master: Stop managing one more mediawiki git submodule https://review.opendev.org/673339 | 17:02 |
| clarkb | infra-root ^ that should fix that git submodule error | 17:02 |
| *** goldyfruit has quit IRC | 17:02 | |
| *** ramishra has quit IRC | 17:03 | |
| *** xek has quit IRC | 17:07 | |
| *** xek has joined #openstack-infra | 17:08 | |
| *** ykarel is now known as ykarel|away | 17:10 | |
| clarkb | fungi: http://paste.openstack.org/show/755083/ do we have to restart the server to get /afs working? | 17:10 |
| clarkb | or modprobe maybe? | 17:10 |
| fungi | oh, yes likely | 17:11 |
| *** happyhemant has quit IRC | 17:12 | |
| fungi | i doubt our ansible does that for us currently | 17:12 |
| clarkb | fungi: should I go ahead and reboot it then? | 17:12 |
| fungi | go ahead, it's not in production/use right now | 17:12 |
| clarkb | /afs has content now post reboot | 17:13 |
| *** e0ne has joined #openstack-infra | 17:14 | |
| fungi | excellent | 17:17 |
| *** e0ne has quit IRC | 17:20 | |
| donnyd | clarkb: I am still not seeing any http server on the FN mirror, I am assuming ansible hasn't run yet | 17:20 |
| *** dpawlik has joined #openstack-infra | 17:22 | |
| *** ricolin_ has quit IRC | 17:24 | |
| *** jamesmcarthur_ has quit IRC | 17:24 | |
| clarkb | donnyd: the earlier run failed due to missing afs mounts, those exist now so next run should set itup | 17:24 |
| fungi | it has to rerun after the reboot clarkb performed to get openafs lkm loaded | 17:24 |
| *** ralonsoh has quit IRC | 17:25 | |
| *** eernst_ has joined #openstack-infra | 17:26 | |
| *** eernst_ has quit IRC | 17:28 | |
| *** eernst has joined #openstack-infra | 17:28 | |
| *** whoami-rajat has joined #openstack-infra | 17:39 | |
| donnyd | is there a way to tell nodepool to boot from volume? I am trying to speed up instance boot times | 17:41 |
| clarkb | yes, we use it with vexxhost. We do have aproblem with leaking images and volumes we havent fully tracked down though | 17:42 |
| *** trident has quit IRC | 17:47 | |
| *** ykarel|away has quit IRC | 17:47 | |
| *** trident has joined #openstack-infra | 17:51 | |
| clarkb | fungi: donnyd http://mirror.regionone.fortnebula.opendev.org/ it lives | 17:57 |
| donnyd | YESSSS!!! | 17:57 |
| clarkb | mordred: corvus frickler have a moment for https://review.opendev.org/#/c/673338/ ? we are almost done with that task. Thank you fungi! | 17:58 |
| donnyd | thank you for getting it back running again | 17:58 |
| clarkb | fungi: hopefully this has been a decent introduction to working with docker too :) | 17:58 |
| *** goldyfruit has joined #openstack-infra | 17:59 | |
| corvus | +3 | 18:00 |
| clarkb | tyty | 18:03 |
| *** electrofelix has quit IRC | 18:06 | |
| fungi | clarkb: yes, i now only mostly don't understand docker ;) | 18:09 |
| fungi | also, the git gc on 05 did complete, so it's entirely ready to serve requests once that change lands | 18:10 |
| fungi | getting 07/08 replaced and filing a cve request are my two main tasks for the afternoon | 18:10 |
| fungi | oh, also i still need to work out why those two vexxhost tenant entries were gumming up playbooks/clouds_layouts.yml | 18:11 |
| clarkb | fungi: I'm not sure I'm familiar with that last item. Are there more details I can read somewhere? | 18:12 |
| *** jamesmcarthur has joined #openstack-infra | 18:12 | |
| fungi | clarkb: starts at http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-07-27.log.html#t2019-07-27T19:12:24 | 18:14 |
| fungi | not much to add other than it was just the generic "needs authentication" exception you get out of openstacksdk when auth fails | 18:15 |
| clarkb | fungi: ya the cloud launcher fails fast so the first problem it encounters causes it to not run anything past there | 18:15 |
| fungi | so either those are old entries we forgot to remove or we missed renaming them or something has ceased working for them | 18:15 |
| clarkb | fungi: in the past it has complained that the hash of the keys in $cloud has changed or something (maybe it is a uuid not a hash?) | 18:15 |
| clarkb | that causes it to try and recreate the keys but they can't have identical names or something | 18:16 |
| fungi | i don't think it was getting that far | 18:16 |
| clarkb | I think we just delete them out of the cloud then cloud launcher creates them again | 18:16 |
| * clarkb looks for logs | 18:16 | |
| fungi | i think it was trying to use the credentials we have on file for those two accounts and failing to authenticate to keystone | 18:16 |
| fungi | they're nodepool accounts not currently in use by nodepool so i suspect some sort of bitrot | 18:17 |
| clarkb | ya I see that now "The error was: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401)" | 18:17 |
| clarkb | we actually do use those accounts with nodepool for the gpu instances | 18:17 |
| fungi | oh, hrm | 18:18 |
| clarkb | whether or not anything consumes those isntances I don't know | 18:18 |
| fungi | well, anyway i can take a look in a bit. i don't think that's urgent since i managed to get it to do a full run on saturday and the data doesn't change often | 18:19 |
| fungi | but we will need it sorted before we update security groups or try to add a new provider | 18:19 |
| clarkb | is it possible that when mnaser said he disabled things that was how he did it? | 18:20 |
| clarkb | back on thursday? whenever the ceph problem happened | 18:20 |
| fungi | oh, mayhaps. could explain why nobody noticed until i was trying to get keypairs into fortnebula on saturday | 18:21 |
| mnaser | hi sorry | 18:21 |
| mnaser | i had disabled the account | 18:21 |
| mnaser | because it was a firedrill | 18:21 |
| mnaser | i will enable it again now | 18:21 |
| fungi | mnaser: no worries! just glad the mystery is solved. thanks! | 18:21 |
| clarkb | mnaser: fwiw if you reduce quota to 0 nodepool should honor that | 18:21 |
| clarkb | (for a more graceful disabling) | 18:22 |
| mnaser | clarkb: yeah but that was before we merged that at the time | 18:22 |
| mnaser | should be ok now, forgort about it | 18:22 |
| clarkb | mnaser: I mean on the cloud side | 18:22 |
| mnaser | enabled openstackjenkins | 18:22 |
| fungi | mnaser: not sure if you were aware but we saw another cephfs problem on friday | 18:22 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: Revert "Disable sjc" https://review.opendev.org/673346 | 18:22 |
| mnaser | there shouldn't have been anything on friday afaik, everything should be humming along well | 18:23 |
| mnaser | ^ so we get more capacity agian | 18:23 |
| fungi | interesting. kernels on guests logged write failures | 18:23 |
| clarkb | I confirm that using osc against vexxhost now works | 18:23 |
| fungi | on multiple guests | 18:23 |
| mnaser | hmm | 18:23 |
| mnaser | that's odd | 18:23 |
| mnaser | i wonder if it happened because of a rebalance | 18:24 |
| fungi | i'll get you the timestamp | 18:24 |
| fungi | [Fri Jul 26 19:38:44 2019] INFO: task jbd2/vda1-8:248 blocked for more than 120 seconds. | 18:25 |
| fungi | mnaser: that was one of them | 18:25 |
| mnaser | weird | 18:25 |
| fungi | we saw it reported on multiple server instances | 18:25 |
| fungi | around the same timeframe | 18:25 |
| fungi | these were on the gitea backends, so sjc1 | 18:26 |
| *** kopecmartin is now known as kopecmartin|off | 18:26 | |
| openstackgerrit | Merged opendev/system-config master: Swap gitea05 into service and bring down 07 and 08 https://review.opendev.org/673338 | 18:34 |
| openstackgerrit | Merged opendev/puppet-mediawiki master: Stop managing one more mediawiki git submodule https://review.opendev.org/673339 | 18:36 |
| *** jamesmcarthur has quit IRC | 18:52 | |
| *** aedc has joined #openstack-infra | 18:53 | |
| openstackgerrit | Merged openstack/project-config master: Revert "Disable sjc" https://review.opendev.org/673346 | 18:55 |
| *** jamesmcarthur has joined #openstack-infra | 19:04 | |
| clarkb | fungi: donnyd anything else we need to do to turn fn back on? | 19:07 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-jobs master: Add auth config to kubelet user for buildset registries https://review.opendev.org/673351 | 19:07 |
| * clarkb waiting patiently for puppet to run on wiki-dev02 | 19:07 | |
| donnyd | fans | 19:08 |
| donnyd | they will be here tomorrow | 19:08 |
| donnyd | but nothing from infra for now clarkb | 19:08 |
| fungi | i was gonna say, you have plenty of fans in here already ;) | 19:08 |
| fungi | thanks again! | 19:08 |
| donnyd | :) | 19:14 |
| openstackgerrit | Jeremy Stanley proposed opendev/zone-opendev.org master: Update IP addresses for gitea07 and gitea08 https://review.opendev.org/673353 | 19:19 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add gitea07 and gitea08 replacements to inventory https://review.opendev.org/673355 | 19:21 |
| fungi | once those merge i can get data onto them | 19:21 |
| *** e0ne has joined #openstack-infra | 19:22 | |
| *** yamamoto has joined #openstack-infra | 19:25 | |
| *** jcoufal_ has joined #openstack-infra | 19:28 | |
| *** jcoufal has quit IRC | 19:31 | |
| *** yamamoto has quit IRC | 19:32 | |
| *** jcoufal_ has quit IRC | 19:34 | |
| *** goldyfruit has quit IRC | 19:37 | |
| *** rascasoft has quit IRC | 19:39 | |
| *** rascasoft has joined #openstack-infra | 19:44 | |
| clarkb | fungi: I think puppeting the wiki dev server is working now | 19:50 |
| fungi | woo! it was that close? | 19:52 |
| clarkb | I guess so. I don't see any errors at least | 19:52 |
| fungi | finally can start testing to see how working the resulting mediawiki install is | 19:52 |
| *** jamesmcarthur has quit IRC | 19:54 | |
| *** jamesmcarthur has joined #openstack-infra | 19:58 | |
| *** jamesmcarthur has quit IRC | 20:03 | |
| *** rfarr has quit IRC | 20:03 | |
| *** rfarr has joined #openstack-infra | 20:04 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-jobs master: Add auth config to kubelet user for buildset registries https://review.opendev.org/673351 | 20:06 |
| *** goldyfruit has joined #openstack-infra | 20:16 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Allow ensure-tox to upgrade tox version https://review.opendev.org/672760 | 20:17 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: use opendev image building system for zuul-operator test https://review.opendev.org/673020 | 20:23 |
| *** jamesmcarthur has joined #openstack-infra | 20:25 | |
| *** jamesmcarthur has quit IRC | 20:31 | |
| *** efried has joined #openstack-infra | 20:34 | |
| efried | o/ | 20:35 |
| efried | could someone please help me bootstrap talking to gerrit via the REST API? | 20:35 |
| efried | I'm led to believe I can start curl'ing some URL at some port, but my initial trial-and-error attempts at figuring out what URL and port are not working. | 20:35 |
| *** slaweq has quit IRC | 20:35 | |
| mnaser | efried: anything in particular you are trying to pull out? | 20:36 |
| efried | mnaser: I want to bot-abandon a bunch of patches | 20:36 |
| mnaser | efried: ah you might be better off using the 'ssh' api imho | 20:37 |
| efried | okay; does that let me use my pubkey to bypass the auth step? | 20:37 |
| clarkb | https://review.opendev.org/Documentation/rest-api.html is a useful set of docs. https://review.opendev.org/#/c/671777/1/tools/gerrit-success-rate.py is a script I wrote recently that uses the api if you want an example | 20:37 |
| mnaser | ssh mnaser@review.opendev.org -p29418 should workl | 20:37 |
| mnaser | though i dont know if the api is open for users too via ssh | 20:38 |
| clarkb | oh ya for doing review vote type things the ssh api is easy | 20:38 |
| mnaser | ah yes | 20:38 |
| clarkb | mnaser: it should be available to everyone | 20:38 |
| mnaser | ssh mnaser@review.opendev.org -p29418 gerrit review --help | 20:38 |
| mnaser | i guess gerrit review 1 2 3 4 5 -- --abandon | 20:38 |
| clarkb | another trick is gertty lets you modify chagnes in bulk too | 20:38 |
| mnaser | you can also add a message, but yeah, that should get ya goin :p | 20:39 |
| efried | mnaser: Yes, I'll want to add a message. This wfm. Thanks! | 20:39 |
| *** whoami-rajat has quit IRC | 20:39 | |
| *** aedc has quit IRC | 20:40 | |
| fungi | efried: the big gotchas with the rest api are you need to go to your preferences in the webui and generate an http password, authenticated operations are under the a/ subtree, and responses get a xss-busting prefix of a bunch of closing brackets and quotes prepended which you need to strip off. or use pygerrit or one of the other libraries out there which handles this stuff for you | 20:40 |
| *** aedc has joined #openstack-infra | 20:40 | |
| efried | cool | 20:40 |
| fungi | but also yes bulk abandon with a message via gertty's process-marking feature also works well (and i've used it a bunch) | 20:40 |
| efried | what I want to do isn't really rocket science, query | xargs abandon, so ssh sounds pretty doable. | 20:41 |
| fungi | yep, the ssh cli is pretty easy for simple shell scripting | 20:42 |
| fungi | ssh -p 29418 efried@review.opendev.org "gerrit help" | 20:43 |
| fungi | ought to get you going | 20:43 |
| clarkb | also https://review.opendev.org/Documentation/cmd-index.html | 20:43 |
| *** xek has quit IRC | 20:43 | |
| fungi | er, not "gerrit help" but rather "gerrit" by itself | 20:43 |
| fungi | (but subcommands are prefixed like "gerrit ...") | 20:44 |
| efried | beaut, I'm off to the races. Thanks mnaser clarkb fungi | 20:44 |
| *** e0ne has quit IRC | 20:46 | |
| *** sean-k-mooney has joined #openstack-infra | 20:48 | |
| *** trident has quit IRC | 20:49 | |
| *** aedc has quit IRC | 20:49 | |
| clarkb | corvus: I rechecked https://review.opendev.org/#/c/671852/ hopefully that gets through soon and we get up to date js in production. While I'm waiting I should review those js updates for logs | 20:50 |
| corvus | efried: btw, gertty can mass-abandon changes | 20:50 |
| efried | corvus: ack | 20:51 |
| corvus | efried: if you can make a query that matches the changes you want, you can hit them all with the 'process mark' then ctrl-a to abandon all the marked changes | 20:51 |
| efried | which presumably lets me enter a message too | 20:51 |
| corvus | yep | 20:51 |
| clarkb | though I'm quickly getting lost in the js | 20:52 |
| *** trident has joined #openstack-infra | 20:52 | |
| sean-k-mooney | o/ so i did something dumb with my Depends-on: ... in https://review.opendev.org/#/c/602432/12//COMMIT_MSG and that is why my change was failing. anyone know which of the url style actully works? | 20:53 |
| corvus | clarkb: lemme know if you have questions, i may actually be able to answer them :) | 20:53 |
| sean-k-mooney | oh... im still useing review.openstack.org in that | 20:53 |
| sean-k-mooney | i bet its not following the redirect | 20:53 |
| corvus | sean-k-mooney: yep, sorry about that | 20:54 |
| sean-k-mooney | corvus: why are you sorry | 20:54 |
| sean-k-mooney | its not your fault i never updated the url | 20:54 |
| corvus | sean-k-mooney: because it's an easy mistake to make, and we have no way of reporting it. it just silently ignores the header | 20:54 |
| sean-k-mooney | ya and in my case the neutron patch is to fix a race which i won on enough case to have several jobs pass | 20:55 |
| sean-k-mooney | so i only figured it out be cause i saw a log message ideleted in neutron haha | 20:56 |
| sean-k-mooney | anyway i seam to have answered my own question ill replace it with https://review.opendev.org/#/c/640258 | 20:56 |
| sean-k-mooney | and then it should be happy instead of https://review.opendev.org/#/c/640258 | 20:57 |
| corvus | ++ | 20:57 |
| sean-k-mooney | * https://review.openstack.org/640258 | 20:57 |
| fungi | both forms should work fine | 20:57 |
| fungi | er, both with and without #/c/ i mean | 20:57 |
| corvus | yeah, it's not picky about "#/c" | 20:57 |
| sean-k-mooney | well as long as i updated to open dev it should be happy | 20:57 |
| corvus | fungi: i want to draw your attention to this msg: http://lists.openstack.org/pipermail/openstack-discuss/2019-July/008107.html | 20:58 |
| fungi | ooh, thanks! | 20:58 |
| fungi | that does indeed look nifty | 20:59 |
| fungi | seems like a great idea. will reply shortly | 20:59 |
| corvus | kk | 20:59 |
| *** jamesmcarthur has joined #openstack-infra | 20:59 | |
| *** joeguo has joined #openstack-infra | 20:59 | |
| *** tdasilva has quit IRC | 20:59 | |
| *** tdasilva has joined #openstack-infra | 20:59 | |
| corvus | i have not read the moz paper yet, but i have read enough readme + code that i think i understand what's going on :) | 20:59 |
| fungi | yeah, i'll do a deeper dive into the literature first, but from the brief description i have a pretty good idea of what they're trying to do | 21:00 |
| *** jamesmcarthur has quit IRC | 21:03 | |
| *** goldyfruit has quit IRC | 21:07 | |
| clarkb | fungi: I somehow completely missed those two gitea changes | 21:11 |
| clarkb | but both +2'd from me | 21:11 |
| fungi | thanks clarkb! | 21:12 |
| fungi | if anybody else has time to review 673353 (dns) and 673355 (system-config) changes, i can load data onto the last replacement gitea servers this afternoon | 21:12 |
| fungi | thanks corvus! | 21:12 |
| corvus | fungi: both are ok to +3 right? | 21:12 |
| fungi | yep! | 21:13 |
| corvus | done | 21:13 |
| fungi | much appreciated | 21:13 |
| corvus | right back atcha | 21:13 |
| fungi | there will be one final change to add them back into haproxy and drop the new project creation exclusion for them once they're up and ready to serve | 21:13 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-jobs master: Add auth config to kubelet user for buildset registries https://review.opendev.org/673351 | 21:13 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Update IP addresses for gitea07 and gitea08 https://review.opendev.org/673353 | 21:19 |
| *** dpawlik has quit IRC | 21:21 | |
| openstackgerrit | Merged opendev/system-config master: Add gitea07 and gitea08 replacements to inventory https://review.opendev.org/673355 | 21:37 |
| *** jamesmcarthur has joined #openstack-infra | 21:39 | |
| *** jamesmcarthur has quit IRC | 21:40 | |
| *** jamesmcarthur_ has joined #openstack-infra | 21:40 | |
| openstackgerrit | Merged opendev/puppet-zuul master: Update to new zuul JS content url https://review.opendev.org/671852 | 21:41 |
| efried | corvus: whassa translation from gerrit web UI query (either querystring or search field) to gertty? straight paste from either of those doesn't seem to be working. | 21:41 |
| efried | corvus: fwiw here's my working web UI query: https://review.opendev.org/#/q/project:openstack/nova-specs+status:open+path:%255Especs/.*/approved/.*+NOT+path:%255Especs/train/.*+NOT+path:%255Especs/backlog/.* | 21:41 |
| clarkb | efried: I think you need to use the "rendered" form that gerrit shows you in the search box? | 21:42 |
| *** betherly has joined #openstack-infra | 21:42 | |
| clarkb | project:openstack/nova-specs status:open path:^specs/.*/approved/.* NOT path:^specs/train/.* NOT path:^specs/backlog/.* | 21:42 |
| efried | Syntax error at "^specs/.*/approved/.* NOT │ | 21:43 |
| efried | │path:^specs/train/.* NOT path:^specs/backlog/.*" in search│ | 21:43 |
| efried | │string "project:openstack/nova-specs status:open │ | 21:43 |
| efried | │path:^specs/.*/approved/.* NOT path:^specs/train/.* NOT │ | 21:43 |
| efried | │path:^specs/backlog/.*" (col 46) | 21:43 |
| corvus | hrm, i'll dig into that | 21:43 |
| efried | sorry, this probably isn't the gertty support line :( | 21:43 |
| corvus | no better place; i'm happy to work on it here until someone kicks us out :) | 21:44 |
| *** betherly has quit IRC | 21:47 | |
| *** bdodd has quit IRC | 21:47 | |
| *** betherly has joined #openstack-infra | 21:48 | |
| corvus | efried: it looks like gertty doesn't understand "path:" but does understand "file:" i'm looking to see the difference | 21:50 |
| efried | corvus: iiuc file: is basepath and doesn't like regexes, but I could be misremembering. | 21:50 |
| corvus | re file: Regular expression matching can be enabled by starting the string with ^. In this mode file: is an alias of path: (see above). | 21:51 |
| efried | oh, path component | 21:51 |
| corvus | https://gerrit-review.googlesource.com/Documentation/user-search.html#file | 21:51 |
| efried | cool, lemme dork with that... | 21:51 |
| corvus | efried: so (a) should be easy to add "path:" to gertty and (b) we can change your "path" entries to "file" | 21:51 |
| efried | corvus: I'm about to try, but file: may only be matching path components | 21:52 |
| corvus | efried: and when i do that, i get some results back -- does that look right for you? (project:openstack/nova-specs status:open file:^specs/.*/approved/.* NOT file:^specs/train/.* NOT file:^specs/backlog/.*) | 21:52 |
| efried | corvus: appears to be the same results in UI, trying gertty... | 21:53 |
| corvus | efried: but your terms all start with "^" which enables regex searching for the full patch (ie the "is an alias for path" thing above) | 21:53 |
| *** betherly has quit IRC | 21:53 | |
| efried | corvus: in gertty it retrieves more results... | 21:54 |
| efried | 19 vs 13 | 21:54 |
| corvus | efried: more is better, right? :) | 21:54 |
| efried | not when I'm going to bulk abandon :P | 21:54 |
| corvus | i'll subscribe to nova-specs so i can try to reproduce | 21:54 |
| corvus | it'll take a few mins to catch up | 21:55 |
| corvus | 375221 is one that i have on my gertty screen that doesn't show up on my gerrit screen | 21:59 |
| corvus | oops, sorry that one was just out of date | 22:00 |
| corvus | it merged | 22:00 |
| corvus | (i had some old stuff sitting around from before i subscribed) | 22:00 |
| corvus | okay, looks like 580336 is one | 22:01 |
| corvus | and it shouldn't match because it has a file in "specs/train/" | 22:02 |
| corvus | so i think that's the case to debug | 22:02 |
| corvus | efried: since you're going to use the process mark anyway, you can probably manually find the ones which don't match the ui and make sure they aren't marked | 22:03 |
| corvus | efried: i'll try to figure out why this isn't working | 22:03 |
| efried | corvus: Ack, and thanks. | 22:03 |
| corvus | efried: thanks for the testing :) | 22:03 |
| efried | corvus: wonder if it has to do with the fact that earlier patch sets had files in those paths... | 22:08 |
| corvus | oooh | 22:08 |
| corvus | quite possibly | 22:08 |
| efried | the false positives seem to have that in common. | 22:08 |
| efried | while ones that are conspicuously missing... don't. | 22:08 |
| corvus | in the query that's used, i don't think i see a term that would restrict it to the latest patchset | 22:09 |
| efried | whereas presumably the gerrit web ui does that automatically, or something. | 22:09 |
| *** mriedem has quit IRC | 22:11 | |
| efried | corvus: gertty is going to take some getting used to (this is the third time I've tried) but was mega useful for the two bulk abandons I just pushed. Thanks for producing & maintaining it! | 22:28 |
| corvus | efried: np -- feel free to throw feedback at me (maybe i can improve or better document something) | 22:29 |
| efried | corvus: in the past the main blocker has been the sync time. Maybe it's just because I'm working against massive repos (e.g. nova), or over a slow connection, or not giving it long enough for the initial sync or... | 22:31 |
| corvus | efried: that's entirely possible. i don't subscribe to nova, but probably should just so i feel the pain | 22:31 |
| *** tosky has quit IRC | 22:32 | |
| efried | corvus: the 'sync' number at the top - that's the number of things it thinks it still needs to sync? | 22:32 |
| corvus | (i subscribe to a lot of repos, but that may not be an adequate substitute to subscribing to nova) | 22:32 |
| *** michael-beaver has quit IRC | 22:32 | |
| corvus | efried: yes, and it can go up even while it goes down as tasks explode into more subtasks | 22:32 |
| efried | cause all I've done thus far is | 22:33 |
| efried | install | 22:33 |
| efried | set up my config (the openstack-y template) | 22:33 |
| efried | subscribe to nova-specs | 22:33 |
| efried | and it's at 80-odd still | 22:33 |
| corvus | but once it hits zero, it's updates only | 22:33 |
| corvus | hrm. i managed to sync nova-specs in about 5-10 mins iirc | 22:33 |
| *** rfarr_ has joined #openstack-infra | 22:33 | |
| efried | right, it must be doing something else for the initial bootstrap? | 22:33 |
| corvus | once it clones the repo (which it will do for the first change it syncs) that should be it for bootstrap stuff | 22:35 |
| *** mattw4 has quit IRC | 22:35 | |
| *** mattw4 has joined #openstack-infra | 22:35 | |
| corvus | efried: if you run with "-d" you can see the tasks it's running in .gertty.log | 22:35 |
| clarkb | fungi: are we just waiting on 07 and 08 to sync git repos now | 22:36 |
| efried | assume I can kill it rn and restart it thusly... | 22:36 |
| clarkb | fungi: that and then GC I suppose | 22:36 |
| corvus | efried: you can "ctrl-q" to quit at any time and it should be fine when you restart | 22:36 |
| corvus | efried: that's probably nicer than kill :) | 22:36 |
| *** rfarr has quit IRC | 22:37 | |
| corvus | (but even that *should* be okay) | 22:37 |
| efried | yeah, ^q is what I meant | 22:37 |
| efried | okay, yeah, so it's doing a bunch of stuff in repos I haven't subscribed to. | 22:38 |
| efried | ...or it was. It seems to be done now. Which belies the sync counter from right before I quit... | 22:38 |
| corvus | maybe something left over from a previous run? hrm. | 22:39 |
| efried | is ^s on the project list screen supposed to find things off the current page? | 22:39 |
| *** rfarr_ has quit IRC | 22:39 | |
| *** eernst has quit IRC | 22:40 | |
| *** rfarr_ has joined #openstack-infra | 22:40 | |
| efried | ah, repeat ^s | 22:40 |
| corvus | efried: yes, it should scroll, but you may have to hit "^s" again to get to the first one | 22:40 |
| corvus | that seems like it could be better, huh :) | 22:40 |
| efried | corvus: It's kinda throwing me for a loop, intuitive-ui-wise, that some stuff is emacs-y and some stuff is very very not. | 22:41 |
| fungi | clarkb: yeah, i'm watching the gerrit replication tasks burn down while i eat dinner, and will kick off git gc on them as soon as replication finishes | 22:41 |
| fungi | (replication's about half done) | 22:41 |
| efried | corvus: I think it was the (lack of) highlighting contrast that made it unclear to me that the search had "worked" and landed on my subscribed project, which was the only matching thing on the current screen. nbd, now that I understand :) | 22:42 |
| corvus | efried: yeah, that one is supposed to be more emacsy than it is :) | 22:42 |
| *** diablo_rojo has quit IRC | 22:45 | |
| efried | okay, I'm going to blow this popsicle stand before I do any more damage. Thanks again corvus. | 22:45 |
| * fungi remapped ^q to :q to make himself feel more at home | 22:45 | |
| corvus | i've figured out how to fix the query in sql, but i haven't figured out how to get sqlalchemy to produce that sql yet | 22:45 |
| *** eharney has quit IRC | 22:51 | |
| *** diablo_rojo has joined #openstack-infra | 22:52 | |
| corvus | okay, i got 13 results! | 22:52 |
| corvus | 141219 449155 529135 530214 552924 572805 581638 591037 593197 595236 599957 603417 626055 | 22:53 |
| corvus | let's see if those are the ones efried abandoned | 22:53 |
| *** tkajinam has joined #openstack-infra | 22:54 | |
| corvus | yep, exact match | 22:55 |
| efried | sweet | 22:56 |
| openstackgerrit | James E. Blair proposed ttygroup/gertty master: Only search files from the most recent revision https://review.opendev.org/673370 | 22:57 |
| donnyd | clarkb: when will the new images be loaded? | 22:59 |
| corvus | i think the implementation for "file:" is actually correct for "path" but incorrect for "file" | 22:59 |
| *** rfarr_ has quit IRC | 23:00 | |
| *** rfarr_ has joined #openstack-infra | 23:01 | |
| *** rcernin has joined #openstack-infra | 23:02 | |
| clarkb | donnyd: I would've expected them to try and upload already? | 23:08 |
| clarkb | fungi: ^ | 23:08 |
| *** tjgresha has joined #openstack-infra | 23:08 | |
| openstackgerrit | James E. Blair proposed ttygroup/gertty master: Correct file search implementation https://review.opendev.org/673371 | 23:09 |
| corvus | efried: ^ there's the other thing :) | 23:09 |
| openstackgerrit | James E. Blair proposed ttygroup/gertty master: Correct file search implementation https://review.opendev.org/673371 | 23:11 |
| corvus | i think with both of those, the original gerrit search string would produce identical results | 23:12 |
| *** rfarr_ has quit IRC | 23:13 | |
| *** rfarr_ has joined #openstack-infra | 23:14 | |
| *** rfarr_ has quit IRC | 23:20 | |
| *** rfarr_ has joined #openstack-infra | 23:20 | |
| fungi | clarkb: no reason i can think of on our end nodepool should be unhappy, the fn creds are the same as before | 23:21 |
| fungi | (tested them myself to be sure) | 23:21 |
| fungi | but maybe it gave up trying after a while? or cached something wrong via sdk while stuff was still in flight? | 23:22 |
| *** n-saito has joined #openstack-infra | 23:30 | |
| clarkb | fungi: looks like replication is done? | 23:30 |
| fungi | most recent either nodepool builder mentions anything to do with fn is on the 25th (thursday) around 21:00z | 23:30 |
| clarkb | I'll manually ask nodepool to upload an image now | 23:30 |
| fungi | clarkb: yeah, replication done, git gc for both have been running for a while, should hopefully finish soon | 23:31 |
| fungi | i'll push up the changes to re-add them to haproxy so those are ready to go | 23:31 |
| clarkb | I guess we don't have a command to force an upload anymore? have to delete existing images to do that? | 23:32 |
| * clarkb tries this | 23:32 | |
| clarkb | I wonder if we've managed to confuse nodepool by removing the images out from udner it? | 23:34 |
| clarkb | hrm no it seems to have removed that image properly. Now waiting for it to decide it needs to upload a new one | 23:35 |
| clarkb | but ps says we've got a ton of dib processes running, something seems to have gone sideways | 23:36 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Put gitea07 and gitea08 back into service https://review.opendev.org/673373 | 23:37 |
| clarkb | I'm going to reboot nb01 | 23:38 |
| clarkb | and if that makes it happier I'll repeat with nb02 | 23:38 |
| fungi | heh, *that* explains why i got disconnected right after running image-list ;) | 23:39 |
| clarkb | nb01 is back up and looks a bit happier. Going to repeat on 02 now. Then I guess we check in the morning to see if they have leaked a bunch of dib processes again? or maybe ianw will be able to look at it? | 23:42 |
| clarkb | ianw: ^ are you around today and if so would you prefer that I not reboot nb02 so that you can see what it is doing with the dib processes? | 23:42 |
| *** rfarr__ has joined #openstack-infra | 23:43 | |
| ianw | sure, sorry didn't know it had gone haywire | 23:45 |
| clarkb | ok I deleted an image in fn for which there was a newer one on nb01 and now that image is getting uploaded | 23:45 |
| clarkb | ianw: it seems nb02 isn't uploading images for which it has the newer version of and there are many more disk-image-create processes than expected | 23:45 |
| fungi | yeah, we apparently hadn't uploaded most of our images anywhere for over 4 days | 23:45 |
| *** rfarr_ has quit IRC | 23:45 | |
| clarkb | fwiw nb01 was in the same state and rebooting it seems to have made it happier | 23:45 |
| ianw | really should setup alerts for it. "when did thing happen" isn't grafana's strong point, but it can be done | 23:46 |
| ianw | "when did thing last happen" i mean | 23:46 |
| *** rfarr__ has quit IRC | 23:46 | |
| clarkb | ianw: also have a quick moment to approve https://review.opendev.org/#/c/673373/1, will complete the gitea rebuild process fungi has been pushing | 23:46 |
| fungi | git gc is still underway on both of them but i expect it to complete any minute now | 23:46 |
| clarkb | last dib release was a couple weeks ago so that is unlikely to be the culprit | 23:47 |
| ianw | it seems to have a bunch of zypper processes | 23:48 |
| clarkb | checking dmesg I don't see anything amiss there (getting itno the habit of checking that now that we've had a couple disk issues) | 23:48 |
| *** diablo_rojo has quit IRC | 23:48 | |
| ianw | ianw@nb02:~$ sudo strace -p 23186 | 23:49 |
| ianw | strace: Process 23186 attached | 23:49 |
| ianw | write(1, "Additional rpm output:\nrpm: RPM "... | 23:49 |
| *** sthussey has quit IRC | 23:51 | |
| ianw | yeah so nodepool-builder is looping | 23:52 |
| clarkb | oh ya nb01 already has two disk-image-create proceses | 23:52 |
| ianw | bt of nb2 looping process ... http://paste.openstack.org/show/755086/ | 23:54 |
| ianw | no that's not quite right. there's 250+ nodepool builder threads on nb02 | 23:56 |
| *** mattw4 has quit IRC | 23:56 | |
| ianw | http://paste.openstack.org/show/755087/ | 23:56 |
| ianw | i have dejavu, i'm sure we've seen thsi before | 23:56 |
| ianw | looking at a thread, it's not seemingly yet in nodepool code as such -> http://paste.openstack.org/show/755088/ | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
