| *** aaronsheffield has quit IRC | 00:06 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** gagehugo has quit IRC | 00:12 | |
| *** goldyfruit___ has joined #openstack-infra | 00:13 | |
| *** slaweq has quit IRC | 00:16 | |
| *** rcernin has quit IRC | 00:30 | |
| *** rcernin has joined #openstack-infra | 00:31 | |
| *** jamesmcarthur has joined #openstack-infra | 00:33 | |
| *** mriedem_afk has quit IRC | 00:43 | |
| *** ociuhandu has joined #openstack-infra | 01:01 | |
| *** armax has quit IRC | 01:04 | |
| *** armax has joined #openstack-infra | 01:05 | |
| *** ociuhandu has quit IRC | 01:06 | |
| *** jamesmcarthur has quit IRC | 01:17 | |
| *** nhicher has quit IRC | 01:26 | |
| *** sgw has quit IRC | 01:45 | |
| *** armax has quit IRC | 01:45 | |
| *** apetrich has quit IRC | 02:10 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** slaweq has quit IRC | 02:15 | |
| *** gagehugo has joined #openstack-infra | 02:16 | |
| *** bobh has joined #openstack-infra | 02:24 | |
| *** bobh has quit IRC | 02:24 | |
| *** rcernin has quit IRC | 02:27 | |
| *** rcernin has joined #openstack-infra | 02:27 | |
| *** markvoelker has joined #openstack-infra | 02:38 | |
| *** markvoelker has quit IRC | 02:43 | |
| *** markvoelker has joined #openstack-infra | 03:26 | |
| *** markvoelker has quit IRC | 03:31 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** slaweq has quit IRC | 04:16 | |
| *** kjackal has joined #openstack-infra | 04:37 | |
| *** whoami-rajat has joined #openstack-infra | 04:49 | |
| *** markvoelker has joined #openstack-infra | 04:55 | |
| *** markvoelker has quit IRC | 05:00 | |
| *** kjackal has quit IRC | 05:26 | |
| *** prometheanfire has quit IRC | 05:39 | |
| *** prometheanfire has joined #openstack-infra | 05:40 | |
| *** kjackal has joined #openstack-infra | 05:41 | |
| *** ykarel|away has joined #openstack-infra | 05:45 | |
| *** zzehring has quit IRC | 05:46 | |
| *** lajoskatona has left #openstack-infra | 06:00 | |
| *** slaweq has joined #openstack-infra | 06:05 | |
| *** slaweq has quit IRC | 06:10 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** slaweq has quit IRC | 06:15 | |
| *** igordc has quit IRC | 06:31 | |
| *** markvoelker has joined #openstack-infra | 06:44 | |
| *** markvoelker has quit IRC | 06:49 | |
| openstackgerrit | Merged zuul/nodepool master: k8s: make context optional https://review.opendev.org/682130 | 07:21 |
|---|---|---|
| *** sshnaidm_ has joined #openstack-infra | 07:23 | |
| *** sshnaidm|off has quit IRC | 07:25 | |
| AJaeger | corvus: your rename of check is a nice idea. Still, in most cases I need to remove templates as well... | 07:32 |
| *** slaweq has joined #openstack-infra | 07:54 | |
| *** xenos76 has joined #openstack-infra | 08:11 | |
| *** ykarel|away has quit IRC | 08:38 | |
| *** ykarel|away has joined #openstack-infra | 08:39 | |
| *** markvoelker has joined #openstack-infra | 08:44 | |
| *** markvoelker has quit IRC | 08:48 | |
| *** rcernin has quit IRC | 08:49 | |
| *** tosky has joined #openstack-infra | 08:54 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Start train translations https://review.opendev.org/682205 | 09:20 |
| AJaeger | ianychoi: could you review this, please? ^ | 09:20 |
| *** xenos76 has quit IRC | 09:21 | |
| *** xenos76 has joined #openstack-infra | 09:27 | |
| *** tosky_ has joined #openstack-infra | 09:34 | |
| *** tosky has quit IRC | 09:38 | |
| *** whoami-rajat has quit IRC | 09:39 | |
| *** xenos76 has quit IRC | 09:50 | |
| *** ociuhandu has joined #openstack-infra | 09:51 | |
| *** xenos76 has joined #openstack-infra | 09:51 | |
| *** ociuhandu has quit IRC | 10:05 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Remove doc files and autoremove packages https://review.opendev.org/675546 | 10:34 |
| *** tosky_ is now known as tosky | 10:43 | |
| *** ykarel|away has quit IRC | 11:04 | |
| *** lpetrut has joined #openstack-infra | 11:06 | |
| *** lpetrut has quit IRC | 11:28 | |
| *** e0ne has joined #openstack-infra | 11:51 | |
| *** tosky has quit IRC | 12:55 | |
| *** e0ne has quit IRC | 13:07 | |
| *** xenos76 has quit IRC | 13:08 | |
| *** xenos76 has joined #openstack-infra | 13:28 | |
| *** exsdev has quit IRC | 13:44 | |
| *** rf0lc0 has joined #openstack-infra | 13:46 | |
| *** exsdev has joined #openstack-infra | 13:49 | |
| donnyd | So I have been watching the logs for messages of not being able to connect to the mirror and I don't see any for the last 24 hours | 13:55 |
| donnyd | I am going to mark this network issue solved. | 13:58 |
| donnyd | I have also looked at all of the failures from the last 12 hours and from what I can see none of them are related to FN | 13:59 |
| *** kjackal has quit IRC | 14:09 | |
| *** dulek has quit IRC | 14:26 | |
| *** whoami-rajat has joined #openstack-infra | 14:27 | |
| *** pcaruana has joined #openstack-infra | 14:42 | |
| *** ykarel|away has joined #openstack-infra | 14:57 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Switch project-config to promote jobs [1]: Disable publish https://review.opendev.org/682211 | 14:57 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Switch project-config to promote jobs [3]: Update building https://review.opendev.org/682212 | 14:57 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Switch project-config to promote jobs [2]: Update build jobs https://review.opendev.org/682213 | 14:57 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Switch project-config to promote jobs [2]: Update build jobs https://review.opendev.org/682213 | 14:59 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Switch project-config to promote jobs [1]: Disable publish https://review.opendev.org/682211 | 14:59 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Switch project-config to promote jobs [3]: Update building https://review.opendev.org/682212 | 14:59 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Switch project-config to promote jobs [3]: Update building https://review.opendev.org/682212 | 15:01 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Switch project-config to promote jobs [1]: Disable publish https://review.opendev.org/682211 | 15:03 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Switch project-config to promote jobs [3]: Update building https://review.opendev.org/682212 | 15:03 |
| *** dave-mccowan has joined #openstack-infra | 15:05 | |
| fungi | donnyd: that sounds like great news--thanks for checking! | 15:29 |
| donnyd | fungi: next I am working on moving the object storage to all nvme | 15:29 |
| donnyd | granted there will be less available space, it will be much mo faster | 15:30 |
| fungi | sounds awesome | 15:30 |
| donnyd | there is currently just under 1TB of object space used and around 1.6TB of actual space used. How long is our log retention planned to be? | 15:32 |
| donnyd | I am trying to estimate how much storage I will need. Looks like about 8TB of space will be available in replica 2 and not sure how much if I go erasure coded | 15:33 |
| donnyd | maybe a bit more, but I need to ask swift if erasure coding is good for my use case | 15:33 |
| *** armax has joined #openstack-infra | 15:40 | |
| fungi | we were managing ~1 month of retention on a 14tb ext4 filesystem with a bit of breathing room (but not much) | 15:41 |
| fungi | so how much it takes to match that will depend on how many providers we spread it over | 15:41 |
| fungi | we'd ideally like to have more retention, but for now we're (afaik) setting something like a 30-day expiration on objects we're writing | 15:42 |
| clarkb | the goal was that spreading this out would get us back to 6 months but it is currently at 30 days | 15:45 |
| clarkb | currently spread between 6 regions | 15:46 |
| clarkb | (with maybe a seventh possible if wesortour ara for ceph) | 15:46 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Switch project-config to promote jobs [4]: Add promote jobs https://review.opendev.org/682214 | 15:49 |
| donnyd | so it should be hitting somewhere around 4TB of space used | 15:50 |
| donnyd | Based on current usage it will be somewhere between 4-4.5 | 15:51 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Switch project-config to promote jobs [5]: Update specs, cleanup https://review.opendev.org/682215 | 15:53 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Switch project-config to promote jobs [4]: Add promote jobs https://review.opendev.org/682214 | 15:54 |
| fungi | oh, also it's possible the changes we made to drop the default ara shaves some off it | 15:55 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Switch project-config to promote jobs [6]: Remove publish-openstack-specs https://review.opendev.org/682216 | 15:56 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Switch project-config to promote jobs [5]: Update specs, cleanup https://review.opendev.org/682215 | 15:56 |
| *** dave-mccowan has quit IRC | 15:58 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Switch project-config to promote jobs [3a]: DNM: Test building https://review.opendev.org/682217 | 15:59 |
| donnyd | I think even worst case we should still be ok | 16:00 |
| donnyd | Spinny drive make lots of heat, so getting rid of them really makes a difference here... also zoom zoom | 16:01 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Switch project-config to promote jobs [4]: Add promote jobs https://review.opendev.org/682214 | 16:02 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Switch project-config to promote jobs [6]: Remove publish-openstack-specs https://review.opendev.org/682216 | 16:02 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix dib job https://review.opendev.org/682218 | 16:08 |
| AJaeger | ianw: do you want to update diskimage-builder to require flake8 and cut a new release or should we do it like above? ^ See https://review.opendev.org/682212 for the failure | 16:08 |
| AJaeger | clarkb, fungi, I just did my weekend fun hacking project: Switched specs-site and infra/index.html to promote jobs, topic:promote-project-config is something for reviewing next week ... | 16:11 |
| fungi | donnyd: well, also waste heat == electrical load, so reducing it means longer backup on ups, (slightly?) lower power bill, et cetera | 16:11 |
| fungi | not to mention longevity on the storage since mtbf of moving parts is a lot worse | 16:12 |
| fungi | AJaeger: awesome, thanks! | 16:12 |
| *** signed8b_ has joined #openstack-infra | 16:13 | |
| donnyd | Oh I totally forgot to say my generator fuel supply is being hooked up today, so that won't matter anymore either.. but yes more time on UPS | 16:13 |
| donnyd | it costs 8 degrees to have the spinny servers up and running in my current setup... I about it a month it won't really matter for 6 more.. but in June/July/August it really matters | 16:14 |
| fungi | yeah, back when i had a basement office/lab/workshop with racks full of server and network hardware (and an ancient risc systems zoo), it stayed nice and toasty throughout the winter | 16:15 |
| donnyd | Currently my setup runs at 80F | 16:16 |
| donnyd | with the Storage servers down its 72-74F | 16:16 |
| fungi | nice! | 16:16 |
| donnyd | So my only option is to run swift as instance on Openstack... which isn't the best way to do it... but also costs a lot less $$$ | 16:17 |
| *** signed8b_ has quit IRC | 16:17 | |
| *** signed8bit has quit IRC | 16:17 | |
| fungi | i also had an auxiliary free-standing air conditioner i ducted out through a window panel to get by in the warm months | 16:17 |
| donnyd | I should say instances (12 to be precise) | 16:17 |
| donnyd | I used to use aux A/C, but I figured out that with the right amount of airflow it made very little in the way of difference in DC heat | 16:18 |
| fungi | and covered the insides of all the windows with aluminum foil to cut down on extra sunlight-induced heat during the day | 16:18 |
| donnyd | Like 2 degrees... So I moved to a system that just moves a massive amount of air into the front (cool) side of the room, and the back side (hot) can either exhaust or recycle the air | 16:19 |
| fungi | the mini-crac was necessary in my case because the central ac in that townhouse was already undersized | 16:19 |
| donnyd | depending on the temps outside would determine the amount of recycle.. In winter its 90% recycle, in summer its 10% recycle | 16:19 |
| donnyd | Ah yes, that will do it | 16:20 |
| donnyd | I really need to get my central A/C upsized.. Its on the list, but not currently critical | 16:20 |
| fungi | yeah, so do we. the one in this house is also not big enough to deal with the conditions here, though a lot of that is because we live right on the water and the whole structure is leaky so it's hard to keep our inside relative humidity below 50% | 16:22 |
| fungi | the whole place badly needs to be sealed and insulated | 16:23 |
| *** rf0lc0 has quit IRC | 16:23 | |
| fungi | i'd love to get one with dedicated humidity control. it would come in really handy here | 16:24 |
| donnyd | I am working on the swift nvme deployment, but its probably going to be a bit before its done | 16:24 |
| donnyd | fungi that sounds $$$ | 16:25 |
| *** xek has joined #openstack-infra | 16:25 | |
| fungi | yeah, it will be | 16:25 |
| donnyd | I have been trying to find a cost effective way to move some of this stuff offsite... but I am finding that to be impossible | 16:27 |
| donnyd | I would even be ok with a little less scale... But I can't afford to pay someone else 4K a month for DC space | 16:28 |
| *** xek has quit IRC | 16:33 | |
| *** markvoelker has joined #openstack-infra | 16:55 | |
| *** markvoelker has quit IRC | 17:00 | |
| *** xek has joined #openstack-infra | 17:05 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Always publish infra documents https://review.opendev.org/682222 | 17:10 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove release-tox-docs-infra https://review.opendev.org/682223 | 17:11 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Always publish infra documents https://review.opendev.org/682222 | 17:11 |
| *** xek_ has joined #openstack-infra | 17:13 | |
| *** xek has quit IRC | 17:16 | |
| *** dave-mccowan has joined #openstack-infra | 17:17 | |
| *** dave-mccowan has quit IRC | 17:23 | |
| *** jamesmcarthur has joined #openstack-infra | 17:29 | |
| *** dave-mccowan has joined #openstack-infra | 17:44 | |
| fungi | donnyd: yeah, colocation is not cheap. more than a decade ago i recall the colocation company i worked for charged ~us$2k/mo for a half-cab, and that was with a fairly meager amount of complimentary network bandwidth | 18:00 |
| *** efried_afk has quit IRC | 18:06 | |
| *** dave-mccowan has quit IRC | 18:08 | |
| *** ykarel|away has quit IRC | 18:12 | |
| *** ykarel|away has joined #openstack-infra | 18:12 | |
| *** whoami-rajat has quit IRC | 18:13 | |
| *** bobh has joined #openstack-infra | 18:17 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add promote pipeline to grafana https://review.opendev.org/682225 | 18:21 |
| *** bobh has quit IRC | 18:30 | |
| *** kjackal has joined #openstack-infra | 18:30 | |
| *** bobh has joined #openstack-infra | 18:35 | |
| *** kjackal has quit IRC | 18:38 | |
| *** bobh has quit IRC | 18:40 | |
| *** markvoelker has joined #openstack-infra | 18:50 | |
| *** markvoelker has quit IRC | 18:55 | |
| *** bobh has joined #openstack-infra | 19:00 | |
| *** jamesmcarthur has quit IRC | 19:02 | |
| *** jamesmcarthur has joined #openstack-infra | 19:04 | |
| *** jamesmcarthur has quit IRC | 19:09 | |
| *** jamesmcarthur has joined #openstack-infra | 19:33 | |
| *** rf0lc0 has joined #openstack-infra | 19:35 | |
| *** jamesmcarthur has quit IRC | 19:39 | |
| openstackgerrit | Merged openstack/infra-manual master: Update bug status link. https://review.opendev.org/674218 | 19:43 |
| donnyd | So my generator is running, and I am about to start conducting tests to ensure it actually functions. Should we turn FN off or leave it on | 19:46 |
| donnyd | I can just adjust quota down to zero so we won't need any code changes | 19:46 |
| donnyd | but I just wanted to know if we should test real world or not? | 19:46 |
| fungi | well, if you don't have a strong expectation it's going to take the swift endpoint offline then it's probably fine | 19:52 |
| fungi | if you think you're probably going to lose power to it or need to shut it down for some reason we should temporarily stop writing logs there because it'll cause jobs to end in a post_failure | 19:53 |
| fungi | hopefully soon we'll have a more robust retry mechanism in place for archiving logs to object storage | 19:54 |
| donnyd | yea, so I reread the manual and I need to start testing with all of the loads off... so I think I will need to remove FN | 19:55 |
| fungi | okay, yeah i'm less worried about the nova endpoint but right now we're not very robust when it comes to handling archiving stuff to swift | 19:56 |
| *** slaweq has quit IRC | 19:56 | |
| openstackgerrit | Donny Davis proposed openstack/project-config master: Temp Shutdown of FN for backup generator testing https://review.opendev.org/682226 | 19:56 |
| openstackgerrit | Donny Davis proposed opendev/base-jobs master: Temp Shutdown of FN for backup generator testing https://review.opendev.org/682227 | 19:58 |
| donnyd | yea we don't need another oops with the swift stuff again | 19:58 |
| donnyd | :) | 19:59 |
| *** jamesmcarthur has joined #openstack-infra | 20:05 | |
| donnyd | thanks for being around today fungi | 20:08 |
| donnyd | :) | 20:08 |
| fungi | okay, those are approved and should hopefully merge in a few minutes | 20:08 |
| fungi | hey, no thanks needed. it's just a few button clicks for me ;) | 20:08 |
| fungi | (well, keypresses, but who's counting?) | 20:08 |
| donnyd | I am projecting it will be 2.5 hours till all the workloads are spun down and then 1.5 hours for the actual test | 20:08 |
| donnyd | so it should only be a 4 hour outage | 20:09 |
| fungi | there's no rush... i'm heading out to eat shortly but will certainly be around again by then | 20:09 |
| fungi | i'll happily keep an eye out for additional changes when you're ready to spin stuff back up | 20:10 |
| *** slaweq has joined #openstack-infra | 20:11 | |
| *** jamesmcarthur has quit IRC | 20:14 | |
| openstackgerrit | Merged opendev/base-jobs master: Temp Shutdown of FN for backup generator testing https://review.opendev.org/682227 | 20:14 |
| openstackgerrit | Merged openstack/project-config master: Temp Shutdown of FN for backup generator testing https://review.opendev.org/682226 | 20:15 |
| *** slaweq has quit IRC | 20:15 | |
| donnyd | fungi: how long should i wait for the swift stuff to go active? | 20:27 |
| clarkb | the swift change sgould take effect for new jobs that start after the merge | 20:35 |
| *** eharney has quit IRC | 20:36 | |
| fungi | yeah, there's somewhat of a half-life involved | 20:36 |
| fungi | technically three hours is the maximum we allow builds to execute | 20:37 |
| fungi | but i wouldn't wait anywhere near that long | 20:37 |
| fungi | and if you shut down the compute nodes before all builds have completed, it's not the end of the world since they'll just be retried | 20:39 |
| clarkb | ya its the weekend. now is fine | 20:39 |
| fungi | i mean, there's a 25-change-long gate queue, but... meh | 20:39 |
| donnyd | okie dokie | 20:40 |
| fungi | by 21:15 any builds which complete within an hour are clear from a swift perspective, which is the more touchy bit | 20:40 |
| donnyd | Well the sooner I can do the tests, the sooner i can bring it back up to help out with that queue | 20:41 |
| fungi | we strongly encourage projects to keep individual build times under an hour (ideally 45 minutes or less) | 20:41 |
| donnyd | The hope is that nothing actually goes down at all | 20:41 |
| fungi | and now i disappear to get dinner... bbiaw | 20:41 |
| donnyd | with the load gone, now my ups say I have 26 minutes... which they should only be out of power for around 5 | 20:43 |
| *** jamesmcarthur has joined #openstack-infra | 20:44 | |
| *** ykarel|away has quit IRC | 20:44 | |
| donnyd | ok, FN is running off generator power right noiw | 20:59 |
| donnyd | looks to me like it all stayed up | 21:00 |
| donnyd | I have to let this run for the next 30 or so minutes to ensure everything is working properly. Then FN will be placed back into operation | 21:10 |
| *** jamesmcarthur has quit IRC | 21:20 | |
| *** jamesmcarthur has joined #openstack-infra | 21:24 | |
| *** pcaruana has quit IRC | 21:24 | |
| *** trident has quit IRC | 21:30 | |
| donnyd | There is one important cavet to my system... I can't run the A/C.... so depending on the weather time may be limited | 21:31 |
| *** jamesmcarthur has quit IRC | 21:41 | |
| *** trident has joined #openstack-infra | 21:44 | |
| prometheanfire | requirements gate is very finicky atm | 21:51 |
| *** tosky has joined #openstack-infra | 22:08 | |
| openstackgerrit | Donny Davis proposed opendev/base-jobs master: revert "Temp Shutdown of FN for backup generator testing" https://review.opendev.org/682231 | 22:09 |
| openstackgerrit | Donny Davis proposed openstack/project-config master: revert "Temp Shutdown of FN for backup generator testing" https://review.opendev.org/682232 | 22:11 |
| *** tosky has quit IRC | 22:11 | |
| *** rf0lc0 has quit IRC | 22:13 | |
| donnyd | .ok gen tests done, FN is back online. I ran some quick validations to make sure everything is working. It did go down when it switched back to main power. I am running down why | 22:16 |
| fungi | donnyd: cool! and i'm back from dinner errands | 22:47 |
| fungi | approving the reverts now. lmk if i shouldn't | 22:48 |
| *** markvoelker has joined #openstack-infra | 22:52 | |
| openstackgerrit | Merged opendev/base-jobs master: revert "Temp Shutdown of FN for backup generator testing" https://review.opendev.org/682231 | 22:54 |
| *** markvoelker has quit IRC | 22:56 | |
| openstackgerrit | Merged openstack/project-config master: revert "Temp Shutdown of FN for backup generator testing" https://review.opendev.org/682232 | 22:57 |
| *** bobh has quit IRC | 23:05 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** slaweq has quit IRC | 23:15 | |
| *** xenos76 has quit IRC | 23:24 | |
| *** xek_ has quit IRC | 23:45 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!