| *** mlavalle has quit IRC | 00:15 | |
| *** DSpider has quit IRC | 00:18 | |
| openstackgerrit | Merged opendev/system-config master: Make a review-test that we run ansible on https://review.opendev.org/735602 | 00:19 |
|---|---|---|
| openstackgerrit | Merged opendev/system-config master: Add playbook for syncing state from review to review-test https://review.opendev.org/735610 | 00:19 |
| ianw | again there's a bunch of hung ssh to 104.130.246.44 elasticsearch07.openstack.org. | 00:40 |
| ianw | does the same for me, ssh just hangs, rebooting it | 00:42 |
| ianw | "hung process" stuff on console, nothing else | 00:42 |
| *** aannuusshhkkaa has quit IRC | 01:00 | |
| *** factor__ has quit IRC | 01:53 | |
| *** factor__ has joined #opendev | 01:53 | |
| *** factor__ has quit IRC | 02:02 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: mirror-update: restore -t to rsync commands https://review.opendev.org/736880 | 02:29 |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: Remove mirror-update01 https://review.opendev.org/736860 | 02:34 |
| *** sgw has quit IRC | 02:42 | |
| openstackgerrit | Merged opendev/zone-opendev.org master: Remove mirror-update01 https://review.opendev.org/736860 | 02:56 |
| *** sgw has joined #opendev | 03:00 | |
| ianw | logstashworker09 is also causing problems. that just reejcts ssh connections quickly | 03:22 |
| ianw | that has oom messages on it | 03:22 |
| ianw | #status log reboot elasticsearch07.openstack.org and logstashworker09.openstack.org, both had become unresponsive | 03:26 |
| openstackstatus | ianw: finished logging | 03:26 |
| openstackgerrit | Merged opendev/system-config master: mirror-update: restore -t to rsync commands https://review.opendev.org/736880 | 03:34 |
| *** diablo_rojo has quit IRC | 03:40 | |
| *** ysandeep|away is now known as ysandeep | 04:05 | |
| *** sgw has quit IRC | 04:08 | |
| *** sgw has joined #opendev | 04:09 | |
| AJaeger | ianw: could you review https://review.opendev.org/#/c/736848/1 , please? | 04:42 |
| ianw | AJaeger: ok .. it seems fine but why did tox go missing? | 04:45 |
| AJaeger | guess it was on the images before? | 04:46 |
| *** ykarel|away is now known as ykarel | 04:51 | |
| openstackgerrit | Merged openstack/project-config master: Add ensure-tox to pti-python-tarball https://review.opendev.org/736848 | 04:54 |
| *** sgw has quit IRC | 05:03 | |
| *** sgw has joined #opendev | 05:10 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove mirror-update.openstack.org from graphite iptables https://review.opendev.org/736940 | 06:08 |
| *** tosky has joined #opendev | 06:30 | |
| *** rpittau|afk is now known as rpittau | 06:44 | |
| iurygregory | frickler, hey you around? Do you know if there was any progress on the missing logs after the ansible update? =) | 06:51 |
| frickler | iurygregory: yes, the instance I found should be fixed by https://review.opendev.org/736714 , but it is possible that other code parts are affected, too. let us know if you find any other weirdness | 07:07 |
| frickler | but at least I'm know getting the logs in devstack again that were missing earlier | 07:08 |
| iurygregory | frickler, cool! | 07:08 |
| frickler | s/know/now/ | 07:08 |
| openstackgerrit | Merged opendev/system-config master: Remove mirror-update.openstack.org from graphite iptables https://review.opendev.org/736940 | 07:09 |
| *** DSpider has joined #opendev | 07:21 | |
| *** hashar has joined #opendev | 07:30 | |
| *** bhagyashris|away is now known as bhagyashris | 07:39 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:03 | |
| *** ykarel is now known as ykarel|lunch | 08:45 | |
| *** mugsie has quit IRC | 08:48 | |
| *** mugsie has joined #opendev | 08:48 | |
| iurygregory | frickler, we have the ironic logs back \o/ tks! | 09:03 |
| *** tkajinam has quit IRC | 09:21 | |
| *** clarkb has quit IRC | 09:23 | |
| *** clarkb has joined #opendev | 09:28 | |
| *** ysandeep is now known as ysandeep|brb | 09:29 | |
| *** priteau has joined #opendev | 09:35 | |
| *** ykarel|lunch is now known as ykarel | 09:48 | |
| openstackgerrit | Merged openstack/project-config master: Complete retirement of mox3 https://review.opendev.org/736760 | 10:03 |
| *** calcmandan has quit IRC | 10:11 | |
| *** calcmandan has joined #opendev | 10:12 | |
| *** rpittau is now known as rpittau|bbl | 10:22 | |
| *** calcmandan has quit IRC | 10:30 | |
| *** calcmandan has joined #opendev | 10:32 | |
| openstackgerrit | Hervé Beraud proposed openstack/project-config master: Add Backport-Candidate label for oslo deliverables https://review.opendev.org/734096 | 10:49 |
| AJaeger | infra-root, the deploy job infra-prod-manage-projects failed on https://review.opendev.org/736760 | 10:54 |
| frickler | the failure seems to be only on review-test, so not too critical I hope | 11:33 |
| *** tkajinam has joined #opendev | 12:04 | |
| *** ysandeep|brb is now known as ysandeep | 12:11 | |
| *** rpittau|bbl is now known as rpittau | 12:19 | |
| openstackgerrit | Javier Peña proposed opendev/system-config master: openafs-client: Ensure make is installed for CentOS https://review.opendev.org/736995 | 12:48 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: Support CentOS for AFS mirror https://review.opendev.org/736996 | 12:48 |
| openstackgerrit | Hervé Beraud proposed openstack/project-config master: Add Backport-Candidate label for oslo deliverables https://review.opendev.org/734096 | 13:03 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror https://review.opendev.org/736996 | 13:08 |
| openstackgerrit | Hervé Beraud proposed openstack/project-config master: Add Backport-Candidate label for oslo deliverables https://review.opendev.org/734096 | 13:09 |
| *** ykarel is now known as ykarel|afk | 13:28 | |
| *** auristor has quit IRC | 13:39 | |
| *** auristor has joined #opendev | 13:44 | |
| *** rpittau is now known as rpittau|brb | 13:54 | |
| openstackgerrit | Donny Davis proposed openstack/project-config master: Reduce max servers on OE - air handler needs repairs https://review.opendev.org/737009 | 13:54 |
| *** mlavalle has joined #opendev | 13:54 | |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror https://review.opendev.org/736996 | 13:54 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: Make the set-hostname role compatible with non-Debian distros https://review.opendev.org/737010 | 13:54 |
| donnyd | So I need to reduce the number of GP builds so I can turn off some of the hypervisors until I get my air handler issues resolved. Heat is currently overrunning the air system | 13:54 |
| mordred | frickler: oh - that's ok, but I should maybe exclude review-test from that | 13:55 |
| mordred | donnyd: that is reasonable :) | 13:55 |
| donnyd | Not sure where the issue with the air handler is at - but I am working it and hopefully will have it resolved soon | 13:56 |
| donnyd | not cutting the custom jobs though, I know people are dependent on them ATM | 13:56 |
| AJaeger | donnyd: will single-core approve | 13:58 |
| fungi | donnyd: you even increased the custom node count slightly | 13:59 |
| fungi | presumably that was intentional | 13:59 |
| AJaeger | fungi, do you want to push 737010 to gate directly? | 13:59 |
| fungi | yeah, i can | 14:00 |
| *** rpittau|brb is now known as rpittau|afk | 14:00 | |
| AJaeger | fungi: or should we change the 12 back to 10 on the custom nodes first? | 14:01 |
| AJaeger | we can leave as is as well... | 14:01 |
| fungi | i assume it was increased to 12 intentionally | 14:01 |
| AJaeger | ok | 14:01 |
| donnyd | I changed it to 12 so airship can run more than one job | 14:01 |
| AJaeger | fungi: you're right ;) | 14:01 |
| donnyd | i do believe they are running 4 nodes at a time, and the math on 10 as a max is bad | 14:02 |
| donnyd | lol | 14:02 |
| donnyd | so now they should be able to do 3 at a time | 14:02 |
| *** guillaumec has joined #opendev | 14:03 | |
| fungi | makes sense | 14:04 |
| donnyd | no emergency, just need to start taking some action to fix this for good this time around | 14:06 |
| *** tosky has quit IRC | 14:08 | |
| openstackgerrit | Merged openstack/project-config master: Reduce max servers on OE - air handler needs repairs https://review.opendev.org/737009 | 14:13 |
| *** tkajinam has quit IRC | 14:13 | |
| *** ysandeep is now known as ysandeep|away | 14:14 | |
| fungi | once the deploy jobs run for that, you should be all set | 14:20 |
| fungi | looks like it's just the nodepool deploy job and it's in progress | 14:22 |
| *** hashar has quit IRC | 14:27 | |
| *** ykarel|afk is now known as ykarel | 14:28 | |
| *** rchurch has quit IRC | 14:29 | |
| *** rchurch has joined #opendev | 14:33 | |
| *** chandankumar is now known as raukadah | 14:45 | |
| mnaser | mordred: i need to get around updating your shell script to work with zsh now | 14:58 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update host_vars and sync-to-review-test playbook https://review.opendev.org/737023 | 15:02 |
| mordred | mnaser: you accepted the update? I did not :) | 15:05 |
| mordred | (although I will obviously accept that patch) | 15:05 |
| mordred | infra-root: that patch ^^ is based on running the sync playbook by hand - could I get a quick +A on it? | 15:05 |
| *** sgw has quit IRC | 15:06 | |
| *** sgw has joined #opendev | 15:08 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update host_vars and sync-to-review-test playbook https://review.opendev.org/737023 | 15:09 |
| mordred | infra-root: ^^ sorry, missed one thing | 15:09 |
| clarkb | mordred: I left a note on ps1, I don't think its really a thing, but its early for me and figured I'd mention it | 15:09 |
| mordred | clarkb: yeah, I agree - running that in the wrong direction would be ++bad | 15:11 |
| mordred | (although this is definitely a good argument for using rsync in a shell and not synchronize: :) ) | 15:12 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update host_vars and sync-to-review-test playbook https://review.opendev.org/737023 | 15:13 |
| mordred | clarkb: one more thing. sorry. | 15:13 |
| mordred | clarkb: do you remember why we write mysql creds to /root/.gerrit_db.cnf instead of /root/.my.cnf? | 15:19 |
| clarkb | mordred: I think they started as backup specific and we may have restricted them relative to a full root connection but they ended up equivalent over time? | 15:20 |
| *** mlavalle has quit IRC | 15:21 | |
| mordred | clarkb: nod | 15:21 |
| mordred | clarkb: I'll have one more patch coming for you in just a sec | 15:21 |
| *** mlavalle has joined #opendev | 15:25 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update host_vars and sync-to-review-test playbook https://review.opendev.org/737023 | 15:25 |
| *** sgw has quit IRC | 15:32 | |
| *** rchurch has quit IRC | 15:32 | |
| *** calcmandan has quit IRC | 15:32 | |
| *** moppy has quit IRC | 15:32 | |
| *** sgw has joined #opendev | 15:32 | |
| *** rchurch has joined #opendev | 15:32 | |
| *** calcmandan has joined #opendev | 15:32 | |
| *** moppy has joined #opendev | 15:32 | |
| *** Eighth_Doctor has quit IRC | 15:36 | |
| clarkb | mordred: https://review.opendev.org/#/c/737023/3..4/playbooks/sync-to-review-test.yaml is another scary thing because we could be writing back over the production db? | 15:39 |
| mordred | clarkb: yah. I have verified that the creds on review-test are correctly not the creds on review01 | 15:39 |
| clarkb | I guess in my half awake brain I'm trying to come up with "what is the proper separation between the two hosts and checks that -test is configured properly before landing the scary bits" | 15:39 |
| clarkb | like maybe we need two playbooks? a sync of the data, then a mv of the data into fake production | 15:40 |
| mordred | clarkb: would that really be any safer? we'd still worry that the creds on review-test were pointing at prod | 15:41 |
| mordred | (I'm with you - it's a terrifying playbook - I'm just not sure any of the thoughts I've had so far are any less terrifying) | 15:41 |
| mordred | fungi, corvus: if you have a sec, would appreciate thoughts on this ^^ | 15:42 |
| clarkb | mordred: ya I'm not sure myself. | 15:42 |
| clarkb | the plays are very explicit about where they should run which is probably 95% of what we need | 15:43 |
| clarkb | and its probably better to write it down and have it reviewed than try to ad hoc | 15:43 |
| clarkb | thinking about the two step process if we copied from prod:/prod/path to test:/test/path then had a separate step of apply test:/test/path to test:/prod/path we'd likely fail on prod simply because the prod:/test/path doesn't exist at all | 15:45 |
| clarkb | But that is still vulnerable to applying things on the wrong host which is the biggest issue | 15:45 |
| clarkb | and so it probably doesn't win much overall | 15:45 |
| corvus | mordred: ack; i'll mull over breakfast | 15:46 |
| fungi | my first impression is we do the best we can, but first make sure our backups are recent (and intact) | 15:49 |
| fungi | you can never be 100% positive you're not going to accidentally blow away production data, that's one of the main reasons we make backups | 15:49 |
| *** tosky has joined #opendev | 15:50 | |
| fungi | so a bit of uncertainty is probably fine, even expected, and so time spent avoiding disaster should ideally be less than the time required to deal with the fallout from that potential disaster multiplied by the chance of it occuring | 15:53 |
| AJaeger | here's a simple change to update system-config after refstack rename, it could use another review, please - https://review.opendev.org/735425 | 15:53 |
| AJaeger | thanks, clarkb | 15:59 |
| AJaeger | fungi, your twine change https://review.opendev.org/735932 has enough +2s, please self-approve anytime | 16:00 |
| clarkb | https://github.com/go-gitea/gitea/tree/v1.12.0 exists now. I'm working on updating my change to run that version | 16:00 |
| corvus | clarkb, fungi, mordred: yeah, i think the best things we can do are review carefully and make sure backups are current. should we do a restore test for review.o.o? | 16:01 |
| corvus | (just a simple one -- not a new server, just try to extract the current state on the backup server) | 16:01 |
| clarkb | thinking out loud here, maybe thats the way to sync to review-test | 16:01 |
| clarkb | from backup and not directly from prod | 16:01 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror https://review.opendev.org/736996 | 16:02 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: Use the ansible service task to reload iptables on CentOS https://review.opendev.org/737042 | 16:02 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: Make the base role and playbook compatible with CentOS https://review.opendev.org/737043 | 16:02 |
| *** ykarel is now known as ykarel|away | 16:03 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Update to gitea 1.12.0 https://review.opendev.org/729659 | 16:10 |
| clarkb | infra-root ^ fyi | 16:10 |
| clarkb | I'm going to pop out on a bike ride before the sun makes things too warm | 16:11 |
| fungi | i was hoping to mow the lawn today, first time in a week there's not been standing water in the yard, but rain's coming back :/ | 16:12 |
| openstackgerrit | Merged opendev/system-config master: Update refstack reference after rename https://review.opendev.org/735425 | 16:15 |
| mordred | clarkb: hrm. that's not a bad idea | 16:16 |
| mordred | clarkb: are we backing up the git repos though? | 16:17 |
| clarkb | We should be | 16:21 |
| clarkb | we exclude not include backup input | 16:21 |
| fungi | yeah, i was just poking around on review.o.o checking for the excludes | 16:22 |
| fungi | though i don't recall us specifically excluding the path to the git repos | 16:22 |
| mordred | still - it might be a little more dangerous - because we'd have to make sure we ran ansible on review-test after the backup restore to be sure we haven't written review.o.o credentials or config | 16:22 |
| mordred | like - that actually makes me more apprehensive than the current sync script | 16:23 |
| mordred | (althogh there is obvious benefit) | 16:23 |
| clarkb | ya may want to limit it to just the git repos and the sql dump and nothing else | 16:23 |
| clarkb | basically replace your rsyncs with bup | 16:23 |
| mordred | yea | 16:23 |
| mordred | also - fwiw, I think our log rotation is doing something weird with mysql backups | 16:24 |
| mordred | -rw-r--r-- 1 root root 0 Jun 19 06:25 gerrit.sql.gz | 16:24 |
| fungi | i suppose we can be reasonably sure gids/uids match between those servers, so won't need to chown the restored repos? | 16:24 |
| fungi | mordred: it is weird, but it's explainable | 16:25 |
| mordred | gerrit.sql.gz is zero bytes - but the others are all correct | 16:25 |
| fungi | logrotate creates an empty file | 16:25 |
| fungi | then mysqldump later writes to that file | 16:25 |
| mordred | nod. just as long as we're not backing up only the gerrit.sql.gz to bup :) | 16:25 |
| fungi | and then logrotate renames it out of the way and creates a new empty file | 16:25 |
| fungi | so yes, the most recent mysqldump will be in gerrit.sql.gz.1 or whatever | 16:26 |
| fungi | but yeah, this seems like a worthwhile experiment because at the very least it'll help us confirm whether our backups are good | 16:32 |
| fungi | also if we want to be extra safe we can run another mysqldump and bup pass immediately before we try the automation, just to be sure we have a little data loss as possible if something does blow up the production server somehow | 16:32 |
| fungi | er, AS little data loss as possible | 16:33 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror https://review.opendev.org/736996 | 16:34 |
| *** ChanServ has quit IRC | 16:59 | |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror https://review.opendev.org/736996 | 16:59 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror https://review.opendev.org/736996 | 17:00 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: Make the base role and playbook compatible with CentOS https://review.opendev.org/737043 | 17:01 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror https://review.opendev.org/736996 | 17:01 |
| mtreinish | clarkb, fungi, mordred: do you have any experience with: https://github.com/pypa/setuptools_scm/ ? Someone just submitted a bunch of PRs to projects I maintain (for example https://github.com/Qiskit/retworkx/pull/89 ) that proposes switching to it | 17:10 |
| mtreinish | I tend to be skeptical of anything that relies on pyproject.toml because it's been pretty fragile in my experience for what little I've used it | 17:11 |
| fungi | mtreinish: i haven't used it at all recently. i think maybe setuptools_scm was one of the contemporary solutions we looked at years ago before writing pbr | 17:15 |
| mordred | yup. we replaced use of it with pbr | 17:16 |
| mordred | or - at least its predecessor | 17:17 |
| fungi | i now don't recall what our problems with it were, but odds are it's improved in the many years since | 17:17 |
| mtreinish | heh, well that was my first reaction looking at it, "this is like pbr" :) | 17:17 |
| mordred | yeah - I think they mocked us for writing pbr, then went and wrote the same thing but slightly different iside their little in-crowd of people | 17:17 |
| mordred | I dont' have any negative feelings about that at all clearly :) | 17:18 |
| fungi | but yeah, pbr was effectively a fork of distutils2, right? | 17:18 |
| mordred | yup | 17:18 |
| mordred | infra-root: I'm afk for the next several hours - gotta go get the kittens vaccinated | 17:18 |
| fungi | we obviously picked the wrong in-crowd | 17:18 |
| fungi | mordred: have fun | 17:18 |
| mordred | mtreinish: but in direct answer - I have not played with that in earnest in quite some time because pbr is pretty good at what it does and I don't see the need to | 17:18 |
| mtreinish | mordred: ok, thanks | 17:19 |
| mtreinish | have fun with your cats | 17:19 |
| mordred | mtreinish: that said - I do think anything that uses declarative metadata is better than having _anything_ in setup.py | 17:19 |
| mordred | so I'd be inclined to accept that pr because it is a definite improvement over code in setup.py | 17:20 |
| mordred | mtreinish: thanks! there are ... many of them :) | 17:20 |
| mordred | (and then you can tell us whether setuptools_scm works :) ) | 17:20 |
| mtreinish | I definitely agree with that. I just get worried with changes for retworkx and a couple of other projects with compiled extensions (cython and c++ instead of rust) that rely on build time dependencies being present before setup.py is run | 17:22 |
| mtreinish | because it was really finicky to get things in a place where you could install those from sdist without it erroring because something was missing (despite having a build dep set in the metadata) | 17:22 |
| mtreinish | which is why the setup.py in that pr subprocesses out to pip install setuptools-rust, which was the best answer I found | 17:23 |
| *** ChanServ has joined #opendev | 17:37 | |
| *** tepper.freenode.net sets mode: +o ChanServ | 17:37 | |
| *** mlavalle has quit IRC | 17:40 | |
| openstackgerrit | Rafael Folco proposed openstack/diskimage-builder master: DNM: Debug py3 on dib 7 https://review.opendev.org/736421 | 17:49 |
| *** jaicaa_ has quit IRC | 18:11 | |
| *** mlavalle has joined #opendev | 18:11 | |
| *** jaicaa has joined #opendev | 18:19 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Fix and test multiarch docker builds in a release pipeline https://review.opendev.org/737059 | 18:27 |
| openstackgerrit | Sergii Golovatiuk proposed zuul/zuul-jobs master: Install source Python for supported OSes https://review.opendev.org/737060 | 18:27 |
| *** calcmandan has quit IRC | 18:31 | |
| *** calcmandan has joined #opendev | 18:32 | |
| clarkb | infra-root I'm thinking it might be a good idea to deploy https://review.opendev.org/#/c/729659/ on a single gitea backend | 18:35 |
| clarkb | largely to see what the memory use looks like with the new caching | 18:35 |
| clarkb | I'm not quite sure how to do that yet, but if I can get some reviews on the chnage (without approvals) I can sort that out this afternoon | 18:35 |
| clarkb | I think it will be something like put gitea01 in the emergency file, edit the docker compose file to pull the image for that change, docker-compose down then up? | 18:36 |
| fungi | that sounds like a reasonable goal... maybe you can stick gitea01 into the emergency disable list and locally edit the compose file? | 18:36 |
| fungi | heh, yep | 18:36 |
| clarkb | also I'm going to grab some quick render times for nova and neutron and friends to compare against post upgrade | 18:37 |
| clarkb | https://etherpad.opendev.org/p/gitea-1.12.0-upgrade-notes put some timing notes there. Its not terribly scientific but should give us an idea if its better or not | 18:47 |
| clarkb | fungi: the other way we could do it is put 02-08 into emergency and then land the change and have it update 01 only | 19:00 |
| fungi | yep, i thought about that possibility as well | 19:01 |
| fungi | though it does result in more to back out if something doesn't go as expected | 19:02 |
| clarkb | ya probably just editing 01 pre merge is simplest | 19:03 |
| clarkb | fungi: maybe you can give that change a look over and if you and someone else think it looks good I'll start on 01 | 19:07 |
| fungi | yeah, sorry, had to step away to shape pizza dough, back for a while now | 19:31 |
| clarkb | no worries I'm trying to figure out what lunch looks likehere and now I want pizza | 19:32 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Fix zuul-jobs-test-ensure-python-pyenv files https://review.opendev.org/737068 | 19:32 |
| clarkb | we're seeing peoplecontinue to use meetpad | 19:45 |
| clarkb | whih does make me think that sunsetting pbx is the right move | 19:46 |
| clarkb | (ironic used it recently and interop is planningto use it) | 19:46 |
| fungi | yeah, makes sense. also means we should maybe talk about whether to keep the account for the dial-in line, and how we might want to incorporate it into jitst-meet if so | 20:01 |
| clarkb | fungi: those comments explaining deltas from upstream html templates are good feedback I'll see about adding that to a followup | 20:20 |
| *** aannuusshhkkaa has joined #opendev | 20:24 | |
| fungi | yeah, not critical, just things which sprang to mind as i was reading through, certainly nothing to hold up that patch | 20:25 |
| *** _mlavalle_1 has joined #opendev | 20:25 | |
| *** mlavalle has quit IRC | 20:26 | |
| *** priteau has quit IRC | 20:36 | |
| clarkb | mordred: when cats are happy do you think you could give the gitea 1.12.0 change a look over again? If it still looks good I'll get gitea01 updated | 20:38 |
| *** hashar has joined #opendev | 20:54 | |
| *** josephillips has joined #opendev | 21:04 | |
| josephillips | hey | 21:04 |
| josephillips | guys | 21:04 |
| josephillips | question | 21:04 |
| *** factor has joined #opendev | 21:04 | |
| josephillips | if i want to perform a fork from a actual openstack project can i keep it on opendev? | 21:04 |
| josephillips | or you recommend use github for this | 21:04 |
| josephillips | ? | 21:04 |
| clarkb | josephillips: I think our strong suggestion would be to work with that project directly | 21:05 |
| clarkb | we've historically not prevented forks for packaging purposes (though that also means we've got like 2 extra copies of a bunch of repos that are completely dead now) | 21:05 |
| josephillips | i wish work on charms repo | 21:06 |
| josephillips | but for things i probably will use only for us | 21:06 |
| clarkb | another thing to consider is if the fork is intended to be a long term situation. If not then working with the existing project becomes more important imo | 21:07 |
| clarkb | josephillips: what is the goal here? | 21:07 |
| josephillips | well modify the original charm to add features that will be requiered for own infrastructure , my goal is keep a fork in opendev and merge from new changes to my fork when they commit to they master branch | 21:08 |
| clarkb | josephillips: and that can't be done with the existing project? | 21:09 |
| josephillips | i tried but they told me that these changes need to be discussed with the cannonical team etc etc | 21:10 |
| josephillips | and this why i prefer keep a fork | 21:10 |
| josephillips | and probably will be thing of months | 21:10 |
| clarkb | got it. In that case a proper fork is probably not the end of the world. Another potential option depending on how much you think you'll diverge is to use proposed state in gerrit | 21:10 |
| clarkb | (which is definitely less nice thant git clone and go) | 21:11 |
| josephillips | for something that i already have working on my infrastructure im just looking a way to keep updated my charm and when if someone else look utils then can use it too | 21:11 |
| clarkb | josephillips: ya so you can fetch gerrit changes and use them directly but that won't be already on the master branch and ready to go after a clone (would need a separate fetch and checkout) | 21:12 |
| clarkb | that's the very lightweight fork option where you know they won't merge things in the near term but you can still consume the change from refs/changes/xy/abcxy | 21:12 |
| josephillips | you mean commit to the original project my changes when they perform the review just will rejected but keep the changes on this way? | 21:14 |
| clarkb | ya | 21:15 |
| clarkb | I don't know that it is the best option here, but wanted to point out it is an option | 21:15 |
| josephillips | yeah i know | 21:15 |
| josephillips | im just thinking if is fair with the reviewers do that | 21:15 |
| josephillips | lol | 21:16 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add tests for upload-docker-image https://review.opendev.org/735402 | 21:29 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Fix and test multiarch docker builds in a release pipeline https://review.opendev.org/737059 | 21:29 |
| corvus | clarkb: i'm thinking of landing zk now: https://review.opendev.org/736309 | 21:31 |
| corvus | load isn't crazy; so i'll land that then shutdown and bringup all the zk hosts real quick | 21:31 |
| clarkb | corvus: rgr. I'm around if I can help | 21:32 |
| clarkb | I started looking at my gitea plan more and realized we don't have the change specific tag on github? | 21:32 |
| clarkb | er docker hob | 21:32 |
| clarkb | https://hub.docker.com/r/opendevorg/gitea/tags | 21:32 |
| clarkb | so not sure that plan is going to work and am poking around at it to see what options I've got | 21:33 |
| corvus | clarkb: yeah, we clean those up | 21:33 |
| clarkb | I think build the image myself or do the 02-08 in emergency and have just 01 update | 21:33 |
| fungi | i'm mostly around, though going to see if i can still get to the lawn | 21:33 |
| corvus | clarkb: you want to manually deploy a gitea image we built in check or gate? | 21:33 |
| corvus | clarkb: if so, you can use the intermediate registry | 21:34 |
| corvus | clarkb: link me to the change that built the image you want | 21:34 |
| clarkb | corvus: the previously discussed plan would've been check, but now I'm thinking just approving the change with 02-08 in emergency is the path forward and that will use gate's promoted image as usual | 21:34 |
| clarkb | corvus: https://review.opendev.org/#/c/729659/ | 21:34 |
| corvus | clarkb: let's follow through the check thing first i think it might work | 21:34 |
| clarkb | corvus: k | 21:34 |
| corvus | clarkb: docker://insecure-ci-registry.opendev.org:5000/opendevorg/gitea:0e136fdfd57a47eb8ca3d9ddd9794d7e_latest | 21:35 |
| corvus | clarkb: docker://insecure-ci-registry.opendev.org:5000/opendevorg/gitea-openssh:0e136fdfd57a47eb8ca3d9ddd9794d7e_latest | 21:35 |
| clarkb | corvus: cool so I can docker pull that | 21:35 |
| corvus | clarkb: those are from https://zuul.opendev.org/t/openstack/build/0e136fdfd57a47eb8ca3d9ddd9794d7e | 21:35 |
| corvus | clarkb: right click on the artifacts and copy their urls | 21:35 |
| corvus | clarkb: yep | 21:35 |
| clarkb | then the local image table will find it when docker-compose restarts it | 21:35 |
| corvus | should be able to confirm by just running locally on a workstation too | 21:35 |
| clarkb | ya I'll start there | 21:36 |
| corvus | "docker run insecure-ci-registry.opendev.org:5000/opendevorg/gitea:0e136fdfd57a47eb8ca3d9ddd9794d7e_latest" does something for me :) | 21:36 |
| clarkb | josephillips: to recap I think the prefernece is to at least have the groups talk about it first and ensure we aren't avoiding potential for collaboration or adoption of projects that need help. It sounds like you've done that and the two ideas are still independent so a fork isn't the end of the world | 21:37 |
| corvus | mnaser: ^ fyi since you're doing more with images -- the intermediate registry can be a useful debugging tool; lets you manually run what was built in check or gate. | 21:37 |
| clarkb | corvus: heh I was going to set up a docker-compose file. I think I'll still do that but good to know the super simple case is happy | 21:37 |
| mnaser | ooou, that's REALLY neat | 21:37 |
| corvus | honestly, if openstack uses images more and more, this could be an excellent way for devs to reproduce problems. | 21:37 |
| corvus | mnaser: yeah, it was one of two main reasons we wrote zuul-registry | 21:38 |
| corvus | that plus the split-horizon thing with it shadowing multiple upstream registries | 21:38 |
| corvus | our first intermediate registry, which was just the upstream docker registry, couldn't handle the split authentication needed for auth-write / anon-read | 21:39 |
| corvus | clarkb: well i mean the 'docker run' pulled an image for me, i didn't try to use gitea or anything :) | 21:40 |
| clarkb | ya I'm putting those images into the docker compose file to see if that just works (tm) (I'm also commenting out the volumes because I don't want to set those up on my laptop but this should at least confirm docker-compose does what we expect) | 21:43 |
| clarkb | cool docker-compose is happy with that but it results in errors due to perms but thats fine that is due to me commenting out the bind mounts | 21:46 |
| clarkb | corvus: fungi: should I go ahead and put gitea01 in emergency now and run gitea off the images corvus linked above? or would we like to see mordred review the change first? | 21:47 |
| clarkb | I'm happy to wait for zk thing sfirst too | 21:48 |
| fungi | i think it's fine to go ahead, it's just a test anyway, no real commitment and if something goes sideways we can roll back | 21:49 |
| clarkb | fungi: yes, but rollback may mean rebuilding on the previous version | 21:49 |
| clarkb | I'm not sure how safe it is to go backwards with the db | 21:49 |
| clarkb | but we do have 7 so we can do that with little impact | 21:49 |
| fungi | right, worst case raze the earth as far as 01 is concerned | 21:51 |
| corvus | i agree with fungi; and no need to wait for zk | 21:52 |
| clarkb | k putting 01 in emergency now | 21:53 |
| *** hashar has quit IRC | 21:54 | |
| clarkb | stopping services on 01 now | 21:55 |
| clarkb | gitea01 services are up | 21:59 |
| clarkb | it seems to be working | 21:59 |
| clarkb | loading nova root first time took 8 seconds. Second time 1 second give or take | 22:00 |
| clarkb | previously it was just under 4 seconds | 22:00 |
| clarkb | definitely seems like cold is still slow but warm is quicker | 22:01 |
| fungi | improvement! | 22:01 |
| clarkb | wich isn't surprising given its a cache that is used | 22:01 |
| clarkb | the other thing I want to monitor is memory use | 22:02 |
| clarkb | which may be a check cacti on monday item and if happy land the change | 22:02 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=66633&rra_id=all is the graph to monitor | 22:04 |
| clarkb | #status Log put gitea01 in the emergency file and manually upgraded gitea to 1.12.0 there. This allows us to check the performance updates and memory use after the addition of caching. | 22:05 |
| openstackstatus | clarkb: finished logging | 22:05 |
| fungi | yeah, i don't have confidence our workload between now and monday will be sufficient to really identify memory increases | 22:06 |
| clarkb | at the very least it hasn't immediately imploded and it seems to be quicker for our large repos. I'll call that a win :) | 22:09 |
| clarkb | https://gitea01.opendev.org:3000/openstack/nova/activity is a new thing that we might want to pull out of the UI | 22:11 |
| clarkb | its not a direct problem but it doesn't show anything useful either and people may read into that too much | 22:11 |
| openstackgerrit | Merged opendev/system-config master: Zookeeper: listen on plain and TLS ports https://review.opendev.org/736309 | 22:12 |
| clarkb | oh actually it may be more useful than I thought | 22:13 |
| clarkb | ya I think we can keep it | 22:13 |
| corvus | oh right, this change runs all the deploy jobs | 22:17 |
| corvus | so it's going to be hours until it gets around to running infra-prod-service-zuul | 22:18 |
| corvus | mordred: ^ what did we say we were going to do about that? | 22:18 |
| corvus | i guess i'm done for the day, and maybe we can restart zk sometime this weekend :/ | 22:19 |
| clarkb | corvus: I think the idea was to start being specific about which bits of host vars and inventory map to which jobs so that changes to one small corner of that information doens't trigger everything | 22:20 |
| corvus | clarkb: i agree we should yank activity | 22:20 |
| clarkb | corvus: ya its like almost useful | 22:21 |
| corvus | clarkb: yeah, it's technically correct | 22:23 |
| *** _mlavalle_2 has joined #opendev | 22:23 | |
| corvus | but i'm not sure 'excluding merges' is the right way to characterize our development :) | 22:23 |
| corvus | the branch list is not sorted; that seems weird. | 22:24 |
| corvus | rather, it's not alpha-sorted | 22:24 |
| *** _mlavalle_1 has quit IRC | 22:24 | |
| corvus | that's not a change though, just something i noticed now | 22:25 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Small repo template cleanups in Gitea https://review.opendev.org/737098 | 22:27 |
| clarkb | fungi: corvus ^ I did the repo template cleanups in a followup to help make it more clear that the first thing is what we're testing | 22:27 |
| fungi | thanks, i needed an excuse to sit down and not push the mower for a few minutes anyway | 22:28 |
| *** _mlavalle3 has joined #opendev | 22:29 | |
| *** _mlavalle_2 has quit IRC | 22:31 | |
| *** _mlavalle3 has quit IRC | 22:57 | |
| *** DSpider has quit IRC | 22:58 | |
| *** tosky has quit IRC | 23:45 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!