| corvus | we need to have our chinese summits for letters with consonants, not vowels. | 00:01 |
|---|---|---|
| clarkb | twice we've failed at that alignment problem :) | 00:01 |
| corvus | ya... i'm afraid we're going to end up with another 'icehouse' :| | 00:02 |
| corvus | if we're lucky | 00:02 |
| clarkb | now just waiting on github and k8s gitea to replicate openstack/project-config | 00:02 |
| clarkb | (if we don't want to wait for the other 17k or so events to process | 00:02 |
| fungi | at least icehouse was viable courtesy of hk's long history as an english conquest^H^H^H^H^H^Hlony | 00:03 |
| fungi | we won't have so much of that to fall back on for u | 00:03 |
| mordred | corvus: I think we should just go full-on insensitivity and name it ulaanbaatar | 00:04 |
| clarkb | should we see what it looks like to update project-config on review.o.o now? | 00:05 |
| clarkb | then do some jeepyb sanity checking before we restart? | 00:05 |
| mordred | clarkb: ++ | 00:05 |
| corvus | clarkb: yeah, looks like openstack/project-config has cleared the queue | 00:05 |
| *** jamesmcarthur has quit IRC | 00:05 | |
| clarkb | github and k8s gitea still queued | 00:05 |
| clarkb | but cgit and gitea proper are done | 00:06 |
| corvus | yeah, but we won't be pulling from there | 00:06 |
| clarkb | I'm going to stop the tail on review.o.o's screen | 00:06 |
| corvus | ++ | 00:06 |
| clarkb | and we can do a git remote update and see how that goes | 00:06 |
| fungi | cool | 00:06 |
| clarkb | let me read really quickly on how manage-projects looks at that content | 00:06 |
| mordred | the cache isn't going to be valid for a bunch of the projects | 00:07 |
| *** Lucas_Gray has quit IRC | 00:07 | |
| mordred | so it'll actually talk to github | 00:07 |
| clarkb | /home/gerrit2/projects.yaml | 00:07 |
| clarkb | we copy it from /etc/project-config to there I think | 00:07 |
| clarkb | then if that file updates manage-projects runs | 00:07 |
| mordred | I believe ou are correct | 00:07 |
| clarkb | I'm going to do a git remote update in /etc/project-config now | 00:08 |
| clarkb | I might need to be root | 00:08 |
| *** michael-beaver has quit IRC | 00:08 | |
| fungi | probably, yes | 00:08 |
| clarkb | yup I'm root and running now in a new window on the screen ( does multiattach change windows with me when I change or do you all see the old window?) | 00:08 |
| mordred | I'm in the old window | 00:09 |
| fungi | looks like it doesn't change automatically for us, no | 00:09 |
| mordred | but I'm imagining you doing fun things | 00:09 |
| fungi | ctrl+a,n | 00:09 |
| *** Lucas_Gray has joined #openstack-infra | 00:09 | |
| clarkb | I did a remote update which pulled some stuff. then did a git show origin/master to see what it pulled | 00:09 |
| clarkb | the next step is to reset master to origin master | 00:10 |
| fungi | looks properly up to date there, yes | 00:10 |
| clarkb | hrm it looks like it checks it out as a detached head? | 00:10 |
| clarkb | anways I'll update master to origin/master because that seems saner | 00:11 |
| clarkb | unless we want to just checkout origin/master like what puppet appears to do | 00:11 |
| clarkb | that reset look right i can never remember if I need to do --hard origin/master or --hard origin master | 00:11 |
| mordred | nah. I'm sure puppet will do its thing when it does it | 00:11 |
| clarkb | ? | 00:12 |
| mordred | origin/master | 00:12 |
| clarkb | ok master is up to date now | 00:12 |
| corvus | hopefully we didn't actually move fit-review :) | 00:12 |
| mordred | hahaha | 00:13 |
| clarkb | ha I looked I believe it is just a commit message issue | 00:13 |
| corvus | ("more reps!") | 00:13 |
| fungi | hah | 00:13 |
| clarkb | ok now to copy the gerrit/projects.yaml and the acls | 00:13 |
| fungi | i missed the typo in rapidly assembling that commit, sorry | 00:13 |
| corvus | i'm not sorry; it makes me giggle | 00:14 |
| fungi | plain brown wrapper and mislabeled | 00:14 |
| clarkb | that copy look right? | 00:14 |
| corvus | y | 00:14 |
| fungi | i think so | 00:14 |
| clarkb | ok now the incantation to copy acls | 00:15 |
| fungi | been ages since i've seen midnight commander used | 00:16 |
| fungi | oh | 00:17 |
| fungi | speaking of typos! | 00:17 |
| clarkb | my turn for a typo | 00:17 |
| fungi | yeah, that ought to work now | 00:17 |
| clarkb | corvus: mordred ^ that command look good to you too? I think I need to chown gerrit2:gerrit2 when I'm done too | 00:17 |
| mordred | ++ | 00:17 |
| mordred | chown -R of course | 00:18 |
| corvus | ++ | 00:18 |
| clarkb | ya | 00:18 |
| *** aaronsheffield has quit IRC | 00:19 | |
| *** ijw has quit IRC | 00:20 | |
| clarkb | I feel like I'm missing something | 00:20 |
| *** ijw has joined #openstack-infra | 00:20 | |
| corvus | acs/openstack-infra/git-review.config exists | 00:21 |
| corvus | er acls | 00:21 |
| clarkb | ya I think the mv didn't do anything? | 00:21 |
| fungi | /etc/project-config/gerrit/acls/opendev/git-review.config is there | 00:21 |
| clarkb | there is no acls.old but I also didn't get an error | 00:21 |
| *** auristor has quit IRC | 00:21 | |
| clarkb | shoudl I just do the mv to start then the copy and see if that produces saner output? | 00:21 |
| corvus | did you clear the # when you ran it? | 00:22 |
| clarkb | I thought I did | 00:22 |
| clarkb | but maybe I didn't let me try again being extra sure of that | 00:22 |
| fungi | lgtm now | 00:23 |
| clarkb | ya that must've been it derp | 00:23 |
| clarkb | /home/gerrit2/projects.ini is the other related file that puppet manages | 00:23 |
| clarkb | I think that looks ok as is (since github shouldn't work except for the openstack projects) | 00:24 |
| *** ijw has quit IRC | 00:24 | |
| corvus | ++ | 00:24 |
| corvus | note that /opt/lib/git is a replication target | 00:25 |
| corvus | so if m-p expects to find any current data in there, we may need to think about that | 00:25 |
| *** ijw has joined #openstack-infra | 00:25 | |
| clarkb | mordred: ^ you added that stuff to manage-projects any thoughts | 00:25 |
| clarkb | one thing we can try is running it on a single project say opendev/git-review | 00:25 |
| clarkb | before we do the full list | 00:26 |
| mordred | nope. I have absolutely no memory of /opt/lib/git interactions- looking | 00:26 |
| corvus | local-git-dir=/opt/lib/git | 00:26 |
| corvus | is the thing in the config file | 00:26 |
| corvus | so maybe a grep of that in the jeepyb source would turn things up | 00:26 |
| mordred | it just creates the dir | 00:26 |
| mordred | so it's about creating the local mirror target | 00:27 |
| mordred | so that gerrit can replicate there | 00:27 |
| corvus | got it. so that should be ok. | 00:27 |
| mordred | yeah | 00:27 |
| corvus | that all got handled by the rename script | 00:27 |
| clarkb | alright I still think running against a single project to start is a good idea so lets do that with opendev/git-review | 00:28 |
| mordred | ++ | 00:28 |
| fungi | yep, good thinking | 00:28 |
| clarkb | how does the command look | 00:28 |
| clarkb | oh! | 00:29 |
| clarkb | I didn't chown the acls | 00:29 |
| mordred | chown those acls | 00:29 |
| clarkb | good? | 00:29 |
| *** ijw has quit IRC | 00:29 | |
| mordred | ++ | 00:29 |
| clarkb | ok now I'm ready to run manage-projects | 00:30 |
| *** hwoarang has quit IRC | 00:30 | |
| clarkb | the log file looks ok but you all should look it over too | 00:31 |
| corvus | looks like it correctly nooped | 00:31 |
| corvus | (i did look at the log) | 00:32 |
| clarkb | mordred: fungi ^ if that looks good to you I think the next step is a full run | 00:33 |
| mordred | do it | 00:33 |
| fungi | yeah, seems we're on track | 00:33 |
| clarkb | note I'm changing the log file namepath | 00:34 |
| clarkb | and starting it now | 00:34 |
| *** hwoarang has joined #openstack-infra | 00:34 | |
| corvus | still looking good | 00:35 |
| clarkb | ya | 00:35 |
| *** jamesmcarthur has joined #openstack-infra | 00:35 | |
| corvus | so when this is done, we stage the system-config change on bridge (it's probably still in place) then kick? or something like that? | 00:36 |
| clarkb | corvus: yes. Then I think we put your zuul config updates for the config parser config and kick zuul | 00:36 |
| clarkb | and also before we restart gerrit we can remove the cgit and k8s gitea replication | 00:37 |
| clarkb | note the puppet kick is likely to remove the firewall rules if we run base | 00:37 |
| corvus | clarkb: we can stop gerrit before the kick and it won't start it, right? | 00:38 |
| clarkb | that is my understanding | 00:38 |
| clarkb | corvus: actually did you update iptables manually or via the config files? | 00:38 |
| corvus | config files | 00:39 |
| corvus | so i do expect it to be reverted | 00:39 |
| clarkb | k then ya running puppet will update the rules I think | 00:39 |
| clarkb | ya we run base | 00:39 |
| clarkb | we could remove base.yaml from our run | 00:39 |
| corvus | i'm inclined to block access via the firewall again until we're ready to bring zuul back up | 00:39 |
| fungi | agreed | 00:39 |
| clarkb | if we remove base.yaml from our ansible kick then I don't think firewalls will be touched | 00:40 |
| corvus | is that easy? it's also easy for me to just redo the rules | 00:40 |
| clarkb | ya just remove it out of the script or make a new script or call ansible manually | 00:40 |
| corvus | ok that's the plan then | 00:40 |
| corvus | so about zuul, it's the connection info that needs to be updated, right? | 00:41 |
| corvus | so that's another stage a change on bridge and kick? | 00:41 |
| clarkb | correct | 00:41 |
| corvus | why don't i go ahead and cherry-pick that onto bridge's system-config? | 00:42 |
| clarkb | wfm | 00:42 |
| *** jamesmcarthur has quit IRC | 00:42 | |
| corvus | 7c0358a1e (HEAD -> master) Update gerrit connection in Zuul | 00:44 |
| corvus | 3b9b29741 Redirect review.openstack.org to review.opendev.org | 00:44 |
| corvus | 6747cf236 (origin/master, origin/HEAD) Merge "Update nodepool servers to puppet 4" | 00:44 |
| corvus | that's the current state of /opt/system-config on bridge | 00:44 |
| clarkb | that looks right to me | 00:44 |
| corvus | i'm updating the etherpad alound lines 54-60 that look right? | 00:45 |
| *** armax has quit IRC | 00:45 | |
| clarkb | current plan with us running 4.5 now 4.5) manually update project-config on review.o.o and run jeepy manage-projects 5) kick 6) edit replication config to remove k8s gitea and cgit 7) restart gerrit 8) kick zuul | 00:45 |
| * clarkb compares ^ to etherpad | 00:45 | |
| fungi | yeah, commit looks right on bridge, though no humorous commit message typos | 00:46 |
| corvus | fungi: nobody's perfect | 00:46 |
| mordred | oh for the love of ... sorry - had internet tethering derps | 00:46 |
| clarkb | corvus: one note otherwise that seems to match ^ | 00:46 |
| corvus | clarkb: oh yeah, let's bump that down | 00:47 |
| corvus | that was already there and nearby | 00:47 |
| corvus | i think i moved it one line and it became my color | 00:47 |
| fungi | i've had to switch 'puters... what's the url to the pad again? | 00:47 |
| corvus | https://etherpad.openstack.org/p/the-great-opendev-git-migration | 00:48 |
| fungi | thanks! | 00:48 |
| fungi | i should have memorized it because the title is too great | 00:48 |
| fungi | yeah, lgtm | 00:49 |
| clarkb | heh just got cert renewal notices for git.airshipit.org and git.starlingx.io | 00:49 |
| mordred | SO IMPORTANT | 00:49 |
| clarkb | we are nearing the end of the projects.yaml file | 00:50 |
| corvus | the "z"s | 00:50 |
| mordred | zzzzzzz | 00:50 |
| corvus | i think it's done | 00:50 |
| clarkb | and done | 00:51 |
| mordred | and nothing caught on fire | 00:51 |
| mordred | that's so exciting | 00:51 |
| clarkb | no that looked sane | 00:51 |
| corvus | how did we grow to 13026 tasks | 00:51 |
| corvus | last i looked it was 10345 tasks | 00:51 |
| *** jamesmcarthur has joined #openstack-infra | 00:52 | |
| mordred | corvus: when a task and a task love each other very much ... | 00:52 |
| clarkb | the only git pushes are see are for recordsasnible meta config updates | 00:52 |
| corvus | windmill-ops is at the end of the list... and i don't see any m-p pushes for it | 00:53 |
| fungi | is it possible that some gerrit tasks can trigger other tasks? | 00:53 |
| clarkb | corvus: are they anotated (retry) ? | 00:53 |
| corvus | clarkb: no | 00:53 |
| *** seyeongkim has joined #openstack-infra | 00:53 | |
| corvus | fungi: yes but i don't think that should be happening now with the system quiesced | 00:53 |
| clarkb | corvus: git log on the review_site repo for windmill ops shows the opendev migration commit | 00:56 |
| corvus | yeah and "git show refs/meta/config" shows nothing new | 00:56 |
| corvus | those are the 2 things i'd be worried about | 00:56 |
| clarkb | and manage projects only shows those two pushes of meta/config | 00:56 |
| clarkb | I think its ok | 00:57 |
| corvus | ++ | 00:57 |
| *** jamesmcarthur has quit IRC | 00:57 | |
| corvus | oh i think m-p replicates every project :) | 00:57 |
| clarkb | oh bah | 00:58 |
| mordred | of course | 00:58 |
| corvus | so yeah, that's expected and harmless | 00:58 |
| mordred | whoops! | 00:58 |
| clarkb | alright I'm good with proceeding with the plan then :) | 00:58 |
| mordred | me too | 00:58 |
| corvus | clarkb: con is still yours | 00:58 |
| clarkb | ok why don't I start putting together the ansible command on bridge screen | 00:58 |
| clarkb | can someone else stop gerrit and update replication config? | 00:59 |
| corvus | i will stop and upgrade repl | 00:59 |
| *** gyee has quit IRC | 01:00 | |
| clarkb | mordred: did you catch the bit about not running base.yaml so we don't update the firewall rules? | 01:00 |
| clarkb | mordred: do you think that is unsafe for any reason? | 01:00 |
| corvus | dude. *emacs* is telling me i'm using obsolete commands. i feel old. | 01:00 |
| mordred | clarkb: I do not think it's unsafe for any reason | 01:00 |
| mordred | corvus: WOW | 01:00 |
| clarkb | corvus: obsolete from xenial era emacs too :) | 01:01 |
| mordred | corvus: when you upgrade to bionic, you might not be able to computer | 01:01 |
| corvus | ok i have removed gitea-k8s and git01--git08 from the replication config | 01:01 |
| clarkb | I think the bridge screen ansible-playbook command is ready for review | 01:02 |
| *** jamesmcarthur has joined #openstack-infra | 01:02 | |
| corvus | lgtm | 01:02 |
| clarkb | mordred: fungi ianw ^ want to look? | 01:03 |
| corvus | i have confirmed that gerrit is not running | 01:03 |
| fungi | yeah, i skimmed the playbook, seems reasonable | 01:03 |
| clarkb | oh its not review.openstack.org its review01.openstack.org right? | 01:03 |
| corvus | i've also saved a copy of my new replication config because i assume kick will overwrite it. | 01:04 |
| fungi | good point, ansible knows it by the latter | 01:04 |
| clarkb | yup its review01.openstack.org udpating | 01:04 |
| clarkb | corvus: oh good point | 01:04 |
| clarkb | ok ready? | 01:04 |
| mordred | clarkb: looks good | 01:04 |
| corvus | t | 01:04 |
| clarkb | here goes | 01:04 |
| corvus | you did that thing again :) | 01:04 |
| corvus | better safe than sorry tho | 01:04 |
| *** openstackgerrit has quit IRC | 01:05 | |
| corvus | is "TASK [puppet : create environment.conf from template]" really taking a long time? | 01:06 |
| fungi | i guess so | 01:06 |
| clarkb | no you don't see the log until ti finishes | 01:06 |
| fungi | done now though | 01:06 |
| clarkb | yup and the puppet error is ok I think | 01:06 |
| corvus | fatal: [review01.openstack.org]: FAILED! => {"changed": false, "msg": "/usr/bin/timeout -s 9 30m /opt/puppetlabs/bin/puppet apply /opt/system-confi | 01:06 |
| corvus | g/production/manifests/site.pp --logdest syslog --environment 'production' --no-noop --detailed-exitcodes failed with return code: 6", "rc": 6, "st | 01:06 |
| corvus | derr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} | 01:06 |
| clarkb | it tried to run manage projects because it updated perms on the acl files | 01:07 |
| clarkb | btu that failed because port 29418 is not listening | 01:07 |
| *** jamesmcarthur has quit IRC | 01:07 | |
| corvus | agreed | 01:07 |
| clarkb | but it also may have removed the acls we want | 01:07 |
| clarkb | :/ | 01:07 |
| corvus | Apr 20 01:06:12 review01 puppet-user[66376]: (/Stage[main]/Openstack_project::Gerrit/File[/home/gerrit2/acls/ttygroup]/ensure) removed | 01:08 |
| fungi | huh, why? | 01:08 |
| corvus | good thing that was safe | 01:08 |
| mordred | we updated the copy of project-config on review right? | 01:08 |
| clarkb | I did and puppet put it back | 01:09 |
| clarkb | ok so where the heck is it getting its copy from | 01:09 |
| corvus | is it doing the project-config ref thing? | 01:09 |
| corvus | where it checks out the ref that's on bridge? | 01:09 |
| fungi | pushing from bridge i guess? | 01:09 |
| mordred | maybe just pull latest to bridge too? | 01:09 |
| clarkb | corvus: I think that may be it | 01:09 |
| clarkb | ok this is safe as long as we don't start gerrit | 01:10 |
| corvus | https://opendev.org/opendev/system-config/src/branch/master/playbooks/remote_puppet_git.yaml#L1-L11 | 01:10 |
| corvus | hrm | 01:11 |
| corvus | but remote_puppet_adhoc doesn't do that | 01:11 |
| corvus | where's our puppet role? | 01:12 |
| clarkb | Apr 20 01:05:09 review01 puppet-user[66376]: (/Stage[main]/Project_config/Vcsrepo[/etc/project-config]/ensure) 054c0a0712d9a4dee8cd76cd430f07b1e4966a26 | 01:12 |
| clarkb | corvus: ansible-role-puppet | 01:12 |
| clarkb | so it does seem like we told puppet to ensure that hash | 01:12 |
| *** hwoarang has quit IRC | 01:13 | |
| clarkb | I wonder if we cached that git fact | 01:13 |
| clarkb | s/git/puppet/ | 01:13 |
| corvus | clarkb: that's the only thing i can think of | 01:15 |
| fungi | yeah, i have no other ideas to explain it | 01:15 |
| corvus | okay, so -- we update that adhoc playbook to do the set-fact thing? | 01:16 |
| corvus | and we update project-config on bridge | 01:16 |
| *** auristor has joined #openstack-infra | 01:16 | |
| clarkb | that should work | 01:17 |
| corvus | then we run the playbook once, expecting it to write out the acls and fail to run m-p | 01:17 |
| corvus | then we run it again expecting it to noop | 01:17 |
| clarkb | "facter_project_config_ref": "054c0a0712d9a4dee8cd76cd430f07b1e4966a26" is in /var/cache/ansible/facts/review01.openstack.org.yaml | 01:17 |
| clarkb | corvus: ++ | 01:18 |
| corvus | i have updated project-config on bridge to: 1e6705f72 (HEAD -> master, origin/master, origin/HEAD) Move fit-review from openstack to opendev | 01:18 |
| *** hwoarang has joined #openstack-infra | 01:18 | |
| fungi | fit to go | 01:18 |
| clarkb | I think we should have the adhoc playbook not use the fact cache | 01:18 |
| clarkb | mordred: ^ is that possible | 01:18 |
| corvus | i have verified gerrit is still not running on review | 01:18 |
| mordred | uh | 01:18 |
| mordred | I have literally no idea if that's possible | 01:19 |
| corvus | how about i edit the playbook in screen? | 01:19 |
| clarkb | corvus: ++ | 01:19 |
| *** ijw has joined #openstack-infra | 01:19 | |
| *** yamamoto has quit IRC | 01:20 | |
| corvus | how about we run that for starters? | 01:21 |
| clarkb | ++ | 01:22 |
| corvus | then uncomment the second play and run again | 01:22 |
| corvus | that sha lgtm | 01:23 |
| clarkb | 1e6705f722355537e08a0187dfc957949389dc09 is what we want lgtm | 01:23 |
| corvus | to avoid dealing with ansible weirdness, i removed the limit from the command line and made the play just for review01 | 01:23 |
| corvus | so how's the playbook look now? | 01:23 |
| mordred | corvus: yes | 01:24 |
| fungi | yeah, that hopefully ought to do the trick | 01:24 |
| corvus | err... now :) | 01:24 |
| mordred | yeah. without the comments | 01:24 |
| clarkb | that matches remote puppet git | 01:24 |
| clarkb | lgtm | 01:24 |
| *** jamesmcarthur has joined #openstack-infra | 01:25 | |
| clarkb | Apr 20 01:25:18 review01 puppet-user[68852]: (/Stage[main]/Project_config/Vcsrepo[/etc/project-config]/ensure) 1e6705f722355537e08a0187dfc957949389dc09 | 01:25 |
| corvus | first run as expected i think. agree and proceed with second run? | 01:26 |
| clarkb | corvus: agreed | 01:26 |
| clarkb | I do note that the ansible fact cache did not update | 01:26 |
| clarkb | but that doesn't seem to have mattered | 01:26 |
| corvus | could that cause problems later? | 01:26 |
| fungi | the acls look right now | 01:27 |
| *** yamamoto has joined #openstack-infra | 01:27 | |
| clarkb | corvus: I worry that it might but I'm not sure | 01:27 |
| clarkb | mordred: ^ do you know? | 01:27 |
| * mordred is looking at things | 01:27 | |
| clarkb | I think it is theoretically safe to delete the cache files in that dir | 01:27 |
| corvus | clarkb: where is the file? | 01:27 |
| clarkb | maybe we want to do that then do our second run and see if it populates? | 01:27 |
| * fungi is not entirely sure how that gets cached to start with | 01:27 | |
| clarkb | /var/cache/ansible/facts/review01.openstack.org | 01:27 |
| clarkb | corvus: ^ | 01:28 |
| mordred | I'm honestly not sure that that fact cache is used as part of normal operation | 01:29 |
| corvus | it has a timestamp form 1:04 | 01:29 |
| mordred | yeah. like - go ahead and delete it - I think it's only used when we run ansible on the remote host | 01:29 |
| clarkb | mordred: I'm fairly sure that is where the first run got the wrong hash from | 01:29 |
| clarkb | and I'm guessing setting it explicitly liek corvus just did overrides teh cache value | 01:30 |
| mordred | good point | 01:30 |
| clarkb | but it probably has to timeout before it writes back even if we have overridden it? | 01:30 |
| corvus | oh maybe | 01:30 |
| clarkb | so ya I think we should remove the file and do our second run. That will hopefully populate it at the good ref we want | 01:30 |
| corvus | so yeah, let's delete and run again. i will do that. | 01:30 |
| mordred | ++ | 01:30 |
| clarkb | ++ | 01:30 |
| clarkb | -rw-r--r-- 1 root root 51709 Apr 20 01:30 review01.openstack.org | 01:31 |
| clarkb | so it did update | 01:31 |
| corvus | "facter_project_config_ref": "1e6705f722355537e08a0187dfc957949389dc09", | 01:31 |
| corvus | that's the good value | 01:31 |
| mordred | woot | 01:31 |
| corvus | m-p nooped as expected | 01:32 |
| clarkb | and puppet "nooped" as ya | 01:32 |
| corvus | i copied the "new" replication config into place | 01:32 |
| *** Lucas_Gray has quit IRC | 01:32 | |
| clarkb | lets double check iptables | 01:32 |
| clarkb | iptables lgtm | 01:33 |
| *** jamesmcarthur has quit IRC | 01:33 | |
| corvus | agreed | 01:33 |
| clarkb | (and ip6tables) | 01:33 |
| corvus | i think we can start gerrit now? | 01:33 |
| fungi | yeah, should be safe | 01:33 |
| clarkb | yes I think that is our next step | 01:33 |
| corvus | i will do that | 01:33 |
| fungi | and then it's the online reindex? | 01:33 |
| clarkb | fungi: ya | 01:34 |
| mordred | yah | 01:34 |
| clarkb | and kicking and starting zuul. I wonder if zuul needs a similar project_config fix | 01:34 |
| mordred | but we need to wait for reindex to be done before kicking zuul yeah? | 01:34 |
| corvus | gerrit is building up the replication queue | 01:35 |
| fungi | i believe so, yes | 01:35 |
| clarkb | should I start reindexing? | 01:35 |
| *** Lucas_Gray has joined #openstack-infra | 01:35 | |
| fungi | yeah, may as well | 01:35 |
| mordred | do y'all mind if I step away for a chunk while we wait on reindexing? | 01:35 |
| clarkb | mordred: no | 01:35 |
| corvus | 17887 tasks | 01:35 |
| corvus | and falling | 01:35 |
| mordred | cool. back in a few | 01:35 |
| clarkb | I'm starting the reindexing now | 01:35 |
| mordred | that's a good tasks number | 01:35 |
| corvus | i also would like to eat dinner now | 01:35 |
| fungi | i don't think waiting for replication to finish before starting reindexing is going to get it done faster anyway | 01:35 |
| corvus | agree we should start reindexing, let both it and replication run | 01:36 |
| fungi | may as well let them go concurrently | 01:36 |
| corvus | i'll eat now and check back in when i'm done | 01:36 |
| clarkb | corvus: roger | 01:36 |
| clarkb | I have started the two reindexing commands | 01:36 |
| clarkb | anyone know how to monitor indexing progress by the online indexer? | 01:37 |
| paladox | clarkb https://gerrit-review.googlesource.com/Documentation/cmd-show-queue.html | 01:38 |
| clarkb | oh its show queue too thanks | 01:38 |
| *** jamesmcarthur has joined #openstack-infra | 01:38 | |
| *** imacdonn has quit IRC | 01:38 | |
| fungi | replication also marks overall start and end in the log | 01:38 |
| clarkb | the zuul01.openstack.org ansible fact cache has the bad sha1 too | 01:39 |
| clarkb | when peopel get back from food I'm somewhat inclined to delete the entire fact cache | 01:40 |
| fungi | er, i meant reindexing, sorry | 01:40 |
| clarkb | I think reindexing may queue behind replication | 01:42 |
| clarkb | :/ | 01:42 |
| clarkb | but down to 18k ish tasks including reindexing so hopefully not long | 01:42 |
| fungi | if it is, it is | 01:42 |
| fungi | though... | 01:43 |
| fungi | didn't we determine that touching the replication config while gerrit is running now aborts the current replication tasks? | 01:43 |
| clarkb | fungi: we theorized it but did not confirm | 01:43 |
| fungi | i wonder whether we should "confirm" it now | 01:44 |
| clarkb | http://paste.openstack.org/show/749559/ good thing that worked as expected :) | 01:45 |
| clarkb | fungi: considering we didn't do a complete replication last time I think we should let this one complete | 01:45 |
| clarkb | oh I take that back it is reindexing | 01:46 |
| clarkb | its just not fast | 01:46 |
| clarkb | ok problem solved | 01:46 |
| fungi | good, good | 01:46 |
| clarkb | it went from 2096 tasks for indexing to 2092 over ~10 minutes | 01:46 |
| paladox | clarkb fungi i think https://bugs.chromium.org/p/gerrit/issues/detail?id=10260 may be related to your issue. | 01:46 |
| fungi | i believe it now starts with the repos containing the most changes so that they'll even out across the available threads | 01:47 |
| clarkb | fungi: oh that is a good thing then | 01:47 |
| fungi | so measuring by repos processed it will seem to speed up over time | 01:47 |
| clarkb | 2091 nwo so it is moving | 01:47 |
| *** jamesmcarthur has quit IRC | 01:48 | |
| fungi | paladox: thanks. not a huge issue, i was more suggesting we abuse that suspected bug to get the replication tasks to abort so it might reindex faster | 01:48 |
| clarkb | now 2089 | 01:49 |
| clarkb | and 15.7k replication tasks | 01:49 |
| paladox | ok :) | 01:49 |
| fungi | but glad to know that's what caused what we saw last week | 01:49 |
| clarkb | ianw: ^ fyi paladox knows what that happened | 01:50 |
| *** bgmccollum has quit IRC | 01:53 | |
| *** jamesmcarthur has joined #openstack-infra | 01:54 | |
| clarkb | 14.8k on replication and 2076 on indexing | 01:55 |
| clarkb | so it is moving | 01:55 |
| clarkb | so for anyone following along wondering when they will get gerrit back: we are doing the last set of data integrity udpates across the system | 01:57 |
| clarkb | we are reindexing gerrit's change and account indexes so that changes can merge and be searched on etc and we are pushing all of fungi's updates to rename things in teh zuul config to our git mirror farm | 01:58 |
| clarkb | when that is done the next step is to get zuul up and talking to it | 01:58 |
| clarkb | and then presumeably it will be wild and free in the open again | 01:58 |
| *** bgmccollum has joined #openstack-infra | 01:58 | |
| clarkb | our free range diffy kung fu cuckoo | 01:58 |
| clarkb | for the zuul connection we need 1) clear ansible fact cache. 2) kick.sh zuul servers 3) accept review.opendev.org's ssh host key on zuul servers? | 02:00 |
| fungi | i think so | 02:02 |
| *** bgmccollum has quit IRC | 02:05 | |
| *** jamesmcarthur has quit IRC | 02:08 | |
| *** jamesmcarthur has joined #openstack-infra | 02:08 | |
| * corvus is sated | 02:10 | |
| clarkb | 1800 or so projects need changes indexed and 12k replication events | 02:10 |
| corvus | the fog just arrived; it's summer! | 02:13 |
| *** ijw has quit IRC | 02:13 | |
| paladox | lol | 02:13 |
| clarkb | 1300 indexing events so ya seems to really be picking up steam | 02:13 |
| *** ijw has joined #openstack-infra | 02:14 | |
| *** Lucas_Gray has quit IRC | 02:15 | |
| clarkb | nova neutron openstack-manuals are all that are left in the indexer queue | 02:16 |
| clarkb | I'm guessing they have been running the entire time | 02:16 |
| fungi | yeah, they're the largest 3 | 02:16 |
| clarkb | 11.3k replication events | 02:17 |
| clarkb | I need to stop watching paint dry | 02:17 |
| clarkb | now down to just nova and openstack-manuals | 02:17 |
| clarkb | go go go diffy | 02:17 |
| *** harlowja has quit IRC | 02:18 | |
| fungi | i'm cuckoo for lucene reindexing | 02:19 |
| fungi | (and cocoa puffs) | 02:20 |
| *** ijw has quit IRC | 02:20 | |
| corvus | you have somehow managed to make me hungry again | 02:20 |
| fungi | oops | 02:20 |
| *** ijw has joined #openstack-infra | 02:20 | |
| *** eernst has joined #openstack-infra | 02:22 | |
| *** ijw has quit IRC | 02:22 | |
| clarkb | now just noav indexing | 02:26 |
| clarkb | noav is the new openstack home audio management system | 02:26 |
| clarkb | its like serverless but for your tunes | 02:26 |
| fungi | it's serverless *and* wireless | 02:29 |
| *** ijw has joined #openstack-infra | 02:30 | |
| *** armax has joined #openstack-infra | 02:37 | |
| *** bhavikdbavishi has quit IRC | 02:38 | |
| mordred | it seems like things are still chugging - and potentially not on fire | 02:38 |
| corvus | i think we're just waiting on nova to finish indexing before we start up zuul | 02:39 |
| *** bhavikdbavishi has joined #openstack-infra | 02:39 | |
| clarkb | is the specific concern with starting zuul before indexing completels that changes would not be found by zuul or mergeable? | 02:39 |
| mordred | not be mergable | 02:40 |
| clarkb | I wonder if nova is ok since we didn't moev it? | 02:40 |
| mordred | yeah | 02:40 |
| clarkb | (I don't actually know) | 02:40 |
| mordred | probably? | 02:40 |
| mordred | also - as long as gerrit is firewalled, we can probably go ahead | 02:40 |
| corvus | well, we can't start zuul with gerrit firewalled | 02:40 |
| clarkb | ya zuul will just be sad | 02:40 |
| mordred | ah | 02:40 |
| mordred | yeah | 02:40 |
| mordred | that's an excellent | 02:40 |
| mordred | point | 02:41 |
| clarkb | 02:00:38 clarkb | for the zuul connection we need 1) clear ansible fact cache. 2) kick.sh zuul servers 3) accept review.opendev.org's ssh host key on zuul servers? | 02:41 |
| mordred | so - you know - other than gerrit being firewalled, there's nothing stopping us :) | 02:41 |
| clarkb | is what I had written earlier about this | 02:41 |
| corvus | but i think you're right; if we're not worried about the nova index being wrong.. we can probably get started on that ^ | 02:41 |
| mordred | agree | 02:41 |
| clarkb | corvus: I'm not sure if you expect known hosts to be correct already | 02:41 |
| clarkb | but otherwise I think we can probably proceed | 02:41 |
| corvus | clarkb: i dunno, if it's broke we can fix it :) | 02:41 |
| clarkb | I want to say that when we have done reindexing before we haven't waited for it to finish across the baord and things have mostly worked so I think it will liekly use the existing data | 02:42 |
| clarkb | I wonder if gerrit tries to index the openwrt image in nova | 02:43 |
| corvus | the whaaaaaanevermind | 02:43 |
| clarkb | mordred: corvus should we delete everything in the ansible fact cache? | 02:43 |
| corvus | clarkb: i'm okay with that | 02:44 |
| mordred | ++ | 02:44 |
| clarkb | ok I'll do that now | 02:44 |
| clarkb | except for maybe review01.openstack.org since we know it is happy | 02:44 |
| clarkb | I have started with the zuul nodes and those are done | 02:45 |
| corvus | i think the host keys will be a problem | 02:45 |
| corvus | so maybe i'll make another change to system-config, we can stage it on bridge before we kick | 02:46 |
| *** ijw_ has joined #openstack-infra | 02:47 | |
| *** armax has quit IRC | 02:47 | |
| *** ijw has quit IRC | 02:47 | |
| corvus | clarkb, mordred: can you look at the bridge screen and see if that change lgty? | 02:48 |
| *** bhavikdbavishi1 has joined #openstack-infra | 02:49 | |
| mordred | corvus: yes- I think that looks right | 02:49 |
| mordred | yeah | 02:49 |
| *** bhavikdbavishi has quit IRC | 02:50 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:50 | |
| clarkb | you just prepended the review.opendev.org name right? | 02:50 |
| clarkb | i think that is correct | 02:50 |
| clarkb | I've removed everything from that cache dir but the review01.openstack.org file | 02:50 |
| corvus | clarkb: that was my intention -- to prepend "[review.opendev.org]:29418," | 02:50 |
| corvus | hopefully the intention and the actualization are both correct :) | 02:50 |
| mordred | that is what is appears to be | 02:51 |
| clarkb | that is how I read it so intention seems to be achived (but I'm a human and the computers are pickier | 02:51 |
| mordred | but it would be easier if you could use fit-review to submit that | 02:51 |
| corvus | so i think when we kick, we'll take care of your step 3 too. | 02:51 |
| *** ijw_ has quit IRC | 02:51 | |
| clarkb | we are all gonna have a fit-review alias tomorrow aren't we | 02:51 |
| clarkb | corvus: yup | 02:51 |
| clarkb | corvus: I think you should reuse your custom kick playbook too and replace the groups for the second play | 02:52 |
| corvus | how should we kick -- should we modify the one-off playbook we did for review to run on the zuul hosts? (so we get the same project-config sha behavior?) | 02:52 |
| *** ijw has joined #openstack-infra | 02:52 | |
| corvus | sounds like a yes :) | 02:52 |
| mordred | ++ | 02:52 |
| clarkb | is it commas or : | 02:53 |
| corvus | something like that? | 02:53 |
| clarkb | I think it might be : as separator | 02:53 |
| corvus | ya i think so | 02:53 |
| fungi | oh, yep it's : | 02:53 |
| mordred | yes : | 02:53 |
| clarkb | before we run that do we need to open the firewall or do you do that after you start zuul? | 02:53 |
| fungi | and sorry, back now. we've got 70mph gusts with the front coming through | 02:54 |
| corvus | i don't think we need to open the firewall until we're ready to start zuul | 02:54 |
| clarkb | roger | 02:54 |
| clarkb | I actually need a minute but I think that looks good so I would run it without me | 02:54 |
| mordred | fungi: we may have had that storm a couple of days ago | 02:54 |
| corvus | oh we forgot to restart gitea; we should remember to do that after replication finishes. | 02:54 |
| fungi | just making sure to batten all the hatches | 02:55 |
| corvus | running playbook now | 02:55 |
| corvus | i guess this will be slow because of -f1 but that's okay | 02:55 |
| clarkb | I'm back | 02:56 |
| corvus | should i restart it with -f20? | 02:57 |
| *** ijw has quit IRC | 02:57 | |
| clarkb | we don't puppet zp01 | 02:57 |
| corvus | oh, heh, i shouldn't have added zuul-preview, but it will correctly noop | 02:57 |
| clarkb | ya it should try to run the command that doesn't exist | 02:57 |
| corvus | i believe that technically that means that i failed a sanity check. | 02:58 |
| clarkb | I agree | 02:58 |
| mordred | we're still assuming there are remaining pieces of sanity? | 02:58 |
| clarkb | mordred: I expect most of us in this cahnnel fail that check | 02:59 |
| corvus | mordred: you added that sanity check, and i failed it! | 02:59 |
| corvus | Apr 20 03:00:27 ze01 puppet-user[31446]: (/Stage[main]/Project_config/Vcsrepo[/etc/project-config]/ensure) Updating to latest '1e6705f722355537e08a0187dfc957949389dc09' revision | 03:00 |
| corvus | Apr 20 03:00:44 ze01 puppet-user[31446]: (/Stage[main]/Zuul/File[/etc/zuul/zuul.conf]/content) content changed '{md5}f48c7b0e12823050b85d6bdc38595517' to '{md5}9b7b0b71d221d315b6a052514de75275' | 03:00 |
| corvus | Apr 20 03:00:47 ze01 puppet-user[31446]: (/Stage[main]/Zuul::Known_hosts/File[/home/zuul/.ssh/known_hosts]/content) content changed '{md5}73caf003f374090b50e18bebf5aa0180' to '{md5}d4b32bed12c11da9ea4e7307b4361a51' | 03:00 |
| mordred | I'm just going to assume that's the right md5 | 03:01 |
| clarkb | it is the right sha1 for the git repo so thats nice | 03:01 |
| corvus | the content of both of those files lgtm | 03:01 |
| *** eernst_ has joined #openstack-infra | 03:01 | |
| *** eernst has quit IRC | 03:01 | |
| clarkb | at the summit/ptg we should give nova a prize along the lines of "Gerrits problem child" | 03:02 |
| clarkb | :P | 03:02 |
| mordred | clarkb: ++ | 03:02 |
| corvus | wow we figured out a way to watch 2 boring tasks at the same time | 03:06 |
| fungi | multi-tasked boring | 03:06 |
| clarkb | down to 3k push tasks | 03:07 |
| clarkb | if anyone was wondering | 03:07 |
| clarkb | nova still reindexing | 03:07 |
| corvus | my music player has decided to play the riven soundtrack, which is about the most boring piece of music i have | 03:07 |
| clarkb | corvus: if you upgrade to noav it fixes that problem by playing no music | 03:07 |
| clarkb | (ok that nova typo joke is probably done with) | 03:08 |
| mordred | clarkb: did lennart write noav then? | 03:08 |
| clarkb | I dunno he wrote pulseaudio so I think he is in the av camp | 03:09 |
| *** bgmccollum has joined #openstack-infra | 03:09 | |
| clarkb | its done | 03:09 |
| clarkb | it being the ansible + puppet | 03:09 |
| clarkb | zp01 failed but the others look ok? | 03:09 |
| corvus | yeah, i think we can tweak the firewall, poke at gerrit a bit, then try starting zuul? | 03:09 |
| clarkb | seems reasonable | 03:10 |
| paladox | People still have music players? :) | 03:10 |
| * mordred hopes the logo works | 03:10 | |
| clarkb | mordred: hahaha | 03:10 |
| clarkb | I think the other logo was svg too right? so in theory it will just get squashed into the space for it | 03:10 |
| mordred | yeah. that's the theory I'm operating under | 03:11 |
| fungi | also gitea restarts? | 03:11 |
| corvus | fungi: after replication finishes | 03:11 |
| fungi | yeah | 03:11 |
| clarkb | 2520 replication tasks | 03:11 |
| corvus | actually, let's check and see how many of those are non-gitea | 03:11 |
| clarkb | 1675 are gitea | 03:12 |
| corvus | 1676 are gitea | 03:12 |
| corvus | so i guess we still have a little while; it's at that point that i think we can relatively safely restart gitea | 03:12 |
| corvus | i've updated the iptables rules files, should i reload them? | 03:12 |
| clarkb | I'm ssh'd in so if that has a sad I have like 2 connections | 03:13 |
| mordred | sure | 03:13 |
| clarkb | so ya I say go for it | 03:13 |
| corvus | good, the riven soundtrack has gotten to the tense part | 03:13 |
| corvus | the logo could use some work :) | 03:13 |
| clarkb | holy carp it worked | 03:13 |
| clarkb | should I test a login? | 03:13 |
| mordred | yeah. wow logo | 03:14 |
| * clarkb tests a login | 03:14 | |
| mordred | that does not look good | 03:14 |
| mordred | I have logged in | 03:14 |
| mordred | and it worked | 03:14 |
| clarkb | it works | 03:14 |
| corvus | https://review.opendev.org/#/c/653910/ is a thing in a moved project | 03:14 |
| *** eernst_ has quit IRC | 03:14 | |
| clarkb | it finds my changes | 03:14 |
| mordred | ok. who knows how to fix the logo | 03:14 |
| clarkb | mordred: I hear tristanC | 03:15 |
| mordred | clarkb: ++ | 03:15 |
| corvus | i'm going to upload a change? | 03:15 |
| mordred | or - hey mnaser you around? | 03:15 |
| clarkb | corvus: ya I think that is the next thing to test | 03:15 |
| mnaser | hi | 03:15 |
| clarkb | and you have changes to upload :) | 03:15 |
| mnaser | someone want HTML help? Ha | 03:15 |
| mnaser | Let me open up my laptop quick | 03:16 |
| mordred | mnaser: yes! go check out review.opendev.org | 03:16 |
| clarkb | or review.openstack.org | 03:16 |
| mordred | and see how ugly it is | 03:16 |
| clarkb | beause one is the other now | 03:16 |
| clarkb | like inception magic | 03:16 |
| mnaser | Least I can do after y’alls hard work is polish it up | 03:16 |
| *** openstackgerrit has joined #openstack-infra | 03:17 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Update known_hosts for zuul https://review.opendev.org/653955 | 03:17 |
| clarkb | \o/ | 03:17 |
| clarkb | thinking about the logo maybe we want to drop the text? | 03:17 |
| clarkb | though that requires a new image | 03:17 |
| mnaser | do we have a logo thats like | 03:18 |
| corvus | or do a side-by-side logo and type | 03:18 |
| mnaser | ^^ | 03:18 |
| mordred | ++ | 03:18 |
| mnaser | what corvus said | 03:18 |
| clarkb | mnaser: we can get them I just don't know if we have them this moment | 03:18 |
| mnaser | lemme try and see if I can craft one quickly from the one that's there | 03:18 |
| clarkb | changing the image seems like a thing for when we have run_all.sh running | 03:18 |
| corvus | it's svg so we can probably do something passable | 03:18 |
| clarkb | but I'm game for people to show me I'm wrong :) | 03:18 |
| corvus | should we start zuul now? | 03:18 |
| clarkb | nova is still idnexing (if we care about that) | 03:19 |
| clarkb | corvus: zuul is next on our list | 03:19 |
| clarkb | I'm ready if you are | 03:19 |
| *** hwoarang has quit IRC | 03:19 | |
| clarkb | 452 gitea replication tasks | 03:19 |
| clarkb | maybe start zuul when that is done so zuul sees all the fungi updates? | 03:20 |
| corvus | derp | 03:20 |
| corvus | clarkb: zuul only talks to gerrit | 03:20 |
| clarkb | oh good | 03:20 |
| corvus | zuul is starting. | 03:20 |
| tristanC | clarkb: mordred: adding 'width: 42px' to system-config/modules/openstack_project/files/gerrit/GerritSiteHeader.html img style should do the trick | 03:20 |
| fungi | yeah, and so restarting gitea with zuul up will presumably be a nonevent | 03:20 |
| tristanC | err, height* | 03:20 |
| clarkb | fungi: no we wanted gitea to get all the updates before we restarted it | 03:21 |
| *** hwoarang has joined #openstack-infra | 03:21 | |
| fungi | right, which it almost has now | 03:21 |
| clarkb | ya just a minute or two left | 03:21 |
| corvus | i think fungi was saying starting zuul won't prohibit us from restarting gitea | 03:21 |
| clarkb | oh gotcha | 03:21 |
| fungi | that, yes | 03:21 |
| fungi | i am running low on words as it's getting late over on this end of the rock | 03:21 |
| corvus | it's getting late on *this* end, i think it already got late over there | 03:22 |
| clarkb | I think all of us are probably feeling it a bit at this point (maybe more than a bit) so no worries | 03:22 |
| corvus | merger jobs are out | 03:22 |
| *** jamesmcarthur has quit IRC | 03:22 | |
| corvus | i think the mergers are atad slower because they're cloning and updating | 03:22 |
| clarkb | ya they have to pull down all the "new" repos | 03:23 |
| corvus | but they are completing | 03:23 |
| clarkb | (we should try to remember to clear out the old ones. maybe just delete the entire dir in a rolling fashion at some point) | 03:23 |
| clarkb | gitea replication seems done | 03:24 |
| pabelanger | grafana.o.o has some merger stats, so that is good | 03:24 |
| mordred | clarkb: that's so cloud native of you | 03:24 |
| pabelanger | http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1&from=now-1h&to=now&refresh=5s&fullscreen&panelId=31 | 03:24 |
| mordred | \o/ | 03:24 |
| corvus | how about i restart gitea then? | 03:24 |
| clarkb | corvus: I think that should be fine now | 03:24 |
| corvus | doing so | 03:25 |
| fungi | thanks! | 03:25 |
| fungi | that gets us the .git clone urls working, right? | 03:25 |
| corvus | should yes | 03:26 |
| corvus | gitea is restarted | 03:26 |
| clarkb | I'll test a .git clone | 03:26 |
| clarkb | seems to be working. waiting for it to complete | 03:27 |
| clarkb | it completed and git log -1 in project-config is the fit review change | 03:28 |
| clarkb | I even cloned via the openstack.org/openstack-infra redirects | 03:28 |
| clarkb | dtroyer: ^ I have no idea if yo uare still paying attention but I think we sorted the issue you found. And by we I really mean corvus the gitea wizard | 03:28 |
| * mordred declares corvus to be the king of golang programming for infra | 03:30 | |
| corvus | we're at 3900 cat jobs | 03:30 |
| mnaser | (ignore the choice of site, this was the first result for "svg image upload"): https://svgur.com/s/C_F | 03:30 |
| mnaser | if that looks nice I can push up a patch which should make it fit nicely :> | 03:30 |
| corvus | mnaser: looks good! | 03:30 |
| mnaser | by no means I expect that to land today :-) | 03:30 |
| mnaser | did a pull on my local system-config copy "warning: redirecting to https://opendev.org/opendev/system-config/" | 03:31 |
| mnaser | working as usual :) | 03:31 |
| clarkb | ya if you get the patch in I think the next stuff ( assuming zuul is generally happy ) for us (tomorrow probably?) is to get normal deployment stuff working | 03:31 |
| clarkb | and if your patch is ready we should be able to get that out in a reasonable amount of time | 03:31 |
| corvus | zuul created a layout | 03:32 |
| clarkb | I think its up | 03:32 |
| mordred | \o/ | 03:32 |
| clarkb | and in true openstack fashion there are alredy 4 chagnes scheduled | 03:32 |
| clarkb | 5! | 03:32 |
| corvus | and zero (?) configuration errors? | 03:32 |
| clarkb | ya I'm surprised by that but I'm willing to take it | 03:32 |
| corvus | that's all on fungi :) | 03:33 |
| fungi | shocked | 03:33 |
| clarkb | ++ | 03:33 |
| fungi | maybe i should take that as the universe telling me to get a good night's sleep | 03:33 |
| *** yamamoto has quit IRC | 03:33 | |
| * corvus restarts gertty | 03:34 | |
| mordred | fungi: ++ | 03:35 |
| mnaser | tested it locally :) https://usercontent.irccloud-cdn.com/file/b7TB8DeB/image.png | 03:35 |
| clarkb | I see zero warnings bells | 03:35 |
| clarkb | on any tenant | 03:35 |
| clarkb | that is seriously impressive fungi | 03:35 |
| prometheanfire | testing still paused? | 03:35 |
| clarkb | prometheanfire: we just turned it on | 03:35 |
| prometheanfire | kk | 03:35 |
| clarkb | prometheanfire: and are kicking the tires on it | 03:35 |
| prometheanfire | I'll wait | 03:36 |
| clarkb | mnaser: that looks nice | 03:36 |
| clarkb | prometheanfire: well you can kick the tires too probably and if we have to restart things again you know how to recheck :) | 03:36 |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: assets: replace logo with side-by-side logo and type https://review.opendev.org/653956 | 03:36 |
| mnaser | voila ^ | 03:36 |
| corvus | a job completed successfully! | 03:36 |
| corvus | on a moved repo no less | 03:36 |
| clarkb | nice! | 03:37 |
| corvus | http://logs.openstack.org/06/621906/2/check/openstack-tox-docs/8931d2c/html/ | 03:37 |
| mordred | that's so exciting | 03:37 |
| corvus | that's for x/mogan | 03:37 |
| * fungi is awed and amazed | 03:37 | |
| mordred | corvus: something ran a job on x/mogan? | 03:37 |
| clarkb | nova specs just passed pep8 | 03:37 |
| mnaser | empty gate | 03:37 |
| clarkb | the docs build should finish soon and hopefully we get our first +1 | 03:37 |
| mnaser | clouds all just for me :-P | 03:37 |
| mordred | I love that we turned on the firewall and patches came immediately | 03:37 |
| corvus | https://review.opendev.org/#/c/621906/ | 03:38 |
| mordred | after the service being down for like 12 hours | 03:38 |
| clarkb | corvus: our first +1 | 03:38 |
| prometheanfire | kk | 03:38 |
| clarkb | if we werent' all remote I think this is the point of the day where we all dance like lunatics | 03:38 |
| clarkb | maybe we do that at home anyway | 03:38 |
| fungi | it's like pulling on the change tap and pouring a pint of zuul | 03:38 |
| corvus | gertty users: gertty does detect project renames (and implements them as deletes and adds). if you restart gertty, it should take care of that automatically, but it will take a while to start syncing because we have so many. | 03:39 |
| mordred | fungi: we should pass out bottles of zuul in denver | 03:39 |
| clarkb | I suppose I should start drafting a "where we are" email | 03:39 |
| fungi | i will save that bit of excitement for tomorrow | 03:39 |
| clarkb | https://etherpad.openstack.org/p/eEJT9ssDG5 I'll compose there | 03:40 |
| fungi | "where we are: welcome to opendev" | 03:40 |
| mnaser | "tomorrow we will start moving to gerrit 3.0" | 03:40 |
| clarkb | mnaser: please no | 03:40 |
| mnaser | :-P | 03:40 |
| paladox | Lol | 03:41 |
| *** yamamoto has joined #openstack-infra | 03:41 | |
| fungi | i'll need a break before that ;) | 03:41 |
| * mordred just heard mnaser volunteer for the gerrit upgrade | 03:43 | |
| dtroyer | clarkb: woo! Looks good on the first two random ones I tried, thanks corvus | 03:43 |
| mnaser | sorry, no root access, no can do | 03:43 |
| mnaser | :-P | 03:43 |
| corvus | status ok The OpenDev Gerrit and Zuul are back online; you may need to update your git remotes for projects which have moved. | 03:43 |
| dtroyer | and thanks all of youse guys for pushing through this | 03:43 |
| mnaser | I wonder if the openstack ansible bootstrapping process still works, lemme see | 03:43 |
| mordred | mnaser: root not needed- it's all ansible and puppet my friend :) | 03:43 |
| corvus | anything else we should add into that ^ status? | 03:43 |
| mordred | corvus: I thnik that's great | 03:43 |
| clarkb | ya we can keep that short and my email should hopefully have enough detail | 03:44 |
| mordred | corvus: unless you want to say "please ignore the oversized logo, mnaser is fixing it" | 03:44 |
| mnaser | s/fixing/fixed with 653956/ | 03:44 |
| fungi | lgtm corvus | 03:44 |
| corvus | #status ok The OpenDev Gerrit and Zuul are back online; you may need to update your git remotes for projects which have moved. | 03:44 |
| openstackstatus | corvus: sending ok | 03:44 |
| *** bhavikdbavishi has quit IRC | 03:45 | |
| mnaser | aw man | 03:46 |
| mnaser | osa is completely broken | 03:46 |
| mnaser | (in the gate) | 03:46 |
| clarkb | mnaser: sorry | 03:46 |
| mnaser | (hard path dependency on openstack/foo) | 03:46 |
| mnaser | its all good | 03:46 |
| *** yamamoto has quit IRC | 03:46 | |
| mnaser | probably a good time to get rid of that hardcode I gues | 03:46 |
| mnaser | esp if we move to our own namespace :-) | 03:47 |
| mordred | mnaser: well - let us know if you need help unwinding | 03:47 |
| mordred | mnaser: NO - NO MORE MOVES ;) | 03:47 |
| mnaser | hahaha | 03:47 |
| mnaser | I think a big reason we can't move is the fact that the one redirect rule is a thing | 03:47 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 03:47 | |
| -openstackstatus- NOTICE: The OpenDev Gerrit and Zuul are back online; you may need to update your git remotes for projects which have moved. | 03:47 | |
| clarkb | mnaser: I think git-review said that is a non issue | 03:47 |
| mordred | mnaser: what one redirect rule? | 03:47 |
| clarkb | mnaser: we went from openstack-infra/git-review to openstack/ to opendev/ and it works | 03:48 |
| mordred | mnaser: yeah - git-review has already been moved twice in gitea | 03:48 |
| fungi | clarkb: should the status e-mail mention the "x" namespace too? | 03:48 |
| mnaser | I think fungi mentioned that by default redirects with git can only happen once unless you have some setting set differnetly | 03:48 |
| mordred | mnaser: we read more about that | 03:48 |
| mnaser | (that was me skimming stuff here though, so I guess if thats fine, cool) | 03:48 |
| mordred | mnaser: it'll do as many redirects as needed to get you to the first data operation | 03:48 |
| corvus | and that's how rumors get started! | 03:48 |
| mnaser | :P | 03:48 |
| clarkb | fungi: ++ | 03:48 |
| * mnaser hides | 03:48 | |
| *** yamamoto has joined #openstack-infra | 03:48 | |
| mordred | mnaser: but once you actually get to content, that content needs to be at the same place | 03:49 |
| mnaser | gotcha | 03:49 |
| mordred | we'd be so screwed if that wasn't true :) | 03:49 |
| fungi | mnaser: subsequent requests git makes assume content is served from the same place its first request (eventually) ended up | 03:49 |
| fungi | i wonder if we should (yet) mention how/where/when to start making the inevitable additional namespace move requests | 03:51 |
| openstackstatus | corvus: finished sending ok | 03:51 |
| clarkb | fungi: how does that read? re x/ | 03:51 |
| mnaser | aw man | 03:51 |
| mnaser | I really don't wanna be that guy but | 03:51 |
| mnaser | https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=af41b096a5a0852a15c3989ec48b956433c47873 redirects to https://opendev.org/openstack/requirements/raw/branch/af41b096a5a0852a15c3989ec48b956433c47873/upper-constraints.txt | 03:52 |
| fungi | clarkb: aha, you did mention subsequent renames too, lgtm | 03:52 |
| mnaser | when it should be | 03:52 |
| mnaser | https://opendev.org/openstack/requirements/raw/commit/af41b096a5a0852a15c3989ec48b956433c47873/upper-constraints.txt | 03:52 |
| mordred | mnaser: cool - that's just apache rewrte rules | 03:52 |
| corvus | mnaser: that's *probably* easyish to fix :) | 03:52 |
| mnaser | sorry to throw in more things to do when its gotten so late | 03:52 |
| mnaser | feel free to come at it again tomorrow | 03:52 |
| fungi | yeah, it's all in one .htaccess file in system-config | 03:52 |
| fungi | (in opendev/system-config as of today) | 03:53 |
| clarkb | I'll also send a note to the infra list asking that we be slushy except for changes to get run_all.sh working or fixing bugs. Basically lets not add new projects or similar until that works | 03:53 |
| clarkb | any objections to ^ | 03:53 |
| fungi | sounds about right | 03:53 |
| clarkb | also last call on https://etherpad.openstack.org/p/eEJT9ssDG5 I think that is about ready to go out | 03:53 |
| corvus | mnaser: ... where did you get tha url? | 03:54 |
| mordred | mnaser: https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/git-redirect.htaccess | 03:54 |
| corvus | that means "the head named | 03:54 |
| mordred | in case you get bored | 03:54 |
| mnaser | corvus: its what we use in OSA to get requirements at a specific commit | 03:54 |
| corvus | er, the head named "af41b096a5a0852a15c3989ec48b956433c47873" | 03:54 |
| corvus | mnaser: you want "id=" | 03:54 |
| fungi | clarkb: should we make any mention of github replication? like telling people not to rely on the github mirrors being up to date for now? | 03:54 |
| pabelanger | clarkb: so, cloning from review.o.o over SSH no longer works, is that correct? | 03:54 |
| corvus | mnaser: i do not believe it's possible to get cgit to produce that url | 03:55 |
| clarkb | pabelanger: no it should work | 03:55 |
| pabelanger | Hmm | 03:55 |
| corvus | mnaser: it's probably something that happens to work | 03:55 |
| mordred | pabelanger: it should still work. cloning from gitea over ssh does not | 03:55 |
| clarkb | I haven't tested it but review.openstack.org and review.opendev.org are the same host | 03:55 |
| mnaser | corvus: I don't know how we ended up with this url, this predates my usage of opennstack ansible | 03:55 |
| clarkb | possible that the canonical hostname plays into the ssh cloning? | 03:55 |
| clarkb | pabelanger: its possible if ^ is a thing that that might be something that doesn't work | 03:55 |
| mnaser | but it does mean all of our released/tagged versions for openstackansible are now non-functional.. so I guess I might have to pay the price of figuring this out myself because of the pain we created ourselves | 03:56 |
| pabelanger | clarkb: yah, I think I may need to update my zuul config for zuul.a.o. I seem to be getting permission denied error | 03:56 |
| mordred | pabelanger: might be host key | 03:56 |
| pabelanger | yah | 03:56 |
| fungi | host key should still be the same | 03:57 |
| mordred | yeah - but the entry for it in known hosts might be mapped to old name? | 03:57 |
| corvus | mnaser: i think if you wanted to write a redirect rule that heuristically determined that h=SHA was a sha and not a branch, that would be ok | 03:57 |
| clarkb | mordred: well the old name has the same ip addr as the new name | 03:57 |
| mnaser | ^ yeah I had to accept a new key for my known_hosts | 03:57 |
| mordred | indeed | 03:57 |
| fungi | mordred: sounds like he was still trying to use the old name | 03:57 |
| * mordred has just learned his laptop battery is about to die | 03:57 | |
| corvus | mnaser: https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/git-redirect.htaccess#L57-L62 is the rule in question | 03:58 |
| corvus | mnaser: or actually https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/git-redirect.htaccess#L64-L67 | 03:58 |
| * mordred will be back on tomorrow to work through issues we find | 03:58 | |
| clarkb | mordred: good night | 03:58 |
| clarkb | also ya I'll be around tomorrow too | 03:58 |
| fungi | something like [0-9a-f]{40} should match a sha | 03:58 |
| * mordred waves goodnight to everyone | 03:58 | |
| corvus | mordred: night! thanks! | 03:58 |
| fungi | night mordred! | 03:58 |
| clarkb | ready for me to send that email in the etherpad? | 03:59 |
| clarkb | then I need to stop typing so loudly as kids are unhappy | 03:59 |
| corvus | clarkb: lgtm | 03:59 |
| fungi | clarkb: should we make any mention of github replication? like telling people not to rely on the github mirrors being up to date for now? | 03:59 |
| pabelanger | clarkb: I'm seeing a new SSH host key, did that change? | 03:59 |
| corvus | replication has finished. | 03:59 |
| clarkb | pabelanger: no the shs key shouldn't have changed | 04:00 |
| pabelanger | blarg | 04:00 |
| pabelanger | port 22 | 04:00 |
| clarkb | pabelanger: corvus' change to update our known_hosts was to just add the new name in front of the key | 04:00 |
| fungi | pabelanger: yep, that'll do it! | 04:00 |
| clarkb | indeed | 04:00 |
| clarkb | alright I'm sending that email now | 04:00 |
| paladox | It must be midnight in some parts of the us? | 04:00 |
| pabelanger | fungi: yah, late here too | 04:00 |
| fungi | clarkb: should we make any mention of github replication? like telling people not to rely on the github mirrors being up to date for now? | 04:00 |
| pabelanger | I'm going to bail on debugging this tonight and look in the morning | 04:01 |
| * fungi apologizes for asking that 3 times, not sure if it was missed in the other ensuing discussions | 04:01 | |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: redirects: add support for plain h=<sha> https://review.opendev.org/653957 | 04:01 |
| clarkb | fungi: oh sorry I thougth I responded to that then I think mordred said goodnight and i got idstracted | 04:02 |
| clarkb | fungi: dmsimard sent a really great email on the topic already | 04:02 |
| mnaser | thanks to fungi small insight .. the above is an attempt. I dunno if it works or whatnot :) | 04:02 |
| fungi | clarkb: okay, that's cool | 04:02 |
| clarkb | I'm inclined to not bother for now? I'd like us to get through the cleanup phase of this before fixing everyone's github | 04:02 |
| fungi | just figure folks may be confused seeing stuff which actually moved to x still being in the openstack namespace there but not updating any longer. probably not a big deal | 04:03 |
| mnaser | system-config gate is broken fwiw (rather 'post'): "Warning: Permanently added '104.239.149.122' (ECDSA) to the list of known hosts.\r\nrsync: change_dir \"/home/zuul//src/opendev.org/opendev/system-config/applytest\" failed: No such file or directory (2)\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1655) [Receiver=3.1.1]\nrsync: [Receiver] write error: Broken pipe (32)" | 04:04 |
| fungi | if they do come to us we can remind them to use our official remotes instead | 04:04 |
| clarkb | mnaser: it is tools/applytest iirc | 04:05 |
| fungi | thanks for the heads up mnaser! i think we expect lots of jobs doing interesting things with file paths to probably need further fixing | 04:05 |
| *** yamamoto has quit IRC | 04:05 | |
| corvus | i think we're at a reasonably good stopping point... so if no one else has anything urgent for me, i'm going to go eat some new york super fudge chunk and come back in the morning | 04:05 |
| fungi | thanks corvus! ejoy! | 04:06 |
| fungi | eNjoy too | 04:06 |
| clarkb | corvus: one quick quetion | 04:06 |
| corvus | clarkb: ya | 04:06 |
| clarkb | for my email to the infra list I was going to say fixes for jobs are safe but new projects and things that would otherwise update servers will need work | 04:06 |
| clarkb | I haven't said anything wrong with that have I? | 04:07 |
| corvus | clarkb: as long as run_all is not running we're fine i think | 04:07 |
| corvus | so yeah, i would say that and mention please don't enable run_all | 04:07 |
| clarkb | ++ thanks | 04:07 |
| corvus | goodnight! | 04:08 |
| clarkb | you too | 04:09 |
| fungi | i'm going to knock off now as well. thanks for sending the updates clarkb! and thanks everyone for a very successful maintenance... sorry i had to skip out on the first part | 04:10 |
| pabelanger | clarkb: k, so this should not work right? because ssh:// http://paste.openstack.org/show/749560/ ? | 04:11 |
| pabelanger | https:// works correctly | 04:11 |
| clarkb | pabelanger: for that I think you need openstack/project-config | 04:12 |
| clarkb | because we cannot redirect ssh | 04:12 |
| clarkb | hrm but that doesn't work for me and neither does using review.opendev.org | 04:13 |
| clarkb | so how did zuul do it | 04:13 |
| fungi | or switch to https | 04:13 |
| pabelanger | clarkb: okay, thats fine. I'll update zuul here to do clones via HTTPs, because right now doing via ssh and failing | 04:14 |
| fungi | doesn't zuul use https for its git operations now? | 04:14 |
| clarkb | my fail with opendev.org might be pebkac | 04:14 |
| clarkb | its working now | 04:14 |
| pabelanger | fungi: yes, I think that is right | 04:14 |
| clarkb | pabelanger: let me test with review.openstack.org in a minute | 04:14 |
| prometheanfire | now I need to go run sed on all the repos locally cloned | 04:14 |
| prometheanfire | unless tonyb[m] has a script :P | 04:14 |
| fungi | tonyb[m] *is* a script | 04:15 |
| fungi | okay, really, i swear, i'm actually putting down the computer now. g'night all! | 04:15 |
| prometheanfire | well, true | 04:15 |
| prometheanfire | nn | 04:15 |
| clarkb | prometheanfire: it works for me if I use the new path | 04:15 |
| clarkb | prometheanfire: sorry that was for pabelanger | 04:16 |
| clarkb | pabelanger: so the issue is we can't redirect the path component over ssh | 04:16 |
| pabelanger | clarkb: ++ | 04:16 |
| clarkb | pabelanger: so you'll want to update the path component | 04:16 |
| pabelanger | okay, thanks for confirming | 04:16 |
| pabelanger | will deal with that in the morning | 04:16 |
| tristanC | congrats everyone, the migration is working as expected from here, great work! | 04:16 |
| clarkb | I saw a change in the post queue | 04:19 |
| clarkb | we merged something (it went away as soon as zuul status reloaded) | 04:19 |
| clarkb | https://review.opendev.org/#/c/653917/ a thing that merged | 04:22 |
| clarkb | alright everyone I think it is tiem for me to call ti a day too | 04:22 |
| clarkb | I'll be back in the morning. good night | 04:22 |
| clarkb | nova finished indexing | 04:22 |
| *** yamamoto has joined #openstack-infra | 04:33 | |
| *** yamamoto has quit IRC | 04:34 | |
| *** yamamoto has joined #openstack-infra | 04:34 | |
| *** yamamoto has quit IRC | 04:42 | |
| *** yamamoto has joined #openstack-infra | 04:43 | |
| *** yamamoto has quit IRC | 04:48 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:48 | |
| *** hwoarang has quit IRC | 05:03 | |
| *** Goneri has quit IRC | 05:04 | |
| *** hwoarang has joined #openstack-infra | 05:07 | |
| *** hwoarang has quit IRC | 05:14 | |
| *** yamamoto has joined #openstack-infra | 05:16 | |
| *** yamamoto has quit IRC | 05:18 | |
| *** hwoarang has joined #openstack-infra | 05:21 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Sort gerritbot/channels.yaml https://review.opendev.org/653928 | 05:31 |
| *** yamamoto has joined #openstack-infra | 05:31 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Sort zuul/main.yaml again https://review.opendev.org/653919 | 05:34 |
| *** bhavikdbavishi1 has joined #openstack-infra | 05:35 | |
| *** bhavikdbavishi has quit IRC | 05:36 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 05:36 | |
| *** ykarel|away has joined #openstack-infra | 05:38 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix files for linting after OpenDev change https://review.opendev.org/653919 | 05:56 |
| *** whoami-rajat has joined #openstack-infra | 06:23 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix files for linting after OpenDev change https://review.opendev.org/653919 | 06:50 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix starlingx queue https://review.opendev.org/653923 | 06:51 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add system-required back for repos https://review.opendev.org/653924 | 06:51 |
| *** slaweq has joined #openstack-infra | 07:07 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add system-required back for repos https://review.opendev.org/653924 | 07:15 |
| *** yamamoto has quit IRC | 07:16 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Sort gerritbot/channels.yaml https://review.opendev.org/653928 | 07:17 |
| AJaeger | config-core, please review these changes above to allow us to merge project-config changes again ^ | 07:18 |
| *** trident has quit IRC | 07:19 | |
| *** SotK has quit IRC | 07:20 | |
| *** SotK has joined #openstack-infra | 07:20 | |
| *** yamamoto has joined #openstack-infra | 07:21 | |
| *** yamamoto has quit IRC | 07:21 | |
| *** trident has joined #openstack-infra | 07:21 | |
| *** yamamoto has joined #openstack-infra | 07:22 | |
| *** yamamoto has quit IRC | 07:23 | |
| *** yamamoto has joined #openstack-infra | 07:23 | |
| *** yamamoto has quit IRC | 07:23 | |
| *** yamamoto has joined #openstack-infra | 07:24 | |
| *** yamamoto has quit IRC | 07:24 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix proposal scripts https://review.opendev.org/653966 | 07:29 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add system-required back for repos https://review.opendev.org/653924 | 07:35 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add system-required back for repos https://review.opendev.org/653924 | 07:48 |
| AJaeger | infra-root, we still have four active repos (and some retired) in openstack-infra namespace: zone-opendev.org opendev-website/ zone-zuul-ci.org/ netlify-sandbox/ (plus the retired ones). Something to do later? | 07:56 |
| *** yamamoto has joined #openstack-infra | 07:58 | |
| *** hwoarang has quit IRC | 07:58 | |
| *** jamesmcarthur has joined #openstack-infra | 08:00 | |
| *** hwoarang has joined #openstack-infra | 08:01 | |
| *** yamamoto has quit IRC | 08:05 | |
| *** slaweq has quit IRC | 08:11 | |
| *** hwoarang has quit IRC | 08:23 | |
| *** yamamoto has joined #openstack-infra | 08:25 | |
| *** yamamoto has quit IRC | 08:27 | |
| *** hwoarang has joined #openstack-infra | 08:29 | |
| *** whoami-rajat has quit IRC | 08:33 | |
| *** jamesmcarthur has quit IRC | 09:03 | |
| *** yamamoto has joined #openstack-infra | 09:04 | |
| *** yamamoto has quit IRC | 09:10 | |
| *** yamamoto has joined #openstack-infra | 09:49 | |
| *** yamamoto has quit IRC | 09:50 | |
| *** rascasoft has quit IRC | 09:55 | |
| *** rascasoft has joined #openstack-infra | 09:58 | |
| *** jamesmcarthur has joined #openstack-infra | 10:01 | |
| *** jamesmcarthur has quit IRC | 10:05 | |
| *** rh-jelabarre has joined #openstack-infra | 10:06 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Drop legacy-grenade-dsvm-neutron-multinode-live-migration https://review.opendev.org/640209 | 10:08 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: OpenDev changes https://review.opendev.org/653976 | 10:14 |
| *** kaspars__ has quit IRC | 10:16 | |
| *** ykarel|away has quit IRC | 10:22 | |
| *** yamamoto has joined #openstack-infra | 10:30 | |
| *** yamamoto has quit IRC | 10:31 | |
| *** yamamoto has joined #openstack-infra | 10:31 | |
| *** yamamoto has quit IRC | 10:32 | |
| *** yamamoto has joined #openstack-infra | 10:38 | |
| *** rossella_s has quit IRC | 10:40 | |
| *** yamamoto has quit IRC | 10:41 | |
| *** yamamoto has joined #openstack-infra | 10:53 | |
| *** yamamoto has quit IRC | 10:56 | |
| *** jamesmcarthur has joined #openstack-infra | 11:02 | |
| *** jamesmcarthur has quit IRC | 11:07 | |
| *** whoami-rajat has joined #openstack-infra | 11:18 | |
| *** yamamoto has joined #openstack-infra | 11:36 | |
| *** jamesmcarthur has joined #openstack-infra | 11:41 | |
| *** jamesmcarthur has quit IRC | 11:46 | |
| *** Lucas_Gray has joined #openstack-infra | 11:47 | |
| *** tosky has joined #openstack-infra | 11:51 | |
| *** Wryhder has joined #openstack-infra | 12:11 | |
| *** Lucas_Gray has quit IRC | 12:11 | |
| *** Wryhder is now known as Lucas_Gray | 12:12 | |
| *** roman_g has quit IRC | 12:13 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: OpenDev changes https://review.opendev.org/653976 | 12:16 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Update URL for OpenDev https://review.opendev.org/653986 | 12:20 |
| *** yamamoto has quit IRC | 12:21 | |
| *** yamamoto has joined #openstack-infra | 12:21 | |
| openstackgerrit | Andreas Jaeger proposed opendev/base-jobs master: Fix paths for zuul org rename https://review.opendev.org/653431 | 12:25 |
| AJaeger | fungi, want to abandon https://review.opendev.org/653900 ? | 12:28 |
| *** rascasoft has quit IRC | 12:30 | |
| *** rascasoft has joined #openstack-infra | 12:32 | |
| *** yamamoto has quit IRC | 12:42 | |
| *** jamesmcarthur has joined #openstack-infra | 12:42 | |
| *** jamesmcarthur has quit IRC | 12:46 | |
| *** yamamoto has joined #openstack-infra | 12:52 | |
| *** yamamoto has quit IRC | 12:52 | |
| *** yamamoto has joined #openstack-infra | 12:53 | |
| *** yamamoto has quit IRC | 12:54 | |
| *** yamamoto has joined #openstack-infra | 12:54 | |
| openstackgerrit | Luigi Toscano proposed openstack/devstack-gate master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/653988 | 12:55 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: OpenDev changes https://review.opendev.org/653976 | 13:04 |
| mordred | AJaeger: well that's just unfortunate | 13:07 |
| AJaeger | morning, mordred ! If you have some time, could you review the changes with topic:opendev-gerrit, please? | 13:09 |
| mordred | yup! | 13:09 |
| mordred | AJaeger: and it seems we missed 3 openstack-dev repos too | 13:11 |
| AJaeger | yeah, the sandbox ones | 13:12 |
| *** jamesmcarthur has joined #openstack-infra | 13:13 | |
| smcginnis | Could probably take things down and migrate them over the weekend without anyone really noticing. | 13:14 |
| smcginnis | Especially this weekend. | 13:14 |
| *** lmiccini has quit IRC | 13:15 | |
| mordred | smcginnis: what - you think people might not be working this weekend much? | 13:18 |
| mordred | HACK 24x7 BABY | 13:18 |
| mordred | oh wait | 13:18 |
| smcginnis | :) | 13:18 |
| *** jamesmcarthur has quit IRC | 13:19 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Update jobs for OpenDev https://review.opendev.org/654000 | 13:19 |
| AJaeger | ;) | 13:19 |
| dtroyer | New (to me) detail: Does the hostname in the Gerrit Web UI Download commands need to be review.opendev.org? | 13:21 |
| smcginnis | For the Checkout, Cherry-pick, etc? | 13:22 |
| dtroyer | yes | 13:22 |
| smcginnis | I don't think so. | 13:23 |
| dtroyer | git pll worked for me whan I added 'review.' | 13:23 |
| smcginnis | It failed without? | 13:23 |
| dtroyer | it didn't as-is. | 13:23 |
| smcginnis | Hmm | 13:23 |
| smcginnis | Confirmed | 13:24 |
| *** yamamoto has quit IRC | 13:24 | |
| mordred | dtroyer: oh! yeah - that is a regression | 13:24 |
| mordred | I'll get a patch up to fix it in just a smidge, thanks! | 13:25 |
| smcginnis | It appears the ssh option is correct, but anonymous http is not. | 13:25 |
| dtroyer | it isn't a hard blocker or anything, I've just got a coule of devstacks that use that notation to pull specific reviews. | 13:26 |
| *** smarcet has joined #openstack-infra | 13:26 | |
| *** bhavikdbavishi has quit IRC | 13:26 | |
| *** yamamoto has joined #openstack-infra | 13:26 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Update nodepool for OpenDev https://review.opendev.org/654001 | 13:26 |
| openstackgerrit | Merged zuul/zuul-jobs master: Update URL for OpenDev https://review.opendev.org/653986 | 13:28 |
| smcginnis | git review -d at least works correctly. | 13:28 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Update jobs for OpenDev https://review.opendev.org/654000 | 13:29 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update some paths for opendev https://review.opendev.org/654002 | 13:35 |
| mordred | dtroyer: yeah - so - the issue there is that we're not currently replicating refs/changes to the git farm like before due a bug in gitea. I'm not sure we realized we had folks being all sophisticated like you and pulling the gerrit changes from the replicated farm | 13:36 |
| tosky | also, (part of) this review seems to be needed to fix legacy jobs: https://review.opendev.org/#/c/653988/ | 13:36 |
| tosky | otherwise you get errors like https://review.opendev.org/#/c/637265/ | 13:37 |
| mordred | tosky: thanks - looks great | 13:38 |
| mordred | dtroyer: so we can fix the annonymous http link in gerrit pretty easily - the other thing requires some go hacking, so it's probably best for now to just pull changes from review.opendev.org - sorry about that! | 13:38 |
| dtroyer | mordred: changing the host seems to be working, it isn't a problem for a workaround for me | 13:38 |
| mordred | cool | 13:38 |
| mordred | dtroyer: if you feel like some fun golang hacking and want to figure out why gitea is unhappy when we push refs/changes/* to it - you know, I'd at least buy you a mostly not expired beer | 13:39 |
| dtroyer | mordred: ooooo….. tempting… golang >>>>> devstack | 13:40 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Update nodepool for OpenDev https://review.opendev.org/654001 | 13:40 |
| mordred | dtroyer: also - corvus has made several changes to gitea already and says they've done a good job with their internal abstractions and use of go ... most of his patches have been like 3 lines | 13:41 |
| * mordred tries to nerd-snipe dtroyer | 13:42 | |
| mnaser | hi everyone | 13:44 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: OpenDev changes https://review.opendev.org/653976 | 13:44 |
| AJaeger | morning, mnaser ! | 13:44 |
| mnaser | I'll try to go over project-config changes that are straight forward to help unbreak things | 13:45 |
| *** yamamoto has quit IRC | 13:45 | |
| mnaser | hmm | 13:45 |
| mnaser | I think Gitea doesn't support using .git URLs | 13:45 |
| AJaeger | thanks, mnaser. I used topic:opendev-gerrit for my changes to unbreak stuff | 13:45 |
| mnaser | right? | 13:45 |
| dtroyer | mordred: will trade devstack plugin debugging for golang most any day | 13:46 |
| AJaeger | mnaser: have an example? | 13:46 |
| mnaser | nope, I just tested it, it does support urls with trailing .git | 13:46 |
| mnaser | I was just reviewing a change that didn't drop the trailing .git | 13:46 |
| mnaser | git clone https://opendev.org/openstack/devstack.git worked so I think thats ok | 13:46 |
| mordred | mnaser: there was a bit when it didn't - but corvus fixed it and has submitted a PR :) | 13:47 |
| mnaser | ah wonderful | 13:47 |
| mordred | (we're running a mildly forked gitea now) | 13:47 |
| mnaser | I remember there was talks about that | 13:47 |
| mnaser | ill add it as a note in the review | 13:47 |
| AJaeger | mordred: should we remove the .git when we change? | 13:47 |
| mnaser | I'm inclined to say "yes" in case upstream doesn't like that change and we want to stick to what works by default? | 13:49 |
| mordred | AJaeger: it doesn't matter - it supports both | 13:49 |
| mordred | AJaeger: the .git isn't needed - so we can certainly clean it up - but we don't need to | 13:49 |
| mordred | infra-root: I'm working on opendev cleanup patches for system-config | 13:49 |
| mordred | mnaser: they support .git suffixes in browsing, so I doubt they'll be too upset about it ... but yeah | 13:50 |
| *** smarcet has quit IRC | 13:50 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Update jobs for OpenDev https://review.opendev.org/654000 | 13:51 |
| mnaser | AJaeger: I see in https://review.opendev.org/#/c/653919/4/gerrit/projects.yaml you still kept the cgit-alias URL | 13:51 |
| AJaeger | mnaser: fixed the .git ^ | 13:51 |
| mnaser | is that necessary? | 13:51 |
| mnaser | with gitea/opendev/whatever, doesn't that make it obsolete ? | 13:51 |
| AJaeger | mnaser: I only sorted the file without doing any changes... That kind of cleanup can be done later. | 13:51 |
| mnaser | ok cool | 13:51 |
| mnaser | did you automate sorting it (sorry, I ask because that's a lot of things to eye over :p) | 13:52 |
| mnaser | and if mechanically the first few look ok I won't dig through them all | 13:52 |
| AJaeger | yes, used a script for this. | 13:52 |
| AJaeger | (or one script per file - yamlload, sort, dump) - and having the same number of lines at the end was great relief ;) | 13:53 |
| mnaser | yeah, I figured as much | 13:53 |
| mnaser | the only thing is the fact we're changing zuul/main.yaml / gerrit/projects.yaml / zuul.d/projects.yaml and the "don't merge project creations" thing is there | 13:54 |
| mnaser | I know we're not creating anything but maybe it might pass through the 'codepath' that infra was trying to avoid, I know mordred has a +2 so maybe he can clarify if he's thought that part out and it'd make me feel better +2-nig | 13:55 |
| mnaser | +2-ing* | 13:55 |
| mnaser | AJaeger: forgive my questions, proposal jobs... how do they 'accept' the host key by default? | 13:56 |
| AJaeger | mnaser: IMHO not - but mordred et al know this better than I do, so leave a +2 and let others +A... | 13:56 |
| mnaser | do we need to make another change to update whatever fixes known_hosts or dunno how that's handled | 13:57 |
| AJaeger | mnaser: we might. So, this was the first try ;( | 13:57 |
| corvus | i think it's okay; it will trigger that codepath, but should be a noop and won't run until we re-enable the run_all.sh cron anyway, so it should be something we can keep an eye on. | 13:57 |
| mnaser | morning corvus! okay, based on that, I'll +A to help move things along for AJaeger work | 13:58 |
| AJaeger | morning, corvus ! | 13:58 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update some docs for opendev https://review.opendev.org/654004 | 14:00 |
| *** smarcet has joined #openstack-infra | 14:00 | |
| mordred | yay! my first system-config patch fixes the puppet apply tests | 14:01 |
| mordred | it does not fix the beaker tests ... | 14:01 |
| AJaeger | \o/ | 14:02 |
| * corvus subscribes to new projects in gertty | 14:02 | |
| dtroyer | I am seeing some git_clone issues in devstack, where /opt/stack/<repo> is not present when it expects it to be there, has something changes WRT preparing that workspace? | 14:02 |
| dtroyer | http://logs.openstack.org/60/653060/2/check/ironic-tempest-functional-python3/a7859a4/job-output.txt.gz#_2019-04-20_08_20_28_906834 is one example in an ironic job | 14:03 |
| *** smarcet has quit IRC | 14:04 | |
| corvus | dtroyer: iirc devstack is (probably via the new zuulv3 roles) responsible for moving or symlinking the projects into /opt/stack... maybe it has a rule that assumes openstack/ | 14:05 |
| dtroyer | corvus: ah, I'll head over there. I wasn't sure where that lived, thanks | 14:05 |
| AJaeger | dtroyer: looks liek fallout from moving openstack/pyghmi to x/pyghmi | 14:06 |
| openstackgerrit | Merged openstack/project-config master: Fix files for linting after OpenDev change https://review.opendev.org/653919 | 14:06 |
| mnaser | how has the testing of rewrite rules been done? we can land https://review.opendev.org/#/c/653957/ (after fixing gate) but I dunno if that works, it was just some copy pastage | 14:06 |
| dtroyer | AJaeger: yeah, I'm only seeing it on repos that changed namespaces so far | 14:06 |
| AJaeger | dtroyer: but it should work with the redirects we have... | 14:06 |
| corvus | dtroyer: yeah, i think it's the first task in setup-devstack-source-dirs/tasks/main.yaml | 14:07 |
| mnaser | mordred: if you need help figuring out system-config gate, I can volunteer if that helps land that patch, cause All OSA releases depended on that are broken :< | 14:07 |
| *** ykarel|away has joined #openstack-infra | 14:07 | |
| openstackgerrit | Merged openstack/project-config master: Fix starlingx queue https://review.opendev.org/653923 | 14:07 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update some paths for opendev https://review.opendev.org/654002 | 14:07 |
| *** Lucas_Gray has quit IRC | 14:07 | |
| *** yamamoto has joined #openstack-infra | 14:07 | |
| corvus | dtroyer: https://opendev.org/openstack/devstack/src/branch/master/roles/setup-devstack-source-dirs/tasks/main.yaml#L1-L8 | 14:07 |
| *** yamamoto has quit IRC | 14:08 | |
| dtroyer | AJaeger: if only redirectd worked on filesystems too :) | 14:08 |
| *** whoami-rajat has quit IRC | 14:08 | |
| *** yamamoto has joined #openstack-infra | 14:08 | |
| mordred | mnaser: cool - I think that one - and maybe one more should get it - but I may take you up on that | 14:08 |
| mnaser | mordred: I can baby sit that patch and iterate if you wanna get busy with something else if ya want :) | 14:08 |
| corvus | dtroyer: probably quickest fix is just adding another entry for "x/" | 14:08 |
| AJaeger | corvus: good find. | 14:09 |
| dtroyer | corvus: and starlingx, any others that might want that for now? | 14:09 |
| AJaeger | dtroyer: is starlingx using devstack? Then yes! | 14:09 |
| dtroyer | yup, that's been my winter :) | 14:10 |
| corvus | i wonder if we would need "zuul/" for nodepool | 14:10 |
| corvus | dtroyer: won't hurt to add more | 14:10 |
| dtroyer | I'll add zuul/ | 14:11 |
| openstackgerrit | Merged openstack/project-config master: Add system-required back for repos https://review.opendev.org/653924 | 14:11 |
| openstackgerrit | Merged openstack/project-config master: Fix proposal scripts https://review.opendev.org/653966 | 14:11 |
| openstackgerrit | Merged openstack/project-config master: Sort gerritbot/channels.yaml https://review.opendev.org/653928 | 14:12 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-openstack_infra_spec_helper master: Update for opendev https://review.opendev.org/654005 | 14:12 |
| mordred | corvus, mnaser: it's possible we might need to force merge that ^^ | 14:13 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update some paths for opendev https://review.opendev.org/654002 | 14:13 |
| mordred | if it makes that ^^ green (I added a depends-on) | 14:14 |
| mnaser | hmm | 14:14 |
| dtroyer | corvus, AJaeger: https://review.opendev.org/654006 for the devstack update | 14:14 |
| AJaeger | corvus, mordred , could you double check this nodepool change, please? https://review.opendev.org/654001 | 14:14 |
| mnaser | using the cherry-pick url in the top right of gerrit .. is that not supposed to be working right now? | 14:14 |
| mnaser | git fetch https://opendev.org/openstack/openstack-ansible-tests refs/changes/75/653975/1 && git cherry-pick FETCH_HEAD gives me a "fatal: Couldn't find remote ref refs/changes/75/653975/1" | 14:14 |
| mordred | mnaser: no - the http link is broken | 14:14 |
| AJaeger | corvus, mordred, and also review https://review.opendev.org/654000 , please? | 14:14 |
| mordred | mnaser: it needs to be review.opendev.org for now | 14:14 |
| mnaser | ok, thank you mordred | 14:15 |
| *** jamesmcarthur has joined #openstack-infra | 14:15 | |
| tosky | did anyone mention the fact that the "(gitweb)" links are not working? | 14:16 |
| * AJaeger will be back later | 14:16 | |
| mnaser | tosky: hmm, no, but they should dbe | 14:17 |
| mnaser | they seem to redirect somewhat correctly https://opendev.org/openstack/project-config/commit/ee6468b9b4fde94cba32dc4deb194bc1a61737b0 | 14:17 |
| mnaser | oh I see | 14:18 |
| corvus | mordred, mnaser, tosky: the http links pointing to gitea rather that gerrit are breaking gertty as well | 14:18 |
| corvus | since gertty follows whatever gerrit tells it to do | 14:18 |
| mnaser | I guess cgit before had all of the refs uploaded, but now they live in gerrit and we need to get it from there.. right? | 14:18 |
| tosky | oh, I see how this is connected with the issue with the "Download" link | 14:19 |
| corvus | yeah... we had issues replicating them to gitea so we stopped | 14:19 |
| *** jamesmcarthur has quit IRC | 14:19 | |
| corvus | gitHttpUrl = https://opendev.org/ | 14:21 |
| corvus | that's probably what we want to change? | 14:22 |
| corvus | https://review.opendev.org/Documentation/config-gerrit.html#gerrit.gitHttpUrl | 14:22 |
| mnaser | "By default unset, as the HTTP daemon must be configured externally by the system administrator, and might not even be running on the same host as Gerrit." | 14:23 |
| mnaser | so I think that might want to be https://review.opendev.org/ | 14:24 |
| corvus | patch incoming | 14:26 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Set gerrit's gitHttpUrl to itself https://review.opendev.org/654007 | 14:27 |
| corvus | infra-root: ^ i'd like to manually apply that and restart gerrit; ok? | 14:27 |
| * corvus hears no objection :) | 14:28 | |
| clarkb | seems the defauly is to point at itself if you want to unset it entirely (not sure if puppet maoes that easy) | 14:29 |
| clarkb | I'm not quite here yet but no objections | 14:29 |
| corvus | any other things we should fold into a gerrit restart? | 14:30 |
| mnaser | corvus: maybe you could put the logo fix | 14:30 |
| corvus | (since doing so is going to kick off 2 hours of replication tasks) | 14:30 |
| mnaser | https://review.opendev.org/#/c/653956/ | 14:30 |
| mnaser | afaik, I remember gerrit needing a restart for it to catch the new logo at some point | 14:30 |
| corvus | mnaser: ok will do | 14:30 |
| corvus | file is in place and logo is now broken; assume it will fix on a restart | 14:33 |
| corvus | i'll restart now | 14:33 |
| mnaser | oooh I really hope I didn't break things even more :X | 14:34 |
| corvus | is back up with new logo | 14:36 |
| mnaser | yay, cool | 14:36 |
| mnaser | let me test gitweb url | 14:36 |
| mnaser | hmm | 14:37 |
| mnaser | gitweb url still goes to https://git.openstack.org/cgit/openstack/openstack-ansible-tests/commit/?id=e1b839e0f9b4f50db4c793ea18bc5dea6e5349a1 | 14:37 |
| mnaser | but the cherry-pick one has been fixed -- git fetch https://review.opendev.org/openstack/openstack-ansible-tests refs/changes/75/653975/1 && git cherry-pick FETCH_HEAD | 14:37 |
| corvus | oh i guess that's a different setting | 14:37 |
| paladox | what your looking for is https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#gitweb | 14:37 |
| mnaser | I'm guessing for gitweb is https://review.opendev.org/Documentation/config-gitweb.html#_internal_managed_gitweb | 14:38 |
| mnaser | we will likely have to move towards the internal managed one | 14:38 |
| corvus | well, the links *work* for now :) | 14:38 |
| corvus | the links are entirely customizable | 14:39 |
| corvus | so we can have it work natively with gitea | 14:39 |
| clarkb | but only if we push the changes to gitea? | 14:39 |
| corvus | no | 14:39 |
| corvus | i mean they are customizable in gerrit | 14:39 |
| corvus | if i can find the project this is in, i'll show you :) | 14:39 |
| clarkb | system config passes the data to puppet-gerrit iirc | 14:40 |
| corvus | better yet -- here: http://paste.openstack.org/show/749574/ | 14:40 |
| corvus | that's what's on the production server | 14:41 |
| corvus | it'll take a bit to figure out how that gets there | 14:41 |
| corvus | but you can see that we should be able to fully customize it for gitea | 14:41 |
| corvus | mnaser: also, i think i know where those h= urls came from :) | 14:42 |
| mnaser | so we're not doing something wild after all :-p | 14:42 |
| corvus | (we incorrectly used the wrong cgit argument when we created that) | 14:42 |
| corvus | mnaser: yeah, *infra* was, as it turns out | 14:42 |
| mnaser | im assuming the "file = ${project}/tree/${file}?h=${commit}" is the wrong one there | 14:43 |
| corvus | yeah, aiui cgit would have constructed that as id=commit | 14:43 |
| mnaser | and from there hitting 'raw' probably kept the commit | 14:43 |
| mnaser | anyhow, if someone wants to help test out https://review.opendev.org/#/c/653957/ together, as we already have people showing up wondering why their installs don't work :p | 14:44 |
| corvus | mordred, clarkb: oh! should we set gerrit's git http url to "review.opendev.org/p"? i think we have that as the internal http mirror | 14:45 |
| corvus | wow, it's weirdly slower. | 14:46 |
| corvus | time git fetch https://review.opendev.org/p/openstack/project-config refs/changes/24/653924/5 | 14:46 |
| clarkb | gerrit caches aggressively | 14:46 |
| corvus | that takes 2.5s. the non-p version takes 1.5 | 14:46 |
| clarkb | possoble apache needs to warm up? | 14:47 |
| corvus | i tried a few reps. it's consistent | 14:47 |
| corvus | (i don't have an initial time for gerrit, so i don't know gerrit's cold-cache time) | 14:47 |
| clarkb | huh | 14:47 |
| *** quiquell|off has quit IRC | 14:48 | |
| corvus | maybe we try this for a while, see if gerrit crashes, and if it does, take the hit and switch to /p | 14:48 |
| *** jamesmcarthur has joined #openstack-infra | 14:49 | |
| clarkb | wfm | 14:49 |
| corvus | should i put mnaser's h= fix in place manually? | 14:50 |
| dtroyer | mnaser: 653957 looks like it will get things like the "…/requireemnts/plain/upper-contstraints.ttx" bits in tox.ini? | 14:51 |
| clarkb | I havent been able to review it but if it is to the htaccess file on files that is a really simple thing to edit and change so I say go forit | 14:51 |
| mnaser | dtroyer: for me it's fixing the following url redirect -- https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=af41b096a5a0852a15c3989ec48b956433c47873 | 14:52 |
| dtroyer | yes, I did that manually in a couple of places, thanks | 14:53 |
| *** jamesmcarthur has quit IRC | 14:53 | |
| corvus | clarkb: may i un-draft 15918? | 14:54 |
| corvus | clarkb: and perhaps abandon? | 14:54 |
| corvus | clarkb: (changing the git fetch location has suddenly caused gertty to be able to fetch that, but only if it uses ssh, and i do not want my gertty attempting to ssh into gerrit) | 14:54 |
| clarkb | corvus: yes | 14:55 |
| clarkb | to both | 14:55 |
| corvus | mnaser: try now | 14:56 |
| mnaser | hmm, still redirects to /raw/branch/ instead of /raw/commit/ | 14:57 |
| *** whoami-rajat has joined #openstack-infra | 14:57 | |
| mnaser | Location: https://opendev.org/openstack/requirements/raw/branch/af41b096a5a0852a15c3989ec48b956433c47873/upper-constraints.txt | 14:57 |
| corvus | mnaser: original url? | 14:57 |
| mnaser | testing https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=af41b096a5a0852a15c3989ec48b956433c47873 | 14:57 |
| *** yamamoto has quit IRC | 14:57 | |
| corvus | oh i see | 14:59 |
| corvus | i'll fix and update | 14:59 |
| corvus | mnaser: better now? | 15:01 |
| mnaser | Location: https://opendev.org/openstack/requirements/raw/commit/af41b096a5a0852a15c3989ec48b956433c47873/upper-constraints.txt | 15:01 |
| mnaser | yes :D | 15:01 |
| mnaser | thank you | 15:01 |
| mnaser | want me to update patch (or you will?) with whatever that fix was | 15:01 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: redirects: add support for plain h=<sha> https://review.opendev.org/653957 | 15:02 |
| corvus | mnaser: ^ no problem :) | 15:02 |
| fungi | okay, i'm just about caught up on saturday morning chores and errands, need to go grab an early lunch here in a few minutes and then i should be able to see what's left on the docket and dive in | 15:02 |
| fungi | AJaeger: yeah we should be able to move those repos later, it's not critical | 15:03 |
| corvus | i will eat breakfast while gertty syncs 217 changes | 15:04 |
| * mordred is back online - we're driving to lafayette today, so I'll be on and off - but sandy is driving so I'll be on and off all day | 15:07 | |
| *** yamamoto has joined #openstack-infra | 15:07 | |
| *** yamamoto has quit IRC | 15:07 | |
| *** yamamoto has joined #openstack-infra | 15:08 | |
| corvus | i've published/abandoned 2 more drafts (this time in devstack) | 15:09 |
| corvus | it's *really* good we disabled that. | 15:09 |
| *** bhavikdbavishi has joined #openstack-infra | 15:09 | |
| *** yamamoto has quit IRC | 15:10 | |
| *** yamamoto has joined #openstack-infra | 15:10 | |
| fungi | corvus: still catching up, if gitHttpUrl were https://review.opendev.org/p/ then it would use our on-disk git clone via apache instead of hitting the gerrit jvm's jgit right? | 15:11 |
| mordred | corvus, clarkb, fungi, cmurpheus I believe the beaker job is broken and actually has been | 15:12 |
| mordred | it's not using the version of tools/install_modules_acceptance.sh that's in the patch | 15:12 |
| fungi | and as i catch up further on scrollback i see corvus also realized we probably want /p/ | 15:13 |
| fungi | though interesting it was actually slower | 15:13 |
| mordred | oh - wait ... I think it's still opendev related. digging in further | 15:13 |
| fungi | anyway, i have to go grab food but will be back asap to pitch in | 15:15 |
| paladox | fungi cloning over /p/? | 15:15 |
| fungi | at least i'll hopefully have less scrollback to read through | 15:15 |
| fungi | paladox: we've traditionally had gerrit replicate to a local git tree on the server's filesystem and then served that via smart git http backend through apache at a /p/ subpath to put less load on the gerrit jvm | 15:16 |
| fungi | anyway, heading out, bbiab | 15:16 |
| paladox | ah i see. /p/ is broken in gerrit from 2.16 :( (cloning over it, new ui takes over /p/) (we had to do a mass migration at the WMF) | 15:17 |
| corvus | paladox: oh neat; well if we end up using it i guess we could move it to a different location. but that's good to know we'll at least need to adjust our apache config | 15:18 |
| paladox | We had to patch zuul too | 15:18 |
| paladox | just to unblock our upgrade :) | 15:19 |
| corvus | i think that dates from a very early version of gerrit which used /p internally for git operations. | 15:19 |
| paladox | Yup, i tryed fixing /p/ but was told /p/ was completely optional. | 15:20 |
| mordred | corvus: if you get a sec, could you look at http://logs.openstack.org/02/654002/3/check/openstackci-beaker/8982d9f/job-output.txt.gz#_2019-04-20_14_21_16_508068 | 15:21 |
| mordred | corvus: I'm not finding where we're doing whatever we're doing to cause it to clone system-config directly rather than using what's on the filesystem | 15:21 |
| clarkb | mordred: I'll start trying to look too (I'm slowly booting my day) | 15:22 |
| clarkb | it turns out that getting sleep is difficult after an exciting day like yesterday :) | 15:22 |
| mordred | clarkb: right? | 15:23 |
| mordred | clarkb: ok - I think it may involve puppet-openstackci | 15:23 |
| corvus | yeah, that was fun we should do it more often | 15:23 |
| corvus | mordred: i think that gemfile thing is copied into every repo | 15:23 |
| mordred | yes. I found the issue I think | 15:24 |
| corvus | mordred: so the first part of 654002 may need to be applied to openstackci and possibly others | 15:24 |
| clarkb | if we are tracking priority reviews somewhere let me know. I'm making toast then can start reviewing things | 15:25 |
| corvus | clarkb: i don't think so, but i think we are setting topic opendev-gerrit | 15:26 |
| clarkb | k that is probably good enough to get started | 15:26 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-openstackci master: Fix opendev paths https://review.opendev.org/654009 | 15:26 |
| paladox | corvus https://phabricator.wikimedia.org/T214807 is the task we have for patching zuul :) | 15:26 |
| mordred | corvus, clarkb: ^^ we may need to force-merge that patch, because it needs an openstack-spec-helper patch, but the o-s-h patch needs that patch | 15:27 |
| mordred | I *think* if we do, and then recheck the system-config patch that depends on the spec-helper patch, it should be green (we might need to force merge th spec helper patch too) | 15:28 |
| corvus | mordred: i'm down for these force-merges | 15:28 |
| mordred | corvus: wanna just force-merge openstackci and spec-helper on principle? it's not like they can break anything _worse_ | 15:29 |
| corvus | mordred: yeah, what's the spec-helper change? | 15:30 |
| mordred | corvus: https://review.opendev.org/#/c/654005/ | 15:30 |
| mordred | oh - btw - we should be able to remove the trusty and centos jobs from spec helper | 15:31 |
| mordred | we're off trusty now, right? | 15:31 |
| openstackgerrit | Merged opendev/puppet-openstackci master: Fix opendev paths https://review.opendev.org/654009 | 15:31 |
| openstackgerrit | Merged opendev/puppet-openstack_infra_spec_helper master: Update for opendev https://review.opendev.org/654005 | 15:32 |
| mordred | cool. I'll recheck the system-config patch | 15:32 |
| mordred | clarkb, corvus: I started in on all the system-config references - so if the first patch goes green I'll keep slogging through thoe (there's a lot to update) | 15:33 |
| mordred | oh - also - hound is stale | 15:34 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use local fork of gitea https://review.opendev.org/654010 | 15:34 |
| mordred | do we need to kick/hup it? | 15:34 |
| corvus | mordred: well, the plan was to retire it, so it wasn't in the etherpad | 15:34 |
| corvus | if we want to keep it running for now, i suggest we restart it after run_all.sh is working again | 15:35 |
| mordred | corvus: ah - good point | 15:35 |
| corvus | i don't think it's worth manual intervention when it should mostly fix itself once we have that working | 15:35 |
| corvus | (and it'll work "good enough" for the time being as is) | 15:35 |
| mordred | yeah | 15:35 |
| clarkb | ++ | 15:35 |
| clarkb | I'll start with 654010 beacuse I want to make sure that doesn't get lost | 15:38 |
| mordred | corvus, clarkb: also - did y'all see the scrollback from AJaeger that we missed a few openstack-infra and openstack-dev projects? | 15:38 |
| mordred | clarkb: ++ | 15:38 |
| corvus | mordred: yep | 15:38 |
| clarkb | ya :/ fwiw I think we catch those in a followup | 15:39 |
| clarkb | I think for the zonefile stuff its only us that will notice (and we can deal with that for a little while) | 15:40 |
| clarkb | ah I understand the 1.7.3 version on gitea now. corvus' repo doesn't have the 1.7.4 tag | 15:42 |
| corvus | mordred, clarkb: what do you think about force-merging some of the changes that are already in prod? | 15:42 |
| corvus | that way the system is more in sync for when we're finally ready to enable run_all | 15:42 |
| clarkb | that seems reasonable | 15:42 |
| clarkb | things like the githttpurl and logo update? | 15:42 |
| mordred | corvus: ++ | 15:43 |
| corvus | yep. zuul config | 15:43 |
| mordred | corvus: I support force-merging all of the things that we manually applied to production | 15:43 |
| openstackgerrit | Merged opendev/system-config master: Update gerrit connection in Zuul https://review.opendev.org/653877 | 15:43 |
| openstackgerrit | Merged opendev/system-config master: Update known_hosts for zuul https://review.opendev.org/653955 | 15:43 |
| clarkb | mordred: corvus https://review.opendev.org/#/c/653507/ that one too | 15:44 |
| openstackgerrit | Merged opendev/system-config master: assets: replace logo with side-by-side logo and type https://review.opendev.org/653956 | 15:44 |
| clarkb | actually 653507 might have had to go before ^ | 15:44 |
| clarkb | corvus: ^ fyi | 15:44 |
| corvus | clarkb: can you review https://review.opendev.org/653957 before i force-merge it? | 15:44 |
| clarkb | looking | 15:45 |
| openstackgerrit | Merged opendev/system-config master: Redirect review.openstack.org to review.opendev.org https://review.opendev.org/653507 | 15:45 |
| clarkb | and before we start run_all.sh we can update system-config and diff it against what we had on bridge already | 15:46 |
| clarkb | corvus: where are there two new rules in 653957? | 15:47 |
| clarkb | the first one has a $3 glob replacement but no third glob | 15:47 |
| corvus | clarkb: files.openstack.org /var/www/git-redirect/.htaccess | 15:47 |
| corvus | clarkb: yeah, but it matches its friends | 15:47 |
| corvus | i agree it's probably an error, but i think it's a working one | 15:48 |
| clarkb | heh ok. | 15:48 |
| clarkb | +2'd | 15:48 |
| openstackgerrit | Merged opendev/system-config master: redirects: add support for plain h=<sha> https://review.opendev.org/653957 | 15:48 |
| clarkb | should someone repush https://review.opendev.org/653956 ? | 15:49 |
| corvus | clarkb: i don't understand | 15:49 |
| openstackgerrit | Merged opendev/system-config master: Bind to v4 and v6 in haproxy https://review.opendev.org/653874 | 15:49 |
| clarkb | corvus: https://review.opendev.org/#/c/653507/ and https://review.opendev.org/653956 edit the same svg file and with 653507 merging second its content (the stacked logo) will be applied aiui | 15:50 |
| corvus | clarkb: oh, ugh | 15:50 |
| corvus | i'll do it | 15:50 |
| corvus | i wasn't in gertty so i didn't notice it wasn't stacked | 15:51 |
| clarkb | no worries | 15:51 |
| corvus | clarkb: oh neat | 15:53 |
| clarkb | did it merge them? | 15:53 |
| corvus | they actually technically don't conflict | 15:53 |
| corvus | but we do need a patch to fix | 15:53 |
| corvus | i'll do it | 15:53 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Rename gerrit logo file https://review.opendev.org/654012 | 15:55 |
| openstackgerrit | Merged opendev/system-config master: Set gerrit's gitHttpUrl to itself https://review.opendev.org/654007 | 15:56 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update some docs for opendev https://review.opendev.org/654004 | 15:57 |
| corvus | clarkb: if that lgty, i'll submit it and then i think we're caught up | 15:57 |
| clarkb | corvus: I think you need to edit modules/openstack_project/manifests/gerrit.pp for the new filename too possibly? | 15:57 |
| corvus | clarkb: you already did that | 15:57 |
| clarkb | oh sorry I pulled master to git review -s again and then didn't recheck out my change | 15:58 |
| corvus | clarkb: well, master should have your change now | 15:58 |
| corvus | modules/openstack_project/manifests/gerrit.pp: source => 'puppet:///modules/openstack_project/opendev.svg', | 15:58 |
| clarkb | corvus: it does but it hasn't replicated yet | 15:58 |
| clarkb | my correct remotes process was to update from gitea, check that out, git remote rm, git review -s, then I didn't go back to old change | 15:59 |
| corvus | gotcha | 15:59 |
| clarkb | in any case I agree yuor change is good I will +2 as soon as I view the svg locally | 15:59 |
| corvus | 4514 tasks | 15:59 |
| clarkb | corvus: +2'd | 16:00 |
| openstackgerrit | Merged opendev/system-config master: Rename gerrit logo file https://review.opendev.org/654012 | 16:00 |
| corvus | okay, i think we're in sync with production now. | 16:02 |
| mordred | corvus: gerrit seems to think that 654002 is in merge-conflict | 16:02 |
| clarkb | we should allow replication to complete before trying to use any automated deployment stuff | 16:02 |
| mordred | corvus: but if I try to rebase, there's nothing to do and no new changes | 16:02 |
| clarkb | mordred: is it possible you have the stale gitea problem too? | 16:03 |
| mordred | oh - maybe it's just replication related and I'm not actually pulling master | 16:03 |
| mordred | yeah | 16:03 |
| clarkb | mordred: you amy want to update from gerrit | 16:03 |
| corvus | yep, i see conflicts | 16:03 |
| mordred | yup | 16:03 |
| mordred | fixing now | 16:03 |
| clarkb | one weekend where I get good sleep and have nothign to do and it is raining outside I'm going to write a gerrit patch to make replicate on start optional. You can always manually trigger it | 16:04 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update some paths for opendev https://review.opendev.org/654002 | 16:05 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update some docs for opendev https://review.opendev.org/654004 | 16:05 |
| clarkb | with 654002 one thing we may want to test is how vcsrepo (and ansible git module) handle changing a remote in a repo | 16:05 |
| clarkb | I assume it will be all fine, but who knows | 16:05 |
| mordred | yeah - and the redirect should also work properly | 16:06 |
| mordred | if it doesn't update the remote propertly | 16:06 |
| clarkb | ya | 16:06 |
| openstackgerrit | Merged opendev/base-jobs master: Fix paths for zuul org rename https://review.opendev.org/653431 | 16:06 |
| corvus | that was a zuul merge :) | 16:07 |
| clarkb | woot | 16:07 |
| clarkb | there were a couple zuul merges in openstack projects last night before I stopped looking at my monitor | 16:07 |
| clarkb | I need to watch the kids for a bit sometime this morning so that larissa can run errands. Is now a good time given the replication backlog? | 16:08 |
| corvus | 3072 tasks | 16:08 |
| corvus | seems good to me | 16:08 |
| clarkb | ok | 16:08 |
| mordred | our docs have such fun historical easter eggs in them | 16:20 |
| mordred | the paste docs still talk about drizzle | 16:20 |
| pabelanger | yum, easter eggs | 16:20 |
| clarkb | mordred: that is because welove drizzle | 16:21 |
| clarkb | drizzle: openstack's weord uncle | 16:21 |
| clarkb | but weird in a good way | 16:21 |
| mordred | hell yeah | 16:21 |
| * mordred afks for a few | 16:23 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Fix logstash filter location https://review.opendev.org/654016 | 16:30 |
| corvus | that should fix the logstash-filters job | 16:30 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Update some paths for opendev https://review.opendev.org/654002 | 16:31 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Update some docs for opendev https://review.opendev.org/654004 | 16:31 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Fix logstash filter location https://review.opendev.org/654016 | 16:31 |
| corvus | mordred: ^ your readme.md -> rst change was incomplete so i updated it; be sure you pull if you do anything else | 16:31 |
| corvus | that's all the errors i can find in the latest run of jobs | 16:34 |
| mnaser | corvus: 654004 needs a small revision too | 16:39 |
| mnaser | /home/zuul/src/opendev.org/opendev/system-config/doc/source/systems.rst:6:toctree contains reference to nonexisting document 'jjb' | 16:40 |
| mnaser | (both jjb and git need to be removed) | 16:40 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Update some docs for opendev https://review.opendev.org/654004 | 16:41 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Fix logstash filter location https://review.opendev.org/654016 | 16:41 |
| corvus | mnaser: thx! | 16:41 |
| mordred | corvus: cool | 17:02 |
| pabelanger | clarkb: I take it with all the repo renames, we should also update dib cache on images? | 17:06 |
| pabelanger | s/dib/git | 17:06 |
| *** whoami-rajat has quit IRC | 17:06 | |
| *** larainema has quit IRC | 17:08 | |
| mnaser | seems like 654002 is missing still a reference to logstash | 17:08 |
| clarkb | pabelanger: it gets that from the projects.yaml iirc | 17:08 |
| mnaser | clarkb, pabelanger: AJaeger pushed up a patch to fix up the paths for that | 17:08 |
| mnaser | https://review.opendev.org/#/c/654001/ | 17:09 |
| corvus | mnaser: logstash is fixed in https://review.opendev.org/654016 | 17:11 |
| mnaser | corvus: ok I assume then we'll just force merge that entire stack at this point? | 17:12 |
| corvus | mnaser: that's how i'm leaning, once we're happy that the full set of changes fixes the jobs | 17:12 |
| mnaser | fair enough :) and given that the top of the stack is passing, it seems good overall | 17:13 |
| corvus | ya | 17:13 |
| corvus | everything failing in the first change is passing in the third; let's see what #2 does | 17:13 |
| corvus | yeah, everything failing or still running in 654004 is green in 654016 | 17:14 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: OpenDev Migration Patch https://review.opendev.org/654023 | 17:14 |
| mnaser | yep it looks like they indeed progressively fixed more things | 17:14 |
| corvus | mnaser, mordred, clarkb: so i think we're ready to force-merge 654002 and 654004 | 17:15 |
| corvus | we can either force-merg 654016 or let is go through on its own | 17:15 |
| mordred | corvus: kk | 17:15 |
| fungi | and i'm back and catching up on what else i've missed | 17:15 |
| mnaser | infra-puppet-apply-3-ubuntu-trusty hasn't finished yet but xenial passed | 17:16 |
| mnaser | which means in theory the path related things are probably ok | 17:16 |
| fungi | haha, seems i need to update my e-mail filters for the gerrit hostname change | 17:16 |
| mnaser | for 654004 | 17:16 |
| corvus | mnaser: yeah, 3-trusty passed on 654016 though, so should be good | 17:17 |
| mnaser | ah yep, true | 17:17 |
| pabelanger | mnaser: +3 | 17:17 |
| clarkb | sorry not quite back yet but if end of stack passes the merging the intermediates sounds good tk me | 17:17 |
| openstackgerrit | Merged opendev/system-config master: Update some paths for opendev https://review.opendev.org/654002 | 17:18 |
| openstackgerrit | Merged opendev/system-config master: Update some docs for opendev https://review.opendev.org/654004 | 17:18 |
| corvus | clarkb, fungi, mordred: +3 https://review.opendev.org/654016 ? | 17:18 |
| * corvus removes himself from project bootstrappers | 17:19 | |
| openstackgerrit | Merged openstack/devstack-gate master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/653988 | 17:19 |
| mordred | \o/ | 17:19 |
| corvus | (in case there was any doubt, the devstack-gate change was a zuul merge) | 17:20 |
| mnaser | https://review.opendev.org/#/c/654000/3 is a nice clean up patch by AJaeger too btw | 17:20 |
| pabelanger | corvus: +2 with comment on 654016 | 17:20 |
| clarkb | pabelanger: the answer to your question is yes, want to push that patch? | 17:21 |
| clarkb | dtroyer's fix for devstack looks like it will pass tests https://review.opendev.org/#/c/654006/ | 17:22 |
| *** panda has quit IRC | 17:23 | |
| pabelanger | clarkb: kk | 17:23 |
| corvus | i've gone ahead and +2d the devstack change | 17:23 |
| clarkb | corvus: I'll review it momentarilly too | 17:23 |
| clarkb | +3'd | 17:24 |
| *** panda has joined #openstack-infra | 17:25 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: OpenDev changes https://review.opendev.org/653976 | 17:32 |
| *** kaspars__ has joined #openstack-infra | 17:32 | |
| AJaeger | fungi, I'll abandon your change https://review.opendev.org/653900 - that was the project-config "mass rename" that you merged locally. | 17:33 |
| clarkb | looking at git01 apache logs to double check that people have migrated off of it (eg no bad dns caches) I think we are looking good there | 17:33 |
| clarkb | I only see webcrawlers in the access logs | 17:33 |
| openstackgerrit | Merged openstack/project-config master: Update nodepool for OpenDev https://review.opendev.org/654001 | 17:36 |
| openstackgerrit | Merged openstack/project-config master: Update jobs for OpenDev https://review.opendev.org/654000 | 17:36 |
| *** slaweq has joined #openstack-infra | 17:38 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Update grafana for OpenDev https://review.opendev.org/654026 | 17:39 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove running your own doc https://review.opendev.org/654027 | 17:40 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Finish updating docs for opendev https://review.opendev.org/654028 | 17:40 |
| mordred | clarkb, corvus: ok - that should take care of all of system-config docs | 17:40 |
| fungi | AJaeger: thanks, still getting my gertty sync'd up | 17:40 |
| mordred | I left some notes in the commit message there - notably we should probably go ahead and merge the zuul and zuulv3 docs :) | 17:40 |
| AJaeger | could a grafana expert check the first two or three files of 654026, please? I used sed to update git_openstack_org with opendev_org | 17:40 |
| fungi | surprised any web crawlers are hitting the old git farm... ignoring ttls in dns? | 17:40 |
| mordred | but I tried to stay away from getting nerd-sniped into rewriting the docs too much today | 17:40 |
| clarkb | fungi: ya I'm guessing they are trying to get at things on less public internet | 17:41 |
| clarkb | fungi: not really a concern for us to turn it off on them | 17:41 |
| fungi | i wholeheartedly concur | 17:41 |
| * mordred agrees - turn em off! | 17:41 | |
| fungi | yoink! | 17:41 |
| mordred | clarkb: since I'm doing system-config cleanup patches - want me to make one that removes the git.openstack.org servers from the config? | 17:42 |
| clarkb | mordred: on the etherpad I noted cleanign those up during the week. I don't think there ism uch urgency there and we can ignore it to focus on more important bits | 17:42 |
| mordred | ok. | 17:42 |
| AJaeger | mordred: 027 is failing tox-docs, see http://logs.openstack.org/27/654027/1/check/tox-docs/f6553b8/job-output.txt.gz#_2019-04-20_17_42_26_770618 | 17:43 |
| AJaeger | "doc/source/running-your-own.rst:document isn't included in any toctree" | 17:43 |
| AJaeger | you forgot to delete the file... | 17:44 |
| mordred | AJaeger: poo | 17:44 |
| mordred | one sec :) | 17:44 |
| AJaeger | trivial fix ;) | 17:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove running your own doc https://review.opendev.org/654027 | 17:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Finish updating docs for opendev https://review.opendev.org/654028 | 17:44 |
| mordred | AJaeger: fixed! | 17:44 |
| AJaeger | team, https://opendev.org/openstack/openstack gives a 500 error ("internal server errror") | 17:45 |
| AJaeger | mordred: nice | 17:45 |
| mordred | exciting | 17:45 |
| clarkb | AJaeger: my hunch is that all of the submodule urls need updating | 17:46 |
| clarkb | or some of them need updating | 17:46 |
| mordred | yeah | 17:46 |
| AJaeger | clarkb: ah, indeed. | 17:46 |
| mordred | we wrote a script to update it from projects.yaml in governance | 17:46 |
| clarkb | and it is erroring trying to load data about invalid submodules | 17:46 |
| mordred | I can't remember if we landed it or not | 17:46 |
| AJaeger | seems it landed - I'll look into it | 17:47 |
| clarkb | I know cncfs thingamajig uses that repo | 17:47 |
| clarkb | so udpating it would probably make a certain selection of people happy | 17:47 |
| mordred | yes. generate-gitmodules | 17:47 |
| mordred | in the openstack/openstack repo | 17:47 |
| AJaeger | mordred: yes | 17:48 |
| mordred | I'm tethered to phone - and I think it's going to clone a lot of things | 17:48 |
| mordred | so if someone else wants to do that ... the script might also want to be updated | 17:48 |
| mordred | (might not need to be) | 17:48 |
| corvus | mordred: -1 on 654028 because the puppet page is halfway useful | 17:48 |
| corvus | it's like, half right and half wrong | 17:48 |
| mordred | corvus: oh yeah? so undelete it, fix the links and worry about the wrong stuff later? | 17:49 |
| corvus | mordred: yeah; maybe stick a big "this is out of date" note in there if you want | 17:49 |
| corvus | mordred: but i think it's better to keep and update than delete | 17:49 |
| AJaeger | mordred: I'm on it... | 17:49 |
| clarkb | as a friendly reminder to those of us working on a weekend. In a week many of us will be travelling to denver for a really long week of summit and ptg there. All this to say take care of yourselves and don't overdo it between now and then. (I'm feeling pretty wiped out already) | 17:49 |
| mordred | clarkb: ++ | 17:50 |
| corvus | to that end -- should we see about getting run_all.sh running and then run away? :) | 17:51 |
| clarkb | yes I think getting that going is a good goal | 17:51 |
| clarkb | corvus: maybe we want to run the components of that script one by one in the foreground and watch it directly? | 17:52 |
| clarkb | then maybe run it again as the full script in the foreground if it works, then enable the cron after that? | 17:52 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Finish updating docs for opendev https://review.opendev.org/654028 | 17:52 |
| tosky | uh, is the Depends-On: https://review.openstack.org still supposed to work (or at least when the change touches zuul jobs)? I had to change this depends-on to opendev.org: https://review.opendev.org/#/c/638390/ | 17:52 |
| mordred | corvus: k. ^^ that shoudl be good | 17:52 |
| openstackgerrit | Merged opendev/system-config master: Fix logstash filter location https://review.opendev.org/654016 | 17:52 |
| clarkb | tosky: I think because we chagned teh zuul config the old name won't work for that. We can possibly update zuul to have aliases for that though | 17:52 |
| corvus | tosky: no, in fact, all of the depends-on headers will need to be changed | 17:52 |
| corvus | er, footers | 17:53 |
| tosky | oh, ok | 17:53 |
| fungi | i didn't consider that, but great point | 17:53 |
| clarkb | considering we aren't likely to change the gerrit name again I'm not sure how much value there is in updating zuul to support aliases like that | 17:53 |
| mordred | clarkb: we should still probably send out an announcement specifically about it | 17:54 |
| clarkb | ya that is a good idea | 17:54 |
| tosky | that would make sense | 17:54 |
| mordred | since here's likely a ton of those in patches in flight that are now not depending on :) | 17:54 |
| clarkb | I can draft one up really quickly | 17:54 |
| tosky | would it be possible to generate a list of open patches with Depends-On lines, and comment on them? | 17:55 |
| * clarkb checks a query | 17:55 | |
| fungi | tosky: you can query gerrit for that pretty easily i think | 17:55 |
| clarkb | https://review.opendev.org/#/q/message:depends-on+status:open | 17:56 |
| clarkb | that seems to work | 17:56 |
| fungi | maybe message:"depends-on: https://review.openstack" | 17:56 |
| fungi | so as not to catch ones which are fixed | 17:56 |
| clarkb | good point | 17:56 |
| fungi | and also that will skip the old-style depends-on which (ironically) shouldn't be broken by the domain change | 17:56 |
| clarkb | https://review.opendev.org/#/q/message:%22depends-on:+https://review.openstack%22+status:open | 17:56 |
| corvus | since there are already 11 using the new name | 17:57 |
| *** slaweq has quit IRC | 17:58 | |
| corvus | clarkb: i'm inclined to just run the script in the foreground, but not run ic piecemeal | 17:58 |
| fungi | for that matter, there could also be in-flight dependencies on github or other code locations zuul knows about which aren't impacted | 17:58 |
| clarkb | corvus: ok. I think my biggest concern is that ansible/puppet won't update our project-config git repo properly because the remote changes and then maybe it has a sad against eg gerrit. I think the chances of that are low and as mordred points out the old remote works so even lower chance of impact | 17:59 |
| clarkb | fungi: ya I think we still send the email | 17:59 |
| clarkb | https://etherpad.openstack.org/p/eoKVci3kvo I'm drafting that | 17:59 |
| *** dikonoor has quit IRC | 17:59 | |
| AJaeger | clarkb: yeah, all submodule urls need updating - and the script gets them from the governance repo. Unfortunatley governance repo is not updated yet. mnaser, is that on your radar? | 17:59 |
| clarkb | then if someone can script up a helpful comment via that query that can eb done too | 17:59 |
| fungi | clarkb: well, i meant changes in gerrit using "depends-on: https://github.com/..." won't be impacted either, as reasoning for limiting the query to just the old review base url | 18:00 |
| mnaser | AJaeger: trying to get context.. | 18:00 |
| fungi | but yes, there could be impact in the other direction we're not finding too | 18:00 |
| clarkb | fungi: oh ya | 18:01 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update opendev git references in puppet modules https://review.opendev.org/654029 | 18:01 |
| *** ykarel|away has quit IRC | 18:01 | |
| fungi | mnaser: basically asking if we have a change yet to update the governance repo in the wake of repository namespace moves | 18:01 |
| mnaser | nothing proposed and nothing merged afaik | 18:02 |
| fungi | especially the reference/projects.yaml and related files | 18:02 |
| AJaeger | mnaser: governance/reference/projects.yaml does not follow the new changes - and we have some scripts to look at that, e.g. to update openstack/openstack | 18:02 |
| fungi | yeah, that will definitely need updating sooner rather than later. we also have stuff like election tooling which relies on that being corrected | 18:02 |
| mordred | fungi: can we use your script as a base of a script to update the entries there? | 18:03 |
| *** bhavikdbavishi has quit IRC | 18:03 | |
| mordred | or maybe just grab one of the repos.yaml files we already have | 18:03 |
| fungi | mordred: we could, but i think there are some unanswered bits like stuff that moved *out* of governance may need to be split into additional changes | 18:03 |
| mordred | ah - good point | 18:04 |
| mnaser | im assuming a lot of under openstack-infra/<foo> stuff too | 18:04 |
| clarkb | https://etherpad.openstack.org/p/eoKVci3kvo is a complete email now. how does it look? | 18:04 |
| fungi | for example all the things we moved to opendev | 18:04 |
| corvus | clarkb: okay, i could get behind running it piecemeal | 18:04 |
| corvus | clarkb: lgtm | 18:05 |
| mnaser | is there a utility that can help fix up projects.yaml? | 18:05 |
| fungi | clarkb: message lgtm too | 18:05 |
| clarkb | ok I'll send that out probably as a new thread against the 4 lists (not cross posting) | 18:06 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-gerrit master: Fix opendev urls https://review.opendev.org/654031 | 18:07 |
| fungi | mnaser: maybe the opendev-patching script we used to adjust zuul configs could be tweaked to do it, but as noted there are a lot of repo removals from various governance lists which may need to be split into different changes for clarity of voting | 18:08 |
| fungi | and generally removals from reference/projects.yaml need to be coupled with dated additions to reference/legacy.yaml for completeness | 18:09 |
| corvus | infra-root: will you join me in screen on bridge? | 18:09 |
| fungi | joining | 18:09 |
| fungi | this is the existing session with a root prompt in /home/corvus? | 18:09 |
| corvus | i've created this script to run the playbooks, how does that sound? http://paste.openstack.org/show/749576/ | 18:09 |
| corvus | fungi: yep | 18:09 |
| fungi | cool, i was still joined to it from last night | 18:10 |
| clarkb | give me a minute to send these emails then I will join | 18:10 |
| fungi | corvus: this is a subset of run_all.sh? | 18:11 |
| clarkb | fungi: I think it is the whole thing with "breaks" inserted | 18:12 |
| clarkb | corvus: lgtm | 18:12 |
| corvus | correct, whole with breaks | 18:12 |
| clarkb | I even tested the breaks locally with a simple script (beacuase read in bash is not something I use often) | 18:13 |
| corvus | clarkb: you weren't the only one :) | 18:13 |
| clarkb | does it work with tee too? | 18:13 |
| clarkb | looks like yes | 18:14 |
| fungi | oh, hah, my eye totally skipped over the "read" commands interspersed throughout | 18:14 |
| fungi | okay, yes i agree that looks like the whole of run_all.sh | 18:14 |
| corvus | ready to run that command? | 18:14 |
| fungi | yep | 18:14 |
| clarkb | I am too | 18:14 |
| corvus | i made it bash -x; good? | 18:15 |
| clarkb | ++ | 18:15 |
| fungi | thanks | 18:15 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-gerrit master: Add support for gitea links https://review.opendev.org/654033 | 18:15 |
| clarkb | ok that didn't update the /opt/system-config remote ( I think it will do that on the next run) but the commit is the logtash fix which is I think what we want? | 18:16 |
| corvus | i believe that should be head yes | 18:16 |
| clarkb | clarkb@bridge:/etc/puppet/modules/accessbot has the new remote and its HEAD is fungi's opendev migration change | 18:17 |
| clarkb | so spot check on the puppet module updates looks good | 18:17 |
| corvus | anythinge else, or proceed? | 18:17 |
| corvus | next up is: # Update the code on bridge | 18:17 |
| corvus | timeout -k 2m 10m ansible-playbook ${ANSIBLE_PLAYBOOKS}/bridge.yaml | 18:17 |
| mordred | corvus: we haven't landed a change to remove cgit farm from replication | 18:18 |
| * AJaeger is giving up on openstack/openstack - with all the renames, this needs more than I can do right now ;( Sorry | 18:18 | |
| clarkb | /etc/ansible/roles are installed via galaxy and seem to not be git repos? | 18:18 |
| corvus | mordred: ha! | 18:18 |
| mordred | corvus: I can make one real quick | 18:18 |
| clarkb | I'm gonna see if I can find evidence that our puppet ansible role is up to date | 18:18 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use gitea links in gerrit https://review.opendev.org/654034 | 18:18 |
| corvus | mordred: ++ i'm inclined to abort this run, land that quickly and start again | 18:18 |
| corvus | clarkb, fungi, mordred: ^ sound good? | 18:19 |
| clarkb | the .gitreview in /etc/ansible/roles/puppet says opendev so I think it also successfully updated so that first bit lgtm | 18:19 |
| mordred | clarkb: ++ | 18:19 |
| clarkb | corvus: +1 to aborting then starting again when ready with replication update | 18:19 |
| mordred | corvus: want me to include not replicating to k8s too? | 18:19 |
| corvus | mordred: yes pls | 18:19 |
| fungi | yes, i agree | 18:19 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop replicating to cgit and k8s-gitea https://review.opendev.org/654035 | 18:20 |
| mordred | corvus: ^^ there we go | 18:20 |
| AJaeger | config-core, please review https://review.opendev.org/653976 to fix openstack-zuul-jobs. It needed updating of the fetch-zuul-cloner test, I'm not sure whether that change is what we want. | 18:20 |
| mordred | fwiw - I also did a puppet-gerrit and system-config patch just now to support gitea links and to update gerrit config to use that - but that's less urgent :) | 18:20 |
| clarkb | AJaeger: will try to though probably focused on the run_all stuff for the next little while. Thank you for poking at these things | 18:21 |
| AJaeger | config-core, https://review.opendev.org/654026 updates grafana - please review as well. | 18:21 |
| AJaeger | thanks, clarkb | 18:21 |
| * AJaeger signs off for now | 18:22 | |
| fungi | thanks AJaeger! | 18:22 |
| corvus | i'll direct-enqueue 654035 | 18:23 |
| mordred | fungi: thanks for the +2 on the puppet-gerrit - we need one on https://review.opendev.org/#/c/654031/1 as well (parent change) | 18:24 |
| fungi | i thought i did | 18:24 |
| fungi | oh, i did not! | 18:24 |
| corvus | what's with the merge failures on that? | 18:25 |
| clarkb | merge failures on what specifically? (I'm not seeing them on the last two changes listed above) | 18:26 |
| corvus | mordred's gitea link changes | 18:26 |
| mordred | corvus: it's a conflict - I'm about to push up a fix | 18:26 |
| corvus | mordred: https://review.opendev.org/654029 conflict there | 18:27 |
| mordred | yeah - it's on the logstash filter url | 18:27 |
| *** pcaruana has joined #openstack-infra | 18:27 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove running your own doc https://review.opendev.org/654027 | 18:27 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Finish updating docs for opendev https://review.opendev.org/654028 | 18:27 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update opendev git references in puppet modules https://review.opendev.org/654029 | 18:27 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use gitea links in gerrit https://review.opendev.org/654034 | 18:27 |
| corvus | cool, that's all normal behavior then :) | 18:27 |
| *** Lucas_Gray has joined #openstack-infra | 18:27 | |
| mordred | do we know if the legacy-logstash-filters job failures are a thing we need to fix - or have they been fixed already? | 18:28 |
| clarkb | I believe they have been fixed | 18:29 |
| mordred | cool | 18:29 |
| mordred | that stack should show us if we're wrong :) | 18:29 |
| corvus | the fix for the logstash filters job was self-tested. | 18:32 |
| mordred | cool | 18:32 |
| mordred | then that stack should not show us that we are wrong :) | 18:32 |
| corvus | it better not! | 18:32 |
| clarkb | zuul says eta 12 minutes on the replication fix | 18:32 |
| mordred | should we run a dev gitea for review-dev? or should we replicate review-dev to opendev.org? or neither? | 18:34 |
| mordred | review-dev is currently configured to show web links to git.openstack.org - which isn't correct for review-dev anyway | 18:34 |
| clarkb | mordred: I think the ideal here is moving more twoards easily deploying locally? but maybe we can run a gitea on the same host for now? | 18:35 |
| mordred | so I'm thinking just update it to point to opendev for now, so that the config is the same - and we can sort out what, if anyhting, we want to do with replicating those repos later | 18:35 |
| mordred | clarkb: good call on both | 18:35 |
| mordred | clarkb: maybe once we get our gerrit container going we can have a nice easy gitea+gerrit docker compose | 18:35 |
| clarkb | ya | 18:36 |
| *** Lucas_Gray has quit IRC | 18:37 | |
| clarkb | a dev gitea + review-dev might be a good place to test full gerrit replication to gitea if/when that is addressed | 18:38 |
| *** Lucas_Gray has joined #openstack-infra | 18:39 | |
| corvus | infra-puppet-apply-3-ubuntu-trusty failed | 18:40 |
| corvus | is that known flaky? | 18:40 |
| corvus | http://logs.openstack.org/35/654035/1/gate/infra-puppet-apply-3-ubuntu-trusty/bdc47f8/applytest/puppetapplytest51.final.out.FAILED | 18:40 |
| corvus | copying puppet modules for the ask server is what failed | 18:41 |
| clarkb | that should be a cp/rsync from /home/zuul/src to /etc/puppet/modules iirc | 18:41 |
| clarkb | oh maybe it isn't | 18:41 |
| corvus | it passed it check | 18:41 |
| clarkb | since that is the actual ansible copy | 18:42 |
| corvus | it passed in check | 18:42 |
| clarkb | I'm inclined to treat that as flaky for now and we can fix later if we want to merge it assuming everything else looks happy | 18:42 |
| corvus | i think if all the other jobs pass we should force-merge it | 18:42 |
| clarkb | we also get puppet modules from github | 18:42 |
| clarkb | so it could've failed there | 18:42 |
| clarkb | corvus: +1 | 18:42 |
| corvus | if that's the source of flakiness, we could maybe have zuul do that | 18:43 |
| corvus | via required-projects | 18:43 |
| clarkb | ya hard to say for sure with the no log on that play (maybe we can reduce the logging it does but still get the error message? I think it is no log simply because its huge) | 18:44 |
| corvus | if so maybe we could change to 'command' instead of 'synchronize' | 18:44 |
| openstackgerrit | Merged opendev/system-config master: Stop replicating to cgit and k8s-gitea https://review.opendev.org/654035 | 18:45 |
| corvus | replication is complete; shall we run the script again? | 18:46 |
| clarkb | yes and lets double check that /opt/system-config's remote changes after this second run | 18:46 |
| clarkb | mordred: fungi ^ fyi I think we are starting again | 18:46 |
| mordred | clarkb: cool | 18:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Finish removing git.openstack.org references https://review.opendev.org/654040 | 18:47 |
| mordred | wow. that's it for system-config | 18:47 |
| clarkb | it is still git.openstack.org making me think we are missing a url update somewhere | 18:47 |
| mordred | that was not quick :) | 18:47 |
| mordred | clarkb: what's still git.openstack.org ? | 18:47 |
| fungi | following | 18:47 |
| clarkb | https://review.opendev.org/#/c/654040/1/playbooks/update-system-config.yaml is the fix I think | 18:47 |
| clarkb | mordred: /opt/system-config remotes | 18:47 |
| clarkb | mordred: your change above is the fix I think | 18:48 |
| mordred | ah - yeah. cool | 18:48 |
| clarkb | 9fdd8ca3d26d6430cbb8356695f1b4ac4112b9f0 is checked out in /opt/system-config | 18:48 |
| corvus | system-config head lgtm | 18:48 |
| clarkb | which is the stop replication commit | 18:48 |
| clarkb | corvus: yup | 18:48 |
| clarkb | /etc/ansible/roles/* lgtm and spot check of /etc/puppet/modules also looksgood | 18:49 |
| corvus | proceed or more checks? | 18:49 |
| clarkb | I'm ready to proceed if others are | 18:49 |
| corvus | fungi, mordred: ? | 18:50 |
| fungi | yeah, looks right to me | 18:50 |
| corvus | mordred has probably been pulled over, so i'll continue | 18:51 |
| clarkb | I think that just enabled the cron | 18:52 |
| clarkb | yes, any objection to disabling it? | 18:52 |
| corvus | i disabled it | 18:52 |
| clarkb | ty | 18:52 |
| clarkb | and yes looks disabled to me | 18:52 |
| corvus | next is # Run k8s-on-openstack | 18:53 |
| corvus | timeout -k 2m 120m ${SYSTEM_CONFIG}/run_k8s_ansible.sh | 18:53 |
| corvus | which shouldn't be very interesting as i think it's still semi-broken | 18:53 |
| fungi | looks commented back out to me | 18:53 |
| clarkb | I didn't see anything scary in the output of bridge.yaml other than the cron being enabled | 18:53 |
| clarkb | I think we can likely proceed unless we want to verify our ansible installation by hand | 18:54 |
| clarkb | (presumably the next playbook will smoketest that anyway) | 18:54 |
| corvus | the next playbook should verify a bunch of stuff :) | 18:54 |
| clarkb | this is a chatty one | 18:54 |
| fungi | i'm all for having the computer verify the other computers there | 18:54 |
| fungi | something it's far better at than i am | 18:55 |
| * fungi suspects that's the true origin of skynet | 18:55 | |
| clarkb | we hit our first error :) | 18:55 |
| clarkb | pip3 freeze | grep os-client-config shows no entries | 18:56 |
| corvus | i assume that's just it being broken | 18:56 |
| clarkb | does the cloud launcher playbook not rely on os-client-config? | 18:57 |
| fungi | i can't immediately think of anything we would have done to bring that condition about | 18:57 |
| clarkb | but ya it may just be it broken | 18:57 |
| corvus | Friday 19 April 2019 06:32:20 +0000 (0:00:00.057) 0:01:02.065 ********** | 18:57 |
| corvus | fatal: [localhost]: FAILED! => {"changed": false, "msg": "os-client-config is required for this module"} | 18:57 |
| fungi | so likely preexisting | 18:57 |
| corvus | that's pre-existing | 18:57 |
| fungi | indeed | 18:57 |
| clarkb | /root/.pip/pip.log doesn't have anything useful | 18:57 |
| clarkb | ok cool | 18:57 |
| corvus | proceed? | 18:57 |
| clarkb | I think so | 18:57 |
| fungi | yep! | 18:57 |
| clarkb | next is bootstrap k8s nodes | 18:58 |
| corvus | which will probably also fail | 18:58 |
| corvus | guess not :) | 18:58 |
| clarkb | next is update_puppet_version which should go through all our puppet hosts andi nstall puppet4 if they are in the puppet4 group and don't already have it installed | 18:58 |
| clarkb | that should noop | 18:59 |
| clarkb | (because we haven't merged any new puppet-4ification changes) | 18:59 |
| corvus | this usually does not have errors | 18:59 |
| corvus | oh no it does | 19:00 |
| clarkb | they are ssh connection errors I think | 19:00 |
| fungi | the citycloud-sto2 unreachable bit or something i didn't spot? | 19:00 |
| clarkb | fungi: there were a few like that ya | 19:00 |
| corvus | there are about 7 in the old logs | 19:00 |
| clarkb | corvus: what is the tee logfile location? | 19:01 |
| clarkb | I guess ps will tell me | 19:01 |
| fungi | could be we've missed cleaning up the inventory for some recent-ish server deletions | 19:01 |
| clarkb | /root/opendev-run-all-log | 19:01 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: OpenDev changes https://review.opendev.org/653976 | 19:01 |
| corvus | the current errors seem to match the old log | 19:02 |
| clarkb | corvus: including the complaints about no useable tmp? | 19:02 |
| corvus | yes | 19:02 |
| clarkb | k I'm good with proceeding then | 19:02 |
| corvus | that's on paste and pbx | 19:02 |
| clarkb | next up is base.yaml | 19:02 |
| corvus | sounds important | 19:02 |
| clarkb | it does have broad impact | 19:02 |
| corvus | oh, this may take a while since we cleared the cache | 19:03 |
| clarkb | it should mostly be a noop as it creates users, configures exim and the firewall etc | 19:03 |
| corvus | i want to say it takes a few seconds for each host to gather facts | 19:03 |
| clarkb | ah | 19:03 |
| fungi | but it still has to check all those things | 19:03 |
| fungi | yep, that | 19:03 |
| mnaser | silly question, where does codesearch get its source from? | 19:05 |
| clarkb | mnaser: "hound" its from pinterest iirc | 19:05 |
| clarkb | it is on github | 19:05 |
| mnaser | okay let me rephrase, what does it index, git.openstack.org ? | 19:05 |
| clarkb | oh | 19:05 |
| mnaser | i.e. do we have to maybe fix it up to point at Gitea or I dunno how that whole thing works | 19:06 |
| corvus | it needs its cofig updated, which will happen shortly if we're successful | 19:06 |
| mnaser | okay sounds good :) | 19:06 |
| clarkb | for the remote_puppet_git we should expect manage projects to run there with ajaegers sorted list if that merged | 19:07 |
| clarkb | I don't expect that to be an issue but that may also cause it to be a slow run | 19:07 |
| mordred | bad internet - don't expect me to be very responsive for the next hour or so | 19:10 |
| clarkb | mordred: that is better than getting pulled over like corvus theorized | 19:11 |
| *** slaweq has joined #openstack-infra | 19:11 | |
| fungi | mnaser: also in the not-too-distant future i think we want to deprecate and/or take down codesearch.o.o in favor of gitea's source code search functionality, but it's not a 1:1 match (expression language differs, and there are some shortcomings around number of returned results folks may be interested in patching) | 19:11 |
| corvus | our ansible processes are stopped with sigtou | 19:14 |
| corvus | sigttou | 19:14 |
| corvus | is ansible unfriendly with tee? | 19:15 |
| clarkb | I was running the stuff yesterday with tee, that is odd though | 19:15 |
| clarkb | would the use of read give ansible a stdin to block on in some cases? | 19:16 |
| clarkb | er tou is out not in | 19:17 |
| corvus | 12050 is suspended? | 19:18 |
| corvus | maybe someone accidentally did that, and we should just send it a resume signal | 19:19 |
| fungi | stopped on sigttou would make me think it's been disassociated from the controlling terminal somehow | 19:19 |
| clarkb | I didn't but maybe mordreds flaky internet somehow did? | 19:19 |
| fungi | or paused, maybe, yes | 19:19 |
| fungi | like if someone sent ctrl+z or (maybe in some cases) ctrl+s? | 19:20 |
| corvus | the timeout process is suspended, everything else is tou | 19:20 |
| fungi | and timeout's the main parent | 19:20 |
| *** armax has joined #openstack-infra | 19:20 | |
| fungi | well, after the shell i guess | 19:20 |
| corvus | kill -18 12050 ? | 19:21 |
| clarkb | worth a try I guess | 19:21 |
| corvus | no joy | 19:21 |
| clarkb | also in double checking the signal number different arches use different signal numbers amazing | 19:22 |
| corvus | should we ctrl-c and try again? | 19:22 |
| clarkb | corvus: ya I think so | 19:22 |
| corvus | look good? | 19:23 |
| clarkb | yes | 19:23 |
| fungi | yep | 19:24 |
| fungi | and maybe make sure the sigint propagated to all the child processes | 19:24 |
| clarkb | I had to double check that those nodepool magnum nodes were really in inventory (they are) | 19:24 |
| clarkb | fungi: there were no remaining ansible processes (I checked) | 19:24 |
| fungi | perfect | 19:25 |
| clarkb | getting lots of ok's which is what we want | 19:27 |
| clarkb | (signals a noop) | 19:27 |
| *** Lucas_Gray has quit IRC | 19:28 | |
| *** tosky has quit IRC | 19:29 | |
| clarkb | on git01 I've started a tail -f /var/log/messages | grep puppet-user and watch git log -1 --oneline in different windows so that we can hopefully see the project-config sha1 update there in near real time | 19:33 |
| clarkb | (this will happen in the next playbook) | 19:33 |
| johnsom | Just an FYI, the commit gitweb links from gerrit are not working (404). An example: https://opendev.org/openstack/octavia/commit/95b3edbc6a405991656cff6ff5b6ac5e41f7b32e (this is by far not critical functionality) | 19:33 |
| corvus | clarkb: git01? | 19:33 |
| corvus | ok yeah, i guess they're still in inventory | 19:34 |
| clarkb | corvus: yes we will still puppet them so its a good canary | 19:34 |
| clarkb | johnsom: ya mordred has some changes up to fix that I think | 19:34 |
| clarkb | johnsom: so it is on our list (and thanks for double checking on it) | 19:34 |
| johnsom | Cool! Overall it looks good! | 19:35 |
| clarkb | grafana01 failed to update apt cache. This may be related to ianws work to update grafana there | 19:35 |
| clarkb | (they moved repos) | 19:35 |
| clarkb | we appear to sigsus again? | 19:36 |
| clarkb | I don't get it | 19:36 |
| fungi | odd | 19:36 |
| corvus | i don't see any "T" this time | 19:37 |
| corvus | both of the child ansible-playbook processes are busy-waiting | 19:37 |
| clarkb | ah ok | 19:37 |
| clarkb | I'm just paranoid then | 19:37 |
| corvus | on what i don't know | 19:38 |
| clarkb | TASK [puppet-install : Install puppet packages] was the last task recorded by the log | 19:39 |
| clarkb | which should be a noop | 19:39 |
| clarkb | lists.katacontainers.io is the host that it is sh'd to | 19:39 |
| corvus | neat it moved | 19:41 |
| clarkb | syslog showed it finished around 19:35 | 19:41 |
| clarkb | on katacontainers.io | 19:41 |
| clarkb | maybe a network blip getting that back to ansible | 19:42 |
| clarkb | corvus: there was a cahnged item on adns1 | 19:43 |
| clarkb | hanlder to reload named. I would expect we only do that if zones change? | 19:43 |
| clarkb | I can still resolve opendev.org against ns1 so maybe nothing to be concerned about | 19:44 |
| clarkb | corvus: ^ maybe you want to sanity check the ns servers before we continue? but otherwise that looked really good to me | 19:44 |
| clarkb | I need a quick break | 19:44 |
| clarkb | back in a minute | 19:44 |
| corvus | 2019-04-20 19:42:32,297 p=12940 u=root | changed: [adns1.opendev.org] => (item={'name': 'zone-opendev.org', 'url': 'https://git.openstack.org/openstack-infra/zone-opendev.org'}) | 19:45 |
| corvus | i think that's pulling down the opendev migration commits in the zone repos | 19:45 |
| corvus | so it did change, but the data didn't | 19:45 |
| corvus | should be fine i expect | 19:45 |
| clarkb | back and gotcha | 19:46 |
| clarkb | and th change is the redirect? | 19:47 |
| clarkb | beacuse that is a reop that didn't get renamed (accidentally)? | 19:47 |
| clarkb | cee7b8ce87135dd5a6ac54bedab0333df063b4cd is what I expect project config to update to on the next bit | 19:47 |
| fungi | also i think one of mordred's proposed changes solves that one | 19:48 |
| fungi | or rather updates it to opendev | 19:48 |
| clarkb | fungi: hopefully it doesn't change the path too :) | 19:48 |
| clarkb | oh the next bit is actually more k8s gitea stuff | 19:48 |
| clarkb | but after that is remote_puppet_git | 19:48 |
| clarkb | I think I'm happy with how base.yaml went | 19:49 |
| fungi | yeah, log looks reasonable to me | 19:50 |
| clarkb | I vote to proceed and do gitea k8s things | 19:50 |
| *** armax has quit IRC | 19:53 | |
| clarkb | did we lose corvus | 19:56 |
| fungi | it was a fine time for a break since there was a lot to look over after that phase completed | 19:57 |
| corvus | who? | 19:57 |
| corvus | yeah, i stuffed my face | 19:57 |
| clarkb | corvus: you :P just checking given fungi and mordred's internet woes recently | 19:57 |
| corvus | i will hit enter now | 19:57 |
| clarkb | k | 19:58 |
| clarkb | that one seemed happy. There are two more k8s gitea ones | 19:58 |
| corvus | i'll plow through the next 2 | 19:58 |
| corvus | ready for remote git? | 19:59 |
| clarkb | I've still got my git01 canary up | 19:59 |
| fungi | yup | 19:59 |
| clarkb | git01 updated to cee... | 20:00 |
| clarkb | so that looks good | 20:00 |
| clarkb | create cgit repos failed | 20:01 |
| fungi | missing parent dirs for new namespaces? | 20:01 |
| corvus | we moved the repos on cgit so that should be ok | 20:01 |
| clarkb | ya I'm not too worried about that failing. It does mean that nothing else in that playbook will run though | 20:02 |
| clarkb | meaning no review01 | 20:02 |
| clarkb | (or gitea) | 20:02 |
| corvus | right, so we should fix this i think | 20:02 |
| fungi | oh, right, the rename_repos playbook should have taken care of adding new namespaces | 20:02 |
| fungi | hrm... | 20:02 |
| clarkb | Apr 20 20:00:45 git01 puppet-user[10266]: (/Stage[main]/Openstack_project::Git_backend/Exec[create_cgitrepos]/returns) os.symlink(alias_repo_path, alias_link_path) is what failed | 20:02 |
| corvus | ooooh | 20:02 |
| corvus | it's probably something about the cgit aliases for eg git.zuul-ci.org | 20:03 |
| mnaser | previously, we used https://opendev.org/openstack/openstack-ansible-tests/src/branch/master/gen-projects-list.sh#L73-L75 to get a list of openstack-ansible projects when we use the proposal bot.. that no longer works, do we have a suggested alternative | 20:03 |
| fungi | symlink errors | 20:03 |
| clarkb | corvus: and airship and starlingx ya | 20:03 |
| fungi | yeah, just found those in the log | 20:03 |
| corvus | mnaser: you can do an equivalent with either gerrit or gitea | 20:04 |
| fungi | mnaser: as in official deliverable repos of the openstack ansible team? or something different? | 20:04 |
| mnaser | fungi: pretty much I guess, just getting list of all repos that are openstack/openstack-ansible-* | 20:04 |
| clarkb | corvus: fungi so one idea would be to remove git0* from our ansible + puppetry | 20:04 |
| clarkb | we can do that temporarily via the emergency file | 20:04 |
| mnaser | if there's somewhere you can point me at, I can go there and get testing | 20:05 |
| corvus | so i think we can either remove the git fram or clean up the projects list to remove the alias stuff | 20:05 |
| corvus | really, i guess we should do both :) | 20:05 |
| fungi | mnaser: one option would be to parse the governance reference/projects.yaml, another would be to hit the gerrit projects api | 20:05 |
| corvus | i'll start working on the projects.yaml cleanup | 20:06 |
| clarkb | I can look at our inventory | 20:06 |
| mnaser | hmm, is the gerrits project api usable anonymously? | 20:06 |
| fungi | mnaser: gerrit projects list can be accessed anonymously from https://review.opendev.org/projects/ | 20:06 |
| clarkb | mnaser: a lot of it is | 20:06 |
| mnaser | ok lovely, ill try my way that way | 20:06 |
| fungi | mnaser: it's json, but needs the first ~5 bytes stripped to be parseable | 20:07 |
| mnaser | ah json, the current script is bash.. but I think I might have no choice but to switch to python | 20:07 |
| mnaser | or add jq, idk, it'll be some form of work :) | 20:07 |
| fungi | mnaser: or use a command-line parser, right | 20:07 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Remove cgit aliases https://review.opendev.org/654052 | 20:08 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Stop ansipuppeting the old cgit farm https://review.opendev.org/654053 | 20:12 |
| clarkb | corvus: I'm not sure if ^ is complete but is a start | 20:12 |
| clarkb | mordred: ^ I know I said that shouldn't matter but it apparently does | 20:12 |
| *** dciabrin__ has joined #openstack-infra | 20:13 | |
| corvus | clarkb: lgtm | 20:13 |
| clarkb | corvus: I've approved the project-config change | 20:13 |
| mnaser | clarkb, fungi: I've found a workaround, given this is a proposal job, I can use the proposal user and I came up with this: ssh -p 29418 review.opendev.org gerrit ls-projects --prefix openstack/openstack-ansible | 20:14 |
| clarkb | mnaser: oh ya you have ssh creds and that is nicely parseable | 20:14 |
| fungi | sure, that's an option too | 20:14 |
| fungi | ls-projects also supports a regex option if you prefer | 20:14 |
| corvus | clarkb, fungi: for our next run, after those changes merge, should we run the update-system-config playbook then remote git? | 20:15 |
| clarkb | corvus: ++ | 20:15 |
| fungi | i think so. almost done reviewing the second now | 20:15 |
| *** kaspars__ has quit IRC | 20:15 | |
| fungi | okay, both lgtm | 20:16 |
| fungi | also i guess this means we can drop centos from our battery of infra puppet tests? | 20:16 |
| clarkb | fungi: ++ that would be a thing I missed :) | 20:16 |
| *** dciabrin_ has quit IRC | 20:17 | |
| fungi | those were our last centos servers since pbx moved to ubuntu, right? | 20:17 |
| clarkb | should be | 20:17 |
| * mordred is back | 20:25 | |
| mordred | clarkb: did I miss anything fun? | 20:25 |
| clarkb | mordred: we discovered that jeepyb fails to configure cgit beacuse what were once symlinks are now real dirs | 20:25 |
| mordred | oh fun! | 20:26 |
| clarkb | mordred: so we are removing the cgit alias config from project-config and I pushed a chagne to stop ansipuppeting cgit farm | 20:26 |
| mordred | ++ | 20:26 |
| clarkb | so I was wrong about that not being important :/ | 20:26 |
| mordred | well - such is life | 20:26 |
| clarkb | anyways both are approved and we are waiting for them to merge before resuming the run_all.sh walkthough | 20:26 |
| clarkb | everything up to the remote_puppet_git playbook looked great though | 20:26 |
| mordred | it's exciting to clean up house a bit | 20:26 |
| mordred | \o/ | 20:26 |
| corvus | delete all things | 20:26 |
| clarkb | I think we just discovered the fastest way to automate ourselves out of jobs | 20:27 |
| clarkb | just delete everything | 20:27 |
| clarkb | wcpgw | 20:27 |
| mordred | clarkb: let's do it | 20:27 |
| openstackgerrit | Merged openstack/project-config master: Remove cgit aliases https://review.opendev.org/654052 | 20:27 |
| clarkb | corvus: ^ do we want to wait for system-config too or proceed as is? | 20:27 |
| clarkb | I'm happy either way | 20:28 |
| fungi | may as well give it a few | 20:28 |
| fungi | probably make up for lost time in the ansible run anyway | 20:28 |
| corvus | i think it'll take another 25m for system config, even if i direct-enqueue it now | 20:28 |
| fungi | oh, maybe not then | 20:29 |
| corvus | we could take a small shortcut, wait 9 mins for tests to pass then force-merge it | 20:29 |
| corvus | hrm | 20:29 |
| corvus | we have other changes in gate though, so that's a bad idea | 20:29 |
| corvus | so yeah, it's either proceed or wait 30m | 20:29 |
| fungi | i think we can just keep moving in that case | 20:29 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Rename cgit_file to git_file https://review.opendev.org/654055 | 20:30 |
| corvus | k. there's a chance we may still hit cgit errors, but hopefully not. | 20:30 |
| fungi | since the outstanding one is really more cleanup | 20:30 |
| mordred | I haven't gotten enough stackalytics points yet today | 20:30 |
| corvus | if we do, we'll just end up waiting 25m | 20:30 |
| clarkb | corvus: ya that is what I figured | 20:30 |
| corvus | okay, run now? | 20:31 |
| clarkb | I've got my watches going on git01 again | 20:31 |
| clarkb | I'm ready if you are | 20:31 |
| fungi | yeah, ready | 20:31 |
| fungi | if it breaks on more cgit stuff, well the fix will be that much closer to merging by then anyway | 20:32 |
| corvus | clarkb: apply failed on centos4 | 20:32 |
| corvus | clarkb: maybe should remove the job with that change? | 20:32 |
| clarkb | I can update then maybe we direct enqueue? | 20:33 |
| clarkb | I'll do that after watching this run? | 20:33 |
| fungi | wfm | 20:33 |
| corvus | clarkb: http://logs.openstack.org/53/654053/1/check/infra-puppet-apply-4-centos-7/18889b5/job-output.txt.gz#_2019-04-20_20_30_51_035615 | 20:33 |
| corvus | yep that's the fix | 20:33 |
| corvus | clarkb: want me to continue, or take this pause to update and enqueue? | 20:33 |
| clarkb | why don't you continue and we can update my patch if necessary | 20:33 |
| clarkb | f4f2d56 is what I got on git01 and it seemed to have succeeded there | 20:34 |
| clarkb | so that bit lgtm | 20:35 |
| clarkb | we'll rerun manage-projects on review01 when it gets there due to the projects.yaml updates | 20:35 |
| clarkb | but it should be fine as long as it too gets f4f2d56 | 20:35 |
| clarkb | are we happy with that gitea output? | 20:35 |
| fungi | i think so | 20:36 |
| corvus | i think it's skipping all the actionable bits | 20:36 |
| openstackgerrit | Merged opendev/system-config master: Remove running your own doc https://review.opendev.org/654027 | 20:37 |
| openstackgerrit | Merged opendev/system-config master: Finish updating docs for opendev https://review.opendev.org/654028 | 20:39 |
| mordred | mmm. watching text roll by terminal windows | 20:42 |
| mordred | corvus: are all of those skippings skipping already existing projects in gitea? | 20:43 |
| corvus | mordred: yep | 20:43 |
| mordred | awesome | 20:43 |
| clarkb | seems like some giteas are faster than others | 20:43 |
| corvus | maybe we should turn it into a python script | 20:43 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Stop ansipuppeting the old cgit farm https://review.opendev.org/654053 | 20:43 |
| clarkb | lets see if ^ works better now | 20:43 |
| corvus | should i direct-enqueue that? | 20:45 |
| * mordred has +2d | 20:45 | |
| clarkb | I'm ok with direct enqueue if others are. If I got something wrong it should fail in the gate | 20:46 |
| mordred | ++ | 20:46 |
| corvus | enqueued it is | 20:46 |
| corvus | once we move that repo over to opendev, we won't have clean-check anyway | 20:47 |
| mordred | \o/ yay no clean check | 20:47 |
| clarkb | ok gitea done now on to review01 | 20:48 |
| clarkb | Apr 20 20:48:40 review01 puppet-user[10213]: (/Stage[main]/Project_config/Vcsrepo[/etc/project-config]/ensure) f4f2d560f57bca8103c630a746883731031984ae that looks good to me | 20:48 |
| *** openstackgerrit has quit IRC | 20:49 | |
| clarkb | manage projects is running if you want to follow its log file | 20:49 |
| clarkb | I think it is fetching the github info because everything else is in the cache | 20:50 |
| clarkb | I have a fix for caching the github stuff too but never had time to test it | 20:50 |
| clarkb | that is to say it appears to be nooping | 20:50 |
| corvus | i don't see any crazy new projects in the gitea explorer | 20:50 |
| clarkb | its done | 20:51 |
| mordred | corvus: I dunno - opendev/system-config is pretty crazy | 20:51 |
| corvus | openstack/openstack is crazy crazy | 20:51 |
| mordred | yeah it is | 20:51 |
| clarkb | Apr 20 20:49:08 review01 puppet-user[10213]: (/Stage[main]/Gerrit/File[/home/gerrit2/review_site/etc/gerrit.config]/content) content changed '{md5}4ed3385a8a9073580edd90df4ea40923' to '{md5}17255eb12dcf2af038db617dd991c422' is the only thing that looks weird to me | 20:52 |
| clarkb | I'm gonna see if I can dig out the old file in the puppet bucket to see what changed | 20:52 |
| clarkb | maybe it was corvus fix for githttpurl? | 20:52 |
| corvus | clarkb: gerrit does have that weird habit of editing its configs to mess with tabs | 20:52 |
| clarkb | I should know in a minute | 20:52 |
| fungi | clarkb: mordred: https://review.opendev.org/654057 may be relevant to the openstack/openstack superrepo update | 20:52 |
| corvus | the content of gerrit.config lgtm | 20:53 |
| clarkb | `diff /var/lib/puppet/clientbucket/4/e/d/3/3/8/5/a/4ed3385a8a9073580edd90df4ea40923/contents /home/gerrit2/review_site/etc/gerrit.config` its just whitespace and quoting | 20:54 |
| clarkb | we might want to figureout how gerrit wants that so it doesn't undo our changes but I agree it lgtm | 20:54 |
| fungi | i think it's mostly to do with handling of tabs | 20:55 |
| fungi | ahh, right, quoting differences too | 20:55 |
| clarkb | it appears to be tabs in both sides, but ya the differences are in serialization format not content | 20:55 |
| clarkb | so I think we are good | 20:55 |
| corvus | fungi: you set that to formal-vote; could it pass through under a housekeeping rule? | 20:56 |
| fungi | quite possibly. mnaser ^ what do you think about expedited tc housekeeping rules for 654057? | 20:57 |
| clarkb | I am ready to proceed onto remote_puppet_afs if others are | 20:57 |
| mordred | fungi: I gave it a +1 anyway | 20:57 |
| * mnaser reads buffer | 20:57 | |
| mordred | the patch in questio didn't apply any shifts in or out of governance, which I think is the right first patch | 20:57 |
| fungi | mnaser: in short, the openstack/openstack superrepo update script consumes the openstack governance projects list | 20:57 |
| fungi | so this brings governance in-line with reality from a repository name perspective | 20:58 |
| mordred | it's also clear where there are some things that should be proposed for removal - what with the x/ prefixes and all :) | 20:58 |
| * mnaser parsing | 20:58 | |
| corvus | fungi, mnaser: i think there are 2 house rules that could be useful. either "typo fixes" or "other project team updates" with the "time-sensitive" exception (that one requires a report in the TC update) | 20:58 |
| fungi | i used a stripped-down version of the script i wrote for all the offline repo updates and applied the same repository mapping we used in the maintenance to generate that change, fwiw | 20:58 |
| corvus | https://governance.openstack.org/tc/reference/house-rules.html#typo-fixes or https://governance.openstack.org/tc/reference/house-rules.html#other-project-team-updates | 20:59 |
| corvus | maybe https://governance.openstack.org/tc/reference/house-rules.html#code-changes too | 21:00 |
| fungi | this also, less urgently, fixes the data from the perspective of our election tooling | 21:00 |
| corvus | that's also a stretch | 21:00 |
| corvus | clarkb, fungi, mordred: proceed? | 21:00 |
| *** openstackgerrit has joined #openstack-infra | 21:00 | |
| openstackgerrit | Merged opendev/system-config master: Update opendev git references in puppet modules https://review.opendev.org/654029 | 21:00 |
| clarkb | corvus: yes please | 21:00 |
| mnaser | ok, so I think this falls under project-update | 21:00 |
| fungi | corvus: yeah, i'm cool with moving on to the next playbook | 21:00 |
| clarkb | the two tests that failed on my system-config change in check were due to doing docker things in limestone | 21:01 |
| clarkb | the puppetry for afs is fairly minimal I don't expect much on this playbook | 21:01 |
| mnaser | can I have a very brief reasoning as-to why we need to do this *now* (sorry, just catching up) | 21:01 |
| clarkb | mnaser: I think beacuse that is an input to the script that updates openstack/openstack's git submodules | 21:01 |
| clarkb | mnaser: currently openstack/openstack is unhappy about submodules pointing at invalid locations | 21:02 |
| fungi | which cncf's "cloud-native landscape" is also relying on to report openstack activity volume... i guess some people care about that | 21:02 |
| clarkb | puppet log on afs01 is just deprecation warnings | 21:03 |
| corvus | i'm happy. proceed? | 21:03 |
| fungi | mnaser: i don't know that it's on-fire-fix-it-now urgent, but would be nice to have it in fairly soon | 21:03 |
| fungi | rather than waiting the usual formal voting period | 21:03 |
| clarkb | yup just finished checking afsdb01 too and same thing as afs01.dfw | 21:03 |
| clarkb | corvus: I think we can proceed. Next playbook is the everything else puppet run | 21:03 |
| fungi | corvus: yep, lgtm | 21:03 |
| corvus | that error was paste and pbx again | 21:04 |
| corvus | i expect we'll see that same set of 7 hosts error | 21:04 |
| mnaser | the way I see it, it helps you unbreak things, it has no actual governing change, and it reflects a correct reality of things (who owns what repo) rather than the inaccurate state it is in now. the biggest concern is if someone is unhappy of where something was renamed to.. but then we can always change that specifically (because it's already too late to say "wow I don't want this thing to live under x" | 21:04 |
| mnaser | because it already does. | 21:04 |
| clarkb | the reason for pbx errors is a full disk | 21:05 |
| clarkb | explains the no suitable /tmp | 21:05 |
| fungi | oh, yep, rootfs full on paste.o.o too | 21:05 |
| fungi | i'll take a quick look at it | 21:06 |
| clarkb | /var/log/asterisk is pbx's problem | 21:06 |
| clarkb | there is a very large messages.1 | 21:06 |
| fungi | yeesh | 21:06 |
| corvus | mnaser: agreed; i think linking to the change in the update email is the best way to deal with that | 21:06 |
| clarkb | I don't think I'll fix that right this moment | 21:06 |
| corvus | probably all sip auth errors | 21:07 |
| fungi | /var/backups/mysql_backups on paste.o.o is consuming >15gb of the 20gb rootfs | 21:07 |
| clarkb | fungi: wow | 21:07 |
| clarkb | we can probably reduce retention there if we need to since we should have off host backups 9lets check that too) | 21:07 |
| clarkb | mnaser: corvus I also mentioend in my email last night that there will be time to make changes in the fturue I just want to think about stabilizing where we are at now and the summit and ptg and other things for a bit :) | 21:08 |
| fungi | looks like it stopped having enough room to rotate the db backups after april 3 | 21:08 |
| fungi | i've deleted all but the most recent 10 compressed backups | 21:09 |
| clarkb | and maybe have a day off between now and may 5 | 21:09 |
| *** openstackstatus has quit IRC | 21:09 | |
| fungi | looks like a compressed backup of the paste db is running around 0.5gb now | 21:09 |
| clarkb | fungi: one thing I was thinking about was converting to .xz | 21:10 |
| mnaser | clarkb, fungi, corvus: I've +W'd the proposal and sent a notification to the ML, will ping channel as well | 21:10 |
| mnaser | (it's in the gate now) | 21:10 |
| fungi | thanks mnaser! i didn't mean to suggest it was that urgent, but much appreciated! | 21:10 |
| *** openstackstatus has joined #openstack-infra | 21:10 | |
| *** ChanServ sets mode: +v openstackstatus | 21:10 | |
| mnaser | it's fine, no worries | 21:10 |
| clarkb | hrm we have a grafana01 and grafana02. Will have to ask ianw about removing the old server | 21:11 |
| clarkb | infra-root the summary is looking mostly ok to me | 21:11 |
| clarkb | status failed becuase we couldn't update our vcsrepo for reviewday | 21:12 |
| corvus | clarkb: graeed | 21:12 |
| mordred | corvus: that should be a word | 21:12 |
| clarkb | it uses git:// is why | 21:12 |
| fungi | that'd certainly do it | 21:13 |
| corvus | mordred: braeeens! | 21:13 |
| mordred | we missed a git:// ? | 21:13 |
| fungi | or we missed merging ianw's fix for one | 21:13 |
| clarkb | ya I think we likely just failed to merge the change | 21:13 |
| clarkb | health01 failed beacuse the irc library requires python3.4 or greater now and we are trying to isntall it under python2 | 21:14 |
| corvus | i feel like the superrepo is a "good to get done before monday" kinda thing, so merging that will help :) | 21:14 |
| fungi | yep | 21:14 |
| clarkb | pbx and paste we already know why | 21:14 |
| clarkb | storyboard-dev failed on Apr 20 21:07:42 storyboard-dev01 puppet-user[26995]: (/Stage[main]/Storyboard::Load_projects/Exec[load-projects-yaml]/returns) sqlalchemy.orm.exc.StaleDataError: DELETE statement on table 'project_group_mapping' expected to delete 44 row(s); Only 45 were matched. | 21:15 |
| clarkb | fungi: ^ did we need to also update storyboard-dev's db when we did the great renaming? | 21:15 |
| clarkb | and then wiki02 I'm not even gonna look at >_> | 21:15 |
| clarkb | overall that looks pretty good to me | 21:15 |
| fungi | clarkb: nah, storyboard-dev isn't critical to have match | 21:15 |
| clarkb | fungi: well I wonder if it not matching is why ^ that failed | 21:16 |
| fungi | oh | 21:16 |
| fungi | ummm | 21:16 |
| fungi | i wonder why we're doing anything with the storyboard-dev db | 21:16 |
| clarkb | that was from a db migration | 21:16 |
| clarkb | I think because we "udpated" storboard, likely just the opendev migration change, then it runs migrations after and failed | 21:17 |
| clarkb | storyboard-db-manage is the command | 21:17 |
| clarkb | infra-root I think we should be safe to enable run_all.sh | 21:18 |
| clarkb | and we'll have to fix those issues above out of band | 21:18 |
| mordred | clarkb: ++ | 21:18 |
| corvus | #status log re-enabled run_all.sh cron | 21:18 |
| openstackstatus | corvus: finished logging | 21:19 |
| fungi | yeah, we may be better off blowing away the storyboard-dev database and starting fresh. it's overdue for a cleansing | 21:19 |
| corvus | i believe the system is in normal condition now? | 21:20 |
| mordred | corvus: i believe it is | 21:20 |
| clarkb | yes | 21:20 |
| clarkb | do we still want a hold on new project creations so that we can all be comfortable with whatever new rules we end up with? | 21:20 |
| clarkb | I can followup to my email to the infra list sasying things should eb good otherwise | 21:20 |
| clarkb | or do we not want to worry too much about new projects either? | 21:21 |
| fungi | looks like maybe we use a puppet exec to create project-groups in storyboard corresponding to groups entries in projects.yaml, and do the same on storyboard-dev | 21:21 |
| corvus | we should probably flush out our cleanup list before new projects | 21:21 |
| clarkb | corvus: I like that plan | 21:21 |
| corvus | (i think we found some places where new orgs may be a problem) | 21:21 |
| clarkb | oh right | 21:22 |
| fungi | yeah, i'm not keen on opening the floodgates for new repos or renames just yet | 21:22 |
| corvus | shouldn't be much though, can probably get it going first of the week | 21:22 |
| clarkb | ok I'll send an update saying things are generally back to normal but please hold on new projects | 21:22 |
| clarkb | and then I've noticed the sun just came out and after yesterday's downpour I'm gonna go outside and enjoy it for a bit | 21:22 |
| fungi | did we get mnaser's .htaccess addition merged to redirect h= shas? | 21:23 |
| clarkb | fungi: ya I think we applied that | 21:23 |
| clarkb | fungi: you should be able to check now that we ran puppet :) | 21:23 |
| fungi | right, was gonna try in a sec | 21:24 |
| clarkb | alrihgt email sent and now time for some sun | 21:25 |
| clarkb | before I go are we good to shutdown the screen on review01? | 21:26 |
| clarkb | should I exit out of it instead of dettaching? | 21:26 |
| clarkb | also tomorrow I've got family functions. Easter at grandmothers so I won't be around a good chunk of the day like today | 21:26 |
| corvus | i think we can close all the screens | 21:27 |
| fungi | i agree | 21:27 |
| clarkb | ok I detacched from the review01 one if someone else wants to close it | 21:27 |
| clarkb | I'll detach from bridge too | 21:27 |
| fungi | hrm, https://git.openstack.org/cgit/openstack-infra/system-config/tree/?h=8dd82494cd66825b005ba82e3e3dbd47326f3edc redirects to https://opendev.org/opendev/system-config/src/branch/8dd82494cd66825b005ba82e3e3dbd47326f3edc/ (which is a 404) instead of https://opendev.org/opendev/system-config/src/commit/8dd82494cd66825b005ba82e3e3dbd47326f3edc/ | 21:28 |
| fungi | on files01 /var/www/git-redirect/.htaccess has a last modified timestamp from ~22 minutes ago | 21:29 |
| fungi | and seems to include the desired rule addition | 21:31 |
| clarkb | that should be right. corvus hand edited the file hours ago then puppet wouldve updated to remove my comment from testing redirects yestderday | 21:32 |
| fungi | oh, it only has matches for urls including /plain/ not /tree/ | 21:32 |
| fungi | aha, i think the change has a copy-paste error so the existing rules for /tree/ got duplicated rather than adding a different one | 21:34 |
| fungi | also i should probably start looking into how to get git notes replicating. i'm missing them already | 21:36 |
| mordred | fungi: ++ | 21:36 |
| mordred | fungi: I'm guessing it's going to be a patch to gitea - when we tried replicating all refs originally, gitea-side there were errors - but at the time we didn't dig in to them very much | 21:36 |
| mordred | fungi: but I thnik it would be great if we can go back to repliating refs/changes and refs/notes | 21:37 |
| *** rf0lc0 has quit IRC | 21:38 | |
| fungi | hrm, yeah, alternatively maybe i can figure out how to convince git to fetch notes from review.opendev.org instead of opendev.org | 21:38 |
| clarkb | do you use the notes to look at review data? | 21:40 |
| clarkb | (I almost never look atnotes so am curious) | 21:40 |
| fungi | i do, yes | 21:40 |
| fungi | particularly often i use it to grab review urls | 21:40 |
| fungi | i'm looking at a commit and right there inline with the commit message it tells me the url to the review and when it actually merged (not just the authored/committed date) which is often useful for me as well | 21:41 |
| fungi | also tells me who +2'd it and who approved it | 21:42 |
| fungi | so faster to follow up with reviewers/approvers without a round-trip through gerrit's webui or gertty to find that information | 21:42 |
| clarkb | I usually search the changeid in gerrit | 21:43 |
| clarkb | and ya not fast | 21:43 |
| fungi | also those are non-unique | 21:43 |
| fungi | so sometimes you get multiple results and then need to figure out which project and branch you're looking for | 21:43 |
| corvus | fungi: are you fixing the rules or should i? | 21:44 |
| fungi | i have a feeling gitea may also display git notes along with the commit details if it has them, like github does | 21:44 |
| fungi | corvus: i just got finished confirming there's not a paste error in the earlier commit, it's simply missing additional rules so i'm adding those now | 21:45 |
| fungi | should have a change up in moments | 21:45 |
| corvus | kk | 21:45 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add some more legacy cgit h= redirects https://review.opendev.org/654065 | 21:57 |
| fungi | corvus: mnaser: ^ | 21:57 |
| corvus | fungi: lgtm! | 22:01 |
| *** pcaruana has quit IRC | 22:02 | |
| fungi | thanks | 22:02 |
| *** mriedem has joined #openstack-infra | 22:05 | |
| *** mriedem has quit IRC | 22:28 | |
| * paladox is already planning on a gerrit 3.0 upgrade :) (we are planning on getting rid of our db :)) | 22:30 | |
| *** yamamoto has quit IRC | 22:40 | |
| fungi | yeah, we're excited for that as well | 22:43 |
| paladox | Our releng department is going to discuss the 2.16 update next week :) (already tested and have all the changes up that are required) | 22:44 |
| *** yamamoto has joined #openstack-infra | 22:46 | |
| * paladox is very excited for that update, our theme goes very nicely in pg. | 22:46 | |
| *** slaweq has quit IRC | 22:47 | |
| *** yamamoto has quit IRC | 22:50 | |
| *** yamamoto has joined #openstack-infra | 22:54 | |
| *** slaweq has joined #openstack-infra | 22:58 | |
| paladox | fungi upstream were very kind to implement notedb.config (on my request) for users that use puppet for notedb migrations. We have been on NoteDB since June last year. So it will be very exciting when we do our 2.16 update to finally drop the db! (we are planning to do it at the same time as the upgrade). This will allow us to restart our replica which currently cannot connect to the db :) | 23:02 |
| *** slaweq has quit IRC | 23:03 | |
| *** yamamoto has quit IRC | 23:09 | |
| *** yamamoto has joined #openstack-infra | 23:12 | |
| *** tosky has joined #openstack-infra | 23:18 | |
| fungi | yeah, previously when we'd been trying to work out a viable upgrade path we were facing lengthy incremental upgrades to get the migration from mysql to notedb to happen (and i think mordred submitted some fixes upstream for some of the migration routines they had at the time) | 23:20 |
| fungi | i gather it's become easier since and you can skip versions now | 23:20 |
| fungi | we just haven't had time to revisit it yet | 23:20 |
| clarkb | fungi: I dont think the myswl patches went upstream :/ | 23:25 |
| *** yamamoto has quit IRC | 23:26 | |
| *** yamamoto has joined #openstack-infra | 23:31 | |
| paladox | clarkb mysql patches? | 23:33 |
| clarkb | paladox: for the new db, it didnt work with myswl | 23:33 |
| clarkb | I forget its name but people must only h2 it? | 23:34 |
| clarkb | or postgres? | 23:34 |
| *** auristor has quit IRC | 23:34 | |
| paladox | h2 is for cache. | 23:34 |
| paladox | But we are going to move to h2 so we can drop our db (on mysql) | 23:35 |
| paladox | there's only one table left that requires the db in 2.16. | 23:35 |
| clarkb | accountpatchreviewdb | 23:35 |
| paladox | ah | 23:35 |
| paladox | you use mysql with accountpatchreviewdb? | 23:35 |
| clarkb | it doesnt or didnt work with mysql due to index restrictions | 23:35 |
| clarkb | we do | 23:36 |
| paladox | Ah i see. We use h2 for accountpatchreviewdb :) | 23:36 |
| clarkb | so we have a patch from monty that fixes the indexes | 23:36 |
| fungi | oh, right, the available migration path for us at the time was mysql->h2->notedb | 23:37 |
| fungi | but now seems like they've made it possible to skip straight from mysql to notedb for that as well? | 23:37 |
| paladox | NoteDB was very easy to migrate to :) | 23:37 |
| paladox | after upstream kindness | 23:38 |
| clarkb | fungi: I think it is online now too | 23:38 |
| clarkb | which is new since we seriously looked at it and got scared away | 23:38 |
| paladox | clarkb fungi https://github.com/wikimedia/puppet/commit/06c8e4122c37508045d84840ac1cb23f4f7d9011#diff-4c58f684fb8a36946bc7616d35570c00 | 23:38 |
| paladox | we then set https://github.com/wikimedia/puppet/commit/d0b08b9675438fe637374a165fdf28c375c3510a after the migration | 23:39 |
| paladox | Our 2.16 upgrade will be like: https://gerrit.wikimedia.org/r/#/q/topic:gerrit-upgrade-2.16+(status:open+OR+status:merged) | 23:48 |
| paladox | (with switching the db last). | 23:48 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
