| opendevreview | Merged opendev/system-config master: Update opensuse mirror script to more completely clean up https://review.opendev.org/c/opendev/system-config/+/913608 | 08:55 |
|---|---|---|
| *** amoralej_ is now known as amoralej | 11:40 | |
| *** blarnath is now known as d34dh0r53 | 12:49 | |
| fungi | infra-root: does anyone have information on how we utilize our channel prefix control in oftc to gain ops/founder on or register a non-empty unregistered channel with no operators? | 14:40 |
| fungi | not finding it noted in our credential/contacts file nor docs in system-config | 14:40 |
| frickler | fungi: I don't know really, but would simply try asking in #oftc | 14:45 |
| fungi | sure, that's going to be my fallback if we don't have a process or contact documented somewhere | 14:46 |
| fungi | just want to make sure i'm not overlooking something | 14:46 |
| frickler | I'm assuming we never had to do this since we moved over from freenode? | 14:49 |
| fungi | entirely possible, but also possible that we've done it once or twice and i just don't remember the details | 14:49 |
| clarkb | fungi: I think corvus set that up a long time ago? | 15:08 |
| clarkb | but I want to say that several of us are registered with oftc somehow as having prefix ownership/stewardship | 15:08 |
| clarkb | and that if we make requests to the admins they're able to look that up and trust our requests as a result? | 15:08 |
| clarkb | Looking at grafana the opensuse afs volume looks much better now. And navigating the content via a mirror it looks like I expect | 15:09 |
| clarkb | fungi: re rax MFA I realized on the school walk today that you have an account and I (selfishly) wondered if you might be willing to opt into MFA for that account before we do our accounts. Then you can test the api_key there as well as see what the process is like | 15:10 |
| fungi | clarkb: sure, i can do that in a bit and report on the overall experience/process | 15:18 |
| clarkb | thanks! | 15:19 |
| fungi | i have also been avoiding setting up the (now required) mfa for my pypi and github accounts | 15:20 |
| clarkb | github does FIDO which I've got set up (and vastly prefer for personal accounts) | 15:21 |
| fungi | sadly, the sign-up for mfa is a little opaque for anything that isn't a smartphone | 16:05 |
| clarkb | I was worried about that. The smartphone signup implies totp though and as long as they show you the secret you should be able to enroll it | 16:06 |
| fungi | i need to figure out which "phone app" i'm going to impersonate (authy, duo, or google authenticator) | 16:06 |
| clarkb | also this would be good feedback to rax I think | 16:06 |
| clarkb | "if you do totp have an escape hatch for people who know what 'totp' means" | 16:07 |
| fungi | it has steps like "use the camera to scan this barcode in order to pair your app" | 16:08 |
| clarkb | fungi: I don't think the app matters as long as whichver one you choose shares the shared secret and not just a qr code to scan | 16:08 |
| clarkb | I think you might be able to decode the qr code to decode the info too; however, I've luckily not run into any applications that have required me to do that. They've all shared a secret in human readable fashion too | 16:09 |
| fungi | it provides a copyable string below the qr code, thankfully | 16:09 |
| clarkb | there are theoretically a bunch of inputs to totp (hash, time hash is good for, secret (and secret encoding), and name). But in reality everyone does shasomething, 30 seconds, and hex or decimal (I forget which) then the name is arbitrary | 16:10 |
| clarkb | oh and number of digits (but again everyone just does 6) | 16:11 |
| clarkb | looking at ykman its sha1, 30 seconds, 6 digits, the secret, and $NAME | 16:12 |
| clarkb | ykman expects the secret to be base32 encoded. IIRC ubuntu one did hex and so I had to convert. I gave them some feedback on that when we enrolled I wonder if they every changed it | 16:12 |
| fungi | okay, i managed to pair a totp slot on one of my librem keys and authenticate with it. doing my backup key now | 16:14 |
| fungi | also took some time to rebuild the latest version of nitrocli from source | 16:14 |
| clarkb | fungi: did they make you repeat back two codes to confirm it registered on your end properly? | 16:16 |
| clarkb | I think that is the most important bit for our purposes just to be sure we have an incantation locally that will work | 16:16 |
| fungi | in a way, yes. it gives you the secret string and has a form field for an authentication code which was really just the result of me generating an auth number from my key after i stored the secret in that totp slot | 16:23 |
| fungi | then it automatically logged me out and i had to log back in | 16:23 |
| fungi | so that was effectively two codes | 16:23 |
| fungi | though for safety, i should have generated and recorded recovery codes *first* just in case it didn't go as planned | 16:24 |
| fungi | (i've done that now at least) | 16:24 |
| clarkb | but it did require at least one code before accepting it that is good. ++ to recording recovery codes first though | 16:25 |
| fungi | yes, it wasn't enrolled until i gave it a working response from the key | 16:26 |
| clarkb | so for our purposes the process should be something like 1) record recovery codes 2) figure out how to use cli totp tool to emit codes 3) register with generated codes | 16:26 |
| clarkb | I'm thinking that counter intuitively we want to do the DNS management account first, then the control plan account, then the nodepool account | 16:26 |
| clarkb | since the quickest noticeable impact will be in reverse of that order | 16:27 |
| clarkb | fungi: did the api_key with openstacksdk continue to work afterwards? | 16:27 |
| fungi | yeah, it's *possible* that the link to generate recovery codes doesn't appear until you've enabled mfa by enrolling a key, which could explain why i didn't do it first | 16:27 |
| clarkb | but ya I think if the api key continues to work we've done about as much checking as we can for now and should proceed with our accounts | 16:28 |
| fungi | confirmed, i can `openstack server list` with my clouds.yaml configured only for the api key/rackspaceauth lib | 16:29 |
| fungi | so i think it's safe for us to enable mfa for one of our accounts... control plane or nodepool first? | 16:30 |
| fungi | oh, you already suggested a sequence | 16:30 |
| clarkb | fungi: ya the sequence I suggested is basically "if something breaks how soon will someone notice before we have time to fix it" | 16:31 |
| fungi | yeah, my actual next step is to refresh my memory on how to sync our totp client on bridge for a new secret | 16:31 |
| clarkb | I could see going in the other direction since it is easier to turn off nodepool than deal with dns problems should they arise though | 16:31 |
| clarkb | fungi: I think we provide the secret to the command at invocation time and so ist msotly a matter of having the correct invocation and putting the whole thing in the secrets file | 16:32 |
| clarkb | I believe that is what we did for other totp cases | 16:32 |
| fungi | yeah | 16:33 |
| fungi | cool, found it, so there's really no configuration to do | 16:35 |
| clarkb | fungi: not after you figure out the invocation. I think it is theoretically possible we may have to set flags if they aren't using sha1 or 6 digits etc | 16:35 |
| clarkb | but you didn't mention any of that so its probably the straightforward case | 16:36 |
| fungi | should i record more than one secret (a backup key), or just rely on the recovery codes for that purpose? | 16:36 |
| clarkb | I think we can rely on the recovery codes for that purpose. I only bother with backups if they are stored separately or a different method. That way if I lose a physical device I can go to a different physical device or if the method stops working for some reason we have a way around it | 16:37 |
| clarkb | in this case we would store them in the same location but have different methods | 16:37 |
| fungi | yeah, having two physical keys makes sense so you can carry one and store one. i agree having two in the same place (in this case same file on the same system) is fairly pointless | 16:38 |
| fungi | okay, so with that figured out... do our dns management account first and then... what are we doing before deciding to move forward with the control plane account and then the nodepool account? | 16:40 |
| fungi | just have another one of us confirm they're also able to log into the dashboard or something? | 16:40 |
| clarkb | for the dns account I think we just verify we can log in and log out in the dashboard | 16:40 |
| fungi | cool, that wfm | 16:40 |
| fungi | i need to put the kettle on, and can get started in a few minutes. i'll add some placeholders to our credential file so i minimize the editing i'll be doing to add details for the second and third accounts | 16:41 |
| clarkb | sounds good. I expect to be around and can help test etc | 16:43 |
| clarkb | maybe I should make some tea too | 16:44 |
| fungi | clarkb: one more idea... should we temporarily take rackspace regions out of our log upload destinations list, then test re-adding them in base-test after we switch that account over? | 16:49 |
| clarkb | fungi: ya we can do that if we want to be extra cautious | 16:51 |
| clarkb | I can push that change up momentarily | 16:52 |
| fungi | thanks, yeah looking through our list of rackspace credentials and thinking back to how these accounts are set up, it may want us to enable mfa for that user as well | 16:53 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Remove rax from zuul log upload config https://review.opendev.org/c/opendev/base-jobs/+/913821 | 16:55 |
| clarkb | fungi: hrm that would effectively make using rax for swift log uploads not possible? | 16:56 |
| clarkb | I had thought they were swift specific credentials and was hoping that would be similar to the nova api key stuff | 16:56 |
| clarkb | but maybe not and ya being cautious can only help | 16:56 |
| fungi | i'm not sure, it may mean we have to use an api key for swift uploads after that point | 16:57 |
| fungi | from the notes in the credentials list, it looks like we tied the rw swift user to the same tenant we use for nodepool | 16:58 |
| fungi | so it will be the last of the three we deal with anyway | 16:58 |
| clarkb | fungi: looks like my foundation email address is the contact email addr for that account (I just got notified of the enrollment). While you are in there you may want to add infra-root? | 17:05 |
| clarkb | I guess this isn't urgent though | 17:05 |
| clarkb | maybe deal with that after everything else is done | 17:05 |
| fungi | there's a phone number tied to that technical contact | 17:09 |
| clarkb | fungi: is it mine? | 17:09 |
| fungi | number ends in 846 | 17:09 |
| clarkb | not me | 17:10 |
| fungi | interesting | 17:10 |
| fungi | austin tx area code | 17:11 |
| clarkb | I think jimmy may have set up that account for us when we unshared account creds | 17:11 |
| clarkb | perhaps it is jimmy's number | 17:11 |
| fungi | possible, i think he created that login for us | 17:11 |
| clarkb | reading https://docs-ospc.rackspace.com/cloud-files/v1/general-api-info/role-based-access-control.html I think you may be right about the swift creds. However, I suspect it may only be an issue if we try to login as that account and/or change credentials? | 17:12 |
| clarkb | since that seems to imply the accounts are fairly separate from one another | 17:12 |
| clarkb | basically meaning we may be able to get away with not modifying the swift stuff for a bit | 17:13 |
| clarkb | but ya I guess we proceed as planned and test and see where we end up | 17:13 |
| fungi | clarkb: okay, 2fa details are added to the list for that account, feel free to test at your convenience and then i'll move on to the control plane account | 17:14 |
| clarkb | ok looking now | 17:14 |
| clarkb | fungi: I think you hvae a running gpg agent that will conflict if I try to open the file? | 17:15 |
| fungi | there should not be, i instantiate a temporary daemon when decrypting | 17:16 |
| clarkb | hrm maybe not the timestamp on that process is from march 14 | 17:16 |
| fungi | i use `gpg-agent --daemon gpg ...` to avoid it | 17:16 |
| corvus1 | could be me; i thought we had a script and i used it | 17:16 |
| fungi | so my gpg process always uses the temporary agent i've instantiated and it terminates as soon as the child process dies | 17:17 |
| corvus1 | isn't that what the edit-secrets script does? | 17:17 |
| fungi | but it could explain why i didn't notice another one someone left running | 17:18 |
| clarkb | corvus1: yes the script it meant to clean up. But als o istarted the script and it ran fine so maybe this isn't an issue | 17:18 |
| fungi | yeah, same thing the edit-secrets script does | 17:18 |
| clarkb | I logged in just fine. I'm going to logout and log into the nodepool creds to see if the swift stuff makes any sense to me | 17:19 |
| fungi | clarkb: probably someone ran gpg a different way and left an agent running, but the way edit-secrets invokes its own agent should result in the other one getting ignored anyway, i think | 17:20 |
| corvus1 | well, last says fungi was the only one who logged in on mar 14; looks like my edit-secrets session was on either mar 7 or mar 20 | 17:20 |
| clarkb | fungi: got it | 17:20 |
| * fungi tries to remember what he might have been doing 6 days ago | 17:20 | |
| clarkb | under the nodepool account if I go account -> user management I see the RO and RW accounts for cloud files. And there is a column in that table to indicate if MFA is enabled or not and it isn't for those accounts | 17:23 |
| fungi | aha, that was from me creating the new signing key. the agent is using a completely separate directory/keychain | 17:23 |
| fungi | it has "--homedir /root/signing.gnupg" in the command line | 17:23 |
| clarkb | unfortunately I think that means fungi's suspicions are correct that even though these should be scoped to files only we'll be fored to mfa with them? | 17:23 |
| clarkb | maybe we can setup openstacksdk with the rackspaceauth plugin inside of zuul and then use the api key though | 17:24 |
| clarkb | however, I also think it is safe to proceed with our plan since we won't be modifying those accounts at all so they shouldn't be affected yet | 17:24 |
| clarkb | fungi: aha | 17:24 |
| fungi | i can kill that, but yeah we should probably experiment with some adjustments to the key creation process | 17:24 |
| clarkb | hrm if I click on one of the accounts it says these accounts don't have product access just ticket access and the spot where there would be an api key has an alert saying the page you are looking for has moved or no longer exists | 17:26 |
| clarkb | so maybe we are using some other type of credential for swift? | 17:26 |
| fungi | very odd, but yeah it could be that these aren't affected after all | 17:26 |
| clarkb | I'm beginning to think we should try decrypting those secrets to confirm or not how the uploads are working | 17:29 |
| fungi | looks like they have two object storage products | 17:30 |
| clarkb | the files are uploaded to the nodepool account I can browse the containers and objects there just fine | 17:31 |
| clarkb | maybe we are using these accounts its just not super apparent that the rbac stuff is setup for that in the user browser | 17:32 |
| fungi | https://www.rackspace.com/cloud/object-storage seems to not be cloudfiles | 17:32 |
| clarkb | interesting and ya cloud files == swift | 17:33 |
| fungi | well, pseudoswift anyway | 17:33 |
| clarkb | I think I'm back to believing we must use these accounts for swift and the web gui is just not reflecting the access in an intuitive manner | 17:37 |
| clarkb | and in that case we may have to deal with api keys in the future but for now I hope we can just leave these accounts as is since we don't actively make changes to them | 17:38 |
| fungi | yes, there was probably some extra dance we did through the api to set that up, and they just needed corresponding unprivileged accounts in the dashboard | 17:38 |
| fungi | likely the error in the api key field reflects a lack of rackspace api key integration in cloudfiles | 17:39 |
| clarkb | either that or the other idea I had was maybe you can only see the api key if you log in as that user rather than as the parent user | 17:39 |
| clarkb | (I'm logged in as the parent not the user itself) | 17:39 |
| fungi | but we probably want to take the rackspace regions out of our upload targets before we switch this account to mfa and then not add it back until after the 26th just in case we were supposed to add separate mfa to those logins | 17:40 |
| fungi | oh, perhaps | 17:40 |
| clarkb | I also remember now that image uploads go to swift first then are imported to glance | 17:40 |
| clarkb | whcih means nodepool builders will potentially be impacted by this change if the machinery to do that doesn't work in openstacksdk with rackspaceauth | 17:40 |
| clarkb | however, we've already been using rackspace auth so we should be abel to check that real quick | 17:41 |
| fungi | good point, thought those are using full access accounts not swift-only accounts | 17:41 |
| clarkb | correct | 17:41 |
| fungi | but i agree, try logging in as the limited account and maybe it will display an api key there | 17:41 |
| clarkb | we have day old ubuntu jammy images in all of rax. That is current with the latest ubuntu jammy image build and newer than the clouds.yaml update for config | 17:42 |
| clarkb | I think that means in theory worst case we're converting the auth creds to api keys if we can find them to do log uploads | 17:43 |
| clarkb | fungi: re disabling rax until the 26th I think we can land my change and trigger a base-test run and see if still works after we update the nodepool creds and if so revert the change. Then next week we can disable again and test around the 27th? | 17:44 |
| clarkb | anyway I'm back to thinking this is largely ok if not ideal and we can proceed with our plan. Unless you have any new concerns after all of this discussion | 17:44 |
| corvus1 | clarkb: ++ | 17:46 |
| fungi | yeah, that seems fine to me too, as long as we remember | 17:52 |
| corvus1 | if we forget, and there's a problem, we will find out quickly | 17:53 |
| clarkb | fungi: next step is the control plane account then? | 17:54 |
| clarkb | and then we can run launch node and make sure it still works? | 17:54 |
| fungi | yep! | 17:57 |
| corvus1 | i'm going to restart zuul-web: 1) we should get the new filtering for certain config errors; 2) there's a change in review for substring searching that depends on a web api change that has merged but not deployed, so we can resume work on that. | 18:05 |
| clarkb | ack | 18:05 |
| fungi | okay, i've switched the control plane account over to mfa and recorded the details in the list. also realized i forgot to do recovery codes for the dns account, so those are added now | 18:14 |
| fungi | testing launch-node now | 18:15 |
| corvus1 | #status log restarted zuul-web to pick up some recent changes | 18:19 |
| corvus1 | https://zuul.opendev.org/t/openstack/config-errors?name=Project+Not+Found <-- new error filter | 18:20 |
| fungi | woo! awesome, that's going to be extra helpful | 18:22 |
| fungi | and also the exceptions are being used so we have useful names for a lot more stuff | 18:22 |
| fungi | so, i was able to use launch-node on bridge to create a new server, then openstackclient to list and delete it, no new problems with the api key after enabling mfa on the account. i guess we can do nodepool after 913821 deploys | 18:28 |
| fungi | oh, actually just after it merges is sufficient | 18:29 |
| clarkb | fungi: that seems reasonable. I guess did you also test logging into the dashboard with the totp token? I think you said they make you do this so ya | 18:29 |
| fungi | i did test logging in with it, yes | 18:29 |
| clarkb | then ya that seems like a plan | 18:29 |
| opendevreview | Merged opendev/base-jobs master: Remove rax from zuul log upload config https://review.opendev.org/c/opendev/base-jobs/+/913821 | 18:34 |
| fungi | okay, getting to work on the last account now | 18:34 |
| fungi | it's done and recorded in the list along with recovery codees | 18:39 |
| clarkb | so now we watch nodepool for boot health and image upload success. And then separtely we can run some jobs against base-test to see if that had any impact on log uploads | 18:39 |
| fungi | also i confirmed, their ui doesn't give you the option to generate recovery codes before you enroll at least one device, so that explains why i kept doing it after i guess | 18:40 |
| clarkb | I think I have a chaneg I can recheck somewhere for base-test testing | 18:40 |
| clarkb | https://review.opendev.org/c/zuul/zuul-jobs/+/680178 has been rechecked | 18:40 |
| fungi | confirmed openstackclient is still working with the api key for that account too after switching it, `openstack --os-cloud=openstackjenkins-rax --os-region-name=DFW server list` returns lots of nodepool nodes (many in error state unfortunately) | 18:42 |
| clarkb | ya you'll likely need to check nodepool logs for successful boots instead | 18:42 |
| clarkb | one of those errored nodes is that centos 7 node I found | 18:42 |
| fungi | oh of course, i just wanted to test osc first as a safety | 18:43 |
| clarkb | ++ | 18:43 |
| fungi | last mention of rackspace in the debug log on nl01 was from a leaked node delete attempt almost 14 hours ago | 18:46 |
| clarkb | fungi: 0037117004 is a node that I believe booted after the change and is in use | 18:49 |
| clarkb | you can grep that string in the debug log to see the progression | 18:49 |
| fungi | yeah, looks good | 18:49 |
| clarkb | if you grep rax-iad/rax-dfw/rax-ord etc you'll also see the provider logs for those providers. But ya I agree this look good | 18:50 |
| clarkb | so the other thing to check is that we have new ubuntu jammy images tomorrow | 18:50 |
| clarkb | or whichever image we upload daily (I think jammy is one of them) | 18:50 |
| fungi | right, i figure it's too soon to know whether image uploads are broken | 18:50 |
| fungi | but they've (in theory) been using api keys so far | 18:50 |
| clarkb | yup | 18:51 |
| clarkb | https://zuul.opendev.org/t/zuul/build/8799eaafa98943bd9cdebb58900a1702/logs seems to show our log uplaods are not affected | 18:51 |
| clarkb | note the raw link is to rax cdn and the job started after we merged the removal from prod uploads | 18:51 |
| clarkb | so ya I think we can revert that base-jobs change then plan to reapply it again next week and test again after the 26th deadline has passed | 18:51 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Revert "Remove rax from zuul log upload config" https://review.opendev.org/c/opendev/base-jobs/+/913839 | 18:55 |
| clarkb | and now lunch | 18:56 |
| opendevreview | Merged opendev/base-jobs master: Revert "Remove rax from zuul log upload config" https://review.opendev.org/c/opendev/base-jobs/+/913839 | 19:34 |
| fungi | on my way out to get dinner, but shouldn't be more than an hour if something comes up | 20:42 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!