| airship-irc-bot | <mattmceuen> Hey @sreejith.punnapuzha, re: what we talked about this morning, where the deployment scripts remove the master taint to allow scheduling, and we want to do something smarter | 00:02 |
|---|---|---|
| airship-irc-bot | <mattmceuen> Looking at the functions, it appears that the CAPI components (and HelmOperator) should already have tolerations for the master taint | 00:03 |
| airship-irc-bot | <mattmceuen> So let me know what your experiment uncovers, now I think I understand the problem less than before :slightly_smiling_face: | 00:03 |
| *** rezroo has joined #airshipit | 03:21 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #airshipit | 05:33 | |
| *** awander has quit IRC | 07:33 | |
| *** roman_g has joined #airshipit | 08:03 | |
| *** rezroo has quit IRC | 08:14 | |
| airship-irc-bot | <victor.melin> document pull | 10:59 |
| airship-irc-bot | <craig.anderson> image-builder patchset could use some reviews still. thanks. https://review.opendev.org/#/c/730777/ | 12:58 |
| roman_g | Added ^^^ to requests for review https://etherpad.opendev.org/p/airship-meeting-2020-11-10 | 13:15 |
| airship-irc-bot | <craig.anderson> thx | 13:44 |
| *** awander has joined #airshipit | 13:56 | |
| mattmceuen | Hi all, our team meeting will begin in ~1hr. Agenda: https://etherpad.opendev.org/p/airship-team-meeting | 14:03 |
| mattmceuen | @roman_g we're using a rolling agenda etherpad now, would you mind copying your item to this one^? | 14:03 |
| roman_g | mattmceuen OK. Wiki needs to be updated | 14:04 |
| mattmceuen | thanks | 14:04 |
| *** SRao has joined #airshipit | 14:04 | |
| roman_g | Wiki is updated | 14:12 |
| *** dwalt has joined #airshipit | 14:25 | |
| mattmceuen | ty! | 15:00 |
| dwalt | #startmeeting airship | 15:00 |
| openstack | Meeting started Tue Nov 10 15:00:51 2020 UTC and is due to finish in 60 minutes. The chair is dwalt. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| *** openstack changes topic to " (Meeting topic: airship)" | 15:00 | |
| openstack | The meeting name has been set to 'airship' | 15:00 |
| mattmceuen | o/ | 15:01 |
| dwalt | Hey everyone, we'll get started in a few minutes. The design call is still wrapping up | 15:01 |
| airship-irc-bot | <ih616h> o/ | 15:04 |
| airship-irc-bot | <mf4716> o/ | 15:04 |
| airship-irc-bot | <james.gu> o/ | 15:04 |
| airship-irc-bot | <ak3216> o/ | 15:04 |
| roman_g | o/ | 15:04 |
| dwalt | Welcome to everyone filtering in from the design call. Here is our agenda today | 15:04 |
| dwalt | #link https://etherpad.opendev.org/p/airship-team-meeting | 15:04 |
| *** sreejithp has joined #airshipit | 15:05 | |
| airship-irc-bot | <j_t_williams> o/ | 15:06 |
| dwalt | Alrighty, let's get things started | 15:06 |
| dwalt | #topic Zuul gate problems | 15:06 |
| airship-irc-bot | <sean.eagan> o/ | 15:06 |
| *** openstack changes topic to "Zuul gate problems (Meeting topic: airship)" | 15:06 | |
| mattmceuen | Folks have probably noticed, we have a pair of unrelated but brutal issues with our gates at the moment | 15:06 |
| jemangs | o/ | 15:07 |
| mattmceuen | First: infrastructure availability for our 16gb VMs | 15:07 |
| mattmceuen | Of the two providers that supply them, we're back down to one working at the moment -- resulting in a lot of NODE_FAILURE type errors | 15:07 |
| airship-irc-bot | <lb4368> o/ | 15:07 |
| airship-irc-bot | <mb551n> o/ | 15:07 |
| sreejithp | o/ | 15:07 |
| jemangs | (The same problem as earlier?) | 15:08 |
| mattmceuen | Arijit has been working to get 3rd-party gating going per patchset | 15:08 |
| mattmceuen | similar @jemangs | 15:08 |
| mattmceuen | I think this may be a networking problem, previously a cooling problem, or something | 15:08 |
| mattmceuen | Currently the 3rd party gates are non-voting, and not (yet) reporting status up to patchsets -- that's a work in progress | 15:08 |
| roman_g | jemangs, similar. Previously that was AC failure, now – router malfunction. | 15:09 |
| mattmceuen | https://jenkins.nc.opensource.att.com | 15:09 |
| mattmceuen | While we get status reporiting to patchsets working, developers can manually check for their job's status in that UI^ | 15:09 |
| mattmceuen | And we'd like to switch (when ready) to making the 3rd party deployment gate our voting gate, instead of the zuul deployment gate | 15:10 |
| mattmceuen | to be clear: all the things that can be done in "normal size" VMs -- linting, image build, etc -- would still be run via zuul | 15:11 |
| mattmceuen | I would propose that for the time being, we make the zuul deployment gate *on-merge only*, and then check in jenkins for per-patch testing status | 15:11 |
| mattmceuen | This will get us unblocked | 15:12 |
| mattmceuen | Thoughts/concerns? | 15:12 |
| dwalt | This is great, thanks mattmceuen and sreejithp | 15:12 |
| airship-irc-bot | <ih616h> +1 | 15:12 |
| dwalt | Does that mean the job would be disabled for check, and enabled voting for gate? | 15:12 |
| mattmceuen | yes, disabled for check -- leaving more of our limited capacity available for the merge job | 15:13 |
| roman_g | dwalt , Yes, but you can trigger job manually by leaving "check experimental" comment to the patch set in Gerrit | 15:13 |
| airship-irc-bot | <mf4716> Did we hear any more about resolution to the router malfunction? | 15:13 |
| mattmceuen | +1 | 15:13 |
| dwalt | That makes a lot of sense. More reliable than relying on the cores to verify that the job passed. +1 from me | 15:13 |
| mattmceuen | The longer-term (but hopefully still soon) idea would be to make the 3rd party gating Voting, and make the zuul gate non-voting | 15:14 |
| mattmceuen | We should pull the trigger on that once we have status reporting back to the PS in place, and are comfortable with it | 15:14 |
| roman_g | mf4716 no. It's non-commercial cloud provider (basically home-based servers farm). | 15:15 |
| mattmceuen | Any other thoughts/comments before we move on to the other terrible gate issue? :) | 15:15 |
| mattmceuen | yeah - really useful from a community perspective, but can be problematic when we're relying on it for our day jobs | 15:15 |
| airship-irc-bot | <mf4716> any ETA roman? | 15:15 |
| mattmceuen | at least, when the third party gates break, it's our job to fix them :) | 15:15 |
| roman_g | mf4716 no ETA. | 15:16 |
| mattmceuen | So issue #2: | 15:16 |
| mattmceuen | And this one may be more challenging to work around. | 15:17 |
| mattmceuen | dockerhub has instituted rate limiting on image pulls, limiting to 100 pulls per-IP per month | 15:17 |
| mattmceuen | I believe we thought we'd been using some kind of image caching, but weren't, and now we've exausted our quota for the month from some of the open infra IPs | 15:18 |
| mattmceuen | A few ways we could potentially approach this: | 15:18 |
| mattmceuen | 1. migrate to using the 3rd party gates as Voting ASAP, and make sure they have caching enabled -- they'll have different IPs and so won't be hurt by the quota | 15:18 |
| mattmceuen | 2. migrate to a non-dockerhub mirror for all images hosted in dockerhub | 15:19 |
| mattmceuen | 3. upgrade to a paid plan | 15:19 |
| dwalt | Do we have an estimate on the number of images impacted? | 15:19 |
| mattmceuen | This is a promising (incomplete) list: | 15:21 |
| mattmceuen | [madgin@leviathan:~/airship2/airshipctl]$ grep -r "image: docker" manifests | 15:21 |
| mattmceuen | manifests/function/helm-operator/deployment.yaml: image: docker.io/fluxcd/helm-operator:1.2.0 | 15:21 |
| mattmceuen | [madgin@leviathan:~/airship2/airshipctl]$ | 15:21 |
| airship-irc-bot | <rp2723> Can we put the point in time images we care about in quay and use that instead | 15:21 |
| dwalt | lol. That is indeed promising | 15:21 |
| dwalt | If it's one image, migrating it to Quay seems like it would be feasible | 15:22 |
| airship-irc-bot | <rp2723> It’s a single image ?? The word list is rhetorical | 15:22 |
| mattmceuen | lol I should have googled earlier, mea culpa | 15:23 |
| mattmceuen | there could be other images in other projects though | 15:23 |
| mattmceuen | but getting airshipctl unblocked would be great | 15:23 |
| airship-irc-bot | <sean.eagan> helm-controller/source-controller are now in github container registry | 15:23 |
| dwalt | \o/ | 15:23 |
| airship-irc-bot | <sean.eagan> (the replacements for helm operator) | 15:23 |
| airship-irc-bot | <sean.eagan> https://github.com/orgs/fluxcd/packages/container/package/helm-controller | 15:24 |
| mattmceuen | what about helm-operator? We're not quite ready to switch to the helm-controller yet, are we @sean.eagan? | 15:24 |
| dwalt | So is it fair to say that we can push the operator image to quay, since we are moving away from it? | 15:24 |
| mattmceuen | yes | 15:24 |
| airship-irc-bot | <sean.eagan> https://review.opendev.org/#/c/758615/ | 15:24 |
| mattmceuen | +1 | 15:24 |
| airship-irc-bot | <sean.eagan> ^ it's merged | 15:24 |
| sreejithp | mattmceuen, we also need the go docker image "docker.io/golang:1.13.1-stretch" | 15:25 |
| sreejithp | we use this for building airshipctl binary | 15:25 |
| *** SRao has quit IRC | 15:26 | |
| mattmceuen | ahh you're right | 15:26 |
| mattmceuen | do we even still need helm-operator now that we have the helm-controller merged Sean? | 15:26 |
| mattmceuen | if we simply switch the test-site to use helm-controller, -operator wouldn't get run by the tests | 15:27 |
| airship-irc-bot | <sean.eagan> the function for helm-operator is still in place (for a short migration period), but it's not actually used by the default phases included in airshipctl | 15:27 |
| mattmceuen | then it must only be the golang image that's hurting the gates, right? | 15:28 |
| dwalt | Is the gate runner failing in the pre-run playbooks or later on? | 15:28 |
| dwalt | I think that's when we run `make images` | 15:29 |
| mattmceuen | (treasuremap is using the helm-operator, so we'll need to handle that) | 15:29 |
| airship-irc-bot | <sean.eagan> https://review.opendev.org/#/c/761666/ | 15:29 |
| airship-irc-bot | <sean.eagan> ^ treasuremap migration | 15:29 |
| mattmceuen | nice! | 15:30 |
| dwalt | That's great sean.eagan. Between pushing those two images to quay and finishing the helm controller migration, it sounds like this is largely under control | 15:32 |
| mattmceuen | I'm trying to find a failed job example and am not quickly... was seeing a few yesterday, so that's interesting | 15:33 |
| airship-irc-bot | <ih616h> Here's one that failed on the golang image https://zuul.opendev.org/t/openstack/build/36ce6912b90b45378d246273d193ebb6 | 15:33 |
| dwalt | Yeah. Mostly seeing the node failures on the front page | 15:33 |
| airship-irc-bot | <ih616h> (if that's what you're looking for) | 15:33 |
| mattmceuen | ahh yep, it's for the golang | 15:34 |
| airship-irc-bot | <ih616h> which makes sense - that image is pulled for even the linting jobs | 15:34 |
| roman_g | https://review.opendev.org/#/c/755456 Docker cache for airship/airshipctl Zuul jobs | 15:34 |
| mattmceuen | so migrating to an alternate source for that is a top priority, any volunteers? | 15:34 |
| dwalt | If we're just going to do a quick push, probably needs to be one of us from the wc :) | 15:35 |
| dwalt | I can do it | 15:35 |
| mattmceuen | oh I figured there might already be a golang image in gcr land :) | 15:35 |
| mattmceuen | they share a "g" | 15:35 |
| dwalt | oh that's a better idea | 15:36 |
| roman_g | There are also a few images which are pulled from inside of nested VMs. Those also need to be verified on where do they get pulled from. | 15:37 |
| roman_g | PS above is a solution to enable cacheing of container images on OpenDev infrastructure servers. | 15:38 |
| mattmceuen | awesome ty roman_g | 15:38 |
| mattmceuen | dwalt thank you for volunteering for that one, yeah I'd say just spend a few minutes searching for prior work before pushing to quay.io/airshipit as a Plan B | 15:39 |
| dwalt | I'll post to #airshipit if I find one | 15:39 |
| mattmceuen | Good news -- just learned from Arijit that the 3rd-party gates are now reporting back to gerrit! | 15:39 |
| dwalt | nice! | 15:40 |
| mattmceuen | Can I ask for a volunteer to put in a change to make the deployment zuul job on-merge only? | 15:40 |
| mattmceuen | I'd do it but need to step away for a couple hours | 15:40 |
| roman_g | mattmceuen Make airshipctl gate runner script run only on request and on Zuul gate (pre-merge) to reduce workload onto community CI | 15:40 |
| roman_g | https://review.opendev.itorg/762136 | 15:40 |
| roman_g | It's all in meeting Etherpad https://etherpad.opendev.org/p/airship-team-meeting | 15:41 |
| mattmceuen | That's perfect, +2 - ty Roman | 15:41 |
| dwalt | thanks roman_g | 15:41 |
| dwalt | Anything else we need to hash out? | 15:41 |
| mattmceuen | ok dwalt that all I have for our gate shenanigans, THANKS ALL for the working session | 15:41 |
| dwalt | great! | 15:42 |
| dwalt | #topic release tagging | 15:42 |
| *** openstack changes topic to "release tagging (Meeting topic: airship)" | 15:42 | |
| roman_g | Images pulled from inside of nested VMs are not covered. | 15:42 |
| dwalt | I think you're back in the spotlight mattmceuen | 15:42 |
| dwalt | roman_g: do you know those atm? We may need to let the gates run to see | 15:42 |
| mattmceuen | @roman_g agree we need to follow up on that; hopefully the golang change will give us a little bit of respite, esp will switching most of the VM load to 3rd party gates | 15:43 |
| roman_g | dwalt not atm, need to grep code a bit | 15:43 |
| dwalt | Sounds good. Let's go ahead and cover our last agenda item and circle back if we have some remaining time | 15:43 |
| mattmceuen | Tagging: so we'd decided when we cut our Beta release that we'd tag the quay.io image, but not the repo | 15:44 |
| mattmceuen | For the life of me, I don't recall why we thought that was a good idea -- seems like we should be doing both; @sean.eagan brought up yesterday wanting to do git diffs and needing to dig for commit hashes | 15:44 |
| mattmceuen | In addition, we're not actually labelling our tagged quay.io images with their git hash, so it's not easy to trace from "v2.0.0-beta.1" tag to commit hash in quay | 15:45 |
| mattmceuen | Are their any concerns with additionally labelling the airshipctl repo (and others in the future) with release tags which match the built quay images? | 15:45 |
| dwalt | I can't think of any, but I missed Sean's concern the other day | 15:47 |
| airship-irc-bot | <sean.eagan> use case is starting to explore publishing actual github releases with release notes etc: https://github.com/airshipit/airshipctl/issues/390 | 15:47 |
| dwalt | Do we need to start publishing our images with the commit sha mattmceuen? | 15:47 |
| mattmceuen | I think that would be a good thing to do additionally as well dwalt | 15:47 |
| mattmceuen | Cool, I will tag the repo later today then unless anyone objects. | 15:48 |
| mattmceuen | all from me dwalt :) | 15:48 |
| dwalt | Sound good. Thanks mattmceuen | 15:48 |
| dwalt | I'll create an issue for the image publishing. We can discuss on the flight plan call | 15:48 |
| dwalt | #topic roundtable | 15:48 |
| *** openstack changes topic to "roundtable (Meeting topic: airship)" | 15:48 | |
| dwalt | Okay folks, 10 mins left. Anything else we need to discuss? | 15:49 |
| dwalt | #topic reviews | 15:50 |
| *** openstack changes topic to "reviews (Meeting topic: airship)" | 15:50 | |
| dwalt | #link https://review.opendev.org/#/c/730777/ | 15:50 |
| dwalt | Just one review for today. And with that, we can adjourn. Thanks, everyone! Have a great day | 15:51 |
| dwalt | #endmeeting | 15:51 |
| *** openstack changes topic to "https://opendev.org/airship || https://wiki.openstack.org/wiki/Airship || https://review.opendev.org/#/q/projects:airship+status:open+NOT+label:Verified%253D-1+NOT+label:Workflow%253D-1+NOT+message:DNM+NOT+message:WIP" | 15:51 | |
| openstack | Meeting ended Tue Nov 10 15:51:16 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:51 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/airship/2020/airship.2020-11-10-15.00.html | 15:51 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/airship/2020/airship.2020-11-10-15.00.txt | 15:51 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/airship/2020/airship.2020-11-10-15.00.log.html | 15:51 |
| mattmceuen | ty @dwalt! | 15:51 |
| airship-irc-bot | <mf4716> ty all - make it a great day! | 15:53 |
| airship-irc-bot | <kk6740> @mattmceuen do u know by any chance where did `dev` tag for templater plugin docker image go? trying to find it in the repository seems to be missing https://quay.io/repository/airshipit/templater?tab=tags | 15:55 |
| airship-irc-bot | <kk6740> at the same time, we have it in our manifests https://github.com/airshipit/airshipctl/blob/master/manifests/function/airshipctl-base-catalogues/env-vars-template.yaml#L10 | 15:56 |
| roman_g | Add commit and other labels to the airshipctl container image https://review.opendev.org/762164 | 15:58 |
| airship-irc-bot | <victor.melin> Hi, is it normal that the docker image "quay.io/airshipit/templater:dev" is still referenced in the Airshipctl functions while it does not exist in the docker registry? | 16:05 |
| airship-irc-bot | <kk6740> @victor.melin That is what i am working on currently | 16:06 |
| airship-irc-bot | <kk6740> @victor.melin, @sidney.shiba I have added correct tag, can u try now ? | 16:11 |
| airship-irc-bot | <dwalt> `gcr.io/gcp-runtimes/go1-builder ` is a potential candidate for go. 8 MB smaller :) | 16:13 |
| airship-irc-bot | <victor.melin> oh sorry, I did not see your message.. | 16:29 |
| airship-irc-bot | <victor.melin> yes it works thanks ! | 16:30 |
| airship-irc-bot | <sb464f> is there a way to login to ephemeral node if cloud init is not ran and no user got created ..? | 17:01 |
| airship-irc-bot | <craig.anderson> with image-builder, yes | 17:06 |
| airship-irc-bot | <sb464f> the current airshipctl using image builder ..? | 17:21 |
| airship-irc-bot | <sb464f> I mean the latest. | 17:21 |
| *** muhaha has joined #airshipit | 17:24 | |
| airship-irc-bot | <craig.anderson> the one we integrated together | 17:52 |
| *** muhaha has quit IRC | 18:03 | |
| *** roman_g has quit IRC | 18:10 | |
| *** roman_g has joined #airshipit | 18:11 | |
| *** roman_g has quit IRC | 18:12 | |
| *** uzumaki has joined #airshipit | 20:00 | |
| uzumaki | hey mattmceuen , you there? | 20:00 |
| uzumaki | need to lend your thoughts for a couple minutes | 20:01 |
| *** GoldenBear_ has joined #airshipit | 20:05 | |
| *** GoldenBear has quit IRC | 20:05 | |
| *** dwalt has quit IRC | 20:05 | |
| airship-irc-bot | <mattmceuen> hey @uzumaki yep I'm here | 20:19 |
| uzumaki | how've you been? | 20:19 |
| uzumaki | mattmceuen, | 20:19 |
| airship-irc-bot | <mattmceuen> Can't complain :) | 20:19 |
| airship-irc-bot | <mattmceuen> you??? | 20:19 |
| uzumaki | I'm well, thanks. | 20:20 |
| uzumaki | I'm sure everybody's occupied with the airship 2.0 release coming up, I'm just wondering, what are some of the "should haves" or "good to haves" for airship in the coming months? | 20:21 |
| uzumaki | I know it's a very vague question, but I'm just curious, since we plan to make more contributions mattmceuen | 20:22 |
| airship-irc-bot | <mattmceuen> Hey that's awesome and I'm happy to hear that. One thing I'll offer is taking a look at the notes from the PTG for anything that aligns with what you guys would benefit from: https://etherpad.opendev.org/p/wallaby-ptg-airship | 20:25 |
| airship-irc-bot | <mattmceuen> We talked a lot about post-2.0 scope (sometimes in vague terms, with more detailed design to follow) | 20:25 |
| uzumaki | Oh, that's great! | 20:25 |
| airship-irc-bot | <mattmceuen> But multi-cluster management, airshipui, out-of-box-user-experience, disaster recovery and others | 20:26 |
| airship-irc-bot | <mattmceuen> More notes from the post-2.0 scope were captured here: https://hackmd.io/9eXSGC2WQOOFiVMrAocB3Q?both | 20:27 |
| airship-irc-bot | <mattmceuen> If any of those things sound like fun, we could start deeper discussion in the design call with you | 20:27 |
| uzumaki | That sounds GREAT! Unfotunately I missed this PTG and I'd be catching up with the recordings and it'll be awesome to catch up on the design call, sounds prefect! | 20:28 |
| uzumaki | Both the sources are very helpful, merci beaucoup! | 20:28 |
| airship-irc-bot | <mattmceuen> oh fantastic :slightly_smiling_face: yeah let me know what you think! de rien mon ami! | 20:29 |
| uzumaki | au revoir! o/ merci encore! | 20:29 |
| *** sreejithp has quit IRC | 22:24 | |
| *** ianychoi has quit IRC | 22:52 | |
| *** ianychoi has joined #airshipit | 22:52 | |
| *** uzumaki has quit IRC | 23:19 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!