Tuesday, 2019-05-28

*** jamesgu has joined #airshipit00:18
*** jamesgu__ has quit IRC00:19
*** jamesgu__ has joined #airshipit00:19
*** jamesgu has quit IRC00:49
*** sreejithp has joined #airshipit01:27
*** sreejithp has quit IRC01:31
*** calw has joined #airshipit02:29
*** licanwei has joined #airshipit03:20
*** krypto has quit IRC04:55
*** jamesgu__ has quit IRC05:19
*** jamesgu__ has joined #airshipit05:20
*** jamesgu__ has quit IRC05:54
*** jamesgu__ has joined #airshipit05:55
*** aojea has joined #airshipit06:16
*** mbeierl has quit IRC06:25
*** anyrude10_ has quit IRC06:57
*** anyrude10_ has joined #airshipit06:58
anyrude10_<roman_g> Thanks for your help. I am again getting an Invalid Document Format error while running the command ./tools/airship promenade build-all -o /target/bundle /target/collect/*.yaml /target/certs/*.yaml.     ERROR    req_id=- ctx=- end_user=- user=- promenade.exceptions:exceptions.py:222:display The provided documents failed schema validation07:00
openstackgerritASHWINI GHANWAT proposed airship/promenade master: Set default option for log_level in controller-manager  https://review.opendev.org/66128207:03
*** jhesketh has quit IRC07:11
*** jhesketh has joined #airshipit07:30
*** happyhemant has joined #airshipit07:31
*** mikakoski has joined #airshipit08:04
mikakoskisudo apt install ./slack-desktop-*.deb08:06
*** jesusaur has quit IRC09:17
*** jesusaur has joined #airshipit09:20
*** sreejithp has joined #airshipit09:28
*** sreejithp has quit IRC09:33
anyrude10_Hi, Trying to create Airskiff setup using the link https://airship-treasuremap.readthedocs.io/en/latest/airskiff.html, but facing error in the step Setup OpenStack Client.  2019-05-28 08:34:01.011 1 ERROR armada.cli     config_persister=config_persister) 2019-05-28 08:34:01.011 1 ERROR armada.cli   File "/usr/local/lib/python3.6/dist-packages/kubernetes/config/kube_config.py", line 503, in _get_kube_config_loader_for_yaml_file 209:40
anyrude10_PermissionError: [Errno 13] Permission denied: '/armada/.kube/config'09:40
roman_ganyrude10_: verify permissions10:14
roman_gprobably user id's under which you are running container and user id within a container do not match10:15
*** mikakoski has quit IRC10:49
anyrude10_<roman_g> I am running it as root11:05
roman_ganyrude10_: inside the container use is armada, thus it has no access rights to write files owned by root:root mounted from your host to within a container.11:20
roman_g*inside the container user is armada11:21
roman_g`export USER=$(id -u) GROUP=$(id -g)`11:22
anyrude10_<roman_g> so what you suggest? I should run all these scripts as non-user?11:22
roman_gand add `-u "${USER}:${GROUP}"` to docker command as a parameter (`docker run -u "${USER}:${GROUP}" ...` inside ./tools/airship or other script you use11:24
roman_g)11:24
*** mbeierl has joined #airshipit11:55
*** henriqueof has quit IRC12:56
*** bh526r has joined #airshipit12:57
*** kranthikirang has joined #airshipit12:59
*** kguttikonda has joined #airshipit13:00
*** kranthikirang has quit IRC13:03
*** henriqueof has joined #airshipit13:04
*** howell has joined #airshipit13:04
*** aaronsheffield has joined #airshipit13:20
*** sreejithp has joined #airshipit13:26
openstackgerritMichael Beaver proposed airship/shipyard master: Store status of deployment in a ConfigMap  https://review.opendev.org/66019713:32
*** sreejithp has quit IRC13:32
*** ab2434_ has joined #airshipit13:34
*** alexanderhughes has joined #airshipit13:35
*** henriqueof has quit IRC13:49
*** calw has quit IRC14:01
*** MattCoachCarter has joined #airshipit14:13
*** henriqueof has joined #airshipit14:20
*** aojea has quit IRC14:23
*** aojea has joined #airshipit14:23
*** dasp has quit IRC14:36
*** StaceyF has joined #airshipit15:07
openstackgerritIan Pittwood proposed airship/pegleg master: Allow decryption of directories  https://review.opendev.org/65740515:08
*** roman_g has quit IRC15:11
*** michael-beaver has joined #airshipit15:15
*** ian-pittwood has joined #airshipit15:21
openstackgerritMerged airship/shipyard master: Add deployment status generation to Shipyard  https://review.opendev.org/65963715:21
mattmceuenGM/GE all -- the agenda for our meeting in a half hour is here, please add anything you'd like to discuss today: https://etherpad.openstack.org/p/airship-meeting-2019-05-2815:24
*** arunkant has joined #airshipit15:28
arunkantHi, can any of the core reviewers review this and possibly merge this. https://review.opendev.org/#/c/638301/ . It has been pending for a while.15:30
openstackgerritMerged airship/pegleg master: Allow decryption of directories  https://review.opendev.org/65740515:37
*** nishantkr has joined #airshipit15:39
*** jamesgu__ has quit IRC15:41
*** jamesgu__ has joined #airshipit15:41
*** jamesgu__ has quit IRC15:41
*** mcfuller has joined #airshipit15:44
*** jamesgu has joined #airshipit15:48
openstackgerritIan Pittwood proposed airship/pegleg master: Specify collection for upload command  https://review.opendev.org/65937515:57
mattmceuen#startmeeting Airship16:00
openstackMeeting started Tue May 28 16:00:03 2019 UTC and is due to finish in 60 minutes.  The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: Airship)"16:00
mattmceuen#topic Rollcall16:00
openstackThe meeting name has been set to 'airship'16:00
*** openstack changes topic to "Rollcall (Meeting topic: Airship)"16:00
alexanderhughes\o/16:00
mattmceuenGood morning / evening everyone!16:00
ian-pittwoodo/16:00
mattmceuenOur illustrious agenda for today https://etherpad.openstack.org/p/airship-meeting-2019-05-2816:00
mattmceuenLet's give folks a couple minutes; I know we have a number of folks still out for the US holiday16:01
*** levmorgan has joined #airshipit16:01
michael-beavero/16:01
levmorgano/16:01
evgenylHi!16:01
jamesguo/16:02
mattmceuenthanks all for joining us today!  let's get started:16:02
mattmceuen#topic OpenSuse Airskiff Gates16:03
*** openstack changes topic to "OpenSuse Airskiff Gates (Meeting topic: Airship)"16:03
arunkanto/16:03
mattmceuenSo, as discussed at the PTG, there are both technical and roadmap-type challenges associated with doing full-airship-1.0-stack integrated testing using OpenSuse16:04
mattmceuenSUSE doesn't plan to use maas, and so it's not really a helpful test, among other reasons16:04
mattmceuenSo some of the validation that would be most valuable would be what airskiff gives you -- bring your own bare metal and k8s16:05
mattmceuenSo today I just wanted to chat through the work involved in making Airskiff gates for non-ubuntu images16:05
mattmceuenDo we have dwalt here by chance?16:05
jamesguso that will covers shipyard, armada, dechhand and pegleg, right/16:06
dwalto/16:06
mattmceuenYup exactly, with gates on the first three projects (not on pegleg itself) is how we do it today16:06
mattmceueno/ dwalt16:06
mattmceuenThe airskiff manifests live here, as a site definition in treasuremap: https://opendev.org/airship/treasuremap/src/branch/master/site/airskiff16:07
mattmceuenthey pull from the global versions.yaml, which defines the image overrides16:07
evgenylIs there a plan on how images overrides should be implemented?16:08
dwaltThat is going to be tricky16:08
dwaltWe may need to leverage the cicd layer to override images16:09
mattmceuenso I think the two broad approaches to non-ubuntu airskiff would be either "make a new site", e.g. airskiff-opensuse, which would be highly redundant; or, use the existing site definition but somehow solve for it to have multiple sets of overrides (which gets a little non-declarative)16:09
mattmceuenthere is not evgenyl, I think this is the first we've discussed16:10
evgenyl+1 for the second one, even if it would involve a hack with some `cp override.yaml site/airskiff..`16:10
jamesgudwalt: cicd layer you mean use ansible templates?16:10
dwaltSo, if we used the cicd layer, I think we would only need to override three charts. That would also keep us declarative16:10
dwaltno. There is an additional cicd layer in treasuremap16:11
dwalt#link https://opendev.org/airship/treasuremap/src/branch/master/global/layering-policy.yaml#L1116:11
mattmceuenthis would be a nice use case for having another layer in our definition hierarchy, if that existed -- e.g. something in between type and site, or a site definition that could inherit from/override another site.  But we don't have that today.16:11
dwaltmattmceuen: I think we do have something like that. I linked above :)16:11
evgenylHow do we enable this layer only for specific gates?16:12
mattmceuenoh hey - I didn't realize that was working yet dwalt :)16:12
mattmceuenthat is perfect16:13
evgenylI assume it's a site config change.16:13
dwaltgreat question, I think the best way would be to have a separate manifest at that layer. Then, it could be toggled in the pipeline16:13
dwaltI don't know if we can select layers using the site configuration. But that would be sweet16:13
mattmceuenwhere does that cicd layer get used today?16:14
mattmceuene.g. I don't see a `/cicd` in treasuremap -- does it get used dynamically only?16:14
dwaltI don't think it is used at all, but MattCoachCarter has been playing around with it for the new Deckhand integration gates16:14
dwaltmattmceuen: it's possible that's happening in one of the pipelines16:15
dwaltI hope we can walk away with a solution that doesn't do that for airskiff16:15
mattmceuengit blame tells me that kaspars added that cicd layer - I don't think we have him here today16:16
evgenylmattmceuen: I think it is used somehow in BM pipeline, but there is some jenkins magic that adds required overrides, so they are not in the repo.16:16
mattmceuenIs there any reason not to add a `/cicd` to treasuremap, and create an `airskiff-opensuse` under that, with very minimal document replacement -- versions.yaml alone hopefully?16:17
dwalt++ I like that approach16:17
dwaltExcept -- it would be the individual charts themselves, not versions.yaml16:17
evgenylCan we override versions.yaml?16:18
mattmceuenWhy not versions.yaml, to just override the image definitions?16:18
evgenylIt would be useful to be able to use an existing updater.py to do uplift.16:18
dwaltNot with the way it's setup today, the Airskiff site does the overrides there.16:18
dwaltI think we could probably fix that16:18
jamesgumattmceun: I like it... but I think there is more than versions.yaml... some osh charts have distro specific overrides, e.g., https://review.opendev.org/gitweb?p=openstack/openstack-helm.git;a=tree;f=horizon/values_overrides;h=1b85f9e317676e59ca8392aa529cbc0b8edf36ec;hb=refs/heads/master16:18
mattmceuenaha, good point jamesgu16:19
dwaltActually, I don't think we can fix that. Someone tell me if this reasoning is off16:19
jamesguwould be nice that airship can reference the overrides subdir for both distro and releases in airship/airskiff16:20
dwaltWe would "layer" to replace the versions.yaml, but the chart would use "substitution" to utilize the image. Since substitution happens before layering, we would be grabbing the unmodified images for the charts.16:20
evgenyljamesgu: Hmm, how these overrides are being used now? Are they just passed to helm to override the defaults (ubuntu)?16:20
jamesguI believe osh gate scripts are just passing in multiple overrides paths at the moment16:21
mattmceuendwalt:  no, it should work fine16:22
mattmceuendwalt: trying to think of an example :)16:22
dwaltmattmceuen: no worries! I'll accept that answer for now for timing purposes16:23
evgenylI would assume pegleg resolves the dependencies and does some sort of topological sort for dependencies.16:23
mattmceuenevgenyl: yep16:23
mattmceuenjamesgu:  I think we'll need to duplicate those configs into the new cicd/site definition16:23
jamesgutht's our experience re. versions and substitution.. +1 to what mattmceun said :-016:24
mattmceuenas they are raw values overrides in OSH, rather than in deckhand-friendly docs16:24
dwaltThat's good news. We should probably take care of modifying the current Airskiff setup first to avoid any unexpected consequences when implementing the cicd layer overrides.16:24
mattmceuenagree, would be good to get that done first16:25
evgenyl++16:25
mattmceuenCool - well I think the work in treasuremap should be the "interesting" part, and then copying / pasting the gates in armada, shipyard, deckhand should be pretty straightforwardf16:25
mattmceuenanything else to cover on this topic?16:25
dwaltJust that we would need to build the correct image for each component16:26
dwaltSo, that means environment variables, I believe16:26
mattmceuenahh - good point16:26
jamesguand add the ability to plug in image tags to the shipyard and pegleg tools script16:27
dwaltFor example, in Deckhand, the command would be `make images DISTRO=opensuse_15`16:27
mattmceuenjamesgu: I think that should be taken care of declaratively by using the opensuse-specific definition, right?16:27
mattmceuendwalt: would it be as simple as exporting DISTRO prior to running the airskiff scripts?16:28
mattmceuen(i.e. no changes needed to the scripts)16:28
dwaltCurrently, that happens in this playbook, so we could use Zuul variables https://opendev.org/airship/deckhand/src/branch/master/tools/gate/playbooks/airskiff-deploy.yaml#L2616:28
dwaltmattmceuen: correct, no script changes needed. Only zuul.yaml and playbook16:29
mattmceuenawesome16:29
jamesgumattmceund: I remember the shipyard tools is hard coded to master16:29
mattmceuenoh are you talking about this guy, jamesgu: https://opendev.org/airship/shipyard/src/branch/master/tools/shipyard.sh16:31
mattmceuenor are you talking about the airskiff script16:31
jamesguyes, the shipyard.sh16:31
jamesgudoes Airskiff still invoke it?16:31
*** kskels has joined #airshipit16:31
mattmceuencool - looks like that can be overridden using SHIPYARD_IMAGE as well16:31
jamesguah okay16:31
evgenylI think airskiff uses https://opendev.org/airship/treasuremap/src/branch/master/tools/airship now16:32
mattmceuengood catch16:32
dwaltAirskiff now uses a new utility script in treasuremap:16:32
dwalt#link https://opendev.org/airship/treasuremap/src/branch/master/tools/airship16:32
jamesguah okay!16:32
jamesgunice!16:32
mattmceuendwalt: will the `airship` command use the shipyard image from the rendered software-versions doc, or from the global values.yaml?16:33
mattmceuenlooks like the latter unfortunately16:34
kskelsright - reading versions.yaml directly16:34
dwaltmattmceuen: Looks like you're right. Why unfortunately?16:34
dwaltoh, I see. We need rendered.16:34
mattmceuenbecause the global versions.yaml won't have the distro-specific override for the SY image in it16:34
kskelsthe YAML reading of python I believe was already meant for rendered document.. e.g. it should traverse docs..16:35
dwaltDoes anyone think that introducing an environment variable override for images to that script would be inappropriate?16:35
kskelsso it should be possible to add that functionalitty to the script rather straight forward as it already has the pegleg command also16:35
mattmceuenkskels: that would be great16:36
dwalt++ that would be preferable to what I suggested16:36
kskelsI think adding envs is nice also16:36
kskelsso you can use that script for more generic case..16:36
mattmceuenagree - glad we uncovered that here16:37
dwalttrue. We can still do that but use the declarative option :)16:37
mattmceuenOk - we need to keep moving! :)  a couple more agenda items16:37
mattmceuenI think we're in good shape with next steps on the airskiff genericization16:37
mattmceuen#topic AIAB -> Treasuremap update16:38
*** openstack changes topic to "AIAB -> Treasuremap update (Meeting topic: Airship)"16:38
mattmceuenevgenyl - this is yours, you've been doing a lot of work here.  Want to give us an update?16:38
evgenylYep, in the last few days I've been working on moving AIAB into treasuremap repo.16:38
evgenylReviews are welcome https://review.opendev.org/#/c/656900/16:39
mattmceuen\o/16:39
evgenylThere is a comment from kskels , I have addressed it in a separate PS.16:39
evgenylhttps://review.opendev.org/#/c/661363/16:39
dwaltthis is fantastic evgenyl!16:39
kskelsI'll review today once more, thank you! this is awesome16:40
evgenylAs soon as we get it merged and it is stable, we can start looking into multinode gate.16:40
evgenylkskels: thanks!16:40
mattmceuenYes, this'll be very valuable in helping to keep AIAB up to date in its definitions, which will make it much more helpful to the community16:40
mattmceuenPlease give that a review - and if anyone out there has had issues with AIAB in the past, please give this a try as well -- it should have the latest and greatest airship definitions16:41
mattmceuenAnything else on this one?16:41
evgenylThat is it from me, thanks!16:42
mattmceuenAlright - thanks evgenyl.  Moving on:16:42
mattmceuen#topic airskiff deployment gate is failing on shipyard project, and maybe others16:42
*** openstack changes topic to "airskiff deployment gate is failing on shipyard project, and maybe others (Meeting topic: Airship)"16:42
mattmceuenMattCoachCarter, what's the damage16:42
MattCoachCarterI've been noticing this on recent PSs in the shipyard project.16:42
MattCoachCarterThe gate seems to fail around 12m in trying to install docker I think.16:42
MattCoachCarterI don't know much about these gates, I just wanted to bring it to everyone's attention.16:43
MattCoachCarterIt's non-voting, so it isn't blocking anything, but I think it's a useful check and I'd like to see it stable again.16:43
mattmceuenfor context if anyone didn't know:  the airskiff gates ar non-voting, so that cross-project integration against master is not a hard requirement of making per-project changes16:44
MattCoachCarterNot sure who the best person to look into this would be. But that's about it for this topic.16:44
mattmceuen2019-05-23 17:11:38.391285 | primary | 2019-05-23 17:11:38.390 1 ERROR armada.handlers.wait [-] [chart=ucp-deckhand]: Timed out waiting for jobs (namespace=ucp, labels=(release_group=airship-ucp-deckhand)). These jobs were not ready=['deckhand-db-init', 'deckhand-db-sync'][00m16:45
mattmceuenwe'll need to look in the logs to see why armada was timing out on deckhand16:45
dwalt`E: Unable to locate package libxtables12` ... hmm. It looks like new dependencies were added16:46
MattCoachCarterYeah I saw that too.16:46
dwaltmattmceuen: I think that one may be a read-herring16:46
dwaltred*16:46
MattCoachCarterAnyway, it's not something we need to debug and solve during this meeting. As long as we know it's something that needs fixing.16:47
mattmceuenyep, thanks for raising awareness MattCoachCarter.  Non-voting gates are only as valuable as the people that watch for them to fail :)16:47
mattmceuenOk final topic -16:48
mattmceuen#topic Deckhand multi-dstro review pending for core reviewer attention16:48
*** openstack changes topic to "Deckhand multi-dstro review pending for core reviewer attention (Meeting topic: Airship)"16:48
mattmceuenhttps://review.opendev.org/#/c/638301/16:48
mattmceuenthis been waiting for review since we worked around the old deckhand gate issue - would appreciate additional review!16:49
mattmceuenthat's a good transition into16:49
mattmceuen#topic Roundtable16:49
*** openstack changes topic to "Roundtable (Meeting topic: Airship)"16:50
mattmceuenAnything else that needs some eyeballs today?16:50
*** aojea has quit IRC16:50
mattmceuenor any other topics on your minds16:51
mattmceuenIn that case - I will give everyone a couple minutes rest before the next meeting (or a couple extra minutes of sleep if you're jayahn)16:52
mattmceuenThanks everyone, great meeting!16:52
evgenylThanks!16:52
mattmceuen#endmeeting16:52
*** openstack changes topic to "airshipit.org || General Review Dashboard: https://review.opendev.org/#/q/project:%255Eairship.*+status:open,n,z"16:52
openstackMeeting ended Tue May 28 16:52:47 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:52
openstackMinutes:        http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-05-28-16.00.html16:52
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-05-28-16.00.txt16:52
openstackLog:            http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-05-28-16.00.log.html16:52
*** vladyslav has joined #airshipit16:53
*** jamesgu_ has joined #airshipit16:53
evgenylOh, I forgot to bring up the discussion on MaaS in AIAB..16:59
evgenylWe can discuss this next week.16:59
*** MattCoachCarter has quit IRC17:03
*** irclogbot_3 has quit IRC17:17
*** irclogbot_2 has joined #airshipit17:17
*** sreejithp has joined #airshipit17:25
*** mbeierl has quit IRC17:33
*** mbeierl has joined #airshipit17:49
*** bh526r has quit IRC18:17
*** jesusaur has quit IRC18:30
clarkbmattmceuen: fyi https://review.opendev.org/#/c/656008/2 needs a rebase ( and I believe we are still on track to do that this friday )18:37
mattmceuenthanks clarkb!  That's great news, and I'll rebase now18:37
*** kguttikonda has quit IRC18:51
openstackgerritAlexander Hughes proposed airship/pegleg master: Fix --save-location error in decrypt command  https://review.opendev.org/66182718:54
*** arunkant has quit IRC18:54
openstackgerritLev Morgan proposed airship/pegleg master: Fix multiple I/O issues in cert generation  https://review.opendev.org/64367818:56
*** kskels has quit IRC19:01
mattmceuenhey clarkb, we now have spyglass-core and pegleg-core groups, but I can't add anyone because I'm not a member.  How do we bootstrap somebody into those groups so we can add the right folks?19:18
clarkbmattmceuen: a gerrit admin needs to seed the group with an initial member who can then manage things. Should that initial member be you?19:19
mattmceuenthat would be awesome - I can take it from there. ty!19:19
clarkbmattmceuen: I added your account with the att email address. There is another email addr though so let me know if I need to swap them19:20
mattmceuennope that's fine -- appreciate your help19:20
*** henriqueof has quit IRC19:32
openstackgerritLev Morgan proposed airship/pegleg master: Fix multiple I/O issues in cert generation This patch handles the case where CA certs or authorities are loaded as byte strings. It also disables parsing YAML documents with python/object types directly into (non-dict) Python objects (which is PyYaml's de  https://review.opendev.org/64367819:33
openstackgerritLev Morgan proposed airship/pegleg master: Fix multiple I/O issues in cert generation  https://review.opendev.org/64367819:36
*** ian-pittwood has quit IRC19:42
openstackgerritMerged airship/spyglass master: Revise CLI documentation  https://review.opendev.org/65013719:47
*** licanwei has quit IRC19:56
openstackgerritMerged airship/pegleg master: Make -u required in CLI when required by repo  https://review.opendev.org/64917619:56
jayahnmattmceuen: thanks for giving me a minutes of extra sleep. ;)19:58
*** kranthikirang has joined #airshipit19:59
*** levmorgan has quit IRC20:06
*** dasp has joined #airshipit20:20
*** sreejithp_ has joined #airshipit20:30
*** sreejithp has quit IRC20:31
openstackgerritMerged airship/shipyard master: Create Xcom Pusher/Puller for concurrency_check  https://review.opendev.org/66113520:35
openstackgerritIan Pittwood proposed airship/spyglass master: [WIP] Data objects for Spyglass  https://review.opendev.org/65891720:38
*** dasp has quit IRC20:38
*** dasp has joined #airshipit20:38
openstackgerritIan Pittwood proposed airship/spyglass master: Separate plugins from Spyglass  https://review.opendev.org/65355520:45
*** alexanderhughes has quit IRC20:45
openstackgerritIan Pittwood proposed airship/spyglass-plugin-xls master: Update plugin docs  https://review.opendev.org/66184120:49
openstackgerritIan Pittwood proposed airship/pegleg master: Specify collection for upload command  https://review.opendev.org/65937521:00
openstackgerritAlexander Vlasov proposed airship/promenade master: Apply docker-default AppArmorProfile for calico-etcd pod  https://review.opendev.org/65918821:02
*** howell has quit IRC21:51
*** kranthikirang has quit IRC22:05
openstackgerritMichael Beaver proposed airship/shipyard master: Follow up change for deployment-status ConfigMap  https://review.opendev.org/66185422:13
*** Talion has joined #airshipit22:24
*** ab2434_ has quit IRC22:30
*** StaceyF has quit IRC22:52
openstackgerritMerged airship/pegleg master: Specify collection for upload command  https://review.opendev.org/65937522:53
*** aaronsheffield has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!