| openstackgerrit | Matt McEuen proposed openstack/airship-treasuremap master: WIP: airskiff: Use global Airship charts https://review.openstack.org/635218 | 00:19 |
|---|---|---|
| openstackgerrit | Arun Kant proposed openstack/airship-deckhand master: Adding opensuse image build for deckhand https://review.openstack.org/638301 | 00:37 |
| openstackgerrit | Arun Kant proposed openstack/airship-deckhand master: Adding opensuse image build for deckhand https://review.openstack.org/638301 | 00:43 |
| openstackgerrit | Arun Kant proposed openstack/airship-deckhand master: Adding opensuse image build for deckhand https://review.openstack.org/638301 | 00:51 |
| openstackgerrit | Arun Kant proposed openstack/airship-deckhand master: Adding opensuse image build for deckhand https://review.openstack.org/638301 | 00:54 |
| openstackgerrit | Arun Kant proposed openstack/airship-deckhand master: Adding opensuse image build for deckhand https://review.openstack.org/638301 | 00:57 |
| openstackgerrit | Arun Kant proposed openstack/airship-deckhand master: Adding opensuse image build for deckhand https://review.openstack.org/638301 | 00:58 |
| *** nishant__ has quit IRC | 02:13 | |
| *** roman_g has quit IRC | 03:19 | |
| *** mbologna_ has joined #airshipit | 07:05 | |
| *** mbologna has quit IRC | 07:06 | |
| *** pkaralis has quit IRC | 08:31 | |
| *** roman_g has joined #airshipit | 08:32 | |
| *** georgk3 has joined #airshipit | 09:29 | |
| *** georgk has quit IRC | 09:30 | |
| *** arunkant has quit IRC | 09:35 | |
| *** arunkant has joined #airshipit | 09:36 | |
| *** roman_g has quit IRC | 09:45 | |
| *** mbologna_ has quit IRC | 10:23 | |
| *** mbologna has joined #airshipit | 10:27 | |
| openstackgerrit | Smruti Soumitra Khuntia proposed openstack/airship-shipyard master: User context tracing through logging https://review.openstack.org/633873 | 11:38 |
| openstackgerrit | Smruti Soumitra Khuntia proposed openstack/airship-shipyard master: User context tracing through logging https://review.openstack.org/633873 | 11:38 |
| *** roman_g has joined #airshipit | 12:23 | |
| *** mbologna_ has joined #airshipit | 12:26 | |
| *** mbologna has quit IRC | 12:30 | |
| *** openstackgerrit has quit IRC | 12:30 | |
| *** dimitris_ has quit IRC | 13:03 | |
| *** hemanth_n has joined #airshipit | 13:32 | |
| *** sgrasley has joined #airshipit | 13:46 | |
| *** michael-beaver has joined #airshipit | 13:51 | |
| *** dustinspecker has joined #airshipit | 13:54 | |
| *** levmorgan has joined #airshipit | 13:57 | |
| mattmceuen | #startmeeting airship | 14:00 |
| mattmceuen | #topic Rollcall | 14:00 |
| openstack | Meeting started Tue Feb 26 14:00:55 2019 UTC and is due to finish in 60 minutes. The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:01 |
| *** openstack changes topic to " (Meeting topic: airship)" | 14:01 | |
| mattmceuen | Hello everyone! | 14:01 |
| openstack | The meeting name has been set to 'airship' | 14:01 |
| *** openstack changes topic to "Rollcall (Meeting topic: airship)" | 14:01 | |
| dwalt | o/ | 14:01 |
| michael-beaver | o/ | 14:01 |
| mattmceuen | Here's our agenda for today - please take a moment to add anthing you'd like to discuss: https://etherpad.openstack.org/p/airship-meeting-2019-02-26 | 14:01 |
| georgk3 | hi | 14:01 |
| levmorgan | o/ | 14:01 |
| portdirect | ヅ | 14:01 |
| *** aaronsheffield has joined #airshipit | 14:01 | |
| arunkant | o/ | 14:02 |
| b-str | o/ | 14:02 |
| roman_g | o/ | 14:02 |
| mattmceuen | #topic Consider what onboarding environment seems the most valuable | 14:02 |
| *** openstack changes topic to "Consider what onboarding environment seems the most valuable (Meeting topic: airship)" | 14:02 | |
| aaronsheffield | o/ | 14:02 |
| mattmceuen | So this one was sthussey's; unfortunately he's not in today | 14:03 |
| pas-ha | o/ | 14:03 |
| mattmceuen | o/ everyone | 14:03 |
| *** Nishant_ has joined #airshipit | 14:03 | |
| mattmceuen | sthussey had brought up a good point that we have a number of different environments that are recommended for onboarding / reference type things | 14:03 |
| pas-ha | my question would also be 'is there a doc describing a dev process' | 14:04 |
| pas-ha | like if I need to patch and test | 14:04 |
| mattmceuen | Airskiff | 14:04 |
| mattmceuen | Airship-in-a-bottle local VM | 14:04 |
| mattmceuen | Airship-in-a-bottle multi-VM | 14:04 |
| mattmceuen | Treasuremap - Seaworthy | 14:04 |
| mattmceuen | There is a little bit of that in the Airskiff documentation, but not much to speak of pas-ha. I think that's a very good value-add for the docs | 14:05 |
| pas-ha | like a path from local repo -> patch -> build new container? -> push it to local docker? -> redeploy? | 14:05 |
| mattmceuen | A "day in the life of a developer" kind of thing | 14:05 |
| pas-ha | yes, smth like that | 14:05 |
| georgk3 | +1 | 14:05 |
| pas-ha | figuring that out myself for now | 14:05 |
| mattmceuen | #action mattmceuen to create a storyboard item for documenting the Airship development lifecycle | 14:06 |
| *** levmorgan_ has joined #airshipit | 14:06 | |
| *** levmorgan has quit IRC | 14:06 | |
| *** mbeierl has joined #airshipit | 14:07 | |
| mattmceuen | My thought is that for the different "things" up above, they are each a combo of a couple different characteristics: | 14:07 |
| mattmceuen | 1) a deployment approach | 14:07 |
| mattmceuen | 2) a collection of manifests | 14:07 |
| mattmceuen | Whatever else we do, I want to try to align our manifests to one another as much as possible | 14:07 |
| mattmceuen | dwalt has a patchset out for example that takes the airskiff manifests and aligns them more closely to the treasuremap globals | 14:08 |
| mattmceuen | I think maintenance of different, segmented manifests sets has been part of the maintenance challenge that sthussey wants to avoid | 14:09 |
| dwalt | This might be a step back, but as far as lifecycle goes, I usually try to run the component locally before deploying any changes into one of our "development environments (i.e. Airskiff, AIAB) | 14:09 |
| dwalt | For example, you can run Armada as a local Python project against KubeADM cluster, and it will find Tiller | 14:09 |
| dwalt | Deckhand and Shipyard also support similar dev setups, but they are buried a bit in the docs | 14:10 |
| mattmceuen | Promenade has a great "prom only" test environment, which spins up a 4-node VM cluster and beats it up. This is what Airship-in-a-Bottle multinode (the tooling) was modeled after | 14:11 |
| mattmceuen | Airship-in-a-Bottle (the manifest set) is something that could be aligned into the treasuremap globals | 14:11 |
| mattmceuen | Let me ask for y'alls opinions -- what do you find valuable amongst these things? I've heard the opinion that "developers don't tend to use airship single node", but that's the main onboarding path for new folks. Would AiaB multinode serve that purpose, perhaps even better? | 14:12 |
| mattmceuen | I for one use Airskiff and AiaB multinode for my own work, depending on what I'm working on | 14:14 |
| roman_g | For professional development people do probably use big bare-metal servers with lots of RAM. | 14:15 |
| *** dimitris_ has joined #airshipit | 14:15 | |
| roman_g | This is not something to expect to have from those who only want to have a look at Airship | 14:15 |
| dwalt | For me, it depends on the component. I tend to use Airskiff (single node) for the software delivery components and AIAB multimode in other cases | 14:15 |
| michael-beaver | I for one use the multinode a lot, but I think it is very prohibitive for someone without a dedicated lab | 14:15 |
| mattmceuen | The nice thing about multinode is that it can deploy a variable number of nodes, based on configs... even perhaps a single node :) | 14:16 |
| mattmceuen | So that's where my head is at with, we could do more things with less moving pieces if we aligned our manifests together and configured AiaB multinode scripts to be the single-node demo environment as well | 14:17 |
| mattmceuen | Any disagreement with this general sentiment? | 14:17 |
| levmorgan_ | Sounds good to me. | 14:18 |
| georgk3 | I like the multinode environment a lot, so +1 | 14:18 |
| michael-beaver | Maybe this is trivial, but another problem with running it is you need a Host OS capable of nested-virt, so windows using virtualbox is a no go ATM, but overall I don't disagree with the sentiment that the multinode gate would help us have less moving pieces | 14:18 |
| dwalt | ++ mattmceuen. It'd be a great step moving our development environments closer in-line with our "reference" architecture | 14:19 |
| mattmceuen | Cool, let's circle back with sthussey when he's back | 14:20 |
| *** dimitris_ has quit IRC | 14:20 | |
| mattmceuen | good point michael-beaver and we should keep that in mind | 14:20 |
| mattmceuen | we've also been talking about aligning AiaB manifests to treasuremap for a long time | 14:20 |
| mattmceuen | #action mattmceuen create a Storyboard item for aligning AiaB manifests to Treasuremap | 14:20 |
| mattmceuen | anything else on this topic for now guys? | 14:21 |
| mattmceuen | #topic Ironic driver spec | 14:22 |
| *** openstack changes topic to "Ironic driver spec (Meeting topic: airship)" | 14:22 | |
| mattmceuen | pas-ha and some other folks have been going some work on the spec for adding Ironic support to Drydock: | 14:22 |
| mattmceuen | https://review.openstack.org/#/c/613358/ | 14:23 |
| mattmceuen | roman_g, you wanted to sync up on that here, right? | 14:23 |
| roman_g | mattmceuen: not really to sync up, but may be we have Hemmanth here? | 14:25 |
| roman_g | hemanth_n: | 14:25 |
| roman_g | If not, let's move on. | 14:25 |
| mattmceuen | Ok. I need to catch up on the latest discussion in the PS as well; pas-ha & roman_g should we plan to touch on this next week? | 14:26 |
| roman_g | Yes. Move to the next week. | 14:26 |
| roman_g | Would also be good to discuss on design call | 14:26 |
| roman_g | probably | 14:26 |
| mattmceuen | If there's anything specific we need to grab hemanth_n's attention on we can mailing list it as well | 14:26 |
| roman_g | True. | 14:27 |
| mattmceuen | Next up: | 14:28 |
| mattmceuen | #topic Airship talks accepted into the Denver Summit | 14:28 |
| *** openstack changes topic to "Airship talks accepted into the Denver Summit (Meeting topic: airship)" | 14:28 | |
| mattmceuen | So, we have some Airship talks accepted into the Denver Summit! | 14:28 |
| mattmceuen | Unfortunately, the summit website appears to not be functioning today, so I can't tell you what they all are! :D | 14:28 |
| mattmceuen | So many great Airship talks that the database crashed perhaps | 14:29 |
| mattmceuen | will share next time | 14:29 |
| mattmceuen | #topic Design meeting(s) for Docker, Kernel, OS, Security Patches, etc | 14:29 |
| *** openstack changes topic to "Design meeting(s) for Docker, Kernel, OS, Security Patches, etc (Meeting topic: airship)" | 14:29 | |
| mattmceuen | Just wanted to make sure folks were aware that Rodolfo has scheduled a design call later today (and may end up being recurring as needed) to dive deep into the topic of how to best orchestrate heavyweight, low-level changes across Airship sites | 14:30 |
| *** dimitris_ has joined #airshipit | 14:30 | |
| mattmceuen | Things like operating system upgrades, kubelet upgrades, docker, kernel, etc | 14:30 |
| mattmceuen | As well as simple cluster-wide reboots when needed | 14:30 |
| georgk3 | interesting, thanks for the info | 14:31 |
| roman_g | interesting. not in my calendar | 14:31 |
| evgenyl | Where can I get an invite? | 14:31 |
| roman_g | will ask. thank you | 14:31 |
| mattmceuen | The intention has always been for Airship to be able to reprovision hosts in a way that preserves the local data for the running workloads (e.g. VMs) . This is not quite finished yet, but will allow for e.g. host-by-host or failure-domain-by-domain reprovisionings of the cluster with e.g. a new patched kernel | 14:32 |
| mattmceuen | However, that's also a heavyweight operation and there are lighterweight things we can do as well | 14:32 |
| mattmceuen | Anyway - please join if you'd like to discuss more | 14:32 |
| roman_g | Good to have it recorded and published | 14:33 |
| mattmceuen | The invite should be out on the Mailing List, if you go back to the archives from yesterday or Friday I believe | 14:33 |
| evgenyl | Oh, sorry, it was in the mailing list, I somehow missed it. | 14:33 |
| mattmceuen | Don't worry evgenyl, I am good at missing all kinds of email | 14:33 |
| mattmceuen | three and a half hours from now | 14:33 |
| mattmceuen | Moving on: | 14:33 |
| mattmceuen | #topic Divingbell gates are broken in upstream | 14:33 |
| *** openstack changes topic to "Divingbell gates are broken in upstream (Meeting topic: airship)" | 14:33 | |
| mattmceuen | roman_g this is yours, go for it | 14:34 |
| roman_g | Pretty much everything is in the description https://etherpad.openstack.org/p/airship-meeting-2019-02-26 | 14:34 |
| roman_g | Update of Helm from version 2.11.0 to 2.12.1 and then to 2.12.3 broke host and label `overrides:` functionality for the Airship Divingbell. | 14:34 |
| roman_g | This affects Divingbell development (voting gates not working), and partially other projects like openstack/openstack-helm-infra, which have Divingbell as a non-voting gate | 14:34 |
| roman_g | Roman is looking for an assistance | 14:34 |
| roman_g | Issue in one line: secret contents which is used for the host falling under override is identical to the secret used by default | 14:35 |
| Nishant_ | I see a topic dedicated to the overrides issue in openstack-helm meeting today starting 9 AM CST | 14:35 |
| mattmceuen | awesome | 14:35 |
| mattmceuen | Yes this is definitely something we need to get fixed ASAP, thank you roman_g for your analysis and summary | 14:36 |
| roman_g | I was offline probably. Will look at the logs. | 14:36 |
| mattmceuen | Nope, it's coming up in a half hour :) | 14:36 |
| roman_g | Ah, today. Good. | 14:37 |
| roman_g | Nishant_: thank you! | 14:37 |
| mattmceuen | Shall we see how the discussion goes on the OSH side, and if confirmed to be an OSH issue we can help dev/test as possible? And otherwise circle back in the chat room roman_g? | 14:37 |
| roman_g | Yes, sure. I have a testbed running, and can test easily. | 14:38 |
| mattmceuen | sweet | 14:38 |
| mattmceuen | #topic Deckhand & Shipyard - OpenSUSE builds | 14:38 |
| *** openstack changes topic to "Deckhand & Shipyard - OpenSUSE builds (Meeting topic: airship)" | 14:38 | |
| roman_g | Arun has started to work on building OpenSUSE-based images here: https://review.openstack.org/#/q/status:open+branch:master+topic:airship_suse | 14:38 |
| roman_g | Does not align good with https://airship-specs.readthedocs.io/en/latest/specs/approved/multi-linux-distros.html we have approved | 14:38 |
| roman_g | #action Need to follow-up with/between Arun and James Gu | 14:38 |
| mattmceuen | What is the main difference between the approaches? | 14:39 |
| roman_g | Incomplete approach. | 14:39 |
| mattmceuen | ok | 14:40 |
| roman_g | Not distro-independent, no tests, only image build part | 14:40 |
| arunkant | mattmceuen: hi, currently we are working on building images. So adding another variable input to ask for distro flavor and that is used in jobs | 14:40 |
| roman_g | arunkant: are those 2 changes complete or WIP? | 14:41 |
| arunkant | roman_g: its work-in-progress, so trying to add check and gate jobs for building images. Is there any specific tests are needed other than adding jobs ? | 14:41 |
| roman_g | arunkant: check the spec via the link above. There are other things which need to be done. | 14:42 |
| roman_g | Also please don't override ubuntu-based :latest with opensuse-based :latest image on quay | 14:43 |
| arunkant | roman_g: Also need clarification where the docs need to be provided for building opensuse images . Is it updated spec or some other doc section ? | 14:43 |
| roman_g | in deckhand and in shipyard docs | 14:43 |
| roman_g | spec is used to discuss implementation, collect feedback | 14:44 |
| mattmceuen | arunkant, can you please give the spec a read, and see if it covers all the bases or if there are any gaps from your perspective? | 14:45 |
| roman_g | Can you reach out to James Gu (SUSE Gmbh.) and may be talk to him on implementation specifics detailed in the spec he wrote? | 14:45 |
| arunkant | roman_g: okay. Will look into docs for those components. | 14:45 |
| mattmceuen | feel free to shout here in the chat room any time, really awesome to see your work on this, and looking forward to helping. Thanks for your efforts here | 14:46 |
| arunkant | roman_g: Yes, I work with James Gu so will clarify the details with him. | 14:46 |
| mattmceuen | Anything else on this topic all? | 14:47 |
| roman_g | Actually, my top list of what I would love to see in Airship is to see it working on SLES & RHEL (OpenSUSE & CentOS) in addition to Ubuntu. | 14:47 |
| roman_g | So, thank you for your work, arunkant | 14:47 |
| arunkant | roman_g: will clarify in chat about the image tag question as oepnsuse images uses different tag (opensuse_15_latest) | 14:47 |
| roman_g | arunkant: ahha, good. | 14:48 |
| roman_g | mattmceuen | 14:48 |
| arunkant | roman_g: yes that's the plan to make airship components to work with opensuse | 14:49 |
| mattmceuen | ++ | 14:49 |
| mattmceuen | #topic Patchset reviews requested | 14:50 |
| *** openstack changes topic to "Patchset reviews requested (Meeting topic: airship)" | 14:50 | |
| mattmceuen | https://review.openstack.org/#/c/615892/ - docs: Add use cases for each of the mutation operations | 14:50 |
| mattmceuen | Hooray for docs! | 14:50 |
| mattmceuen | oh that guy's merged already | 14:50 |
| mattmceuen | I should have checked :D | 14:50 |
| mattmceuen | Any other patchsets that folks would like to request some solid review? | 14:51 |
| mattmceuen | #topic Roundtable | 14:51 |
| *** openstack changes topic to "Roundtable (Meeting topic: airship)" | 14:51 | |
| mattmceuen | Any other topics, all? | 14:51 |
| roman_g | Nothing. Thank you, Matt. | 14:53 |
| mattmceuen | Alright - thanks everyone for joining us today | 14:53 |
| mattmceuen | Have a great week! | 14:53 |
| mattmceuen | #endmeeting | 14:53 |
| *** openstack changes topic to "airshipit.org || General Review Dashboard: https://review.openstack.org/#/q/project:%255Eopenstack/airship.*+status:open,n,z" | 14:53 | |
| openstack | Meeting ended Tue Feb 26 14:53:55 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:53 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-02-26-14.00.html | 14:53 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-02-26-14.00.txt | 14:53 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-02-26-14.00.log.html | 14:54 |
| *** levmorgan_ has quit IRC | 14:58 | |
| *** michael-beaver has quit IRC | 14:58 | |
| *** DanCrank has joined #airshipit | 14:59 | |
| *** sreejithp has joined #airshipit | 15:08 | |
| *** jamesgu has joined #airshipit | 15:23 | |
| *** jamesgu__ has joined #airshipit | 15:24 | |
| *** jamesgu__ has quit IRC | 15:24 | |
| *** michael-beaver has joined #airshipit | 15:27 | |
| *** hemanth_n has quit IRC | 15:55 | |
| vdrok | hey folks, I have a question from one of mirantis' ceph guys, are there any plans to improve the ceph deployment strategy in openstack-helm? One of the things he mentioned was deprecation of ceph-disk for mimic (it seems to be used at least during osd initialization), also maybe using rook instead of bash scripts there currently are? | 16:08 |
| roman_g | vdrok: probably beter to ask in #openstack-helm. It's silent here. | 16:22 |
| *** michael-beaver has quit IRC | 16:22 | |
| vdrok | roman_g: okie, will do | 16:23 |
| *** sgrasley has quit IRC | 16:31 | |
| *** michael-beaver has joined #airshipit | 16:38 | |
| *** michael-beaver has quit IRC | 16:39 | |
| *** michaelbeaver has joined #airshipit | 16:39 | |
| *** openstackgerrit has joined #airshipit | 16:50 | |
| openstackgerrit | Drew Walters proposed openstack/airship-treasuremap master: WIP: airskiff: Use global Airship charts https://review.openstack.org/635218 | 16:50 |
| *** jsun3 has joined #airshipit | 17:09 | |
| *** jsun3 has quit IRC | 17:17 | |
| *** dustinspecker has quit IRC | 18:02 | |
| openstackgerrit | Anthony Bellino proposed openstack/airship-treasuremap master: Secret rotation and validation https://review.openstack.org/635694 | 18:22 |
| openstackgerrit | Alexander Hughes proposed openstack/airship-pegleg master: [WIP] Update pki generation code to make certificate duration configurable https://review.openstack.org/639414 | 18:40 |
| *** jsun3 has joined #airshipit | 18:41 | |
| *** JosW has joined #airshipit | 18:43 | |
| *** JosW has left #airshipit | 18:43 | |
| openstackgerrit | Alexander Hughes proposed openstack/airship-pegleg master: Unwrap managed documents before linting https://review.openstack.org/638689 | 19:42 |
| *** georgk3 has quit IRC | 20:19 | |
| *** georgk33 has joined #airshipit | 20:19 | |
| openstackgerrit | Alexander Hughes proposed openstack/airship-pegleg master: [WIP] Update pki generation code to make certificate duration configurable https://review.openstack.org/639414 | 20:34 |
| openstackgerrit | Jared Miller proposed openstack/airship-maas master: Disable tgt service for maas 2.3 https://review.openstack.org/639432 | 20:35 |
| openstackgerrit | Rick Bartra proposed openstack/airship-divingbell master: Run Divingbell containers as unprivileged https://review.openstack.org/639435 | 20:48 |
| openstackgerrit | Stacey Fletcher proposed openstack/airship-pegleg master: Unwrap managed documents before linting https://review.openstack.org/638689 | 20:51 |
| openstackgerrit | Stacey Fletcher proposed openstack/airship-pegleg master: Unwrap managed documents before linting https://review.openstack.org/638689 | 20:51 |
| openstackgerrit | Rick Bartra proposed openstack/airship-divingbell master: Run Divingbell containers as unprivileged https://review.openstack.org/639435 | 20:55 |
| evgenyl | dwalt: Hi Drew, I have an uplift job for treasuremap failing (http://paste.openstack.org/show/746301/) so I was wondering if you can help me to understand if it can be possible related to http://paste.openstack.org/show/746301/ which comes with the latest uplift | 21:13 |
| evgenyl | dwalt: sorry, second link should be https://review.openstack.org/#/c/618585/ :) | 21:15 |
| evgenyl | dwalt: I also looked into the logs of airship-neutron-test pod, rally finished successfully there. | 21:17 |
| dwalt | evgenyl: Hey! It's possible. A timeout of 300, the new default value for Helm tests in Armada, is large enough for most tests. I have observed that the Neutron and Nova tests are a bit longer than average, so setting a timeout value for those might fix this. | 21:21 |
| dwalt | Does Helm history shows that the actual Helm test did not timeout? | 21:21 |
| evgenyl | dwalt: I've just looked into Pod's description, it finished at 19:53:33 and the error in Armada happened at 20:03:49. How do I find this info in helm? Is there some specific command that I can use? | 21:25 |
| dwalt | I think if you do helm history airship-neutron, I think it will indicate if the test itself timed out | 21:27 |
| dwalt | You'll probably see SUCCESS though, since you said the logs looked clean | 21:27 |
| dwalt | err, DEPLOYED, not SUCCESS | 21:28 |
| evgenyl | dwalt: Yes, it looks good http://paste.openstack.org/show/746307/ | 21:28 |
| dwalt | Yep, that looks good. We can try bumping the timeout value for the Nova and Neutron tests based on the previous upgrade timeout | 21:29 |
| dwalt | It looks like it used the default wait timeout, so that would be 900 | 21:30 |
| dwalt | I would image it doesn't require the whole value of 900, so we can probably tune that down a bit and be safe | 21:32 |
| evgenyl | I've just checked and init + ks-users + the test took 302 seconds, I'm wondering though, why does Tiller returns `pods \"airship-neutron-test\" already exists`with status `FAILURE`? | 21:35 |
| evgenyl | What does happen under the hood? | 21:35 |
| Nishant_ | evgenyl: Is this a GF deployment? | 21:37 |
| evgenyl | Nishant_: Yes, it is. | 21:38 |
| dwalt | evgenyl: I didn't see any Tiller messages in your paste. Where did you see the Tiller message? | 21:39 |
| Nishant_ | evgenyl: I have seen similar issue mostly during an upgrade scenario if the pre-upgrade delete hook for test is not enabled. However with automatic test pod deletion change in place that should be cared for already. | 21:42 |
| evgenyl | dwalt: Isn't this message in Armada logs a response from Tiller http://paste.openstack.org/show/746311/ ? | 21:43 |
| dwalt | evgenyl: Sorry! I didn't see the whole paste | 21:43 |
| dwalt | test pod deletion should be automated after this patch: https://review.openstack.org/629313 | 21:45 |
| dwalt | I'm assuming your uplift includes it | 21:45 |
| dwalt | but since there is no wait configuration for this chart, no test pods are getting deleted | 21:46 |
| Nishant_ | For GF it shouldn't matter as there would be no pods existing | 21:46 |
| dwalt | The recommended approach here would be to add the proper wait labels to the chart so that Helm tests do not fail on retries | 21:46 |
| Nishant_ | dwalt: Agreed. | 21:54 |
| Nishant_ | evgenyl: would be good to check in logs if the test pod was ever run earlier during deployment | 21:55 |
| evgenyl | Nishant_: Yes, it was run at 19:53:16, and there were three more attempts that failed with "already exists" error at 20:00, 20:01 and 20:03. | 22:04 |
| evgenyl | So the suggestion is to add both wait labels and also increase the timeout for the test, is this correct? | 22:05 |
| openstackgerrit | Rick Bartra proposed openstack/airship-divingbell master: Run Divingbell containers as unprivileged https://review.openstack.org/639435 | 22:06 |
| Nishant_ | evgenyl: I didn't follow as to why the test failed initially, if it was due to a timeout issue then maybe increasing the timeout would make sense | 22:19 |
| evgenyl | Nishant_: The assumption is this is due to a timeout, I checked the containers start and finish times, and it took about 302 seconds to execute airship-neutron-test, but I'm still wondering why Tiller does not show that it failed to deploy the release due to a timeout... | 22:21 |
| evgenyl | By the way, Nishant_ and dwalt, if I add wait.resources.labels, should I delete upgrade.pre.delete? https://github.com/openstack/airship-treasuremap/blob/master/global/software/charts/osh/openstack-compute-kit/neutron.yaml#L197-L201 | 22:23 |
| dwalt | evgenyl: Yes, as Armada can automatically delete test pods now using the wait.labels or wait.resources.labels | 22:24 |
| dwalt | also, if Helm does not show that the test timed out, it's very possible it only timed out on the Armada side | 22:24 |
| dwalt | In that case, adding a test timeout and the proper wait labels should be a good, permanent solution | 22:25 |
| Nishant_ | evgenyl: the existing upgrade.pre.delete hook in the chart is for `job` type, I am not sure if that is being taken care off in the automatic deletion | 22:35 |
| openstackgerrit | Rick Bartra proposed openstack/airship-divingbell master: Run Divingbell containers as unprivileged https://review.openstack.org/639435 | 22:41 |
| openstackgerrit | Evgeniy L proposed openstack/airship-treasuremap master: Uplift Armada and fix neutron configs https://review.openstack.org/639462 | 22:41 |
| openstackgerrit | Drew Walters proposed openstack/airship-treasuremap master: WIP: airskiff: Use global Airship charts https://review.openstack.org/635218 | 22:42 |
| evgenyl | Nishant_: good, point, I will leave it as is for now. | 22:42 |
| evgenyl | Nishant_: dwalt Can you please let me know if this makes sense? https://review.openstack.org/#/c/639462/1/global/software/charts/osh/openstack-compute-kit/neutron.yaml | 22:42 |
| Nishant_ | evgenyl: LGTM | 22:47 |
| evgenyl | Nishant_: dwalt Alright, thank you guys, really appreciate your help with that, I'm re-triggering the job and we will see if it helps :) | 22:49 |
| openstackgerrit | Michael Beaver proposed openstack/airship-armada master: [WIP] Move tiller into Armada POD https://review.openstack.org/632788 | 23:03 |
| *** sreejithp has quit IRC | 23:10 | |
| openstackgerrit | Georg Kunz proposed openstack/airship-in-a-bottle master: [WIP] Configuration for testing DPDK in multi-node AIAB https://review.openstack.org/634207 | 23:14 |
| *** michaelbeaver has quit IRC | 23:34 | |
| *** jesusaur has quit IRC | 23:51 | |
| *** jsun3 has quit IRC | 23:52 | |
| *** jsun3 has joined #airshipit | 23:52 | |
| *** DanCrank has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!