| airship-irc-bot1 | <rishabh.k.jain> Hi Team, please review https://review.opendev.org/c/airship/airshipctl/+/772425 | 01:17 |
|---|---|---|
| *** uzumaki has joined #airshipit | 11:38 | |
| *** raildo has joined #airshipit | 13:34 | |
| *** uzumaki has quit IRC | 13:40 | |
| *** thansen has quit IRC | 14:22 | |
| *** thansen has joined #airshipit | 14:22 | |
| airship-irc-bot1 | <sirishagopigiri> Hi Team, I would like to request some reviews on these PSs: https://review.opendev.org/c/airship/hostconfig-operator/+/775332 https://review.opendev.org/c/airship/hostconfig-operator/+/773389 https://review.opendev.org/c/airship/airshipctl/+/777930 Thank you in advance! | 14:52 |
| airship-irc-bot1 | <sirajudeen.yasin> Team, can i request some reviews for these PSs https://review.opendev.org/c/airship/treasuremap/+/777797 => Treasuremap zuul specific issues addressed https://review.opendev.org/c/airship/airshipctl/+/778124 => collect only current apache2 log and not all the archived logs ( gz ). on CI env with static VM, it goes up to several MBs if not cleaned up https://review.opendev.org/c/airship/airshipctl/+/769617 => wait logic for | 14:56 |
| airship-irc-bot1 | tigerastatus updated as per review comments | 14:56 |
| *** uzumaki has joined #airshipit | 14:58 | |
| *** roman_g has joined #airshipit | 15:04 | |
| *** uzumaki has quit IRC | 16:08 | |
| *** uzumaki has joined #airshipit | 16:21 | |
| airship-irc-bot1 | <ih616h> Good morning team. I've been trying to deploy the SIP samples, but I'm getting an error because the sample BMHs for SIP reference non-existent secrets: https://github.com/airshipit/sip/blob/master/config/samples/bmh/bmh.yaml#L58 | 16:43 |
| airship-irc-bot1 | <ih616h> I can probably just remove those BMHs, but does anyone know why they're there in the first place? | 16:44 |
| airship-irc-bot1 | <alexander.hughes> https://review.opendev.org/c/airship/sip/+/777693 | 16:44 |
| airship-irc-bot1 | <ih616h> :slightly_smiling_face: | 16:44 |
| airship-irc-bot1 | <ih616h> thanks! | 16:44 |
| airship-irc-bot1 | <alexander.hughes> I honestly thought that was merged already :slightly_smiling_face: | 16:45 |
| airship-irc-bot1 | <ih616h> any idea why it isn't merging? Looks like Zuul doesn't want to run gates | 16:47 |
| airship-irc-bot1 | <alexander.hughes> I was just wondering the same thing | 16:47 |
| airship-irc-bot1 | <alexander.hughes> I'm going to rebase the change and see what happens | 16:48 |
| airship-irc-bot1 | <alexander.hughes> patch passed the checks, it's going through gates now to merge. hopefully will be merged in next ~20 minutes | 17:13 |
| airship-irc-bot1 | <ih616h> awesome, thanks again | 17:13 |
| airship-irc-bot1 | <alexander.hughes> my pleasure | 17:14 |
| airship-irc-bot1 | <steven.fitzpatrick> Hey all, anyone familiar with how airshipctl phases run, and specifically how they wait for resources to become `AllCurrent`? In [this PS](https://review.opendev.org/c/airship/treasuremap/+/769096) I'm adding this lma-infra composite to the initinfra-target phase. I ran the phase via `./tools/deployment/31_deploy_initinfra_target_node.sh`, and it seemed to deploy everything successfully: ```steven@airship-3:~$ kc get | 17:29 |
| airship-irc-bot1 | helmrepository -A NAMESPACE NAME URL READY STATUS AGE lma-infra banzaicloud https://kubernetes-charts.banzaicloud.com True Fetched revision: 2021-03-02T16:44:02Z 7h38m lma-infra prometheus-community https://prometheus-community.github.io/helm-charts True Fetched revision: | 17:29 |
| airship-irc-bot1 | 2020-09-27T05:31:40.762116-04:00 7h38m steven@airship-3:~$ kc get hr -n lma-infra NAME READY STATUS AGE kube-prometheus-stack True Release reconciliation succeeded 7h29m logging-operator True Release reconciliation succeeded 7h29m logging-operator-logging True Release reconciliation succeeded 7h29m``` However, airshipctl never picked up on these items, | 17:29 |
| airship-irc-bot1 | eventually timing out the phase. The last I can see in the `--debug` output is that these lma resources are `Unknown`. Strangely, if I delete the lma resources and run the phase again, everything becomes `Current` quite quickly. Looking to better understand what's going on. Logs attached. Thanks! | 17:29 |
| airship-irc-bot1 | <sirajudeen.yasin> @steven.fitzpatrick,this might be something related to what we had in tigera-operator | 17:30 |
| airship-irc-bot1 | <sirajudeen.yasin> one of the resource of tigera operator ( installations ) did not implement status ( condition ) , due to which the wait never met the condition and it fails.. so we had to implement a special nowait phase executor to overcome this situation.. | 17:31 |
| airship-irc-bot1 | <sirajudeen.yasin> can you please check the status field of your CR.. if it has conditions | 17:32 |
| airship-irc-bot1 | <steven.fitzpatrick> Ahhh, okay. Sure | 17:32 |
| airship-irc-bot1 | <sirajudeen.yasin> but now we have better ways to overcome this .. may be generic container is one of the options discussed.. i will wait for others suggestions | 17:35 |
| airship-irc-bot1 | <kk6740> @steven.fitzpatrick can u try to isolate the resources that are actually not reaching current state in a first run? | 17:39 |
| airship-irc-bot1 | <steven.fitzpatrick> Yeah, you're right. Most of the prometheus / logging operator CRs do not seem to have Status or Condition :thinking_face: Why would the phase complete nominally when I run it manually though? | 17:39 |
| airship-irc-bot1 | <steven.fitzpatrick> @kk6740 do you know if this is summarized someplace? I don't see a "These resources were not Current" kind of message in the log, or in the phase configmap | 17:41 |
| airship-irc-bot1 | <steven.fitzpatrick> Based on what I saw, the resources added in my PS were last in 'Unknown', in the first run of the phase | 17:41 |
| airship-irc-bot1 | <kk6740> we use external lib to do all the magic, unfortunately it is not as verbose as we would like to about which resources dind’t reach the state | 17:42 |
| airship-irc-bot1 | <kk6740> it logs the changes basically, so if resource status doesn’t change, it will not belogged | 17:42 |
| airship-irc-bot1 | <kk6740> but, to start, i would create a new phase | 17:43 |
| airship-irc-bot1 | <kk6740> for debugging | 17:43 |
| airship-irc-bot1 | <kk6740> so that u throw out initinfra original items | 17:43 |
| airship-irc-bot1 | <steven.fitzpatrick> I had it like that before actually - using an 'lma-infra' phase that ran after workload-target, everything deployed normally. Even when I re-run initinfra-target now, it seem to work just fine | 17:46 |
| airship-irc-bot1 | <kk6740> hm | 17:49 |
| airship-irc-bot1 | <kk6740> i don’t know internals | 17:49 |
| airship-irc-bot1 | <kk6740> of what is there in lma-infra | 17:49 |
| airship-irc-bot1 | <kk6740> are there helm charts? | 17:49 |
| airship-irc-bot1 | <steven.fitzpatrick> Yeah, it is 2 helmrepositories and then 3 helmreleases | 17:50 |
| airship-irc-bot1 | <steven.fitzpatrick> Those are explicit in the treasuremap documents. Those helmreleases deploy other things, CRDs & CRs, deployements, etc | 17:52 |
| airship-irc-bot1 | <kk6740> so basically, if you separate this into two phases: 1. deploys operator && operator CRs 2. deploys operators CRS it works ? | 17:55 |
| airship-irc-bot1 | <steven.fitzpatrick> In this case it isn't so easy to separate, since the HelmRelease is creating the CRDs + deploying some CRs as part of the release | 17:58 |
| airship-irc-bot1 | <steven.fitzpatrick> Also, check out that manual-rerun.log file. You can see I delete the helmreleases and CRDS, then run the phase again manually. It's able to create everything successfully and end normally. | 18:00 |
| airship-irc-bot1 | <steven.fitzpatrick> Just not while the playbook was running `./tools/deployment/31_deploy_initinfra_target_node.sh` ? | 18:00 |
| airship-irc-bot1 | <kk6740> @steven.fitzpatrick sorry for misunderstanding here. I mean separate helm operator and helm operator CRD deployments, from helm operator CRs | 18:10 |
| airship-irc-bot1 | <kk6740> because right now they are piled up in the same phase | 18:10 |
| airship-irc-bot1 | <steven.fitzpatrick> Oh, I see what you mean. Because `flux-helm` composite and mine are installed in the same phase, right? | 18:11 |
| airship-irc-bot1 | <kk6740> yes | 18:12 |
| airship-irc-bot1 | <kk6740> i am not sure that that will help | 18:12 |
| airship-irc-bot1 | <kk6740> but it would be first step for me to help investigate further | 18:12 |
| airship-irc-bot1 | <steven.fitzpatrick> Sure, let me arrange that | 18:16 |
| airship-irc-bot1 | <steven.fitzpatrick> @kk6740 might take a little while. I will ping you when I have another update | 19:12 |
| airship-irc-bot1 | <kk6740> sure | 19:12 |
| airship-irc-bot1 | <kk6740> :+1: | 19:12 |
| airship-irc-bot1 | <steven.fitzpatrick> @sirajudeen.yasin Could you set up a jenkins build for treasuremap using my PS? https://review.opendev.org/c/airship/treasuremap/+/769096/ | 19:13 |
| airship-irc-bot1 | <steven.fitzpatrick> While I try to isolate the issue in my environment, maybe we can see if it's reproducible in jenkins | 19:13 |
| airship-irc-bot1 | <sirajudeen.yasin> sure @steven.fitzpatrick, I will update if i can run a build with your PS | 19:14 |
| airship-irc-bot1 | <steven.fitzpatrick> tyvm :+1: | 19:15 |
| airship-irc-bot1 | <steven.fitzpatrick> @kk6740 Watching this jenkins console output, it seems to be stuck at the exact same place as in my build. I wonder if it could be due to these ```helmrelease.helm.toolkit.fluxcd.io/kube-prometheus-stack created helmrelease.helm.toolkit.fluxcd.io/logging-operator created helmrelease.helm.toolkit.fluxcd.io/logging-operator-logging created``` being created before | 20:30 |
| airship-irc-bot1 | ```customresourcedefinition.apiextensions.k8s.io/helmreleases.helm.toolkit.fluxcd.io is Current: CRD is established``` I wish I could `kubectl get helmrelease -n lma-infra` on this jenkins site, because if it's like my lab then those HR are deployed and Ready, just not recognized by airshipctl. I'm getting ready to push the PS with the separated phases | 20:30 |
| *** raildo_ has joined #airshipit | 21:53 | |
| *** raildo has quit IRC | 21:55 | |
| *** raildo_ has quit IRC | 22:01 | |
| airship-irc-bot1 | <steven.fitzpatrick> With a separate phase, there is no issue: https://jenkins.nc.opensource.att.com/job/development/job/Treasuremap/job/Treasuremapv2-AirshipctlKnownState/129/console | 22:11 |
| airship-irc-bot1 | <steven.fitzpatrick> However, we would like to include lma-infra in the initinfra phase, since it is infra after all | 22:11 |
| *** roman_g has quit IRC | 22:19 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!