Monday, 2019-04-22

« Sunday, 2019-04-21 Index Tuesday, 2019-04-23 »
*** cheng1 has quit IRC03:57
*** kaspars__ has quit IRC06:43
*** pkaralis has joined #airshipit07:23
*** aojea has joined #airshipit07:58
*** cheng1 has joined #airshipit09:47
*** dims has joined #airshipit12:26
*** aaronsheffield has joined #airshipit12:53
*** bh526r has joined #airshipit12:57
*** Nishant_ has joined #airshipit13:06
*** mbeierl has joined #airshipit13:07
*** sthussey has joined #airshipit13:18
*** kranthikirang has joined #airshipit13:52
*** kranthikirang1 has joined #airshipit13:59
*** kranthikirang has quit IRC14:03
*** michael-beaver has joined #airshipit14:04
kranthikirang1@team, can you push this https://review.opendev.org/#/c/649155/?14:56
*** kaspars__ has joined #airshipit15:01
*** openstackgerrit has joined #airshipit16:19
openstackgerritEvgeniy L proposed airship/deckhand master: CI: Add Airskiff check  https://review.opendev.org/60077116:19
kranthikirang1@team, I just saw a stacktrace with armada container; http://paste.openstack.org/show/749602/16:21
kranthikirang1this is while running update_software; It waited for nova_vnc_proxy and timedout and after a min saw this16:21
openstackgerritMerged airship/pegleg master: zuul: Add python37 job  https://review.opendev.org/64064616:23
evgenylkranthikirang1: I have seen something similar when configuring PROXY env variables for Armada, unfortunately I don't remember exact details.16:23
kranthikirang1evgenyl: ok; you mean proxy_server as a parameter to charts? Since I didn't pass proxy values to armada pods16:24
evgenylkranthikirang1: I recall that in my case it was related to env variables and not to charts configuration. Sorry for not giving too much info on that, I encountered something similar a long time ago.16:27
kranthikirang1evgenyl: OK. So, there is some issue; Who can take a look? Any details we can collect?16:28
evgenylkranthikirang1: I ended up reverting my changes and trying to figure which of the parameters broke the container.16:28
evgenylkranthikirang1: in a backtrace I can see `google/protobuf`, it is used by Armada in grpc client, can you check what armada related config changes you have? And maybe reverting some of those.16:31
kranthikirang1evgenyl: honestly there are no configuration changes that I have applied for Armada in spec16:33
kranthikirang1I need one more help16:33
kranthikirang12019-04-22 16:31:35.992 9 ERROR armada.api [-] Failed to apply manifest: Exception deploying charts: ['nova', 'neutron']: armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['nova', 'neutron']16:33
kranthikirang1I see neutron-test takes a bit long and nova-test as well (when the pod comes up in genesis host). Tests are being passed but before that armada timeout happens. How can I disable the tests or increase the timeout; I understand for ucp components but not for osh components through armada16:34
openstackgerritEvgeniy L proposed airship/treasuremap master: Update docs, clarify the requirements on DNS names registration  https://review.opendev.org/65455916:38
openstackgerritMerged airship/treasuremap master: CI: Update repository references  https://review.opendev.org/65402516:40
openstackgerritDrew Walters proposed airship/shipyard master: CI: Add Airskiff check  https://review.opendev.org/65303516:40
openstackgerritEvgeniy L proposed airship/treasuremap master: Pipeline improvements for Airship Seaworthy  https://review.opendev.org/65427616:40
openstackgerritDrew Walters proposed airship/deckhand master: CI: Add Airskiff check  https://review.opendev.org/60077116:40
evgenylkranthikirang1: Here is an example on how to disable the tests https://review.opendev.org/#/c/633243/2/site/airskiff/software/charts/osh/openstack-compute-kit/neutron.yaml16:44
kranthikirang1evgenyl: Thank you16:45
openstackgerritMerged airship/maas master: Additional Rules for optional MAAS api only  https://review.opendev.org/64950616:56
openstackgerritEvgeniy L proposed airship/treasuremap master: Add metadata proxy shared secret for Nova and Neutron  https://review.opendev.org/63918317:00
openstackgerritKaspars Skels proposed airship/treasuremap master: Change default number of l3 agents from 5 to 3  https://review.opendev.org/63917917:02
openstackgerritKaspars Skels proposed airship/treasuremap master: Integrate support for MAAS HA and VIPs  https://review.opendev.org/65043217:08
openstackgerritKaspars Skels proposed airship/maas master: Support for MAAS URL overrides  https://review.opendev.org/65385317:10
openstackgerritKaspars Skels proposed airship/maas master: Support for MAAS URL overrides  https://review.opendev.org/65385317:13
openstackgerritEvgeniy L proposed airship/treasuremap master: Fix chart repo urls from openstack to opendev  https://review.opendev.org/65457917:18
openstackgerritKaspars Skels proposed airship/maas master: Support for MAAS URL overrides  https://review.opendev.org/65385317:44
openstackgerritKaspars Skels proposed airship/maas master: Support for MAAS URL overrides  https://review.opendev.org/65385317:46
openstackgerritKaspars Skels proposed airship/maas master: Support for MAAS URL overrides  https://review.opendev.org/65385317:52
openstackgerritKaspars Skels proposed airship/treasuremap master: Enable nested virtualization by default  https://review.opendev.org/65213917:55
openstackgerritKaspars Skels proposed airship/maas master: Support for MAAS URL overrides  https://review.opendev.org/65385318:01
openstackgerritKaspars Skels proposed airship/maas master: Support for MAAS URL overrides  https://review.opendev.org/65385318:01
openstackgerritMatt McEuen proposed airship/promenade master: Add optional routable_ip to promjoin  https://review.opendev.org/65016818:06
openstackgerritAnthony Bellino proposed airship/shipyard master: Update ha_celery policy pattern  https://review.opendev.org/65459818:17
*** aojea has quit IRC18:31
openstackgerritEvgeniy L proposed airship/treasuremap master: Update links in the docs to refer to opendev  https://review.opendev.org/65460218:33
*** bh526r has quit IRC18:37
openstackgerritDrew Walters proposed airship/deckhand master: CI: Update OSH relative paths for OpenDev  https://review.opendev.org/65460418:40
openstackgerritEvgeniy L proposed airship/in-a-bottle master: Update links in the docs to refer to opendev  https://review.opendev.org/65460518:42
openstackgerritIan Pittwood proposed airship/spyglass master: Refactors CLI to match style of Airship CLIs  https://review.opendev.org/64945218:46
openstackgerritDrew Walters proposed airship/deckhand master: CI: Add Airskiff check  https://review.opendev.org/60077118:46
openstackgerritIan Pittwood proposed airship/spyglass master: PEP-8 code styling and linting  https://review.opendev.org/64969218:46
openstackgerritDrew Walters proposed airship/deckhand master: CI: Add Airskiff check  https://review.opendev.org/60077118:48
openstackgerritDrew Walters proposed airship/deckhand master: CI: Add Airskiff check  https://review.opendev.org/60077118:50
openstackgerritMerged airship/treasuremap master: Integrate support for MAAS HA and VIPs  https://review.opendev.org/65043219:00
openstackgerritDrew Walters proposed airship/treasuremap master: Update Zuul refs for airship-treasuremap to treasuremap  https://review.opendev.org/65425119:01
openstackgerritMatt McEuen proposed airship/promenade master: Add optional routable_ip to promjoin  https://review.opendev.org/65016819:07
openstackgerritEvgeniy L proposed airship/treasuremap master: Pipeline improvements for Airship Seaworthy  https://review.opendev.org/65427619:08
openstackgerritDrew Walters proposed airship/treasuremap master: global: Uplift helm-toolkit  https://review.opendev.org/64522619:12
openstackgerritDrew Walters proposed airship/treasuremap master: airskiff: Use global Airship charts  https://review.opendev.org/63521819:12
openstackgerritKaspars Skels proposed airship/maas master: Support for MAAS URL overrides  https://review.opendev.org/65385319:13
openstackgerritEvgeniy L proposed airship/treasuremap master: Fix chart repo urls from openstack to opendev  https://review.opendev.org/65457919:13
openstackgerritEvgeniy L proposed airship/treasuremap master: Update links in the docs to refer to opendev  https://review.opendev.org/65460219:13
openstackgerritKaspars Skels proposed airship/maas master: Support for MAAS URL overrides  https://review.opendev.org/65385319:15
openstackgerritLev Morgan proposed airship/pegleg master: Improve files.write, add decrypt output to file  https://review.opendev.org/65349319:30
*** sgrasley has joined #airshipit19:36
openstackgerritKaspars Skels proposed airship/treasuremap master: Uplift tiller/help to 2.13.1 and Airship/OSH charts  https://review.opendev.org/65396319:59
openstackgerritKaspars Skels proposed airship/treasuremap master: Support latest MAAS in Airsloop  https://review.opendev.org/65385720:04
*** mbeierl has quit IRC20:06
openstackgerritKaspars Skels proposed airship/treasuremap master: Update Zuul refs for airship-treasuremap to treasuremap  https://review.opendev.org/65425120:09
openstackgerritMatt McEuen proposed airship/promenade master: CoreDNS: remove external liveness dependency  https://review.opendev.org/65462520:15
kranthikirang1kaspars__, evgenyl: at last I have got Successfully performed update_software20:20
kranthikirang1+ echo -e '\n'20:20
kranthikirang1+ break20:20
kranthikirang1+ [[ Complete == \C\o\m\p\l\e\t\e ]]20:20
kranthikirang1+ exit 020:20
kaspars__!! meaning openstack is up, etc? you should be able to reach Horzion at dashboard.<domain> you chose20:21
openstackkaspars__: Error: "!" is not a valid command.20:21
openstackgerritDrew Walters proposed airship/deckhand master: CI: Add Airskiff check  https://review.opendev.org/60077120:22
openstackgerritDrew Walters proposed airship/deckhand master: CI: Update OSH relative paths for OpenDev  https://review.opendev.org/65460420:23
openstackgerritDrew Walters proposed airship/deckhand master: CI: Add Airskiff check  https://review.opendev.org/60077120:23
kranthikirang1kaspars__: yes, I can open the horizon dashboard with the dashboard.<domain>20:28
kaspars__congrats! :)20:29
kranthikirang1only thing is I can't find the horizon admin username :) I don't see that in secrets/passphrases folder20:29
kaspars__admin/password12320:29
kranthikirang1aha20:29
kranthikirang1perhaps we should update that in documentation20:29
kaspars__it's under servers osh_keystone_password or similar20:29
kranthikirang1osh_keystone_admin_password?20:29
kaspars__yes20:29
kaspars__if you updated it - use that password20:29
kaspars__domain -> default20:30
kranthikirang1perfect20:30
kranthikirang1Able to get in20:30
kranthikirang1thanks a lot; I will update the documentation for tenent-ceph osd as well and this20:30
kaspars__great!! here is also a version of CLI you can use https://opendev.org/airship/treasuremap/src/branch/master/tools/openstack20:30
kranthikirang1and give the information for you to review20:30
kaspars__yes - I think for all the proxy stuff - might be good to have whole chapter of what needs to be done20:31
kranthikirang1OK; cool; thanks for the CLI ref. However we can download that from dashboard as well20:31
kranthikirang1for proxy sure; I have all the details required; Except for glance-test since that requires giving proxy values to rally test pod as well20:32
kranthikirang1for now I have hosted internally a web server with cirros image and updated specs to use that20:32
kaspars__yes - that is def a good internal way of doing it20:33
kaspars__just for my own curiosity - did you deploy the whole 6 node reference?20:36
kaspars__3 control + 3 computes?20:36
kranthikirang1not really; I have total 6 nodes; So used one for genesis (so 4 CP) + 2  computes20:37
*** mbeierl has joined #airshipit20:37
kranthikirang1I have used gen9 HP; I see after some load eventually executing the containers taking a bit time; especially coming towards test pods20:37
kranthikirang1I am able to run CLI commands as well20:38
kaspars__pefect! the requirement erally is 3 control - genesis is 1 of the control nodes20:38
kranthikirang1Happy that I know the deployment procedure and tweaks to make this work in different h/w20:38
kaspars__so normally our seaworthy ref is genesis + 2 additional control, and then 3 computes20:38
openstackgerritMerged airship/promenade master: CoreDNS: remove external liveness dependency  https://review.opendev.org/65462520:38
kaspars__3 computes mostly so that tenant ceph has 3 domains to replicate towards20:39
kranthikirang1oh; got it20:39
kaspars__bit surprised it didn't complain about replication when you deployed stuff20:39
kaspars__awesome though! also getting it to work via proxy takes quite a bit of digging into details!20:40
kranthikirang1since ceph-mon uses different ports for ucp and tenant-ceph and nodes are labeled with tenan-ceph-mon=enabled it doesn't matter20:41
openstackgerritMatthew Carter proposed airship/shipyard master: Add validation support for deployment-version document  https://review.opendev.org/65384720:42
kranthikirang1Only thing is these HP gen9 hosts are having 56 CPU and 256 GB memory and still I am seeing slower processing while deploying workloads20:42
openstackgerritPRATEEK REDDY DODDA proposed airship/armada master: [WIP] Implement Security Context for Armada  https://review.opendev.org/63920720:43
kaspars__yes - both control planes of ceph run on the same control nodes - but the OSDs don't - so actual ceph tenant storage (OSDs) will be running only on compute nodes.20:43
kranthikirang1ok; but I guess only ceph-mon requires a quoram; not osd20:44
kaspars__right - OSDs mostly for replicating data20:44
kranthikirang1failure_domain by default is set to host?20:44
kaspars__usually we require 3x replication on 3 different domains (servers)20:44
kaspars__you may check if everything is healthy by doing20:45
kaspars__"sudo kubectl exec -it -n tenant-ceph ceph-mon-znkkx -- ceph -s"20:45
kranthikirang1agreed; I thought about that but after seeing workload sloppiness thought better to use 4 CP20:45
openstackgerritPRATEEK REDDY DODDA proposed airship/armada master: [WIP] Implement Security Context for Armada  https://review.opendev.org/63920720:47
kaspars__interesting observation.. generally airship seaworthy servers are also very similar20:47
kaspars__56vCPU and 256G of memory20:47
kaspars__they are DELL servers though20:47
kaspars__but sure - adding more control nodes is very much supported and can be done if required20:47
kranthikirang1health: HEALTH_WARN20:47
kranthikirang1            2/3549 objects misplaced (0.056%)20:47
kranthikirang1            Degraded data redundancy: 1181/3549 objects degraded (33.277%), 11 pgs degraded, 71 pgs undersized20:47
openstackgerritPRATEEK REDDY DODDA proposed airship/armada master: [WIP] Implement Security Context for Armada  https://review.opendev.org/63920720:48
kaspars__yes :)20:48
kaspars__you are missing 1/3 of data redundancy20:48
kranthikirang1mon <node-1-genesis> is low on available space20:48
kranthikirang1last thing I didn't understand; how the mon is low on available space?20:48
kranthikirang1:)20:48
kaspars__it may be using the same disk as operating system20:48
kaspars__you can just check if your regular disk for OS might be filling up20:49
kaspars__in other words for tenant ceph - would be nice if you can find extra server20:49
kaspars__otherwise you may update your replication to only be 2x20:49
openstackgerritMatthew Carter proposed airship/shipyard master: Add validation support for deployment-version document  https://review.opendev.org/65384720:49
kranthikirang1well, I guess that's true right; we keep mon data in /var/lib/openstack-helm/ceph/mon dir20:50
kaspars__https://opendev.org/openstack/openstack-helm-infra/src/branch/master/ceph-client/values.yaml#L25220:50
kaspars__see replication section..20:50
kranthikirang1and giving 30GB for / and keeping docker images there; we will fill the space every quickly right?20:51
openstackgerritEvgeniy L proposed airship/maas master: Support for MAAS URL overrides  https://review.opendev.org/65385320:51
kaspars__docker imgaes are stored under /var/lib20:51
kranthikirang1yeah, in order to correct this I will have to redeploy20:51
kaspars__that has much more space..20:51
kaspars__so if you gave the other paritions similarly to our reference it shoul dbe good20:52
kaspars__root / is not used for much other then OS itself20:52
*** v1k0d3n has quit IRC20:52
kranthikirang1(e.g., 30G for /, 1G for /boot, 100G for /var/log, and all remaining storage for /var). Note that the volume size syntax looking like >300g in Drydock means that all remaining disk space is allocated to this volume, and that volume needs to be at least 300G in size.20:52
kranthikirang1Oh /var20:52
kranthikirang1OK20:52
kaspars__https://www.irccloud.com/pastebin/PbZBOX1s/20:53
kranthikirang1I guess in order to correct this; I have to deploy the env once again by correcting site specs to use 3 CP and 3 DP20:53
kranthikirang1sorry, I can't open irccloud behind proxy :)20:53
*** michael-beaver has quit IRC20:54
*** v1k0d3n has joined #airshipit20:54
kaspars__yeah - I guess the space issue is only on genesis, right?20:54
kaspars__as other nodes are automatically created with the correct paritions?20:54
kaspars__yes - you would need to re-deploy..20:54
kranthikirang1yeah20:54
kaspars__though probably a good idea to move to 3 and 320:54
kaspars__as you will get more computes out of it as well20:54
*** dwalt has quit IRC20:55
kranthikirang1yeah, Will do that20:55
*** sthussey has quit IRC20:55
kaspars__https://www.irccloud.com/pastebin/5jUD3q4J/20:56
kaspars__fyi - here is what I have on my genesis (to match other control nodes)20:56
*** michael-beaver has joined #airshipit20:57
kranthikirang1can you put it in paste.openstack.org?20:57
*** dwalt has joined #airshipit20:57
*** v1k0d3n has quit IRC20:59
kaspars__http://paste.openstack.org/show/749614/21:00
kranthikirang1kaspars__: In three CP two would be primary and 1 secondary?21:00
kranthikirang1thanks for the genesis conf21:00
kaspars__all primary21:00
kranthikirang1ok21:01
kaspars__secondary is if you are adding 4th or more21:01
kaspars__you should be able to match it exactly as reference site21:01
kranthikirang1guess, When I redeployed genesis I didn't make the partitions proper then21:01
kranthikirang1ok21:01
kranthikirang1changing in baremetal/nodes.yaml would be sufficient right?21:02
kranthikirang1node to correct profile21:02
kaspars__yes - that should be it21:02
kaspars__https://opendev.org/airship/treasuremap/src/branch/master/site/airship-seaworthy/baremetal/nodes.yaml21:02
kranthikirang1perfect; Will redeploy tomorrow and let you know21:02
*** sthussey has joined #airshipit21:02
*** dwalt has quit IRC21:02
*** mnaser has quit IRC21:04
*** sthussey has quit IRC21:07
*** kranthikirang1 has quit IRC21:07
*** dwalt has joined #airshipit21:08
*** v1k0d3n has joined #airshipit21:08
*** mnaser has joined #airshipit21:08
*** sthussey has joined #airshipit21:09
openstackgerritLev Morgan proposed airship/pegleg master: Added cleartext option to passphrase generation  https://review.opendev.org/64501721:13
openstackgerritKaspars Skels proposed airship/maas master: Support for MAAS URL overrides  https://review.opendev.org/65385321:15
openstackgerritMerged airship/armada master: Introduce v2 docs  https://review.opendev.org/64824621:18
openstackgerritEvgeniy L proposed airship/maas master: Support for MAAS URL overrides  https://review.opendev.org/65385321:18
*** mbeierl has quit IRC21:24
openstackgerritSean Eagan proposed airship/armada master: [WIP]: Improve wait API and semantics in v2 docs  https://review.opendev.org/63644021:27
openstackgerritMerged airship/treasuremap master: Update Zuul refs for airship-treasuremap to treasuremap  https://review.opendev.org/65425121:29
openstackgerritKaspars Skels proposed airship/treasuremap master: Uplift tiller/help to 2.13.1 and Airship/OSH charts  https://review.opendev.org/65396321:39
openstackgerritKaspars Skels proposed airship/maas master: Support for MAAS URL overrides  https://review.opendev.org/65385321:41
openstackgerritEvgeniy L proposed airship/maas master: Support for MAAS URL overrides  https://review.opendev.org/65385321:43
openstackgerritDimitrios Markou proposed airship/treasuremap master: [WIP] Migrate single node manifests to treasuremap  https://review.opendev.org/65454821:43
*** mbeierl has joined #airshipit21:47
openstackgerritKaspars Skels proposed airship/treasuremap master: Support latest MAAS in Airsloop  https://review.opendev.org/65385721:48
*** mbeierl has quit IRC21:51
*** irclogbot_3 has quit IRC21:59
*** irclogbot_2 has joined #airshipit22:01
*** mbeierl has joined #airshipit22:05
*** sthussey_ has joined #airshipit22:05
*** sthussey has quit IRC22:07
*** sthussey_ is now known as sthussey22:07
*** mbeierl has quit IRC22:10
*** kinrui has joined #airshipit22:14
*** irclogbot_2 has quit IRC22:16
*** pkaralis has quit IRC22:16
*** gagehugo has quit IRC22:16
*** fungi has quit IRC22:16
*** irclogbot_2 has joined #airshipit22:17
*** kinrui is now known as fungi22:19
*** irclogbot_2 has quit IRC22:19
*** irclogbot_2 has joined #airshipit22:19
*** mbeierl has joined #airshipit22:32
*** mbeierl has quit IRC22:37
*** mbeierl has joined #airshipit22:42
*** mbeierl has quit IRC22:46
*** aaronsheffield has quit IRC23:19
*** sthussey has quit IRC23:28
openstackgerritMerged airship/treasuremap master: Enable nested virtualization by default  https://review.opendev.org/65213923:34
openstackgerritMerged airship/treasuremap master: Change default number of l3 agents from 5 to 3  https://review.opendev.org/63917923:34
« Sunday, 2019-04-21 Index Tuesday, 2019-04-23 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!