Tuesday, 2019-11-19

openstackgerritGary W. Smith proposed airship/airshipui master: Upgrade to octant 0.9.1  https://review.opendev.org/69485800:04
*** alexanderhughes has quit IRC00:08
*** nishantkr has quit IRC00:28
openstackgerritMerged airship/airshipctl master: Update README with a more accurate project description  https://review.opendev.org/69442600:46
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Verify Docker Default apparmor for Openstack Utility Container.  https://review.opendev.org/69438501:00
openstackgerritAlan Meadows proposed airship/airshipctl master: Correct README link to developer documentation  https://review.opendev.org/69486501:05
*** redrobot has quit IRC01:07
*** jamesgu has quit IRC01:53
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Verify Docker Default apparmor for Openstack Utility Container.  https://review.opendev.org/69438502:05
*** roman_g has quit IRC02:33
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Verify Docker Default apparmor for Openstack Utility Container.  https://review.opendev.org/69438502:37
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Verify Docker Default apparmor for Openstack Utility Container.  https://review.opendev.org/69438503:23
openstackgerritGary W. Smith proposed airship/airshipui master: Add argo-ui as an iframe  https://review.opendev.org/69489603:34
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Verify Docker Default apparmor for Openstack Utility Container.  https://review.opendev.org/69438503:48
openstackgerritGary W. Smith proposed airship/airshipui master: Add argo-ui as an iframe  https://review.opendev.org/69489604:01
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Verify Docker Default apparmor for Openstack Utility Container.  https://review.opendev.org/69438504:16
*** openstackstatus has joined #airshipit06:08
*** ChanServ sets mode: +v openstackstatus06:08
*** timClicks has quit IRC06:20
*** tumtarm has joined #airshipit06:21
openstackgerritRoman Gorshunov proposed airship/airshipctl master: Minor README formatting fix  https://review.opendev.org/69491006:42
openstackgerritRoman Gorshunov proposed airship/airshipctl master: Minor README formatting fix  https://review.opendev.org/69491006:46
openstackgerritRoman Gorshunov proposed airship/airshipctl master: Minor README formatting fixes  https://review.opendev.org/69491006:49
openstackgerritRoman Gorshunov proposed airship/airshipctl master: Minor README formatting fixes  https://review.opendev.org/69491006:52
*** dpawlik has joined #airshipit07:03
openstackgerritMerged airship/airshipctl master: Correct README link to developer documentation  https://review.opendev.org/69486507:05
openstackgerritRoman Gorshunov proposed airship/airshipctl master: Minor README formatting fixes  https://review.opendev.org/69491007:16
openstackgerritRoman Gorshunov proposed airship/airshipctl master: Minor README formatting fixes  https://review.opendev.org/69491007:20
*** tumtarm has quit IRC07:39
*** dpawlik has quit IRC07:43
*** pgaxatte has joined #airshipit07:44
*** dpawlik has joined #airshipit07:45
*** tumtarm has joined #airshipit07:52
*** tumtarm has quit IRC07:56
*** kskels has quit IRC08:02
*** tumtarm has joined #airshipit08:22
*** tumtarm has quit IRC08:26
*** tumtarm has joined #airshipit08:39
*** tumtarm has quit IRC08:59
*** tumtarm has joined #airshipit09:31
*** tumtarm has quit IRC09:33
*** dpawlik has quit IRC10:45
*** tumtarm has joined #airshipit10:59
*** pgaxatte has quit IRC11:03
*** tumtarm has quit IRC11:03
*** dpawlik has joined #airshipit11:30
*** roman_g has joined #airshipit11:30
*** tumtarm has joined #airshipit11:51
*** roman_g has quit IRC11:53
*** roman_g has joined #airshipit11:53
*** tumtarm has quit IRC11:54
*** tumtarm has joined #airshipit12:02
*** tumtarm has quit IRC12:17
*** dpawlik has quit IRC12:28
*** dpawlik has joined #airshipit12:31
*** roman_g has quit IRC12:37
*** roman_g has joined #airshipit12:39
*** tumtarm has joined #airshipit12:39
*** tumtarm has quit IRC12:50
*** pgaxatte has joined #airshipit12:56
*** alexanderhughes has joined #airshipit13:16
*** hiregoudar has joined #airshipit13:24
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Verify Docker Default apparmor for Openstack Utility Container.  https://review.opendev.org/69438513:29
*** hiregoudar has quit IRC13:44
*** bh526r has joined #airshipit13:55
*** Madhumati has joined #airshipit14:00
*** uzumaki has joined #airshipit14:01
*** SRao has joined #airshipit14:02
*** rajat143 has joined #airshipit14:04
*** rajat143 has quit IRC14:05
*** RaviWaghmare has joined #airshipit14:07
*** rajat143 has joined #airshipit14:08
*** pgaxatte has quit IRC14:15
*** rajat143 has quit IRC14:18
*** pgaxatte has joined #airshipit14:19
*** aaronsheffield has joined #airshipit14:21
*** rajat143 has joined #airshipit14:21
*** rajat143 has quit IRC14:23
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Verify Docker Default apparmor for Openstack Utility Container.  https://review.opendev.org/69438514:25
*** ab2434_ has joined #airshipit14:35
*** pgaxatte has quit IRC14:35
*** GarimaM has joined #airshipit14:36
*** uzumaki has quit IRC14:36
openstackgerritMerged airship/airshipctl master: Add `make tidy` to check for up-to-date go.mod  https://review.opendev.org/69325014:37
openstackgerritMerged airship/airshipctl master: Bump kubernetes 1.16.3 as a main dependency  https://review.opendev.org/69457214:37
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Verify Docker Default apparmor for Openstack Utility Container.  https://review.opendev.org/69438514:41
*** roman_g has quit IRC14:51
dwaltHey team! IRC meeting begins at the top of the hour. Here is the etherpad: https://etherpad.openstack.org/p/airship-meeting-2019-11-1914:54
openstackgerritRahul Khiyani proposed airship/porthole master: Adjust code to project conventions  https://review.opendev.org/69140014:56
*** spilla_ has joined #airshipit14:56
*** nishantkr has joined #airshipit14:57
dwalt#startmeeting airship15:00
openstackMeeting started Tue Nov 19 15:00:21 2019 UTC and is due to finish in 60 minutes.  The chair is dwalt. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: airship)"15:00
openstackThe meeting name has been set to 'airship'15:00
*** Madhumati has quit IRC15:00
dwalthello all! Agenda is here: https://etherpad.openstack.org/p/airship-meeting-2019-11-1915:00
dwaltAs expected, the crowd will probably be light. We have quite a few team members at Kubecon, where it is only 07:00!15:01
*** Madhumati has joined #airshipit15:01
*** ian-pittwood has joined #airshipit15:02
ian-pittwoodo/15:02
dwalto/ ian-pitwood15:03
*** pgaxatte has joined #airshipit15:03
dwaltit looks like nothing got added, so we can just proceed with the announcements for posterity and close15:03
seaneagano/15:03
dwalto/ seaneagan15:04
dwalt#topic annoncements and roundtable15:04
nishantkro/15:04
*** openstack changes topic to "annoncements and roundtable (Meeting topic: airship)"15:04
dwaltJust a reminder, the meeting time has moved an hour later for DST in the states. That's 9:00 CST and 15:00 UTC15:04
*** AshuKumar has joined #airshipit15:04
dwalt(right now)15:04
*** diga has joined #airshipit15:05
alexanderhughesdo you know if the meeting times have been updated for all the events - SIGs especially?15:05
dwaltFor anyone interested in the mini-PTG that occurred at Kubecon yesterday, the etherpad is here: https://etherpad.openstack.org/p/airship-kubecon-san-diego15:05
dwaltalexanderhughes: I believe those also shifted15:05
alexanderhughesthere was an action item last week to meet with SIG owners to check with them to verify they are moving their times with DST15:05
dwaltI have not heard anything of that15:05
*** shubham_kaushal has joined #airshipit15:06
dwaltin the interest of consistency, maybe they should all shift like this one?15:06
alexanderhugheswhen I say move, I mean update wiki to show the UTC time changed by 1 hr as result of DST.  Matt M was working on it15:06
dwaltah. I just updated the IRC one this morning15:06
dwaltI don't think anything else has been updated15:06
alexanderhugheslet's add as followup - TC, WC, Sigs, community meeting let's verify with the owners of each meeting and make sure wikis/event invites are all accurate15:07
*** SRao has quit IRC15:07
alexanderhughesI think most of this is done, just want to double check before we get the larger audience back next week15:07
dwalt#action verify ALL meeting times15:07
dwaltsounds like a good idea15:07
dwaltBack on Kubecon, I expect those folks will fill us in next week. In the meantime, we can read the etherpad and get an idea for what's happening :)15:08
dwaltAnything else before we close team?15:08
*** michael-beaver has joined #airshipit15:08
*** neha-alhat has joined #airshipit15:08
shubham_kaushalI have query on meatl3 project..15:08
dwaltgo for it!15:08
shubham_kaushalI have seen metal3 project under:15:09
shubham_kaushalhttps://github.com/metal3-io/15:09
shubham_kaushaland under https://github.com/Nordix/15:09
*** RaviWaghmare has quit IRC15:09
shubham_kaushalwhich one I should follow?15:09
*** DanCrank has joined #airshipit15:10
dwaltI believe the first link is the official one. The second one looks like a fork of the official project15:10
*** rajat143 has joined #airshipit15:10
dwalt#link https://github.com/metal3-io/15:10
*** dpawlik has quit IRC15:11
shubham_kaushalok.. I shall look at https://github.com/metal3-io .. thank you @dwalt15:11
dwaltsure thing!15:12
dwaltany final thoughts team?15:12
*** RaviWaghmare has joined #airshipit15:12
digaHey Digambar Here15:12
dwalto/ diga15:13
digaI attended Airship PTG at Shanghai and working on Airship now :)15:13
dwaltWelcome! Glad to have you on board :)15:13
digaI have couple of questions to ask but will reserve it for next meeting :)15:13
shubham_kaushalwelcome diga!15:13
digaThank you dwalt15:13
diga:)15:13
dwaltSounds good! Most of our team is in San Diego for Kubecon, so the next meeting should be pretty busy :)15:13
digashould be :)15:14
digaYes15:14
openstackgerritMerged airship/airshipctl master: Minor README formatting fixes  https://review.opendev.org/69491015:14
dwaltAlrighty, I think we can wrap this one up. Thanks for coming everyone! Have a great day/evening15:14
dwalt#endmeeting15:14
*** openstack changes topic to "https://opendev.org/airship || https://wiki.openstack.org/wiki/Airship || https://review.opendev.org/#/q/projects:airship+status:open+NOT+label:Verified%253D-1+NOT+label:Workflow%253D-1+NOT+message:DNM+NOT+message:WIP"15:14
openstackMeeting ended Tue Nov 19 15:14:52 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:14
openstackMinutes:        http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-11-19-15.00.html15:14
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-11-19-15.00.txt15:14
openstackLog:            http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-11-19-15.00.log.html15:14
*** shubham_kaushal has quit IRC15:16
*** neha-alhat has quit IRC15:16
*** uzumaki has joined #airshipit15:19
*** rajat143 has quit IRC15:19
*** uzumaki has quit IRC15:19
*** AshuKumar has quit IRC15:22
openstackgerritDoug Aaser proposed airship/promenade master: [WIP/DNM] Uplift etcd to v3.4.2  https://review.opendev.org/69342015:22
*** roman_g has joined #airshipit15:31
openstackgerritMerged airship/airship-in-a-bottle master: Enable VNC Console for vms  https://review.opendev.org/69417615:32
openstackgerritDoug Aaser proposed airship/promenade master: [WIP/DNM] Uplift etcd to v3.4.2  https://review.opendev.org/69342015:32
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Verify Docker Default apparmor for Openstack Utility Container.  https://review.opendev.org/69438515:33
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Verify Docker Default apparmor for Openstack Utility Container.  https://review.opendev.org/69438515:35
*** RaviWaghmare has left #airshipit15:38
*** roman_g has quit IRC15:53
*** kskels has joined #airshipit15:57
*** sreejithp has joined #airshipit15:59
*** roman_g has joined #airshipit16:03
openstackgerritSamuel Pilla proposed airship/promenade master: [WIP/DNM] Upgrade Hyperkube version for k8s 1.16  https://review.opendev.org/69372616:04
*** Madhumati has quit IRC16:10
openstackgerritvaragini karthik proposed airship/treasuremap master: [WIP] Integrate postgresql utility container in airship/treasuremap  https://review.opendev.org/68863116:15
openstackgerritvaragini karthik proposed airship/treasuremap master: [WIP] Integrate postgresql utility container in airship/treasuremap  https://review.opendev.org/68863116:26
openstackgerritSamuel Pilla proposed airship/promenade master: [WIP/DNM] Upgrade Hyperkube version for k8s 1.16  https://review.opendev.org/69372616:32
*** SRao has joined #airshipit16:33
*** ian-pittwood has quit IRC16:33
openstackgerritPrateek Dodda proposed airship/divingbell master: Implement Security Context for Divingbell  https://review.opendev.org/69308716:35
openstackgerritIan Pittwood proposed airship/pegleg master: Move generate-pki under generate sub-command  https://review.opendev.org/69481016:38
openstackgerritvaragini karthik proposed airship/treasuremap master:  [WIP] Integrate postgresql utility container in airship/treasuremap  https://review.opendev.org/69503216:47
*** SRao has quit IRC16:48
*** SRao has joined #airshipit16:49
*** cps1408 has joined #airshipit16:50
*** pgaxatte has quit IRC16:54
*** SRao has quit IRC16:57
openstackgerritvaragini karthik proposed airship/treasuremap master:  [WIP] Integrate postgresql utility container in airship/treasuremap  https://review.opendev.org/69503217:02
*** SRao has joined #airshipit17:02
openstackgerritDoug Aaser proposed airship/promenade master: [WIP/DNM] Uplift etcd to v3.4.2  https://review.opendev.org/69342017:04
*** cps140834 has joined #airshipit17:09
*** cps140834 has quit IRC17:10
*** cps140852 has joined #airshipit17:11
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Enable Docker-default Apparmor Profile to  Ceph Utility Container.  https://review.opendev.org/69503717:11
*** cps1408 has quit IRC17:12
*** cps140852 has left #airshipit17:12
*** SRao has quit IRC17:25
*** GarimaM has quit IRC17:26
*** michael-beaver has quit IRC17:27
openstackgerriteric welch proposed airship/deckhand master: Yaml linting gates (second commit) Change-Id: I4cf78172b71e8fa0c4de5072b51a1fd90fee595a  https://review.opendev.org/69393517:30
openstackgerritdiwakar thyagaraj proposed airship/treasuremap master:  [WIP] Integrate postgresql utility container in airship/treasuremap  https://review.opendev.org/69503217:34
*** roman_g has quit IRC17:36
*** roman_g has joined #airshipit17:36
*** bh526r has quit IRC17:45
openstackgerritDoug Aaser proposed airship/promenade master: [WIP/DNM] Uplift etcd to v3.4.2  https://review.opendev.org/69342018:00
openstackgerritDoug Aaser proposed airship/promenade master: [WIP/DNM] Uplift etcd to v3.4.2  https://review.opendev.org/69342018:20
*** jamesgu has joined #airshipit18:28
*** dpawlik has joined #airshipit18:30
*** diga has quit IRC18:31
openstackgerritDoug Aaser proposed airship/promenade master: [WIP/DNM] Uplift etcd to v3.4.2  https://review.opendev.org/69342018:41
openstackgerritDoug Aaser proposed airship/promenade master: Add debug reporting to genesis gate  https://review.opendev.org/69483818:45
*** mnaser has joined #airshipit18:57
mnaseris airship 2.0 meant to have an upgrade path from 1.0?18:57
openstackgerritDoug Aaser proposed airship/promenade master: [WIP/DNM] Uplift etcd to v3.4.2  https://review.opendev.org/69342018:58
openstackgerritDoug Aaser proposed airship/promenade master: Add debug reporting to genesis gate  https://review.opendev.org/69483819:32
openstackgerritDoug Aaser proposed airship/promenade master: [WIP/DNM] Uplift etcd to v3.4.2  https://review.opendev.org/69342019:32
*** dpawlik has quit IRC19:41
*** dpawlik has joined #airshipit19:48
openstackgerritIan Howell proposed airship/airshipctl master: Utilize Docker caching to speed up image building  https://review.opendev.org/69485720:00
*** michael-beaver has joined #airshipit20:02
nishantkrmnaser: We have had some discussions around it in the past but I don't think we have a fully laid out plan, most folks are out attending kubecon and I think it was one of the discussion points in yesterday's airship discussion, you can probably refer this etherpad, L135- https://etherpad.openstack.org/p/airship-kubecon-san-diego20:03
mnasernishantkr: cool, gotcha20:03
mnasernishantkr: currently, i'm interested in leveraging promenade but curious about how the genesis node gets self managed eventually20:06
mnasera lot of the airship 2.0 concepts though seem much more forward looking (i.e. replacing things with kubeadm, kustomize, etc)20:07
openstackgerritTrung Thai proposed airship/maas master: Add facility to rsyslog to configure log levels in the Chart.  https://review.opendev.org/69479420:11
openstackgerritTrung Thai proposed airship/maas master: Add facility to rsyslog to configure log levels in the Chart.  https://review.opendev.org/69479420:13
nishantkrmnaser: genesis node in 2.0 will be provisioned remotely by our new CLI tool airshipctl which is a client external to the site, the idea is to generate an ISO image and then use it to stand up your ephemeral(first) node, post that it will be the bootstrap flow as followed by the cluster api project. (not sure if that answers your question)20:19
*** dpawlik has quit IRC20:22
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Verify Docker Default apparmor for Openstack Utility Container.  https://review.opendev.org/69438520:22
nishantkrmnaser: If you haven't already read, This blog gives a pretty decent high level overview of how baremetal provisioning would look like in Airship 2.0 - https://www.airshipit.org/blog/airship-blog-series-5-drydock-and-its-relationship-to-cluster-api.html20:26
*** openstackgerrit has quit IRC20:35
*** openstackgerrit has joined #airshipit20:38
openstackgerritdiwakar thyagaraj proposed airship/treasuremap master:  [WIP] Integrate postgresql utility container in airship/treasuremap  https://review.opendev.org/69503220:38
*** dpawlik has joined #airshipit20:45
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Enable Docker-default Apparmor Profile to  Calicoctl Utility Container  https://review.opendev.org/69479320:56
*** dpawlik has quit IRC20:58
mnasernishantkr: yep i saw those articles, really like them, just wondering how far away are we from airship 2.0 :p21:00
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Update Calicoctl Utility Container to use Ubuntu-Xenial Image  https://review.opendev.org/69480321:02
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Enable Docker-default Apparmor Profile to  Calicoctl Utility Container  https://review.opendev.org/69479321:02
nishantkrmnaser: Well, the plan is to have a 2.0 release by mid of next year21:04
*** dpawlik has joined #airshipit21:18
*** dpawlik has quit IRC21:23
openstackgerritAlexander Noskov proposed airship/treasuremap master: Uplift treasuremap + fixes  https://review.opendev.org/69508621:24
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Update Calicoctl Utility Container to use Ubuntu-Xenial Image  https://review.opendev.org/69480321:35
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Update Calicoctl Utility Container to use Ubuntu-Xenial Image  https://review.opendev.org/69480321:39
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Update Calicoctl Utility Container to use Ubuntu-Xenial Image  https://review.opendev.org/69480321:42
openstackgerritSean Eagan proposed airship/promenade master: [WIP] Add kubelet_config chart  https://review.opendev.org/69187621:42
openstackgerritSamuel Pilla proposed airship/promenade master: [WIP/DNM] Upgrade Hyperkube version for k8s 1.16  https://review.opendev.org/69372621:43
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Verify Docker Default apparmor for Openstack Utility Container.  https://review.opendev.org/69438521:55
openstackgerritdiwakar thyagaraj proposed airship/porthole master: Verify Docker Default apparmor for Openstack Utility Container.  https://review.opendev.org/69438522:11
*** jamesgu has quit IRC22:37
*** aaronsheffield has quit IRC22:41
*** sreejithp has quit IRC22:41
*** sreejithp has joined #airshipit22:42
JaredSchooleySo I was able to resolve the issue I was having, with kubernetes API not deploying.. Now I'm having issues with mariaDB and postgresql not working.22:46
JaredSchooleylooking through the logs, the issues with mariadb-server seems to be:22:46
JaredSchooley'Failed to open file '/var/lib/mysql//galera.cache': 1 (Operation not permitted)\n"'22:46
JaredSchooleypostgresql seems to be causing issues by:22:49
JaredSchooley'data directory \"/var/lib/postgresql/pgdata\" has wrong ownership\n","stream":"stderr","time":"2019-11-19T22:46:21.465266284Z"}22:49
JaredSchooley{"log":"postgresql: 2019-11-19 22:46:21 UTC [698]: [2-1] 5dd470bd.2ba 0     2019-11-19 22:46:21.465 UTC HINT:  The server must be started by the user that owns the data directory.\n","stream":"stderr","time":"2019-11-19T22:46:21.465294063Z"}'22:49
JaredSchooleyI'm running the airship-in-a-bottle script as root by doing sudo -i so I'm not sure what the issue is? Does there need to be a certain user account with root?22:50
*** alexanderhughes has quit IRC22:56
*** DanCrank has quit IRC22:59
JaredSchooleyIt seems to get stuck in a CrashLoopBackOff state as well, I also see this is an issue currently on atlassian. Is anyone aware of a work-around?23:00
JaredSchooleyhttps://airship.atlassian.net/browse/AIR-131?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel23:00
kskelsare you running treasuremap airship in a bottle?23:03
kskelshttps://github.com/airshipit/treasuremap/tree/master/tools/deployment/aiab23:04
kskelshave a try with v1.5 as well23:04
kskelsmake sure ceph is in healthy state23:04
kskelsthese are quick tips - I can try help more tomorrow23:04
kskelswe are running airship in a bottle from treasyremap in daily tests - it seems very stable right now23:04
kskelsso it might likely be envirionment, perhaps IOPS related23:05
kskelsI've hard people have succeeded by making VM larger if htis is running on some public cloud service23:05
*** spilla_ has quit IRC23:06
openstackgerritKaspars Skels proposed airship/treasuremap master: Upgrade Tiller version for k8s 1.16  https://review.opendev.org/69460423:06
JaredSchooleyyes23:08
JaredSchooleyok ill look at that, the VM right now has been allocated 100gb of disk space, 48gb of ram, and 16 threads23:09
JaredSchooleyalso, yes, I am running it from treasuremap, I've tried v1.5 and the latest. I can try v1.4 and v1.5 again23:10
kskelsis ceph healthy?23:10
JaredSchooleyIt seems like eventually mariaDB does come up23:10
kskelskubectl exec -it -n ceph <ceph-mon pod> -- ceph -s23:10
kskelscheck `kubectl exec -it -n ceph <ceph-mon pod> -- ceph -s`23:10
kskelsyou can also measure IOPS from the VM23:11
kskelslet me send a command23:11
kskels`apt-get install fio`23:11
kskels`fio   --randrepeat=1     --ioengine=libaio     --direct=1     --gtod_reduce=1     --name=test     --filename=.fiotest     --bs=4k     --iodepth=64     --size=1G     --readwrite=randrw     --rwmixread=50 `23:11
kskelscan you run this on your VM?23:11
michael-beaverAs an extra note, if you are finding that some of the services that are failing do eventually come up without any more issues, you can extend their timeouts in the manifests, here is an example location for mariadb: https://opendev.org/airship/treasuremap/src/branch/master/global/software/charts/ucp/core/mariadb.yaml#L5723:14
kskelsthe env we are running `aiab` is around 500-1k IOPS23:15
kskelsI believe you will need at least couple of hundred for things to run smooth23:15
kskelsif you are running on a public cloud, you may try higher IOPS options23:15
*** sreejithp has quit IRC23:15
JaredSchooleyyea I'm checking, sudo kubectl exec -it -n ceph <ceph-mon pod> --ceph -s isn't executing correctly, im trying to figure out why it isnt being listed23:15
kskels`<ceoh-monpod>`23:16
kskelsyou need to replace by the ceph pod name23:16
kskels`kubectl get pods -n ceph -o wide |grep -v Completed`23:16
kskelsyou will see a name for `ceph-mon-xyz`23:16
JaredSchooleyYes, I figured as much, none is being returned23:17
JaredSchooleywhich, if its as you say, could def be the issue23:17
JaredSchooleyIt did create a new user on the VM though called 'Ceph storage service'23:17
kskelscan you share what you see23:17
kskelsin kubectl get pods -n ceph23:17
kskelsit could be that ceph failed23:17
JaredSchooleykube-system   airship-coredns-test                                      0/1     Completed   0          85m23:17
JaredSchooleykube-system   airship-haproxy-haproxy-test                              0/1     Completed   0          84m23:18
JaredSchooleykube-system   airship-kubernetes-calico-etcd-etcd-test                  0/1     Completed   0          87m23:18
JaredSchooleykube-system   airship-kubernetes-etcd-etcd-test                         0/1     Completed   0          84m23:18
JaredSchooleykube-system   apiserver-key-rotate-5ppb4                                0/1     Completed   0          82m23:18
JaredSchooleykube-system   auxiliary-etcd-airshipvm2inabottle                        3/3     Running     0          90m23:18
JaredSchooleykube-system   bootstrap-armada-airshipvm2inabottle                      4/4     Running     0          90m23:18
JaredSchooleysorry it didnt appear to paste the pod list23:18
JaredSchooleyWhere should I check to see if it has failed?23:18
kskelshttp://paste.openstack.org/23:18
kskelsyou may use this as well23:18
kskels`kubectl get pods -n ceph -o wide`23:18
kskelswill show you `ceph` namespace and if pods are running23:19
JaredSchooleyNo resources found23:19
JaredSchooleykubectl get pods --all-namespaces returns23:20
JaredSchooleyhttp://paste.openstack.org/show/786374/23:20
kskelsah never mind - I forgot we are using NFS for aiab as a shortcoming23:20
kskelsI think I've heard of this NFS bug before23:20
kskelsrelated to postgres and usually in public cloud environments23:21
JaredSchooleyI found a few refrences to a similar issue online23:21
kskelsmattmceuen: do you remember if we had solution for NFS/postgresql issue?23:21
JaredSchooleyWell the system I'm running this on is personal, its running ubuntu 18.04 and the VM's are running through KVM on 16.04 - as requested by the docs.23:21
kskelsyes, this sounds good23:22
kskelslet's see if Matt or others have any tips - I know some people had this before, I'm not sure why23:22
JaredSchooleyOk, thank you for the help!23:22
JaredSchooleyAlso, I ran fio as requested.23:28
JaredSchooleyit returned:23:28
JaredSchooley read : io=524704KB, bw=5882.2KB/s, iops=1470, runt= 89203msec23:28
JaredSchooley  write: io=523872KB, bw=5872.9KB/s, iops=1468, runt= 89203msec23:28
JaredSchooleyalong with some other probably not relevant information23:28
JaredSchooleysince you seemed to be most interested in the iops value.23:28
*** jamesgu has joined #airshipit23:30
*** ab2434_ has quit IRC23:34

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!