Thursday, 2019-08-22

*** nishantkr has quit IRC00:01
*** mcriswel1 has quit IRC00:19
*** rezroo has quit IRC00:30
*** jamesgu_ has quit IRC00:31
*** hogepodge has quit IRC00:55
*** ChanServ has quit IRC00:55
*** jhesketh has quit IRC00:55
*** georgk has quit IRC00:55
*** dpawlik has quit IRC00:55
*** gagehugo has quit IRC00:56
*** jamesgu has quit IRC00:56
*** evrardjp has quit IRC00:56
*** CobHead has quit IRC00:56
*** serverascode has quit IRC00:56
*** kukacz has quit IRC00:56
*** licanwei has quit IRC00:56
*** openstackgerrit has quit IRC00:56
*** dasp has quit IRC00:56
*** alanmeadows has quit IRC00:56
*** irclogbot_2 has quit IRC00:56
*** srwilkers has quit IRC00:56
*** dirk has quit IRC00:56
*** Anticimex has quit IRC00:56
*** jbryce has quit IRC00:56
*** aprice has quit IRC00:56
*** neith has quit IRC00:56
*** jayahn has quit IRC00:56
*** howell has quit IRC00:56
*** pas-ha has quit IRC00:56
*** ildikov has quit IRC00:56
*** seungkyua has quit IRC00:56
*** lamt has quit IRC00:56
*** mugsie has quit IRC00:56
*** masayukig has quit IRC00:56
*** v1k0d3n has quit IRC00:56
*** jemangs has quit IRC00:56
*** cheng1 has quit IRC00:56
*** KeithMnemonic1 has quit IRC00:56
*** arunkant has quit IRC00:56
*** umbSublime has quit IRC00:56
*** kskels has quit IRC00:56
*** iamweswilson has quit IRC00:56
*** evgenyl has quit IRC00:56
*** portdirect has quit IRC00:56
*** mattmceuen has quit IRC00:56
*** dwalt has quit IRC00:56
*** AlexNoskov has quit IRC00:56
*** vdrok has quit IRC00:56
*** mbologna has quit IRC00:56
*** redrobot has quit IRC00:56
*** mbeierl has quit IRC00:56
*** toabctl has quit IRC00:56
*** toabctl has joined #airshipit01:14
*** mbeierl has joined #airshipit01:14
*** redrobot has joined #airshipit01:14
*** mbologna has joined #airshipit01:14
*** evrardjp has joined #airshipit01:14
*** jamesgu has joined #airshipit01:14
*** gagehugo has joined #airshipit01:14
*** dpawlik has joined #airshipit01:14
*** alanmeadows has joined #airshipit01:14
*** dasp has joined #airshipit01:14
*** openstackgerrit has joined #airshipit01:14
*** licanwei has joined #airshipit01:14
*** hogepodge has joined #airshipit01:14
*** howell has joined #airshipit01:14
*** v1k0d3n has joined #airshipit01:14
*** kukacz has joined #airshipit01:14
*** serverascode has joined #airshipit01:14
*** CobHead has joined #airshipit01:14
*** georgk has joined #airshipit01:14
*** jhesketh has joined #airshipit01:14
*** vdrok has joined #airshipit01:14
*** AlexNoskov has joined #airshipit01:14
*** dwalt has joined #airshipit01:14
*** mattmceuen has joined #airshipit01:14
*** portdirect has joined #airshipit01:14
*** evgenyl has joined #airshipit01:14
*** iamweswilson has joined #airshipit01:14
*** ChanServ has joined #airshipit01:14
*** card.freenode.net sets mode: +o ChanServ01:14
*** srwilkers has joined #airshipit01:14
*** cheng1 has joined #airshipit01:14
*** KeithMnemonic1 has joined #airshipit01:14
*** arunkant has joined #airshipit01:14
*** umbSublime has joined #airshipit01:14
*** v1k0d3n is now known as Guest9878601:17
*** mugsie has joined #airshipit01:21
*** Anticimex has joined #airshipit01:21
*** irclogbot_3 has joined #airshipit01:23
*** Kuirong has joined #airshipit01:25
openstackgerritEvgeniy L proposed airship/treasuremap master: Forceful deletion of Stateful set pods  https://review.opendev.org/67778902:18
*** ianychoi has quit IRC02:19
*** ianychoi has joined #airshipit02:19
*** jamesgu_ has joined #airshipit02:33
*** licanwei has quit IRC02:36
*** gkadam has joined #airshipit03:37
*** gkadam is now known as gkadam-afk03:38
*** gkadam has joined #airshipit04:17
*** gkadam-afk has quit IRC04:19
*** lamt has joined #airshipit04:51
openstackgerritMerged airship/porthole master: Chart/Dockerfile for Openstack Utility Container Added Support for rbac  https://review.opendev.org/67467005:58
*** ianychoi has quit IRC06:15
*** jamesgu_ has quit IRC06:17
*** ianychoi has joined #airshipit06:19
openstackgerritLuna Das proposed airship/porthole master: Add Tracking User Identity in openstack utility container logs  https://review.opendev.org/67480806:20
*** ianychoi has quit IRC06:28
*** ianychoi has joined #airshipit06:29
*** cp has quit IRC06:30
*** arunx has joined #airshipit06:30
*** jsuchome has joined #airshipit06:42
openstackgerritSmruti Soumitra Khuntia proposed airship/promenade master: Enabling debug-report.sh to configure pods log duration.  https://review.opendev.org/67744806:44
*** ianychoi has quit IRC06:52
*** ianychoi has joined #airshipit06:54
*** dr_feelgood has joined #airshipit07:13
openstackgerritDmitry Ukov proposed airship/specs master: Introduce isogen subcommand for airshipctl  https://review.opendev.org/67670007:15
*** dirk1 has joined #airshipit07:27
*** dr_feelgood has quit IRC07:29
*** dr_feelgood has joined #airshipit07:29
*** dirk1 is now known as dirk07:30
*** itxaka has joined #airshipit07:32
CobHeadQuestion - do I need to register with Ubuntu One in order to submit PR to Gerrit?07:35
CobHeadNvm - seems like I need it.07:45
*** AlexNoskov has quit IRC07:49
*** ianychoi has quit IRC07:52
*** ianychoi has joined #airshipit07:54
openstackgerritSmruti Soumitra Khuntia proposed airship/promenade master: Enabling debug-report.sh to configure pods log duration.  https://review.opendev.org/67744808:12
*** nmimi has joined #airshipit08:18
*** CobHead has quit IRC08:19
*** itxaka has quit IRC08:23
*** CobHead has joined #airshipit08:34
roman_gCobHead: yes, this is required08:43
roman_gCobHead: https://docs.openstack.org/contributors/common/setup-gerrit.html08:44
roman_gCobHead: https://docs.openstack.org/infra/manual/developers.html08:45
CobHeadThanks :)08:48
*** roman_g has quit IRC08:51
*** dr_feelgood has quit IRC08:59
*** dr_feelgood has joined #airshipit09:02
*** dr_feelgood has joined #airshipit09:02
*** dr_feelgood has quit IRC09:50
*** dr_feelgood has joined #airshipit09:51
*** dr_feelgood has quit IRC09:55
*** dr_feelgood has joined #airshipit10:11
*** dr_feelgood has quit IRC10:12
*** dr_feelgood has joined #airshipit10:13
*** dr_feelgood has quit IRC10:14
*** dr_feelgood has joined #airshipit10:14
*** dr_feelgood has quit IRC10:46
*** dr_feelgood has joined #airshipit10:47
*** dr_feelgood has quit IRC10:52
*** Kuirong has quit IRC11:16
*** dr_feelgood has joined #airshipit11:26
openstackgerritOthar Linaker proposed airship/treasuremap master: Bumped the timeout for the ucp keystone test  https://review.opendev.org/67798011:47
CobHeadI'll be investigating whether or not Armada fails because of a timeout next.11:52
CobHeadI realize I need to wrap my lines - sorry about the commit message ;\11:52
*** souradage has joined #airshipit11:57
*** nmimi has left #airshipit11:58
*** roman_g has joined #airshipit12:06
*** roman_g has quit IRC12:06
*** roman_g has joined #airshipit12:06
*** ianychoi has quit IRC12:07
*** ianychoi has joined #airshipit12:09
*** aaronsheffield has joined #airshipit12:15
*** ianychoi has quit IRC12:27
*** mcriswel1 has joined #airshipit12:27
*** ianychoi has joined #airshipit12:28
*** gkadam has quit IRC12:32
*** mcriswel1 has quit IRC12:39
*** dr_feelgood has quit IRC13:02
*** dr_feelgood has joined #airshipit13:02
*** dr_feelgood has quit IRC13:07
*** dr_feelgood has joined #airshipit13:07
*** alexanderhughes has joined #airshipit13:08
*** alexanderhughes_ has joined #airshipit13:13
*** alexanderhughes has quit IRC13:13
*** alexanderhughes_ is now known as alexanderhughes13:13
openstackgerritSean Eagan proposed airship/armada master: Support builtin chart dependencies  https://review.opendev.org/67683113:14
*** mcriswel1 has joined #airshipit13:20
*** kranthikirang has joined #airshipit13:23
*** kranthikirang has quit IRC13:34
*** AlexNoskov has joined #airshipit13:34
*** kranthikirang has joined #airshipit13:34
*** kranthik_ has joined #airshipit13:38
*** kranthikirang has quit IRC13:41
*** nishantkr has joined #airshipit13:42
*** mcriswel1 has quit IRC13:43
*** altlogbot_0 has joined #airshipit13:46
*** altlogbot_0 has quit IRC13:48
*** altlogbot_0 has joined #airshipit13:51
openstackgerritNishant Kumar proposed airship/treasuremap master: Forceful deletion of Stateful set pods  https://review.opendev.org/67778913:53
*** mcriswel1 has joined #airshipit13:54
openstackgerritAnthony Bellino proposed airship/treasuremap master: Secret rotation and validation  https://review.opendev.org/63569414:05
*** roman_g has quit IRC14:09
*** dr_feelgood has quit IRC14:10
*** spilla has joined #airshipit14:11
*** roman_g has joined #airshipit14:30
*** jhesketh has quit IRC14:44
openstackgerritKaspars Skels proposed airship/treasuremap master: Update expired certs for Seaworthy k8s cluster  https://review.opendev.org/67784414:44
*** kskels has joined #airshipit14:45
openstackgerritNishant Kumar proposed airship/treasuremap master: Forceful deletion of Stateful set pods  https://review.opendev.org/67778914:45
*** michael-beaver has joined #airshipit14:45
*** jhesketh has joined #airshipit14:45
*** rezroo has joined #airshipit14:47
*** jamesgu_ has joined #airshipit15:04
kskelsdwalt: certs had expired for Seaworthy - https://review.opendev.org/#/c/677844/ - once we merge that we can test more Nagios15:12
alexanderhugheskskels:15:27
kskelshey Alex15:27
alexanderhughesI picked a random cert, from that patch it's valid for 5 years.  seems excessive15:27
alexanderhughesis that intended?15:28
kskelshm.. I'm not able to launch k8s without this update15:28
alexanderhughesyeah, just wondering if 5 years is appropriate or if we want to scale that back to like 1 yr15:28
kskelsoh - this was just default from pegleg15:28
alexanderhughespegleg default should be 1yr but you can specify longer15:29
kskelsthis is pure pegleg - no options at all15:29
kskelsjust re-generated new certs15:29
alexanderhugheswith the img from master?15:29
kskelssorry that is promenade15:29
kskelsyes - very much lastest master code15:30
alexanderhughesoh ok15:30
kskelsI think for seaworthy test site - not sure we care - but maybe we want to check Promenade? for good defaults..15:30
alexanderhughesdon't really need to use promenade anymore for cert generation.  I need to go look at the airship script again to see what we can rotate out15:31
alexanderhughespegleg cert feature was finally updated so it can be used directly to rotate all certs, or fill in gaps, with configurable duration (default=365d)15:31
alexanderhughespatch looks good, just thought the 5y was odd so wanted to check with you15:32
kskelsso there is script treasuremap/tools/airship - and it already has pegleg wrapper15:33
kskelslet me know if you have a sample command - I can definetly re-generate the certs with pegleg instead15:33
kskelsit should already have very recent version of pegleg - like a few days old15:33
alexanderhugheshttps://opendev.org/airship/pegleg/src/branch/master/doc/source/cli/cli.rst#generate-pki15:34
dwaltkskels: great, I'll take another look :)15:35
*** roman_g has quit IRC15:36
openstackgerritDrew Walters proposed airship/deckhand master: CI: Update Airskiff full-site manifest location  https://review.opendev.org/67782615:38
openstackgerritPhil Sphicas proposed airship/airship-in-a-bottle master: (fix) Add apparmor profiles on genesis node in multinode gate  https://review.opendev.org/67804815:38
openstackgerritDrew Walters proposed airship/shipyard master: CI: Update Airskiff full-site manifest location  https://review.opendev.org/67782315:39
kskelshttps://www.irccloud.com/pastebin/YhoD0Lkr/15:41
kskelsalexanderhughes:15:41
alexanderhugheshttps://review.opendev.org/#/c/671337/ 2w ago, 4480ab5574b2eec8a3c46b3f3eb63c48e1bd95ed is 3w ago15:42
openstackgerritMerged airship/treasuremap master: Update expired certs for Seaworthy k8s cluster  https://review.opendev.org/67784415:43
alexanderhughesdo I need to update versions.yaml manually?  I thought auto uplift job would have hit it15:44
kskelsexactly..15:44
kskelsit should of been automatically and last one was just a few days back15:44
kskelswe need to check if everyhting works well there15:44
kskelslet me pull it in manually for update now15:44
kskelsah! I'm silly15:44
kskelsI had v1.3 checked out :)15:44
alexanderhughes:)15:44
alexanderhughesso when you run that command it won't generate a new certificates.yaml; instead it will generate each document in its own <document_name>.yaml file.  felt that was easier for humans to track down changes to individual certs/CA/keys etc15:45
alexanderhughesbut if that's not desired I can change it to certificates.yaml15:46
kskelshttps://www.irccloud.com/pastebin/Sm928tem/15:46
kskelswhat!? :)15:46
alexanderhughesanything using pegleg secrets management (certs get checked to see if they should be encrypted) needs pegleg passphrase and salt set15:47
alexanderhughesgenesis bundle also needs promenade_encryption_key15:47
kskelsI see15:47
kskelsare you using just like openssl rand15:47
kskelsto generate some key?15:47
alexanderhugheswhich key?15:48
kskelsthat is PEGLEG_PASSPHRASE15:48
alexanderhughesset manually by user15:48
alexanderhughesif storagepolicy:encrypted it'll consume that to encrypt15:48
kskelsright now we don't have any of the encryption stuff for seaworhty15:48
alexanderhughesif cleartext it'll leave as plain text, but because of the check it expects salt/passphrase there.  starting to think that's a gap15:48
kskelsos there a way to just use it as before..?15:48
alexanderhughesforcing salt/passphrase if encryption/decryption isn't actually used might be overkill15:49
kskelsit may be a bigger effort to move Seaworthy to all encypted15:49
alexanderhughesyeah so if storagepolicy:cleartext you can set pegleg_passphrase and pegleg_salt env vars as anything you want as long as they're 24char or longer.  because storagepolicy:cleartext it won't actually encrypt15:49
alexanderhughesso when I run locally I just do export PEGLEG_PASSPHRASE="twentyfourcharacterpassphrase"; export PEGLEG_SALT=$PEGLEG_PASSPHRASE15:50
kskelsok15:50
kskelslet me try15:50
openstackgerritIan Pittwood proposed airship/pegleg master: [WIP] Fix test_check_expiry  https://review.opendev.org/67805015:58
kskelshttps://www.irccloud.com/pastebin/EVD0NoVr/16:02
*** gkadam has joined #airshipit16:02
kskelslooks like it's working16:02
kskelslooks like I need to manually delete old certificates.yaml file16:02
openstackgerritIan Pittwood proposed airship/pegleg master: [WIP] Fix test_check_expiry  https://review.opendev.org/67805016:03
*** gkadam has quit IRC16:06
alexanderhughesyeah if you're swapping from old certificates.yaml setup to the new one yaml file per doc you'll need to get rid of certificates.yaml16:10
*** altlogbot_0 has quit IRC16:14
*** altlogbot_1 has joined #airshipit16:14
*** ianychoi has quit IRC16:15
*** ianychoi has joined #airshipit16:17
*** ianychoi has quit IRC16:19
*** ianychoi has joined #airshipit16:19
dwaltroman_g: hey! Would appreciate your reconsideration on this patch https://review.opendev.org/66041616:35
dwaltI'll be out for an extended period of time starting tomorrow16:35
*** roman_g has joined #airshipit16:36
openstackgerritKrishna Venkata proposed airship/porthole master: [ceph]: Added procedure to stop the osd pod from being scheduled  https://review.opendev.org/67806016:41
*** roman_g has quit IRC16:43
dwaltTeam, since the Airskiff site in treasuremap moved, the Airskiff CI jobs in other repos (armada, deckhand, shipyard) need to be updated. I have a few patches that change the location and would appreciate reviews: https://review.opendev.org/677822 https://review.opendev.org/677823 https://review.opendev.org/67782616:52
*** vasiler59 has joined #airshipit16:58
openstackgerritKrishna Venkata proposed airship/porthole master: Ceph-Utility implement nccli help command  https://review.opendev.org/67806516:59
openstackgerritDmitrii Kabanov proposed airship/porthole master: Implement Network Policy for Ceph-Utility  https://review.opendev.org/67517317:00
openstackgerritMerged airship/porthole master: Add Tracking User Identity in openstack utility container logs  https://review.opendev.org/67480817:02
openstackgerritKrishna Venkata proposed airship/porthole master: [Ceph]: Ceph-Utility implement nccli help command  https://review.opendev.org/67806517:05
openstackgerritIan Howell proposed airship/airshipctl master: Add document command and its first subcommand  https://review.opendev.org/67682817:10
openstackgerritSmruti Soumitra Khuntia proposed airship/promenade master: Enabling debug-report.sh to configure pods log duration.  https://review.opendev.org/67744817:13
openstackgerritRahul Khiyani proposed airship/porthole master: Implement Network Policy for Ceph-Utility  https://review.opendev.org/67806817:15
openstackgerritDrew Walters proposed airship/treasuremap master: airskiff: Remove package installation script  https://review.opendev.org/67254017:25
*** vasiler has joined #airshipit17:26
openstackgerritMerged airship/porthole master: Initial Commit of compute-utility container code  https://review.opendev.org/67469517:29
openstackgerritIan Pittwood proposed airship/pegleg master: [WIP] Fix test_check_expiry  https://review.opendev.org/67805017:31
*** vasiler has quit IRC17:31
openstackgerritMerged airship/porthole master: Implement Network Policy for Ceph-Utility  https://review.opendev.org/67517317:36
openstackgerritMerged airship/divingbell master: apt: Add full-system upgrade feature  https://review.opendev.org/66041617:43
openstackgerritIan Pittwood proposed airship/pegleg master: Fix OpenSUSE image build  https://review.opendev.org/67694417:49
openstackgerritIan Pittwood proposed airship/pegleg master: Update treasuremap revision for tests  https://review.opendev.org/67805017:52
openstackgerritIan Pittwood proposed airship/promenade master: Pull requirements-direct into requirements  https://review.opendev.org/67749718:06
*** irclogbot_3 has quit IRC18:14
*** irclogbot_2 has joined #airshipit18:14
*** altlogbot_1 has quit IRC18:29
openstackgerritMerged airship/airship-in-a-bottle master: (fix) Add apparmor profiles on genesis node in multinode gate  https://review.opendev.org/67804818:40
CobHeaddwalt: The OS Keystone test is run after Armada has been built, right?19:18
*** cheng1 has quit IRC19:18
*** jsuchome has quit IRC19:24
CobHeadSeems that is indeed the case, judging by the full-site.yaml19:25
openstackgerritMerged airship/treasuremap master: airskiff: Remove package installation script  https://review.opendev.org/67254019:26
dwaltCobHead: yep, after Armada is deployed by the Armada container in 030-armada-bootstrap.sh (I assume you're still referring to Airskiff).19:32
dwaltbasically, that keystone is deployed in the 100-deploy-osh.sh script19:32
CobHeadI see - what could cause Armada to spend eternity running the build action in deploy_site?19:33
openstackgerritRahul Khiyani proposed airship/porthole master: [ceph]: Added procedure to stop the osd pod from being scheduled  https://review.opendev.org/67806019:35
*** cheng1 has joined #airshipit19:35
openstackgerritIan Pittwood proposed airship/pegleg master: Fix OpenSUSE image build  https://review.opendev.org/67694419:38
dwaltCobHead: sorry, I didn't realize you meant the `armada_build` step of the Shipyard action :)19:38
CobHead:)19:38
dwaltyes, that's when the OSH keystone test is run.19:38
CobHeadStill learning my way around19:38
dwaltHow long has it been going?19:38
dwaltthat's my fault :)19:38
CobHeadWell, I just realized that my VM crashed, so I'm trying to identify why19:39
CobHeadPresumably it crashed sometime after the Armada build started.19:39
dwaltOh. Crashed due to resource consumption?19:40
CobHeadUncertain. The syslog is chock full of Kubernetes entries, so I will be spending some time parsing it.19:40
CobHeadBut I find it hard to believe it is due to resource consumption - disk is not full, 25GB of allocated RAM, 8 cores19:41
CobHeadBut we'll see.19:41
dwaltGotcha. I agree, that should be more than sufficient19:41
openstackgerritMerged airship/treasuremap master: Enable back Nagios as part osh-infra monitoring  https://review.opendev.org/67757119:43
openstackgerritIan Pittwood proposed airship/pegleg master: Fix OpenSUSE image build  https://review.opendev.org/67694419:45
openstackgerritIan Pittwood proposed airship/pegleg master: Fix OpenSUSE image build  https://review.opendev.org/67694419:46
CobHeadAug 22 21:09:20 xxxx kernel: [ 7080.861778] Out of memory: Kill process 3666 (qemu-system-x86) score 748 or sacrifice child19:48
CobHeaddwalt ^19:48
CobHeadI'm gonna downscale that allocated memory some.. :d19:48
CobHeadOOM killed it19:48
CobHeadWhich is amusing, as I didn't notice it19:50
*** souradage has quit IRC20:16
openstackgerritRahul Khiyani proposed airship/porthole master: [Ceph]: Ceph-Utility implement utilscli help command  https://review.opendev.org/67806520:18
openstackgerritRahul Khiyani proposed airship/porthole master: [Ceph]: Ceph-Utility implement utilscli help command  https://review.opendev.org/67806520:20
openstackgerritDrew Walters proposed airship/divingbell master: apt: Document management of sources and keys  https://review.opendev.org/67810320:25
*** georgk has quit IRC20:30
*** georgk has joined #airshipit20:30
openstackgerritJames Gu proposed airship/treasuremap master: Add airskiff_suse site  https://review.opendev.org/67810620:58
dwaltCobHead: lol. I didn't expect *too much* memory to be the issue21:07
dwalthope it works out21:07
*** spilla has quit IRC21:09
CobHeaddwalt: Well, so far no dice -> step/01DJXGVQ7TYQEJ3W3VSR115H28/armada_build  9  failed21:19
CobHeadAt this point the script should also terminate, but it moves on to deploying Horizon, and then exits when it is unsuccessful.21:20
CobHeadI have no clue as to whether this is due to a failed build, or if the timeout is hit.21:26
CobHeadTimeout seems to be 3600s -> 1hr, which is probably what it has been running for.21:26
CobHeadI am doing another timeout bump - this time to both Armada timeout and the total deploy.21:34
openstackgerritJames Gu proposed airship/treasuremap master: Add airskiff_suse site  https://review.opendev.org/67810621:49
openstackgerritSean Eagan proposed airship/promenade master: [WIP] Armada metric output for genesis  https://review.opendev.org/67810921:51
openstackgerritMerged airship/armada master: Support builtin chart dependencies  https://review.opendev.org/67683121:54
openstackgerritMerged airship/armada master: Don't log metrics HTTP data  https://review.opendev.org/67778421:54
openstackgerritMerged airship/armada master: CI: Update Airskiff full-site manifest location  https://review.opendev.org/67782222:18
openstackgerritMerged airship/deckhand master: CI: Update Airskiff full-site manifest location  https://review.opendev.org/67782622:26
openstackgerritJames Gu proposed airship/treasuremap master: Add airskiff_suse site  https://review.opendev.org/67810622:43
*** aaronsheffield has quit IRC22:55
*** mcriswel1 has quit IRC22:57
*** AlexNoskov has quit IRC23:25
openstackgerritMerged airship/pegleg master: Update treasuremap revision for tests  https://review.opendev.org/67805023:32

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!