Wednesday, 2019-06-19

*** aaronsheffield has quit IRC00:06
*** nishantkr has quit IRC00:19
*** kskels has joined #airshipit01:37
*** irclogbot_2 has quit IRC05:30
*** irclogbot_3 has joined #airshipit05:30
*** krypto has joined #airshipit05:55
*** phasespace has quit IRC06:14
*** phasespace has joined #airshipit06:42
*** cheng1 has quit IRC06:53
*** pgaxatte has joined #airshipit06:55
*** cheng1 has joined #airshipit06:55
*** amalrajgenocidex has joined #airshipit07:16
amalrajgenocidexHi, I was setting up airsloop. Is it mandatory to have both the servers of same model?07:19
amalrajgenocidexBecause, the pci addresses in hardware profiles are different.07:21
cheng1amalrajgenocidex, you have two nodes right? one as Genesis and the other as work node?07:24
amalrajgenocidexyes, one genesis and one compute(PXE booted)07:25
cheng1The hardware profiles doesn't have affect on the Genesis node, because you install Genesis node manually.07:25
amalrajgenocidexOk. Thanks. So I just need to mention the compute node specs in hardware profile. Thanks :)07:26
cheng1So you need to edit hardware profiles based on your compute node HW info.07:26
cheng1exactly07:27
amalrajgenocidexThank you :)07:27
cheng1amalrajgenocidex, no problem07:27
kskelsadditionally you can create multiple host/hardware profiles and assign them to each node07:34
kskelsyou would just specify in the baremetal/nodes.yaml07:36
kskelsa profile for each node specifically07:37
kskelsso ultimately if you have multiple different computes - they can all be different07:37
amalrajgenocidexOk. Thank you kskels07:37
amalrajgenocidexInitially, I thought the networks and hardware profiles were used by both nodes.07:38
kskelsgenesis in general is deployed manually - so profiles don't apply07:53
kskelsin more generic case, the nodes.yaml will set the host profiule for each individual node07:53
kskelsfor airsloop - you might be adding more computes07:53
kskelsin full seaworthy production deployment - you will have 3 control nodes07:54
kskelsso host profiles might differ between control/compute nodes07:54
*** rustyweise has quit IRC07:58
*** gkadam has joined #airshipit08:00
amalrajgenocidexOk. I understand. I'm changing the profiles now and rerunning site deployment.08:06
openstackgerritKudaka Poorna Rajesh proposed airship/promenade master: Enable probes and mount config for etcd  https://review.opendev.org/65986308:51
*** lemko has joined #airshipit09:02
kskelssounds great! I think you are getting pretty far so feel free to submit some patchset with the things you feel could be improved for docs09:11
kskelsmaybe adding a note for the hw profiles is something in general that could be improved09:11
amalrajgenocidexSure, I will contribute when I have it running. Currently, the PXE boot failed again. failed to get the network. PXE network got assigned to both PXE and data nic.09:33
amalrajgenocidexI don't need to redeploy genesis again after changing the nic values in yaml files, right?09:34
kskelsno - but you must push the new docs into shipyard with --replace option09:34
amalrajgenocidexYes, I did that. but somehow it still didn't work. even after modifying the PCI address09:35
kskelsyou may then ultimately re-deploy genesis09:36
kskelsalthough it shouldn't be needed if the host profile then was updated09:36
kskelsgenerally re-deploying genesis is not too long - so might as well clanup and re-deploy09:38
amalrajgenocidexOk. Let me try that too.09:38
kskelsI think best way so far I've been doing is to disable kubelet/docker09:39
kskelsthen hard-reset the node (via iDRAC/iLO)09:39
kskelsand then run promenade/tools/cleanup.sh -f09:39
amalrajgenocidexOk. I did cleanup too many times. so it's not a problem ;) :)09:42
*** rustyweise has joined #airshipit09:44
*** rustyweise has quit IRC09:47
*** rustyweise has joined #airshipit09:51
amalrajgenocidexkskels, which interfaces needs to be mentioned 'site/airsloop/networks/common-addresses.yaml'09:51
amalrajgenocidexgenesis or compute?09:51
amalrajgenocidexhttps://opendev.org/airship/treasuremap/src/branch/master/site/airsloop/networks/common-addresses.yaml09:52
kskelsah! that does need to match both09:52
kskelsI'm not sure if things will work if your interface name is different09:52
kskelsas neutron and ovs will try to bind to that interface09:53
kskelsand it need to function on both09:53
kskelscalico network I think can work with a regex09:54
kskelse.g.09:54
kskels    ip_autodetection_method: interface=enp67s0f0.72|enp88s0f0.7209:54
kskelsor similar - you may look into the calico docs09:54
kskelsfor ovs and neutron - I'm not sure09:55
kskelsbut you at least should be able to get all the way to joining the nodes09:56
kskelshttps://docs.projectcalico.org/v2.2/reference/node/configuration#ip-autodetection-methods09:58
kskelshere is some description for calico part.. but I'm not sure how to solve ovs/neutron part09:59
amalrajgenocidexMy network in genesis node is 'eth' by default but in compute, it's in 'ens' when airship creates the network. May be I need to setup the same(ens format) in genesis too'09:59
kskelsyes- that would be ideal09:59
amalrajgenocidexOk. noted10:11
*** krypto has quit IRC10:45
*** krypto has joined #airshipit10:45
*** krypto has quit IRC10:46
rustyweiseI'm attempting to make progress on the airsloop deployment now that my cluster is (rather magically) coalesced, with the 'airship shipyard create configdocs' command and I'm getting an error that the command I'm attempting requires authentication.  I can't seem to connect the dots to find the auth keys I should use and where.11:01
kskelsshipyard authenticates via Keystone, e.g. https://github.com/airshipit/treasuremap/blob/master/tools/airship#L3411:13
kskelsneed to be setup correctly with shipyard user and airsloop123 as password11:13
kskelsthe best way to check if you have access to shipyard is11:13
kskelstools/airship shipyard get actions11:13
kskelsthis should return an empty list but will validate if things work11:14
amalrajgenocidexHi rustyweise, refer the docs here " https://airship-treasuremap.readthedocs.io/en/latest/authoring_and_deployment.html#deploy-site-with-shipyard "11:16
amalrajgenocidexyou can export the username and password as mentioned there.11:16
rustyweiseI have set the env vars for keystone, yes11:31
rustyweisethe get actions command also responds with a required auth error so I must have to do something further...#digging11:33
rustyweiselooking through the k8s-keystone-api pod logs it looks like keystone isn't realy running well -->11:39
rustyweise172.22.23.125 - - [19/Jun/2019:11:35:31 +0000] "GET / HTTP/1.1" 300 267 "-" "kube-probe/1.11"11:39
rustyweise2019-06-19 11:35:36.404046 2019-06-19 11:35:36.403 9 INFO keystone.common.wsgi [req-4126d7f4-ba7c-44ff-ae31-fe21138f1750 - - - - -] POST http://keystone-api.ucp.svc.cluster.local:5000/v3/auth/tokens11:39
rustyweise2019-06-19 11:35:36.413247 2019-06-19 11:35:36.403 9 INFO keystone.common.wsgi [req-4126d7f4-ba7c-44ff-ae31-fe21138f1750 - - - - -] POST http://keystone-api.ucp.svc.cluster.local:5000/v3/auth/tokens11:39
rustyweise2019-06-19 11:35:36.463012 2019-06-19 11:35:36.462 9 WARNING keystone.auth.core [req-4126d7f4-ba7c-44ff-ae31-fe21138f1750 - - - - -] Could not find project: service.11:39
rustyweise2019-06-19 11:35:36.468928 2019-06-19 11:35:36.462 9 WARNING keystone.auth.core [req-4126d7f4-ba7c-44ff-ae31-fe21138f1750 - - - - -] Could not find project: service.11:39
rustyweise2019-06-19 11:35:36.469724 2019-06-19 11:35:36.469 9 WARNING keystone.common.wsgi [req-4126d7f4-ba7c-44ff-ae31-fe21138f1750 - - - - -] Authorization failed. The request you have made requires authentication. from 172.22.23.12511:39
rustyweise2019-06-19 11:35:36.471163 2019-06-19 11:35:36.469 9 WARNING keystone.common.wsgi [req-4126d7f4-ba7c-44ff-ae31-fe21138f1750 - - - - -] Authorization failed. The request you have made requires authentication. from 172.22.23.12511:39
rustyweise172.22.23.125 - - [19/Jun/2019:11:35:36 +0000] "POST /v3/auth/tokens HTTP/1.1" 401 114 "-" "shipyard keystoneauth1/3.11.0 python-requests/2.19.1 CPython/3.5.2"11:39
rustyweise2019-06-19 11:35:41.144109 2019-06-19 11:35:41.143 9 INFO keystone.common.wsgi [req-4126d7f4-ba7c-44ff-ae31-fe21138f1750 - - - - -] GET http://10.97.211.235:5000/11:39
rustyweise2019-06-19 11:35:41.147881 2019-06-19 11:35:41.143 9 INFO keystone.common.wsgi [req-4126d7f4-ba7c-44ff-ae31-fe21138f1750 - - - - -] GET http://10.97.211.235:5000/11:39
amalrajgenocidexmay be you need to wait for some time, it might start running after multiple attempts. not sure though11:44
*** alexanderhughes has joined #airshipit12:24
*** howell has joined #airshipit12:24
*** CobHead has joined #airshipit12:39
CobHeadHi lads, I'm trying out Airship-In-A-Bottle (AIAB), but it seems that the postgresql container is stucked in a CrashLoopBackOff. Anyone experienced this and/or have a solution?12:40
*** aaronsheffield has joined #airshipit12:57
*** sthussey has joined #airshipit13:12
kskelsYes.. there has been intermittent issue13:23
kskelsTry re-deploy.. it might help13:23
*** lemko has quit IRC13:32
CobHeadTried it twice yesterday, but I can try again :-)13:39
evgenylCobHead: Hi, what instruction did you use to install AIAB?13:42
*** sri_ has joined #airshipit13:44
CobHeadHi evgenyl - I used this repo: https://github.com/airshipit/treasuremap and ran the airship-in-a-bottle.sh script in the tools/deployment folder (https://github.com/airshipit/treasuremap/tree/master/tools/deployment/aiab)13:52
CobHeadIt's deploying now - I'll report back on how it went in an hour or so13:52
evgenylCobHead: So this is a known issue it happens randomly on some environments, we have patches to update postgresql helm charts to the newest version with a large refactoring, which should help resolving it. Until this patch is landed, you can just retry.13:54
evgenylCobHead: Just curios, what are you VM specs?13:55
evgenylyour*13:55
CobHeadGood to hear the issue has been verified and is known :-) My VM specs are 8 vCPUs, 32GB RAM and 100GB of storage13:55
*** nishantkr has joined #airshipit13:58
openstackgerritRyan van Wyk proposed airship/election master: Adding Ryan van Wyk candidacy for TC role  https://review.opendev.org/66629914:04
*** roman_g has quit IRC14:08
*** phasespace has quit IRC14:11
mattmceuenevgenyl: speaking of that we have two incomplete patchsets for uplifting postgresql -- mine (https://review.opendev.org/#/c/657667/) is missing the new exporter password and service_accounts.yaml setup, and yours (https://review.opendev.org/666299) is missing brownfield upgraded doc  / scripting14:11
mattmceuen(there may be other diffs but those were the obvious ones)14:12
mattmceuenShould we pick one of them and incorporate the diffs from the other one?14:12
CobHeadSeems like it entered the crashloop again14:17
evgenylmattmceuen: Oh, this is simple, you submitted your patch earlier than me so let's get everything updated within your ps.14:19
evgenylCobHead: Can you check the pod's logs to make sure that it is the same problem which we saw earlier? There should be an error about missing postgres database.14:19
openstackgerritIan Pittwood proposed airship/spyglass master: [WIP] Implements data object models  https://review.opendev.org/66209214:20
mattmceuenevgenyl:  sounds good, I'll list you as co-contributor and steal liberally from your patchset :D  thanks14:22
evgenylmattmceuen: Haha, sure :D14:22
*** sreejithp has joined #airshipit14:25
*** obravo has joined #airshipit14:45
*** amalrajgenocidex has quit IRC14:48
*** alexanderhughes has quit IRC14:56
*** michael-beaver has joined #airshipit15:05
*** gkadam has quit IRC15:19
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Set 0640 permissions at time of creation  https://review.opendev.org/66631915:36
openstackgerritScott Hussey proposed airship/airship-in-a-bottle master: (fix) Fix br_netfilter disable persistence  https://review.opendev.org/66632015:38
openstackgerritMerged airship/election master: Adding Ryan van Wyk candidacy for TC role  https://review.opendev.org/66629915:46
CobHeadevgenyl: I pulled the logs - and you are correct. It says "FATAL: database "postgres" does not exist15:54
*** pgaxatte has quit IRC15:56
*** sri_ has quit IRC16:07
*** kranthikirang has joined #airshipit16:15
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Set 0640 permissions at time of creation  https://review.opendev.org/66631916:21
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Set 0640 permissions at time of creation  https://review.opendev.org/66631916:22
openstackgerritMatt McEuen proposed airship/treasuremap master: Update kernel to 4.15.0-46  https://review.opendev.org/66638116:23
rustyweiseI am quite nearly successful in deploying behind a proxy, alas MaaS seems to have changed CLI options from expectations and I'm getting this error in the 'maas-import-resources' pod -->16:27
rustyweisehttp://maas.io/16:28
rustyweiseargument COMMAND: invalid choice: 'admin' (choose from 'refresh', 'logout', 'login', 'list', 'apikey', 'createadmin', 'changepassword')16:28
rustyweise+ cur_val=16:28
rustyweise++ echo true16:28
rustyweise++ tr -d '"'16:28
rustyweise+ desired_val=true16:28
rustyweise+ [[ '' != true ]]16:28
rustyweise+ echo 'Setting MAAS option enable_http_proxy to true'16:28
rustyweise+ maas admin maas set-config name=enable_http_proxy value=true16:28
rustyweiseSetting MAAS option enable_http_proxy to true16:28
rustyweiseusage: maas [-h] COMMAND ...16:28
rustyweiseoptional arguments:16:28
rustyweise  -h, --help      show this help message and exit16:28
rustyweisedrill down:16:28
rustyweise  COMMAND16:28
rustyweise    login         Log in to a remote API, and remember its description and16:28
rustyweise                  credentials.16:28
rustyweise    list          List remote APIs that have been logged-in to.16:28
rustyweise    logout        Log out of a remote API, purging any stored credentials.16:28
rustyweise    refresh       Refresh the API descriptions of all profiles.16:28
rustyweise    apikey        Used to manage a user's API keys. Shows existing keys unless16:28
rustyweise                  --generate or --delete is passed.16:28
rustyweise    createadmin   Create a MAAS administrator account.16:28
rustyweise    changepassword16:28
rustyweise                  Change a MAAS user's password.16:28
rustyweisehttp://maas.io/16:28
openstackgerritMichael Beaver proposed airship/treasuremap master: Add new seaworthy-virt site  https://review.opendev.org/65601516:37
sthusseyProbably just need to delete the pod and let the job recreate it16:37
sthusseySometimes the login to the MAAS api fails, but the MAAS cli returns 0 anyway16:38
*** sreejithp has quit IRC16:43
*** sreejithp has joined #airshipit16:44
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Set 0640 permissions at time of creation  https://review.opendev.org/66631917:00
openstackgerritAlexander Hughes proposed airship/shipyard master: [WIP] Allow user to specify token as env var  https://review.opendev.org/66640217:05
rustyweiseThanks sthussey, I'll give that a try.17:36
rustyweiseit appears that this is the container running a maas cli and the options for the cli have changed out from under what airship was expecting17:41
rustyweiseI deleted the pod and it's still reporting the invalid choice error17:41
rustyweiseI need to leave for the night...chat more tomorrow, thanks guys17:42
*** mbeierl has quit IRC17:42
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Set 0640 permissions at time of creation  https://review.opendev.org/66631917:53
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Set 0640 permissions at time of creation  https://review.opendev.org/66631917:54
openstackgerritAlexander Hughes proposed airship/shipyard master: [WIP] Allow user to specify token as env var  https://review.opendev.org/66640218:03
openstackgerritIan Pittwood proposed airship/spyglass master: [WIP] Poetry for Spyglass  https://review.opendev.org/66641418:09
openstackgerritIan Pittwood proposed airship/spyglass-plugin-xls master: [WIP] Implements data objects in excel extractor  https://review.opendev.org/66209318:16
*** sreejithp has quit IRC18:16
*** sreejithp has joined #airshipit18:16
*** kranthikirang has quit IRC18:19
*** mbeierl has joined #airshipit18:26
openstackgerritIan Pittwood proposed airship/spyglass master: [WIP] Implements data object models  https://review.opendev.org/66209218:35
*** obravo has quit IRC18:39
openstackgerritNishant Kumar proposed airship/maas master: [WIP] Add release uuid annotation to POD spec  https://review.opendev.org/66641818:50
openstackgerritNishant Kumar proposed airship/drydock master: [WIP] Add release uuid annotation to POD spec  https://review.opendev.org/66641918:51
openstackgerritNishant Kumar proposed airship/deckhand master: [WIP] Add release uuid annotation to POD spec  https://review.opendev.org/66642018:51
openstackgerritNishant Kumar proposed airship/promenade master: [WIP] Add release uuid annotation to POD spec  https://review.opendev.org/66642118:53
openstackgerritEvgeniy L proposed airship/treasuremap master: Add new seaworthy-virt site  https://review.opendev.org/65601518:57
mattmceuenhey evgenyl, it looks like the seaworthy job didn't triggered for this PS yesterday: https://review.opendev.org/#/c/665729/19:04
mattmceuenany idea why?19:04
evgenylmattmceuen: usually we run it in a manual mode and the lab is used mostly for uplift/debugging, this is one of the reasons why we want to get all virtual installation in treasuremap to run on every patch. Here is a deployment for aiab https://jenkins.atlantafoundry.com/job/airship-in-a-bottle/289/19:10
mattmceuengot it, thanks!19:11
evgenylmattmceuen: Actually I see kskels re-enabled it. Not sure when this has happened though.19:12
mattmceuenyeah, I saw my PS from this morning get kicked off automatically, that's why I was confused - makes sense now though19:13
openstackgerritIan Pittwood proposed airship/spyglass master: [WIP] Implements data object models  https://review.opendev.org/66209219:23
openstackgerritIan Pittwood proposed airship/spyglass-plugin-xls master: [WIP] Implements data objects in excel extractor  https://review.opendev.org/66209319:28
*** mbeierl has quit IRC19:29
kskelsHaha yes - enabled it as I'm not using it anymore19:30
kskelsThought to see some feedback in general.. feel free to configure it as needed.19:31
openstackgerritIan Pittwood proposed airship/spyglass master: [WIP] Implements data object models  https://review.opendev.org/66209219:40
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Set 0640 permissions at time of creation  https://review.opendev.org/66631919:48
openstackgerritIan Pittwood proposed airship/spyglass master: [WIP] Implements data object models  https://review.opendev.org/66209219:53
openstackgerritIan Pittwood proposed airship/spyglass-plugin-xls master: [WIP] Implements data objects in excel extractor  https://review.opendev.org/66209319:56
openstackgerritMerged airship/treasuremap master: Introduce CoreDNS DaemonSet  https://review.opendev.org/66572919:58
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Set 0640 permissions at time of creation  https://review.opendev.org/66631920:04
openstackgerritIan Pittwood proposed airship/spyglass master: Implements data object models  https://review.opendev.org/66209220:05
openstackgerritIan Pittwood proposed airship/spyglass-plugin-xls master: Implements data objects in excel extractor  https://review.opendev.org/66209320:06
*** mbeierl has joined #airshipit20:06
openstackgerritEvgeniy L proposed airship/treasuremap master: Add new seaworthy-virt site  https://review.opendev.org/65601520:07
openstackgerritDan Crank proposed airship/promenade master: [WIP] Use etcd encryption for webhook-apiserver  https://review.opendev.org/66576120:24
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Set 0640 permissions at time of creation  https://review.opendev.org/66631920:25
*** ian-pittwood has joined #airshipit20:47
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Set 0640 permissions at time of creation  https://review.opendev.org/66631920:54
*** dirk has quit IRC20:55
openstackgerritIan Pittwood proposed airship/spyglass master: Disable voting on openSUSE  https://review.opendev.org/66644120:56
openstackgerritAlexander Hughes proposed airship/pegleg master: Make OpenSuse build non-voting  https://review.opendev.org/66644220:56
openstackgerritIan Pittwood proposed airship/spyglass master: Disable voting on openSUSE  https://review.opendev.org/66644120:58
openstackgerritDan Crank proposed airship/promenade master: [WIP] Use etcd encryption for webhook-apiserver  https://review.opendev.org/66576121:00
*** sreejithp has quit IRC21:02
*** sreejithp_ has joined #airshipit21:03
*** ian-pittwood has quit IRC21:23
openstackgerritEvgeniy L proposed airship/treasuremap master: Add new seaworthy-virt site  https://review.opendev.org/65601521:29
*** howell has quit IRC21:35
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP] Set 0640 permissions at time of creation  https://review.opendev.org/66631921:57
openstackgerritMatt McEuen proposed airship/treasuremap master: Incorporate HA PostgreSQL  https://review.opendev.org/65766722:13
*** sreejithp_ has quit IRC22:31
*** kskels has quit IRC22:46
openstackgerritMerged airship/treasuremap master: [aiab] Fix a path to scripts directory  https://review.opendev.org/66499322:51
openstackgerritMerged airship/treasuremap master: [aiab] Add a site linting gate to Zuul  https://review.opendev.org/66500723:04
*** aaronsheffield has quit IRC23:17
*** sthussey has quit IRC23:41

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!