| *** cheng1 has joined #airshipit | 01:03 | |
| *** cheng1 has quit IRC | 02:29 | |
| *** cheng1 has joined #airshipit | 02:36 | |
| *** roman_g has quit IRC | 02:58 | |
| *** cheng1 has quit IRC | 03:16 | |
| *** juhak has quit IRC | 03:53 | |
| *** juhak has joined #airshipit | 03:54 | |
| *** cheng1 has joined #airshipit | 03:59 | |
| *** cheng1_ has joined #airshipit | 05:41 | |
| *** cheng1 has quit IRC | 05:43 | |
| *** cheng1__ has joined #airshipit | 06:43 | |
| *** cheng1_ has quit IRC | 06:46 | |
| *** Nishant_ has quit IRC | 06:54 | |
| *** cheng1__ has quit IRC | 07:04 | |
| *** cheng1 has joined #airshipit | 07:09 | |
| *** mbologna has quit IRC | 07:26 | |
| *** mbologna has joined #airshipit | 07:27 | |
| *** pkaralis has joined #airshipit | 08:05 | |
| *** dimitris_ has joined #airshipit | 08:15 | |
| georgk | cheng1: hi! | 08:25 |
|---|---|---|
| cheng1 | georgk, hi | 08:25 |
| georgk | thanks for your reviews! very helpful | 08:26 |
| georgk | I just pushed another update | 08:26 |
| cheng1 | sure, I am going to have a look today | 08:26 |
| cheng1 | seems you are working on this akraino pb https://wiki.akraino.org/display/AK/Support+of+OVS-DPDK+in+Airship | 08:27 |
| georgk | yes, those two are connected | 08:28 |
| georgk | actually, I am more intersted in the upstream work in Airship, but OVS dpdk is supposed to be re-used here as well | 08:28 |
| cheng1 | seems need patches in several project, openstack-helm, treasuremap. Can I know what's the progress? If anything I can help | 08:30 |
| georgk | I was about to ask if you are intersted :-) | 08:32 |
| cheng1 | Yes, if anything need to help, please let me know. either openstack-helm or airship :) | 08:33 |
| georgk | well, to my current understanding, there are basically two open work areas left: i) getting the Neutron config right (working on this right now) and ii) figuring out which image to use (and not my personal one) | 08:33 |
| georgk | that's very nice, I will | 08:33 |
| *** nick_kar has joined #airshipit | 08:35 | |
| cheng1 | For neutron config task, I think we can keep going with your patch. Let more people review it :) | 08:36 |
| cheng1 | Regarding the image, the public kolla image doesn't work right? | 08:37 |
| cheng1 | I remember you said there is a problem with that image | 08:38 |
| cheng1 | in your commit msg | 08:39 |
| georgk | yes, I haven't figured out why yet, though. | 08:45 |
| georgk | I could post more detailed logs later today if you are interested. | 08:46 |
| cheng1 | sure | 08:47 |
| georgk | in a nutshell, there is a memory allocation problem when attaching the DPDK device to OVS. This works fine howver, in "my" image, which is just a source build of the kolla image (no changes) | 08:47 |
| cheng1 | your local built image works right? | 08:47 |
| georgk | yes | 08:47 |
| cheng1 | So seems we need to find why the published image doesn't work, then may make some change of the building script. | 08:52 |
| cheng1 | Once we are there, we can use the public image, am I right? | 08:52 |
| georgk | that seems to be way to go | 08:56 |
| openstackgerrit | Hemanth Nakkina proposed openstack/airship-in-a-bottle master: Add seccomp profile on genesis node in multinode gate https://review.openstack.org/644824 | 09:17 |
| *** Nishant_ has joined #airshipit | 09:26 | |
| cheng1 | I am trying to deploy an openstack using openstack-helm, with dpdk enabled. Will see what problem I can meet? georgk | 09:31 |
| cheng1 | will see if I meet the same problem with ovs-dpdk image | 09:32 |
| georgk | ok, thank you | 09:32 |
| georgk | I was also just looking at this; https://github.com/openstack/openstack-helm/tree/master/tools/images/openvswitch | 09:32 |
| georgk | maybe we should update this one | 09:33 |
| cheng1 | yes, as I know, openstack-helm is trying to use self built images | 09:34 |
| cheng1 | But seems it doesn't support dpdk yet. We can make effort on it | 09:35 |
| openstackgerrit | Hemanth Nakkina proposed openstack/airship-in-a-bottle master: Add seccomp profile on genesis node in multinode gate https://review.openstack.org/644824 | 09:43 |
| *** cheng1 has quit IRC | 09:44 | |
| *** roman_g has joined #airshipit | 10:47 | |
| openstackgerrit | Smruti Soumitra Khuntia proposed openstack/airship-armada master: End user logging for audit traceabilty https://review.openstack.org/638124 | 11:18 |
| openstackgerrit | Smruti Soumitra Khuntia proposed openstack/airship-in-a-bottle master: Document End user optional header https://review.openstack.org/642999 | 11:22 |
| georgk | roman_g: hi! | 11:55 |
| *** juhak has quit IRC | 12:52 | |
| *** juhak has joined #airshipit | 12:52 | |
| *** aaronsheffield has joined #airshipit | 12:52 | |
| *** kranthikirang has joined #airshipit | 12:54 | |
| kranthikirang | mattmceuen: Good Morning; Did you get a chance to look into my messages? | 12:56 |
| portdirect | georgk/cheng1: for ovs dpdk support the image will need updated, please feel free to do so | 12:56 |
| *** dims has joined #airshipit | 13:13 | |
| *** ab2434_ has joined #airshipit | 13:32 | |
| *** michael-beaver has joined #airshipit | 13:44 | |
| *** michaelbeaver has joined #airshipit | 13:48 | |
| *** michael-beaver has quit IRC | 13:52 | |
| evrardjp | georgk: cheng1 for osh images you can edit them in openstack-helm-images repo. It's all using openstack-infra tooling for publishing the images. | 13:56 |
| evrardjp | patches welcomed | 13:57 |
| georgk | evrardjp: thanks for the pointer | 13:57 |
| evrardjp | yw | 13:57 |
| evrardjp | if you have further questions, you should discuss it in #openstack-helm imo | 13:59 |
| mattmceuen | kranthikirang: have not yet, still planning on taking a look though! | 14:09 |
| mattmceuen | All: here's the agenda for our meeting in a couple hours, please add anything you'd like to discuss: https://etherpad.openstack.org/p/airship-meeting-2019-03-26 | 14:10 |
| openstackgerrit | Luna Das proposed openstack/airship-promenade master: Add init container to load custom apparmor profile for kube-proxy https://review.openstack.org/647749 | 14:12 |
| mattmceuen | kranthikirang: were you getting those errors on a fresh greenfield deployment of Airship? Or were you updating an existing deployment? | 14:16 |
| mattmceuen | HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Operation cannot be fulfilled on configmaps \"airship-ucp-mariadb-mariadb-state\": the object has been modified; please apply your changes to the latest version and try again","reason":"Conflict","details":{"name":"airship-ucp-mariadb-mariadb-state","kind":"configmaps"},"code":409} | 14:17 |
| mattmceuen | ^ I don't believe I've seen that one before | 14:17 |
| kranthikirang | mattmceuen: It was a fresh install of genesis node; I didn't have anything | 14:17 |
| kranthikirang | mattmceuen: However, I had to clear all the pods and images before re-run the genesis | 14:18 |
| kranthikirang | mattmeceuen: My main concern is rabbitmq timeout; Looks like that's happening because of too much CPU utilization | 14:18 |
| kranthikirang | mattmeceuen: http://paste.openstack.org/show/748392/ | 14:19 |
| kranthikirang | mattmeceuen: I guess we are polling k8s API very frequently and if the chart deployment fails armada starts deploying them again and never stops | 14:20 |
| kranthikirang | mattmeceuen: Please correct me | 14:20 |
| openstackgerrit | Drew Walters proposed openstack/airship-shipyard master: tools: Pin HTK to Helm 2.10.0 compatible version https://review.openstack.org/647751 | 14:22 |
| mattmceuen | I'm not sure about this rabbitmq error: | 14:23 |
| mattmceuen | 2019-03-25 21:19:49.470 [info] <0.244.0> Waiting for Mnesia tables for 30000 ms, 9 retries left | 14:23 |
| mattmceuen | 2019-03-25 21:20:19.478 [warning] <0.244.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]} | 14:23 |
| mattmceuen | kranthikirang: have you seen it before, related to CPU utilization? | 14:23 |
| kranthikirang | mattmceuen: I am pretty sure; I copied the k8s manifests and tried it in another k8s cluster. It came up in 10 seconds | 14:24 |
| mattmceuen | armada does try four times total | 14:24 |
| kranthikirang | mattmceuen: that's not what I am seeing; I see Armada tying to deploy the same charts from yesterday :) | 14:24 |
| kranthikirang | mattmceuen: I guess at least to isloate the issue, I guess we need to increase RedinessProbe values for mariadb and rabbitmq pods; | 14:25 |
| kranthikirang | mattmceuen: Also in ./genesis.sh increase polling seconds? | 14:26 |
| kranthikirang | while true; do | 14:26 |
| kranthikirang | if [[ -e /etc/kubernetes/manifests/bootstrap-armada.yaml ]]; then | 14:26 |
| kranthikirang | sleep 30 | 14:26 |
| kranthikirang | kubectl get pods --all-namespaces || echo "Could not get current pod status." | 14:26 |
| kranthikirang | else | 14:26 |
| kranthikirang | log Armada bootstrap manifest deployed | 14:26 |
| kranthikirang | break | 14:26 |
| kranthikirang | fi | 14:26 |
| kranthikirang | done | 14:26 |
| kranthikirang | Also, do we have a clean up script to run before re-running ./genesis.sh? | 14:27 |
| mattmceuen | it's actually shipyard that does the retries, but it should only be four tries total. I believe there's a cleanup script but am not finding it; evgenyl, where does that cleanup script live? | 14:30 |
| mattmceuen | I don't think a 30-second sleep period between kubectls should be a problem - if it is, then we have bigger problems :) | 14:31 |
| evgenyl | mattmceuen: Are you looking for this one? https://github.com/openstack/airship-promenade/blob/master/tools/cleanup.sh | 14:31 |
| mattmceuen | ahh, that's it. I was looking in the wrong repo. Aren't there additional cleanup steps for cleaning up ceph disks - do you know if we have that documented? | 14:32 |
| evgenyl | mattmceuen: Here how airship-seaworthy does it https://github.com/openstack/airship-treasuremap/blob/master/tools/gate/Jenkinsfile#L174-L177 , but I'm not sure if this step is required unless ceph configs are changed. | 14:35 |
| evgenyl | Here is* | 14:35 |
| openstackgerrit | Lev Morgan proposed openstack/airship-pegleg master: Fix multiple I/O issues in cert generation https://review.openstack.org/643678 | 14:36 |
| mattmceuen | kranthikirang: the readiness probes for mariadb and rabbitmq aren't configurable, but they're not crazy fast either, e.g.: https://github.com/openstack/openstack-helm-infra/blob/master/mariadb/templates/statefulset.yaml#L160-L166 | 14:36 |
| mattmceuen | awesome - thanks evgenyl | 14:36 |
| kranthikirang | mattmceuen: initialDelaySeconds: 30, probably a bit low if this CPU utilization | 14:38 |
| openstackgerrit | Luna Das proposed openstack/airship-promenade master: Add init container to load custom apparmor profile for kube-proxy https://review.openstack.org/647749 | 14:38 |
| kranthikirang | and ideal for normal situations; if there is no way to pass the variables then we have to see other options | 14:38 |
| kranthikirang | Thank you for the clean up script | 14:39 |
| kranthikirang | mattmceuen: I can write script for ceph-osd and ceph-journal | 14:39 |
| mattmceuen | I think that would be awesome kranthikirang - that would make it more reusable, and could go in the tools/ folder in airship-treasuremap. evgenyl do you have any other thoughts on a clean-up script for ceph? | 14:42 |
| mattmceuen | kranthikirang: what do you see using up most of the CPU on the host? We do run airship in VMs and such so I'm optimistic that this is something we can fix | 14:43 |
| kranthikirang | mattmceuen: I guess you have missed this link earlier http://paste.openstack.org/show/748392/ | 14:44 |
| kranthikirang | mattmceuen: its a too much usage | 14:44 |
| kranthikirang | I guess script like this would be sufficient; | 14:45 |
| kranthikirang | for disk in c d b; do | 14:45 |
| kranthikirang | sudo parted -s /dev/sd$disk mklabel gpt | 14:45 |
| kranthikirang | done | 14:45 |
| kranthikirang | sudo rm -rf /var/lib/openstack-helm/ceph | 14:45 |
| kranthikirang | however, depending on how many disks, people have to modify the inputs | 14:46 |
| openstackgerrit | Drew Walters proposed openstack/airship-shipyard master: Shipyard and Airflow: Add pod/container security context https://review.openstack.org/639195 | 14:46 |
| mattmceuen | kranthikirang: curious whether `journalctl -u kubelet` gives any clues? what do `docker ps | wc` and `docker ps -a | wc` show? | 14:49 |
| openstackgerrit | Lev Morgan proposed openstack/airship-pegleg master: Added document wrapping command https://review.openstack.org/644637 | 14:49 |
| kranthikirang | mattmceuen: http://paste.openstack.org/show/748395/ | 14:50 |
| kranthikirang | mattmceuen: journalctl -u kubelet has so many logs | 14:50 |
| mattmceuen | can you grab the full text of some of those " Failed to get status for pod" logs? What version of docker and k8s do you have running? | 14:52 |
| openstackgerrit | Smruti Soumitra Khuntia proposed openstack/airship-drydock master: End user logging for audit traceabilty https://review.openstack.org/638115 | 14:53 |
| kranthikirang | mattmceuen: ./genesis.sh installing docker-ce 17.03 | 14:55 |
| mattmceuen | also `iostat -dx 5` | 14:56 |
| *** michael-beaver has joined #airshipit | 15:06 | |
| kranthikirang | mattmceuen: here is the kubelet "Failed to get status" logs | 15:06 |
| kranthikirang | http://paste.openstack.org/show/748397/ | 15:06 |
| kranthikirang | mattmceuen: http://paste.openstack.org/show/748398/ | 15:07 |
| kranthikirang | mattmeceuen: above is for iostat | 15:07 |
| evgenyl | kranthikirang: Just curios, what version of treasuremap do you deploy? | 15:07 |
| evgenyl | *curious | 15:08 |
| kranthikirang | evgenyl: I have just followed the instructions; How can I check? | 15:08 |
| evgenyl | kranthikirang: Go to airship-treasuremap repo which cloned earlier, and run `git branch -v` | 15:09 |
| *** michaelbeaver has quit IRC | 15:09 | |
| *** kaspars__ has quit IRC | 15:09 | |
| kranthikirang | evgenyl: git branch -v | 15:09 |
| kranthikirang | * master c497783 Merge "Use sudo in systemctl unmask call" | 15:09 |
| evgenyl | kranthikirang: Ok, this one should be good, it's `v19.03.06` tag, just wanted to confirm that it is not a current master, which is broken. | 15:11 |
| kranthikirang | evgenyl: I have HP Gen9 V4 servers; Actually those should be sufficient to run any loads | 15:11 |
| kranthikirang | evgenyl: Ok | 15:11 |
| kranthikirang | evgenyl: mattmceuen: Now I couldn't even call docker ps -a or kubectl version commands :) | 15:16 |
| mattmceuen | from your kubelet logs -- this guy `Get https://127.0.0.1:6553/api/v1/namespaces/kube-system/pods/bootstrap-armada-mtn21c-node-01: dial tcp 127.0.0.1:6553: connect: connection refused` means that armada is having trouble communicating with the apiserver, which is done via haproxy on the node | 15:18 |
| *** levmorgan has joined #airshipit | 15:19 | |
| kranthikirang | mattmceuen: OK | 15:19 |
| *** levmorgan has quit IRC | 15:20 | |
| kranthikirang | mattmceuen: Perhaps, some configuration is missed on genesis node; I have configured OAM interface first and configured all other VLANs including OAM on bond interface | 15:20 |
| mattmceuen | can you please look on each of your nodes (control and compute) and do `wc -l /etc/promenade/haproxy/haproxy.cfg` ? | 15:21 |
| evgenyl | kranthikirang: It should go through the localhost, so I don't think that it's something related to your net configs on this stage. | 15:21 |
| mattmceuen | kranthikirang: and if all the haproxy.cfgs are the same length, then paste an example of one of them | 15:21 |
| evgenyl | kranthikirang: I would also check that you have haproxy container running, `ps aux -H`. | 15:21 |
| kranthikirang | mattmceuen: at this point I have only genesis is being configured | 15:21 |
| *** michaelbeaver has joined #airshipit | 15:21 | |
| mattmceuen | ok cool | 15:21 |
| *** levmorgan_ has joined #airshipit | 15:22 | |
| kranthikirang | mattmceuen: here is the haproxy config in genesis node | 15:23 |
| kranthikirang | http://paste.openstack.org/show/748403/ | 15:23 |
| mattmceuen | There may be a clue in the haproxy logs or kube apiserver logs as to why the connection was refused - it's possible it's a configuration-related thing (which wouldn't show up in the logs) or it's possible haproxy or apiserver is denying the request for some reason | 15:24 |
| *** levmorgan_ has quit IRC | 15:24 | |
| *** levmorgan_ has joined #airshipit | 15:24 | |
| kranthikirang | mattmceuen: netstat -uanltp | grep 6553 | 15:24 |
| kranthikirang | tcp 0 0 0.0.0.0:6553 0.0.0.0:* LISTEN 50840/haproxy | 15:24 |
| kranthikirang | tcp 0 0 0.0.0.0:6553 0.0.0.0:* LISTEN 50840/haproxy | 15:24 |
| kranthikirang | tcp 0 0 127.0.0.1:37704 127.0.0.1:6553 ESTABLISHED 45080/controller-ma | 15:24 |
| kranthikirang | tcp 0 0 127.0.0.1:37744 127.0.0.1:6553 ESTABLISHED 17044/kubelet | 15:24 |
| kranthikirang | tcp 0 0 127.0.0.1:6553 127.0.0.1:37704 ESTABLISHED 50840/haproxy | 15:24 |
| kranthikirang | tcp 0 0 127.0.0.1:6553 127.0.0.1:37866 ESTABLISHED 50840/haproxy | 15:24 |
| kranthikirang | tcp 0 0 127.0.0.1:37812 127.0.0.1:6553 ESTABLISHED 21240/proxy | 15:25 |
| kranthikirang | tcp 0 0 127.0.0.1:6553 127.0.0.1:37744 ESTABLISHED 50840/haproxy | 15:25 |
| kranthikirang | tcp 0 87 127.0.0.1:37854 127.0.0.1:6553 ESTABLISHED 45351/scheduler | 15:25 |
| kranthikirang | tcp 0 0 127.0.0.1:6553 127.0.0.1:37854 ESTABLISHED 50840/haproxy | 15:25 |
| kranthikirang | tcp 0 0 127.0.0.1:6553 127.0.0.1:37812 ESTABLISHED 50840/haproxy | 15:25 |
| kranthikirang | tcp 0 0 127.0.0.1:37866 127.0.0.1:6553 ESTABLISHED 45080/controller-ma | 15:25 |
| *** michael-beaver has quit IRC | 15:25 | |
| *** levmorgan_ has quit IRC | 15:26 | |
| *** levmorgan has joined #airshipit | 15:26 | |
| *** levmorgan has quit IRC | 15:28 | |
| kranthikirang | mattmceuen: I can't access docker container currently; I will try to deploy again and see why haproxy is not working | 15:30 |
| kranthikirang | mattmceuen: Meanwhile in armada <-> tiller container, is there anyway we could interact with tiller using helm commands? | 15:31 |
| mattmceuen | I eyeballed your haproxy.cfg against a known-good one deployed in a treasuremap genesis node, and they look the same, so I don't think that's the issue | 15:32 |
| mattmceuen | re: interact with tiller - sure can | 15:32 |
| mattmceuen | we often use the helm cli for debugging purposes, and the shipyard->armada api for driving changes | 15:33 |
| kranthikirang | mattmceuen: How can I use helm cli; I see armada-tiller container is actually deploying charts but there is no tiller pod running in kube-system namespace | 15:34 |
| kranthikirang | mattmceuen: If there is no haproxy problem, what do you think is the problem? | 15:34 |
| *** levmorgan has joined #airshipit | 15:34 | |
| *** levmorgan has quit IRC | 15:36 | |
| *** levmorgan has joined #airshipit | 15:37 | |
| mattmceuen | ah, that's the bootstrap tiller, which gets replaced by a standalone tiller during the genesis process. You could probably deploy another tiller into the cluster for debugging purposes using `helm init` | 15:40 |
| mattmceuen | I will have to think about this a bit more kranthikirang | 15:40 |
| kranthikirang | mattmceuen: ok | 15:41 |
| mattmceuen | I would say trying to get access to the haproxy and apiserver docker container logs and looking for errors is the next step | 15:41 |
| kranthikirang | mattmceuen: so communication should happen on 127.0.0.1 for haproxy while backend is on s10.23.22.11 ? | 15:41 |
| openstackgerrit | Ahmad Mahmoudi proposed openstack/airship-maas master: [FIX] - Fixed maas-rack reschedule issue https://review.openstack.org/642174 | 15:42 |
| openstackgerrit | PRATEEK REDDY DODDA proposed openstack/airship-divingbell master: Implement Security Context for Divingbell https://review.openstack.org/641706 | 15:43 |
| openstackgerrit | PRATEEK REDDY DODDA proposed openstack/airship-deckhand master: Implement Security Context for Deckhand https://review.openstack.org/639204 | 15:44 |
| mattmceuen | kranthikirang: yep - that 10.23.22.11 IP is the calico IP of the genesis node | 15:45 |
| mattmceuen | which is where the apiserver should be listening | 15:46 |
| kranthikirang | mattmceuen: and kubelet communicate to haproxy on 127.0.0.1 | 15:46 |
| mattmceuen | yep exactly | 15:48 |
| *** michaelbeaver has quit IRC | 15:48 | |
| *** michael-beaver has joined #airshipit | 15:49 | |
| openstackgerrit | Luna Das proposed openstack/airship-promenade master: Add init container to load custom apparmor profile for kube-proxy https://review.openstack.org/647749 | 15:55 |
| mattmceuen | #startmeeting airship | 16:00 |
| mattmceuen | #topic Rollcall | 16:00 |
| openstack | Meeting started Tue Mar 26 16:00:13 2019 UTC and is due to finish in 60 minutes. The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: airship)" | 16:00 | |
| openstack | The meeting name has been set to 'airship' | 16:00 |
| mattmceuen | Hey GM everybody | 16:00 |
| *** openstack changes topic to "Rollcall (Meeting topic: airship)" | 16:00 | |
| dwalt | o/ GM! | 16:00 |
| michael-beaver | o/ GM | 16:00 |
| mattmceuen | Here's our agenda for today: https://etherpad.openstack.org/p/airship-meeting-2019-03-26 | 16:00 |
| Nishant_ | o/ | 16:00 |
| mattmceuen | please add anythingthing you'd like to discuss, as we wait for folks to join | 16:00 |
| evgenyl | Hi everyone! | 16:00 |
| mattmceuen | Alright, let's get started | 16:02 |
| mattmceuen | #topic Chart document naming and labelling conventions | 16:02 |
| *** openstack changes topic to "Chart document naming and labelling conventions (Meeting topic: airship)" | 16:02 | |
| *** AlexanderHughes has joined #airshipit | 16:02 | |
| levmorgan | o/ | 16:02 |
| mattmceuen | deckhand allows us to have multiple layers (global, type, site) of documents, and is very flexible in how you approach naming and labelling docs | 16:03 |
| openstackgerrit | Luna Das proposed openstack/airship-promenade master: Add init container to load custom apparmor profile for kube-proxy https://review.openstack.org/647749 | 16:03 |
| mattmceuen | I think it would be good to come up with a *convention* for how we approach parent-child document naming and labelling, and then apply that convention across treasuremap | 16:03 |
| mattmceuen | as we have a few different conventions that have grown organically | 16:04 |
| mattmceuen | First point: what should the documents themselves be named | 16:04 |
| mattmceuen | If we have a document `foo` defined at the global level, I've seen it named either `foo` or `foo-global` | 16:05 |
| mattmceuen | I'd suggest we just call it `foo` and then | 16:06 |
| mattmceuen | 1) use labels to designate it as global | 16:06 |
| mattmceuen | 2) when re replace at a child level, just retain the same name e.g. `foo` | 16:06 |
| mattmceuen | 3) when we layer without replacement at a child level, give the result a derivitive name | 16:06 |
| mattmceuen | This convention would keep the focus in the "name" on the intent of the content, rather than "where it comes from" | 16:06 |
| mattmceuen | What do y'all think | 16:06 |
| dwalt | When using labels to designate the document as global, are we talking about appending global to the document name label, or adding a separate label? | 16:07 |
| dwalt | Currently, I think we do the former | 16:07 |
| evgenyl | Hmm, e.g. if I have `keystone` what would be a name for the override? `keystone-seaworthy`? | 16:07 |
| mattmceuen | dwalt: yep, we could give it a label of `name: foo-global` which seems to be the most common convention | 16:07 |
| mattmceuen | evgenyl: yep I think something like that makes sense | 16:07 |
| mattmceuen | `keystone-<type or site name>` | 16:08 |
| * dwalt Does anyone like the idea of moving that convention to something like: | 16:08 | |
| dwalt | Sorry for emote | 16:08 |
| dwalt | labels: | 16:08 |
| dwalt | name: keystone | 16:08 |
| dwalt | layer: global | 16:08 |
| dwalt | That way, the name label is consistent with the document name | 16:09 |
| mattmceuen | that is indeed Point #2 to figure out! I don't have strong opinions but have seen both ways :) | 16:09 |
| dwalt | Sorry for skipping ahead :) | 16:09 |
| mattmceuen | No worries, I'm hoping that means we're just aligned on #1; speak now or forever hold your peace on doc names just being `foo` or `keystone` etc | 16:10 |
| Nishant_ | I like #1 i.e. global: keystone-global; type: keystone-type; site: keystone-site | 16:10 |
| dwalt | ++ | 16:10 |
| *** kranthikirang has quit IRC | 16:11 | |
| mattmceuen | I am leaning toward that too | 16:14 |
| evgenyl | Is the suggestion to have `name: keystone-global` label for globals and use different postfixes for different layers? | 16:15 |
| Nishant_ | Yes, that's what I understand | 16:15 |
| evgenyl | And without having `name: keystone` labels at all? | 16:15 |
| *** michaelbeaver has joined #airshipit | 16:16 | |
| evgenyl | I like it because it is so much easier to grep and understand what the child exactly overrides :) | 16:16 |
| mattmceuen | Agree | 16:17 |
| mattmceuen | May require revisiting when we implement service layers | 16:17 |
| openstackgerrit | Luna Das proposed openstack/airship-promenade master: Add init container to load custom apparmor profile for kube-proxy https://review.openstack.org/647749 | 16:17 |
| * mattmceuen mythical service layers | 16:18 | |
| mattmceuen | Ok awesome -- I think we're ready for Point #3 to sort out | 16:18 |
| mattmceuen | Should our default position be to insert labels into documents, so that there is always a hook there for documents to override them at the type or site level? | 16:18 |
| *** michael-beaver has quit IRC | 16:19 | |
| mattmceuen | Or, should we only add labels into a document if and when we decide it should be overridden? | 16:19 |
| evgenyl | I would consider always adding the labels for the documents, it may help to reduce the amount of forks. | 16:20 |
| Nishant_ | With this I am leaning towards the former | 16:20 |
| dwalt | I personally like always having them available to reduce the amounts of extra commits/work when adding new site documents | 16:20 |
| mattmceuen | My opinion is - let's define labels at the global / type level in general. Otherwise, as more folks use treasuremap as a reference for their own sites, we'd be restricting the ways they could extend things | 16:20 |
| mattmceuen | I think we all agree! | 16:20 |
| michaelbeaver | ++ | 16:20 |
| mattmceuen | I think we can leave them out at the site level, since they wouldn't be used for anything | 16:21 |
| evgenyl | ++ only for globals and types | 16:21 |
| mattmceuen | Those are all the convention-y things I wanted to sort out; am I forgetting any other things we should sort out while we're discussing this? | 16:21 |
| dwalt | I think we hit it all | 16:22 |
| mattmceuen | dwalt had put together a user story for this: https://storyboard.openstack.org/#!/story/2004354 | 16:22 |
| mattmceuen | dwalt could you please summarize the results of this discussion in that story? Then we could update the treasuremap peggles accordingly as part of that story | 16:22 |
| dwalt | #action dwalt update treasuremap labels convention story | 16:23 |
| dwalt | will do! | 16:23 |
| mattmceuen | thanks! | 16:23 |
| mattmceuen | alright, moving on: | 16:23 |
| mattmceuen | #topic Season of Docs | 16:23 |
| *** openstack changes topic to "Season of Docs (Meeting topic: airship)" | 16:23 | |
| mattmceuen | last week, roman_g had brought up that google has a new "season of docs" program that takes after their "summer of code" program | 16:24 |
| *** michael-beaver has joined #airshipit | 16:24 | |
| mattmceuen | I looked into this a bit | 16:24 |
| mattmceuen | It's a way for a technical doc writer to get extra experience for their resume while also helping an open source project | 16:24 |
| mattmceuen | OSS organizations that are interested have to apply by April 23rd, and then there's a selection process for both orgs and for individual writers | 16:25 |
| mattmceuen | Then the program itself takes place in the Fall IIRC | 16:25 |
| mattmceuen | I think this is a great idea, and there's no harm in trying for it | 16:26 |
| evgenyl | Docs is always something that we need help with :) | 16:26 |
| mattmceuen | For our application, we'd need a selection of potential projects for the technical document writer to do - we can't wait till the Fall to come up with work :) | 16:26 |
| mattmceuen | And I think compelling projects would give a better chance of being selected | 16:26 |
| mattmceuen | So, keeping in mind that this will be in the Fall (i.e. post-1.0) | 16:27 |
| mattmceuen | Let's brainstorm some ideas for document authoring | 16:27 |
| *** michaelbeaver has quit IRC | 16:27 | |
| evgenyl | We need more user-facing docs, and better organization/referencing of all the projects that we use. | 16:27 |
| dwalt | I think we need more integration diagrams | 16:27 |
| dwalt | especially in the individual components. For example, how Armada uses tiller | 16:28 |
| mattmceuen | Oh my - know what, we're going to be moving toward the k8s cluster API in the future, which would require user-facing docs and diagrams | 16:29 |
| mattmceuen | Would be great to be ahead of the curve from a documentation perspective on that, rather than behind the curve | 16:29 |
| mattmceuen | and I bet things related to the k8s cluster API would be favorable to google :) | 16:29 |
| mattmceuen | evgenyl and kaspars are planning on creating some new "getting started" stripped down reference yamls for bare metal. I bet a good guide walking people through that would be valuable? | 16:31 |
| evgenyl | mattmceuen: Do you want to have some generic k8s cluster api docs? Or something airship related? | 16:31 |
| mattmceuen | def something airship related -- i.e. how airship will integrate with cluster api, how to configure it, etc | 16:31 |
| mattmceuen | side note - if anyone wants to discuss airship-cluster api integration, come to Rodolfo's call on Thursday :) | 16:32 |
| mattmceuen | I will take some notes based on the above, in the agenda | 16:32 |
| dwalt | Sorry if it has already been said, but how long does the program last? | 16:33 |
| mattmceuen | Let's think about this some more, and then revisit next week, since we have a little runway | 16:33 |
| mattmceuen | https://developers.google.com/season-of-docs/docs/timeline | 16:33 |
| mattmceuen | the actual meat of the program is Sept 2 - Nov 22 | 16:33 |
| dwalt | cool, just wanted to understand how much time there is for ramping-up. Thx mattmceuen | 16:34 |
| mattmceuen | hey hogepodge - will want to sync up with you on this at some point too; it looks to me like it would be most appropriate for the Foundation to formally submit the application, so we should make sure there are no gotchas with that | 16:34 |
| mattmceuen | Anything else on the doc front to discuss? | 16:35 |
| mattmceuen | dwalt, michael-beaver, and I divvied up a lot of the dev guide user story tasks; I've been looking for some time to get started on that | 16:35 |
| mattmceuen | #topic roundtable | 16:36 |
| *** openstack changes topic to "roundtable (Meeting topic: airship)" | 16:36 | |
| evgenyl | Just a small announcement, we got fixed AIAB https://review.openstack.org/#/c/644634/ | 16:36 |
| dwalt | mattmceuen: thanks! I've made some good progress, just holding off for the template before I push | 16:36 |
| evgenyl | So gates should be green now, unless there are some problems with the infra (which have a lot recently) | 16:36 |
| dwalt | evgenyl: that's great! | 16:37 |
| evgenyl | So before merging something to AIAB, check the comments if there are failures related to AIAB deployment. | 16:37 |
| evgenyl | And multinode gate is still failing, so we need help from somebody to take a look into that. | 16:38 |
| dwalt | evgenyl: will do. Is there anything we can do to have the results published near Zuul? Or does it need to be a gate | 16:38 |
| dwalt | I have seen other third party CIs do so on other OpenStack projects, not sure if it's feasible | 16:38 |
| mattmceuen | evgenyl: that's awesome! thanks for your work on getting AIAB back up and running :) | 16:39 |
| evgenyl | dwalt: We can definitely look into that, any help would be appreciated, there are quite a few problems with Jenkins stability :( | 16:40 |
| openstackgerrit | Ahmad Mahmoudi proposed openstack/airship-maas master: [FIX] - Fixed maas-rack reschedule issue https://review.openstack.org/642174 | 16:41 |
| evgenyl | That is it from me for roundtable. | 16:41 |
| dwalt | evgenyl: I'll look into it. Would it be better to leave it toggle-able in the meantime? | 16:41 |
| mattmceuen | It would be great if we could get the results published in a non-voting way back up to the green/red CI section at the top of the PS. Hopefully that isn't a big effort, but I haven't done it before | 16:41 |
| dwalt | #action dwalt explore publishing AT&T CI results in PS tables | 16:42 |
| mattmceuen | ty dwalt | 16:42 |
| dwalt | sure thing | 16:42 |
| mattmceuen | any other roundtable topics to discuss today? | 16:42 |
| mattmceuen | In that case - great meeting, thanks everyone for joining | 16:43 |
| mattmceuen | Have a great week. | 16:43 |
| mattmceuen | #endmeeting | 16:43 |
| dwalt | thanks all! | 16:43 |
| *** openstack changes topic to "airshipit.org || General Review Dashboard: https://review.openstack.org/#/q/project:%255Eopenstack/airship.*+status:open,n,z" | 16:43 | |
| evgenyl | Thanks! | 16:43 |
| openstack | Meeting ended Tue Mar 26 16:43:15 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:43 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-03-26-16.00.html | 16:43 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-03-26-16.00.txt | 16:43 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-03-26-16.00.log.html | 16:43 |
| levmorgan | Till next week! | 16:43 |
| *** levmorgan has quit IRC | 16:43 | |
| *** AlexanderHughes has quit IRC | 16:44 | |
| openstackgerrit | Luna Das proposed openstack/airship-promenade master: Add init container to load custom apparmor profile for kube-proxy https://review.openstack.org/647749 | 16:46 |
| openstackgerrit | Drew Walters proposed openstack/airship-armada master: tools: Update Helm to v2.13.1 https://review.openstack.org/647812 | 16:57 |
| *** kranthikirang has joined #airshipit | 17:09 | |
| openstackgerrit | Luna Das proposed openstack/airship-promenade master: Add init container to load custom apparmor profile for kube-proxy https://review.openstack.org/647749 | 17:12 |
| openstackgerrit | Drew Walters proposed openstack/airship-deckhand master: tools: Update Helm to v2.13.1 https://review.openstack.org/647822 | 17:16 |
| openstackgerrit | Drew Walters proposed openstack/airship-drydock master: tools: Update Helm to v2.13.1 https://review.openstack.org/647825 | 17:16 |
| openstackgerrit | Drew Walters proposed openstack/airship-divingbell master: tools: Update Helm to v2.13.1 https://review.openstack.org/647826 | 17:17 |
| openstackgerrit | Drew Walters proposed openstack/airship-maas master: tools: Update Helm to v2.13.1 https://review.openstack.org/647827 | 17:17 |
| openstackgerrit | Drew Walters proposed openstack/airship-promenade master: tools: Update Helm to v2.13.1 https://review.openstack.org/647828 | 17:17 |
| openstackgerrit | Drew Walters proposed openstack/airship-shipyard master: tools: Update Helm to v2.13.1 https://review.openstack.org/647829 | 17:17 |
| openstackgerrit | Luna Das proposed openstack/airship-promenade master: Add init container to load custom apparmor profile for kube-proxy https://review.openstack.org/647749 | 17:19 |
| dwalt | evgenyl: It looks like in order to publish our results in the table, Jenkins needs to comment using HTML that matches | 17:42 |
| dwalt | For example: https://review.openstack.org/627941/ | 17:42 |
| dwalt | The Dell EMC jobs just post their results wrapped in HTML | 17:42 |
| evgenyl | dwalt: Job definition for AIAB is stored here https://github.com/att-comdev/cicd/tree/master/integration/airship/airship-in-a-bottle I'm wondering if there is some ready to use plugin that we can use for that... | 17:45 |
| openstackgerrit | Merged openstack/airship-in-a-bottle master: Fixed sudo env vars for apt https://review.openstack.org/647598 | 17:50 |
| dwalt | evgenyl: is that where it actually publishes a comment? | 17:51 |
| evgenyl | dwalt: I know that logs are published here https://github.com/att-comdev/cicd/blob/master/integration/airship/airship-in-a-bottle/Jenkinsfile#L42-L43 and there is some magic in gerrit plugins that publishes the links, sorry for not giving more details, I'm not really familiar with all this groovy/jenkins stuff :( | 17:54 |
| dwalt | Interesting. I thought that just went here https://github.com/att-comdev/cicd/blob/master/vars/artifactory.groovy#L28 | 17:57 |
| dwalt | I'll keep searching | 17:58 |
| openstackgerrit | PRATEEK REDDY DODDA proposed openstack/airship-armada master: Implement Security Context for Armada https://review.openstack.org/639207 | 18:46 |
| openstackgerrit | Felipe Monteiro proposed openstack/airship-pegleg master: tests: Add CLI tests for site encrypt action https://review.openstack.org/614303 | 18:55 |
| openstackgerrit | Felipe Monteiro proposed openstack/airship-pegleg master: zuul: Remove redundant airship-pegleg-tox-py36 job. https://review.openstack.org/640645 | 18:55 |
| openstackgerrit | Felipe Monteiro proposed openstack/airship-pegleg master: tests: Improve unit tests runtime performance https://review.openstack.org/611435 | 18:57 |
| openstackgerrit | Felipe Monteiro proposed openstack/airship-pegleg master: tests: Improve unit tests runtime performance https://review.openstack.org/611435 | 18:57 |
| openstackgerrit | Felipe Monteiro proposed openstack/airship-pegleg master: tests: Add CLI tests for site encrypt action https://review.openstack.org/614303 | 18:58 |
| openstackgerrit | Felipe Monteiro proposed openstack/airship-pegleg master: zuul: Add python37 job https://review.openstack.org/640646 | 18:58 |
| openstackgerrit | Felipe Monteiro proposed openstack/airship-pegleg master: Embed UML generated diagrams into docs, fix docs build https://review.openstack.org/639676 | 18:59 |
| openstackgerrit | Felipe Monteiro proposed openstack/airship-pegleg master: pki: Add pki_generator unit tests https://review.openstack.org/640622 | 18:59 |
| openstackgerrit | Felipe Monteiro proposed openstack/airship-pegleg master: pki: Add documentation for PKI implementation https://review.openstack.org/611738 | 18:59 |
| openstackgerrit | PRATEEK REDDY DODDA proposed openstack/airship-divingbell master: Implement Security Context for Divingbell https://review.openstack.org/641706 | 19:01 |
| openstackgerrit | Felipe Monteiro proposed openstack/airship-pegleg master: fix: Multiple issues related to lint check P003 https://review.openstack.org/614053 | 19:03 |
| openstackgerrit | PRATEEK REDDY DODDA proposed openstack/airship-divingbell master: Implement Security Context for Divingbell https://review.openstack.org/647883 | 19:12 |
| openstackgerrit | Merged openstack/airship-in-a-bottle master: Add bgp peering in virtual airship https://review.openstack.org/642171 | 19:18 |
| openstackgerrit | Drew Walters proposed openstack/airship-promenade master: tools: Update Helm to v2.13.1 https://review.openstack.org/647828 | 19:24 |
| openstackgerrit | PRATEEK REDDY DODDA proposed openstack/airship-divingbell master: Implement Security Context for Divingbell https://review.openstack.org/647894 | 19:34 |
| kranthikirang | mattmceuen: Haproxy seems ok; We are seeing "Failed to get status of the pods" message during haproxy and kubernetes-api-server restart or config change; Here are the haproxy logs. In one case haproxy kubernetes-be configured with 6444 port instead of 6443. I didn't understand that; | 20:07 |
| kranthikirang | http://paste.openstack.org/show/748425/ | 20:07 |
| openstackgerrit | Aaron Sheffield proposed openstack/airship-deckhand master: [WIP] Updating Docker Gate use of zuul.newrev https://review.openstack.org/645825 | 20:08 |
| kranthikirang | mattmceuen: So, I have increased the kubectl poll time in ./genesis.sh from 30 seconds to 90 seconds which reduced the load on system considerably; and now I can see rabbitmq and mariadb statefulsets are up; However, increasing the Armada timeouts also helped since first time image downloading and extraction also includes; Seems like prior downloading the images will save CPU and time | 20:09 |
| *** JosW has joined #airshipit | 20:13 | |
| *** JosW has left #airshipit | 20:13 | |
| openstackgerrit | PRATEEK REDDY DODDA proposed openstack/airship-divingbell master: Implement Security Context for Divingbell https://review.openstack.org/647894 | 20:21 |
| openstackgerrit | Felipe Monteiro proposed openstack/airship-pegleg master: zuul: Remove redundant airship-pegleg-tox-py36 job. https://review.openstack.org/640645 | 20:38 |
| openstackgerrit | Felipe Monteiro proposed openstack/airship-pegleg master: zuul: Add python37 job https://review.openstack.org/640646 | 20:38 |
| openstackgerrit | Felipe Monteiro proposed openstack/airship-pegleg master: tests: Add CLI tests for site encrypt action https://review.openstack.org/614303 | 20:39 |
| kranthikirang | mattmceuen: How can I increase the timeout for keystone-test for Armada? | 20:45 |
| openstackgerrit | Evgeniy L proposed openstack/airship-treasuremap master: Uplift helm version to introduce `mergeOverwrite` https://review.openstack.org/643105 | 20:54 |
| openstackgerrit | Evgeniy L proposed openstack/airship-treasuremap master: Uplift helm version to introduce `mergeOverwrite` https://review.openstack.org/643105 | 20:54 |
| openstackgerrit | Aaron Sheffield proposed openstack/airship-deckhand master: [WIP] Updating Docker Gate use of zuul.newrev https://review.openstack.org/645825 | 21:17 |
| openstackgerrit | PRATEEK REDDY DODDA proposed openstack/airship-divingbell master: Implement Security Context for Divingbell https://review.openstack.org/647894 | 21:25 |
| kranthikirang | mattmceuen: 2019-03-26 21:29:42.393 3838 INFO armada.handlers.tiller [-] [chart=ucp-keystone]: Running Helm test: release=airship-ucp-keystone, timeout=300 | 21:32 |
| *** kranthikirang has quit IRC | 21:43 | |
| *** ab2434_ has quit IRC | 22:38 | |
| *** michaelbeaver has joined #airshipit | 22:38 | |
| *** michael-beaver has quit IRC | 22:42 | |
| *** michaelbeaver has quit IRC | 22:43 | |
| *** bh526r has joined #airshipit | 22:58 | |
| openstackgerrit | Tin Lam proposed openstack/airship-pegleg master: Remove extraneous dependency https://review.openstack.org/645048 | 23:19 |
| *** aaronsheffield has quit IRC | 23:22 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!