| *** sthussey has quit IRC | 00:13 | |
| *** levmorgan has quit IRC | 00:22 | |
| *** kaspars__ has quit IRC | 00:33 | |
| *** michael-beaver has quit IRC | 00:48 | |
| *** Talion has quit IRC | 00:54 | |
| openstackgerrit | Scott Hussey proposed openstack/airship-promenade master: (apiserver) [WIP] support key rotation https://review.openstack.org/631935 | 01:27 |
|---|---|---|
| openstackgerrit | Alexander Hughes proposed openstack/airship-pegleg master: [WIP] Clean up Pegleg tests directory https://review.openstack.org/653225 | 02:14 |
| *** aaronsheffield has quit IRC | 03:15 | |
| openstackgerrit | Arijit Bose proposed openstack/airship-in-a-bottle master: [site update] Adding gate set up https://review.openstack.org/646796 | 03:51 |
| openstackgerrit | Arijit Bose proposed openstack/airship-in-a-bottle master: [site update] Adding gate set up https://review.openstack.org/646796 | 03:57 |
| *** Gpshz1Ccsms2 has quit IRC | 06:04 | |
| *** aojea has joined #airshipit | 06:06 | |
| *** rihbb has joined #airshipit | 07:06 | |
| *** dimitris_ has quit IRC | 08:03 | |
| *** happyhemant has joined #airshipit | 09:25 | |
| *** aojea has quit IRC | 12:02 | |
| openstackgerrit | Alexander Hughes proposed openstack/airship-pegleg master: [WIP] Clean up Pegleg tests directory https://review.openstack.org/653225 | 12:42 |
| *** aaronsheffield has joined #airshipit | 12:53 | |
| *** sthussey has joined #airshipit | 12:56 | |
| openstackgerrit | Scott Hussey proposed openstack/airship-promenade master: (apiserver) [WIP] support key rotation https://review.openstack.org/631935 | 13:12 |
| openstackgerrit | Bryan Strassner proposed openstack/airship-shipyard master: Update upgrade_airflow_worker.sh script https://review.openstack.org/653434 | 13:40 |
| *** mbologna has quit IRC | 13:46 | |
| *** mbologna has joined #airshipit | 13:47 | |
| *** levmorgan has joined #airshipit | 13:57 | |
| *** mardim has joined #airshipit | 14:11 | |
| mardim | Good morning all | 14:11 |
| mardim | I am trying to add the VIPs functionality in airship-seaworthy | 14:11 |
| mardim | I uplifted the versions of drydock and maas and also the dependent HTK to the latest commit SHA | 14:12 |
| mardim | but I am getting the error | 14:12 |
| mardim | http://paste.openstack.org/show/749429/ | 14:12 |
| mardim | the PS that I have is this https://review.openstack.org/#/c/650432/ | 14:13 |
| openstackgerrit | Scott Hussey proposed openstack/airship-promenade master: (apiserver) [WIP] support key rotation https://review.openstack.org/631935 | 14:13 |
| mardim | Does anybdy knows why I get that error ? | 14:13 |
| sthussey | @mardim you'll need to use the latest armada and ensure you are deploying tiller 2.13.1 | 14:16 |
| sthussey | There was a breaking change in the OSH helm-toolkit functionality that requires 2.13.1 | 14:16 |
| openstackgerrit | Drew Walters proposed openstack/airship-shipyard master: CI: Add Airskiff check https://review.openstack.org/653035 | 14:16 |
| mardim | sthussey: oh ok didn't know | 14:18 |
| mardim | sthussey: So that means that I need to hange only the versions in those two in versions.yaml right ? | 14:18 |
| mardim | change* | 14:18 |
| sthussey | I think that is correct | 14:19 |
| mardim | when you are saying tiller 2.13.1 | 14:21 |
| mardim | you mean the docker container right ? | 14:21 |
| mardim | sthussey: ^ | 14:21 |
| sthussey | yes | 14:21 |
| mardim | ok thanks | 14:22 |
| mardim | let me change those and see | 14:22 |
| *** jemangs has joined #airshipit | 14:27 | |
| *** stefanb has joined #airshipit | 14:28 | |
| openstackgerrit | Dimitrios Markou proposed openstack/airship-treasuremap master: [WIP] Integrate VIPs in airship-seaworthy https://review.openstack.org/650432 | 14:31 |
| *** michael-beaver has joined #airshipit | 14:37 | |
| openstackgerrit | Merged openstack/airship-shipyard master: Update upgrade_airflow_worker.sh script https://review.openstack.org/653434 | 14:48 |
| *** kaspars__ has joined #airshipit | 14:59 | |
| jemangs | We're trying to install the v19.02.04-release on Dell R630s. We're experiencing completely random failures in MaaS. We've tried all kinds of things, Dell firmware upgrades, replaced networking (switches, NICs, cables etc.), switched to 10 GbE for PXE etc. Nodes fail in random during drydock/MaaS without any understandable pattern. Does anyone have any clue how to debug this? Right now we're sort of out of ideas. /Jan-Eri | 15:00 |
| jemangs | k & Stefan | 15:00 |
| jemangs | BWT, what does the three different timeouts in global/software/charts/ucp/drydock mean? Anything we need to tune? | 15:02 |
| kaspars__ | Is the failures mostly IPMI related or actual PXE boot/install? might be good to share any drydock logs for help.. | 15:03 |
| kaspars__ | have you tried to run ipmitool manually (many times) to see if the requests are stable - I would suggest to run if from Genesis node as that's where drydock/maas will call from | 15:04 |
| kaspars__ | ipmitool -I lanplus -H ${GENESIS_IPMI_IP} -U \$IUSER -P \$IPASS chassis power on (and off) | 15:06 |
| kaspars__ | would suggest to run this from genesis towards one of the nodes you are going to deploy.. | 15:06 |
| kaspars__ | if the failures are mostly not in the IPMI area - I could also suggest to try using GA instead of HWE kernel and see if that helps. | 15:07 |
| *** rihbb has quit IRC | 15:08 | |
| stefanb | On the same topic as @jemangs: we have seen temporary failures in IPMI tools command, but only for a minute or two | 15:08 |
| *** Gpshz1 has joined #airshipit | 15:09 | |
| stefanb | In MAAS logs there was sometimes things like: "Error changing power state (on) of node" (...) "provisioningserver.drivers.power.PowerAuthError: Incorrect password. Check BMC configuration and try again". | 15:09 |
| jemangs | The network genesis -> ipmi is solid, I guess iDRAC is what it is, we've upgraded firmware, I do "racadm racreset soft" periodically etc. | 15:09 |
| openstackgerrit | Kaspars Skels proposed openstack/airship-treasuremap master: Add tools/airship as wrapper for deployment tools https://review.openstack.org/651917 | 15:09 |
| stefanb | But there have also been MAAS failures that did not have this error | 15:09 |
| *** sreejithp has joined #airshipit | 15:10 | |
| openstackgerrit | Dimitrios Markou proposed openstack/airship-treasuremap master: [WIP] Integrate VIPs in airship-seaworthy https://review.openstack.org/650432 | 15:12 |
| kaspars__ | yeah - so if you are hitting actual IPMI issues then in my experience is either a) issues with firmware version (maybe downgrade could help), or b) there is some network issue from genesis towards the oob (as it's separate NIC on the board and usually a switch) | 15:13 |
| jemangs | @kaspars__, kubectl logs on drydock containers after failure? | 15:14 |
| kaspars__ | auth error - interesting! so only thing you need to make sure is to update the ipmi password under secrets | 15:14 |
| *** rihbb has joined #airshipit | 15:14 | |
| kaspars__ | I think Evgenyi had a similar issue that you needed to reset password on iDRAC | 15:15 |
| kaspars__ | so potentially I could suggest to either reset password (even if it's the same), or perhaps create another user | 15:15 |
| jemangs | The password issue is not permanent, it's just one of the things that sometimes can show up | 15:16 |
| kaspars__ | right - I think we had to "reset" it to get it working stable in one of the envs | 15:16 |
| kaspars__ | even if we did set/reset it to the same password then it was before.. there was some bug in iDRAC or so | 15:17 |
| kaspars__ | I have seen it only 1x ever | 15:17 |
| kaspars__ | I think that's all the tips I can think of right now | 15:18 |
| stefanb | We can try, but I'm pretty sure I saw failed MAAS deployments that did not have auth error | 15:18 |
| stefanb | Right now we run a deployment that seems to go through | 15:18 |
| stefanb | Next time, which of the 7 drydock pods should we ask for logs? | 15:18 |
| kaspars__ | haha - only 2 of interest | 15:19 |
| kaspars__ | drydock-api | 15:19 |
| jemangs | kaspars, I saw that error too (password reset) but that was a couple of months ago | 15:19 |
| kaspars__ | one of them will be standby, and one active | 15:19 |
| kaspars__ | you only need the active logs | 15:19 |
| stefanb | Ok, yes, I can see the API ones; they are the only ones that keep on running. | 15:20 |
| kaspars__ | yes | 15:20 |
| stefanb | Right now (as things work well this time) they give only one error: "DEBUG - api_client.py:_send_request - Received error response - URL: GET http://10.23.22.11:30001/MAAS/api/2.0/nodes/wd8wrf/blockdevices/16/partitions/ - RESPONSE: 500" | 15:21 |
| kaspars__ | to rule out all the networking issues - you may check some 'dmesg' and /var/log/syslog to see if bond or links migh tbe going up and down | 15:21 |
| kaspars__ | you can ignore that | 15:21 |
| kaspars__ | that's mostly for empty partition tables that are not setup yet and return errors (instead of empty list) | 15:21 |
| jemangs | (I have to sign off for a while) | 15:23 |
| *** jemangs has quit IRC | 15:23 | |
| openstackgerrit | Kaspars Skels proposed openstack/airship-treasuremap master: Uplift helm/tiller and most charts https://review.openstack.org/653470 | 15:32 |
| *** rihbb has quit IRC | 15:37 | |
| *** stefanb has left #airshipit | 15:43 | |
| openstackgerrit | Sirajudeen proposed openstack/airship-in-a-bottle master: Multinode support for promenade encryption https://review.openstack.org/652735 | 15:50 |
| openstackgerrit | Stas Egorov proposed openstack/airship-promenade master: Replace kubectl and kubelet binary with hyperkube image https://review.openstack.org/653479 | 15:53 |
| openstackgerrit | Ian Pittwood proposed openstack/airship-spyglass master: [WIP] Implements PBR for packaging https://review.openstack.org/653480 | 15:56 |
| mardim | kaspars__: evgenyl Should we do a meeting to coordinate our endeavors? | 15:57 |
| mardim | I noticed that you kaspars | 15:57 |
| mardim | you have a PS | 15:57 |
| mardim | that uplifts some version | 15:57 |
| mardim | versions* | 15:57 |
| kaspars__ | yes please | 15:57 |
| mardim | and also in my PS i uplift some other versions | 15:58 |
| mardim | this is going to be a mess | 15:58 |
| mardim | if we do not do some coordination | 15:58 |
| openstackgerrit | Ian Pittwood proposed openstack/airship-spyglass master: [WIP] Implements PBR for packaging https://review.openstack.org/653480 | 15:58 |
| mardim | kaspars__: can you please create an invitation for the meeting ? | 15:58 |
| kaspars__ | please check private messages | 15:59 |
| mardim | ok I saw that | 16:04 |
| openstackgerrit | Sean Eagan proposed openstack/airship-armada master: [WIP]: Improve wait API and semantics in v2 docs https://review.openstack.org/636440 | 16:07 |
| *** dims has quit IRC | 16:07 | |
| *** mardim has quit IRC | 16:08 | |
| openstackgerrit | Merged openstack/airship-maas master: CI: Add chart build job for latest Helm toolkit https://review.openstack.org/648548 | 16:12 |
| openstackgerrit | Ian Pittwood proposed openstack/airship-spyglass master: PEP-8 code styling and linting https://review.openstack.org/649692 | 16:13 |
| openstackgerrit | Merged openstack/airship-divingbell master: CI: Add chart build job for latest Helm toolkit https://review.openstack.org/648544 | 16:17 |
| openstackgerrit | Merged openstack/airship-armada master: CI: Add chart build job for latest Helm toolkit https://review.openstack.org/648443 | 16:19 |
| openstackgerrit | Merged openstack/airship-in-a-bottle master: Multinode support for promenade encryption https://review.openstack.org/652735 | 16:19 |
| openstackgerrit | Drew Walters proposed openstack/airship-treasuremap master: CI: Add airsloop site linting gate https://review.openstack.org/653485 | 16:20 |
| openstackgerrit | Ian Pittwood proposed openstack/airship-spyglass master: PEP-8 code styling and linting https://review.openstack.org/649692 | 16:25 |
| openstackgerrit | Merged openstack/airship-drydock master: CI: Add chart build job for latest Helm toolkit https://review.openstack.org/648543 | 16:26 |
| openstackgerrit | Merged openstack/airship-shipyard master: CI: Add chart build job for latest Helm toolkit https://review.openstack.org/648546 | 16:28 |
| openstackgerrit | Kaspars Skels proposed openstack/airship-treasuremap master: [WIP] Integrate VIPs in airship-seaworthy https://review.openstack.org/650432 | 16:29 |
| openstackgerrit | Ian Pittwood proposed openstack/airship-spyglass master: Implements PBR for packaging https://review.openstack.org/653480 | 16:32 |
| openstackgerrit | Kaspars Skels proposed openstack/airship-treasuremap master: Uplift helm/tiller and most charts https://review.openstack.org/653470 | 16:34 |
| *** mardim has joined #airshipit | 16:40 | |
| openstackgerrit | Merged openstack/airship-deckhand master: CI: Add chart build jobs https://review.openstack.org/649441 | 16:43 |
| openstackgerrit | Ian Pittwood proposed openstack/airship-spyglass master: PEP-8 code styling and linting https://review.openstack.org/649692 | 16:51 |
| *** dims has joined #airshipit | 17:02 | |
| openstackgerrit | Lev Morgan proposed openstack/airship-pegleg master: Improve files.write, add decrypt output to file https://review.openstack.org/653493 | 17:07 |
| *** krypto has joined #airshipit | 17:26 | |
| openstackgerrit | Evgeniy L proposed openstack/airship-treasuremap master: Change default number of l3 agents from 5 to 3 https://review.openstack.org/639179 | 17:32 |
| openstackgerrit | Evgeniy L proposed openstack/airship-treasuremap master: Add metadata proxy shared secret for Nova and Neutron https://review.openstack.org/639183 | 17:32 |
| openstackgerrit | Evgeniy L proposed openstack/airship-treasuremap master: Do not rerun promjoin on every reboot https://review.openstack.org/642571 | 17:32 |
| openstackgerrit | Evgeniy L proposed openstack/airship-treasuremap master: Enable nested virtualization by default https://review.openstack.org/652139 | 17:32 |
| *** rihbb has joined #airshipit | 17:41 | |
| openstackgerrit | Merged openstack/airship-promenade master: Promenade: Add security context for pod/container https://review.openstack.org/651416 | 17:58 |
| openstackgerrit | Drew Walters proposed openstack/airship-treasuremap master: global: Uplift helm-toolkit https://review.openstack.org/645226 | 18:16 |
| openstackgerrit | Drew Walters proposed openstack/airship-treasuremap master: airskiff: Reduce postgresql PVC size https://review.openstack.org/639123 | 18:16 |
| openstackgerrit | Drew Walters proposed openstack/airship-treasuremap master: airskiff: Use global Airship charts https://review.openstack.org/635218 | 18:16 |
| *** happyhemant has quit IRC | 18:23 | |
| openstackgerrit | Scott Hussey proposed openstack/airship-promenade master: (apiserver) [WIP] support key rotation https://review.openstack.org/631935 | 18:45 |
| openstackgerrit | Drew Walters proposed openstack/airship-promenade master: CI: Use official OpenStack Train Python jobs https://review.openstack.org/653069 | 18:52 |
| openstackgerrit | Kaspars Skels proposed openstack/airship-treasuremap master: Uplift helm/tiller and most charts https://review.openstack.org/653470 | 19:13 |
| openstackgerrit | Ian Pittwood proposed openstack/airship-spyglass master: PEP-8 code styling and linting https://review.openstack.org/649692 | 19:15 |
| openstackgerrit | Sreejith Punnapuzha proposed openstack/airship-promenade master: [fix] etcd backup to use different filename https://review.openstack.org/648745 | 19:16 |
| openstackgerrit | Kaspars Skels proposed openstack/airship-treasuremap master: Uplift helm/tiller and most charts https://review.openstack.org/653470 | 19:33 |
| openstackgerrit | Dimitrios Markou proposed openstack/airship-treasuremap master: [WIP] Integrate VIPs in airship-seaworthy https://review.openstack.org/650432 | 19:35 |
| openstackgerrit | Scott Hussey proposed openstack/airship-maas master: Add explicit rackd image sync https://review.openstack.org/653551 | 20:24 |
| openstackgerrit | Stas Egorov proposed openstack/airship-in-a-bottle master: utility script to exctract binary data from genesis.sh https://review.openstack.org/650427 | 20:27 |
| openstackgerrit | Drew Walters proposed openstack/airship-treasuremap master: airskiff: Use global Airship charts https://review.openstack.org/635218 | 20:40 |
| openstackgerrit | Ian Pittwood proposed openstack/airship-spyglass master: [WIP] Separate plugins from Spyglass https://review.openstack.org/653555 | 20:59 |
| openstackgerrit | Dimitrios Markou proposed openstack/airship-treasuremap master: [WIP] Integrate VIPs in airship-seaworthy https://review.openstack.org/650432 | 21:08 |
| openstackgerrit | Kaspars Skels proposed openstack/airship-treasuremap master: Uplift helm/tiller and most charts https://review.openstack.org/653470 | 21:08 |
| openstackgerrit | Dimitrios Markou proposed openstack/airship-treasuremap master: [WIP] Integrate VIPs in airship-seaworthy https://review.openstack.org/650432 | 21:12 |
| openstackgerrit | Drew Walters proposed openstack/airship-treasuremap master: airskiff: Use global Airship charts https://review.openstack.org/635218 | 21:20 |
| openstackgerrit | Scott Hussey proposed openstack/airship-maas master: syslog listener for physical nodes https://review.openstack.org/627415 | 21:20 |
| openstackgerrit | Bryan Strassner proposed openstack/airship-shipyard master: Update mq queue mirror directive to match vhost https://review.openstack.org/653611 | 21:34 |
| *** levmorgan has quit IRC | 21:35 | |
| evgenyl | In the last few day when I answer questions on airship ML, my message goes to a moderator approval due to "Message body is too big: 40974 bytes with a limit of 40 KB", I think this happens because of html formatting that is used in the emails, is there a way to increase it to say 400KB? | 21:38 |
| *** rihbb has quit IRC | 21:39 | |
| kaspars__ | mattmceuen: | 21:40 |
| kaspars__ | (learning to tag people..) | 21:40 |
| evgenyl | :) | 21:43 |
| *** fdegir has quit IRC | 21:55 | |
| *** georgk has quit IRC | 21:55 | |
| openstackgerrit | Merged openstack/airship-armada master: CI: Add Airskiff check https://review.openstack.org/599020 | 21:56 |
| openstackgerrit | Drew Walters proposed openstack/airship-shipyard master: CI: Add Airskiff check https://review.openstack.org/653035 | 22:03 |
| openstackgerrit | Drew Walters proposed openstack/airship-deckhand master: CI: Add Airskiff check https://review.openstack.org/600771 | 22:04 |
| openstackgerrit | Kaspars Skels proposed openstack/airship-treasuremap master: Add tools/airship as wrapper for deployment tools https://review.openstack.org/651917 | 22:08 |
| *** mardim has quit IRC | 22:08 | |
| *** sreejithp has quit IRC | 22:15 | |
| openstackgerrit | Merged openstack/airship-shipyard master: Update mq queue mirror directive to match vhost https://review.openstack.org/653611 | 22:19 |
| openstackgerrit | Merged openstack/airship-treasuremap master: CI: Add airsloop site linting gate https://review.openstack.org/653485 | 22:21 |
| openstackgerrit | Stas Egorov proposed openstack/airship-in-a-bottle master: utility script to exctract binary data from genesis.sh https://review.openstack.org/650427 | 22:27 |
| openstackgerrit | Stas Egorov proposed openstack/airship-promenade master: Replace kubectl and kubelet binary with hyperkube image https://review.openstack.org/653479 | 22:32 |
| mattmceuen | kaspars__ heyo! | 22:32 |
| mattmceuen | This is mildly amusing: https://www.airship.com/blog/why-airship/ | 22:33 |
| kaspars__ | haha btw, see "In the last few day when I answer questions on airship ML, my message goes to a moderator approval due to "Message body is too big: 40974 bytes with a limit of 40 KB", I think this happens because of html formatting that is used in the emails, is there a way to increase it to say 400KB?" | 22:33 |
| kaspars__ | from evgenyl | 22:34 |
| mattmceuen | evgenyl: that's probably a setting that can be tweaked somewhere. Do you know why the message is so big though | 22:34 |
| mattmceuen | e.g. are there attachments or some such | 22:35 |
| openstackgerrit | Stas Egorov proposed openstack/airship-promenade master: Replace kubectl and kubelet binary with hyperkube image https://review.openstack.org/653479 | 22:36 |
| openstackgerrit | Kaspars Skels proposed openstack/airship-treasuremap master: Integrate support for MAAS HA and VIPs https://review.openstack.org/650432 | 22:41 |
| openstackgerrit | Kaspars Skels proposed openstack/airship-treasuremap master: Uplift helm/tiller and most charts https://review.openstack.org/653470 | 22:43 |
| mattmceuen | also - here's a recording of the webcast that evgenyl and I gave the other day: https://www.brighttalk.com/webcast/12229/354156/elevate-your-infrastructure-using-airship | 23:18 |
| mattmceuen | Turned out good I think! | 23:19 |
| * mattmceuen aside from typical "I hate the sound of my own voice and say um too much" :) | 23:19 | |
| *** aaronsheffield has quit IRC | 23:21 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!