| *** goldyfruit_ has joined #openstack-kolla | 00:06 | |
| *** k_mouza has joined #openstack-kolla | 00:30 | |
| *** yingjun has joined #openstack-kolla | 00:31 | |
| *** k_mouza has quit IRC | 00:35 | |
| *** oneswig has quit IRC | 01:11 | |
| *** BjoernT has joined #openstack-kolla | 02:26 | |
| *** zhangfei has joined #openstack-kolla | 02:57 | |
| *** BjoernT has quit IRC | 03:01 | |
| *** BjoernT has joined #openstack-kolla | 03:02 | |
| *** yingjun has quit IRC | 03:24 | |
| *** dave-mccowan has quit IRC | 03:36 | |
| *** zhangfei has quit IRC | 03:49 | |
| *** zhangfei has joined #openstack-kolla | 04:07 | |
| *** janki has joined #openstack-kolla | 04:34 | |
| *** pcaruana has joined #openstack-kolla | 04:46 | |
| *** BjoernT has quit IRC | 04:48 | |
| *** Luzi has joined #openstack-kolla | 04:59 | |
| *** radeks has joined #openstack-kolla | 05:04 | |
| *** hjensas|afk has quit IRC | 06:02 | |
| *** cah_link has joined #openstack-kolla | 06:07 | |
| *** igordc has joined #openstack-kolla | 06:17 | |
| *** igordc has quit IRC | 06:19 | |
| *** zhangfei has quit IRC | 06:32 | |
| openstackgerrit | Qitao proposed openstack/kolla-ansible master: Update stackalytics link for README https://review.opendev.org/682806 | 06:40 |
|---|---|---|
| yoctozepto | morning | 06:41 |
| *** k_mouza has joined #openstack-kolla | 06:41 | |
| *** cah_link has quit IRC | 06:43 | |
| *** k_mouza has quit IRC | 06:46 | |
| osmanlicilegi | morning | 06:49 |
| *** zhangfei has joined #openstack-kolla | 06:57 | |
| *** tonythomas has joined #openstack-kolla | 07:01 | |
| *** cah_link has joined #openstack-kolla | 07:08 | |
| *** luksky has joined #openstack-kolla | 07:13 | |
| *** hjensas|afk has joined #openstack-kolla | 07:17 | |
| mnasiadka | morning | 07:29 |
| yoctozepto | morning mnasiadka | 07:30 |
| yoctozepto | mnasiadka: https://review.opendev.org/682600 - this is for staters, OL is broken for dessert: https://bugs.launchpad.net/kolla/+bug/1844484 | 07:31 |
| openstack | Launchpad bug 1844484 in kolla train "oraclelinux builds always fail (package gone)" [Critical,Triaged] | 07:31 |
| yoctozepto | welcome to centos 7.7 8-) | 07:31 |
| *** ivve has joined #openstack-kolla | 07:32 | |
| *** rpittau|afk is now known as rpittau | 07:32 | |
| mnasiadka | yoctozepto: well, based on discussion on centos-devel, it seems we might need epel-rpm-macros to work around this python36 vs python3 crap ;) | 07:38 |
| *** cah_link has quit IRC | 07:38 | |
| yoctozepto | mnasiadka: so there are also python3 issues? :O | 07:39 |
| yoctozepto | the OL issue is far from python atm ;p | 07:39 |
| mnasiadka | OL? I thought we deprecated that ;) | 07:40 |
| yoctozepto | mnasiadka: we don't build that for train yeah | 07:40 |
| yoctozepto | but for stein- we do | 07:40 |
| mnasiadka | oh crap | 07:41 |
| yoctozepto | mnasiadka: but in train we still support building it | 07:42 |
| mnasiadka | that's why I said oh crap | 07:42 |
| yoctozepto | just that we don't offer the releases | 07:42 |
| yoctozepto | yeah, I'm just giving more details | 07:42 |
| yoctozepto | ;p | 07:43 |
| mnasiadka | can't we just pin OL to 7.6 and be gone? :D | 07:43 |
| yoctozepto | mnasiadka: you mean playing with access to vault? xD | 07:43 |
| yoctozepto | gl&hf :D | 07:43 |
| yoctozepto | I think we should just centosify ol more | 07:44 |
| yoctozepto | and treat them equally | 07:44 |
| mnasiadka | so then it's no more OL | 07:44 |
| mnasiadka | :) | 07:44 |
| yoctozepto | no, it has the whole base of OL | 07:44 |
| yoctozepto | which should be binary compatible | 07:44 |
| yoctozepto | with centos anyways | 07:44 |
| yoctozepto | just slap the extras so that they work the same | 07:45 |
| yoctozepto | and not play with direct url | 07:45 |
| mnasiadka | so go for it and be gone ;) | 07:49 |
| hrw | just drop OL | 07:50 |
| *** gfidente has joined #openstack-kolla | 07:50 | |
| hrw | ;d | 07:50 |
| yoctozepto | hrw: +3 | 07:50 |
| yoctozepto | morning you joker | 07:50 |
| hrw | but as Red Hat employee I probably can not do it myself | 07:51 |
| *** gfidente has quit IRC | 07:51 | |
| *** cah_link has joined #openstack-kolla | 07:51 | |
| yoctozepto | mnasiadka: yeah, after finishing local higher prio stuff ;-) | 07:51 |
| hrw | ok, I returned ppc64le machine | 07:51 |
| hrw | in case of need will grab another one. but that's October | 07:52 |
| yoctozepto | you two better review those RP+2 changes I and Mark have | 07:52 |
| yoctozepto | so that we get modernized sooner | 07:52 |
| yoctozepto | than later | 07:52 |
| *** radeks has quit IRC | 07:55 | |
| *** radeks has joined #openstack-kolla | 07:55 | |
| hrw | moment, let me read apache arrow changes first | 07:56 |
| hrw | upstream developer took my changes and got them nicely integrated | 07:56 |
| hrw | yoctozepto: https://review.opendev.org/682600 https://review.opendev.org/682437 and? | 07:59 |
| *** Reepicheep has quit IRC | 08:00 | |
| yoctozepto | hrw: thanks, 2nd no, w-1 slapped just now | 08:00 |
| mnasiadka | yoctozepto: can we agree, that we do RP+1/+2 only when Zuul is green? :) | 08:00 |
| yoctozepto | then one in ansible awaits | 08:00 |
| yoctozepto | https://review.opendev.org/670690 | 08:00 |
| yoctozepto | mnasiadka: doing that unless not | 08:01 |
| *** Reepicheep has joined #openstack-kolla | 08:01 | |
| yoctozepto | heh :D | 08:01 |
| mnasiadka | yoctozepto: well, "RP+1 recheck" doesn't sound good for me :D | 08:01 |
| hrw | mnasiadka: some patches are just important enough to get RP+1/+2 right after sending | 08:01 |
| yoctozepto | mnasiadka: when I know the issue is random | 08:01 |
| yoctozepto | then it's ok ;-) | 08:01 |
| hrw | mnasiadka: it also shows "this is important! if it fail then those CI errors need to be fixed FRIST!" | 08:02 |
| mnasiadka | yoctozepto: let's agree when the issue is sorted out - do it then, I don't like looking in Zuul logs with random issues :D | 08:02 |
| mnasiadka | hrw: this is not the case | 08:02 |
| yoctozepto | FRIST indeed | 08:02 |
| mnasiadka | https://review.opendev.org/#/c/682437/ | 08:02 |
| yoctozepto | mnasiadka: I invite you to look at the code | 08:02 |
| mnasiadka | I don't even know why it is priority? | 08:02 |
| hrw | yoctozepto: I looked at Mark's change and do not understand it. too k-a for me | 08:03 |
| yoctozepto | mnasiadka: that was a bad example, hence rp-1 removed | 08:03 |
| yoctozepto | ;-) | 08:03 |
| mnasiadka | yeah, let's agree RP+1 only for those changes that are related to a) CI breakage b) features on the priority list | 08:04 |
| mnasiadka | I haven't seen Zun on priority list :) | 08:04 |
| yoctozepto | RP+2 for CI breakage | 08:04 |
| yoctozepto | RP+1 for priority | 08:04 |
| mnasiadka | good :) | 08:04 |
| yoctozepto | sure, that was some sloppy mouse click :-) | 08:04 |
| yoctozepto | mnasiadka: anyway - https://review.opendev.org/670690 | 08:05 |
| yoctozepto | too ansibly for hrw | 08:06 |
| yoctozepto | co-authored by me and Mark | 08:06 |
| yoctozepto | so mostly for you to review | 08:06 |
| yoctozepto | it will have to march back to rocky to unbreak upgrade jobs mind you | 08:07 |
| mnasiadka | yoctozepto: tell me why this happens: https://zuul.opendev.org/t/openstack/build/ccbd5ed4cb8740ab991775116905899f/log/primary/logs/ansible/upgrade#22838 | 08:08 |
| *** dougsz has joined #openstack-kolla | 08:09 | |
| yoctozepto | mnasiadka: we don't support moving from non-ha to ha | 08:09 |
| yoctozepto | but this very much beyond the scope of this change | 08:09 |
| yoctozepto | remember we ran multinode without ha | 08:09 |
| mnasiadka | And what now? We fail all Ceph upgrade jobs? that's the approach? | 08:09 |
| yoctozepto | mnasiadka: read me above | 08:09 |
| yoctozepto | we are enabling ha in ci in previous releases | 08:10 |
| mnasiadka | Well - I don't like it, fix it. | 08:10 |
| mnasiadka | :) | 08:10 |
| yoctozepto | well, I tried | 08:10 |
| yoctozepto | the readdressing is the issue | 08:10 |
| yoctozepto | we would have to support it | 08:10 |
| mnasiadka | Well, I just don't like the fact, that we add some CI feature and then break something else | 08:11 |
| mnasiadka | here it's even better: https://zuul.opendev.org/t/openstack/build/7b35fc9c870646e7ae6ba37702b1da7a/console | 08:12 |
| mnasiadka | :) | 08:12 |
| yoctozepto | mnasiadka: it's a chicken and egg problem really | 08:12 |
| yoctozepto | mnasiadka: that is CI for you | 08:12 |
| yoctozepto | we get this from time to time | 08:12 |
| yoctozepto | seems you are not acquiainted with CI features :-) | 08:12 |
| mnasiadka | yoctozepto: I understand the technical issue at hand, but if we don't care about Ceph-upgrade jobs - let's just remove them. | 08:12 |
| yoctozepto | mnasiadka: we care, these are only multinode upgrade jobs we have | 08:13 |
| yoctozepto | ;p | 08:13 |
| mnasiadka | What is the sense of having a CI that always fails and is non-voting? :) | 08:13 |
| mgoddard | morning | 08:13 |
| yoctozepto | mnasiadka: you need to eat some snickers | 08:14 |
| yoctozepto | morning mgoddard | 08:14 |
| mnasiadka | yoctozepto: naah, I just don't like breaking things - and you tend to not think about it :) | 08:14 |
| yoctozepto | mnasiadka: I told you it is fixed by backporting to stein and then for stein to rocky | 08:14 |
| yoctozepto | I think long distance | 08:15 |
| *** gfidente has joined #openstack-kolla | 08:15 | |
| yoctozepto | ;-) | 08:15 |
| mnasiadka | yoctozepto: I don't think it's a feature we should back port - unless we have no other choice. | 08:15 |
| yoctozepto | let's agree on the general functionality, behavior and code | 08:16 |
| yoctozepto | mnasiadka: why not, it makes CI more realistic | 08:16 |
| yoctozepto | else we can change the upgrade jobs if you think no-backport | 08:16 |
| yoctozepto | so that we modify the pre-upgrade settings | 08:16 |
| yoctozepto | it might be better if we agree on how to tackle this | 08:17 |
| yoctozepto | I am fine with either option as long as we reach consensus | 08:17 |
| mnasiadka | well, I thought the back port policy is not back porting new features, but maybe I'm wrong :) | 08:17 |
| yoctozepto | mgoddard: please join us in our pre-meeting discussion :-) | 08:17 |
| yoctozepto | mnasiadka: surely not backporting Ipv6 support | 08:17 |
| yoctozepto | hell no | 08:17 |
| yoctozepto | :D | 08:18 |
| yoctozepto | but CI is a gray zone | 08:18 |
| yoctozepto | more of paying the technical debt | 08:18 |
| mnasiadka | no grey zones please | 08:18 |
| yoctozepto | than feature adding | 08:18 |
| yoctozepto | CI is not the product, sir :D | 08:18 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe stable/stein: Fix kayobe-overcloud-centos CI job flakiness https://review.opendev.org/682829 | 08:18 |
| mnasiadka | yoctozepto: why not? :) | 08:20 |
| yoctozepto | mnasiadka: k-a users don't install it maybe? I feel you are in some strange mood today - but you made me think and for that I thank you - I think what should be done is this: | 08:21 |
| yoctozepto | current branch should control configuring of pre-upgrade env | 08:22 |
| yoctozepto | this way we fix this thing you don't like now | 08:22 |
| yoctozepto | but also | 08:22 |
| yoctozepto | this crosses out the dependency that has bitten us already | 08:22 |
| yoctozepto | e.g. the comment I left for hrw on his docker change | 08:22 |
| yoctozepto | about being careful with those variables | 08:22 |
| yoctozepto | since they bite later | 08:23 |
| yoctozepto | let's make later never happen | 08:23 |
| mnasiadka | well, for some features we need to support both approaches (pre-change and post-change), I know it's additional work - but nothing is easy :) | 08:23 |
| yoctozepto | indeed, in the same sense we should handle pre-upgrade config locally | 08:24 |
| yoctozepto | rather than from pre-upgrade branch | 08:24 |
| mnasiadka | If there is no other way - it would be nice to have some explanation in commit message or a comment in the review itself - I'm not asking for much :) | 08:24 |
| yoctozepto | it handles its own issues already | 08:24 |
| hrw | yoctozepto: 'backport this that those to keep new CI upgrade jobs working' ends with "why I cannot upgrade from stein to train while it is supported" | 08:33 |
| hrw | mnasiadka hit my common issue | 08:33 |
| hrw | for me commit message should say what change does so I do not need to look at code to find out | 08:33 |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: [prometheus] Added support for extra options https://review.opendev.org/682834 | 08:34 |
| mgoddard | ok, finally caught up on the chat | 08:37 |
| mgoddard | the solution is simples | 08:37 |
| mgoddard | add a variable to configure enable_haproxy in globals.yml in CI, backport it, set it when available | 08:38 |
| *** dciabrin_ has quit IRC | 08:38 | |
| mgoddard | so first iteration adds VXLAN but keeps haproxy disabled (at least) on upgrades | 08:38 |
| mgoddard | once the stable branch has the flag, enable haproxy on upgrades too | 08:39 |
| yoctozepto | mgoddard: I think it's cleaner what mnasiadka made me invent - configure pre-upgrade using current (adapted file for pre) | 08:40 |
| mgoddard | I'll take another look at the code | 08:40 |
| yoctozepto | then we squash both current problems | 08:40 |
| yoctozepto | and any future one arising from this dependency | 08:40 |
| mgoddard | hmm | 08:42 |
| yoctozepto | then we avoid "omg, this backport broke master jobs" | 08:43 |
| yoctozepto | we survived that already | 08:43 |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: [prometheus] Added support for extra options https://review.opendev.org/682834 | 08:43 |
| mgoddard | true | 08:44 |
| mgoddard | I do quite like the realistic approach of having the initial install decoupled from the upgrad | 08:44 |
| mgoddard | (that we have now) | 08:45 |
| yoctozepto | mgoddard: code is still decoupled | 08:45 |
| mgoddard | sort of | 08:45 |
| yoctozepto | only config may differ | 08:45 |
| mgoddard | we might need to add 'if upgrade else...' | 08:45 |
| mgoddard | in globals.yml | 08:45 |
| yoctozepto | yeah, we have such a variable | 08:46 |
| mgoddard | we do | 08:46 |
| yoctozepto | and we will be using it locally now | 08:46 |
| yoctozepto | no longer "wait for backport to discover the issue in upgrade" | 08:47 |
| *** k_mouza has joined #openstack-kolla | 08:47 | |
| mgoddard | that can still happen with code :) | 08:47 |
| yoctozepto | mgoddard: true but that's how life goes | 08:51 |
| yoctozepto | shouldn't if we stick to nice backports | 08:51 |
| yoctozepto | the same cannot be told about config | 08:51 |
| yoctozepto | because those were clean backports | 08:51 |
| mgoddard | this will add another TODO at the end of a release - remove all the 'if upgrade' bits | 08:53 |
| mgoddard | recent docker config changes for example | 08:54 |
| yoctozepto | mgoddard: true, on branching off | 08:54 |
| mgoddard | yep | 08:54 |
| yoctozepto | need to clean master | 08:54 |
| *** KeithMnemonic1 has joined #openstack-kolla | 08:55 | |
| mgoddard | we already have a list of things to do, it's not the end of the world | 08:55 |
| *** KeithMnemonic has quit IRC | 08:55 | |
| yoctozepto | indeed, CI usually breaks then anyway :-) | 08:55 |
| *** KeithMnemonic1 has quit IRC | 08:55 | |
| yoctozepto | let me try this out then | 08:56 |
| *** KeithMnemonic1 has joined #openstack-kolla | 08:56 | |
| mnasiadka | well, at least we have a list now :D | 08:56 |
| yoctozepto | dumb approach comes first | 08:56 |
| *** pcaruana has quit IRC | 08:57 | |
| *** pcaruana has joined #openstack-kolla | 09:01 | |
| *** priteau has joined #openstack-kolla | 09:07 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: WIP: Configure the upgrade jobs from the current branch https://review.opendev.org/682842 | 09:09 |
| yoctozepto | ^ probably gonna fail | 09:10 |
| yoctozepto | but it's a start | 09:10 |
| yoctozepto | then we don't need to backport config changes not being a fix to CI operations just to satisfy newer branches | 09:11 |
| mgoddard | yoctozepto: so we're doing it for all configs? could get messy :( | 09:12 |
| yoctozepto | mgoddard: does it make sens to omit some? | 09:13 |
| yoctozepto | in global picture | 09:13 |
| mgoddard | yoctozepto: inventory can change over time | 09:13 |
| mgoddard | but normally only additions | 09:13 |
| yoctozepto | I am worried too but we have control over it in one place now | 09:13 |
| yoctozepto | so it's a tradeoff | 09:13 |
| mgoddard | we should stop templating daemon.json, as it gets overwritten now | 09:14 |
| mgoddard | but still need it for pre-upgrade ) | 09:14 |
| mgoddard | I'll make these comments in the review | 09:15 |
| yoctozepto | k thanks mgoddard | 09:18 |
| yoctozepto | we are failing already as expected | 09:18 |
| yoctozepto | well, not entirely as expected: | 09:19 |
| yoctozepto | Destination directory /etc/kolla/config/nova does not exist | 09:19 |
| yoctozepto | seems our little role did more magic | 09:19 |
| mgoddard | yoctozepto: git diff origin/stable/stein tests/templates/ | 09:21 |
| mgoddard | you'll need to change the docker config, add back removed groups in inventory | 09:21 |
| yoctozepto | mgoddard: yeah, that would be the next step if only it did not fail the actual templating :D | 09:22 |
| yoctozepto | brb | 09:23 |
| hrw | mgoddard: daemon.json content is merged | 09:23 |
| hrw | mgoddard: not overwritten | 09:23 |
| hrw | bbl | 09:24 |
| mgoddard | hrw: we didn't do that in the end | 09:24 |
| mgoddard | I'll fix that one up | 09:25 |
| openstackgerrit | Merged openstack/kolla master: CI: CentOS - override mirrors ASAP + crmsh mirror override https://review.opendev.org/682600 | 09:31 |
| mgoddard | ^ we're back in the game | 09:32 |
| mgoddard | on the OL issue, we should just remove it | 09:32 |
| mgoddard | we didn't promise to keep it | 09:32 |
| mgoddard | just haven't removed yet | 09:32 |
| yoctozepto | mgoddard: what about stein- ? | 09:33 |
| mgoddard | yoctozepto: we should fix that | 09:33 |
| mgoddard | assuming it's not too much effort | 09:34 |
| yoctozepto | mgoddard: if my plan works | 09:34 |
| yoctozepto | it will require only as much as centos | 09:34 |
| yoctozepto | so gets free | 09:34 |
| yoctozepto | ok, back on track with ansible | 09:34 |
| mgoddard | well let's go for the smallest solution on stable branches | 09:34 |
| mgoddard | I don't fully understand the problem yet | 09:35 |
| mgoddard | (haven't tried) | 09:35 |
| yoctozepto | mgoddard: package is gone :D | 09:35 |
| yoctozepto | we used absolute paths instead of extras repo | 09:35 |
| yoctozepto | (no idea why though) | 09:35 |
| yoctozepto | as for the ansible | 09:35 |
| yoctozepto | I convince myself we should only apply the change to globals.yml | 09:35 |
| yoctozepto | otherwise it gets messy for no gain | 09:35 |
| yoctozepto | as globals is the only current source of our issues to this point | 09:36 |
| yoctozepto | wdyt | 09:36 |
| yoctozepto | mgoddard, | 09:37 |
| mgoddard | sounds inconsistent :) | 09:37 |
| yoctozepto | indeed | 09:38 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: CI: Reinstate use of Docker registry mirror https://review.opendev.org/682846 | 09:38 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 09:38 |
| mnasiadka | yoctozepto: maybe it would be good to check what else we don't have mirrored (excluding the GitHub releases we fetch for some Docker images) and add that - I'll look into that later. | 09:41 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: [designate] Fix admin api configuration https://review.opendev.org/682847 | 09:45 |
| yoctozepto | mnasiadka: yeah, we should add some debug to name resolution to trace this | 09:46 |
| hrw | http://paste.debian.net/1101394/ - OL fix | 09:47 |
| hrw | to review or not to? | 09:47 |
| yoctozepto | hrw: check if it's the only one missing | 09:48 |
| yoctozepto | they did major rebuild there | 09:48 |
| hrw | yoctozepto: in progress | 09:48 |
| hrw | INFO:kolla.common.utils:Successfully built images | 09:48 |
| hrw | INFO:kolla.common.utils:========================= | 09:48 |
| hrw | INFO:kolla.common.utils:base | 09:48 |
| yoctozepto | but yeah, works fine as the smallest fix for sure | 09:48 |
| yoctozepto | ok, so review it | 09:48 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: base: fix OracleLinux build https://review.opendev.org/682849 | 09:49 |
| hrw | small, should be backportable | 09:49 |
| mnasiadka | and then we can remove OL ;) | 09:49 |
| yoctozepto | mnasiadka: yeah, I don't think it brings any benefits over centos as far as containers are concerned | 09:50 |
| yoctozepto | hrw: we like them small | 09:50 |
| mnasiadka | yoctozepto: and I don't think OL has any future | 09:50 |
| yoctozepto | mnasiadka: their UEK was nice | 09:51 |
| yoctozepto | is* | 09:51 |
| yoctozepto | see, you already made me doubt about OL future | 09:51 |
| yoctozepto | ;p | 09:51 |
| hrw | speaking of OL... my (personal) opinion is: no one in core team uses it, no one likes it so drop it. if someone care then they will complain and can bring it back | 09:52 |
| hrw | for br in stein rocky queens pike;do git checkout origin/stable/$br -b test/$br;git am 0001-base-fix-OracleLinux-build.patch && ~/devel/docker-utils/kolla-build-images.sh oraclelinux binary ^base $br;done | 09:52 |
| hrw | bbl | 09:53 |
| mnasiadka | yoctozepto: well, they released 8.0 | 09:56 |
| mnasiadka | yoctozepto: so maybe it has some pulse | 09:56 |
| mgoddard | hrw: https://bugs.launchpad.net/kolla/+bug/1814233 what do you mean we are hitting this again? Is rabbitmq failing to build and we don't know it? | 09:59 |
| openstack | Launchpad bug 1814233 in kolla "rabbitmq image fails to build" [Critical,Fix released] | 09:59 |
| mgoddard | hrw: is it actually the same bug or a similar one? | 09:59 |
| mgoddard | hrw: I'll add a patch to reinstate OL testing to test your fix | 10:00 |
| mgoddard | (DNM) | 10:00 |
| mgoddard | actually, it will be easier to just backport it | 10:03 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/stein: base: fix OracleLinux build https://review.opendev.org/682852 | 10:04 |
| *** openstackgerrit has quit IRC | 10:06 | |
| *** openstackgerrit has joined #openstack-kolla | 10:09 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/stein: base: fix OracleLinux build https://review.opendev.org/682852 | 10:09 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Configure the upgrade jobs from the current branch https://review.opendev.org/682842 | 10:10 |
| *** zbr has quit IRC | 10:11 | |
| yoctozepto | mgoddard: ^ | 10:11 |
| yoctozepto | leaving to grab some food | 10:11 |
| *** dciabrin has joined #openstack-kolla | 10:12 | |
| *** zbr has joined #openstack-kolla | 10:13 | |
| *** dciabrin has quit IRC | 10:17 | |
| openstackgerrit | Merged openstack/kolla master: Add effective and permitted capability to blackbox exporter https://review.opendev.org/677156 | 10:33 |
| openstackgerrit | Merged openstack/kolla-ansible master: Update stackalytics link for README https://review.opendev.org/682806 | 10:33 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Configure the upgrade jobs from the current branch https://review.opendev.org/682842 | 10:37 |
| *** dciabrin has joined #openstack-kolla | 10:37 | |
| *** jdwidari has joined #openstack-kolla | 11:01 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Configure the upgrade jobs from the current branch https://review.opendev.org/682842 | 11:01 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Reinstate use of Docker registry mirror https://review.opendev.org/682846 | 11:05 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Configure the upgrade jobs from the current branch https://review.opendev.org/682842 | 11:07 |
| openstackgerrit | Dincer Celik proposed openstack/kolla master: build: enable Octavia for Ubuntu/binary https://review.opendev.org/682671 | 11:14 |
| *** pcaruana has quit IRC | 11:19 | |
| *** kplant has joined #openstack-kolla | 11:19 | |
| *** dave-mccowan has joined #openstack-kolla | 11:23 | |
| openstackgerrit | Dincer Celik proposed openstack/kolla master: build: enable Octavia for Ubuntu/binary https://review.opendev.org/682671 | 11:24 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Remove support for OracleLinux https://review.opendev.org/682858 | 11:26 |
| *** pcaruana has joined #openstack-kolla | 11:28 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 11:32 |
| *** Luzi has quit IRC | 11:35 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Remove support for OracleLinux https://review.opendev.org/682865 | 11:41 |
| *** zhangfei has quit IRC | 11:42 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix prometheus-alertmanager cluster bug https://review.opendev.org/682132 | 11:43 |
| openstackgerrit | Merged openstack/kolla-ansible master: Update python_path for "source" install type https://review.opendev.org/675822 | 11:43 |
| openstackgerrit | Merged openstack/kolla-ansible master: Ignore create_cells and discover_computes when nova-api is disabled https://review.opendev.org/681376 | 11:43 |
| openstackgerrit | Merged openstack/kolla master: Update skydive to v0.25.0 https://review.opendev.org/681768 | 11:43 |
| *** Luzi has joined #openstack-kolla | 11:49 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 11:58 |
| *** AJaeger has joined #openstack-kolla | 12:02 | |
| hrw | mgoddard: commented | 12:02 |
| *** AJaeger has left #openstack-kolla | 12:02 | |
| openstackgerrit | Dincer Celik proposed openstack/kolla master: build: enable Octavia for Ubuntu/binary https://review.opendev.org/682671 | 12:03 |
| *** janki has quit IRC | 12:07 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: base: enable rabbitmq and elasticsearch repos for ppc64le https://review.opendev.org/682618 | 12:09 |
| hrw | rebased | 12:09 |
| *** goldyfruit_ has quit IRC | 12:15 | |
| *** k_mouza has quit IRC | 12:16 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 12:17 |
| hrw | yoctozepto: how goes mirroring that opensuse repo? | 12:18 |
| mnasiadka | I think it's merged | 12:26 |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: Fixes destroys. https://review.opendev.org/668288 | 12:26 |
| *** redrobot has joined #openstack-kolla | 12:26 | |
| hrw | mnasiadka: but is it mirrored... | 12:26 |
| mnasiadka | yup - http://mirror.iad.rax.openstack.org/opensuse/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/ | 12:27 |
| *** luksky has quit IRC | 12:27 | |
| hrw | they are | 12:28 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Remove support for OracleLinux https://review.opendev.org/682865 | 12:43 |
| *** scottsol has joined #openstack-kolla | 12:48 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Fix prometheus-alertmanager cluster bug https://review.opendev.org/682888 | 12:50 |
| *** janki has joined #openstack-kolla | 12:52 | |
| *** janki has quit IRC | 12:53 | |
| *** vmixor has joined #openstack-kolla | 13:02 | |
| yoctozepto | hrw: yeah, today merged our usage of it | 13:09 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Configure the upgrade jobs from the current branch https://review.opendev.org/682842 | 13:11 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Configure the upgrade jobs from the current branch https://review.opendev.org/682842 | 13:11 |
| *** luksky has joined #openstack-kolla | 13:19 | |
| hrw | yoctozepto: series of recheck sent | 13:21 |
| yoctozepto | hrw: good | 13:22 |
| *** k_mouza has joined #openstack-kolla | 13:22 | |
| yoctozepto | it's worth doing what mnasiadka suggested - confirming we mirror everything we need and use only mirrored | 13:22 |
| *** k_mouza has quit IRC | 13:26 | |
| *** scottsol has quit IRC | 13:29 | |
| *** k_mouza has joined #openstack-kolla | 13:29 | |
| *** scottsol has joined #openstack-kolla | 13:30 | |
| *** goldyfruit_ has joined #openstack-kolla | 13:30 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla master: Add volume-related packages to zun-compute image https://review.opendev.org/682437 | 13:34 |
| *** hjensas|afk has quit IRC | 13:36 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Add support to enable l3 port-forwarding plugin https://review.opendev.org/681636 | 13:42 |
| *** BjoernT has joined #openstack-kolla | 13:48 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Zun: add volume support https://review.opendev.org/682900 | 13:48 |
| *** Luzi has quit IRC | 13:49 | |
| *** k_mouza has quit IRC | 13:51 | |
| *** JamesBenson has joined #openstack-kolla | 14:02 | |
| andrein | mgoddard: got the VLAN stuff sorted out! YAY! | 14:08 |
| mgoddard | andrein: nice work | 14:09 |
| andrein | now, the next two things in my todo list: setting up LACP and Software RAID (https://storyboard.openstack.org/#!/story/2004581 - I know, I'm going to have to upgrade to master to get it) | 14:09 |
| andrein | any pointers for the LACP? | 14:09 |
| mgoddard | andrein: we normally configure bonds to fall back to a single interface on the switch, to allow PXE to work | 14:11 |
| kplant | ^ | 14:11 |
| mgoddard | it depends on the switch model if/how to do it | 14:11 |
| kplant | junos is force-up | 14:11 |
| kplant | extremely useful | 14:11 |
| mgoddard | +1 | 14:11 |
| kplant | make sure you only do it on one lag member though | 14:11 |
| kplant | i also recommend doing active lacp and 1s (fast) instead of 30s (slow) intervals | 14:12 |
| andrein | mgoddard: ahead of you on the switch config, it's all there :) | 14:12 |
| kplant | as well as layer2 and layer3 lb | 14:13 |
| andrein | kplant: | 14:13 |
| andrein | kplant: why do it on a single LAG member though? | 14:13 |
| kplant | so i'll use mgoddard's example of PXE | 14:13 |
| mgoddard | andrein: so now you hit the problem that bifrost can't configure bond interfaces | 14:13 |
| kplant | if both interfaces are up/up, and the switch is configured to force-up the lag from both members | 14:14 |
| kplant | it will try to send traffic over both lag members | 14:14 |
| mgoddard | we do it after deploy using kayobe, although it doesn't currently work with a VLAN subinterface and a bond | 14:14 |
| kplant | so in the case of pxe, where both ports are L1 up, it will try to send traffic to both interfaces | 14:14 |
| kplant | where really you only want the interface that's doing pxe | 14:14 |
| kplant | make sense? | 14:14 |
| kplant | force-up basically says to send traffic over the lag member if it's not speaking lacp | 14:14 |
| andrein | kplant: yeah, makes sense, i'll double check with my nework engineer, thanks for the heads up! | 14:15 |
| kplant | yeah i got burned by that early on | 14:15 |
| kplant | pxe over a vlan without a native vlan is difficult @mgoddard | 14:16 |
| kplant | sometimes the NIC will just put the tag on for you, sometimes it doesn't | 14:16 |
| kplant | blah | 14:16 |
| yoctozepto | my fujitsu systems do not support pxe with tagged-only interfaces ;/ | 14:21 |
| *** openstackgerrit has quit IRC | 14:21 | |
| andrein | mgoddard: no vlans over bonds? :( | 14:23 |
| yoctozepto | mgoddard: some ironic failure in CI: https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_3af/682618/2/check/kolla-ansible-centos-source-ironic/3af4e4a/primary/logs/ansible/test-ironic - it is easier for you to digest probably | 14:26 |
| mgoddard | yoctozepto: nom nom nom | 14:27 |
| *** openstackgerrit has joined #openstack-kolla | 14:27 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Remove support for OracleLinux https://review.opendev.org/682858 | 14:27 |
| openstackgerrit | Merged openstack/kolla stable/stein: base: fix OracleLinux build https://review.opendev.org/682852 | 14:27 |
| yoctozepto | mgoddard: yeah, iron is healthy | 14:28 |
| *** wuchunyang has joined #openstack-kolla | 14:39 | |
| mgoddard | yoctozepto: which patch? it's hard to see what actually went wrong | 14:40 |
| mgoddard | possibly that it's SHUTOFF? | 14:41 |
| mgoddard | could be related to new nova-ironic callbacks | 14:41 |
| mgoddard | config added for it here: https://review.opendev.org/680737 | 14:42 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 14:46 |
| *** priteau has quit IRC | 14:47 | |
| yoctozepto | mgoddard: unrelated hrw's patch | 14:47 |
| hrw | ? | 14:47 |
| yoctozepto | https://review.opendev.org/#/c/682618/ | 14:47 |
| *** k_mouza has joined #openstack-kolla | 14:48 | |
| *** luksky has quit IRC | 14:48 | |
| yoctozepto | hrw: I was asking mgoddard about the failure reason | 14:48 |
| hrw | ah | 14:48 |
| mgoddard | meeting in 10 minutes | 14:50 |
| mgoddard | mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne | 14:50 |
| hrw | ok | 14:51 |
| *** priteau has joined #openstack-kolla | 14:51 | |
| mnasiadka | so, is there a way to get ARA visualisation for steps inside kolla-ansible deploy now? Or do we only get what is in Zuul console tab, which is Zuul level? | 14:53 |
| *** wuchunyang has quit IRC | 14:53 | |
| *** gkadam has joined #openstack-kolla | 14:54 | |
| mnasiadka | yoctozepto: nice error, timeout connecting to Docker API on localhost? :) | 14:55 |
| mnasiadka | https://zuul.opendev.org/t/openstack/build/6fba26aab357413e8e6dc416891db478/log/primary/logs/ansible/deploy#81125 | 14:55 |
| *** gkadam_ has joined #openstack-kolla | 14:55 | |
| *** gkadam has quit IRC | 14:56 | |
| *** priteau has quit IRC | 14:56 | |
| yoctozepto | mnasiadka: we need to add argument to ara so that it generates a static version of it | 14:57 |
| yoctozepto | so far been lazy | 14:58 |
| mnasiadka | yoctozepto: yeah, I thought so | 14:58 |
| *** wuchunyang has joined #openstack-kolla | 14:58 | |
| *** priteau has joined #openstack-kolla | 14:58 | |
| yoctozepto | is that one of my changes? | 14:58 |
| yoctozepto | with that sad error | 14:58 |
| *** chason has joined #openstack-kolla | 14:58 | |
| mnasiadka | yoctozepto: the one you referenced a couple of lines earlier :) | 14:59 |
| yoctozepto | ah, the nfv case | 15:00 |
| hrw | ding! 1700 | 15:00 |
| yoctozepto | there was some loss of network it seems | 15:01 |
| yoctozepto | mgoddard | 15:01 |
| yoctozepto | do your job | 15:01 |
| yoctozepto | :D | 15:01 |
| mgoddard | zzz | 15:01 |
| mgoddard | #startmeeting kolla | 15:01 |
| openstack | Meeting started Wed Sep 18 15:01:24 2019 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| *** openstack changes topic to " (Meeting topic: kolla)" | 15:01 | |
| openstack | The meeting name has been set to 'kolla' | 15:01 |
| mgoddard | #topic rollcall | 15:01 |
| *** openstack changes topic to "rollcall (Meeting topic: kolla)" | 15:01 | |
| mgoddard | \o | 15:01 |
| hrw | /o\ | 15:01 |
| yoctozepto | o/ | 15:01 |
| scottsol | o/ | 15:01 |
| chason | o/ | 15:01 |
| mnasiadka | o/ | 15:01 |
| generalfuzz | o/ | 15:01 |
| priteau | o | 15:02 |
| priteau | o/ | 15:02 |
| yankcrime | \o | 15:02 |
| mgoddard | welcome back yankcrime | 15:02 |
| yankcrime | hi mgoddard | 15:02 |
| *** wuchunyang has quit IRC | 15:03 | |
| mgoddard | #topic agenda | 15:03 |
| *** openstack changes topic to "agenda (Meeting topic: kolla)" | 15:03 | |
| mgoddard | * Roll-call | 15:03 |
| mgoddard | * Announcements | 15:03 |
| mgoddard | ** Vote for Kayobe Train planning meeting time: https://doodle.com/poll/y7fakbhbkfx5hqyk | 15:03 |
| mgoddard | * Review action items from last meeting | 15:03 |
| mgoddard | * CI status | 15:03 |
| mgoddard | * Train release planning | 15:03 |
| mgoddard | * Review priorities | 15:03 |
| mgoddard | * Kayobe Train planning | 15:03 |
| mgoddard | #topic announcements | 15:03 |
| *** openstack changes topic to "announcements (Meeting topic: kolla)" | 15:03 | |
| mgoddard | #info Vote for Kayobe Train planning meeting time | 15:03 |
| mgoddard | #link https://doodle.com/poll/y7fakbhbkfx5hqyk | 15:03 |
| mgoddard | Will discuss this a little more later | 15:04 |
| mgoddard | #info CentOS 7.7 released yesterday | 15:04 |
| mgoddard | only minimal CI breakage this time | 15:04 |
| *** gkadam_ has quit IRC | 15:04 | |
| mgoddard | #info CentOS 8 released next week, 24th | 15:05 |
| *** jovial[m] has joined #openstack-kolla | 15:05 | |
| hrw | In october I may take a look at centos:8 builds | 15:06 |
| mgoddard | #info OpenStack projects releasing RC1 next week, final release 14-18 Oct | 15:06 |
| mgoddard | that will be good hrw | 15:06 |
| mgoddard | we should try to get ahead as soon as possible in case there are problems we need to report back | 15:07 |
| mgoddard | Any others? | 15:07 |
| yoctozepto | ipv6 is not a goal of deployment projects in train | 15:07 |
| yoctozepto | probably in ussuri | 15:07 |
| mgoddard | Right | 15:07 |
| yoctozepto | so we are step ahead 8-) | 15:07 |
| mgoddard | still nice to have | 15:07 |
| mgoddard | always | 15:07 |
| *** chason has quit IRC | 15:07 | |
| yoctozepto | indeed nice | 15:07 |
| mgoddard | (when complete) | 15:07 |
| mgoddard | #topic Review action items from last meeting | 15:08 |
| *** openstack changes topic to "Review action items from last meeting (Meeting topic: kolla)" | 15:08 | |
| mgoddard | There were none | 15:08 |
| *** chason has joined #openstack-kolla | 15:08 | |
| mgoddard | #topic CI status | 15:08 |
| *** openstack changes topic to "CI status (Meeting topic: kolla)" | 15:08 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/rocky: base: fix OracleLinux build https://review.opendev.org/682927 | 15:08 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/queens: base: fix OracleLinux build https://review.opendev.org/682928 | 15:09 |
| mgoddard | We have issues with oraclelinux on stable branches. Fixed in stein, looks like yoctozepto is backporting as we speak | 15:09 |
| yoctozepto | getting unbroken as we speak 8-) | 15:09 |
| hrw | +2 on both | 15:09 |
| yoctozepto | ol is not an issue of train | 15:09 |
| yoctozepto | and we are dropping it nicely | 15:09 |
| mgoddard | we had some flakiness in kayobe CI bare metal deploys, 'fixed' with iptables | 15:10 |
| yoctozepto | why 'fixed'? | 15:10 |
| mgoddard | https://review.opendev.org/#/c/681700/ | 15:10 |
| yoctozepto | weird indeed | 15:11 |
| yoctozepto | at least it should always fail | 15:11 |
| mgoddard | cause was docker CE adding a default DROP policy to the FORWARD chain, in combination with bridge-nf-call-iptables | 15:11 |
| mgoddard | I never really understood why it only sometimes failed, and exactly what the problem was | 15:11 |
| mgoddard | but a blanket ACCEPT on the FORWARD chain did the trick | 15:12 |
| yoctozepto | worth noting if it does not break real deploys somewhere | 15:12 |
| mgoddard | +1, docker's policy has caused a few different issues for us | 15:12 |
| priteau | I've had issues on a Queens deployment using DVR | 15:12 |
| priteau | I ended up running docker with --iptables=false | 15:13 |
| yoctozepto | maybe it does something funny on the lower layers as well (raw, mangle, nat stuff) | 15:13 |
| yoctozepto | I seem to forget about raw and it can be tricky left unchecked :D | 15:14 |
| yoctozepto | (vide that ipv6 problem with firewalld on) | 15:14 |
| mnasiadka | maybe it's time to add iptables: false to /etc/docker/daemon.json ;-) | 15:14 |
| mgoddard | priteau: that is tempting. It wouldn't hurt kolla-ansible, I was just worried in case people run other containers without net=host | 15:14 |
| priteau | There are security implications | 15:15 |
| yoctozepto | mnasiadka, mgoddard, priteau: zun impact | 15:15 |
| mnasiadka | yoctozepto: I don't care about Zun, who uses it anyway :) | 15:15 |
| mgoddard | ouch | 15:16 |
| yoctozepto | behave, mnasiadka, behave | 15:16 |
| mnasiadka | Frankly speaking I never understood why we use the same container runtime for control plane and Zun | 15:16 |
| yoctozepto | mnasiadka: because we cannot otherwise | 15:17 |
| mnasiadka | yoctozepto: I would challenge that, but this is not the meeting to do it :) | 15:17 |
| yoctozepto | yeah, I would like to hear that | 15:17 |
| mgoddard | I think we have digressed | 15:17 |
| yoctozepto | indeed | 15:17 |
| mgoddard | #topic Train release planning | 15:17 |
| *** openstack changes topic to "Train release planning (Meeting topic: kolla)" | 15:17 | |
| *** ivve has quit IRC | 15:18 | |
| mgoddard | According to our release status section, we have 2 weeks until our feature freeze | 15:18 |
| mgoddard | Kolla feature freeze: Sep 30 - Oct 04 | 15:18 |
| mgoddard | I think we'll need to exclude centos8 & centos py3 from that | 15:19 |
| mgoddard | Let's assess what other priorities look like | 15:19 |
| hrw | yep | 15:19 |
| yoctozepto | mgoddard: or do the standard 2.5 month delay | 15:20 |
| mgoddard | support matrix won't be affected by freeze | 15:20 |
| *** goldyfruit_ has quit IRC | 15:20 | |
| mgoddard | yoctozepto: we still froze a few weeks after everyone else IIRC | 15:20 |
| yoctozepto | mgoddard: seems it got hit by hot winds then | 15:20 |
| mgoddard | we'd just need to backport anything after RC1 | 15:21 |
| openstackgerrit | Merged openstack/kayobe stable/stein: Fix kayobe-overcloud-centos CI job flakiness https://review.opendev.org/682829 | 15:21 |
| mgoddard | I think that's still a good approach, agree? | 15:21 |
| mgoddard | ^ oh nice I can mark kayobe CI GREEN again | 15:21 |
| mgoddard | Nova cells could be at risk - dougsz is away next week | 15:22 |
| mgoddard | I'd like to get the nova part of that work merged this cycle | 15:23 |
| mgoddard | (at least) | 15:23 |
| mgoddard | https://review.opendev.org/675659 | 15:23 |
| mgoddard | We finished the ansible upgrade \o/ | 15:24 |
| mgoddard | Internal TLS is coming along nicely, first patch merged | 15:24 |
| mgoddard | I don't know if we'll get the backend patch in this cycle | 15:25 |
| mgoddard | but we'll see | 15:25 |
| mgoddard | this one is now a dependency of TLS: https://review.opendev.org/#/c/587590 | 15:26 |
| mgoddard | and then it's this one: https://review.opendev.org/#/c/664517 | 15:26 |
| mnasiadka | I love those multi file small fixes :) | 15:26 |
| hrw | maybe we should start use BP+1/-1 to mark what we need in Train and what goes for Ussuri? | 15:27 |
| mgoddard | oh yeah, get all those CI jobs going | 15:27 |
| mgoddard | I've been adding RP+1 for train priorities | 15:27 |
| mgoddard | RP-1 seems a bit mean :) | 15:28 |
| mgoddard | but maybe nearer the freeze it will help | 15:28 |
| mgoddard | what's the status of health checks mnasiadka? | 15:28 |
| mnasiadka | I doubt I'll get them in this cycle, looks like some debugging needed on docker-py level to get this working - and then there's a lot of work ahead | 15:29 |
| yoctozepto | :-( | 15:29 |
| mgoddard | k, good to know so we can focus elsewhere | 15:29 |
| mgoddard | yoctozepto: reckon IPv6 can land in time? | 15:30 |
| yoctozepto | mgoddard: I confirm, it works, CI is just grumpy :-) | 15:30 |
| yoctozepto | (well, ubuntu worked well) | 15:30 |
| mgoddard | ok | 15:31 |
| mgoddard | I added patches to generate PDF docs | 15:32 |
| mgoddard | https://review.opendev.org/682610 | 15:32 |
| mgoddard | https://review.opendev.org/682611 | 15:32 |
| mgoddard | https://review.opendev.org/682616 | 15:32 |
| mgoddard | shamelessly adapted from ironic's hard work | 15:32 |
| mgoddard | it's a community goal, so I'll mark RP+1 | 15:32 |
| yoctozepto | seems we need more irrelevant files around | 15:33 |
| mgoddard | maybe | 15:33 |
| yoctozepto | no need to test docs with ansible | 15:33 |
| mgoddard | had to fix requirements.txt for those: https://review.opendev.org/#/c/682646 | 15:33 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 15:34 |
| hrw | +2 on pdf | 15:34 |
| mgoddard | I think that's all the big stuff. I marked some things as being pushed out to Ussuri | 15:34 |
| mgoddard | Did I miss anything? | 15:35 |
| mgoddard | #topic Review priorities | 15:36 |
| *** openstack changes topic to "Review priorities (Meeting topic: kolla)" | 15:36 | |
| mgoddard | Anything we should focus on this week? | 15:36 |
| *** zbr has quit IRC | 15:36 | |
| mgoddard | already marked PDFs | 15:36 |
| mgoddard | Cells | 15:37 |
| yoctozepto | vxlan | 15:37 |
| *** zbr has joined #openstack-kolla | 15:37 | |
| yoctozepto | ipv6 | 15:37 |
| yoctozepto | I do, you review | 15:37 |
| hrw | -OL | 15:37 |
| mgoddard | I think -OL went in already? | 15:37 |
| yoctozepto | same here | 15:37 |
| mgoddard | ah, not kolla | 15:37 |
| mgoddard | added | 15:38 |
| hrw | OL stuff: https://review.opendev.org/#/c/682927/ https://review.opendev.org/#/c/682928/ https://review.opendev.org/#/c/682865/ | 15:38 |
| mgoddard | yoctozepto: could you add bp/ topic to your IPv6 patches? | 15:39 |
| mgoddard | https://review.opendev.org/#/q/topic:bp/ipv6-control-plane | 15:39 |
| mgoddard | yoctozepto: feel free to add RP+1 to your IPv6 patches | 15:40 |
| mnasiadka | mgoddard: what's the plan with ceph-ansible spec? do we want to merge it now, in Ussuri or never? ;-) | 15:40 |
| mgoddard | mnasiadka: hmm | 15:40 |
| mnasiadka | Ussuri might make sense, we'll know more after the experimental CI job with ceph-ansible will be merged ;-) | 15:40 |
| mgoddard | mnasiadka: +2 | 15:41 |
| yoctozepto | mgoddard: ok, right when done | 15:41 |
| mgoddard | mnasiadka: I think the direction is sound, even if we don't know when we'll do it | 15:42 |
| mgoddard | Kayobe priorities this week? | 15:42 |
| mgoddard | priteau, jovial[m]? | 15:42 |
| *** k_mouza has quit IRC | 15:42 | |
| priteau | I am going over the storyboard and identifying things we can fix in Train | 15:43 |
| mgoddard | great. we can discuss the train planning in the next section, just looking for review priorities for this week atm | 15:44 |
| mgoddard | we have a few bug fixes waiting for review, I'll pick some and mark RP+1 | 15:45 |
| priteau | Oh, sorry | 15:45 |
| mgoddard | priteau: np | 15:45 |
| mgoddard | Anyone know about kolla-cli | 15:45 |
| priteau | Review priority: stable/queens backport https://review.opendev.org/#/c/679269/ | 15:45 |
| mgoddard | priteau: +1. Feel free to add RP+1 | 15:45 |
| mgoddard | Is kolla-cli going to be ready to release? | 15:46 |
| yoctozepto | mgoddard: it looks it was a temporary activity burst | 15:46 |
| jovial[m] | I'll try and review 679269 now | 15:46 |
| mgoddard | yoctozepto: I have seen a few patches recently | 15:47 |
| mnasiadka | mgoddard: do you have any contact with somebody that is merging that work? ;-) | 15:48 |
| yoctozepto | they don't seem to visit irc channel | 15:48 |
| mnasiadka | probably great firewall of China is to blame | 15:48 |
| yoctozepto | probably | 15:48 |
| mgoddard | yeah | 15:48 |
| yoctozepto | let's mail them then | 15:49 |
| mgoddard | mnasiadka: not really. Jeffrey4l is the only one from 99cloud who is on IRC AFAIK | 15:49 |
| mgoddard | ok, last 10 minutes | 15:49 |
| mnasiadka | mgoddard: I guess this is not very easy for him, I worked with some Chinese people in IBM, and they had to do a lot of tunnelling :-) | 15:49 |
| mgoddard | I expect so | 15:50 |
| yoctozepto | mgoddard: maybe he has higher clearance ;-) | 15:50 |
| mgoddard | #action mgoddard to email Jeffrey4l & co about kolla-cli | 15:50 |
| mgoddard | #topic Kayobe Train planning | 15:50 |
| *** openstack changes topic to "Kayobe Train planning (Meeting topic: kolla)" | 15:50 | |
| mgoddard | Ok kayobians | 15:50 |
| mgoddard | priteau & jovial[m] | 15:50 |
| * yoctozepto just eavesdropping | 15:51 | |
| mgoddard | Currently no slot next week suits everyone :( | 15:51 |
| mgoddard | If I allow for even hours does that help? | 15:51 |
| mgoddard | couldn't see any way to overlap in doodle | 15:51 |
| mgoddard | I think I'll just need to pick one and apologise | 15:53 |
| mgoddard | We should have a think about what to talk about in advance. priteau started an etherpad | 15:53 |
| jovial[m] | +1 | 15:54 |
| mgoddard | #link https://etherpad.openstack.org/p/kayobe-train-release | 15:54 |
| *** cah_link has quit IRC | 15:54 | |
| *** k_mouza has joined #openstack-kolla | 15:55 | |
| mgoddard | I think we could follow a similar pattern to kolla PTG | 15:55 |
| mgoddard | dump a list of things to discuss, then vote on priorities | 15:55 |
| mgoddard | bearing in mind we don't have much time | 15:55 |
| mgoddard | but priorities can always roll over | 15:55 |
| mgoddard | so, bring your ideas for discussion, and comb storyboard for anything that we should be picking up | 15:56 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 15:56 |
| mgoddard | anything else we should be doing in those 2 hours? | 15:56 |
| priteau | Discuss path forward for Kolla and Kayobe? | 15:57 |
| jovial[m] | is remaining patches for stuff that already has a change up for review? | 15:57 |
| priteau | Are there things we need to change in how the two integrate together | 15:57 |
| mgoddard | priteau: good question | 15:57 |
| mgoddard | it's a good time for open questions like that | 15:57 |
| mgoddard | chuck it on the pad | 15:58 |
| mgoddard | the format of the pad probably needs cleaning up | 15:58 |
| mgoddard | it's more of a status tracking pad | 15:58 |
| mgoddard | should we have a fresh one for ths discussion? | 15:59 |
| priteau | +1 | 15:59 |
| mgoddard | ok | 15:59 |
| mgoddard | https://etherpad.openstack.org/p/kayobe-train-planning | 15:59 |
| priteau | Since we just released Stein, this is a kind of PTG meeting | 15:59 |
| mgoddard | #link Welcome to Etherpad Lite! | 15:59 |
| mgoddard | This pad text is synchronized as you type, so that everyone viewing this page sees the same text. This allows you to collaborate seamlessly on documents! | 15:59 |
| priteau | without the snow | 15:59 |
| mgoddard | Etherpad Lite on Github: http://j.mp/ep-lite | 15:59 |
| mgoddard | n | 15:59 |
| mgoddard | oops | 15:59 |
| mgoddard | #link https://etherpad.openstack.org/p/kayobe-train-planning | 15:59 |
| mgoddard | could be snowing somewhere | 16:00 |
| mgoddard | it's 1600UTC, regrettably we must end | 16:01 |
| mgoddard | thanks everyone | 16:01 |
| mgoddard | the train is fast approaching | 16:01 |
| mgoddard | #endmeeting | 16:01 |
| *** openstack changes topic to "Topic for #openstack-kolla is: support: ask.openstack.org | New to Kolla: docs @ https://docs.openstack.org/kolla/latest/ | | Kolla IRC meetngs on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b" | 16:01 | |
| openstack | Meeting ended Wed Sep 18 16:01:51 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:01 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-09-18-15.01.html | 16:01 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-09-18-15.01.txt | 16:01 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-09-18-15.01.log.html | 16:01 |
| *** henriqueof has quit IRC | 16:02 | |
| *** rpittau is now known as rpittau|afk | 16:03 | |
| yankcrime | fyi i called kolla 'fucking magic' during a lightning talk i did at the london openstack meetup | 16:04 |
| scottsol | ^^ he did | 16:04 |
| yankcrime | just thought you should know i'm still doing my advocacy bit for the project ❤️ | 16:04 |
| openstackgerrit | Merged openstack/kolla master: base: enable rabbitmq and elasticsearch repos for ppc64le https://review.opendev.org/682618 | 16:05 |
| *** zbr is now known as zbr|ruck | 16:07 | |
| *** hjensas|afk has joined #openstack-kolla | 16:08 | |
| mnasiadka | mgoddard: keystone role refactor change is merging - do we still have any usage for kolla_keystone* modules? | 16:08 |
| mgoddard | mnasiadka: boom | 16:09 |
| yoctozepto | mnasiadka: now tell me about your idea for running a separate zun backend | 16:09 |
| mnasiadka | mgoddard: I think it should be at least boom^2 | 16:09 |
| mgoddard | I don't believe so. But we should wait before removing to provide a transition | 16:09 |
| mgoddard | maybe deprecate for removal in U? | 16:09 |
| mnasiadka | yoctozepto: I don't know Zun, but what's the problem to run multiple container runtimes on one host? | 16:10 |
| mnasiadka | mgoddard: yeah, I was thinking about that | 16:10 |
| mgoddard | nice | 16:10 |
| mgoddard | yankcrime: you legend | 16:10 |
| yoctozepto | mnasiadka: what multiple you mean | 16:10 |
| yoctozepto | port kolla to podman and run zun on podman-deployed docker? :D | 16:11 |
| mgoddard | yankcrime: how was the meetup? | 16:11 |
| yankcrime | down on numbers, but it's been nearly 2 years since the last one so it's lost all momentum | 16:12 |
| yankcrime | hopefully it'll pick up again though, we've got the next one scheduled for november 28th at the microsoft reactor of all places | 16:12 |
| *** gfidente has quit IRC | 16:12 | |
| noxoid | yoctozepto, you good with me just copying what you did for cinder for designate's coordination backend? https://review.opendev.org/#/c/682095/ | 16:12 |
| yoctozepto | noxoid: sure, we can later globalize it | 16:13 |
| noxoid | perfect. ill try to get that done in the next day or two | 16:13 |
| yoctozepto | refer the bug as related | 16:13 |
| noxoid | thanks | 16:13 |
| noxoid | yep | 16:14 |
| mnasiadka | yoctozepto: no, run another docker/crio/podman in addition :) | 16:14 |
| mnasiadka | yoctozepto: never done that? :) | 16:15 |
| *** radeks has quit IRC | 16:16 | |
| mnasiadka | yoctozepto: there's dind I guess, but that's not really production grade :) | 16:17 |
| yoctozepto | mnasiadka: what benefits does it bring though? If I enable iptables in either it's going to feel like the king of firewall anyway :-) | 16:17 |
| mnasiadka | yoctozepto: I just mean that we do crazy stuff with docker daemon config for control plane, which might influence it's behaviour | 16:18 |
| mnasiadka | crazy == configure Etcd :) | 16:18 |
| yoctozepto | mnasiadka: I see | 16:18 |
| yoctozepto | well, it's food for thought re: stability of solution | 16:18 |
| mnasiadka | yoctozepto: maybe I'm just over-cautious, or something | 16:19 |
| yoctozepto | nah, that's ok | 16:19 |
| yoctozepto | mgoddard, mnasiadka: https://review.opendev.org/682842 <- works | 16:20 |
| yoctozepto | now can rebase vxlan on that | 16:20 |
| yoctozepto | and make mnasiadka happy again | 16:20 |
| mnasiadka | lol | 16:20 |
| yoctozepto | make CI green again 8-) | 16:20 |
| *** chason has quit IRC | 16:21 | |
| *** gmann is now known as gmann_afk | 16:23 | |
| *** goldyfruit_ has joined #openstack-kolla | 16:28 | |
| *** ivve has joined #openstack-kolla | 16:32 | |
| *** k_mouza has quit IRC | 16:43 | |
| *** scottsol has quit IRC | 16:50 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Use VXLAN overlay network https://review.opendev.org/670690 | 16:52 |
| *** dougsz has quit IRC | 16:56 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Use VXLAN overlay network https://review.opendev.org/670690 | 16:58 |
| andrein | mgoddard: are you still around? | 17:04 |
| *** luksky has joined #openstack-kolla | 17:06 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Use VXLAN overlay network https://review.opendev.org/670690 | 17:07 |
| openstackgerrit | Merged openstack/kolla-ansible master: Build pdf doc https://review.opendev.org/682611 | 17:11 |
| shubjero | I am trying to follow the multinode instructions for kolla-ansible and its saying a docker registry is mandatory but I've never done that and the instructions dont tell me how to get the 'kolla' directory to run a 'start-registry'. Do i just need to clone the specific github repo? https://docs.openstack.org/project-deploy-guide/kolla-ansible/rocky/multinode.html | 17:14 |
| kplant | it's not actually mandatory | 17:15 |
| kplant | if you don't specify one, it will use docker.io | 17:15 |
| shubjero | oh | 17:16 |
| shubjero | for testing thats fine thenn, thanks kplant | 17:16 |
| kplant | np | 17:16 |
| kplant | if you're wanting to setup a registry though it's not hard, you can just plop docker on a machine/vm start a registry container and start pushing images to it | 17:17 |
| kplant | i think it needs to be tls with a legitimate certificate though | 17:17 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Use VXLAN overlay network https://review.opendev.org/670690 | 17:17 |
| shubjero | yeah i did a bit of reading and looked in to the start-registry file on github so now i know what it does | 17:17 |
| shubjero | i'll get there eventually | 17:18 |
| kplant | it's definitely the way to go, way faster and more reliable | 17:18 |
| kplant | you don't want docker.io/cloudflare being down stopping your deployment | 17:18 |
| shubjero | exactly | 17:18 |
| openstackgerrit | Merged openstack/kayobe stable/queens: Allocate IPs separately from configuring network https://review.opendev.org/679269 | 17:24 |
| shubjero | fatal: [kolla-c1]: FAILED! => {"changed": false, "msg": "Unable to find any of pip3 to use. pip needs to be installed."} | 17:27 |
| shubjero | hmm? | 17:28 |
| shubjero | this is when im doing a multinode bootstrap | 17:28 |
| kplant | did you get kolla via git or pip | 17:29 |
| shubjero | 'pip install kolla-ansible' | 17:29 |
| kplant | hmm | 17:30 |
| shubjero | there was one option i was not confident in selecting.. the binary/source choice in globals.yml | 17:31 |
| shubjero | i chose source | 17:31 |
| kplant | yeah source is the better choice | 17:31 |
| kplant | what about version? | 17:31 |
| shubjero | rocky | 17:31 |
| kplant | i'm unsure why it's trying to use py3 | 17:31 |
| kplant | i've deployed stein on centos and i don't have python3 at all :-\ | 17:32 |
| shubjero | the underlying OS is 18.04.. I know it said for new users to use 16.04 but our prod env is all 18.04 now so i didnt want to back pedal | 17:32 |
| shubjero | (ubuntu) | 17:32 |
| shubjero | 1s | 17:33 |
| kplant | hmm, i've only ever done centos | 17:33 |
| kplant | no me gusta ubuntu | 17:33 |
| openstackgerrit | Merged openstack/kolla-ansible master: Refactor service, endpoint and user registration https://review.opendev.org/587590 | 17:34 |
| shubjero | i told ansible to use python3 as the ansible intpreter in the inventory file.. let me inset that | 17:34 |
| shubjero | *unset | 17:34 |
| kplant | oh yeah | 17:34 |
| kplant | don't do that | 17:34 |
| shubjero | yup, fixed | 17:34 |
| kplant | are you using virtualenvs on the target hosts? | 17:34 |
| shubjero | dont think so. I'm just running 'kolla-ansible -i ./multinode bootstrap-servers' at this point | 17:35 |
| kplant | then you're fine not specifying | 17:35 |
| shubjero | i spun up 3 controller nodes and 1 compute node and just testing kolla for the first time | 17:35 |
| *** zbr|ruck is now known as zbr | 17:36 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Use VXLAN overlay network https://review.opendev.org/670690 | 17:40 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Use VXLAN overlay network https://review.opendev.org/670690 | 17:51 |
| *** priteau has quit IRC | 18:01 | |
| openstackgerrit | Merged openstack/kolla master: Remove support for OracleLinux https://review.opendev.org/682865 | 18:11 |
| openstackgerrit | Merged openstack/kolla master: build: enable Octavia for Ubuntu/binary https://review.opendev.org/682671 | 18:11 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: Fix for haproxy precheck failing on CentOS running non-root https://review.opendev.org/682957 | 18:14 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Use VXLAN overlay network https://review.opendev.org/670690 | 18:15 |
| *** k_mouza has joined #openstack-kolla | 18:19 | |
| *** k_mouza has quit IRC | 18:23 | |
| *** openstackgerrit has quit IRC | 18:37 | |
| *** goldyfruit___ has joined #openstack-kolla | 18:38 | |
| *** goldyfruit_ has quit IRC | 18:41 | |
| *** openstackgerrit has joined #openstack-kolla | 19:02 | |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for self-signed SSL certificates https://review.opendev.org/664517 | 19:02 |
| *** jbadiapa has quit IRC | 19:44 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 19:44 |
| *** kplant has quit IRC | 19:46 | |
| *** jbadiapa has joined #openstack-kolla | 19:57 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 20:02 |
| *** pcaruana has quit IRC | 20:10 | |
| openstackgerrit | Joseph M proposed openstack/kolla-ansible master: [designate] Add coordination backend for designate workers https://review.opendev.org/682095 | 20:20 |
| openstackgerrit | Joseph M proposed openstack/kolla-ansible master: [designate] Add coordination backend for designate workers https://review.opendev.org/682095 | 20:22 |
| *** goldyfruit___ has quit IRC | 20:44 | |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for self-signed SSL certificates https://review.opendev.org/664517 | 20:50 |
| *** goldyfruit___ has joined #openstack-kolla | 20:56 | |
| *** signed8bit has quit IRC | 21:01 | |
| *** signed8bit has joined #openstack-kolla | 21:02 | |
| *** vmixor has quit IRC | 21:14 | |
| *** goldyfruit_ has joined #openstack-kolla | 21:22 | |
| *** goldyfruit___ has quit IRC | 21:25 | |
| *** JamesBenson has quit IRC | 21:25 | |
| *** JamesBenson has joined #openstack-kolla | 21:30 | |
| *** JamesBenson has quit IRC | 21:35 | |
| *** jonaspaulo has quit IRC | 21:56 | |
| *** goldyfruit_ has quit IRC | 22:15 | |
| *** goldyfruit_ has joined #openstack-kolla | 22:40 | |
| *** vmixor has joined #openstack-kolla | 23:05 | |
| *** luksky has quit IRC | 23:13 | |
| *** goldyfruit_ has quit IRC | 23:24 | |
| *** goldyfruit_ has joined #openstack-kolla | 23:29 | |
| *** BjoernT has quit IRC | 23:30 | |
| *** JamesBenson has joined #openstack-kolla | 23:46 | |
| *** ivve has quit IRC | 23:48 | |
| *** JamesBenson has quit IRC | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!