| *** schwicht has quit IRC | 00:07 | |
| *** zhanglong has joined #openstack-kolla | 00:08 | |
| *** schwicht has joined #openstack-kolla | 00:12 | |
| *** kplant has quit IRC | 00:21 | |
| *** schwicht has quit IRC | 00:48 | |
| *** brett-soric has joined #openstack-kolla | 01:24 | |
| *** zhanglong has quit IRC | 01:41 | |
| *** zhanglong has joined #openstack-kolla | 01:43 | |
| *** brett-soric has left #openstack-kolla | 02:11 | |
| *** zhanglong has quit IRC | 02:51 | |
| *** zhanglong has joined #openstack-kolla | 03:01 | |
| *** Sravan has joined #openstack-kolla | 03:17 | |
| *** Sravan has quit IRC | 03:30 | |
| *** zhanglong has quit IRC | 03:47 | |
| *** Sravan has joined #openstack-kolla | 04:08 | |
| *** Sravan has quit IRC | 04:16 | |
| *** Sravan has joined #openstack-kolla | 04:45 | |
| *** Sravan has quit IRC | 04:46 | |
| *** Sravan has joined #openstack-kolla | 05:02 | |
| *** shyamb has joined #openstack-kolla | 05:24 | |
| *** zhanglong has joined #openstack-kolla | 05:38 | |
| *** Sravan has quit IRC | 05:44 | |
| *** jbadiapa has joined #openstack-kolla | 05:45 | |
| *** rpittau|mtg is now known as rpittau|afk | 05:49 | |
| *** Sravan has joined #openstack-kolla | 05:59 | |
| *** Sravan has quit IRC | 06:04 | |
| *** cah_link has joined #openstack-kolla | 06:10 | |
| *** dpawlik has joined #openstack-kolla | 06:15 | |
| yoctozepto | morning | 06:18 |
|---|---|---|
| *** luksky has joined #openstack-kolla | 06:28 | |
| yoctozepto | re: https://bugs.launchpad.net/kolla-ansible/+bug/1834811 | 06:29 |
| openstack | Launchpad bug 1834811 in kolla-ansible "Unable to Pull Kuryr Docker image" [Undecided,Invalid] | 06:29 |
| yoctozepto | should not we default to 'source' since it is better supported? ;D | 06:29 |
| *** Dubla has joined #openstack-kolla | 06:43 | |
| *** shyamb has quit IRC | 06:44 | |
| *** Dubla has quit IRC | 06:50 | |
| *** skramaja has joined #openstack-kolla | 06:53 | |
| *** zhanglong has quit IRC | 06:58 | |
| *** ivve has joined #openstack-kolla | 07:05 | |
| *** Dubla has joined #openstack-kolla | 07:05 | |
| *** lemko has joined #openstack-kolla | 07:11 | |
| *** shyamb has joined #openstack-kolla | 07:13 | |
| stingrayza | morning folks :) | 07:24 |
| stingrayza | mgoddard: re: https://review.opendev.org/#/c/668153/ - should I do a diff removing version_check for stein branch? | 07:24 |
| stingrayza | *restoring version_compare I mean | 07:25 |
| yoctozepto | stingrayza, mgoddard: can't we up the ansible version for stein since it is not released yet anyway? | 07:28 |
| yoctozepto | but for queens/rocky definitely need to support older ansible | 07:28 |
| *** zhanglong has joined #openstack-kolla | 07:29 | |
| *** heikkine has joined #openstack-kolla | 07:30 | |
| stingrayza | I suppose it's whether you want to increase it this late in the release cycle | 07:32 |
| openstackgerrit | Raimund Hook (stingrayza) proposed openstack/kolla-ansible stable/queens: Updating Jinja filters to conform to Ansible 2.5+ https://review.opendev.org/668160 | 07:42 |
| *** pcaruana has joined #openstack-kolla | 07:43 | |
| yoctozepto | stingrayza: I know it's not nice in general but we are still at RC, ansible older than 2.6 is not supported upstream and all distros seem to provide easy access to newer ansible versions | 07:44 |
| *** rpittau|afk is now known as rpittau | 07:44 | |
| yoctozepto | so it's not nice but sensible | 07:44 |
| stingrayza | you'll find no arguments from me :) | 07:45 |
| yoctozepto | stingrayza: yeah, figured that | 07:45 |
| yoctozepto | the thing is we don't really test that we didn't break compatibility with 2.4 anyway | 07:46 |
| yoctozepto | so it's a bit of hypothesizing :P | 07:46 |
| stingrayza | that's a good point - there _should_ be a test for that if it's a hard requirement | 07:47 |
| stingrayza | I'm waiting for 2.9 to come out anyway, for this to come into play :D https://github.com/ansible/ansible/pull/56453 | 07:47 |
| yoctozepto | stingrayza: ah, the feeling of having your features released ;D | 07:50 |
| mnasiadka | morning | 07:50 |
| *** Sravan has joined #openstack-kolla | 07:50 | |
| *** Sravan has quit IRC | 07:55 | |
| mgoddard | morning all | 07:58 |
| *** zhanglong has quit IRC | 07:59 | |
| *** shyamb has quit IRC | 07:59 | |
| *** zhanglong has joined #openstack-kolla | 08:01 | |
| stingrayza | yoctozepto: yeah - this open source contribution thing is becoming addictive :) | 08:05 |
| *** dougsz has joined #openstack-kolla | 08:08 | |
| *** k_mouza has joined #openstack-kolla | 08:13 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Bump minimum Ansible version to 2.5 https://review.opendev.org/668387 | 08:38 |
| mgoddard | yoctozepto: stingrayza: I proposed ^ to bump minimum to 2.5 on master. I would be open to doing the same on stein given the age of 2.4 | 08:45 |
| *** k_mouza has quit IRC | 08:46 | |
| mgoddard | stingrayza: looks like your patches for rocky and queens will need to go though - those tests don't exist in 2.4 :( | 08:46 |
| *** k_mouza has joined #openstack-kolla | 08:47 | |
| stingrayza | mgoddard: I updated the queens one to revert back to 'success' - the rocky one doesn't have any success->successful changes | 08:48 |
| mgoddard | stingrayza: yeah, but even success doesn't seem to work as a test, only as a filter | 08:48 |
| mgoddard | stingrayza: if you can make it pass on queens, I would accept the same changes in rocky. In queens CI we test against ansible 2.4 | 08:49 |
| stingrayza | ah, right. :( ok. I suppose if you're still on rocky/queens you'll have to live with the deprecation messages :) | 08:50 |
| stingrayza | (neither bothers me, I'm dropping Stein in prod at the moment ;) ) | 08:53 |
| *** Sravan has joined #openstack-kolla | 08:53 | |
| mgoddard | stingrayza: is this a fresh deploy, not an upgrade? | 08:55 |
| stingrayza | yeah, I'm deploying fresh | 08:55 |
| mgoddard | ok | 08:56 |
| mgoddard | upgrades on centos not working yet | 08:56 |
| *** Sravan has quit IRC | 08:58 | |
| *** shyamb has joined #openstack-kolla | 08:58 | |
| mgoddard | should we be testing against our minimum supported version of ansible? | 09:00 |
| mgoddard | rather than the latest | 09:00 |
| stingrayza | probably. it would catch any issues where anybody uses newer ansible features (intentionally or not) | 09:06 |
| yoctozepto | <mgoddard> should we be testing against our minimum supported version of ansible? | 09:09 |
| yoctozepto | +1 | 09:09 |
| yoctozepto | in fact we should also be testing the latest, probably as non-voting | 09:09 |
| yoctozepto | to catch failures of future | 09:09 |
| yoctozepto | maybe even just as periodic job | 09:09 |
| mnasiadka | periodic jobs suck, in the end nobody looks at them :) | 09:09 |
| yoctozepto | mnasiadka: yup, they should be a more aggressive with notifications | 09:10 |
| yoctozepto | a bit* | 09:10 |
| *** Sravan has joined #openstack-kolla | 09:40 | |
| *** Sravan has quit IRC | 09:44 | |
| *** adrianreza has quit IRC | 09:48 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible stable/stein: Remove FWaaS v1 related options https://review.opendev.org/668403 | 09:51 |
| *** whoami-rajat has joined #openstack-kolla | 09:52 | |
| *** chaptertoaster has joined #openstack-kolla | 09:54 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Remove FWaaS v1 related options https://review.opendev.org/668406 | 09:59 |
| chaptertoaster | Is something up with kolla on dockerhub? | 10:13 |
| *** zhanglong has quit IRC | 10:15 | |
| openstackgerrit | Dincer Celik proposed openstack/kolla master: Fixes wrong local_setting.py file issue for Ubuntu binary deployment. https://review.opendev.org/668409 | 10:20 |
| yoctozepto | chaptertoaster: what do you mean? | 10:22 |
| chaptertoaster | https://hub.docker.com/u/kolla is a 404 for me | 10:25 |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: Fixes wrong local_setting.py file issue for Ubuntu binary deployment. https://review.opendev.org/668410 | 10:26 |
| yoctozepto | chaptertoaster: odd indeed | 10:27 |
| yoctozepto | https://hub.docker.com/r/kolla/ubuntu-source-cinder-volume | 10:27 |
| yoctozepto | though it managed to upload new images | 10:27 |
| yoctozepto | like 3 hours ago | 10:27 |
| chaptertoaster | I don't seem to be able to pull any of those images. | 10:27 |
| chaptertoaster | Error response from daemon: manifest for kolla/ubuntu-source-cinder-volume:latest not found | 10:28 |
| *** skramaja has quit IRC | 10:28 | |
| yoctozepto | https://hub.docker.com/u/centos | 10:28 |
| yoctozepto | too | 10:28 |
| yoctozepto | seems like dockerhub gone awry | 10:28 |
| chaptertoaster | darn | 10:28 |
| chaptertoaster | i was using that | 10:28 |
| yoctozepto | chaptertoaster: no u/ seems to work for me | 10:29 |
| yoctozepto | must be their internal issue | 10:29 |
| yoctozepto | though publishers/ work | 10:29 |
| yoctozepto | https://hub.docker.com/publishers/oracle | 10:29 |
| mgoddard | mnasiadka: I check periodic jobs sometimes. It's a good way to see jobs that are expected to pass | 10:30 |
| mgoddard | chaptertoaster: use 'master' or a release e.g. 'stein' for image tag | 10:32 |
| chaptertoaster | yep, looks like that works | 10:33 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Test minimum supported and latest versions of Ansible https://review.opendev.org/668413 | 10:38 |
| mgoddard | could someone review https://review.opendev.org/#/c/667665/ please? | 10:41 |
| yoctozepto | mgoddard: before slapping rechecks, let's first unbreak rocky CI: https://review.opendev.org/668335 | 10:44 |
| mgoddard | recheck yoctozepto | 10:44 |
| yoctozepto | ouch | 10:44 |
| yoctozepto | CI violence | 10:44 |
| mgoddard | do you know why it changed? | 10:45 |
| yoctozepto | mgoddard: nah, though in Stein and master we have it relative so it still works (the change that introduced that was irrelevant) | 10:46 |
| yoctozepto | might be Zuul doing something differently now | 10:46 |
| yoctozepto | ceph is not always tested | 10:46 |
| yoctozepto | yet from the kolla perspective | 10:46 |
| yoctozepto | rare job -> rare error | 10:46 |
| yoctozepto | I lost some time trying to figure out what started happening in master with fluentd | 10:47 |
| yoctozepto | would like someone else to check | 10:47 |
| *** shyamb has quit IRC | 10:52 | |
| yoctozepto | mgoddard: read Zuul's docs - found the reason you sought | 11:03 |
| yoctozepto | in rocky and queens ansible jobs do not list kolla-ansible as a required project | 11:03 |
| mgoddard | yoctozepto: ah, that makes sense | 11:03 |
| yoctozepto | though they work because Zuul clones the repo anyway | 11:04 |
| yoctozepto | it is just not linked in work dir | 11:04 |
| yoctozepto | so depending on how you use it | 11:04 |
| yoctozepto | you may fail or not | 11:04 |
| mgoddard | why does it clone the repo? because the job is defined in that repo? | 11:04 |
| yoctozepto | yeah, seems so, it needs the playbooks ;D | 11:04 |
| yoctozepto | seems it only does not link it in work/ if not affected by depends-on or required-projects | 11:05 |
| yoctozepto | new lesson learnt | 11:05 |
| yoctozepto | now somebody help me with fluentd failing all over the place in master kolla-ansible jobs when run from kolla ;D | 11:06 |
| mgoddard | yoctozepto: does it happen every time? | 11:08 |
| yoctozepto | mgoddard: yup | 11:09 |
| *** kplant has joined #openstack-kolla | 11:09 | |
| yoctozepto | for all flavors | 11:09 |
| *** shyamb has joined #openstack-kolla | 11:09 | |
| yoctozepto | but I just cannot wrap my head around it | 11:09 |
| yoctozepto | it only happens when images are rebuilt | 11:09 |
| yoctozepto | oh, and we are talking about upgrades | 11:10 |
| mgoddard | yoctozepto: bear in mind this is the first image | 11:10 |
| yoctozepto | mgoddard: good point | 11:11 |
| mgoddard | yoctozepto: https://review.opendev.org/#/c/666141/ | 11:11 |
| yoctozepto | gee | 11:11 |
| yoctozepto | http: server gave HTTP response to HTTPS client | 11:11 |
| mgoddard | yoctozepto: wrong one | 11:11 |
| mgoddard | https://review.opendev.org/#/c/666780/ | 11:11 |
| yoctozepto | missed that message in that chunk | 11:11 |
| mgoddard | turns out that "unnecessary logic" was necessary :) | 11:12 |
| yoctozepto | mgoddard: yeah, I felt the same after finding that message | 11:12 |
| mgoddard | need to be careful with zuul - the ground shifts with and without depends-on | 11:12 |
| yoctozepto | I just saw 500 and then why fluentd blah | 11:12 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Revert "CI - remove unnecessary logic when building images for upgrade" https://review.opendev.org/668431 | 11:12 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Revert "CI - remove unnecessary logic when building images for upgrade" https://review.opendev.org/668432 | 11:13 |
| yoctozepto | mgoddard: nah, let me check for the real reason why it failed, by all means it seemed redundant | 11:13 |
| yoctozepto | then we fix the real bug | 11:13 |
| yoctozepto | :D | 11:14 |
| mgoddard | it would be quite a coincidence | 11:14 |
| mgoddard | "when building images for upgrade" | 11:14 |
| yoctozepto | mgoddard: yes, it is this one but it should be there in place anyway | 11:14 |
| yoctozepto | as explained in the commit message | 11:14 |
| yoctozepto | ;D | 11:14 |
| mgoddard | can check config files in zuul logs | 11:15 |
| yoctozepto | yeah, cecking | 11:15 |
| yoctozepto | damn, we don't preserve the docker config | 11:19 |
| yoctozepto | must have been reset | 11:19 |
| yoctozepto | but why | 11:19 |
| yoctozepto | we do template: | 11:20 |
| yoctozepto | src: docker_systemd_service.j2 | 11:20 |
| yoctozepto | dest: /etc/systemd/system/docker.service.d/kolla.conf | 11:20 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: DNM: Grab some more configs https://review.opendev.org/668445 | 11:35 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Use template-overrides.j2 from kolla https://review.opendev.org/667971 | 11:36 |
| *** pcaruana has quit IRC | 11:36 | |
| *** Sravan has joined #openstack-kolla | 11:46 | |
| *** luksky has quit IRC | 11:46 | |
| *** shyamb has quit IRC | 11:48 | |
| yoctozepto | mgoddard: I -1 to let others know I am working on the root cause | 11:50 |
| *** Sravan has quit IRC | 11:50 | |
| yoctozepto | (and also to dislike reverting my work ofc ;P ) | 11:51 |
| yoctozepto | I might later change that DNM to a proposal for more logging | 11:51 |
| *** skramaja has joined #openstack-kolla | 12:01 | |
| *** shyamb has joined #openstack-kolla | 12:02 | |
| *** schwicht has joined #openstack-kolla | 12:17 | |
| mgoddard | yoctozepto: more logging is usually good | 12:24 |
| *** schwicht has quit IRC | 12:30 | |
| *** schwicht has joined #openstack-kolla | 12:31 | |
| *** luksky has joined #openstack-kolla | 12:37 | |
| *** schwicht has quit IRC | 12:50 | |
| *** iclon_ has quit IRC | 12:52 | |
| *** shyam89 has joined #openstack-kolla | 12:54 | |
| *** shyamb has quit IRC | 12:54 | |
| *** iclon has joined #openstack-kolla | 13:00 | |
| *** shyam89 has quit IRC | 13:04 | |
| *** ohwhyosa has joined #openstack-kolla | 13:16 | |
| *** schwicht has joined #openstack-kolla | 13:17 | |
| *** Sravan has joined #openstack-kolla | 13:20 | |
| *** skramaja has quit IRC | 13:23 | |
| *** Sravan has quit IRC | 13:25 | |
| ohwhyosa | Hi guys! | 13:35 |
| ohwhyosa | I'm doing another deployment with more options enabled in globals.yml | 13:35 |
| ohwhyosa | Should I do the bootstrap step again? | 13:35 |
| ohwhyosa | Because it complains, on the newly added services, that ImageNotFound: 404 Client Error: Not Found (\"pull access denied for kolla/centos-binary-monasca-api, repository does not exist or may require \\'docker login\\'\") | 13:36 |
| ohwhyosa | I did the prechecks and then deploy | 13:36 |
| ohwhyosa | Twice (it failed first with dpdk and then with monasca) | 13:41 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: During deploy, always sync DB https://review.opendev.org/650962 | 13:46 |
| mgoddard | Hi ohwhyosa, monasca is not available for binary deploys | 13:46 |
| mgoddard | ohwhyosa: https://docs.openstack.org/kolla-ansible/latest/reference/logging-and-monitoring/monasca-guide.html | 13:46 |
| ohwhyosa | Oh!! Great! Thanks mgoddard! Is there a general list with which services are available for binary or source? | 13:47 |
| ohwhyosa | Else I can try and make it to pay it forward a bit, though I'll probably be a pain hahahaha | 13:47 |
| mgoddard | ohwhyosa: not documented, but checkout kolla/image/build.py:SKIPPED_IMAGES | 13:48 |
| ohwhyosa | Perfect! I'll check that out! | 13:49 |
| ohwhyosa | Gotta go now, we have a meeting! | 13:49 |
| *** Sravan has joined #openstack-kolla | 13:52 | |
| mgoddard | yoctozepto: did you get anywhere with the CI issue? If not I'll push those two reverts through | 13:54 |
| mgoddard | can always try the change again once you know what needs to be done | 13:55 |
| *** Sravan has quit IRC | 13:56 | |
| *** schwicht has quit IRC | 14:00 | |
| *** dpawlik has quit IRC | 14:05 | |
| *** pcaruana has joined #openstack-kolla | 14:06 | |
| *** schwicht has joined #openstack-kolla | 14:09 | |
| *** pcaruana has quit IRC | 14:56 | |
| *** michaelbarkdoll has joined #openstack-kolla | 15:00 | |
| openstackgerrit | Michael proposed openstack/kolla stable/stein: Update neutron to 14.0.2. https://review.opendev.org/#/c/663122/ https://review.opendev.org/668486 | 15:01 |
| michaelbarkdoll | Hi, I just submitted an update for neutron to version 14.0.2, which was merged yesterday. This fixes support for multiple external networks in L3 agent and addresses an issue related to dhcp agent configured with mismatching domain and host entries. | 15:08 |
| *** factor has joined #openstack-kolla | 15:09 | |
| *** icarusfactor has quit IRC | 15:10 | |
| mnasiadka | mgoddard: sometimes is the keyword :) no blaming anybody, I had a rule that watched for publish job failures, but since we had a very long history of failed publish jobs on ocata - I disabled it :) | 15:10 |
| *** icarusfactor has joined #openstack-kolla | 15:11 | |
| *** factor has quit IRC | 15:11 | |
| *** BjoernT has joined #openstack-kolla | 15:12 | |
| *** factor has joined #openstack-kolla | 15:14 | |
| *** icarusfactor has quit IRC | 15:15 | |
| *** icarusfactor has joined #openstack-kolla | 15:21 | |
| openstackgerrit | Michael proposed openstack/kolla stable/stein: Update versions: ./tools/version-check.py https://review.opendev.org/668491 | 15:23 |
| *** factor has quit IRC | 15:24 | |
| *** nde has joined #openstack-kolla | 15:29 | |
| *** ivve has quit IRC | 15:30 | |
| michaelbarkdoll | no | 15:36 |
| *** goldyfruit has quit IRC | 15:39 | |
| *** cgrosjean has joined #openstack-kolla | 15:40 | |
| openstackgerrit | Michael proposed openstack/kolla stable/stein: Update Neutron to 14.0.2 Update ironic-inspector to 8.2.2 https://review.opendev.org/668496 | 15:41 |
| michaelbarkdoll | Ug, I give up on learning git for this morning :( | 15:42 |
| michaelbarkdoll | sorry, was trying to squash the changes into one commit for my previous push | 15:43 |
| openstackgerrit | Merged openstack/kolla-ansible master: Revert "CI - remove unnecessary logic when building images for upgrade" https://review.opendev.org/668431 | 15:48 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Revert "CI - remove unnecessary logic when building images for upgrade" https://review.opendev.org/668432 | 15:50 |
| *** pcaruana has joined #openstack-kolla | 15:53 | |
| openstackgerrit | Michael proposed openstack/kolla stable/stein: Update Neutron to 14.0.2 Update ironic-inspector to 8.2.2 https://review.opendev.org/668496 | 15:54 |
| *** luksky has quit IRC | 15:56 | |
| *** ohwhyosa has quit IRC | 15:59 | |
| *** whoami-rajat has quit IRC | 16:01 | |
| *** Dubla has quit IRC | 16:03 | |
| mnasiadka | michaelbarkdoll: everyone can have a bad day :) | 16:08 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: DNM: try waiting for mariadb to become operational https://review.opendev.org/668514 | 16:16 |
| *** Sravan has joined #openstack-kolla | 16:17 | |
| yoctozepto | mgoddard: no, sorry, been busy, guess I have to propose the breaking change again and depend on it ;D | 16:17 |
| yoctozepto | it bothers me that it caused the issue | 16:17 |
| mgoddard | yoctozepto: sure, although it doesn't fail if you depend on it does it? | 16:17 |
| yoctozepto | mgoddard: no idea, we tested upgrade then but it did not fail so... | 16:18 |
| yoctozepto | could be some strange Zuul thing | 16:19 |
| yoctozepto | but bothering no matter | 16:19 |
| michaelbarkdoll | mgoddard, would it be alright if I just leave https://review.opendev.org/#/c/668491/ open and abandon my original PR? | 16:20 |
| mgoddard | michaelbarkdoll: that's fine | 16:20 |
| yoctozepto | mgoddard: something rewrote the file completely http://logs.openstack.org/71/667971/10/check/kolla-ansible-centos-source-upgrade-ceph/524c6b0/primary/logs/kolla_configs/systemd-system/docker.service.d/kolla.conf | 16:20 |
| michaelbarkdoll | Thanks, sorry for the noise. | 16:21 |
| yoctozepto | ignoring the insecure part | 16:21 |
| yoctozepto | very strange | 16:21 |
| mgoddard | yoctozepto: bear in mind that during an upgrade the previous release is used initially | 16:21 |
| *** goldyfruit has joined #openstack-kolla | 16:21 | |
| *** Sravan has quit IRC | 16:21 | |
| *** cgrosjean has quit IRC | 16:22 | |
| *** shaan9 has joined #openstack-kolla | 16:25 | |
| yoctozepto | mgoddard: sure, right, it was stein's patch that caused the breakage! you are a genius, we did not see the failure because master broke the future master and stein broke master | 16:26 |
| yoctozepto | (presumably) | 16:27 |
| *** ohwhyosa has joined #openstack-kolla | 16:27 | |
| yoctozepto | now onto why | 16:27 |
| ohwhyosa | Yo people! Kayobe was Ubuntu 16/Centos 7 only, but Kolla-ansible has no problem on ubuntu 18 right? | 16:32 |
| ohwhyosa | As in, I deployed it on ubuntu 18 and it works marvels | 16:32 |
| yoctozepto | mgoddard: ok, clear, because templating occurs only with the previous release, which now explicitly had no information that this insecure one is used just later | 16:32 |
| yoctozepto | the quick and dirty way is to rebootstrap | 16:32 |
| mgoddard | ohwhyosa: correct, from rocky onwards | 16:33 |
| ohwhyosa | but I wanna know if it might bring some problem in the long road | 16:33 |
| ohwhyosa | good, good | 16:33 |
| *** jonaspaulo has joined #openstack-kolla | 16:33 | |
| mgoddard | yoctozepto: quick and dirty way is to move on with life - it works :) | 16:33 |
| mgoddard | yoctozepto: or maybe just don't backport to stein? | 16:33 |
| ohwhyosa | is there a patch in progress regarding the deprectaion warnings with result | failed? | 16:33 |
| yoctozepto | mgoddard: and wait for train to fail | 16:34 |
| yoctozepto | nah, I think let's keep it this way for now | 16:34 |
| mgoddard | yoctozepto: why would it fail? | 16:34 |
| yoctozepto | because it would still not retemplate the file | 16:34 |
| yoctozepto | unless we change that | 16:34 |
| mgoddard | but didn't we add the insecure support in stein(?) so upgrades on master should be ok? | 16:34 |
| mgoddard | ohwhyosa: yes, but only backporting to stein | 16:35 |
| yoctozepto | mgoddard: the thing you reverted has us safe now | 16:35 |
| yoctozepto | without it, it does not hold | 16:35 |
| yoctozepto | because it templates the docker override without knowledge of registry | 16:35 |
| yoctozepto | it's a failed assumption here | 16:36 |
| yoctozepto | good to remember | 16:36 |
| yoctozepto | if it bites us from another side | 16:36 |
| ohwhyosa | I've had a problem with stein, with every deployment I've tried (on kolla I'm using rocky), with the novnc, but I believe it's a websockify problem after talking with openstack-ansible and openstack-nova people, have you people had a similar problem? | 16:36 |
| *** rpittau is now known as rpittau|afk | 16:36 | |
| yoctozepto | I know, I will improve the comment thing so that it keeps us informed at least, and I can use the stein explicit approach to insecure | 16:37 |
| yoctozepto | just watch | 16:37 |
| ohwhyosa | mgoddard and do you want it also on the rocky one or it's not a priority? I could take a look on that | 16:37 |
| mgoddard | ohwhyosa: no we can't backport to rocky because we need to support ansible 2.4 | 16:37 |
| *** schwicht has quit IRC | 16:37 | |
| openstackgerrit | Alfredo Moralejo proposed openstack/kolla stable/pike: Move pike jobs to use validated RDO Trunk https://review.opendev.org/668522 | 16:38 |
| mgoddard | ohwhyosa: I'm not aware of that issue with noVNC, but try searhing launchpad | 16:38 |
| yoctozepto | nah, just the comment because the registry address has to be known upfront too | 16:39 |
| *** k_mouza_ has joined #openstack-kolla | 16:40 | |
| ohwhyosa | mgoddard, I did, no luck! (there was someone, but no answers), but i meant to ask if you tried stein with novnc | 16:41 |
| yoctozepto | mgoddard: "is success" does not work in 2.4 at all? | 16:42 |
| mgoddard | yoctozepto: nope | 16:42 |
| mgoddard | ohwhyosa: I haven't tried it personally | 16:43 |
| yoctozepto | mgoddard: pitiful to deprecate the only way in the next point release | 16:43 |
| yoctozepto | ;P | 16:43 |
| *** k_mouza has quit IRC | 16:43 | |
| mgoddard | it's a bit tricksy | 16:43 |
| yoctozepto | ohwhyosa: what the problem is exactly | 16:44 |
| yoctozepto | I think I got stein deploy running with console working, probably novnc unless the default is different? | 16:44 |
| *** k_mouza_ has quit IRC | 16:45 | |
| ohwhyosa | yoctozepto, the problem was that via cli I could get a url with a token, and horizon could get a url with a token, but when trying to connect to the console it said it was unavailable | 16:46 |
| ohwhyosa | digging around a bit | 16:46 |
| ohwhyosa | I reached the conclusion that it was because it was getting a empty token | 16:46 |
| ohwhyosa | on the nova-novncproxyy logs | 16:46 |
| ohwhyosa | And it was hashing it nonetheless, so it did get a hashed token (different from the one given by openstack cli and horizon) | 16:47 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Add note to CI config regarding registry during upgrade https://review.opendev.org/668524 | 16:47 |
| ohwhyosa | And it was throwing an invalid token | 16:47 |
| yoctozepto | ^ do not backport, just make it stay to prevent me from "fixing" | 16:47 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: DNM: try waiting for mariadb to become operational https://review.opendev.org/668514 | 16:48 |
| yoctozepto | ohwhyosa: k, I will check here on my deploy, could be that I checked console for rocky because I deployed rocky first | 16:48 |
| ohwhyosa | tried checking connection to the db and connectivity between containers and stuff, but it seemed to be all working and stuff in that regard | 16:48 |
| ohwhyosa | Thanks yoctozepto! | 16:48 |
| ohwhyosa | Also, websockify was in a continuos broken pipe error loop, though with spice it was the same and it worked (kinda) | 16:49 |
| yoctozepto | mgoddard: how lucky with mariadb? | 16:50 |
| mgoddard | yoctozepto: I added a sleep after bringing the master up and it hasn't failed (in that part) since | 16:50 |
| yoctozepto | mgoddard: hmm, so we just need to wait for master to be really up? | 16:51 |
| mgoddard | yoctozepto: the above patch is trying to wait for wsrep_evs_state to be OPERATIONAL, which is what OSA does when bootstrapping galera | 16:51 |
| yoctozepto | mgoddard: I see | 16:51 |
| yoctozepto | sounds reasonable | 16:51 |
| mgoddard | yeah, just trying to find the right thing to wait for | 16:51 |
| yoctozepto | mgoddard: oh, it could just say "IMOK" when ready, eh? | 16:51 |
| mgoddard | that would be nice | 16:52 |
| mgoddard | we normally just wait for the port to be open and say 'MariaDB', but seems that isn't enough | 16:52 |
| yoctozepto | mgoddard: yeah, I found out that much, seems being ready to accept connections is not the same as being normally operational in here | 16:53 |
| michaelbarkdoll | Nice to hear people are working on that mariadb, not say how many times I've seen my deploy fail due to it waiting for it to be online. | 16:53 |
| yoctozepto | michaelbarkdoll: hmm? deploy | 16:53 |
| yoctozepto | ? | 16:53 |
| michaelbarkdoll | No, maybe not deploy not sure | 16:54 |
| yoctozepto | we are about upgrade in here but deploy could also be affected | 16:54 |
| yoctozepto | though none of mine failed it | 16:54 |
| yoctozepto | and boy, I did a few reruns ;D | 16:54 |
| michaelbarkdoll | Well, last week I saw something about it waiting for the slave mariadb containers | 16:54 |
| yoctozepto | could very well depend on timing | 16:54 |
| mgoddard | yoctozepto: seen this: http://logs.openstack.org/63/667363/11/check/kolla-ansible-centos-source-upgrade-ceph-1/640d468/primary/logs/ansible/test-openstack | 16:55 |
| mgoddard | yoctozepto: seems like it could be related to recent nova compute discovery changes? | 16:55 |
| mgoddard | (that boot would have been on rocky release) | 16:55 |
| *** dougsz has quit IRC | 16:56 | |
| yoctozepto | mhm, looks very likely | 16:56 |
| yoctozepto | but we do discovery of nova after it is running | 16:56 |
| yoctozepto | unless this is another case for | 16:56 |
| yoctozepto | "almost running" | 16:56 |
| yoctozepto | eh ;/ | 16:56 |
| yoctozepto | though we cannot just revert that, it fixed a genuine race condition | 16:57 |
| ohwhyosa | mgoddard, that thing with mariadb is the thing with the upgrade process from rocky to stein you people mentioned the other day? | 16:57 |
| ohwhyosa | I believe yoctozepto mentioned it | 16:57 |
| yoctozepto | ohwhyosa: yup | 16:57 |
| yoctozepto | mgoddard working hard on fix | 16:58 |
| ohwhyosa | oh! Now that I remembered, regarding upgrades and config changes | 16:58 |
| ohwhyosa | I tried changing the config in /etc/kolla/horizon/local_settings | 16:58 |
| ohwhyosa | to change the HORIZON_MULTIDOMAIN setting (not the dropdown one, the other) | 16:59 |
| ohwhyosa | and then kolla-ansible -i all_in_one reconfigure | 16:59 |
| yoctozepto | ohwhyosa: if it was ubuntu binary, then we have a bug with local_settings being ignored afaik | 16:59 |
| ohwhyosa | Went amazing but did nothing | 16:59 |
| ohwhyosa | hahaha yup, exactly, that was it! | 16:59 |
| ohwhyosa | No wait | 16:59 |
| ohwhyosa | The base was ubuntu | 16:59 |
| ohwhyosa | but the container os was Centos | 16:59 |
| yoctozepto | well, no one complained about binary centos | 17:00 |
| yoctozepto | but could be broken too | 17:00 |
| yoctozepto | ;p | 17:00 |
| ohwhyosa | I destroyed that instance | 17:00 |
| ohwhyosa | but in about one hour I'm doing another test deployment | 17:00 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Use template-overrides.j2 from kolla https://review.opendev.org/667971 | 17:00 |
| ohwhyosa | Now on multi-node bare-metal instead of aio vm | 17:01 |
| ohwhyosa | Will report back and have a look at it | 17:01 |
| *** schwicht has joined #openstack-kolla | 17:03 | |
| *** k_mouza has joined #openstack-kolla | 17:04 | |
| *** k_mouza has quit IRC | 17:09 | |
| *** Sravan has joined #openstack-kolla | 17:09 | |
| *** Sravan has quit IRC | 17:11 | |
| *** Sravan has joined #openstack-kolla | 17:12 | |
| ohwhyosa | By the way, is there any plan to support at some point tungsten fabric as a neutron driver? | 17:17 |
| ohwhyosa | I saw you already have opendaylight | 17:17 |
| ohwhyosa | (tungsten fabric, "The artist previously known as OpenContrail" | 17:17 |
| ohwhyosa | ) | 17:17 |
| ohwhyosa | Also, can't remember now, but did I see a Princess Bride reference somewhere in kolla? | 17:18 |
| kplant | tungsten would be neat, i'm setting something up with my juniper rep :> | 17:18 |
| yoctozepto | ohwhyosa, kplant: hereby I nominate you to add tungsten support | 17:20 |
| kplant | no hablo ingles | 17:20 |
| yoctozepto | kplant: que pena! | 17:23 |
| kplant | foiled again | 17:23 |
| yoctozepto | ;P | 17:24 |
| openstackgerrit | Michael Bayer proposed openstack/kolla master: Add python-sqlalchemy-collectd client https://review.opendev.org/659678 | 17:25 |
| ohwhyosa | Yeah!! | 17:28 |
| ohwhyosa | Will be doing some tests with my teammates and try and implement it | 17:28 |
| *** shaan9 has left #openstack-kolla | 17:29 | |
| mgoddard | we are planning to use tungsten with kolla in quite a large deployment | 17:31 |
| yoctozepto | mnasiadka: I wonder if you are perhaps interested in unblocking CI for two releases :-) https://etherpad.openstack.org/p/KollaWhiteBoard | 17:32 |
| mgoddard | there is this thing, although I haven't tried it | 17:32 |
| kplant | mgoddard: that sounds like a lot of fun | 17:32 |
| mgoddard | https://github.com/Juniper/contrail-kolla-ansible | 17:32 |
| mgoddard | kplant: could be, quite a few unknowns though | 17:33 |
| mnasiadka | mgoddard: I don't think that's ,,current'' :) | 17:33 |
| mgoddard | mnasiadka: agreed. Is there something more current? | 17:33 |
| kplant | if you don't have someone above you making up deadlines, it should be fun** | 17:33 |
| mnasiadka | mgoddard: I did something with Juniper guys when I worked in IBM, but the code was left in IBM Github :) | 17:34 |
| mnasiadka | mgoddard: like last year :) | 17:34 |
| mgoddard | doh | 17:34 |
| ohwhyosa | Dwarf Fortress levels of fun | 17:34 |
| mnasiadka | kplant: there are always deadlines :) | 17:34 |
| kplant | for sure, but if there's a little more intelligence than rand() that's always helpful | 17:35 |
| *** luksky has joined #openstack-kolla | 17:39 | |
| *** ivve has joined #openstack-kolla | 17:42 | |
| mgoddard | yoctozepto: nova compute discovery issue is due to not waiting for all computes. It's not caused by your patch but is exacerbated by it | 17:47 |
| mnasiadka | kplant: I live in rand() organisation for a year, don't even try to tell me all the others do the same :) | 17:49 |
| mnasiadka | mgoddard: so now somebody needs to properly fix it :) | 17:49 |
| mgoddard | yoctozepto: https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/nova/tasks/discover_computes.yml#L24 | 17:49 |
| mgoddard | mnasiadka: volunteering? :p | 17:50 |
| mnasiadka | mgoddard: not today, today is not a good day for me :D | 17:50 |
| mgoddard | mnasiadka: sure | 17:50 |
| yoctozepto | mgoddard: sure, meant the same, though having a second check every 60 seconds helps with stabilizing situation | 17:51 |
| mgoddard | yeah | 17:51 |
| yoctozepto | mgoddard: hmm, but we know how many computes we should have, don't we/ | 17:52 |
| mgoddard | I think we should be able to do something nice with jinja & set theory here | 17:52 |
| mgoddard | build a list of expected computes, then ensure they're all present in the service list | 17:52 |
| yoctozepto | mgoddard: exactly | 17:52 |
| yoctozepto | mgoddard: pike and queens waiting for the merge of race patch | 17:54 |
| yoctozepto | https://review.opendev.org/#/c/668104/ | 17:54 |
| yoctozepto | (a side note) | 17:54 |
| mgoddard | yoctozepto: thanks, added -1 for now | 17:55 |
| yoctozepto | mgoddard: thanks for your strange appreciation of my work ;D | 18:03 |
| ohwhyosa | Hey, in case in case we do manage to get to work on the tungsten driver implementation, we have to first do kolla image for tungsten, then kolla-ansible right? And we would submit whatever we do as we do it or wait until we have at list the full role? | 18:03 |
| ohwhyosa | I'm reading the opendaylight playbooks now as a bit of a training | 18:03 |
| mgoddard | ohwhyosa: tungsten has their own images, let's use those and just treat it as external | 18:03 |
| mgoddard | ohwhyosa: we already deploy too many things :) | 18:04 |
| yoctozepto | mgoddard, ohwhyosa: yeah, because there is the problem of curation | 18:04 |
| yoctozepto | it is fun to add stuff | 18:04 |
| yoctozepto | and then noone cares | 18:04 |
| yoctozepto | ;D | 18:04 |
| ohwhyosa | Totally understandable! | 18:04 |
| ohwhyosa | What other service are external now, to read those playbooks? | 18:05 |
| kplant | speaking of adding more stuff.. do you guys plan to back-port masakari to stein? | 18:05 |
| kplant | os is that t and beyond | 18:05 |
| *** igordc has joined #openstack-kolla | 18:09 | |
| *** igordc has quit IRC | 18:14 | |
| *** igordc has joined #openstack-kolla | 18:15 | |
| mgoddard | kplant: we don't typically backport features | 18:15 |
| mgoddard | ohwhyosa: the main external thing now is ceph | 18:15 |
| mgoddard | various things can be internal or external, e.g. ceph, elasticsearch, mariadb, haproxy | 18:16 |
| ohwhyosa | mgoddard, great! we are going to use that too on the final deployment, so it's good to read up on that! | 18:17 |
| *** igordc has quit IRC | 18:17 | |
| mgoddard | ohwhyosa: we are planning to remove ceph support eventually, I would recommend using ceph-ansible | 18:17 |
| ohwhyosa | We are kinda testing waters still, but the starmen are planning on having a prod deployment by september | 18:18 |
| ohwhyosa | We tried to find an openstack expert to guide us a bit either coming or by conference, but turns out there are not many in our country it seems | 18:19 |
| ohwhyosa | we have some k8s pros though | 18:19 |
| ohwhyosa | We were planning on using our own, dear existing cephs, are you dropping kolla support for ceph or configuration for ceph as a backend in general? | 18:20 |
| mgoddard | ohwhyosa: only the kolla deployment of ceph. We'll always support an externally managed ceph | 18:20 |
| ohwhyosa | also, completely offtopic but mnasiadka kplant what is rand()? | 18:20 |
| kplant | a function in c/c++ (and many others) that returns a pseudorandom integer | 18:20 |
| ohwhyosa | mgoddard, oh, good, good, that was our use case! | 18:21 |
| ohwhyosa | Oh, yeah, that one I knew, I thought you meant like a company or something by context | 18:21 |
| ohwhyosa | kplant ^ | 18:21 |
| kplant | nope, i was just somewhat venting about non technical people creating deadlines out of thin air | 18:21 |
| kplant | you know, _those_ people | 18:22 |
| ohwhyosa | Hahahaha yeah, I just called them starmen in my prev paragraphs hahahaha | 18:22 |
| ohwhyosa | Because you know, they are up there in the skies | 18:22 |
| ohwhyosa | Looking from above | 18:22 |
| kplant | but re: ceph-ansible, it's very easy to get a solid working deployment with ceph-ansible | 18:23 |
| kplant | the defaults are pretty sane, there's maybe 10 things you need to change in all.yml | 18:23 |
| ohwhyosa | kplant, great! thanks! | 18:23 |
| ohwhyosa | I'll take a look! | 18:23 |
| kplant | i linked someone a yt video on it a while ago, was it you? | 18:24 |
| kplant | well if it wasn't: https://www.youtube.com/watch?v=dv_PEp9qAqg | 18:26 |
| kplant | it's very helpful, and if i'm around feel free to reach out if you have questions | 18:26 |
| ohwhyosa | I wasn't! thanks kplant! | 18:28 |
| ohwhyosa | Gotta leave now, will read you in the logs until i get my znc working again | 18:29 |
| ohwhyosa | Just remember, always shrink the filesystem before shrinking the lvm | 18:30 |
| ohwhyosa | Else bad things happen | 18:30 |
| ohwhyosa | expanding was so easy and beautiful | 18:30 |
| ohwhyosa | Night! | 18:31 |
| *** ohwhyosa has quit IRC | 18:31 | |
| yoctozepto | mgoddard: so we are planning to do only external ceph and test it in CI using ceph-ansible? | 18:39 |
| mnasiadka | that's more or less the plan | 18:50 |
| *** kplant has quit IRC | 19:17 | |
| *** schwicht has quit IRC | 19:18 | |
| michaelbarkdoll | So, if I bootstrap servers in a virtualenv (e.g., kolla-ansible -e virtualenv=/root/target2 -i /root/multinode bootstrap-servers) then attempt the prechecks in the same virtualenv (e.g., kolla-ansible -e virtualenv=/root/target2 -i /root/multinode prechecks). I'm getting an error due to the roles/prechecks/tasks/package_checks.yml using var {{ ans | 19:24 |
| michaelbarkdoll | ible_python.executable }} which is translating to /usr/bin/python instead of the virtualenv's /root/target2/bin/python. | 19:24 |
| michaelbarkdoll | Essentially, the error is related to the bootstrap install `pip install docker` but the "import docker" is later failing in the prechecks due to the system python version being used. | 19:26 |
| *** Dubla has joined #openstack-kolla | 19:28 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: WIP: Wait for all nova computes to be up before discovery https://review.opendev.org/668553 | 19:31 |
| *** Sravan has quit IRC | 19:37 | |
| michaelbarkdoll | Ah, I think i also need to set ansible_python_interpreter | 19:46 |
| *** hamzy has quit IRC | 20:02 | |
| *** dciabrin has quit IRC | 20:03 | |
| *** hamzy has joined #openstack-kolla | 20:07 | |
| *** Sravan has joined #openstack-kolla | 20:07 | |
| *** nde has quit IRC | 20:08 | |
| *** Sravan has quit IRC | 20:12 | |
| *** cah_link has quit IRC | 20:37 | |
| *** pcaruana has quit IRC | 20:41 | |
| *** Sravan has joined #openstack-kolla | 21:08 | |
| *** Dubla has quit IRC | 21:09 | |
| *** cgrosjean has joined #openstack-kolla | 21:11 | |
| *** dciabrin has joined #openstack-kolla | 21:26 | |
| *** kplant has joined #openstack-kolla | 21:34 | |
| *** BjoernT has quit IRC | 21:52 | |
| *** hamzy has quit IRC | 22:18 | |
| *** jonaspaulo has quit IRC | 22:32 | |
| *** luksky has quit IRC | 22:47 | |
| *** hamzy has joined #openstack-kolla | 22:57 | |
| *** igordc has joined #openstack-kolla | 23:03 | |
| *** igordc has quit IRC | 23:05 | |
| *** igordc has joined #openstack-kolla | 23:06 | |
| *** lemko has quit IRC | 23:17 | |
| *** Sravan has quit IRC | 23:38 | |
| *** cgrosjean has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!