| *** jonaspaulo has quit IRC | 00:05 | |
| *** JamesBenson has joined #openstack-kolla | 00:13 | |
| *** JamesBenson has quit IRC | 00:18 | |
| *** TrevorV has joined #openstack-kolla | 00:33 | |
| *** devfaz has quit IRC | 00:35 | |
| *** devfaz has joined #openstack-kolla | 00:35 | |
| *** zhanglong has joined #openstack-kolla | 00:36 | |
| *** JamesBenson has joined #openstack-kolla | 00:56 | |
| *** LinPeiWen has joined #openstack-kolla | 00:57 | |
| *** JamesBenson has quit IRC | 01:00 | |
| *** zhanglong has quit IRC | 01:31 | |
| *** JamesBenson has joined #openstack-kolla | 01:35 | |
| *** JamesBenson has quit IRC | 01:40 | |
| *** zhanglong has joined #openstack-kolla | 01:46 | |
| *** JamesBenson has joined #openstack-kolla | 02:46 | |
| *** LinPeiWen has quit IRC | 03:04 | |
| *** LinPeiWen has joined #openstack-kolla | 03:19 | |
| *** rphillips has quit IRC | 03:37 | |
| *** rphillips has joined #openstack-kolla | 03:38 | |
| *** TrevorV has quit IRC | 03:43 | |
| *** zhanglong has quit IRC | 03:58 | |
| *** skramaja has joined #openstack-kolla | 04:32 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-kolla | 04:33 | |
| *** vishalmanchanda has joined #openstack-kolla | 04:55 | |
| *** shyamb has joined #openstack-kolla | 05:19 | |
| *** bhagyashris|away is now known as bhagyashris | 05:27 | |
| oklhost | Morning. Hm, I had a look at the vote (channel topic). One sentence puzzles me there. Dropping Ubuntu? | 05:28 |
|---|---|---|
| *** zhanglong has joined #openstack-kolla | 05:34 | |
| *** kevinz has joined #openstack-kolla | 05:34 | |
| *** hjensas has joined #openstack-kolla | 05:39 | |
| *** shyamb has quit IRC | 05:55 | |
| *** cah_link has joined #openstack-kolla | 05:58 | |
| *** jopdorp has quit IRC | 06:01 | |
| *** jopdorp has joined #openstack-kolla | 06:02 | |
| *** shyamb has joined #openstack-kolla | 06:02 | |
| *** r3ap3r has quit IRC | 06:02 | |
| *** stackedsax has quit IRC | 06:02 | |
| *** stackedsax has joined #openstack-kolla | 06:03 | |
| *** nikparasyr has joined #openstack-kolla | 06:04 | |
| *** r3ap3r has joined #openstack-kolla | 06:04 | |
| *** shyam89 has joined #openstack-kolla | 06:20 | |
| *** shyamb has quit IRC | 06:22 | |
| osmanlicilegi | morning | 06:29 |
| osmanlicilegi | yoctozepto: if you need adventure in your life, patching glibc might be a good start :D | 06:31 |
| *** cah_link has quit IRC | 06:31 | |
| *** shyam89 has quit IRC | 06:39 | |
| *** JamesBenson has quit IRC | 06:39 | |
| *** also_stingrayza has joined #openstack-kolla | 07:07 | |
| *** stingrayza has quit IRC | 07:07 | |
| *** bengates has joined #openstack-kolla | 07:08 | |
| *** Tengu has quit IRC | 07:08 | |
| *** Tengu has joined #openstack-kolla | 07:10 | |
| *** shyamb has joined #openstack-kolla | 07:18 | |
| *** shyamb has quit IRC | 07:18 | |
| mnasiadka | osmanlicilegi: it's not adventure, it's like jumping down the cliff without any equipment :) | 07:18 |
| *** shyamb has joined #openstack-kolla | 07:19 | |
| yoctozepto | oklhost: unless we get our hands on focal; but we got them | 07:25 |
| *** sorin-mihai has joined #openstack-kolla | 07:25 | |
| sorin-mihai | hi. can anyone help me figure out why keystone container would keep restarting? i can see this at the end of the log file: ERROR: Primary token 2 is stale. | 07:26 |
| *** e0ne has joined #openstack-kolla | 07:27 | |
| yoctozepto | sorin-mihai: fernet sync has failed most likely; check the other keystone containers | 07:27 |
| yoctozepto | ssh/fernet | 07:27 |
| oklhost | yoctozepto: don't make me afraid. we're using ubuntu in production ;) if there is something I can help with in future, feel free to ping me for a talk. | 07:27 |
| *** shyam89 has joined #openstack-kolla | 07:27 | |
| sorin-mihai | yoctozepto, fernet/ssh don't restart. is there any solution for this? | 07:28 |
| yoctozepto | oklhost: I guess that was a bad joke; we have this joke type that we are dropping things that annoy us | 07:28 |
| yoctozepto | sorin-mihai: do they log anything though? | 07:28 |
| *** shyamb has quit IRC | 07:29 | |
| oklhost | yoctozepto: puh... my heart beat steps up to 300 while reading it :D | 07:30 |
| sorin-mihai | yoctozepto, not sure what to check, i can see the containers are up, docker logs only say how they started (obviously) and that's it. docker logs keystone shows that stale error | 07:32 |
| *** shyamb has joined #openstack-kolla | 07:34 | |
| *** ChanServ sets mode: +o yoctozepto | 07:35 | |
| *** brinzhang0 has joined #openstack-kolla | 07:35 | |
| *** yoctozepto changes topic to "IRC meetings on Wednesdays @ 15:00 UTC - agenda @ https://goo.gl/OXB0DL | Whiteboard: https://bit.ly/2MM7mWF | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b" | 07:35 | |
| *** ChanServ sets mode: -o yoctozepto | 07:36 | |
| yoctozepto | oklhost: got rid of both sources of confusion :-) | 07:36 |
| yoctozepto | sorin-mihai: hmm, maybe mnasiadka recognizes the symptoms? | 07:36 |
| *** shyam89 has quit IRC | 07:37 | |
| oklhost | yoctozepto: yeah, thanks a lot! Nerds and sarcasm, you know ;) | 07:37 |
| *** brinzhang_ has quit IRC | 07:38 | |
| sorin-mihai | i tried both deploy and reconfigure -t keystone (maybe stupid idea) but both failed at trying to create the admin user task (does make sense)... i'm not even sure what is now broken because of this | 07:41 |
| mnasiadka | sorin-mihai: remove keystone docker volumes and run deploy? | 07:45 |
| yoctozepto | oklhost: I'm both :-) | 07:46 |
| sorin-mihai | happy to do that if i don't loose other data.. i have lots of VMs in this deployment... | 07:46 |
| sorin-mihai | only the keystone ones? | 07:46 |
| yoctozepto | sorin-mihai: you won't; but remove only containers and volumes with keystone in their name | 07:46 |
| yoctozepto | sorin-mihai: all permanent data is in mariadb; never touch it ;d | 07:47 |
| sorin-mihai | barbican_keystone_listener also? | 07:47 |
| yoctozepto | sorin-mihai: nah, barbican is fine | 07:47 |
| yoctozepto | sorin-mihai: should have said with keystone in front of their name | 07:47 |
| sorin-mihai | so stop containers, remove containers, remove volumes, deploy -t keystone | 07:48 |
| yoctozepto | sorin-mihai: yes | 07:48 |
| sorin-mihai | just to make sure, can i have a list of keystone related volumes in a working deployment? :D | 07:50 |
| yoctozepto | sorin-mihai: just keystone_fernet_tokens | 07:56 |
| sorin-mihai | great. thank you | 07:56 |
| *** gfidente has joined #openstack-kolla | 08:00 | |
| sorin-mihai | the issues i'm having with this deployment could be caused by the fact that everything is's on a md raid1 made of 2x 5400 rpm HDDs? cinder is separated on a lvm on top of raid10 (also 5400 drives), but i noticed that if there is something writing bigger files continuously on the raid1 mount point (in this case / and say big file would be writen in /home) all sort of things start to timeout. does it make sense or is it just a bad | 08:00 |
| sorin-mihai | interpretation? | 08:00 |
| sorin-mihai | now after deploy -t keystone, although i can login, i can't see any of the instances... | 08:03 |
| *** openstackgerrit has joined #openstack-kolla | 08:05 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla stable/train: Bump up versions for train https://review.opendev.org/749454 | 08:05 |
| yoctozepto | sorin-mihai: they must be still there as you just got rid of stale keys for authentication | 08:07 |
| *** jonaspaulo has joined #openstack-kolla | 08:08 | |
| sorin-mihai | yoctozepto, getting "Error: Unable to retrieve instances." in browser | 08:08 |
| mnasiadka | maybe horizon is using some stale tokens | 08:09 |
| mnasiadka | try openstack CLI | 08:10 |
| sorin-mihai | openstack server list Unknown Error (HTTP 503) | 08:10 |
| mnasiadka | openstack catalog list ? | 08:11 |
| mnasiadka | (just to check if nova has a problem or keystone) :) | 08:11 |
| sorin-mihai | lots of output | 08:11 |
| mnasiadka | so then check nova-api logs what the heck is going on | 08:12 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla stable/ussuri: Bump up versions for ussuri https://review.opendev.org/749458 | 08:17 |
| sorin-mihai | WARNING nova.conductor.api [req-*** ] Timed out waiting for nova-conductor Is it running? Or did this service start before nova-conductor? Reattempting establishment of nova-conductor connection...: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID *** | 08:18 |
| sorin-mihai | but this is in nova-compute-log | 08:18 |
| mnasiadka | yoctozepto: https://zuul.opendev.org/t/openstack/build/2504b8271430421798fbbb356f486241/log/primary/logs/kolla/all-ERROR.txt#31 - interesting... | 08:21 |
| sorin-mihai | last line in nova-api-log says "server certificate does NOT include an ID which matches the server name" but is like from 2 hours ago. before that i have way too many lines like: /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_policy/policy.py:991: UserWarning: Policy os_compute_api:os-hypervisors:list-detail failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. | 08:25 |
| sorin-mihai | This behavior may change in the future where using the intended scope is required. | 08:25 |
| sorin-mihai | given that keystone is now changed, is there some way to just restart all the containers in such a way to make sure that all sync properly with keystone? | 08:26 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: Fix common role when using external mariadb https://review.opendev.org/749463 | 08:27 |
| sorin-mihai | i assume a reboot wouldn't make sense, i guess that is how keystone got busted... | 08:27 |
| *** LinPeiWen has quit IRC | 08:28 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Fix external mariadb documentation for database_user https://review.opendev.org/749464 | 08:31 |
| mnasiadka | sorin-mihai: rather think about restarting nova-api and conductor on the controllers | 08:33 |
| *** cah_link has joined #openstack-kolla | 08:42 | |
| *** shyamb has quit IRC | 08:48 | |
| *** shyamb has joined #openstack-kolla | 08:51 | |
| *** shyamb has quit IRC | 08:55 | |
| mnasiadka | mgoddard, yoctozepto: any idea if we need to do anything to sort out this setuptools._distutils mess? | 09:01 |
| mgoddard | mnasiadka: I haven't been following it, don't really have time to do so | 09:01 |
| mgoddard | mnasiadka, yoctozepto: I will have to miss at least the first part of the meeting today. Can one of you stand in? | 09:04 |
| yoctozepto | mgoddard: ok | 09:09 |
| mgoddard | thanks | 09:10 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: Work around setuptools>50 https://review.opendev.org/749469 | 09:14 |
| *** cah_link has quit IRC | 09:15 | |
| *** LinPeiWen has joined #openstack-kolla | 09:17 | |
| yoctozepto | sorin-mihai: the services should settle themselves after a bit of time | 09:19 |
| yoctozepto | ,m | 09:19 |
| yoctozepto | mnasiadka: vitrage? ;d | 09:19 |
| mnasiadka | yoctozepto: ubuntu-binary seems to ignore requirements.txt in heat |:D | 09:20 |
| yoctozepto | mnasiadka: binary is from deb so obviously it cares not :-) | 09:20 |
| yoctozepto | weird they got this dep wrong | 09:20 |
| mnasiadka | yoctozepto: well, not my problem :) | 09:21 |
| yoctozepto | mnasiadka: well, no; it seems heat just tries finding optional deps: | 09:21 |
| yoctozepto | 2020-08-31 10:04:02.890 7 ERROR heat-engine AttributeError: 'EntryPoint' object has no attribute 'module' | 09:21 |
| yoctozepto | heat's exception handling is busted | 09:21 |
| yoctozepto | or just stevedore | 09:21 |
| yoctozepto | mnasiadka: just write to the ml and ask whether this is a known issue | 09:22 |
| sorin-mihai | yoctozepto, they should indeed, but for some reason they didn't, i attempted a reconfigure, still running and things starting to come back | 09:23 |
| yoctozepto | mnasiadka: I'm not getting that setuptools issue now; I commented on it; just two days ago we thought it has different problems than what kolla could possibly hit | 09:28 |
| mnasiadka | yoctozepto: so, in get-pip.py - we get latest setuptools, which seems to be triggering that issue, and the env var seems to help - I didn't have time to dig deeply into that madness | 09:29 |
| mnasiadka | yoctozepto: there was a requirements change to pin setuptools to before 50, but get-pip.py doesn't really care about constraints :D | 09:29 |
| mnasiadka | either we set the env var, or pin - setting the var was easier to do | 09:30 |
| *** zzzeek has quit IRC | 09:32 | |
| *** zzzeek has joined #openstack-kolla | 09:33 | |
| yoctozepto | mnasiadka: does it affect all branches now? | 09:39 |
| yoctozepto | my concern is, based on my knowledge, we should not be affected (standard installs, venv) | 09:40 |
| *** shyamb has joined #openstack-kolla | 09:40 | |
| *** zzzeek has quit IRC | 09:48 | |
| *** zzzeek has joined #openstack-kolla | 09:49 | |
| mnasiadka | yoctozepto: but we are ;) I noticed it on Ussuri at least | 09:56 |
| *** LinPeiWen has quit IRC | 10:00 | |
| mnasiadka | yeah, it fixes setuptools problems, but bifrost has some problem now - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a47/749469/1/check/kolla-build-centos8-source/a47afd2/kolla/build/000_FAILED_bifrost-base.log | 10:02 |
| *** zhanglong has quit IRC | 10:04 | |
| sorin-mihai | yoctozepto, seems like even if nova services came back up after reconfigure, only one instance was able to boot, the others don't boot and are still in shuttoff state | 10:05 |
| *** zzzeek has quit IRC | 10:08 | |
| *** zzzeek has joined #openstack-kolla | 10:10 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: Work around setuptools>50 and fix bifrost network config https://review.opendev.org/749469 | 10:20 |
| *** k_mouza has joined #openstack-kolla | 10:32 | |
| *** LinPeiWen has joined #openstack-kolla | 10:42 | |
| mnasiadka | yoctozepto: seems todays publish centos8 job only failed on master, and it wasn't due to setuptools, so I'll only fix bifrost ;) | 10:44 |
| mnasiadka | now we get older setuptools installed than 50, even in get-pip.py | 10:46 |
| *** brinzhang_ has joined #openstack-kolla | 10:48 | |
| *** brinzhang0 has quit IRC | 10:51 | |
| *** brinzhang has joined #openstack-kolla | 10:51 | |
| mnasiadka | yoctozepto: but now tell me why ubuntu installs latest setuptools in kolla-ansible-ubuntu-source job? :)https://review.opendev.org/#/c/749454/ | 10:51 |
| patchbot | patch 749454 - kolla (stable/train) - Bump up versions for train - 1 patch set | 10:51 |
| mnasiadka | well, kolla-build-ubuntu-source as well | 10:53 |
| sorin-mihai | is there any way to reset the state of the VMs that are not booting neither through horizon nor through openstack-cli? | 10:53 |
| *** brinzhang_ has quit IRC | 10:54 | |
| *** zzzeek has quit IRC | 10:57 | |
| bodik | i guess nova has reset-state command | 11:00 |
| *** zzzeek has joined #openstack-kolla | 11:00 | |
| *** hjensas has quit IRC | 11:06 | |
| *** hjensas has joined #openstack-kolla | 11:07 | |
| *** k_mouza has quit IRC | 11:09 | |
| sorin-mihai | so, for an instance that is looking like this http://paste.openstack.org/raw/6RmkMfpWOOWPC27IOalb/ what would the safe way to make it boot again? | 11:17 |
| *** jbadiapa has joined #openstack-kolla | 11:20 | |
| mnasiadka | yoctozepto: well, the ENV fixes ubuntu builds... | 11:20 |
| *** shyamb has quit IRC | 11:23 | |
| *** k_mouza has joined #openstack-kolla | 11:30 | |
| *** k_mouza has quit IRC | 11:31 | |
| *** k_mouza has joined #openstack-kolla | 11:32 | |
| *** k_mouza has quit IRC | 11:34 | |
| *** cah_link has joined #openstack-kolla | 11:40 | |
| *** cah_link has quit IRC | 11:44 | |
| *** cah_link has joined #openstack-kolla | 11:54 | |
| *** cah_link has quit IRC | 11:54 | |
| *** JamesBenson has joined #openstack-kolla | 12:01 | |
| *** dave-mccowan has joined #openstack-kolla | 12:18 | |
| *** Reepicheep has quit IRC | 12:49 | |
| *** zhanglong has joined #openstack-kolla | 12:53 | |
| *** Reepicheep has joined #openstack-kolla | 13:02 | |
| *** zhanglong has quit IRC | 13:15 | |
| *** zhanglong has joined #openstack-kolla | 13:16 | |
| *** zhanglong has quit IRC | 13:45 | |
| *** TrevorV has joined #openstack-kolla | 13:46 | |
| *** sluna has joined #openstack-kolla | 14:09 | |
| *** wuchunyang has joined #openstack-kolla | 14:12 | |
| *** rphillips has quit IRC | 14:18 | |
| yoctozepto | mnasiadka: I'm not 100% against it but... it's this time again that my ass-bite-detector's red lamp has fired off :-) | 14:23 |
| yoctozepto | as in we are getting some issue hidden that will bite us | 14:23 |
| *** rphillips has joined #openstack-kolla | 14:26 | |
| hrw | ok, so what is proper setuptools fix? I want to get ussuri/debian/source/aarch64 working ;d | 14:27 |
| yoctozepto | hrw: me too, investigating | 14:27 |
| *** dking has quit IRC | 14:29 | |
| hrw | thx | 14:32 |
| mnasiadka | yoctozepto: well, weird thing is that only kolla-toolbox fails on Ussuri, but openstack-base fails on Train | 14:32 |
| mnasiadka | For now probably would be easier to pin, but how :) | 14:32 |
| yoctozepto | hrw: yw | 14:33 |
| yoctozepto | mnasiadka: is the bifrost thing master-only? | 14:33 |
| yoctozepto | mnasiadka: checked; it should be | 14:34 |
| yoctozepto | mnasiadka: could you propose only the bifrost fix? | 14:34 |
| mnasiadka | yoctozepto: yes, I will drop setuptools crap from change and we can merge it | 14:34 |
| yoctozepto | mnasiadka: thx | 14:34 |
| yoctozepto | master looks worky otherwise | 14:34 |
| *** cah_link has joined #openstack-kolla | 14:35 | |
| osmanlicilegi | folks, I will skip today's meeting as this is my birthday and I will have an unlimited pizza and beer for my pandemic birthday party :))) | 14:40 |
| yoctozepto | osmanlicilegi: aww, that's nice; happy birthday and have a great time, you are more than excused! (why no strawberries though?!) | 14:43 |
| *** cah_link has quit IRC | 14:46 | |
| yoctozepto | mnasiadka, hrw: fwiw, there are no debuntu-relevant changes between current ussuri and master | 14:51 |
| hrw | osmanlicilegi: all the best! | 14:53 |
| *** skramaja has quit IRC | 14:54 | |
| mnasiadka | yoctozepto: I noticed that myself, Sherlock | 14:55 |
| yoctozepto | mnasiadka: I did not know, no need to attack me ;p | 14:56 |
| mnasiadka | yoctozepto: not attacking, just laughing :) | 14:59 |
| mnasiadka | yoctozepto: running the meeting? | 14:59 |
| yoctozepto | mnasiadka: yeah, forgot to notify | 14:59 |
| yoctozepto | mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak - meeting already | 15:00 |
| yoctozepto | #startmeetin kolla | 15:00 |
| yoctozepto | #topic roll-call | 15:00 |
| yoctozepto | #startmeeting kolla | 15:00 |
| yoctozepto | #topic roll-call | 15:00 |
| openstack | Meeting started Wed Sep 2 15:00:21 2020 UTC and is due to finish in 60 minutes. The chair is yoctozepto. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| *** openstack changes topic to " (Meeting topic: kolla)" | 15:00 | |
| openstack | The meeting name has been set to 'kolla' | 15:00 |
| *** openstack changes topic to "roll-call (Meeting topic: kolla)" | 15:00 | |
| yoctozepto | o/ | 15:00 |
| yoctozepto | I wonder if I got delayed network split or everyone is suddenly having their parties thrown :-) | 15:04 |
| hrw | o\ | 15:04 |
| bbezak | o/ | 15:04 |
| yoctozepto | #topic agenda | 15:07 |
| *** openstack changes topic to "agenda (Meeting topic: kolla)" | 15:07 | |
| yoctozepto | * Roll-call | 15:07 |
| yoctozepto | * Announcements | 15:07 |
| yoctozepto | ** Kolla feature freeze fast approaching: Sep 28 - Oct 02 | 15:07 |
| yoctozepto | ** Victoria vPTG schedule https://ethercalc.openstack.org/7xp2pcbh1ncb | 15:07 |
| yoctozepto | ** New Docker Hub rules: https://www.docker.com/blog/scaling-dockers-business-to-serve-millions-more-developers-storage/ and https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/ | 15:07 |
| yoctozepto | ** Open infra summit forum submissions open https://cfp.openstack.org | 15:07 |
| yoctozepto | * Review action items from the last meeting | 15:07 |
| mnasiadka | o/ | 15:07 |
| yoctozepto | * CI status | 15:07 |
| yoctozepto | * Victoria release planning (kayobe) | 15:07 |
| yoctozepto | * Victoria release planning (kolla ansible) | 15:07 |
| yoctozepto | * Victoria release planning (kolla) | 15:07 |
| yoctozepto | * Kolla Kall is tomorrow (2020-09-03) | 15:07 |
| yoctozepto | #topic Announcements | 15:07 |
| *** openstack changes topic to "Announcements (Meeting topic: kolla)" | 15:07 | |
| yoctozepto | #info Kolla feature freeze fast approaching: Sep 28 - Oct 02 | 15:07 |
| yoctozepto | not much time for your fav new features; get them merged before it's too late :-) | 15:08 |
| yoctozepto | I guess we will again allow exceptions | 15:08 |
| yoctozepto | to be discussed | 15:08 |
| yoctozepto | #info Victoria vPTG schedule https://ethercalc.openstack.org/7xp2pcbh1ncb | 15:08 |
| yoctozepto | related to the previous bullet point; the Victoria is getting released so we will be having a vPTG for Wallaby | 15:09 |
| yoctozepto | PTL (mgoddard) needed to schedule time | 15:09 |
| *** LinPeiWen has quit IRC | 15:09 | |
| yoctozepto | we can discuss that later when mgoddard is around again | 15:10 |
| yoctozepto | #info New Docker Hub rules: https://www.docker.com/blog/scaling-dockers-business-to-serve-millions-more-developers-storage/ and https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/ | 15:10 |
| yoctozepto | please read these notices as they affect Kolla to some degree | 15:10 |
| yoctozepto | tl;dr free stuff has to be limited to not overwhelm docker servers | 15:10 |
| hrw | does infra has 'no limit' account or not? | 15:11 |
| yoctozepto | quotes: "starting on November 1, images stored in free Docker Hub repositories that have not had their manifest pushed or pulled in the last 6 months will be removed. This policy does not apply to images stored by paid Docker Hub subscription accounts, Docker Verified Publishers , or Docker Official Images." | 15:11 |
| yoctozepto | quotes: "that will take effect November 1, 2020: | 15:12 |
| yoctozepto | Free plan – anonymous users: 100 pulls per 6 hours | 15:12 |
| yoctozepto | Free plan – authenticated users: 200 pulls per 6 hours | 15:12 |
| yoctozepto | Pro plan – unlimited | 15:12 |
| yoctozepto | Team plan – unlimited" | 15:12 |
| yoctozepto | tripleo is already patching their CI to avoid docker.io | 15:12 |
| mnasiadka | 100 pulls per 6 hours? awesome | 15:12 |
| yoctozepto | hrw: I don't know yet, good point | 15:12 |
| yoctozepto | mnasiadka: indeed | 15:12 |
| mnasiadka | so that means we can nearly stop publishing | 15:12 |
| mnasiadka | because our CI is doing more :) | 15:12 |
| yoctozepto | if we are running on a free account, which is likely, our jobs will be crippled | 15:12 |
| yoctozepto | mnasiadka: stop testing ansible parts; as limits are about pulls | 15:13 |
| mnasiadka | yoctozepto: true, but then it's useless we waste cpu cycles to publish | 15:13 |
| mnasiadka | and network bandwidth | 15:13 |
| yoctozepto | mnasiadka: well, we spend more on building but network bandwidth is indeed wasted quite a bit | 15:14 |
| yoctozepto | I guess we need official opendev registry instead | 15:14 |
| yoctozepto | who wants to investigate this subject? | 15:14 |
| yoctozepto | the first step would be to discuss that with the infra folks | 15:14 |
| yoctozepto | anyone wants to pick it ip? | 15:15 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/train: Revert "openstack-base: pin setuptools < 46 to get horizon working" https://review.opendev.org/749534 | 15:18 |
| yoctozepto | #action yoctozepto to discuss how opendev could help with new docker.io limits | 15:19 |
| yoctozepto | #info Open infra summit forum submissions open https://cfp.openstack.org | 15:19 |
| yoctozepto | feel welcome to submit something* | 15:19 |
| yoctozepto | * make sure it's about kolla! | 15:19 |
| yoctozepto | :-) | 15:20 |
| yoctozepto | #topic Review action items from the last meeting | 15:21 |
| *** openstack changes topic to "Review action items from the last meeting (Meeting topic: kolla)" | 15:21 | |
| yoctozepto | mgoddard to message openstack-discuss about focal & victoria upgrade | 15:21 |
| yoctozepto | he did | 15:21 |
| yoctozepto | #topic CI status | 15:21 |
| *** openstack changes topic to "CI status (Meeting topic: kolla)" | 15:21 | |
| yoctozepto | Kolla is RED in stable branches still | 15:22 |
| yoctozepto | new issues attacked Ussuri and Train, related to setuptools on debuntu | 15:22 |
| mnasiadka | well, master is RED as well - due to bifrost, but it's being fixed | 15:22 |
| yoctozepto | mnasiadka: yeah, I treated that as fixed already :-) | 15:22 |
| yoctozepto | mnasiadka: please repropose soon | 15:23 |
| yoctozepto | ansible seems happy | 15:23 |
| yoctozepto | not sure about kayobe | 15:23 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: Fix bifrost network config https://review.opendev.org/749469 | 15:23 |
| mnasiadka | yoctozepto: and come on, -2? | 15:23 |
| mnasiadka | I feel like smashed in the head using a brick :) | 15:23 |
| *** wuchunyang has quit IRC | 15:24 | |
| yoctozepto | hrw: https://review.opendev.org/749469 | 15:24 |
| patchbot | patch 749469 - kolla - Fix bifrost network config - 3 patch sets | 15:24 |
| yoctozepto | mnasiadka: amended | 15:24 |
| yoctozepto | kayobe looks green too | 15:27 |
| hrw | +2 | 15:27 |
| yoctozepto | so only kolla issues atm | 15:27 |
| yoctozepto | #topic Victoria release planning (kayobe) | 15:27 |
| *** openstack changes topic to "Victoria release planning (kayobe) (Meeting topic: kolla)" | 15:27 | |
| yoctozepto | mnasiadka, bbezak: I can't find anyone else to discuss kayobe today :-) | 15:28 |
| mnasiadka | so then let's skip it for today :) | 15:28 |
| bbezak | :) | 15:29 |
| yoctozepto | :-) | 15:29 |
| yoctozepto | #topic Victoria release planning (kolla ansible) | 15:30 |
| *** openstack changes topic to "Victoria release planning (kolla ansible) (Meeting topic: kolla)" | 15:30 | |
| yoctozepto | I'm back to work on masakari/hacluster integration in kolla-ansible | 15:30 |
| yoctozepto | mgoddard has marked some changes with RP+1 so please have a look at them | 15:31 |
| *** jovial[m] has joined #openstack-kolla | 15:32 | |
| yoctozepto | #topic Victoria release planning (kolla) | 15:36 |
| *** openstack changes topic to "Victoria release planning (kolla) (Meeting topic: kolla)" | 15:36 | |
| yoctozepto | essentially same here | 15:36 |
| yoctozepto | #topic Kolla Kall is tomorrow (2020-09-03) | 15:39 |
| *** openstack changes topic to "Kolla Kall is tomorrow (2020-09-03) (Meeting topic: kolla)" | 15:39 | |
| JamesBenson | :yay: | 15:39 |
| yoctozepto | please feel welcome to join, we will likely continue the group bug triage | 15:39 |
| *** bengates has quit IRC | 15:41 | |
| *** bengates has joined #openstack-kolla | 15:42 | |
| *** gfidente has quit IRC | 15:43 | |
| yoctozepto | #topic Open discussion | 15:45 |
| *** openstack changes topic to "Open discussion (Meeting topic: kolla)" | 15:45 | |
| *** bengates has quit IRC | 15:46 | |
| mnasiadka | seems that ussuri kolla-toolbox setuptools is failing, because of virtualenv --system-site-packages bug https://github.com/pypa/pip/issues/6264, I'll just pin setuptools to lower version - we are not alone in this... | 15:49 |
| yoctozepto | mnasiadka: hmm... but what about master? | 15:51 |
| mnasiadka | yoctozepto: master is focal I think, so maybe it's better there in some sort? | 15:52 |
| yoctozepto | mnasiadka: but it also affects debian, but not on master | 15:53 |
| mnasiadka | it's all weird :) | 15:53 |
| yoctozepto | something else must have changed too | 15:53 |
| yoctozepto | yeah, it's lol | 15:53 |
| *** sorin-mihai_ has joined #openstack-kolla | 15:58 | |
| yoctozepto | anyhow, let's wrap it up now | 15:59 |
| yoctozepto | thanks everyone | 15:59 |
| yoctozepto | #endmeeting | 15:59 |
| *** openstack changes topic to "IRC meetings on Wednesdays @ 15:00 UTC - agenda @ https://goo.gl/OXB0DL | Whiteboard: https://bit.ly/2MM7mWF | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b" | 15:59 | |
| openstack | Meeting ended Wed Sep 2 15:59:24 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:59 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-09-02-15.00.html | 15:59 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-09-02-15.00.txt | 15:59 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-09-02-15.00.log.html | 15:59 |
| *** sorin-mihai has quit IRC | 16:00 | |
| mgoddard | thanks for chairing yoctozepto. Other commitment overran, so didn't get a chance to join | 16:04 |
| mgoddard | catching up no | 16:04 |
| mgoddard | w | 16:04 |
| mnasiadka | yoctozepto: difference is virtualenv version, 20.0.17 in ussuri vs 20.0.31 in master, using 20.0.31 in ussuri seems to fix | 16:06 |
| mnasiadka | I need to sweep the changelog and talk to requirements guys | 16:06 |
| *** sorin-mihai has joined #openstack-kolla | 16:06 | |
| *** sorin-mihai_ has quit IRC | 16:08 | |
| yoctozepto | mnasiadka: we can skip constraints for virtualenv then | 16:08 |
| mnasiadka | yeah, might be a simple solution | 16:09 |
| yoctozepto | mnasiadka: as I commented there, we already skip it one side | 16:09 |
| yoctozepto | that's why only kolla-toolbox is failing in ussuri | 16:10 |
| yoctozepto | now why is the train reversed xD | 16:10 |
| yoctozepto | it could be that | 16:12 |
| yoctozepto | in Train the virtualenv does not have this broken behaviour | 16:13 |
| yoctozepto | but we had the setuptools pinneds | 16:13 |
| yoctozepto | and this triggered some discrepancy issue | 16:13 |
| openstackgerrit | Mariusz Karpiarz proposed openstack/kolla-ansible master: Don't set hostname in monasca-agent-collector's config https://review.opendev.org/749547 | 16:14 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla stable/ussuri: Install newer virtualenv with relaxed importlib https://review.opendev.org/749548 | 16:15 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla stable/ussuri: Install newer virtualenv with relaxed importlib https://review.opendev.org/749548 | 16:19 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla stable/ussuri: Install newer virtualenv with relaxed importlib https://review.opendev.org/749548 | 16:35 |
| mnasiadka | yoctozepto: I feel we'll have more fun with setuptools 50 :) | 16:37 |
| yoctozepto | mnasiadka: revert works on train | 16:43 |
| yoctozepto | just need bifrost fixes | 16:43 |
| yoctozepto | I guess we are in for squash again | 16:43 |
| mnasiadka | yoctozepto: bifrost fix is in the air, but not really priority for Dmitry - https://review.opendev.org/#/c/749014/ | 16:44 |
| patchbot | patch 749014 - bifrost (stable/stein) - [Train and older] Fix several CI problems - 1 patch set | 16:44 |
| mnasiadka | yoctozepto: argh, but for train - then just squash two fixes in one and we're done | 16:46 |
| mnasiadka | then we'll only have stein as broken | 16:46 |
| *** sorin-mihai_ has joined #openstack-kolla | 16:46 | |
| yoctozepto | mnasiadka: ok, I'm taking care of Train | 16:48 |
| *** sorin-mihai has quit IRC | 16:48 | |
| *** dking has joined #openstack-kolla | 16:48 | |
| mnasiadka | yoctozepto: good | 16:49 |
| yoctozepto | mnasiadka: so 7.2.2 should be fine now? | 16:49 |
| yoctozepto | (for bifrost) | 16:49 |
| mnasiadka | yoctozepto: yes | 16:49 |
| yoctozepto | mnasiadka: ack, thanks | 16:50 |
| mnasiadka | yoctozepto: https://review.opendev.org/#/c/749469/3 +w? :) | 16:51 |
| patchbot | patch 749469 - kolla - Fix bifrost network config - 3 patch sets | 16:51 |
| yoctozepto | mnasiadka: with pleasure! | 16:51 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/train: Revert "openstack-base: pin setuptools < 46 to get horizon working" https://review.opendev.org/749534 | 16:53 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/train: Bump up versions for train https://review.opendev.org/749454 | 16:54 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/train: Upgrade Fluentd Monasca plugin https://review.opendev.org/743922 | 16:54 |
| yoctozepto | mnasiadka: qc https://review.opendev.org/749548 | 17:01 |
| patchbot | patch 749548 - kolla (stable/ussuri) - Install newer virtualenv with relaxed importlib - 3 patch sets | 17:01 |
| mnasiadka | yoctozepto: ah right, will do | 17:05 |
| *** mgoddard has quit IRC | 17:12 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Coordinate haproxy and keepalived restarts https://review.opendev.org/723342 | 17:34 |
| yoctozepto | mnasiadka: and please take a look at https://review.opendev.org/723342 it's a keepalived world game changer 8-) | 17:34 |
| patchbot | patch 723342 - kolla-ansible - Coordinate haproxy and keepalived restarts - 11 patch sets | 17:34 |
| yoctozepto | hmm, bifrost on Train fails at | 17:36 |
| yoctozepto | "msg": "The task includes an option with an undefined variable. The error was: 'kolla_internal_address' is undefined\n\nThe error appears to be in '/home/zuul/src/opendev.org/openstack/kolla-ansible/ansible/roles/bifrost/tasks/start.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending | 17:36 |
| yoctozepto | line appears to be:\n\n---\n- name: Starting bifrost deploy container\n ^ here\n" | 17:36 |
| openstackgerrit | Merged openstack/kolla master: Fix bifrost network config https://review.opendev.org/749469 | 18:13 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla stable/ussuri: Install newer virtualenv with relaxed importlib https://review.opendev.org/749548 | 18:29 |
| *** e0ne has quit IRC | 18:29 | |
| *** k_mouza has joined #openstack-kolla | 18:32 | |
| *** k_mouza has quit IRC | 18:36 | |
| *** vishalmanchanda has quit IRC | 18:43 | |
| *** sorin-mihai_ has quit IRC | 18:44 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: [WIP] Add HAcluster Ansible role https://review.opendev.org/670104 | 18:51 |
| *** ftarasenko has joined #openstack-kolla | 19:08 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/train: [CI] Fix Bifrost CentOS 8 job https://review.opendev.org/749576 | 19:08 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/train: Revert "openstack-base: pin setuptools < 46 to get horizon working" https://review.opendev.org/749534 | 19:11 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/train: Revert "openstack-base: pin setuptools < 46 to get horizon working" https://review.opendev.org/749534 | 19:11 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/train: Bump up versions for train https://review.opendev.org/749454 | 19:12 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/train: Upgrade Fluentd Monasca plugin https://review.opendev.org/743922 | 19:12 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/train: Bump up versions for train https://review.opendev.org/749454 | 19:15 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/train: Upgrade Fluentd Monasca plugin https://review.opendev.org/743922 | 19:15 |
| *** zzzeek has quit IRC | 19:17 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/train: Revert "openstack-base: pin setuptools < 46 to get horizon working" https://review.opendev.org/749534 | 19:17 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/train: Bump up versions for train https://review.opendev.org/749454 | 19:18 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/train: Upgrade Fluentd Monasca plugin https://review.opendev.org/743922 | 19:18 |
| *** zzzeek has joined #openstack-kolla | 19:18 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/ussuri: version-check: handle using of stable branch tarballs https://review.opendev.org/748989 | 19:21 |
| *** zzzeek has quit IRC | 19:25 | |
| *** zzzeek has joined #openstack-kolla | 19:27 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix external mariadb documentation for database_user https://review.opendev.org/749464 | 19:34 |
| *** nikparasyr has left #openstack-kolla | 19:35 | |
| *** e0ne has joined #openstack-kolla | 19:51 | |
| *** zzzeek has quit IRC | 19:53 | |
| *** oncall-pokemon has joined #openstack-kolla | 19:56 | |
| *** zzzeek has joined #openstack-kolla | 19:56 | |
| *** qqmber has joined #openstack-kolla | 20:26 | |
| qqmber | Hi! I'm adding a new node to my Openstack Rocky deploy with kolla-ansible. The problem is the installer try to install docker from yum.dockerproject.org, which doesn't exist.. I already install docker in the target server.. How can I disable docker installation in the deploy? thanks! | 20:27 |
| qqmber | help! | 20:36 |
| *** e0ne has quit IRC | 20:44 | |
| *** zzzeek has quit IRC | 20:50 | |
| *** zzzeek has joined #openstack-kolla | 20:51 | |
| *** k_mouza has joined #openstack-kolla | 21:12 | |
| *** k_mouza has quit IRC | 21:21 | |
| *** k_mouza has joined #openstack-kolla | 21:27 | |
| *** k_mouza has quit IRC | 21:31 | |
| qqmber | finally I installed the latest kolla and kolla-ansible and now fails on the installation of deltarpm: fatal: [rancagua.dalmacia.local]: FAILED! => {"changed": false, "msg": "No package matching 'drpm' found available, installed or updated", "rc": 126, "results": ["No package matching 'drpm' found available, installed or updated"]} | 21:46 |
| *** jbadiapa has quit IRC | 21:52 | |
| *** brtknr has quit IRC | 21:59 | |
| *** strobert has quit IRC | 21:59 | |
| *** Wasaac has quit IRC | 21:59 | |
| *** timss has quit IRC | 21:59 | |
| *** strobert has joined #openstack-kolla | 22:00 | |
| *** Wasaac has joined #openstack-kolla | 22:00 | |
| *** brtknr has joined #openstack-kolla | 22:00 | |
| *** timss has joined #openstack-kolla | 22:01 | |
| *** k_mouza has joined #openstack-kolla | 22:06 | |
| *** k_mouza has quit IRC | 22:11 | |
| qqmber | latest version only works with centos 8.. I have 7 and my hardware can't be upgraded to centos 8 | 22:13 |
| *** TrevorV has quit IRC | 22:18 | |
| *** jonaspaulo has quit IRC | 22:36 | |
| *** k_mouza has joined #openstack-kolla | 22:41 | |
| *** k_mouza has quit IRC | 22:47 | |
| openstackgerrit | James Kirsch proposed openstack/kolla master: Enable apache in Neutron server container https://review.opendev.org/749601 | 22:57 |
| *** hjensas has quit IRC | 23:11 | |
| *** k_mouza has joined #openstack-kolla | 23:19 | |
| *** k_mouza has quit IRC | 23:24 | |
| *** irclogbot_0 has quit IRC | 23:29 | |
| *** irclogbot_1 has joined #openstack-kolla | 23:33 | |
| *** Tengu has quit IRC | 23:34 | |
| *** Tengu has joined #openstack-kolla | 23:35 | |
| r3ap3r | qqmber: Train is still supported on CentOS 7 so that may be an option? Maybe not the newest but it gets you off Queens. ;) | 23:40 |
| r3ap3r | I have the same issue with CentOS 8 with my hardware, 11G Dell servers so the Raid controller support killed CentOS 8 support for me. I'm rocking Ubuntu Focal at the moment, trying to prep for Victoria and beyond. | 23:42 |
| r3ap3r | Disclaimer however, Kolla-ansible support for Focal is being built into Victoria and is not currently supported in Ussuri. | 23:43 |
| *** k_mouza has joined #openstack-kolla | 23:57 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!