| *** bsanjeewa has joined #openstack-kolla | 00:18 | |
| *** bsanjeewa has quit IRC | 01:09 | |
| *** zzzeek has quit IRC | 02:00 | |
| *** zzzeek has joined #openstack-kolla | 02:01 | |
| -openstackstatus- NOTICE: We are investigating an issue with our hosted Gerrit services. We will provide an update as soon as we can. If you want to follow the latest, feel free to join #opendev | 03:25 | |
| *** skramaja has joined #openstack-kolla | 03:27 | |
| *** skramaja_ has joined #openstack-kolla | 04:09 | |
| *** skramaja has quit IRC | 04:09 | |
| *** skramaja_ is now known as skramaja | 04:09 | |
| *** vishalmanchanda has joined #openstack-kolla | 04:26 | |
| -openstackstatus- NOTICE: We identified a possible vulnerability in Gerrit and are investigating the potential impact on our services. Out of an abundance of caution we have taken our OpenDev hosted Gerrit system offline. We will update with more information once we are able. | 04:30 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-kolla | 04:33 | |
| *** suryasingh has joined #openstack-kolla | 04:46 | |
| *** kemopq has joined #openstack-kolla | 05:20 | |
| *** gfidente has joined #openstack-kolla | 05:52 | |
| *** zzzeek has quit IRC | 06:01 | |
| *** zzzeek has joined #openstack-kolla | 06:04 | |
| *** cah_link has joined #openstack-kolla | 06:08 | |
| *** zzzeek has quit IRC | 06:32 | |
| *** zzzeek has joined #openstack-kolla | 06:36 | |
| *** wuchunyang has joined #openstack-kolla | 06:56 | |
| yoctozepto | morning | 06:58 |
|---|---|---|
| yoctozepto | hah, so much of my plans for a little upstream in the morning | 06:58 |
| yoctozepto | rescheduling eh | 06:59 |
| *** rpittau|afk is now known as rpittau | 07:04 | |
| *** bengates has joined #openstack-kolla | 07:17 | |
| *** nikparasyr has joined #openstack-kolla | 07:20 | |
| *** shyamb has joined #openstack-kolla | 07:26 | |
| mnasiadka | yoctozepto: well, let's hope they solve it soon | 07:27 |
| *** dougsz has joined #openstack-kolla | 07:33 | |
| *** shyam89 has joined #openstack-kolla | 07:34 | |
| yoctozepto | mnasiadka: infra is comparing backups now | 07:36 |
| yoctozepto | mnasiadka: so it just takes time | 07:36 |
| yoctozepto | no eta as far as I know though | 07:36 |
| mnasiadka | yoctozepto: I'm on #opendev, just read that :) | 07:36 |
| yoctozepto | mnasiadka: OK | 07:37 |
| *** shyamb has quit IRC | 07:37 | |
| mnasiadka | anyway, let's hope summit platform works a bit better today :D | 07:37 |
| osmanlicilegi | morning | 07:39 |
| yoctozepto | mnasiadka: so far it works pretty well | 07:42 |
| *** bsanjeewa has joined #openstack-kolla | 07:42 | |
| yoctozepto | mnasiadka: it worked well yesterday as well but only after the keynotes | 07:42 |
| mnasiadka | that's good, I had some other work to do at that time | 07:43 |
| *** skramaja has quit IRC | 07:47 | |
| *** skramaja has joined #openstack-kolla | 07:47 | |
| mgoddard | morning | 07:49 |
| *** e0ne has joined #openstack-kolla | 08:04 | |
| hrw | which timezone CDT is? I can only imagine Polish name so not everyone will understand | 08:11 |
| yoctozepto | hrw: UTC is given as well | 08:18 |
| yoctozepto | "Central Daylight Time (CDT) is 5 hours behind Coordinated Universal Time (UTC)." | 08:19 |
| hrw | once it load... 'There was a problem with our server, please contact admin.' | 08:19 |
| yoctozepto | geez, again? | 08:19 |
| hrw | this time it was privacy badger ;D | 08:19 |
| yoctozepto | ah, yeah; I just disabled it | 08:20 |
| yoctozepto | let oif track me | 08:20 |
| hrw | I see that messages were written by same team which did translations for my coffee machine... | 08:21 |
| hrw | "Login Required!" "Refill tank!" "Add coffee beans!" | 08:21 |
| hrw | "whatever!" "sensible message!" | 08:22 |
| mnasiadka | "go to hell!" ? :) | 08:22 |
| hrw | 'go to hell' is already aggressive | 08:22 |
| hrw | does not need '!' | 08:23 |
| yoctozepto | 'go to heaven!' then | 08:23 |
| mnasiadka | ultra-aggressive :) | 08:23 |
| yoctozepto | confusing message, eh xD | 08:23 |
| *** vishalmanchanda has quit IRC | 08:25 | |
| *** bengates has quit IRC | 08:30 | |
| hrw | meh. summit agenda is terrible | 08:30 |
| hrw | title, speaker, time. no more information | 08:30 |
| *** bengates has joined #openstack-kolla | 08:31 | |
| mgoddard | hrw: looks like that enum change might have caused a problem: https://zuul.opendev.org/t/openstack/build/b57032f2749b4048a5b74e50fc9a2ffc | 08:31 |
| hrw | hm. 3.25GB container image is not small... but nicer than 11GB before | 08:31 |
| hrw | ERROR:kolla.common.utils.tempest:Failed to download archive: status_code 404 | 08:32 |
| hrw | ERROR:kolla.common.utils.tempest:Failed turning any plugins into a plugins archive | 08:32 |
| hrw | so it failed | 08:32 |
| *** shyam89 has quit IRC | 08:32 | |
| hrw | './tools/build.py tempest' - let's check | 08:33 |
| *** bengates has quit IRC | 08:35 | |
| -openstackstatus- NOTICE: We identified a possible vulnerability in Gerrit and are investigating the potential impact on our services. Out of an abundance of caution we have taken our OpenDev hosted Gerrit system offline. We will update with more information once we are able. | 08:37 | |
| *** ChanServ changes topic to "We identified a possible vulnerability in Gerrit and are investigating the potential impact on our services. Out of an abundance of caution we have taken our OpenDev hosted Gerrit system offline. We will update with more information once we are able." | 08:37 | |
| *** jonaspaulo has joined #openstack-kolla | 08:41 | |
| hrw | INFO:kolla.common.utils.base:Successfully tagged kolla/centos-source-base:10.1.0 | 08:41 |
| hrw | INFO:kolla.common.utils.base:Built at 2020-10-20 10:41:37.021220 (took 0:00:02.704517) | 08:41 |
| hrw | I like how infra change made base image being same for binary/source | 08:42 |
| *** bengates has joined #openstack-kolla | 08:45 | |
| *** priteau has joined #openstack-kolla | 08:46 | |
| mgoddard | hrw: I mean the JSON error at the bottom of the page. I think it is failing due to enum.Status | 08:50 |
| hrw | mgoddard: thanks. wrote a fix. | 08:51 |
| mgoddard | cool | 08:51 |
| hrw | https://paste.centos.org/view/0821bb32 | 08:51 |
| hrw | ERROR:kolla.common.utils:tempest Failed with status: error | 08:52 |
| mgoddard | hrw: yeah I have a fix for that. Just waiting for gerrit to return from holiday | 08:53 |
| hrw | mgoddard: I pasted that it works after fix ;D (works as: shows error properly) | 08:54 |
| mgoddard | ah, nice | 08:54 |
| *** k_mouza has joined #openstack-kolla | 08:56 | |
| *** bengates has quit IRC | 09:10 | |
| priteau | Should we start emailing patches like in the good old days? ;-) | 09:11 |
| *** bengates has joined #openstack-kolla | 09:12 | |
| *** bengates has quit IRC | 09:13 | |
| mgoddard | priteau: I have printed one out and posted it to you | 09:13 |
| *** shyamb has joined #openstack-kolla | 09:23 | |
| wuchunyang | could you open review.opendev.org ? | 09:32 |
| yoctozepto | wuchunyang: down for maintenance | 09:33 |
| wuchunyang | yoctozepto thanks ：-） | 09:37 |
| *** shyam89 has joined #openstack-kolla | 09:39 | |
| *** bengates has joined #openstack-kolla | 09:40 | |
| *** shyamb has quit IRC | 09:42 | |
| *** bsanjeewa has quit IRC | 09:43 | |
| *** bengates has quit IRC | 09:44 | |
| *** bsanjeewa has joined #openstack-kolla | 09:45 | |
| *** bengates has joined #openstack-kolla | 09:45 | |
| *** bsanjeewa has quit IRC | 09:51 | |
| *** shyam89 has quit IRC | 10:02 | |
| *** shyamb has joined #openstack-kolla | 10:03 | |
| *** kevko has joined #openstack-kolla | 10:05 | |
| *** bengates_ has joined #openstack-kolla | 10:05 | |
| kevko | hi guys, can someone advise me how to solve problem with "keystone | creating services" if i turned on ssl on all places ? keystone is reconfigured ..and if i source admin.rc with https://ip:35357/v3 it's working ..but kolla-toolbox creating services failed :( | 10:06 |
| kevko | (turned on and ran reconfigure ) | 10:06 |
| *** bengates_ has quit IRC | 10:07 | |
| *** bengates has quit IRC | 10:07 | |
| *** bengates has joined #openstack-kolla | 10:08 | |
| mgoddard | kevko: have you added your private CA cert to the containers? | 10:12 |
| mgoddard | kevko: https://docs.openstack.org/kolla-ansible/latest/admin/tls.html#adding-ca-certificates-to-the-service-containers | 10:13 |
| kevko | i've added this to globals.yml | 10:13 |
| kevko | kolla_enable_tls_internal: "yes" | 10:13 |
| kevko | kolla_enable_tls_external: "yes" | 10:13 |
| kevko | kolla_enable_tls_backend: "yes" | 10:13 |
| kevko | kolla_copy_ca_into_containers: "yes" | 10:13 |
| mgoddard | and copied the private CA to /etc/kolla/certificates/ca? | 10:14 |
| shyamb | Hi | 10:15 |
| kevko | i've run kolla-ansible -i inventory certificates ...so these files was generated by kolla-ansible ..so i assume it is ok | 10:15 |
| shyamb | Does kolla-ansible ussuri supports TLS on all endpoints? | 10:15 |
| shyamb | mgoddard: ^^ | 10:15 |
| mgoddard | shyamb: yes | 10:15 |
| mgoddard | kevko: that should work then | 10:16 |
| kevko | i think yes it should work | 10:17 |
| *** wuchunyang has quit IRC | 10:17 | |
| shyamb | mgoddard: okay, thanks. | 10:17 |
| kevko | maybe it's problem in kolla-toolbox container ? not imported ca ? | 10:18 |
| mgoddard | kevko: check in /usr/local/share/ca-certificates/ | 10:18 |
| mgoddard | kevko: have you set openstack_cacert? | 10:19 |
| mgoddard | https://docs.openstack.org/kolla-ansible/latest/admin/tls.html#configuring-a-ca-bundle | 10:19 |
| shyamb | mgoddard: Please point me to the TLS configuration doc, if you have it handy | 10:19 |
| mgoddard | shyamb: very handy - I just posted two links to it :) | 10:20 |
| mgoddard | https://docs.openstack.org/kolla-ansible/latest/admin/tls.html | 10:20 |
| shyamb | mgoddard: great, thanks. :) | 10:20 |
| kevko | mgoddard: yes, problem was openstack_cacert missing | 10:22 |
| mgoddard | kevko: if you think the tls doc needs to be clearer, feel free to make suggestions | 10:42 |
| mgoddard | kevko: it's still fresh | 10:42 |
| *** k_mouza has quit IRC | 10:53 | |
| *** bengates has quit IRC | 10:58 | |
| -openstackstatus- NOTICE: Update on gerrit downtime: After investigation, we believe the incident is related to a compromised Gerrit user account rather than a vulnerability in Gerrit software. We are continuing to review activity to verify the integrity of git data and expect to have an additional update with possible service restoration in approximately 2 hours. | 11:05 | |
| *** ChanServ changes topic to "Update on gerrit downtime: After investigation, we believe the incident is related to a compromised Gerrit user account rather than a vulnerability in Gerrit software. We are continuing to review activity to verify the integrity of git data and expect to have an additional update with possible service restoration in approximately 2 hours." | 11:05 | |
| *** shyamb has quit IRC | 11:07 | |
| *** bengates has joined #openstack-kolla | 11:07 | |
| *** shyamb has joined #openstack-kolla | 11:10 | |
| *** vishalmanchanda has joined #openstack-kolla | 11:13 | |
| *** wuchunyang has joined #openstack-kolla | 11:14 | |
| mgoddard | yoctozepto: have you heard anything from infra about Dockerhub? I'm getting worried we will be hit badly soon | 11:34 |
| *** shyamb has quit IRC | 11:36 | |
| *** kevko has quit IRC | 11:38 | |
| *** k_mouza has joined #openstack-kolla | 11:44 | |
| yoctozepto | mgoddard: nothing; considering recent bad luck you might be true ;p | 11:47 |
| *** bsanjeewa has joined #openstack-kolla | 11:51 | |
| *** k_mouza has quit IRC | 11:54 | |
| *** kevko has joined #openstack-kolla | 11:59 | |
| *** wuchunyang has quit IRC | 12:02 | |
| *** bsanjeewa has quit IRC | 12:03 | |
| *** kevko has quit IRC | 12:03 | |
| *** dardelean has joined #openstack-kolla | 12:07 | |
| *** bsanjeewa has joined #openstack-kolla | 12:09 | |
| *** weshay|ruck has joined #openstack-kolla | 12:14 | |
| *** k_mouza has joined #openstack-kolla | 12:50 | |
| *** Luzi has joined #openstack-kolla | 12:55 | |
| *** bengates has quit IRC | 13:00 | |
| *** bengates has joined #openstack-kolla | 13:01 | |
| *** bengates_ has joined #openstack-kolla | 13:01 | |
| *** bengates_ has quit IRC | 13:01 | |
| *** bengates_ has joined #openstack-kolla | 13:02 | |
| *** bengates has quit IRC | 13:03 | |
| *** Fl1nt has joined #openstack-kolla | 13:04 | |
| Fl1nt | Hi everyone, good afternoon! | 13:04 |
| Fl1nt | mgoddard, when you deployed cloudkitty using ES backend, did you used prometheus as a collector too by any chance? | 13:05 |
| mgoddard | Fl1nt: nope, using gnocchi | 13:06 |
| mgoddard | I think we had issues with prometheus collector, although it wasn't me that tried | 13:06 |
| Fl1nt | arff, bad luck :( I get an error too: voluptuous.error.MultipleInvalid: required key not provided @ data['extra_args'] was your issue close to that for instance? | 13:07 |
| Fl1nt | Hold on... I think I know where that come from. | 13:08 |
| Fl1nt | mgoddard, from my understanding of the cloudkitty doc, it seems like metrics.yml is required for some collector such as prometheus, do we want to include the cloudkitty: /etc/cloudkitty/metrics.yml default file within CK role? | 13:29 |
| Fl1nt | they themselves provide it, so I guess it should: https://opendev.org/openstack/cloudkitty/src/branch/master/etc/cloudkitty/metrics.yml | 13:30 |
| mgoddard | Fl1nt: using a custom metrics.yml is supported | 13:31 |
| mgoddard | /etc/kolla/config/cloudkitty/metrics.yml | 13:31 |
| Fl1nt | I know, it need to be validated, but I think it is a requirement to already be there for prometheus (currently checking) so if it's a requirement, we should include it and upload it to the host when prometheus collector is selected. | 13:32 |
| Fl1nt | IMHO, any kolla-ansible role should work out of the box as soon as the ops use a supported option, so as I'm working to get prometheus and ES as available options for CK out of the box I think it may be there too. | 13:33 |
| Fl1nt | at least the default one. | 13:33 |
| -openstackstatus- NOTICE: We've confirmed that known compromised identities have been reset or had their accounts disabled, and we are auditing other service accounts for signs of compromise before we prepare to restore Gerrit to working order. We will update again in roughly 2 hours. | 13:33 | |
| *** ChanServ changes topic to "We've confirmed that known compromised identities have been reset or had their accounts disabled, and we are auditing other service accounts for signs of compromise before we prepare to restore Gerrit to working order. We will update again in roughly 2 hours." | 13:34 | |
| Fl1nt | wooow | 13:36 |
| *** TrevorV has joined #openstack-kolla | 13:37 | |
| *** k_mouza has quit IRC | 13:56 | |
| *** k_mouza has joined #openstack-kolla | 13:57 | |
| *** bengates_ has quit IRC | 14:00 | |
| *** bengates has joined #openstack-kolla | 14:01 | |
| *** cah_link has quit IRC | 14:09 | |
| *** kevko has joined #openstack-kolla | 14:10 | |
| kevko | guys, did someone meet this error while openstack switched to TLS ? | 14:12 |
| kevko | line 2641, in _build_resources reason=six.text_type(exc)) nova.exception.BuildAbortException: Build of instance d7ddd907-67f2-47d9-b4ee-797ac24e601c aborted: Unable to establish connection to http://192.168.205.254:9696/v2.0/ports/aaf04bfc-64e9-4a46-9b00-7893b956efd4: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) | 14:12 |
| kevko | (i really don't know why it is trying to connect to http://) .. | 14:12 |
| kevko | grepped /etc/kolla on all hosts to match http:// <- nothing , restarted controllers ..nothing .. check endpoints ..nothing | 14:13 |
| kevko | list ports via openstack port list <- ok ..i really don't know what to do .. | 14:13 |
| kevko | flushed memcached ..restarted keystones .. no way | 14:14 |
| kevko | any advice ? | 14:14 |
| mnasiadka | is this internal tls? | 14:14 |
| kevko | mnasiadka: kolla_enable_tls_internal: "yes" | 14:15 |
| kevko | kolla_enable_tls_external: "yes" | 14:15 |
| kevko | kolla_copy_ca_into_containers: "yes" | 14:15 |
| kevko | kolla_enable_tls_backend: "yes" | 14:15 |
| kevko | openstack_cacert: "/etc/ssl/certs/ca-certificates.crt" | 14:15 |
| kevko | yes | 14:15 |
| mnasiadka | and it's https:// on the internal endpoint in keystone catalog? | 14:15 |
| kevko | yes ! | 14:16 |
| ozzzo | kevko: what do you get from: openstack endpoint list|grep 9696 | 14:17 |
| priteau | Did you restart nova itself? | 14:17 |
| kevko | priteau: yes | 14:17 |
| kevko | http://paste.openstack.org/show/799212/ << endpoints 9696 | 14:18 |
| *** k_mouza has quit IRC | 14:22 | |
| kevko | it looks like when nova is creating session it is using http schema instead of https .. | 14:22 |
| *** cah_link has joined #openstack-kolla | 14:22 | |
| *** bsanjeewa has quit IRC | 14:33 | |
| *** cah_link has quit IRC | 14:47 | |
| *** cmorey has joined #openstack-kolla | 14:54 | |
| dmsimard | does anyone have an implementation of "nova-manage db archive_deleted_rows" in kolla-ansible ? I'm otherwise trying to find the right pattern to enable it on a single controller | 14:58 |
| yoctozepto | dmsimard: are you trying to have it run on a schedule? | 15:00 |
| dmsimard | yeah, probably weekly in our case | 15:00 |
| dmsimard | I mean, I could probably set up a cron "manually" on a controller that does a "docker exec -it nova_api nova-manage db archive_deleted_rows" but I don't know if that's an antipattern :p | 15:01 |
| dmsimard | I saw there were crons for things like fernet tokens or elasticsearch curators and I'm not sure if I should follow that or not | 15:03 |
| *** bsanjeewa has joined #openstack-kolla | 15:05 | |
| dmsimard | a wild mgoddard has appeared in the keynote :D | 15:28 |
| mgoddard | w00p | 15:29 |
| mgoddard | dmsimard: I have wondered if we need something like that | 15:29 |
| *** skramaja has quit IRC | 15:30 | |
| *** bsanjeewa_ has joined #openstack-kolla | 15:31 | |
| *** bsanjeewa_ has quit IRC | 15:37 | |
| *** kevko has quit IRC | 15:37 | |
| *** bsanjeewa has quit IRC | 15:37 | |
| -openstackstatus- NOTICE: Auditing is progressing but not particularly quickly. We'll keep updating every 2 hours or so. | 15:41 | |
| *** ChanServ changes topic to "Auditing is progressing but not particularly quickly. We'll keep updating every 2 hours or so." | 15:41 | |
| *** bengates has quit IRC | 15:44 | |
| *** bsanjeewa has joined #openstack-kolla | 15:45 | |
| *** bsanjeewa_ has joined #openstack-kolla | 15:45 | |
| *** e0ne has quit IRC | 15:55 | |
| *** dardelean has quit IRC | 16:06 | |
| *** dougsz has quit IRC | 16:31 | |
| *** KurtB has joined #openstack-kolla | 16:36 | |
| *** rpittau is now known as rpittau|afk | 16:38 | |
| yoctozepto | dmsimard: do I need a masterball to have a reasonable chance to catch the legendary mgoddard? | 16:55 |
| yoctozepto | if he's in his cave, I might go away with a bunch of dusk balls... | 16:56 |
| *** cmorey has quit IRC | 17:00 | |
| *** Luzi has quit IRC | 17:09 | |
| dmsimard | haha | 17:18 |
| *** williampiv has quit IRC | 17:34 | |
| *** bsanjeewa has quit IRC | 17:49 | |
| *** bsanjeewa_ has quit IRC | 17:49 | |
| -openstackstatus- NOTICE: Gerrit is offline due to a security compromise. Please refer to https://review.opendev.org/maintenance.html or #opendev for the latest updates. | 18:01 | |
| *** ChanServ changes topic to "Gerrit is offline due to a security compromise. Please refer to https://review.opendev.org/maintenance.html or #opendev for the latest updates." | 18:01 | |
| *** gfidente is now known as gfidente|afk | 18:02 | |
| *** Tengu has quit IRC | 18:06 | |
| *** Tengu has joined #openstack-kolla | 18:08 | |
| *** bsanjeewa_ has joined #openstack-kolla | 18:20 | |
| *** bsanjeewa has joined #openstack-kolla | 18:20 | |
| *** Tengu has quit IRC | 18:21 | |
| *** bsanjeewa_ has quit IRC | 18:27 | |
| *** bsanjeewa has quit IRC | 18:27 | |
| *** Tengu has joined #openstack-kolla | 18:27 | |
| *** bsanjeewa has joined #openstack-kolla | 18:31 | |
| *** bsanjeewa_ has joined #openstack-kolla | 18:31 | |
| *** JamesBenson has joined #openstack-kolla | 18:32 | |
| *** JamesBenson has quit IRC | 18:36 | |
| *** stingrayza has joined #openstack-kolla | 18:37 | |
| *** bsanjeewa_ has quit IRC | 18:37 | |
| *** bsanjeewa has quit IRC | 18:38 | |
| *** also_stingrayza has quit IRC | 18:40 | |
| *** vishalmanchanda has quit IRC | 18:53 | |
| *** nikparasyr has left #openstack-kolla | 18:53 | |
| *** williampiv has joined #openstack-kolla | 18:59 | |
| *** williampiv has quit IRC | 19:09 | |
| *** priteau has quit IRC | 19:16 | |
| *** kemopq has quit IRC | 19:48 | |
| *** williampiv has joined #openstack-kolla | 20:00 | |
| *** williampiv has quit IRC | 20:10 | |
| *** cah_link has joined #openstack-kolla | 20:40 | |
| *** JamesBenson has joined #openstack-kolla | 20:40 | |
| *** cah_link1 has joined #openstack-kolla | 20:44 | |
| *** cah_link has quit IRC | 20:44 | |
| *** cah_link1 is now known as cah_link | 20:44 | |
| *** TrevorV has quit IRC | 20:55 | |
| *** gfidente|afk has quit IRC | 21:28 | |
| *** Fl1nt has quit IRC | 21:32 | |
| *** williampiv has joined #openstack-kolla | 21:35 | |
| *** e0ne has joined #openstack-kolla | 21:47 | |
| *** JamesBenson has quit IRC | 21:54 | |
| *** e0ne has quit IRC | 22:06 | |
| *** jonaspaulo has quit IRC | 22:37 | |
| *** cah_link has quit IRC | 22:46 | |
| *** jbadiapa has quit IRC | 22:54 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!