| *** armax has joined #openstack-lbaas | 00:10 | |
| rm_work | yeah can we just drop like... 3 of them? :D | 01:12 |
|---|---|---|
| *** armax has quit IRC | 01:32 | |
| *** vishalmanchanda has joined #openstack-lbaas | 02:18 | |
| *** Dinesh_Bhor has quit IRC | 02:21 | |
| *** armax has joined #openstack-lbaas | 02:22 | |
| *** psachin has joined #openstack-lbaas | 03:35 | |
| *** armax has quit IRC | 03:56 | |
| *** numans has quit IRC | 06:02 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 06:08 | |
| *** gthiemon1e is now known as gthiemonge | 07:23 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Jobboard based controller https://review.opendev.org/647406 | 07:25 |
| *** ataraday has joined #openstack-lbaas | 07:45 | |
| *** rpittau|afk is now known as rpittau | 08:10 | |
| *** tkajinam has quit IRC | 08:12 | |
| *** gcheresh_ has joined #openstack-lbaas | 08:25 | |
| *** tesseract has joined #openstack-lbaas | 08:36 | |
| *** ivve has joined #openstack-lbaas | 08:37 | |
| *** gcheresh_ has quit IRC | 09:06 | |
| *** kollarl has joined #openstack-lbaas | 09:25 | |
| kollarl | hi | 09:25 |
| kollarl | I would like to install octavia on centos 7 (Stein), octavia-db-manage upgrade head doesn't create db schema at all, thus the db-manage run into error http://paste.openstack.org/show/789272/ | 09:25 |
| *** armax has joined #openstack-lbaas | 09:31 | |
| *** gcheresh_ has joined #openstack-lbaas | 09:45 | |
| *** devfaz has quit IRC | 10:10 | |
| *** baffle has quit IRC | 10:26 | |
| *** devfaz has joined #openstack-lbaas | 10:45 | |
| *** gthiemonge has quit IRC | 10:48 | |
| *** vishalmanchanda has quit IRC | 10:50 | |
| *** gthiemonge has joined #openstack-lbaas | 10:52 | |
| cgoncalves | kollarl, hi. it seems your database is on postgresql. octavia CI only tests on mysql. there are a few reports from other users on postgresql with the same problem as you | 11:04 |
| cgoncalves | https://review.opendev.org/#/c/587400/ | 11:04 |
| cgoncalves | https://review.opendev.org/#/c/625557/ | 11:04 |
| cgoncalves | unfortunately, these patches are still open. you may want to give it a try? patch your system with that last one for example | 11:05 |
| cgoncalves | or create the octavia database on mysql | 11:06 |
| kollarl | cgoncalves, Thank you! I am going to check it! | 11:07 |
| *** rpittau is now known as rpittau|bbl | 11:18 | |
| kollarl | cgoncalves, working with mysql. Thank you! | 11:21 |
| *** kollarl has quit IRC | 11:23 | |
| cgoncalves | oh, kollarl left. well, glad he/she managed with mysql | 11:33 |
| *** baffle has joined #openstack-lbaas | 11:57 | |
| *** nicolasbock has joined #openstack-lbaas | 12:13 | |
| *** spatel has joined #openstack-lbaas | 12:43 | |
| *** spatel has quit IRC | 12:50 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Add optional double check for stale amphora https://review.opendev.org/706493 | 13:10 |
| *** rpittau|bbl is now known as rpittau | 13:14 | |
| *** hongbin has joined #openstack-lbaas | 13:15 | |
| *** gmann_ has joined #openstack-lbaas | 13:40 | |
| *** gmann_ is now known as gmann | 13:40 | |
| *** gmann has quit IRC | 13:40 | |
| *** gmann has joined #openstack-lbaas | 13:41 | |
| *** hongbin has quit IRC | 13:54 | |
| *** yamamoto has joined #openstack-lbaas | 13:58 | |
| *** tesseract-RH has joined #openstack-lbaas | 14:19 | |
| *** tesseract has quit IRC | 14:21 | |
| *** gcheresh_ has quit IRC | 14:27 | |
| *** spatel has joined #openstack-lbaas | 14:30 | |
| *** tesseract has joined #openstack-lbaas | 14:34 | |
| *** spatel has quit IRC | 14:34 | |
| *** tesseract has quit IRC | 14:43 | |
| *** tesseract-RH has quit IRC | 14:43 | |
| *** tesseract has joined #openstack-lbaas | 14:44 | |
| *** rpittau is now known as rpittau|brb | 15:04 | |
| *** TrevorV has joined #openstack-lbaas | 15:08 | |
| *** gcheresh_ has joined #openstack-lbaas | 15:08 | |
| *** yamamoto has quit IRC | 15:12 | |
| *** yamamoto has joined #openstack-lbaas | 15:15 | |
| *** ivve has quit IRC | 15:15 | |
| *** yamamoto has quit IRC | 15:20 | |
| *** rpittau|brb is now known as rpittau | 15:20 | |
| *** gcheresh_ has quit IRC | 15:27 | |
| *** pcaruana has quit IRC | 15:27 | |
| *** pcaruana has joined #openstack-lbaas | 15:33 | |
| brtknr | does neutron lbaas v2 support cascade delete? | 15:44 |
| brtknr | or is this only supported by octavia cli? | 15:47 |
| johnsom | That is an Octavia feature only | 15:51 |
| johnsom | Nuetron-lbaas cannot do cascade delete | 15:52 |
| *** yamamoto has joined #openstack-lbaas | 15:53 | |
| *** yamamoto has quit IRC | 15:58 | |
| *** sapd1_x has joined #openstack-lbaas | 16:13 | |
| *** tesseract has quit IRC | 16:38 | |
| *** hongbin has joined #openstack-lbaas | 16:55 | |
| *** tesseract has joined #openstack-lbaas | 16:57 | |
| *** hongbin has quit IRC | 16:59 | |
| *** rpittau is now known as rpittau|afk | 17:02 | |
| *** yamamoto has joined #openstack-lbaas | 17:04 | |
| *** yamamoto has quit IRC | 17:08 | |
| *** gcheresh_ has joined #openstack-lbaas | 17:17 | |
| *** psachin has quit IRC | 17:21 | |
| *** tesseract has quit IRC | 17:36 | |
| *** ivve has joined #openstack-lbaas | 17:38 | |
| *** yamamoto has joined #openstack-lbaas | 17:40 | |
| *** yamamoto has quit IRC | 17:44 | |
| *** gcheresh_ has quit IRC | 17:45 | |
| *** gcheresh_ has joined #openstack-lbaas | 17:58 | |
| *** yamamoto has joined #openstack-lbaas | 18:56 | |
| *** yamamoto has quit IRC | 19:00 | |
| *** gcheresh_ has quit IRC | 19:14 | |
| *** yamamoto has joined #openstack-lbaas | 19:29 | |
| *** yamamoto has quit IRC | 19:35 | |
| openstackgerrit | Vlad Gusev proposed openstack/octavia stable/train: Update the lb_id on an amp earlier if we know it https://review.opendev.org/706615 | 19:51 |
| openstackgerrit | Vlad Gusev proposed openstack/octavia stable/train: Update the lb_id on an amp earlier if we know it https://review.opendev.org/706615 | 19:53 |
| rm_work | this is the right thing to add to the image properties, right? cgoncalves / johnsom : hw_rng_model='virtio' | 20:21 |
| johnsom | rm_work https://wiki.openstack.org/wiki/LibvirtVirtioRng | 20:23 |
| openstackgerrit | Vlad Gusev proposed openstack/octavia stable/train: Update the lb_id on an amp earlier if we know it https://review.opendev.org/706615 | 20:23 |
| johnsom | Not sure why that patch is so popular... lol | 20:24 |
| *** yamamoto has joined #openstack-lbaas | 20:47 | |
| rm_work | you don't need to do a lot of debugging on amp boot failures? :D | 20:49 |
| rm_work | it's helping me *as we speak*, heh | 20:49 |
| johnsom | Nope, I don't | 20:50 |
| rm_work | i wish we logged a little info about the amps as they're booting actually, too... like their compute-id and maybe IP address, during the "trying to connect" phase | 20:50 |
| rm_work | so we can tell what's what (right now the trying to connect messages are all generic, if you're booting three LBs at once it's all just noise) | 20:50 |
| johnsom | We log both | 20:50 |
| rm_work | where? i am looking at a full set of logs from LB create to amp boot failure | 20:51 |
| rm_work | and I don't see anything about compute-id or IP | 20:51 |
| *** yamamoto has quit IRC | 20:52 | |
| johnsom | https://zuul.opendev.org/t/openstack/build/d9f9e9260e2d4e679b5cf764b8b5644e/log/controller/logs/screen-o-cw.txt#1553 | 20:52 |
| johnsom | Oh, wrong line | 20:52 |
| johnsom | This one: https://zuul.opendev.org/t/openstack/build/d9f9e9260e2d4e679b5cf764b8b5644e/log/controller/logs/screen-o-cw.txt#1582 | 20:53 |
| rm_work | ah damnit, we may not have debug on | 20:53 |
| johnsom | https://zuul.opendev.org/t/openstack/build/d9f9e9260e2d4e679b5cf764b8b5644e/log/controller/logs/screen-o-cw.txt#1605 | 20:53 |
| johnsom | And each connect line has the lb-mgmt-net IP | 20:53 |
| johnsom | Or this line too: https://zuul.opendev.org/t/openstack/build/d9f9e9260e2d4e679b5cf764b8b5644e/log/controller/logs/screen-o-cw.txt#1600 | 20:54 |
| johnsom | But, yeah, those are debug. | 20:54 |
| johnsom | Really, we should do a sweep sometime to create/pull out useful non-debug log info. It's been a while since I have done that. | 20:54 |
| johnsom | But the IP is Warning level | 20:55 |
| johnsom | Though I think we have considered dropping that to a lower log level | 20:55 |
| *** spatel has joined #openstack-lbaas | 20:55 | |
| rm_work | eugh when we rollback a failed amp boot, the record gets deleted from the DB doesn't it T_T | 20:58 |
| johnsom | Yes, we have to | 20:59 |
| rm_work | yeah | 21:00 |
| rm_work | not saying that's wrong, i'm just sad | 21:00 |
| rm_work | because it's making my life harder | 21:00 |
| rm_work | AH but i can do `openstack server list --deleted --project octavia-service | grep '$AMP_ID'` | 21:00 |
| johnsom | Yeah, I was just going to say, go straight to nova as that is what is broken | 21:01 |
| rm_work | eugh but no console logs for deleted servers T_T | 21:01 |
| johnsom | Our log offloading will catch it, if nova even bothered to boot it | 21:01 |
| rm_work | it booted, but then was never networkable | 21:02 |
| rm_work | so no | 21:02 |
| rm_work | maybe the HV keeps it | 21:02 |
| johnsom | Probably stating the obvious, but you remember you can disable reverts in the config file and it won't delete anything right? | 21:04 |
| rm_work | yeah but i need HISTORY | 21:05 |
| rm_work | i don't know how to replicate | 21:05 |
| rm_work | if i knew how to replicate this wouldn't be a problem T_T | 21:05 |
| johnsom | log == history | 21:05 |
| rm_work | i mean, i can't make a new one happen | 21:06 |
| rm_work | so if i could go in the past and make revert be on, great :D | 21:06 |
| johnsom | Right, I get it. Up the logging | 21:06 |
| rm_work | yeah, can try switching debug on | 21:06 |
| rm_work | though the issue is that the server is never connectable, so it doesn't really help me to have octavia logs | 21:07 |
| rm_work | i need to find the nova console logs :D | 21:07 |
| rm_work | but it seems they're deleted off the HV when the instance is deleted | 21:07 |
| johnsom | So you are guessing the VM didn't get an IP for the lb-mgmt-net? | 21:09 |
| rm_work | ahh the MARK ALLOCATED gives me the compute-id | 21:09 |
| rm_work | great | 21:09 |
| rm_work | uhh | 21:09 |
| rm_work | i am guessing it just didn't spin up totally | 21:09 |
| rm_work | need console logs | 21:09 |
| openstackgerrit | Vlad Gusev proposed openstack/octavia stable/train: Update the lb_id on an amp earlier if we know it https://review.opendev.org/706615 | 21:11 |
| johnsom | Ok, lunch is over, back to work. | 21:19 |
| johnsom | I'm close on amphora failover. | 21:19 |
| johnsom | I had hoped I could get away without some of this, but.... | 21:19 |
| johnsom | taskflow.exceptions.MissingDependencies: 'linear_flow.Flow: octavia-failover-amphora-flow(len=12)' requires ['availability_zone', 'flavor', 'loadbalancer', 'loadbalancer_id', 'vip', 'vip_sg_id'] but no other entity produces said requirements | 21:19 |
| johnsom | At least the flow compiles | 21:19 |
| johnsom | Argh, yeah, the reboot brought in this availability zone stuff... | 21:24 |
| johnsom | joy | 21:25 |
| *** yamamoto has joined #openstack-lbaas | 21:25 | |
| cgoncalves | rm_work, https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L665 | 21:30 |
| cgoncalves | the virtio thing you asked about | 21:30 |
| *** yamamoto has quit IRC | 21:30 | |
| johnsom | lol, I just pointed him at the nova docs page | 21:30 |
| cgoncalves | our "docs" are better ;) | 21:31 |
| rm_work | yeah heh but it's not that | 21:31 |
| rm_work | i finally managed to track down console logs | 21:31 |
| cgoncalves | speaking of that, still need to add that to tripleo | 21:31 |
| rm_work | it comes up to login | 21:31 |
| rm_work | but the network interface just ... never comes up | 21:31 |
| johnsom | But, did it get an IP address. | 21:31 |
| rm_work | yes | 21:32 |
| rm_work | so the logs look like this for the good one: | 21:32 |
| rm_work | http://paste.openstack.org/show/7MkHuoiX1K1y6RxkV5ca/ | 21:32 |
| rm_work | well, i put both in there | 21:32 |
| rm_work | it's now been like 10 minutes | 21:32 |
| rm_work | the "good" amp got interface attached quickly | 21:33 |
| rm_work | the "bad" one just... doesn't | 21:33 |
| johnsom | The good one had a VIP applied | 21:33 |
| rm_work | ah | 21:33 |
| rm_work | that's what that is? | 21:33 |
| rm_work | yeah it is isn't it | 21:33 |
| rm_work | fff | 21:33 |
| rm_work | ok | 21:33 |
| rm_work | so that's not actually the issue | 21:33 |
| johnsom | You need to look for a line like this: | 21:34 |
| rm_work | but nova logs in the past have shown us an interface attach delayed by like 20m | 21:34 |
| johnsom | [ 5.237182] cloud-init[544]: ci-info: | ens2 | True | 192.168.0.172 | 255.255.255.0 | global | fa:16:3e:96:e4:23 | | 21:34 |
| johnsom | above that in the log | 21:34 |
| rm_work | i'm just scrolling through a vimdiff of the whole log right now, side by side :D | 21:35 |
| rm_work | good and bad | 21:35 |
| johnsom | This is a "good" log block: | 21:35 |
| johnsom | https://www.irccloud.com/pastebin/7B4AjFj2/ | 21:35 |
| johnsom | You might not have the IPv6 stuff obviously | 21:36 |
| rm_work | yeah ok | 21:38 |
| rm_work | i see that block | 21:38 |
| rm_work | comparing them ... they look ... nearly identical | 21:38 |
| rm_work | the bad one has ... one ADDITIONAL ipv6 route `local`? but that's the only difference | 21:39 |
| JayF | that shouldn't be relevant | 21:39 |
| johnsom | Yep | 21:40 |
| johnsom | Unless your lb-mgmt-net is IPv6 | 21:40 |
| rm_work | OK well FINALLY got access to the bad one via SSH (net came up after about 20m) | 21:40 |
| rm_work | what can i look at in like, dmesg / syslog? | 21:41 |
| JayF | rm_work: you should hangouts me so I can look at your logs in real time <3 | 21:41 |
| rm_work | lol | 21:41 |
| rm_work | k | 21:41 |
| openstackgerrit | Vlad Gusev proposed openstack/octavia stable/train: Update the lb_id on an amp earlier if we know it https://review.opendev.org/706615 | 22:19 |
| JayF | for those following around at home, it appears external to the instance (i.e. the interface didn't get plugged immediately) | 22:21 |
| johnsom | Ah, so some neutron issue | 22:25 |
| rm_work | yeah looks like | 22:31 |
| *** spatel has quit IRC | 22:32 | |
| *** yamamoto has joined #openstack-lbaas | 22:39 | |
| *** spatel has joined #openstack-lbaas | 22:43 | |
| *** yamamoto has quit IRC | 22:44 | |
| *** spatel has quit IRC | 22:48 | |
| *** TrevorV has quit IRC | 22:57 | |
| *** armax has quit IRC | 23:06 | |
| *** yamamoto has joined #openstack-lbaas | 23:12 | |
| *** yamamoto has quit IRC | 23:18 | |
| *** spatel has joined #openstack-lbaas | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!