Monday, 2019-05-06

« Sunday, 2019-05-05 Index Tuesday, 2019-05-07 »
*** yamamoto has joined #openstack-lbaas00:31
*** yamamoto has quit IRC01:33
*** hongbin has joined #openstack-lbaas02:23
*** altlogbot_2 has quit IRC02:41
*** altlogbot_3 has joined #openstack-lbaas02:43
*** yamamoto has joined #openstack-lbaas03:02
*** hongbin has quit IRC03:13
*** psachin has joined #openstack-lbaas03:18
*** yamamoto has quit IRC03:29
*** yamamoto has joined #openstack-lbaas03:29
*** altlogbot_3 has quit IRC03:31
*** altlogbot_1 has joined #openstack-lbaas03:32
*** yamamoto has quit IRC03:32
*** yamamoto has joined #openstack-lbaas03:50
*** sapd1_x has joined #openstack-lbaas03:54
*** ivve has quit IRC04:12
*** yamamoto has quit IRC04:32
*** sapd1_x has quit IRC04:41
*** vishalmanchanda has joined #openstack-lbaas04:49
*** yamamoto has joined #openstack-lbaas04:55
*** yamamoto has quit IRC04:56
*** yamamoto has joined #openstack-lbaas04:58
*** ivve has joined #openstack-lbaas05:15
*** sapd1_x has joined #openstack-lbaas05:25
*** yamamoto has quit IRC05:26
*** ccamposr has joined #openstack-lbaas05:44
*** sapd1_x has quit IRC05:53
*** sapd1_x has joined #openstack-lbaas05:55
*** mkuf has quit IRC05:55
*** yamamoto has joined #openstack-lbaas05:58
*** yamamoto has quit IRC06:11
*** mkuf has joined #openstack-lbaas06:30
*** gcheresh has joined #openstack-lbaas06:41
*** yamamoto has joined #openstack-lbaas06:43
*** gthiemonge has quit IRC06:56
*** yamamoto has quit IRC06:59
*** tesseract-RH has joined #openstack-lbaas07:05
*** AlexStaf has joined #openstack-lbaas07:12
*** altlogbot_1 has quit IRC07:13
*** yamamoto has joined #openstack-lbaas07:15
*** altlogbot_1 has joined #openstack-lbaas07:18
*** yamamoto has quit IRC07:21
openstackgerritOpenStack Proposal Bot proposed openstack/neutron-lbaas-dashboard master: Imported Translations from Zanata  https://review.opendev.org/65732807:34
*** dmellado has quit IRC07:44
*** dmellado has joined #openstack-lbaas07:44
*** nmagnezi has quit IRC07:44
*** yamamoto has joined #openstack-lbaas08:00
*** gthiemonge has joined #openstack-lbaas08:06
*** yamamoto has quit IRC08:07
*** pcaruana has joined #openstack-lbaas08:24
*** rpittau|afk is now known as rpittau08:24
openstackgerritGregory Thiemonge proposed openstack/octavia-tempest-plugin master: WIP Add UDP test scenario  https://review.opendev.org/65651508:39
openstackgerritOpenStack Proposal Bot proposed openstack/octavia-dashboard master: Imported Translations from Zanata  https://review.opendev.org/65734608:40
*** sapd1_x has quit IRC08:40
*** ivve has quit IRC08:40
*** yamamoto has joined #openstack-lbaas08:45
*** ivve has joined #openstack-lbaas08:55
*** mkuf_ has joined #openstack-lbaas08:58
*** mkuf has quit IRC09:02
*** mkuf_ has quit IRC09:10
*** yamamoto has quit IRC09:10
*** altlogbot_1 has quit IRC09:11
*** yamamoto has joined #openstack-lbaas09:11
*** altlogbot_2 has joined #openstack-lbaas09:15
*** altlogbot_2 has quit IRC09:15
*** altlogbot_0 has joined #openstack-lbaas09:16
*** yamamoto has quit IRC09:16
*** yamamoto has joined #openstack-lbaas09:19
*** yamamoto has quit IRC09:19
*** mkuf_ has joined #openstack-lbaas09:38
*** yamamoto has joined #openstack-lbaas09:56
*** yamamoto has quit IRC10:08
*** takamatsu has joined #openstack-lbaas10:36
openstackgerritsahid proposed openstack/octavia master: db: add missing primary key in spares_pool table  https://review.opendev.org/65645910:40
openstackgerritsahid proposed openstack/octavia master: db: add missing primary key in spares_pool table  https://review.opendev.org/65645910:41
*** yboaron has joined #openstack-lbaas11:08
*** yboaron_ has joined #openstack-lbaas11:18
*** altlogbot_0 has quit IRC11:19
*** yboaron has quit IRC11:20
*** altlogbot_3 has joined #openstack-lbaas11:21
*** henriqueof has joined #openstack-lbaas11:44
*** yboaron_ has quit IRC12:07
*** boden has joined #openstack-lbaas12:10
*** altlogbot_3 has quit IRC12:14
*** altlogbot_3 has joined #openstack-lbaas12:20
*** pcaruana has quit IRC12:29
*** happyhemant has joined #openstack-lbaas12:41
*** yamamoto has joined #openstack-lbaas12:49
*** gthiemon1e has joined #openstack-lbaas12:59
*** gthiemonge has quit IRC13:00
*** altlogbot_3 has quit IRC13:15
*** altlogbot_0 has joined #openstack-lbaas13:16
*** pcaruana has joined #openstack-lbaas13:16
*** altlogbot_0 has quit IRC13:21
*** altlogbot_2 has joined #openstack-lbaas13:24
*** altlogbot_2 has quit IRC13:25
*** altlogbot_3 has joined #openstack-lbaas13:26
*** psachin has quit IRC13:28
*** yamamoto has quit IRC13:31
*** yboaron_ has joined #openstack-lbaas13:32
*** gthiemonge has joined #openstack-lbaas13:35
*** gthiemon1e has quit IRC13:36
*** yamamoto has joined #openstack-lbaas13:47
*** ianychoi has quit IRC13:48
*** ianychoi has joined #openstack-lbaas13:52
*** ianychoi has quit IRC14:04
*** Vorrtex has joined #openstack-lbaas14:14
*** ianychoi has joined #openstack-lbaas14:18
*** ivve has quit IRC14:40
*** gcheresh has quit IRC14:56
*** AlexStaf has quit IRC15:16
*** gthiemonge has quit IRC15:16
*** yamamoto has quit IRC15:16
*** ccamposr has quit IRC15:16
*** rpittau is now known as rpittau|afk15:43
*** henriqueof has quit IRC15:53
*** yamamoto has joined #openstack-lbaas15:57
*** yamamoto has quit IRC16:10
*** ivve has joined #openstack-lbaas16:20
*** ivve has quit IRC16:35
*** henriqueof has joined #openstack-lbaas17:07
*** vishalmanchanda has quit IRC17:08
*** ivve has joined #openstack-lbaas17:18
*** tesseract-RH has quit IRC17:24
*** Swami has joined #openstack-lbaas17:29
johnsomHey, nova confirmed the port detach bug, so progress18:09
openstackgerritMerged openstack/octavia-dashboard master: Imported Translations from Zanata  https://review.opendev.org/65734618:12
rm_workyeah :P18:20
rm_workand the ipv6 multinode thing seems fixed, so we can merge that ^_^18:20
rm_workjohnsom: meanwhile i'm getting stuck on the to_data_model object graph mess18:21
johnsomHmmm, for the v1 retirement work?18:23
johnsomI am just now digging out of my e-mail backlog...18:23
johnsom+2'd18:24
*** yboaron_ has quit IRC18:28
*** yboaron_ has joined #openstack-lbaas18:28
rm_workno, for multi-vip18:28
rm_worki started with a TDD approach, wrote the unit test for the api call -- got the model/db work done, and some of the controller work...18:29
rm_workbut stuck on the stupid graph stuff because i must have missed something and it's just returning a graph with the correct NUMBER of vips but all the same one18:29
* rm_work goes back to debugging18:30
johnsomlol18:30
rm_workoh, so yes, the v1 retirement patch is "good to merge" imo -- it gets all the big stuff, what's left would be unwinding some of the crazy compatability code we added to the v2 types or maybe stuff like the data model graph, but that can happen later18:31
rm_workor rather, at least "good to review"18:31
*** KeithMnemonic has joined #openstack-lbaas18:39
rm_workjohnsom: does this look right? http://paste.openstack.org/show/750730/18:44
*** happyhemant has quit IRC18:48
johnsomDo you really want to mess with what is already there?18:57
rm_workerr... kinda have to19:00
rm_workright now the vip table is unique per lb_id19:00
rm_workand to retain which subnet is returned in the "main" vip slot, we need to track that19:01
rm_workwhich are the only two changes19:01
rm_workunless i made another "additional_vip" table which seems ... odd19:02
rm_workor rather, not odd, but ... unnecessary19:02
rm_workthat's just replacing the current "primary key" unique constraint with a compound key of "unique per ip+subnet+lb combo"19:03
rm_workand adding one additional field, with an obvious default value19:04
rm_workahhh forgot to update `_get_unique_key` to deal with the new compound key19:14
openstackgerritMerged openstack/octavia-dashboard master: Changes for url transition to OpenDev  https://review.opendev.org/65446119:25
*** mkuf has joined #openstack-lbaas19:32
*** boden_ has joined #openstack-lbaas19:32
*** boden has quit IRC19:39
*** mkuf_ has quit IRC19:39
*** yboaron_ has quit IRC19:39
*** Swami has quit IRC19:39
*** Vorrtex has quit IRC19:39
*** yboaron_ has joined #openstack-lbaas19:40
*** Swami has joined #openstack-lbaas19:40
*** takamatsu has quit IRC19:51
*** gcheresh has joined #openstack-lbaas19:57
openstackgerritMerged openstack/octavia-dashboard stable/stein: Imported Translations from Zanata  https://review.opendev.org/65674920:05
NobodyCamgood afternoon Folks; is there a way to delete lb's in PENDING_CREATE state?20:20
NobodyCam`Invalid state PENDING_CREATE of loadbalancer resource 7a5622a4-9190-415d-a347-ff2aef3ad2d6 (HTTP 409) (Request-ID: req-9c6d4970-e032-49f7-b0c8-c7fc0b76758d)`20:20
rm_workjohnsom: all of the conversions back and forth between provider/db dicts/objs is insane20:24
rm_workit's hard to even track which format we're in at any given point20:25
rm_workand i swear some of these seem redundant20:25
rm_workbut i'm assuming it's all necessary...20:25
johnsomrm_work: completely agree. Not 100% sure it is ALL necessary though.20:26
johnsomGood thing is the jobboard work will likely clean it up a bit20:26
johnsomNobodyCam: Having rabbit issues?20:27
NobodyCamno20:29
colin-we get that with cloud-provider-openstack created loadbalancers from time to time NobodyCam20:29
colin-still trying to track down why...20:29
NobodyCamI updated mysql to error and was able to delete20:30
colin-brave man :)20:30
colin-s/man/operator/20:30
NobodyCamhehehe20:30
NobodyCamyea we are running magnum20:31
*** Vorrtex has joined #openstack-lbaas20:32
NobodyCamyea we are running magnum20:34
NobodyCamcolin-: how do you deal with such issues?20:34
*** gcheresh has quit IRC20:35
colin-presently we do what you just did, i haven't been able to determine exactly what causes it yet in order to stop it happening20:35
NobodyCam:)20:36
NobodyCamyea last think I see in the log is `Sending create Load Balancer 702673de-8960-48b3-b98f-3aa9c112a515 to provider amphora`20:38
NobodyCams/think/thing/20:39
johnsomThat implies a rabbit queue issue, if you didn’t see it in the worker logs20:42
NobodyCamhumm20:42
NobodyCamoh I do see `<<"/octavia">>, [{<<"connection.blocked">>,bool,true},{<<"authentication_failure_close">>,bool,true}`20:47
NobodyCamthat could be something20:47
*** pcaruana has quit IRC20:51
*** Vorrtex has quit IRC20:51
lxkongrm_work, johnsom, could you please take a look at https://storyboard.openstack.org/#!/story/2005578? not sure it's only affecting us.20:52
johnsomlxkong That is an odd one. We of course run mysql in the gate jobs and the migrations have been fine.20:53
*** imdigitaljim has joined #openstack-lbaas20:53
rm_workyep, that's what i was going to say20:54
*** schaney has joined #openstack-lbaas20:54
rm_workthose are all very well tested20:54
lxkongyeah, i looked at the ci log yesterday, didn't find something suspicious20:55
lxkongnot sure if the mariadb version matters20:55
johnsomThe other issue with that table, that there is no primary key, I understand. That is something that is specific requirement for the Percona engine.20:56
lxkongwe are using the same version in production, that failure blocks us from upgrading to stein :-(20:56
johnsomI wonder if it's a sqlalchemy version issue really, but it might be mariadb20:57
lxkongok, i will dig a little bit further today20:58
johnsomAt least mysql has it in 5.5: https://dev.mysql.com/doc/refman/5.5/en/date-and-time-functions.html#function_current-timestamp20:58
openstackgerritMerged openstack/octavia-tempest-plugin master: Fix ipv6 multinode gate (use private subnet)  https://review.opendev.org/65685221:02
johnsommariadb seems to have it, but they don't say what version they included it in: https://mariadb.com/kb/en/library/current_timestamp/21:02
lxkongi can see the devstack is installing mysql-server-5.721:03
johnsomlxkong https://mariadb.com/kb/en/library/mariadb-1001-release-notes/#new-features-re-implemented-from-a-similar-mysql-feature21:03
johnsomThat implies that mariadb didn't implement it until 10.0.121:04
lxkongjohnsom:  hmm...yeah21:05
johnsomSo... Not sure what to tell you there. OpenStack doesn't have a good story about DB compatibility. Really sqlalchemy should "do the right thing" but....21:06
rm_workjohnsom: ugh, speaking of DB issues... I may have run into one that's going to kill my plan21:07
rm_workto do this the way i was planning, i realized I need to add an `id` field and make that the real primary-key for the `vip` table ... and remove the existing unique constraint on `load_balancer_id`... which makes that table no longer backwards compatible with older controller versions T_T21:09
rm_workand i don't know if there's a way around that with this approach21:09
rm_worki may have to make another table :/21:09
rm_workwhich is super gross, I think. but I'll give it some thought21:10
johnsomYeah, I think you are in "another table" space.  That is what I commented earlier today.21:10
rm_workyeah i thought i might be ok but needing an id field kills it for sure21:10
rm_worki think i see what you meant earlier though, changing that constraint might already have sunk me in the same way21:11
rm_workwhelp21:11
lxkongis it possible we use oslo_db.sqlalchemy.models.TimestampMixin which has `updated_at = Column(DateTime, onupdate=lambda: timeutils.utcnow())` instead of specify the `server_default` in the migration script?21:11
johnsomlxkong There are easier ways to do it than that, but it shifts the time from the DB time to the python time.21:13
johnsomlxkong Also FYI, that keyword was in mysql 5.1 as well: http://download.nust.na/pub6/mysql/doc/refman/5.1/en/date-and-time-functions.html#function_current-timestamp21:13
johnsomSo, mariadb seems to be significantly out of sync21:13
lxkongi think the problem is not mariadb doesn't support CURRENT_TIMESTAMP, but doesn't support CURRENT_TIMESTAMP for DateTime column21:15
*** henriqueof has quit IRC21:16
lxkongif i change DATETIME to TIMESTAMP, it works21:17
johnsomYeah, ok, so this is really a sqlalchemy bug21:17
lxkonghttps://www.irccloud.com/pastebin/2f1zl7tg/21:17
johnsomWe call: sa.Column(u'updated_at', sa.DateTime(), nullable=True,            server_default=sa.func.current_timestamp()))21:17
johnsomSo sqlalchemy is generating invalid SQL for mariadb....21:18
lxkongbecause the code tells sqlalchemy the type is DateTime instead of TIMESTAMP21:19
lxkonghttps://docs.sqlalchemy.org/en/13/core/type_basics.html#sqlalchemy.types.TIMESTAMP21:19
johnsomI mean, sqlalchemy even gives this as an example: https://docs.sqlalchemy.org/en/13/core/defaults.html#client-invoked-sql-expressions21:20
johnsomMy guess is sqlalchemy should be outputing NOW() instead of CURRENT_TIMESTAMP21:20
lxkongthat example is using `onupdate `21:20
lxkonginstead of DEFAULT21:21
lxkongthat's different21:21
johnsomYeah, but it's still a DateTime column21:21
johnsomhttps://docs.sqlalchemy.org/en/13/core/functions.html#sqlalchemy.sql.functions.current_timestamp21:21
johnsomIt also calls out type of DateTime here21:21
lxkongjohnsom: it's too far away for us to fix sqlalchemy or upgrade mariadb at the moment21:25
lxkongfrom octavia's perspective, is it safe if we change DateTime to TIMESTAMP?21:25
lxkongfor spares_pool table?21:26
johnsomNo21:26
johnsomI would instead cook up a fix to the migration script, such as using datetime.datetime.now()21:27
johnsomRight now we aren't really looking at the value of the column, but we may in the future and I'm not sure that TIMESTAMP is fully compatible from a query perspective.21:28
lxkongyeah, fare enough21:28
lxkonglet me do some tests based on your suggestion21:28
johnsomlxkong Actually, we found that the server default doesn't even work.21:30
johnsomlxkong https://review.opendev.org/#/c/649766/1/octavia/db/migration/alembic_migrations/versions/46d914b2a5e5_seed_the_spares_pool_table.py21:30
johnsomWe changed it in the next patch, so just remove the whole server default section and call it a day.21:30
lxkongsigh...21:31
lxkongthanks johnsom21:32
johnsomlxkong I would argue to merge it if you push a patch up that does that.21:32
johnsomplus backport it21:32
lxkongjohnsom: we probably have to carry a private patch by ourselves21:33
johnsomlxkong I'm just saying, if you put up a patch that removes the "server_default=sa.func.current_timestamp()" from 6ffc710674ef_spares_pool_table.py I would support merging it and backporting it to fix this upstream for you.21:33
johnsomI think we can make a good case21:34
johnsomSince it effectively does nothing21:34
lxkongjohnsom: that would be great. I will test first before proposing a patch21:34
johnsomlxkong +121:35
* lxkong goes to stand up meeting and tell a story 21:35
*** boden_ has quit IRC21:45
*** yboaron_ has quit IRC21:54
openstackgerritLingxian Kong proposed openstack/octavia master: Remove the server_default value for spares_pool table creation  https://review.opendev.org/65746222:14
*** rcernin has joined #openstack-lbaas22:48
« Sunday, 2019-05-05 Index Tuesday, 2019-05-07 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!