| *** itlinux has joined #openstack-kolla | 00:48 | |
| *** keper7 has quit IRC | 01:11 | |
| *** dpawlik has joined #openstack-kolla | 01:39 | |
| *** dpawlik has quit IRC | 01:43 | |
| *** itlinux is now known as itlinux-away | 01:45 | |
| *** openstackgerrit has quit IRC | 02:37 | |
| *** k_mouza has joined #openstack-kolla | 02:46 | |
| *** k_mouza has quit IRC | 02:50 | |
| *** dpawlik has joined #openstack-kolla | 03:40 | |
| *** dpawlik has quit IRC | 03:44 | |
| *** gkadam has joined #openstack-kolla | 03:50 | |
| *** gkadam has quit IRC | 03:50 | |
| *** dave-mccowan has quit IRC | 04:41 | |
| *** skramaja has joined #openstack-kolla | 04:49 | |
| *** Luzi has joined #openstack-kolla | 05:05 | |
| *** shyamb has joined #openstack-kolla | 05:11 | |
| *** shyamb has quit IRC | 05:30 | |
| *** openstackgerrit has joined #openstack-kolla | 05:34 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS https://review.opendev.org/675614 | 05:34 |
|---|---|---|
| *** itlinux has joined #openstack-kolla | 05:35 | |
| *** factor has joined #openstack-kolla | 05:51 | |
| *** shyamb has joined #openstack-kolla | 05:52 | |
| *** dpawlik has joined #openstack-kolla | 06:00 | |
| *** dr_feelgood has joined #openstack-kolla | 06:11 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 06:13 |
| *** itlinux is now known as itlinux-away | 06:15 | |
| *** hamdyk has joined #openstack-kolla | 06:18 | |
| yoctozepto | morning channel | 06:31 |
| yoctozepto | mgoddard, mnasiadka, hrw: I have an announcement to make - I will be (much) less available tomorrow and during the upcoming week due to being assigned the role of a co-moderator of a big data conference/workshop in Turkey. This is very recent news because I learned only yesterday's late evening that I'm traveling tomorrow ;-) (hail the bureaucracy :-) ) | 06:39 |
| *** sm806 has quit IRC | 06:41 | |
| *** shyamb has quit IRC | 06:43 | |
| *** gfidente has joined #openstack-kolla | 06:45 | |
| *** shyamb has joined #openstack-kolla | 06:47 | |
| *** dougsz has joined #openstack-kolla | 06:48 | |
| *** dr_feelgood has quit IRC | 06:49 | |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Minimal support for deploying multiple instances of MariaDB https://review.opendev.org/619756 | 06:50 |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Copy Nova role as a basis for the Nova cell role https://review.opendev.org/675658 | 06:50 |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: [WIP] Factor out nova-cell role https://review.opendev.org/675659 | 06:50 |
| mnasiadka | yoctozepto: have fun :) Turkey? Huh | 06:50 |
| *** dr_feelgood has joined #openstack-kolla | 06:51 | |
| *** dr_feelgood has quit IRC | 06:53 | |
| *** dr_feelgood has joined #openstack-kolla | 06:56 | |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: [WIP] Factor out nova-cell role https://review.opendev.org/675659 | 07:02 |
| yoctozepto | mnasiadka: yup, Istanbul to be exact ;-) | 07:09 |
| *** shyamb has quit IRC | 07:13 | |
| *** shyamb has joined #openstack-kolla | 07:20 | |
| *** ivve has joined #openstack-kolla | 07:22 | |
| mnasiadka | yoctozepto: what is irritatingly funny - I'm hitting the neutron bug only on ubuntu-source :) | 07:27 |
| *** jbadiapa has joined #openstack-kolla | 07:28 | |
| *** dr_feelgood has quit IRC | 07:31 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Modernize a way of configuring Docker daemon https://review.opendev.org/672700 | 07:32 |
| yoctozepto | mnasiadka: yeah, this is odd | 07:32 |
| yoctozepto | both images are rebuilt | 07:32 |
| *** dr_feelgood has joined #openstack-kolla | 07:32 | |
| yoctozepto | as if the difference in mysql version was the culprit | 07:32 |
| yoctozepto | i.e. neutron doing nothing bad, only mariadb being broken | 07:33 |
| mnasiadka | well, 10.1 in ubuntu vs 10.3 on centos | 07:35 |
| yoctozepto | yup but they did not update recently | 07:36 |
| yoctozepto | so might be neutron did something incompatible with 10.1 | 07:36 |
| yoctozepto | mnasiadka: I see you there on #neutron | 07:37 |
| yoctozepto | maybe tell them what we have found | 07:37 |
| mnasiadka | I'll talk with a fellow core :) | 07:37 |
| yoctozepto | mnasiadka: ok, who's that? | 07:37 |
| mnasiadka | yoctozepto: another polish guy :) | 07:38 |
| yoctozepto | mnasiadka: PL FTW :D | 07:38 |
| mnasiadka | yoctozepto: well, we have 3 polish cores here :) | 07:38 |
| yoctozepto | PL can into cloud | 07:38 |
| yoctozepto | ;D | 07:38 |
| *** dougsz has quit IRC | 07:40 | |
| mnasiadka | see? doug got offended ;) | 07:42 |
| yoctozepto | mnasiadka: hmm? where? | 07:43 |
| mnasiadka | he quit ;) | 07:43 |
| yoctozepto | ah | 07:43 |
| yoctozepto | I have these filtered out | 07:43 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: DNM: Test Zun now https://review.opendev.org/679195 | 07:43 |
| *** dougsz has joined #openstack-kolla | 07:43 | |
| mnasiadka | yoctozepto: https://review.opendev.org/#/c/677221/ | 07:49 |
| mnasiadka | yoctozepto: this is the change that introduced that db migration, but all checks passed on postgres and mysql 5.x | 07:49 |
| mnasiadka | so it looks like a mariadb bug in 10.1... | 07:49 |
| mnasiadka | I don't like that we have a core service with very old version (but still old-stable-ish) :D | 07:50 |
| mnasiadka | let's wait for mgoddard - he was doing a lot of stuff with mariadb :) | 07:52 |
| *** dr_feelgood has quit IRC | 07:52 | |
| *** shyamb has quit IRC | 07:53 | |
| *** dr_feelgood has joined #openstack-kolla | 07:53 | |
| *** dougsz has quit IRC | 07:53 | |
| *** priteau has joined #openstack-kolla | 07:54 | |
| yoctozepto | mnasiadka: yeah, let's put it on him ;-) | 08:04 |
| yoctozepto | oh well, we might want to use upstream mariadb then :-) | 08:08 |
| yoctozepto | mgoddard gonna love it | 08:08 |
| mgoddard | morning | 08:13 |
| mgoddard | new proposal: deprecate mariadb | 08:16 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Modernize the way of configuring Docker daemon https://review.opendev.org/672700 | 08:16 |
| mgoddard | can we not just treat this as a neutron bug? | 08:16 |
| yoctozepto | mgoddard: well, neutrinos seem happy about what they did :D\ | 08:17 |
| yoctozepto | and it works on centos and some old mysql | 08:18 |
| yoctozepto | looks like regresion in the one shipped with ubuntu | 08:18 |
| yoctozepto | ;/ | 08:18 |
| mgoddard | 19.04 has 10.3 | 08:18 |
| yoctozepto | ubuntu bionic* 18.04 | 08:19 |
| mgoddard | yeah, just saying | 08:19 |
| yoctozepto | ok | 08:19 |
| mgoddard | we normally just test LTS | 08:19 |
| yoctozepto | because we want sanity | 08:19 |
| mgoddard | where possible | 08:20 |
| mnasiadka | mgoddard: so it seems devstack uses mysql 5.7 instead of mariadb | 08:21 |
| mnasiadka | mgoddard: and probably that's the main issue :) | 08:21 |
| mgoddard | yeah | 08:21 |
| mnasiadka | mgoddard: but I asked Slawek Kaplonski (a neutron core) to try to change it together with the QA team | 08:21 |
| mgoddard | isn't he PTL? | 08:22 |
| mnasiadka | At least not yet :) | 08:22 |
| mgoddard | sounds like a way to start a long discussion on the ML | 08:22 |
| mgoddard | but not change it :) | 08:22 |
| mgoddard | unless we could find a majority of deployment tools are using mariadb | 08:23 |
| mgoddard | is there a bug for this issue? | 08:23 |
| mgoddard | we could create in k-a and neutron | 08:24 |
| yoctozepto | mgoddard: mariadb mostly took over mysql | 08:27 |
| yoctozepto | but testing on 10.3 would bring no insight | 08:27 |
| yoctozepto | as only 10.1 on ubuntu is affected so far | 08:27 |
| yoctozepto | could even be by some misapplied ubuntu team patch | 08:27 |
| yoctozepto | who knows | 08:27 |
| yoctozepto | or well, it could be alembic which improperly translates the migration into sql | 08:29 |
| yoctozepto | not detecting that mariadb 10.1 needs another approach | 08:30 |
| yoctozepto | many questions, few answers | 08:30 |
| mgoddard | indeed | 08:34 |
| mgoddard | ultimately we'r surely not the only users running mariadb on bionic, so I think this falls to the neutron team | 08:37 |
| yoctozepto | mgoddard: mhm, let's make the train release most unfortunate for ubuntu users and keep our CI broken | 08:38 |
| yoctozepto | ubuntu unvote? | 08:38 |
| yoctozepto | mnasiadka, mgoddard | 08:38 |
| mgoddard | fine by me :p | 08:38 |
| mnasiadka | other option is to use mariadb repo on ubuntu.... | 08:39 |
| mgoddard | I wonder if there's some manual SQL we could fudge in there | 08:39 |
| mnasiadka | making ubuntu unvoting will make as break it even more in other departments than neutron :) | 08:40 |
| mgoddard | I guess this breaks kolla and kolla-ansible? | 08:41 |
| mnasiadka | yeah | 08:42 |
| yoctozepto | <mnasiadka> other option is to use mariadb repo on ubuntu.... | 08:44 |
| yoctozepto | please propose | 08:44 |
| yoctozepto | we could test that | 08:44 |
| mnasiadka | yeah, will do | 08:45 |
| yoctozepto | thanks mnasiadka | 08:45 |
| mnasiadka | do we want to match version with CentOS? | 08:45 |
| mnasiadka | the latest stable is 10.4, 10.3 is called old-stable :D | 08:45 |
| mgoddard | just asking in OSA | 08:45 |
| mgoddard | looks like they also use mariadb on ubuntu | 08:45 |
| yoctozepto | mgoddard: ok, what did you write (logs are not there yet and I was late?) | 08:46 |
| yoctozepto | ah, you only began | 08:46 |
| yoctozepto | mnasiadka: no need I believe | 08:46 |
| yoctozepto | we could just turn our heads to upstream | 08:47 |
| mgoddard | yoctozepto: are you hit by a CI failure during neutron DB migration? We're seeing it in kolla | 08:47 |
| yoctozepto | mgoddard: thanks | 08:47 |
| yoctozepto | so they freeze | 08:48 |
| mgoddard | we seem to flip between distro and upstream packages :) | 08:48 |
| yoctozepto | mgoddard: yeah, noticed that ;-) | 08:48 |
| yoctozepto | so we wait two weeks till osa broken | 08:48 |
| yoctozepto | then neutron fixes it | 08:48 |
| yoctozepto | me gusta | 08:49 |
| *** shyamb has joined #openstack-kolla | 08:49 | |
| mgoddard | we could freeze :) | 08:50 |
| mgoddard | better solution than non-voting | 08:50 |
| mnasiadka | mgoddard: seems they use mariadb repo and 10.3 - https://github.com/openstack/openstack-ansible-galera_server/blob/master/defaults/main.yml#L44 | 08:51 |
| mnasiadka | and on 10.3 it works | 08:51 |
| yoctozepto | haha, so neutron never gonna fix it | 08:51 |
| yoctozepto | we alone | 08:51 |
| mnasiadka | well, it's us and tripleo | 08:51 |
| mnasiadka | :) | 08:52 |
| yoctozepto | mnasiadka: they are centos | 08:52 |
| mnasiadka | ah | 08:52 |
| mnasiadka | so they are fine | 08:52 |
| yoctozepto | ;-) | 08:52 |
| mnasiadka | so it's again our own hellhole | 08:52 |
| yoctozepto | :D | 08:52 |
| mnasiadka | ok, so I'll try to add the mariadb repo to ubuntu and let's see | 08:52 |
| mgoddard | here's the error: Cannot change column 'network_id': used in a foreign key constraint 'subnets_ibfk_1' | 08:52 |
| mnasiadka | mgoddard: https://review.opendev.org/#/c/677221/ | 08:52 |
| mnasiadka | here is the change that introduced this | 08:53 |
| mnasiadka | happily tested on postgres and mysql 5.7 :) | 08:53 |
| yoctozepto | they are revmoving nullability | 08:53 |
| yoctozepto | should work just fine | 08:53 |
| yoctozepto | for the fk | 08:53 |
| yoctozepto | unless the sql does something awry | 08:53 |
| mnasiadka | works fine on mariadb 10.3 :) | 08:53 |
| yoctozepto | or rbdms broke ;p | 08:53 |
| yoctozepto | yeah, but see alembic translates it based on target server | 08:54 |
| mgoddard | well I expect OSA will hit it eventually | 08:54 |
| mgoddard | what if we temporarily remove the foreign key constraint during the migration? | 08:54 |
| mgoddard | that could be the workaround in neutron | 08:54 |
| yoctozepto | mgoddard: they use 10.3 | 08:54 |
| yoctozepto | they will not hit it ;-) | 08:54 |
| yoctozepto | mnasiadka linked https://github.com/openstack/openstack-ansible-galera_server/blob/master/defaults/main.yml#L44 | 08:55 |
| mgoddard | yeah, looks like they're using upstream | 08:56 |
| yoctozepto | so not affected by ubuntu masters | 08:57 |
| *** gfidente has quit IRC | 08:57 | |
| yoctozepto | and using higher ver | 08:57 |
| yoctozepto | so we ARE alone | 08:57 |
| yoctozepto | align with osa then? | 08:58 |
| yoctozepto | mgoddard, mnasiadka ^ | 08:58 |
| yoctozepto | mnasiadka: preparing patch already? ;D | 08:58 |
| mnasiadka | yeah, started doing the change... | 08:58 |
| yoctozepto | fcuk yeah | 08:58 |
| mgoddard | more mariadb upgrades :( | 08:59 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Modernize the way of configuring Docker daemon https://review.opendev.org/672700 | 08:59 |
| yoctozepto | xD | 08:59 |
| mgoddard | before we charge into this, we're not the only people in the world using ubuntu's distro mariadb | 09:00 |
| mgoddard | I think neutron needs to handle it | 09:00 |
| yoctozepto | or ubuntu | 09:00 |
| yoctozepto | or alembic | 09:00 |
| mgoddard | what are they going to say in the bug report? | 09:00 |
| yoctozepto | or mariadb | 09:00 |
| yoctozepto | "it works for me" | 09:00 |
| mgoddard | oh, you need to upgrade your database | 09:00 |
| mgoddard | it doesn't work for me if you follow the steps to reproduce | 09:01 |
| mgoddard | you can't say it works for me if you do something different! | 09:01 |
| mgoddard | I'll create a bug report | 09:04 |
| mgoddard | and pin neutron | 09:04 |
| *** k_mouza has joined #openstack-kolla | 09:04 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: Move to MariaDB upstream repo on Ubuntu https://review.opendev.org/679209 | 09:07 |
| mnasiadka | mgoddard: I think it's a bigger issue, if all devstack are using postgres or mysql 5.7 | 09:07 |
| mnasiadka | and mariadb 10.1 is let's say the most non-compatible with mysql 5.7 (10.3 is more compatible) | 09:08 |
| mnasiadka | most non-compatible from stable branches :) | 09:08 |
| yoctozepto | well, neutrinos did nothing inherently wrong in that change | 09:10 |
| yoctozepto | I identified 3 vectors of fault so far | 09:10 |
| yoctozepto | could be more | 09:10 |
| yoctozepto | what would you do in their place ;p | 09:11 |
| mnasiadka | so maybe it's an sqlalchemy bug? :D | 09:11 |
| yoctozepto | mnasiadka: that's alembic I mentioned ;-) | 09:11 |
| yoctozepto | ubuntu/alembic/mariadb fault | 09:12 |
| yoctozepto | someone needs to handle it for sure | 09:12 |
| mgoddard | presumably we have the same version of alembic on ubuntu and centos in source images? | 09:14 |
| yoctozepto | mgoddard: but it could produce bad sql (not checked, only guessing) for 10.1 specifically | 09:16 |
| mnasiadka | the only way to verify it is to downgrade MariaDB on CentOS to 10.1 :D | 09:17 |
| yoctozepto | mnasiadka: then you have either mariadb or alembic fault | 09:18 |
| mnasiadka | well again we get shot in the head by using different versions of core packages :) | 09:19 |
| yoctozepto | DROP CONSTRAINT for UNIQUE and FOREIGN KEY constraints was introduced in MariaDB 10.2.22 and MariaDB 10.3.13. | 09:21 |
| yoctozepto | well, that is late | 09:21 |
| yoctozepto | it could be alembic can't handle lack of lifting the constraint for example | 09:22 |
| mnasiadka | uhh | 09:23 |
| yoctozepto | but if train is meant to be tested on mysql 5.7+ | 09:23 |
| yoctozepto | then bionic is outdated | 09:23 |
| yoctozepto | mnasiadka, mgoddard: guys, I really think we should follow in the steps of osa and upstreamify mariadb | 09:24 |
| yoctozepto | maybe it makes upgrades easier/more consistent | 09:25 |
| yoctozepto | as well | 09:25 |
| mnasiadka | yeah, at least we should fail everywhere :) | 09:25 |
| mnasiadka | (if we fail) | 09:25 |
| yoctozepto | (not the first one though ;p) | 09:25 |
| yoctozepto | mnasiadka: and be coordinated with osa ;p | 09:25 |
| mgoddard | well we go back and forth on this. Upstream packages don't always work as well as distro packages | 09:26 |
| mgoddard | and remember how much time we ploughed into the last upgrade | 09:27 |
| mgoddard | so let's not rush into that | 09:27 |
| mgoddard | https://bugs.launchpad.net/kolla/+bug/1841907 | 09:28 |
| openstack | Launchpad bug 1841907 in kolla-ansible train "Neutron bootstrap failing on Ubuntu bionic with Cannot change column 'network_id" [Critical,New] | 09:28 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Workaround: Pin neutron source to avoid migration issue https://review.opendev.org/679213 | 09:31 |
| mgoddard | trying that ^ | 09:34 |
| mnasiadka | yeah, that might be the fastest workaround | 09:35 |
| mgoddard | until binary packages get updated :) | 09:36 |
| mnasiadka | yeah... | 09:36 |
| yoctozepto | binaries are nonvoting | 09:38 |
| yoctozepto | no tears shed ;-) | 09:38 |
| *** gfidente has joined #openstack-kolla | 09:39 | |
| *** shyamb has quit IRC | 09:40 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 09:40 |
| openstackgerrit | Will Szumski proposed openstack/kolla-ansible master: Set my_ip in ironic.conf https://review.opendev.org/679215 | 09:41 |
| yoctozepto | so in centos rdo bumps up mariadb | 09:43 |
| yoctozepto | seems we are not getting the same comfort in ubuntu | 09:44 |
| *** ivve has quit IRC | 09:44 | |
| *** ivve has joined #openstack-kolla | 09:45 | |
| mgoddard | RDO: for your comfort | 09:46 |
| *** shyamb has joined #openstack-kolla | 09:47 | |
| yoctozepto | and sanity | 09:49 |
| yoctozepto | (most of the time) | 09:49 |
| mnasiadka | and insanity when they release and bump up ceph | 09:51 |
| mnasiadka | :) | 09:51 |
| yoctozepto | mnasiadka: deprecate ceph-nfs = profit | 09:53 |
| *** shyamb has quit IRC | 10:03 | |
| mnasiadka | bumping up MariaDB to 10.3 on Ubuntu helped... | 10:04 |
| mgoddard | it passed the migration? | 10:04 |
| mnasiadka | ah sorry, it was the build job, the k-a job is still deploying (but not failed yet) | 10:06 |
| openstackgerrit | Will Szumski proposed openstack/kolla-ansible master: Use secure websocket for nova serial console proxy when TLS enabled https://review.opendev.org/679222 | 10:06 |
| *** radeks has joined #openstack-kolla | 10:09 | |
| *** shyamb has joined #openstack-kolla | 10:11 | |
| openstackgerrit | Pierre Riteau proposed openstack/kolla-ansible master: Remove Heat environment file disabling deprecated plugins https://review.opendev.org/674867 | 10:13 |
| *** radeks_ has joined #openstack-kolla | 10:14 | |
| *** radeks has quit IRC | 10:16 | |
| *** dpawlik has quit IRC | 10:19 | |
| mnasiadka | https://www.irccloud.com/pastebin/lKNfaCrW/ | 10:19 |
| mnasiadka | mgoddard: ring any bell? | 10:20 |
| mnasiadka | you were doing the move to mariabackup for SST? | 10:20 |
| mgoddard | mnasiadka: reminds me of a month of staring at those logs | 10:20 |
| mgoddard | is it a fatal error? | 10:21 |
| mgoddard | unrecoverable suggests yes | 10:21 |
| mgoddard | which job is it from? | 10:22 |
| mgoddard | deploy or upgrade? | 10:22 |
| mnasiadka | deploy | 10:22 |
| mgoddard | hmm | 10:22 |
| mnasiadka | hmm, we don't install mariadb-backup | 10:22 |
| mgoddard | nothing comes to mind | 10:22 |
| mgoddard | is xtrabackup installed? | 10:23 |
| mgoddard | could be 10.1 was using that? | 10:23 |
| mgoddard | looks like k-a always uses mariabackup now | 10:23 |
| mnasiadka | yeah, let me add mariadb-backup package to the list and let's see | 10:25 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: Move to MariaDB upstream repo on Ubuntu https://review.opendev.org/679209 | 10:25 |
| mnasiadka | (at least mariadb.org suggests installing it) | 10:25 |
| *** dpawlik has joined #openstack-kolla | 10:29 | |
| mgoddard | you could check the mariadb logs from 10.1 | 10:33 |
| *** andrein has joined #openstack-kolla | 10:34 | |
| *** shyamb has quit IRC | 10:42 | |
| *** shyamb has joined #openstack-kolla | 10:43 | |
| *** radeks_ has quit IRC | 10:51 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: DNM: Try updating alembic and SQLAlchemy in neutron-server https://review.opendev.org/679228 | 10:53 |
| *** radeks has joined #openstack-kolla | 10:53 | |
| *** radeks has quit IRC | 10:58 | |
| *** shyamb has quit IRC | 11:02 | |
| *** shyamb has joined #openstack-kolla | 11:02 | |
| *** shyamb has quit IRC | 11:06 | |
| *** k_mouza has quit IRC | 11:10 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 11:14 |
| *** radeks has joined #openstack-kolla | 11:17 | |
| *** dave-mccowan has joined #openstack-kolla | 11:18 | |
| *** kplant has joined #openstack-kolla | 11:20 | |
| openstackgerrit | Merged x/kayobe stable/stein: Set provisioning and cleaning networks as non-shared https://review.opendev.org/679129 | 11:27 |
| *** k_mouza has joined #openstack-kolla | 11:31 | |
| mnasiadka | mgoddard: seems adding mariadb-backup package made it work | 11:35 |
| mnasiadka | but I think debian should be broken due to my change, but we don't have a debian CI | 11:36 |
| *** k_mouza has quit IRC | 11:37 | |
| mnasiadka | mgoddard: but still I don't like this: 2019-08-29 10:59:14 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status Joiner --uuid fc8c3eb9-ca4b-11e9-8746-86b9735ff44b --primary yes --index 1 --members | 11:39 |
| mnasiadka | 051e82c5-ca4c-11e9-9ef6-8e61bd0d8fc7/primary/10.176.128.68:3306,0974c821-ca4c-11e9-b033-4f5cb82afe51/secondary1/10.176.128.79:3306,fc8bb98f-ca4b-11e9-9382-2bbcd1845ccb/secondary2/10.176.128.120:3306: 1 (Operation not permitted) | 11:39 |
| mnasiadka | but it works | 11:40 |
| mnasiadka | ;) | 11:40 |
| *** nathharp has joined #openstack-kolla | 11:41 | |
| *** k_mouza has joined #openstack-kolla | 11:49 | |
| *** shyamb has joined #openstack-kolla | 11:49 | |
| openstackgerrit | Ondrej Duchon proposed openstack/kolla-ansible master: Corrected notificaions for auditing. https://review.opendev.org/678544 | 11:51 |
| mgoddard | mnasiadka: yes, we see that always IIRC | 11:54 |
| mnasiadka | mgoddard: then I'm calm :) | 11:54 |
| mgoddard | I think it calls wsrep-notify at some point when the DB isn't ready so it fails. It should call it again later though | 11:55 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Workaround: Pin neutron source to avoid migration issue https://review.opendev.org/679213 | 11:57 |
| *** andrein has quit IRC | 11:58 | |
| *** andrein has joined #openstack-kolla | 11:59 | |
| *** k_mouza has quit IRC | 12:05 | |
| *** k_mouza_ has joined #openstack-kolla | 12:05 | |
| *** scottsol has joined #openstack-kolla | 12:12 | |
| *** k_mouza_ has quit IRC | 12:16 | |
| mnasiadka | mgoddard: although I'm starting to think about adding the same repo to CentOS - if it's the way we want to go... | 12:18 |
| *** k_mouza has joined #openstack-kolla | 12:21 | |
| *** k_mouza has quit IRC | 12:21 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 12:22 |
| *** k_mouza has joined #openstack-kolla | 12:22 | |
| *** k_mouza has quit IRC | 12:25 | |
| *** keper7 has joined #openstack-kolla | 12:33 | |
| *** shyamb has quit IRC | 12:38 | |
| *** shyamb has joined #openstack-kolla | 12:38 | |
| *** andrein has quit IRC | 12:42 | |
| *** Luzi has quit IRC | 12:44 | |
| mgoddard | bumping alembic and sqlalchemy versions didn't work. | 12:49 |
| mnasiadka | so bumping up mariadb in ubuntu did work | 12:54 |
| mnasiadka | https://review.opendev.org/#/c/679209/ | 12:54 |
| mnasiadka | are we going that path? | 12:54 |
| *** keper7 has quit IRC | 12:55 | |
| openstackgerrit | Ondrej Duchon proposed openstack/kolla-ansible master: Corrected notificaions for auditing. https://review.opendev.org/678544 | 12:56 |
| mnasiadka | mgoddard, yoctozepto: ^^ | 12:56 |
| mgoddard | I'm concerned about it. 10.3 upgrade in centos was a pain | 12:59 |
| mgoddard | at least this time we don't have the xtrabackup/mariabackup headache that caused us to shutdown the cluster during upgrade | 12:59 |
| mgoddard | but still, the problems weren't immediately obvious | 13:00 |
| mgoddard | and didn't show up every time | 13:00 |
| yoctozepto | mgoddard, mnasiadka: recheck a few times :-) | 13:01 |
| mnasiadka | mgoddard: well, either way we'll have to go this path, for example when Ubuntu upgrades their packages | 13:02 |
| mgoddard | yes, but the more we flip back and forth the more we headaches we have | 13:03 |
| mnasiadka | and we might end up in a situation in November when we will need to do it last minute | 13:03 |
| yoctozepto | <mgoddard> bumping alembic and sqlalchemy versions didn't work. | 13:03 |
| mgoddard | why? | 13:03 |
| yoctozepto | could be not fixed | 13:03 |
| yoctozepto | <mgoddard> I think it calls wsrep-notify at some point when the DB isn't ready so it fails. It should call it again later though | 13:03 |
| mgoddard | ? | 13:03 |
| yoctozepto | so I thought | 13:03 |
| mnasiadka | yeah, it might be that neutron/alembic/sqlalchemy won't fix it, and we'll need to fix it on our side | 13:04 |
| yoctozepto | I'm ok with mgoddard workaround now | 13:04 |
| mnasiadka | we can pin neutron for now, but I doubt they will flip it back | 13:04 |
| mnasiadka | :) | 13:04 |
| yoctozepto | and having ubuntu patch ready :-) | 13:04 |
| yoctozepto | same here | 13:04 |
| mgoddard | they might be persuaded to make it work | 13:05 |
| yoctozepto | I see, you can blackmail them? ;D | 13:05 |
| mnasiadka | mgoddard: well, I still think it's not Neutron's fault | 13:05 |
| mgoddard | no it's not, but that doesn't mean they don't need to make it work | 13:06 |
| yoctozepto | it's not their fault in the sense that I doing it would not think I did something bad either | 13:06 |
| mgoddard | it's not our fault either, but it is our problem | 13:06 |
| yoctozepto | the "we don't support latest ubuntu lts" is a bad label | 13:06 |
| mnasiadka | well, it's rather a bad label for Ubuntu guys | 13:07 |
| mnasiadka | they should deliver 10.3 as part of UCA and everybody would be happy | 13:07 |
| yoctozepto | who would know 10.1 (or their patches or alembic - we are not sure about that) - would be that faulty | 13:07 |
| mgoddard | software has bugs, new software will still have bugs | 13:07 |
| yoctozepto | indeed mgoddard | 13:07 |
| *** dougsz has joined #openstack-kolla | 13:07 | |
| yoctozepto | the question is who to rely on | 13:08 |
| mgoddard | a lot of software dev is about coping with that | 13:08 |
| yoctozepto | osa decided to rely on upstream mariadb | 13:08 |
| *** zbr has quit IRC | 13:08 | |
| yoctozepto | for either distro afaik | 13:08 |
| *** andrein has joined #openstack-kolla | 13:08 | |
| yoctozepto | that way at least our mariadb issues are synchronized xD | 13:09 |
| yoctozepto | and mariadb upstream might be more inclined to support painless upgrades | 13:09 |
| yoctozepto | lts are more like 'use this oldie, use this oldie, then start anew' | 13:10 |
| yoctozepto | well, at least until new lts is all ready and shining | 13:10 |
| yoctozepto | ;p | 13:10 |
| yoctozepto | I must say the 14.04->16.04->18.04 path was quite nice (simple app+db server) | 13:11 |
| yoctozepto | anyone following my great monologue? ;p | 13:12 |
| yoctozepto | mgoddard, mnasiadka: please share your thoughts ^ | 13:15 |
| mnasiadka | well, maybe I have suicidal thoughts, but I'm really nervous about major differences in critical services versions between distros | 13:17 |
| *** zbr has joined #openstack-kolla | 13:17 | |
| mnasiadka | and I would go for upstream repo for Ubuntu at least, if not for all distros (but then tripleo will be unhappy and they will need to override it) | 13:17 |
| *** shyamb has quit IRC | 13:18 | |
| yoctozepto | mnasiadka: it's really easy to override, we can make an image for them, ubuntu folks don't use us this way | 13:19 |
| yoctozepto | ;-) | 13:19 |
| yoctozepto | we gate together (or will) and they remove our repos so it should be piece of cake ;-) | 13:20 |
| yoctozepto | if we turn to upstream mariadb, we would be obliged to ensure they can use the rdo one | 13:21 |
| yoctozepto | though I like the rdo approach, they are helpful | 13:21 |
| yoctozepto | maybe just for ubuntu and somehow manage to sync version with rdo? | 13:21 |
| yoctozepto | we don't need the latest greatest stuff | 13:21 |
| yoctozepto | though upstream seems to care about last two versions only? | 13:22 |
| yoctozepto | https://downloads.mariadb.org/mariadb/+releases/ | 13:23 |
| yoctozepto | nah, they care about all relevant currently | 13:23 |
| yoctozepto | my heart goes upstream | 13:23 |
| yoctozepto | mnasiadka, mgoddard | 13:23 |
| *** keper7 has joined #openstack-kolla | 13:23 | |
| yoctozepto | for ubuntu | 13:23 |
| yoctozepto | and sync version with rdo | 13:23 |
| yoctozepto | (minor, not patch of course) | 13:24 |
| yoctozepto | would be nice to convince devstack to deploy mariadb instead of mysql in the first place too | 13:24 |
| yoctozepto | ;p | 13:24 |
| mnasiadka | probably this makes sense, we just need to make sure it works :) | 13:24 |
| *** gfidente has quit IRC | 13:27 | |
| *** gfidente has joined #openstack-kolla | 13:27 | |
| mgoddard | are we sure we're not just making work for ourselves here? | 13:28 |
| mgoddard | mnasiadka: have you asked your neutron friend about whether they're likely to fix it? | 13:28 |
| mnasiadka | mgoddard: all tests finished successfully, what's there to fix? :) | 13:29 |
| mgoddard | that's what we said last time! | 13:29 |
| mgoddard | we always have troubles when upgrading an infra component like this - db or mq | 13:30 |
| mnasiadka | yeah, well - I guess we'll start seeing more of these issues if devstack continues to be tested on MySQL 5.7 - which has a lot of incompatibilities with MariaDB 10.1 | 13:34 |
| *** priteau has quit IRC | 13:35 | |
| mnasiadka | I'm not saying it's not adding us work | 13:37 |
| mnasiadka | but there are no more deployment projects using MariaDB 10.1 that we know of :) | 13:38 |
| Chaserjim | Hi Michael ! < jim bagwell here (just sayin hey) | 13:38 |
| mnasiadka | Hi Chaserjim | 13:38 |
| *** hongbin has joined #openstack-kolla | 13:48 | |
| *** gfidente has quit IRC | 13:52 | |
| *** dave-mccowan has quit IRC | 13:54 | |
| *** signed8bit has quit IRC | 13:54 | |
| *** gfidente has joined #openstack-kolla | 13:54 | |
| *** dave-mccowan has joined #openstack-kolla | 13:55 | |
| *** jovial[m] has quit IRC | 13:55 | |
| *** bjolo has quit IRC | 13:57 | |
| *** dougsz has quit IRC | 13:57 | |
| *** roukoswarf has quit IRC | 14:05 | |
| *** dougsz has joined #openstack-kolla | 14:15 | |
| *** dougsz has quit IRC | 14:22 | |
| *** dpawlik has quit IRC | 14:23 | |
| yoctozepto | mgoddard, mnasiadka: https://etherpad.openstack.org/p/mariadb-upstream-yes-or-no | 14:23 |
| yoctozepto | assemble your thoughts | 14:24 |
| openstackgerrit | Pierre Riteau proposed x/kayobe stable/queens: Allocate IPs separately from configuring network https://review.opendev.org/679269 | 14:26 |
| yoctozepto | mgoddard: UCA provides newer mongodb, yet not mariadb: http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/m/ | 14:30 |
| yoctozepto | deprecate mariadb | 14:30 |
| yoctozepto | +10 | 14:30 |
| yoctozepto | noSQL in kolla | 14:31 |
| yoctozepto | (note for those not in the subject: I am joking) | 14:32 |
| *** hamdyk has quit IRC | 14:33 | |
| *** dougsz has joined #openstack-kolla | 14:35 | |
| yoctozepto | ok, the poor man's mind map is ready from my side | 14:39 |
| mgoddard | if bionic does not support train, should we use disco? | 14:40 |
| yoctozepto | mgoddard: according to their notes, they are planning to support up to U on bionic | 14:41 |
| yoctozepto | just they lag with that mariadb... | 14:41 |
| mgoddard | even so, should we just use disco for our ubuntu containers? | 14:41 |
| yoctozepto | mgoddard: does uca build for non-lts? | 14:42 |
| mgoddard | pass | 14:42 |
| yoctozepto | for non-lts we are tied to their openstack release | 14:43 |
| yoctozepto | though it would fix mariadb ;p | 14:43 |
| yoctozepto | argh | 14:43 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: DNM: Update ubuntu image to 19.04 https://review.opendev.org/679277 | 14:43 |
| mgoddard | ^ don't know till you try | 14:44 |
| yoctozepto | mgoddard: let's see if ubunters did not break mariadb on purpose, sure! | 14:45 |
| yoctozepto | I have another idea to test as well | 14:45 |
| *** jbadiapa has quit IRC | 14:47 | |
| *** also_stingrayza has joined #openstack-kolla | 14:49 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla master: DNM: upstream mariadb 10.1 on Ubuntu https://review.opendev.org/679282 | 14:49 |
| yoctozepto | ^ because why not | 14:49 |
| yoctozepto | and we still need to grab some time to migrate percona xtrabackup to mariabackup | 14:51 |
| yoctozepto | 10.3 is not supported with percona | 14:51 |
| yoctozepto | so centos already affected | 14:51 |
| yoctozepto | and ubuntu in plans | 14:51 |
| openstackgerrit | Ondrej Duchon proposed openstack/kolla-ansible master: Corrected notificaions for auditing. https://review.opendev.org/678544 | 14:55 |
| *** dougsz has quit IRC | 14:58 | |
| *** jbadiapa has joined #openstack-kolla | 15:00 | |
| *** Chaserjim has quit IRC | 15:02 | |
| *** cally725 has joined #openstack-kolla | 15:05 | |
| mnasiadka | mgoddard, yoctozepto: so how many versions of the solution we now have? :D | 15:07 |
| *** itlinux has joined #openstack-kolla | 15:08 | |
| *** also_stingrayza has quit IRC | 15:09 | |
| cally725 | Hi is there a way to set the mtu to 9000 in the kola instances | 15:10 |
| yoctozepto | mnasiadka: 10.1 is for pure fun | 15:11 |
| yoctozepto | if it magically passes, then ubuntu folks are to blame -> immediate report to their bugzilla | 15:12 |
| yoctozepto | cally725: what do you mean? you might want to configure neutron for higher mtu for the tenant (tunneled) networks | 15:13 |
| *** scottsol has quit IRC | 15:14 | |
| mnasiadka | mgoddard: non-lts is probably also not the way, because we'll need to change ships often :) | 15:14 |
| yoctozepto | mgoddard, mnasiadka: update on kolla data plane ipv6 | 15:14 |
| mnasiadka | mgoddard: but we could install mariadb 10.3 from non-lts :D | 15:14 |
| yoctozepto | so far it seems that | 15:14 |
| *** scottsol has joined #openstack-kolla | 15:14 | |
| yoctozepto | 1) ipv6 link-local pings do not work (maybe by design?) | 15:15 |
| yoctozepto | 2) ipv6 slaac does not seem to work | 15:15 |
| cally725 | Yes I set global_physnet_mtu = 9000 in the neutron.conf and the path_mtu = 9000 in the ml2.ini | 15:15 |
| yoctozepto | 3) manually assigning delegated ipv6 address results in pingable network | 15:15 |
| yoctozepto | so not that bad | 15:16 |
| yoctozepto | though not good either | 15:16 |
| yoctozepto | mnasiadka: non-lts is bad because uca is for lts | 15:16 |
| yoctozepto | installing non-lts on lts is equally bad | 15:16 |
| yoctozepto | as there is not a single entity willing to support us in that | 15:17 |
| mnasiadka | yoctozepto: so there's only one normal solution, and it seems OSA does the same ;) | 15:17 |
| yoctozepto | I agree, it makes sense since UCA does not update mariadb | 15:17 |
| yoctozepto | eh | 15:17 |
| yoctozepto | have you checked the mind map? | 15:17 |
| cally725 | yoctozepto I see the ovs yoctozepto set to 9000 but br-int and br-tun still at 1500 | 15:17 |
| cally725 | yoctozepto kola rocky | 15:18 |
| yoctozepto | cally725: standard questions - have you confirmed that the new config files are deployed on hosts and that the services were restarted since? | 15:18 |
| cally725 | yoctozepto yes I logged on one of the controller and I see the new config values in the neutron and ml2 files | 15:19 |
| cally725 | yoctozepto and I think that the kola-ansible redeployed restarted the neutron dockers | 15:20 |
| yoctozepto | you can confirm that with docker ps | 15:21 |
| cally725 | this is I am doing but they all show 40 hours up | 15:21 |
| cally725 | yoctozepto so I am not 100% sure then | 15:22 |
| yoctozepto | cally725: it should have restarted them if you used reconfigure with kolla-ansible | 15:23 |
| yoctozepto | but you can restart the relevant ones manually | 15:23 |
| cally725 | yoctozepto this is what I did indeed | 15:23 |
| yoctozepto | I see | 15:23 |
| yoctozepto | did you try rebooting? might be neutron is not recreating those interfaces now | 15:23 |
| *** dpawlik has joined #openstack-kolla | 15:23 | |
| cally725 | yoctozepto no I can try rebooting the 3 servers (2 controllers and on compute) | 15:24 |
| yoctozepto | all right, please note even number for controllers is bad due to lack of quorum | 15:25 |
| yoctozepto | (1/2 never forms a quorum, better use 3 so that you can have 2/3 quorum) | 15:25 |
| cally725 | yoctozepto oups ok so I would be better with 1 than 2 ? | 15:25 |
| cally725 | yoctozepto this is probably why I have restart problems sometime with mariadb | 15:26 |
| yoctozepto | cally725: not exactly always, it only means that having 2 does not really make you have proper HA | 15:26 |
| yoctozepto | mariadb elects a master | 15:27 |
| yoctozepto | so it has problems when you have both servers down at the same time | 15:27 |
| cally725 | yes | 15:27 |
| yoctozepto | rabbitmq is more of an issue here | 15:27 |
| yoctozepto | you should down one at a time | 15:27 |
| yoctozepto | (even with 3 controllers) | 15:27 |
| cally725 | ok | 15:28 |
| cally725 | noted | 15:28 |
| *** dpawlik has quit IRC | 15:28 | |
| *** jbadiapa has quit IRC | 15:30 | |
| cally725 | yoctozepto so the procedure I followed to set the mtu is good then ? | 15:31 |
| yoctozepto | cally725: afaik yes, the two options should make it happen | 15:32 |
| cally725 | yoctozepto ok should all ovs br-xxx be set to 9000 after the modif ? | 15:34 |
| yoctozepto | mgoddard: ubuntu 1904 failing all over the place ;-) | 15:34 |
| mgoddard | it was a long shot | 15:34 |
| yoctozepto | mgoddard: indeed | 15:35 |
| cally725 | yoctozepto what about ubuntu 1804 ? | 15:35 |
| yoctozepto | cally725: I think so, but I am not a neutron expert, probably better to ask in their channel | 15:35 |
| cally725 | yoctozepto can you propose me a channel if you can ? | 15:35 |
| yoctozepto | cally725: don't worry about our internal chat, we are solving some mariadb issue for train on bionic | 15:36 |
| yoctozepto | cally725: #openstack-neutron | 15:36 |
| cally725 | yoctozepto thanks I will check with them | 15:36 |
| yoctozepto | though for help you can also try the mailing list and/or ask.openstack.org | 15:36 |
| yoctozepto | yw | 15:36 |
| cally725 | yoctozepto sorry I am new here which mailing list ? | 15:39 |
| *** skramaja has quit IRC | 15:39 | |
| yoctozepto | cally725: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss | 15:51 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla master: DNM: upstream mariadb 10.1 on Ubuntu https://review.opendev.org/679282 | 15:55 |
| *** hongbin has quit IRC | 15:58 | |
| *** dpawlik has joined #openstack-kolla | 16:01 | |
| *** gfidente has quit IRC | 16:04 | |
| *** dpawlik has quit IRC | 16:06 | |
| *** dougsz has joined #openstack-kolla | 16:14 | |
| *** ivve has quit IRC | 16:24 | |
| *** scottsol has quit IRC | 16:35 | |
| yoctozepto | <yoctozepto> 1) ipv6 link-local pings do not work (maybe by design?) | 16:41 |
| yoctozepto | by design indeed | 16:41 |
| yoctozepto | seems they are filtered out explicitly | 16:41 |
| *** nathharp has quit IRC | 16:42 | |
| *** igordc has joined #openstack-kolla | 16:44 | |
| *** andrein has quit IRC | 16:47 | |
| *** dougsz has quit IRC | 16:56 | |
| *** itlinux is now known as itlinux-away | 17:01 | |
| *** itlinux-away has quit IRC | 17:01 | |
| *** jbadiapa has joined #openstack-kolla | 17:07 | |
| yoctozepto | ubuntu honor restored - upstream 10.1 has the same issue: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_82/679282/2/check/kolla-ansible-ubuntu-source/b95eff9/primary/logs/ansible/deploy | 17:09 |
| yoctozepto | (latest, more recent than ubuntu and uncustomized) | 17:09 |
| yoctozepto | security patches aligned with 10.2 and 10.3 | 17:10 |
| yoctozepto | so it's indeed a thing of mysql incompat | 17:10 |
| yoctozepto | another + for upstream is support for debian | 17:12 |
| yoctozepto | which otherwise becomes another branch ;/ | 17:12 |
| *** ivve has joined #openstack-kolla | 17:31 | |
| *** tonythomas has quit IRC | 17:31 | |
| *** andrein has joined #openstack-kolla | 17:32 | |
| *** dr_feelgood has quit IRC | 17:45 | |
| *** andrein has quit IRC | 17:57 | |
| *** radeks has quit IRC | 18:15 | |
| *** signed8bit has joined #openstack-kolla | 18:15 | |
| *** dpawlik has joined #openstack-kolla | 18:17 | |
| *** dpawlik has quit IRC | 18:22 | |
| *** dr_feelgood has joined #openstack-kolla | 18:23 | |
| *** dr_feelgood has quit IRC | 18:27 | |
| *** jonaspaulo has joined #openstack-kolla | 18:30 | |
| *** andrein has joined #openstack-kolla | 18:34 | |
| *** JamesBenson has joined #openstack-kolla | 18:47 | |
| *** kplant has quit IRC | 18:50 | |
| *** hongbin has joined #openstack-kolla | 18:58 | |
| *** hongbin has quit IRC | 19:03 | |
| *** scottsol has joined #openstack-kolla | 19:06 | |
| *** spsurya has quit IRC | 19:07 | |
| *** KeithMnemonic has quit IRC | 19:09 | |
| *** scottsol has quit IRC | 19:22 | |
| *** tonythomas has joined #openstack-kolla | 19:26 | |
| *** Chaserjim has joined #openstack-kolla | 19:27 | |
| *** dpawlik has joined #openstack-kolla | 20:18 | |
| *** itlinux has joined #openstack-kolla | 20:23 | |
| JamesBenson | Hey all, I'm back from vacation, and working on getting Xen to work again... | 20:23 |
| *** dpawlik has quit IRC | 20:23 | |
| *** scottsol has joined #openstack-kolla | 20:35 | |
| *** factor has quit IRC | 20:42 | |
| *** factor has joined #openstack-kolla | 20:42 | |
| *** watersj has joined #openstack-kolla | 20:43 | |
| watersj | what is proper kolla way to start mariadb, kolla-ansible deploy --tags mariadb ? | 20:44 |
| JamesBenson | @watersj start or deploy? That's the correct code for deploying. | 20:47 |
| watersj | sry I didn't give enough info, if host goes down (dmin trips power cord) from an already deployed environment. What is proper kolla way to restart db? | 20:49 |
| openstackgerrit | Pierre Riteau proposed x/kayobe stable/queens: Allocate IPs separately from configuring network https://review.opendev.org/679269 | 20:50 |
| JamesBenson | I think the proper way is using the recovery.... but not 100% sure. | 20:51 |
| *** icarusfactor has joined #openstack-kolla | 20:53 | |
| *** scottsol has quit IRC | 20:54 | |
| *** factor has quit IRC | 20:54 | |
| noxoid | if mariadb wont start on its own then yea mariadb_recovery would be the best choice | 21:05 |
| *** brtknr has quit IRC | 21:22 | |
| *** brtknr has joined #openstack-kolla | 21:23 | |
| *** JamesBenson has quit IRC | 21:41 | |
| *** JamesBenson has joined #openstack-kolla | 21:43 | |
| *** JamesBen_ has joined #openstack-kolla | 21:44 | |
| *** luksky has joined #openstack-kolla | 21:46 | |
| *** JamesBenson has quit IRC | 21:47 | |
| *** JamesBen_ has quit IRC | 21:49 | |
| *** brtknr has quit IRC | 22:11 | |
| *** jonaspaulo has quit IRC | 22:13 | |
| *** ivve has quit IRC | 22:25 | |
| *** Chaserjim has quit IRC | 22:34 | |
| *** luksky has quit IRC | 22:38 | |
| *** Chaserjim has joined #openstack-kolla | 22:39 | |
| *** JamesBenson has joined #openstack-kolla | 23:39 | |
| *** itlinux is now known as itlinux-away | 23:40 | |
| *** andrein has quit IRC | 23:41 | |
| *** JamesBenson has quit IRC | 23:43 | |
| *** tonythomas has quit IRC | 23:51 | |
| *** hjensas has quit IRC | 23:52 | |
| *** hjensas has joined #openstack-kolla | 23:56 | |
| *** hongbin has joined #openstack-kolla | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!