Wednesday, 2015-04-22

*** markvoelker has joined #openstack-ansible00:02
*** BjoernT has quit IRC00:04
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613400:06
*** markvoelker has quit IRC00:08
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613700:10
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613400:11
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613700:12
cloudnulljaveriak still around ?00:12
javeriakyep00:12
cloudnulli've seen that issue when the lb is not able to contact the galera cluster .00:12
cloudnullyou should be able to ssh to the container that is exhibiting the issue and then simply run `mysql`.00:13
cloudnullif that fails I'd check the lb vip00:13
javeriakssh from the deploy node?00:13
javeriakcloudnull: okay, so i was getting this on the keystone container and running 'mysql' it says " Can't connect to MySQL server on '172.29.236.7"00:15
javeriakand the only place i see that defined is in "etc/rpc_deploy/rpc_user_config.yml" which has the correct value00:17
cloudnullif you run `mysql -h 172.29.236.7` can you connect to it ?00:17
cloudnullalso running haproxy?00:17
javeriakcloudnull: yes "mysql -h 172.29.236.7" gives the same, and haproxy is running00:18
cloudnulldid you run the haproxy play to set that up and if so, can you run `hatop -s /var/run/haproxy.stat` and see that its serving content to the backend vip?00:19
javeriakokay i managed to get the command to run, turns out the container wasnt able to connect to the haproxy node, removed some extra routes00:20
javeriakthanks cloudnull00:20
cloudnullah, that'll do it.00:20
cloudnull:)00:20
cloudnullglad it wasnt something else terrible going on ;)00:21
javeriakyea phew :) .. btw that stat command, so the ones I see down, i should go check correct?00:21
cloudnullyes.00:22
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613700:29
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613400:32
*** jaypipes has quit IRC00:37
*** stevemar has joined #openstack-ansible01:56
*** markvoelker has joined #openstack-ansible02:01
*** darrenc is now known as darrenc_afk02:02
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613402:08
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613702:10
cloudnullpalendae ^ updated based on your comments.02:10
palendaecloudnull: Cool02:10
palendaeConversation in #openstack-infra sounds like rechecks are broken02:11
*** javeriak has quit IRC02:12
*** darrenc_afk is now known as darrenc02:16
miguelgrinbergcloudnull palendae need an opinion. I would like to create a "gate" test suite to replace the "scenario" that we are currently using at the gate. This new group will contain the same scenario tests we are running, plus one basic heat smoke test. Yay or nay?02:20
palendaemiguelgrinberg: So you'd propose making that the default job in the script?02:22
miguelgrinbergright, that's the proposal02:22
miguelgrinbergthere are no heat scenario tests in tempest02:23
palendaeI'm not opposed. We don't have any heat coverage at all right now02:23
miguelgrinbergright, we missed the auth problem last week02:23
*** galstrom_zzz is now known as galstrom02:47
*** sacharya has joined #openstack-ansible02:49
cloudnullmiguelgrinberg i say yay.02:59
miguelgrinbergcloudnull palendae: okay, patch is coming soon03:02
Sam-I-Amstop working :)03:02
miguelgrinbergSam-I-Am: says who?03:03
Sam-I-Amsays me! heh03:04
Sam-I-Ambecause i am too03:04
miguelgrinbergmuch worse for you at 10pm than me at 8pm :)03:05
Sam-I-Amthis is true03:05
cloudnullMiguelgrinberg : with the proposed change , is that something that we could gate on now but could go upstream into tempest as an actual scenario test?03:10
miguelgrinbergcloudnull: wasn't clear. I did not add a scenario test to tempest. I'm enabling an api test in addition to the scenario tests that we currently have.03:10
miguelgrinbergthat's why I created a new test list called "gate", to not mix non-scenario tests in the "scenario" list03:11
cloudnullAh OK.03:11
miguelgrinbergI haven't investigated why, but all the scenario tests for heat were moved into the heat repo03:11
cloudnullSorry. Carry on. Nothing to see here. ;)03:11
cloudnullMiguelgrinberg that's a thing happening to all the projects.03:12
miguelgrinbergwhat's the reason?03:12
cloudnullSo the tempest project doesn't have to maintain them.03:12
cloudnullI don't know all the reasons behind it. But its something that's come up in the defcore meetings.03:13
miguelgrinbergso in the future we'll have to import tests from each project03:13
cloudnullI believe so.03:13
*** javeriak has joined #openstack-ansible03:13
openstackgerritMiguel Grinberg proposed stackforge/os-ansible-deployment: Add a basic orchestration test to the gate check  https://review.openstack.org/17616603:14
miguelgrinbergcloudnull: ^ let me know what you think03:14
cloudnullLooks good to me.03:15
*** galstrom is now known as galstrom_zzz03:15
cloudnullQuestion, could we combine the scenario + api tests as they are?03:16
cloudnullThe gate job will take longer but that's OK for more test coverage.03:17
*** galstrom_zzz is now known as galstrom03:17
*** JRobinson__ is now known as JRobinson__afk03:30
*** galstrom is now known as galstrom_zzz03:43
*** galstrom_zzz is now known as galstrom03:44
*** sacharya1 has joined #openstack-ansible04:01
*** fawadkhaliq has joined #openstack-ansible04:02
*** sacharya has quit IRC04:02
*** sacharya has joined #openstack-ansible04:04
*** sacharya1 has quit IRC04:05
*** mahito has joined #openstack-ansible04:15
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613704:16
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613404:17
*** galstrom is now known as galstrom_zzz04:20
*** sacharya has quit IRC04:24
*** sdake has joined #openstack-ansible04:29
*** sdake_ has joined #openstack-ansible04:32
*** sdake has quit IRC04:33
*** JRobinson__afk is now known as JRobinson__04:38
*** fawadkhaliq has quit IRC04:43
openstackgerritMiguel Grinberg proposed stackforge/os-ansible-deployment: Add a basic orchestration test to the gate check  https://review.openstack.org/17616605:05
*** galstrom_zzz is now known as galstrom05:09
miguelgrinbergcloudnull: if this patch I proposed passed we have a "gate" test list that we can fill out with other api tests. As far as Heat goes not all tests pass, so we can't select all of them at this time, that's something I will probably look into tomorrow. We also need a more convenient way to select tests, the egrep based solution gets tiring for complex selections.05:13
*** galstrom is now known as galstrom_zzz05:15
*** fawadkhaliq has joined #openstack-ansible05:16
*** javeriak has quit IRC05:39
*** sdake_ has quit IRC05:42
*** markvoelker has quit IRC06:10
*** stevemar has quit IRC06:31
*** sdake has joined #openstack-ansible06:34
*** mahito has quit IRC06:38
*** markvoelker has joined #openstack-ansible06:40
*** markvoelker has quit IRC06:45
mancdazsup06:56
matttmorning mancdaz07:02
*** mrodden has quit IRC07:06
*** mrodden has joined #openstack-ansible07:07
*** javeriak has joined #openstack-ansible07:13
*** mrodden has quit IRC07:20
*** mrodden has joined #openstack-ansible07:21
*** JRobinson__ has quit IRC07:25
*** sdake has quit IRC07:35
*** markvoelker has joined #openstack-ansible07:41
*** markvoelker has quit IRC07:46
*** sdake has joined #openstack-ansible07:47
*** sdake has quit IRC08:00
odyssey4mewerd08:01
matttmorning odyssey4me08:03
odyssey4meI see that galera is once again a pain in the behind.08:04
*** sandywalsh has quit IRC08:24
*** sandywalsh has joined #openstack-ansible08:24
*** ccrouch has quit IRC08:31
*** fawadkhaliq has quit IRC08:36
*** markvoelker has joined #openstack-ansible08:42
*** markvoelker has quit IRC08:47
javeriakhey guys, i wrote a new play that is supposed to run on the 'neutron_all' group, i.e. execute only on all neutron-containers, but for some reason it also runs on the compute hosts.08:52
matttjaveriak: the neutron-linuxbridge-agent runs on the compute hosts08:53
matttjaveriak: so neutron_all will contain neutron_linuxbridge_agent which will contain your compute hosts08:55
javeriakmatt: ah okay, is there an easy way to determine what hosts are included in which groups, rpc_enviroment.yml seems to give a top-level distribution08:55
matttjaveriak: you can see this in your rpc_inventory.json file08:56
javeriakmatt: oh okay, i see that only the infra_neurtron_server containers are listed under neutron_server group but neutron conf files are also present on the agents container, will I have to specify both groups for mutual plays09:02
*** fawadkhaliq has joined #openstack-ansible09:07
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613409:22
matttjaveriak: yeah, the neutron_server group is for neutron-server service itself, but you'll find any of the neutron services will have neutron-configurations on them09:23
matttjaveriak: there was a recent change in master (it's probably in kilo branch) to narrow down neutron configurations based on what service you're running -- for example, there is no need for the neutron.conf to have db configurations if you're just using the linuxbridge-agent for example09:24
javeriakmatt: yes that makes more sense, as currently its a bit harder to track down all places configurations may need to be propagated for one service that has many containers09:25
matttjaveriak: in that case you'd be best off using neutron_all09:28
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613409:28
javeriakmatt: but then things like neuton.conf settings error out for compute nodes that are also grouped in neutron_all,09:31
matttjaveriak: why would it error out ?09:34
javeriakmatt: oh wait, if i were to introduce a new compute into an already running system, i would need to add it to the inventory json you mentioned, so that it gets the appropriate configs, hence thats why it cant find neurton.conf09:36
javeriakmatt: the actual neutron server process should only be running on the neutron-server containers? i dont seem to have a problem when i try to restart it for neutron-all09:39
matttjaveriak: if you are adding a new compute node you'd just have to update rpc_user_config.yml, you wouldn't have to modify the inventory json09:39
javeriakmatt: the inventory would have to be regenerated though right?09:40
matttjaveriak: ansible should take care of that as far as i'm aware, i'll need to test first hand but i don't believe anything further is needed09:41
*** markvoelker has joined #openstack-ansible09:43
javeriakmatt: i was working off of juno and i added another compute to the rpc_user_config.yml and ran all playbooks, i did not come into any direct errors, however the new compute seems to be missing some configs as when i run my custom play on it, it cannot find some configs09:43
javeriakmatt: would this have to do with clearing the previous inventory cache? it does not re-generate if it finds the files already there?09:44
matttjaveriak: ah i see what you are saying now, sorry ... yeah there you'd need to run the neutron and nova plays09:44
*** markvoelker has quit IRC09:47
matttjaveriak: you can probably limit the plays that you run to like nova-compute.yml, nova-compute-keys.yml, and neutron-linuxbridge-agent.yml, but you'd have to test this to double-check09:50
javeriakmatt: yes got it, thanks alot09:50
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613409:51
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613409:57
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613409:57
openstackgerritMatt Thompson proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613710:19
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Set gerrit default branch to kilo  https://review.openstack.org/17624610:25
matttjaveriak: cool, any further questions just ping, seems like we're all learning how this stuff works :)10:30
*** daneyon_ has joined #openstack-ansible10:34
*** daneyon has quit IRC10:35
*** markvoelker has joined #openstack-ansible10:43
*** markvoelker has quit IRC10:48
*** markvoelker has joined #openstack-ansible11:44
*** markvoelker has quit IRC11:45
*** markvoelker has joined #openstack-ansible11:45
openstackgerritMatt Thompson proposed stackforge/os-ansible-deployment: Ensure horizon's static content has correct ownership  https://review.openstack.org/17626311:53
openstackgerritMerged stackforge/os-ansible-deployment: Reorder common apt repo tasks  https://review.openstack.org/17555412:05
*** fawadkhaliq has quit IRC12:13
*** jaypipes has joined #openstack-ansible12:15
*** KLevenstein has joined #openstack-ansible12:22
*** mnestheu1 has joined #openstack-ansible12:31
*** galstrom_zzz is now known as galstrom12:50
*** javeriak has quit IRC12:53
*** galstrom is now known as galstrom_zzz12:54
b3rnard0i did include the bug that kevin mentioned but i know we prob need multiple eyes on it13:00
*** galstrom_zzz is now known as galstrom13:12
*** yaya has joined #openstack-ansible13:17
*** galstrom is now known as galstrom_zzz13:19
*** KLevenstein has quit IRC13:25
*** stevemar has joined #openstack-ansible13:29
*** stevemar has quit IRC13:37
*** Mudpuppy has joined #openstack-ansible13:50
*** Mudpuppy has quit IRC13:52
*** Mudpuppy has joined #openstack-ansible13:52
*** Mudpuppy has quit IRC13:57
*** sigmavirus24_awa is now known as sigmavirus2414:05
*** Mudpuppy has joined #openstack-ansible14:05
*** prometheanfire has quit IRC14:06
*** prometheanfire has joined #openstack-ansible14:07
*** stevemar has joined #openstack-ansible14:07
*** KLevenstein has joined #openstack-ansible14:08
*** Mudpuppy has quit IRC14:10
*** Mudpuppy has joined #openstack-ansible14:11
*** erikmwilson is now known as Guest9565814:11
*** Guest95658 has quit IRC14:11
*** erikmwilson has joined #openstack-ansible14:11
*** erikmwilson_ has joined #openstack-ansible14:11
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Set gerrit default branch to kilo  https://review.openstack.org/17624614:16
cloudnullodyssey4me ^ updated dependency to the pin update.14:18
odyssey4mecloudnull yeah, saw that - thanks14:18
odyssey4mepretty much any patches going in now will need that other patch as a dependence14:18
odyssey4me*cy14:18
cloudnullyup. we need to tackle icehouse juno too :'(14:19
odyssey4mesorry about rewriting the galera patches without consulting you, by the way - I figured it was more important to get it done asap14:19
odyssey4meyeah, busy looking into juno, but can't replicate the issue so far14:19
odyssey4meit's there in openstack-infra, but not there in any cloud server AIO or multi-node builds14:20
cloudnullno rewriting them was the right thing to do .14:20
cloudnullso no worries brother .14:20
cloudnullas for the issue you need to make sure your on the latest Ubuntu LTS image release.14:20
cloudnullin the iad3 lab i didn't see the problem which was running 14.04.0114:21
odyssey4mecloudnull before doing a build I made sure to 'apt-get dist-upgrade', does that not achieve the same thing?14:21
cloudnullbut in the qe lab we ran into is running 14.04.0214:21
cloudnulldist-upgrade should do it.14:21
odyssey4meyep: DISTRIB_DESCRIPTION="Ubuntu 14.04.2 LTS"14:22
cloudnullmaybe a "full-upgrade" is what needs to be done?14:22
mattti'm just trying with an image that is already 14.04.2 LTS14:22
palendaeChecking nova image list against RAX, none of the 14.04 images have the 3rd number =\14:23
matttthe previous i built was 14.04.1 i believe and something upgraded it along the way14:23
odyssey4meI suspect that the issue relates to a dependant library somewhere along the way, so when mariadb tries to install the old version we're pinning to apt simply refuses and tries something else.14:25
*** stevemar has quit IRC14:26
matttcloudnull: did you try juno or just kilo/master ?14:26
odyssey4meWe may just be able to update the pinned version to a newer release, but to do that we need to update the right mirror and wait for sync. I just want to make sure that this is the problem before we waste time on that rabbit hole.14:27
sigmavirus24cloudnull: are we going to remove the rpc_deployment directory for Liberty?14:28
cloudnullodyssey4me it is a lib issue for maria this is the output when you run into it.14:28
cloudnull The following packages have unmet dependencies:14:28
cloudnull   libmariadbclient-dev : Depends: libmariadbclient18 (>= 5.5.42+maria-1~trusty) but it is not going to be installed14:28
cloudnull   mariadb-client : Depends: mariadb-client-5.5 (= 5.5.42+maria-1~trusty) but it is not going to be installed14:28
cloudnullsigmavirus24 that was the plan14:28
sigmavirus24:thumbsup:14:29
cloudnullso in truth we could remove it in master whenever14:29
* sigmavirus24 was just checking his santiy14:29
sigmavirus24*sanity14:29
odyssey4mecloudnull in your testing environment, try updating the mariadb repo to this one: http://archive.mariadb.org/mariadb-5.5.42/repo/ubuntu/14:29
* cloudnull doing14:30
odyssey4memattt there was another archive mirror in europe that was faster - what was the URL again?14:30
odyssey4mearchive.mariadb.org had severe rate limiting on it14:30
openstackgerritIan Cordasco proposed stackforge/os-ansible-deployment: Remove the rpc_deployment directory  https://review.openstack.org/17633214:31
matttopenstackgerrit: looking14:31
matttodyssey4me: http://ftp.hosteurope.de/mirror/archive.mariadb.org14:31
sigmavirus24mattt: how nice of you to review my patchset so quickly ;)14:31
cloudnullodyssey4me same error  libmariadbclient18 : Depends: libmysqlclient18 (= 5.5.42+maria-1~trusty) but 5.5.43-0ubuntu0.14.04.1 is to be installed14:31
odyssey4mecloudnull did you clear the packages installed first, do an upt-get update, then run the play?14:34
cloudnulli cleared the package install, and then run update.14:34
cloudnulli was installing by hand14:34
odyssey4mebummer, it's only one patch rev from the version we're pinned to anyway :/14:35
cloudnullbut if i drop the pin to the domain this go14:39
matttcloudnull: so the pin gets dropped on juno already14:41
matttbut we're hitting this same issue14:41
*** Bjoern___ has joined #openstack-ansible14:41
odyssey4mecloudnull to the archive.mariadb domain?14:41
cloudnullmattt the pin that we have in juno / icehouse we pin with an option.14:41
cloudnullwhich is o=MariaDB14:41
cloudnulland now ubuntu's packages satisfy that option.14:42
cloudnullso if we do `Pin: origin <domain>`14:42
odyssey4mecloudnull that only works if the mariadb and ubuntu mirrors used are different host names14:43
*** sdake has joined #openstack-ansible14:43
odyssey4meI tested that this morning, which is why I switch to the org pinning for master/kilo.14:43
cloudnullwhich they are for us14:43
odyssey4mecloudnull they would be for us and for others who're using their own local mirrors... so we need another way to differentiate14:44
*** sdake_ has joined #openstack-ansible14:44
odyssey4medo the ubuntu packages honestly match that now?14:44
cloudnullit seems that way .14:45
matttodyssey4me: doesn't explain why kilo/master are now passing14:45
odyssey4mehow do we see this metadata?14:46
matttodyssey4me: it's the Release file14:46
mattti'm wondering if mirror.rs has some sync issue14:47
cloudnullodyssey4me: apt-cache show14:47
*** sdake has quit IRC14:48
cloudnullmattt just for the sake of argument the boxes im using aren't using the rs mirrors they're pointed at "http://gb.archive.ubuntu.com/ubuntu"14:48
matttorigin in http://gb.archive.ubuntu.com/ubuntu/dists/trusty-updates/universe/binary-amd64/Release is not MariaDB14:52
matttthat'd break a lot of other stuff14:52
odyssey4mecloudnull have you found a way to tell which match the 'release o=MariaDB'14:52
matttodyssey4me: ^^^14:53
odyssey4meversus: http://mirror.rackspace.com/mariadb/repo/5.5/ubuntu/dists/trusty/Release14:53
odyssey4meand http://archive.mariadb.org/mariadb-5.5.42/repo/ubuntu/dists/trusty/Release14:54
odyssey4mewhat I am seeing, though, is that apt-cache show mariadb-common on a server doesn't show Origin: MariaDB for the MariaDB packages14:55
cloudnullso as a compromise to the issue, which will in fact resolve it, we can pin to both the domain and the release.14:57
odyssey4mecloudnull it does fix it? seems odd as the domain is the same for both except for the situation you're in right now14:58
*** galstrom_zzz is now known as galstrom14:58
odyssey4meI'm game for another patching attribute, but ideally something unique14:59
odyssey4meorigin & label perhaps?14:59
cloudnullthat may work.15:00
odyssey4mealternatively the actual file version?15:00
cloudnullthe situation my boxes are in is likely going to be the most common as deployers adopt the system.15:00
odyssey4meyeah, but it's not a guaranteed win15:01
cloudnullits not.15:01
odyssey4methe file version may also work, but I thought we also has a pin for this15:01
cloudnullthe other option here would be to update mariadb to the latest stable.15:01
cloudnullinstead of staying with this old release.15:02
odyssey4mecloudnull the latest for mariadb is only a minor patch release up - that's the archive repo I sent you earlier15:02
odyssey4meunless you're talking v10?15:02
cloudnullno i mean mariadb1015:02
odyssey4meeek, that's a bit rough for icehouse/juno15:02
cloudnullagreeded15:03
cloudnullbut master/kilo could adopt that a more updated system.15:03
odyssey4meagreed - that's a possibility for kilo/master, although we'd need to get upgrades tested15:03
cloudnullas for juno / icehouse pinning to the maria repos as provided by the maria people would make this problem go away as well.15:04
odyssey4methe mariadb version numbers are all 5.5.41+maria-1~trusty15:05
odyssey4medifferent to ubuntu's packages15:05
odyssey4mea version pin may be the simplest way to get this right15:06
odyssey4meI have a pretty simple suggestion for icehouse/juno - we add a pin for the conflicting files to ensure that they use the mariadb versions15:07
odyssey4mewe also pin at a level that replaces any previously installed package by the same name15:08
odyssey4methe conflicts are mysql-common and libmysqlclient1815:08
sigmavirus24andymccr: do you agree with me on https://bugs.launchpad.net/openstack-ansible/+bug/1439892 ?15:08
openstackLaunchpad bug 1439892 in openstack-ansible trunk "Bindmount for Glance" [High,Confirmed] - Assigned to Ian Cordasco (icordasc)15:08
*** galstrom is now known as galstrom_zzz15:08
matttsigmavirus24: i wouldn't make that change15:09
sigmavirus24mattt: the bindmount or the change to the frequency with which we manage the cache?15:10
matttsigmavirus24: not the change you suggested but using bind mount for images :)15:10
odyssey4methe version to pin against is 5.5.41+maria-1~trusty15:11
sigmavirus24mattt: yeah, I think andymccr, you, and I are all in agreement to *not* bindmount the container just because the cache fills up quickly15:11
odyssey4mepackages: mysql-common, libmariadbclient18, libmariadbclient-dev15:11
sigmavirus24mattt: as a compromise I'm thinking of having the cache management scripts run every 12 hours (11 hrs 59 minutes) instead of every 2415:12
andymccryeh i think thats a bad solution sigmavirus24.15:12
sigmavirus24I would hope that would be frequent enough to not cause too many cache misses while preventing the cache from becoming too large15:12
sigmavirus24andymccr: changing the cron job is a bad solution?15:12
*** yaya has quit IRC15:12
cloudnullodyssey4me this seems to work http://cdn.pasteraw.com/tmyov5r4qjul9eoid6vbgy2g4ls2t5h15:13
odyssey4mecloudnull triple whammy!15:14
odyssey4meto replace a file already deployed I think the pin priority needs to be above 1000+15:14
andymccroh no i mean the bind mount sigmavirus24, sorry half dug in on this mysql package and my context switching is clearly terrible15:14
matttodyssey4me: yep15:14
cloudnullbetter one http://cdn.pasteraw.com/chr6z64pkgcfuy1qfmgztglgepuzm0t15:14
sigmavirus24andymccr: no worries15:14
andymccri like your solution much better sigmavirus24. itmakes more sense to me15:14
sigmavirus24andymccr: cool. sending patches now15:15
cloudnullso the version is the highest priority, then the released, then the domain15:15
*** fawadkhaliq has joined #openstack-ansible15:16
cloudnullwe'd need to set the version as a var in the default if we did something like that.15:17
cloudnullit could become part of the hash.15:17
cloudnulland rendered in the template if defined.15:17
odyssey4mecloudnull yeah, that's how it's done in icehouse/juno15:18
cloudnullisnt icehouse / juno doing https://github.com/stackforge/os-ansible-deployment/blob/juno/rpc_deployment/roles/common/templates/mariadb-priority15:19
cloudnullwhich has no version data within it15:19
odyssey4mecloudnull we can use the apt-pinning which is being done for lxc, etc - it's a different task15:21
cloudnullusing this https://github.com/stackforge/os-ansible-deployment/blob/juno/rpc_deployment/roles/common/templates/apt_pinned_packages.j215:22
odyssey4mebut we can just keep it simple in one file15:22
odyssey4meok, I'll put together revised master/kilo patches in a monent - just doing icehouse/juno patches first15:24
*** jwagner_away is now known as jwagner15:25
odyssey4mecloudnull fyi - it appears that for a pin-priority of 1000 and below, apt will allow itself to select an alternative origin... whereas for 1001+ it'll force the pin origin for the install15:26
odyssey4mehttp://blog.opperschaap.net/2009/12/12/using-the-preferences-file-on-debian-and-debian-based-distributions/15:26
cloudnulli used https://wiki.debian.org/AptPreferences for the source of truth15:26
cloudnullbut that sounds about right15:27
*** rromans has quit IRC15:31
openstackgerritMiguel Alejandro Cantu proposed stackforge/os-ansible-deployment: Implement Ceilometer[WIP]  https://review.openstack.org/17306715:31
*** rromans has joined #openstack-ansible15:33
d34dh0r53make it over 900015:36
*** sacharya has joined #openstack-ansible15:39
cloudnullandymccr odyssey4me for master i think we should be doing something like this https://gist.github.com/cloudnull/4d7e2be5e391c1061b9b15:39
cloudnullwhich pins on the version, domain, and release15:40
andymccrcloudnull: so one of the problems is that the package exists already and needs to get downgraded - which it doesnt like.15:40
openstackgerritIan Cordasco proposed stackforge/os-ansible-deployment: Run the glance cache cron jobs more frequently  https://review.openstack.org/17636515:43
odyssey4mecloudnull digging into a possibility, feedback in a bit15:45
openstackgerritIan Cordasco proposed stackforge/os-ansible-deployment: Run the glance cache cron jobs more frequently  https://review.openstack.org/17636815:46
openstackgerritIan Cordasco proposed stackforge/os-ansible-deployment: Run the glance cache cron jobs more frequently  https://review.openstack.org/17637015:47
sigmavirus24andymccr: mattt ^ reviews appreciated15:47
*** sdake_ has quit IRC15:48
*** erikmwilson has quit IRC16:11
*** erikmwilson has joined #openstack-ansible16:11
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Implement higher priority for mariadb packages  https://review.openstack.org/17638216:12
*** erikmwilson has quit IRC16:13
*** erikmwilson has joined #openstack-ansible16:13
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613416:17
odyssey4mecloudnull, we did a bit more testing and found that as long as the pin priority is above 1000 (ie 1001) then the mariadb stuff is properly preferred16:18
cloudnullwe know that will work, until it inevitably doesn't this is the forever game of wack-a-mole16:18
odyssey4methe source of the issue appears to be a mysql library that pre-exists in Ubuntu 14.04.2 which wasn't there in 14.04.1 - which is why upgrading to the new image doesn't display the issue16:18
odyssey4mesetting the pin priority to 1001 forces apt to replace any conflicting packages with the MariaDB versions, whereas anything below it tries to leave it in place16:19
cloudnullwhich is why i think that we need to pin to more things or removing pinning within OSAD16:20
odyssey4mewell, MariaDB is the only thing in OSAD being pinned for kilo/master16:20
cloudnullit is.16:20
ApsuWell technically the pinning isn't the issue exactly. It's part of it, but16:21
ApsuThe issue is that apt-get is somehow not treating -y as "really do it yes" for certain downgrade scenarios16:21
ApsuIt requires --force-yes, which is not one of the underlying dpkg --force-* options16:22
ApsuIt's apt-get itself (config item APT::Get::force-yes)16:22
ApsuBut it also doesn't explain Why it needs further confirmation, aside from the fact we're downgrading16:22
odyssey4meApsu that's resolve by using force in the apt task for ansible16:22
Apsuodyssey4me: Can actually solve it by dropping a file with the config item in /etc/apt/apt.conf.d16:22
cloudnulland its broken. but the greater issue is that this happened exactly like this for libvirt a few months ago. and for rabbit a few months before that, its happened for maria consistently and all throughout the havana / other releases it happened for lots of other packages. im not sure why we keep doing this .16:23
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613416:23
odyssey4meApsu that sounds unmanagable16:23
cloudnullpinning should be a deployer problem not an OSAD one.16:23
d34dh0r53^16:24
odyssey4mecloudnull agreed, which is why pinning in itself needs to move out of OSAD and be the deployer's solution using something like aptly which can aggregate packages, etc16:24
odyssey4mebut if we're using alternative repositories with common files in OSAD, we're forced to deal with pinning - this is the issue for MariaDB16:24
Apsuodyssey4me: cloudnull: d34dh0r53: I don't disagree with where pinning/forcing should live. But wherever it lives and how, right now priority == 1001 doesn't solve it alone in downgrade situations.16:25
odyssey4mewhy are we using a seperate repo for mariadb instead of just the Ubuntu packages?16:25
ApsuThe actual apt-get run needs force-yes enabled.16:25
ApsuSo that needs to happen somehow if we plan to continue managing package versioning and choose a pinning priority designed for downgrading.16:25
cloudnullodyssey4me the ubuntu packages dont have some of the galera packages, or rather didn't.16:25
cloudnullidk if thats still true.16:25
odyssey4meApsu please see https://review.openstack.org/176134 (master) and https://review.openstack.org/176382 (juno) - forced package installs has been included16:26
odyssey4meif there is review consensus for those two patches, the methods can be backported to kilo/icehouse - if not, they can be revised until there is consensus16:28
cloudnullthis doesn't seem to want to work when on the upstream ubuntu repos. this may be a Red herring but i'm looking more into it.16:31
odyssey4mecloudnull the approach I'm taking for kilo/master is to pin as little as possible and add as little extra config as possible16:31
odyssey4meif we really have to also pin on version, this will be messy16:32
andymccrodyssey4me: with force-yes i believe that works16:33
andymccrso i dont think you have to pin on version16:33
andymccri think that change should/would work.16:34
odyssey4mecloudnull not working for you?16:34
andymccrdont be that guy cloudnull16:34
cloudnullit does not. the version table seems to want to only use "5.5.43-0ubuntu0.14.04.1 0" im cleaning up using dpkg to test more.16:38
andymccrugh wtf16:38
andymccrso i had the 5.5.43 but with force-yes it did downgrade it successfully when the priority is 100116:38
cloudnulllet me give that a go.16:39
sigmavirus24So Kevin's (not cloudnull) feedback https://bugs.launchpad.net/openstack-ansible/+bug/1439892/comments/8 on https://bugs.launchpad.net/openstack-ansible/+bug/1439892 is pretty valid16:41
openstackLaunchpad bug 1439892 in openstack-ansible trunk "Bindmount for Glance" [High,In progress] - Assigned to Ian Cordasco (icordasc)16:41
sigmavirus24I can imagine a user creating a snapshot of a VM and trying to boot from it where the VM could be upwards of 20 GB.16:41
sigmavirus24I'm going to see if the glance PTL sees value in having a way to say "Don't cache images over X size"16:42
sigmavirus24Because I can see why caching basic images would be useful while caching snapshots may be less useful16:43
Bjoern___sigmavirus24: thats usually the case for windows guests. They need to have at least 40G instance storage to even get it insalled16:43
Bjoern___also we have customers with 1.2TB glance images16:43
*** Bjoern___ is now known as BjoernT16:43
sigmavirus24Yeah. I think disabling the cache by default isn't the best solution16:43
sigmavirus24That said, for os-a-d as a project, this is an issue16:43
odyssey4mesigmavirus24 I would think that snapshots whould not be cached at all16:44
odyssey4me*should16:44
sigmavirus24Lucky for us we have a glance core who can maybe help add some flexibility to the image service so we don't have to give up the benefits of caching and we can not fill up the container by trying to cache something larger than what we have available16:44
odyssey4methat said, for OSAD we could possibly just make the use of the cache configurable and essentially disable it entirely, but I suppose that makes boot time suffer too much16:45
sigmavirus24odyssey4me: it depends on many factors16:45
sigmavirus24and really it should be a per-installation thing based on the knowledge of the people using that installation16:46
odyssey4mesigmavirus24 unfortunately any changes upstream will only be useful to us when liberty releases, unless it gets sneaked in as a minor patch16:46
sigmavirus24"Do they boot tons of VMs from reasonably sized images? Then caching will probably help"16:46
sigmavirus24odyssey4me: yeah I realize that16:46
sigmavirus24I'm thinking of a more forward looking solution while we figure out what to do for juno/kilo16:46
*** subscope_ has joined #openstack-ansible16:48
*** yaya has joined #openstack-ansible16:49
*** Mudpuppy has quit IRC16:51
*** Mudpuppy has joined #openstack-ansible16:52
*** Mudpuppy has quit IRC16:56
*** subscope_ has quit IRC17:07
*** subscope_ has joined #openstack-ansible17:07
sigmavirus24So row 15 in https://docs.google.com/spreadsheets/d/1vCTZBJKCMZ2xBhglnuK3ciKo3E8UMFo5S5lmIAYMCSE/edit?pli=1#gid=827503418 already has a liberty summit session dedicated to caching17:11
sigmavirus24In the meantime, I'm going to send a separate patch to make the cache configurable (if it isn't already)17:11
*** jwagner is now known as jwagner_lunch17:14
*** sdake has joined #openstack-ansible17:24
*** sdake has quit IRC17:29
*** sdake has joined #openstack-ansible17:30
*** woodard has joined #openstack-ansible17:35
*** Mudpuppy has joined #openstack-ansible17:42
*** Mudpuppy has quit IRC17:43
*** Mudpuppy has joined #openstack-ansible17:44
*** woodard_ has joined #openstack-ansible17:50
*** woodard has quit IRC17:50
*** woodard has joined #openstack-ansible17:58
*** woodard_ has quit IRC17:59
*** jwagner_lunch is now known as jwagner18:02
*** javeriak has joined #openstack-ansible18:25
*** stevemar has joined #openstack-ansible18:50
*** Shrews has joined #openstack-ansible18:54
*** Shrews has left #openstack-ansible18:55
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613418:59
cloudnullso with origin and release pinning set to > 1000 my issues on the physical gear were resolved.19:01
cloudnulland in the data structure version pinning is also a possibility, should the deployer decide to do that.19:03
palendaeI like that approach19:03
palendaeAt the very least, exposing it should deployers wish to use it19:04
*** lkoranda has quit IRC19:04
palendaeThe omit filter might be of use in combination with those (not for mariadb, but offering pinning priorities) http://docs.ansible.com/playbooks_filters.html#omitting-undefined-variables-and-parameters19:04
cloudnullyes thats interesting default filter. i've not used it in the past, but i can see the potential .19:05
palendaeYeah, someone showed it to me the other day. New in 1.8, so we've not been using versions that include it for too long19:06
palendaeSo what can I do on an AIO to help test the galera patch? I ran an earlier patch set, before the > 1000 values were put in, and had a success19:08
palendaeSo I'm guessing I didn't have the right set up - I was on 14.04.219:08
ApsuThere's two separate issues or concerns.19:09
ApsuOne is the ability to install an older package from a 3rd-party repo when a newer version is available in a base repo19:09
ApsuP > 1000 solves that19:09
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613419:10
ApsuThe other issue is that if you already have a newer version of a package *installed*, apt-get -y isn't sufficient to downgrade it.19:10
cloudnullpalendae ^ updated for a doc blurb19:10
ApsuThat requires --force-yes, which Ansible's apt module "force: yes" parameter enables.19:10
ApsuHaving a newer version installed already has two concerns around it19:11
ApsuOne, dealing with regressions where we want to downgrade as part of a new release19:11
*** lkoranda has joined #openstack-ansible19:11
palendaeOk, so I'm testing with an AIO that's only been upgraded to 14.04.2,s o it's not going to have an existing install19:11
palendaeSo I might want to try with 2 AIOs19:11
ApsuTwo, deployment on base installs that for some reason or another (such as hardware monitoring packages) already have a newer package on19:11
palendaeAhh, ok19:12
ApsuIf you just test on our public cloud images, you probably won't run into it19:12
palendaeAlright, so AIO/the gate are not too useful for this19:12
ApsuI think the pinning priority and the force: yes should be separately dealt with, because they have very different concerns and potential impacts19:12
palendaeAt least without installing some conflicting patches19:12
palendaes/patches/packages/19:12
Apsuright19:12
ApsuShould also note that --force-yes is not a dpkg option.19:12
ApsuIt's unrelated to --force-things, --force-downgrade or --force-all19:13
palendaecloudnull: Thanks for the doc change19:13
ApsuIt's an apt-get option specifically (APT::Get::force-yes), and Only designed to make sure that by default you have to explicitly say "yes" to potentially dangerous things, like downgrading.19:13
ApsuThat's why apt-get -y doesn't work. Just apt-get then saying "yes" does work19:14
cloudnullso are we good with the changes that have gone into juno https://review.openstack.org/#/c/176382/ ?19:15
cloudnullif so then we need to bp to icehosue .19:15
*** stevemar has quit IRC19:15
cloudnull*icehouse19:15
clacoI guess the Q is around the force and downgrades19:16
Apsucloudnull: I don't think so. Like I said, I think the forcing should be separated out and we need to discuss it in depth, since it has an upgrade/regression impact.19:16
cloudnulli tend to agree19:16
ApsuNamely, whether we want that to be automatic (and potentially break things) or manual, as release-notes19:16
cloudnulland while pinning this will once again for for a time, it is only a matter of time before its all broken again for some other reason.19:17
ApsuSure19:17
cloudnullnoting from the debian docs "...Debian does not encourage its use without thorough consideration."19:18
Apsuindeed.19:19
*** javeriak has quit IRC19:19
ApsuDowngrading in general is considered dangerous on Debian-based systems.19:19
ApsuMostly because dpkg does not do dependency checking19:19
cloudnull^ this !19:19
ApsuRight now, that's probably ok. But at any moment, those packages could change which libraries they link against (soname bumps, their own regressions/bugfixes)19:20
ApsuAnd if they happen to drop them into the 3rd-party repo too, our pinning will pick them up19:20
ApsuPotentially breaking the entire system19:20
clacoor as you mentioned, the monitoring stuff for omsa or hp doing dep things b4 we even get started19:20
ApsuRight19:20
clacothe trisk is to get over the hump and I'll make sure we communicate up and outword that pinning is also volitile and repo tjhings need some prio love19:21
ApsuCommented and -1'd appropriately: https://review.openstack.org/#/c/176382/19:22
*** jaypipes has quit IRC19:24
*** javeriak has joined #openstack-ansible19:26
*** woodard has left #openstack-ansible19:27
*** sandywalsh has quit IRC19:30
*** sandywalsh has joined #openstack-ansible19:31
palendaeYeah...it sounds like without some hardware that we have said agents on in automatic testing, it should be a big message to deployers19:35
palendaeSo juno has the force: yes, but master does not?19:36
ApsuDon't see the force in kevin's patch19:38
Apsucloudnull: I see there's a urlfilters filter plugin combined in your patch. Was that intentional?19:38
cloudnullyes. that adds the origin type pin19:39
Apsukk. I thought that's what it looked like, just making sure :)19:40
*** javeriak_ has joined #openstack-ansible19:40
cloudnullyea wanted to provide for origin, release and version type pins if needed.19:40
Apsuplus juan19:40
palendaeOnly one actual use of the filter in this current revision19:41
palendaeThat I see, anyway19:41
*** javeriak has quit IRC19:43
cloudnullpalendae it should be in the pin file for both client and server19:43
palendaeYeah, sorry, 219:44
palendaenetloc19:44
*** stevemar has joined #openstack-ansible19:44
Apsu-1, under loc quota19:45
cloudnullhuh?19:45
palendaeline of code quota19:46
cloudnullhahaha19:46
Apsulolol19:48
*** openstackgerrit has quit IRC19:54
*** openstackgerrit has joined #openstack-ansible19:55
*** subscope_ has quit IRC19:59
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Implement higher priority for mariadb packages  https://review.openstack.org/17645820:04
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles (juno)  https://review.openstack.org/17645820:06
*** alextricity_r has joined #openstack-ansible20:08
*** alextricity_r has quit IRC20:13
*** fawadkhaliq has quit IRC20:15
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613420:16
cloudnulland now we wait for gerrit.20:16
cloudnullhttps://review.openstack.org/176458 <- Apsu pinning without the forcing20:17
clacowho's on e the icehouse bp20:17
cloudnullthese solution follow the upstream documentation from mariadb so they should go "https://blog.mariadb.org/dotdeb-repository-problems-with-mariadb-5-5-solution/"20:18
cloudnullim porting the juno fix to icehouse.20:18
clacowe should lob that solution link towards docs20:19
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles (icehouse)  https://review.openstack.org/17646320:19
openstackgerritIan Cordasco proposed stackforge/os-ansible-deployment: Allow users to configure the Glance flavor in use  https://review.openstack.org/17646420:20
cloudnullSam-I-Am: >"https://blog.mariadb.org/dotdeb-repository-problems-with-mariadb-5-5-solution/"<20:20
* cloudnull lobbing 20:20
openstackgerritIan Cordasco proposed stackforge/os-ansible-deployment: Run the glance cache cron jobs more frequently  https://review.openstack.org/17636520:21
openstackgerritIan Cordasco proposed stackforge/os-ansible-deployment: Run the glance cache cron jobs more frequently  https://review.openstack.org/17637020:22
cloudnullclaco https://review.openstack.org/#/c/176463/ <- icehouse fix20:22
clacowe'll see :-)20:23
openstackgerritIan Cordasco proposed stackforge/os-ansible-deployment: Run the glance cache cron jobs more frequently  https://review.openstack.org/17636820:23
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles (icehouse)  https://review.openstack.org/17646320:28
sigmavirus24I updated https://review.openstack.org/#/q/Ie7311f70656447382c2403e79453fe88401efca1,n,z because I realized that 5h 59m is what we wanted, not 6h 59m so I updated the changes for that20:28
palendaehttps://review.openstack.org/#/c/176134/ +2+W20:28
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613420:29
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles (juno)  https://review.openstack.org/17645820:29
cloudnullApsu palendae  sorry updated commit messages to document why.20:29
*** sandywalsh has quit IRC20:33
*** sandywalsh has joined #openstack-ansible20:34
matttpalendae: yeah if you want to test this, first install libmysqclient18 before you run ansible20:35
*** Mudpuppy has quit IRC20:37
*** alextricity_r has joined #openstack-ansible20:37
*** alextricity_r has quit IRC20:40
*** alextricity_r has joined #openstack-ansible20:41
*** stevemar has quit IRC20:41
*** Mudpuppy has joined #openstack-ansible20:42
*** KLevenstein has quit IRC20:42
*** stevemar has joined #openstack-ansible20:45
*** Mudpuppy_ has joined #openstack-ansible20:46
*** Mudpuppy_ has quit IRC20:47
*** Mudpuppy_ has joined #openstack-ansible20:47
*** Mudpuppy has quit IRC20:49
*** KLevenstein has joined #openstack-ansible20:50
*** Mudpuppy_ is now known as Mudpuppy20:53
openstackgerritIan Cordasco proposed stackforge/os-ansible-deployment: Update openstack/requirements SHA for 11.0.0rc3 release  https://review.openstack.org/17648721:02
clacoerwut21:05
*** KLevenstein has quit IRC21:08
*** erikmwilson has quit IRC21:11
*** stevemar has quit IRC21:13
*** openstackgerrit has quit IRC21:29
*** openstackgerrit has joined #openstack-ansible21:30
*** sacharya has quit IRC21:38
*** sdake has quit IRC21:44
palendaeclaco: I think mattt was referring to replicating the downgrading behavior on an AIO21:45
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles (icehouse)  https://review.openstack.org/17646321:53
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles (juno)  https://review.openstack.org/17645821:56
*** yaya has quit IRC21:56
*** alextricity_r has quit IRC21:57
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613421:58
cloudnullso hopfully zuul is now going . . .21:59
*** BjoernT has left #openstack-ansible22:03
*** Mudpuppy has quit IRC22:08
palendaed34dh0r53: Those 3 reviews need another +222:09
*** openstackgerrit has quit IRC22:11
*** openstackgerrit has joined #openstack-ansible22:11
clacoright about now is when they pull a maint window, because st murphy.22:15
*** jaypipes has joined #openstack-ansible22:20
*** yaya has joined #openstack-ansible22:30
*** sdake has joined #openstack-ansible22:31
*** jwagner is now known as jwagner_away22:36
*** yaya has quit IRC22:39
*** JRobinson__ has joined #openstack-ansible22:44
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Fix the package pinning problem within the galera roles  https://review.openstack.org/17613422:45
openstackgerritIan Cordasco proposed stackforge/os-ansible-deployment: Remove the rpc_deployment directory  https://review.openstack.org/17633222:50
*** alextricity_r has joined #openstack-ansible22:56
*** mnestheu1 has quit IRC22:56
*** alextricity_r has quit IRC22:57
cloudnullso the ppa system seems to be down. so st murphy is winning here.22:59
cloudnullfailed: [aio1_horizon_container-f93da9b5] => (item={'repo': 'ppa:adiscon/v8-stable', 'state': 'present'}) => {"attempts": 5, "failed": true, "item": {"repo": "ppa:adiscon/v8-stable", "state": "present"}}23:00
clacoof course. never fails.23:00
*** stevemar has joined #openstack-ansible23:01
*** fawadkhaliq has joined #openstack-ansible23:15
*** fawadkhaliq has quit IRC23:20
*** markvoelker has quit IRC23:26
sigmavirus24I'll be back on to help with the tagging of things in an hour or two23:27
*** sigmavirus24 is now known as sigmavirus24_awa23:27
*** sdake has quit IRC23:39
palendaeSeems like Zuuls either really backed up, or f'ed23:40
claconot sure if laugh or cry23:41

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!