| morganfainberg | ok | 00:00 |
|---|---|---|
| morganfainberg | it's ironic -> neutron issue | 00:00 |
| morganfainberg | great i know what to look for then. | 00:00 |
| morganfainberg | thnx. | 00:00 |
| morganfainberg | i'm keeping an eye on the keystone side seeing what might be going on there. | 00:00 |
| *** lucas-dinner has quit IRC | 00:01 | |
| jogo | we are adding os-collect-config.txt logs into lostash but they aren't getting parsed properly | 00:02 |
| jogo | lifeless: ^ | 00:03 |
| jogo | so we are wasting space for nothingness | 00:06 |
| jogo | filed a bug: bug 1373672 | 00:09 |
| uvirtbot | Launchpad bug 1373672 in tripleo "overcloud-controller0_logs/os-collect-config.txt isn't getting parsed correctly in elasticsearch" [Undecided,New] https://launchpad.net/bugs/1373672 | 00:09 |
| *** dsneddon has quit IRC | 00:10 | |
| *** dsneddon has joined #tripleo | 00:12 | |
| *** bkero has quit IRC | 00:12 | |
| *** bkero has joined #tripleo | 00:15 | |
| *** openstackgerrit has quit IRC | 00:16 | |
| lifeless | jogo: agree, we should fix the filter | 00:16 |
| lifeless | tchaypo: ironic doesn't add a port, it updates one. | 00:17 |
| *** openstackgerrit has joined #tripleo | 00:17 | |
| lifeless | tchaypo: (minor but key nit :)) | 00:17 |
| *** spzala has quit IRC | 00:25 | |
| *** jml has quit IRC | 00:28 | |
| *** dsneddon has quit IRC | 00:31 | |
| *** jml has joined #tripleo | 00:32 | |
| *** dsneddon has joined #tripleo | 00:32 | |
| *** yamahata_ has joined #tripleo | 00:36 | |
| *** dkehn has quit IRC | 00:49 | |
| *** ChuckC has quit IRC | 00:49 | |
| *** dkehn has joined #tripleo | 00:51 | |
| *** dsneddon has quit IRC | 00:53 | |
| *** dsneddon has joined #tripleo | 00:53 | |
| *** dshulyak_ has quit IRC | 01:00 | |
| tchaypo | back | 01:02 |
| *** dshulyak_ has joined #tripleo | 01:07 | |
| *** dshulyak_ has quit IRC | 01:07 | |
| morganfainberg | tchaypo, i don't think i've seen a failed | 01:12 |
| morganfainberg | but i honestly have been looking limitedly at keystone | 01:12 |
| morganfainberg | and ironic | 01:12 |
| morganfainberg | whoopse. that was a mistake | 01:13 |
| morganfainberg | hm | 01:13 |
| morganfainberg | tchaypo, well that doesn't look like failures | 01:14 |
| morganfainberg | ? | 01:14 |
| tchaypo | I think the failures are being disguised by the fact that nova is retrying 50x each node | 01:15 |
| morganfainberg | bah we don't have an index on 'valid' tokens. | 01:15 |
| tchaypo | but there's only 4 running, I would have expected 5 by now | 01:15 |
| *** jang has quit IRC | 01:17 | |
| *** jang has joined #tripleo | 01:18 | |
| morganfainberg | tchaypo, is it ok if I do a couple ugly queries against the DB? | 01:19 |
| morganfainberg | tchaypo, or would you prefer i didn't (might cause some issues with tokens) | 01:19 |
| lifeless | morganfainberg: selects orupdates? | 01:21 |
| morganfainberg | lifeless, selects | 01:21 |
| lifeless | morganfainberg: how would selects cause issues? | 01:21 |
| morganfainberg | lifeless, but they're whole table scans (32k rows) | 01:21 |
| lifeless | still, thats tiny | 01:21 |
| morganfainberg | lifeless, unindexed columns. | 01:21 |
| lifeless | still tiny | 01:21 |
| morganfainberg | yeah might cause token read locks which are unfun some times. | 01:21 |
| morganfainberg | but shouldn't be awful | 01:21 |
| morganfainberg | hmm. | 01:22 |
| morganfainberg | mysql> select id, expires, valid, trust_id, user_id from token where valid = 0; | 01:24 |
| morganfainberg | Empty set (1 min 30.46 sec) | 01:24 |
| morganfainberg | that took a lot longer than i expected. | 01:24 |
| *** ci-overcloud has joined #tripleo | 01:25 | |
| ci-overcloud | ************** ci-overcloud complete status=1 ************ | 01:25 |
| *** ci-overcloud has quit IRC | 01:25 | |
| morganfainberg | ok so, spot check, no tokens should be invalid | 01:25 |
| morganfainberg | and no 404s returned by keystone (according to the log) | 01:27 |
| lifeless | there may be a mysqlconfig issue then | 01:28 |
| lifeless | tchaypo: check the mysql memory param is set in the deploy config for hp2 ;) | 01:28 |
| tchaypo | if this run ever ends, I'm going to set max retrys back to 3, since I think the value of 50 is helping to mask the problem | 01:29 |
| tchaypo | but heck maybe the value of 50 is what we need to make it work and this whole thing is going to magically coalesce into a working cloud one day | 01:29 |
| lifeless | ruh roh | 01:31 |
| morganfainberg | tchaypo, uh i think we need to find out why retry=50 is needed then :P | 01:31 |
| lifeless | that bash exploit is rather bad | 01:31 |
| morganfainberg | lifeless, ? | 01:31 |
| lifeless | morganfainberg: | 01:31 |
| lifeless | http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-6271 | 01:31 |
| uvirtbot | lifeless: ** RESERVED ** This candidate has been reserved by an organization or individual that will use it when announcing a new security problem. When the candidate has been publicized, the details for this candidate will be provided. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-6271) | 01:31 |
| tchaypo | I've heard rumours that XSA-108 is going to be bad as well | 01:31 |
| tchaypo | but http://xenbits.xen.org/xsa/ still just says it | 01:31 |
| morganfainberg | oh | 01:31 |
| morganfainberg | gah | 01:31 |
| tchaypo | s "pre-released but embargoed" | 01:32 |
| lifeless | tchaypo: I think you'll need to upgrade bash on all the externally listening things asap :) | 01:32 |
| morganfainberg | lifeless, that looks like a rough one | 01:32 |
| * morganfainberg goes and checks bash on his server | 01:33 | |
| * tchaypo apt-get upgrades | 01:35 | |
| morganfainberg | phew mine is good | 01:35 |
| morganfainberg | tchaypo, just checked the bastion i'm logged into, yeah needs update | 01:35 |
| tchaypo | brb | 01:35 |
| *** tchaypo has quit IRC | 01:35 | |
| *** tchaypo has joined #tripleo | 01:36 | |
| *** sseago has quit IRC | 01:36 | |
| lifeless | sowe need to update the bastion and the seed and the uc and the oc | 01:38 |
| lifeless | oc - should build fresh images to pull it in | 01:38 |
| tchaypo | okay, that's better | 01:39 |
| tchaypo | upgrading bastion.hp2 now | 01:40 |
| *** marun has joined #tripleo | 01:44 | |
| *** nosnos has joined #tripleo | 01:45 | |
| *** ramishra has joined #tripleo | 01:46 | |
| * tchaypo reads the kolla thread | 01:47 | |
| tchaypo | it'd be really nice if the undercloud could resolve names | 01:53 |
| lifeless | tchaypo: there is a bug open on that | 01:56 |
| tchaypo | ... nova list is taking a while.. | 02:01 |
| StevenK | Wheee. http://www.ubuntu.com/usn/usn-2362-1/ | 02:10 |
| tchaypo | morganfainberg: take a look on window 1 - theres a bunch that have now failed and are being deleted and re-built.. and one that's ERROR|deleting just for fun | 02:10 |
| morganfainberg | tchaypo, cool thanks | 02:10 |
| *** ramishra has quit IRC | 02:11 | |
| lifeless | ok, hp1 bastion, seed, uc, oc1 and oc2 controllers upgraded | 02:11 |
| tchaypo | in unrelated news, <1sec between lightning and thunder. | 02:11 |
| tchaypo | hp2 bastion, seed, uc all upgraded. | 02:11 |
| morganfainberg | tchaypo, huh, ok so keystone doens't seem to be saying invalid tokens. | 02:11 |
| morganfainberg | according to the db | 02:11 |
| tchaypo | wheee | 02:11 |
| tchaypo | not seeing WARNINGs from ironic-conductor nor neutron-server either. | 02:13 |
| morganfainberg | oooh | 02:15 |
| morganfainberg | i wonder... | 02:15 |
| tchaypo | neutron-server does mention "no more dhcp-agents" once or twice but that doesn't seem to be fatal | 02:16 |
| morganfainberg | did we stop everything? | 02:16 |
| morganfainberg | or are somethings still running | 02:16 |
| morganfainberg | cause the keystone log stopped | 02:16 |
| tchaypo | I haven't tried to stop anything | 02:16 |
| morganfainberg | i'm *thinking* i know what is going on here. | 02:17 |
| tchaypo | ... I'm suddenly worried about what the apt-get upgrade did | 02:17 |
| morganfainberg | nah | 02:17 |
| morganfainberg | not too worried aobut that | 02:17 |
| *** marun has quit IRC | 02:17 | |
| morganfainberg | at least i don't *think* that is related here | 02:17 |
| tchaypo | What are you thinking? | 02:17 |
| lifeless | dum dum dum daaaa | 02:18 |
| morganfainberg | i think this is an issue with token delete / invalidation is causing SQL issues | 02:18 |
| morganfainberg | Sep 25 02:13:02 undercloud-undercloud-4ijtx4oh5yh7 keystone: 2014-09-25 02:13:02.349 10114 DEBUG keystone.notifications [-] Invoking callback _delete_user_tokens_callback for event identity user deleted for{'resource_info': u'a37337710c814ab1a69c80123c62d5c6'} notify_event_callbacks /opt/stack/venvs/keystone/local/lib/python2.7/site-packages/keystone/notifications.py:203 | 02:18 |
| morganfainberg | last thing in the log | 02:18 |
| morganfainberg | doing a select to see how many tokens that user has | 02:18 |
| morganfainberg | it's *really* slow select | 02:18 |
| tchaypo | lots of them? | 02:18 |
| tchaypo | or lack of index? | 02:18 |
| morganfainberg | either that or the token table is locked | 02:18 |
| morganfainberg | and we're waiting on a flush | 02:18 |
| morganfainberg | nope | 02:19 |
| morganfainberg | zero tokens for that user | 02:19 |
| tchaypo | getting back to lifeless question about memory for mysql earlier - i can't see any overall memory setting in the logs (lots of settings for memory for specific things) | 02:19 |
| tchaypo | so I'm not sure what he was referring to | 02:19 |
| lifeless | we have a parameter | 02:19 |
| lifeless | in the heat template | 02:20 |
| lifeless | that configures the mysql slab cache | 02:20 |
| lifeless | stack-show should let you see its setting | 02:20 |
| morganfainberg | oh dear. | 02:21 |
| lifeless | ? | 02:21 |
| morganfainberg | some of these keystone queries are consistently in the slow log | 02:21 |
| tchaypo | in undercloud-source.yaml? | 02:22 |
| lifeless | morganfainberg: see, as a keystone dev you should totally be running a production keystone :_ | 02:22 |
| lifeless | :) | 02:22 |
| lifeless | tchaypo: heat stack-show undercloud ... | 02:22 |
| morganfainberg | crap. why are where claused on this user_id bit | 02:22 |
| morganfainberg | ... | 02:22 |
| morganfainberg | http://paste.openstack.org/show/115155/ | 02:23 |
| morganfainberg | W.T.F. | 02:23 |
| morganfainberg | why are we where claused on the user_id.... | 02:23 |
| tchaypo | heat stack-show undercloud | grep mysql shows nothing | 02:23 |
| StevenK | tchaypo: grep -i ? | 02:23 |
| lifeless | MysqlInnodbBufferPoolSize is the parameter | 02:24 |
| morganfainberg | as much as i'd hate to do it, i bet if we used the memcache token backend it would "just workâ˘" (no we shouldn't really do this) | 02:24 |
| morganfainberg | oh yep there it is. | 02:25 |
| morganfainberg | we're deleting users. | 02:25 |
| morganfainberg | yep, that is the culprit it looks like it makes token table unhappy | 02:27 |
| lifeless | missing FK ? | 02:27 |
| lifeless | or cascade delete? | 02:27 |
| morganfainberg | lifeless, missing index it looks like | 02:28 |
| morganfainberg | it's a patch i was pushing back on this cycle because it's an alter to the token table. | 02:28 |
| morganfainberg | and no one could point out why the search on uder_id was happening as much as it was | 02:28 |
| morganfainberg | well.. looks like we delete users. | 02:28 |
| morganfainberg | lifeless, it also *feels* like a mysql tuning issue as well | 02:29 |
| morganfainberg | combine both and you get ick. | 02:29 |
| tchaypo | gah. | 02:29 |
| lifeless | heat creates temporary users | 02:29 |
| tchaypo | "MysqlInnodbBufferPoolSize": "100", | 02:29 |
| morganfainberg | doing a profile of the query | 02:29 |
| morganfainberg | it's really ugly atm | 02:29 |
| lifeless | tchaypo: that would be a little small | 02:29 |
| morganfainberg | 100? haahah | 02:29 |
| morganfainberg | oh yeah | 02:29 |
| StevenK | Isn't that the default? | 02:30 |
| StevenK | I think | 02:30 |
| morganfainberg | StevenK, mysql's defaults are insane for any load beyond "toy" level | 02:30 |
| morganfainberg | lifeless, tchaypo, http://paste.openstack.org/show/115156/ | 02:30 |
| StevenK | morganfainberg: I'd bang the Postgres drum a little here, but people don't like me doing that. :-P | 02:31 |
| morganfainberg | with the small bufferpoolsize we're causing backups on that type of query with even moderate roles. | 02:31 |
| morganfainberg | erm s/roles/tokens | 02:31 |
| morganfainberg | ack s/roles/rows | 02:31 |
| lifeless | StevenK: its a dev default for vms | 02:32 |
| lifeless | StevenK: and a case where I don't like having CI/prod be the ones to make it different | 02:32 |
| morganfainberg | lifeless, the sad part is... that query returns 0 rows. | 02:32 |
| lifeless | morganfainberg: much work | 02:32 |
| lifeless | morganfainberg: nothing to show for it | 02:32 |
| lifeless | morganfainberg: wow | 02:32 |
| morganfainberg | hahah | 02:32 |
| morganfainberg | so query | 02:32 |
| tchaypo | 100 is not the default | 02:32 |
| morganfainberg | http://paste.openstack.org/show/115157/ | 02:32 |
| tchaypo | 100 is hard-coded in devtest_undercloiud.sh | 02:33 |
| *** dshulyak_ has joined #tripleo | 02:33 | |
| StevenK | Hah. Nice. | 02:33 |
| morganfainberg | hm, let me see if i can bump that up inside mysql | 02:33 |
| morganfainberg | i'd like to see if it affects things *a lot* | 02:33 |
| StevenK | It's also hard-coded inside devtest_overcloud.sh | 02:34 |
| tchaypo | but we can override that with $HEAT_ENV | 02:34 |
| *** jml has quit IRC | 02:34 | |
| lifeless | tchaypo: it is meant to be overridden via the env file for the uc | 02:34 |
| tchaypo | actually looking at http://git.openstack.org/cgit/openstack/tripleo-incubator/tree/scripts/devtest_undercloud.sh#n235 I'm not sure if the HEAT_ENV we read in on line 196 gets respected or tampled | 02:35 |
| lifeless | tchaypo: suggest you check its there and if not add it (thats in tie) | 02:35 |
| tchaypo | HEAT_ENV? | 02:35 |
| lifeless | tchaypo: there's a heat env for the uc and oc for hp1 and hp2 in tie | 02:35 |
| lifeless | tchaypo: you may have not used it, which would cause issues ;) | 02:36 |
| *** jml has joined #tripleo | 02:36 | |
| tchaypo | "MysqlInnodbBufferPoolSize": 4096, | 02:36 |
| *** sseago has joined #tripleo | 02:37 | |
| * StevenK glares at his mirror pulse including new releases of firefox for each supported release. | 02:37 | |
| morganfainberg | ugh, ok can't set the pool size live | 02:38 |
| morganfainberg | lame | 02:38 |
| tchaypo | apparently I did not use it. | 02:38 |
| morganfainberg | also we might want more than 1 buffer pool. | 02:38 |
| StevenK | tchaypo: Huh. We have sun here now | 02:39 |
| morganfainberg | tchaypo, lifeless, mind if i index that column live? | 02:39 |
| morganfainberg | tchaypo, lifeless, see how it affects things. | 02:39 |
| tchaypo | rain is easing off for me, no more thunder | 02:39 |
| tchaypo | morganfainberg: go for it | 02:39 |
| tchaypo | i'm going to have to rebuild the undercloud soon but let's do what we can to troubleshoot first | 02:40 |
| lifeless | tchaypo: why rebuild? | 02:40 |
| StevenK | morganfainberg: Using CONCURRENTLY ? | 02:40 |
| lifeless | morganfainberg: lets fix the buffer pool size first :) | 02:40 |
| morganfainberg | lifeless, ah just tried adding the index. | 02:41 |
| morganfainberg | can drop the index at any point anyway | 02:41 |
| morganfainberg | easy enough | 02:41 |
| lifeless | it was more that its going to disrupt while it scans | 02:41 |
| lifeless | so we may as well suck up the disruption and fix the pool :) | 02:42 |
| morganfainberg | yeah sorry :P | 02:42 |
| morganfainberg | give me a minute or so to finish the indexing :( sorry already started. | 02:42 |
| tchaypo | In that case, I may as well kick off the undercloud build now? | 02:42 |
| tchaypo | kk | 02:42 |
| lifeless | tchaypo: what build? | 02:42 |
| morganfainberg | wait please don't rebuild yet! | 02:42 |
| morganfainberg | we can test to see if this will have an impact without needing to rebuild. | 02:43 |
| StevenK | morganfainberg: Huh, I don't like mysql's EXPLAIN output. But I've been using postgres' EXPLAIN for years | 02:43 |
| morganfainberg | StevenK, it's a bit wonky | 02:43 |
| morganfainberg | StevenK, have a POSTGRES example? haven't seen it in a while | 02:43 |
| morganfainberg | ok so now the question is how big of an index do we need for this.. but wow. yeah this is going to be much better with an index | 02:44 |
| tchaypo | lifeless: of the undercloud. But actually that's probably overkill for now | 02:44 |
| morganfainberg | i think we could get away with an index(6) and be overkill | 02:44 |
| StevenK | Huh. Clearly cleaning up my Launchpad developer enivronment has removed my postgres clients | 02:44 |
| lifeless | tchaypo: so when you rebuild | 02:44 |
| tchaypo | we can tweak mysql.cnf by hand for now and make sure that the next build uses the heat env | 02:44 |
| lifeless | tchaypo: its a great way to *test* that what you've fixed is fixed | 02:44 |
| lifeless | tchaypo: but its a terrible way to debug and fix issues | 02:45 |
| lifeless | tchaypo: doing a stack-update is entirely the right way to apply configuration changes | 02:45 |
| morganfainberg | http://paste.openstack.org/show/115158/ that is with index(10) | 02:45 |
| lifeless | tchaypo: but that != rebuild | 02:45 |
| morganfainberg | yeah ok i'm convinced we need the index. | 02:45 |
| morganfainberg | ok lets apply the buffer pool change and i'll check the stats as well | 02:45 |
| StevenK | morganfainberg: http://www.postgresql.org/docs/9.3/static/sql-explain.html has some small examples | 02:46 |
| tchaypo | to do a stack update I'd have to rebuild the heat template, then heat stack-update from the new template? | 02:46 |
| lifeless | tchaypo: huh | 02:46 |
| morganfainberg | lifeless, tchaypo, without losing the ksytone state (db) how do we update the my.cnf values/innodb buffer pool sizes? | 02:46 |
| StevenK | morganfainberg: I can hit up a friend to get more complex EXPLAIN output if you wish | 02:46 |
| lifeless | tchaypo: update the env file, then stack-update -e envfile undercloud | 02:46 |
| morganfainberg | StevenK, nah, that should be fine. | 02:46 |
| morganfainberg | StevenK, just curious | 02:46 |
| morganfainberg | StevenK, more like http://paste.openstack.org/show/115159/ | 02:47 |
| morganfainberg | still i can see why you like the pgsql one | 02:47 |
| StevenK | morganfainberg: I found it invaluable for working out if an index is being used, how much the postgres stats were used, and the column width of what postgres is doing | 02:47 |
| tchaypo | stack-update shouldn't lose the keystone state, should it? | 02:47 |
| morganfainberg | so i think we also need to index trust_id in the token table. | 02:48 |
| morganfainberg | grr, this makes for the migration no fun for real production loads. | 02:48 |
| morganfainberg | unless they do flush-tokens | 02:48 |
| *** rhallisey has quit IRC | 02:50 | |
| lifeless | morganfainberg: if mysql supports partial indices | 02:51 |
| morganfainberg | lifeless, it does. | 02:51 |
| lifeless | morganfainberg: you could index where the token timestamp is > time-migration-is-run | 02:51 |
| morganfainberg | lifeless, oh wait uhm. | 02:51 |
| lifeless | morganfainberg: colour me evil :) | 02:51 |
| morganfainberg | lifeless, hmm... it'd be easier to token_flush | 02:51 |
| morganfainberg | :P | 02:51 |
| morganfainberg | actually i want (if we're using sql as a backend for tokens still) to move to automatic partition tables | 02:52 |
| morganfainberg | with rotation | 02:52 |
| morganfainberg | but that is going to require fancy code in oslo.db i think | 02:52 |
| *** jml has quit IRC | 02:54 | |
| *** jml has joined #tripleo | 02:56 | |
| lifeless | morganfainberg: not to mention deploymnet | 02:57 |
| lifeless | morganfainberg: or a distributed lock manager :) | 02:57 |
| morganfainberg | lifeless, yeah. | 02:57 |
| morganfainberg | lifeless, well... new enough mysql has store proceedures (don't shoot me) | 02:58 |
| lifeless | morganfainberg: and if three clients call the stored procedure at the same time on 3 different galera nodes? | 02:58 |
| morganfainberg | i'm trying to think if I can fix this without needing to migrate the token table. | 02:58 |
| morganfainberg | low impact change. | 02:58 |
| *** untriaged-bot has joined #tripleo | 03:00 | |
| untriaged-bot | Untriaged bugs so far: | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1373063 | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1369256 | 03:00 |
| uvirtbot | Launchpad bug 1373063 in tripleo "port not added to bridge after a reboot" [Undecided,In progress] | 03:00 |
| uvirtbot | Launchpad bug 1369256 in tripleo "ubuntu: hash sum mismatch when installing packages" [Undecided,New] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1366164 | 03:00 |
| uvirtbot | Launchpad bug 1366164 in tripleo "Add domain roles for admin user" [Undecided,New] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1367742 | 03:00 |
| uvirtbot | Launchpad bug 1367742 in tripleo "keepalived brings the VIP up in multiple places on a controller reboot" [Undecided,New] | 03:00 |
| *** untriaged-bot has quit IRC | 03:00 | |
| morganfainberg | lifeless, hey don't poke too many holes in my hopes and dreams :P | 03:00 |
| tchaypo | is it okay for me to go ahead with the stack update? | 03:02 |
| morganfainberg | tchaypo, i have no issues with it | 03:04 |
| * tchaypo crosses fingers | 03:05 | |
| morganfainberg | lifeless, this also comes down to the size of the token, 42k rows should not be that hard to scan.... unless each one contains ~10k+ of data in it :( | 03:05 |
| morganfainberg | or 20k | 03:07 |
| lifeless | postgresql pushes such data out to separate storage automatically | 03:07 |
| lifeless | so table scans that don't actually filter on token content would be fast | 03:08 |
| morganfainberg | lifeless, the page sizes internal to mysql get some wonkyness. | 03:08 |
| lifeless | each row just has a uuid to the TOAST entry | 03:08 |
| morganfainberg | lifeless, the issue isn't the scan in this case, it's also the blob loading in the horrible extra field | 03:08 |
| morganfainberg | at 15k rows to scan, we're loading (bad math) ~150000000 bytes | 03:09 |
| morganfainberg | since we are selecting the data already. | 03:09 |
| lifeless | ah | 03:09 |
| morganfainberg | so we're hitting the 100M buffer pool easily | 03:09 |
| lifeless | one thing you can do in this sort of case is manually do what pg does | 03:10 |
| morganfainberg | yeah. | 03:10 |
| lifeless | that is, have a k:v table for bulk data | 03:10 |
| lifeless | and a FK reference in the main table | 03:10 |
| morganfainberg | the whole "extra" blob in keystone is awful. | 03:10 |
| lifeless | and the main table only has the fields that you select on | 03:10 |
| morganfainberg | we use it in too many places | 03:10 |
| morganfainberg | this issue screams of three things: bad schema design (we've been slowing improving it), known token issues, and then small buffers/caches | 03:12 |
| * morganfainberg sulks. | 03:12 | |
| morganfainberg | this might be another RC issue (better to find it now than need to backport it though) | 03:12 |
| *** Chao_Yan has quit IRC | 03:12 | |
| *** pcrews has quit IRC | 03:22 | |
| *** otoolee- has joined #tripleo | 03:23 | |
| tchaypo | http://www.youtube.com/watch?v=oVa4_2xXwGE | 03:24 |
| tchaypo | apparently AWS have informed people to expect their nodes to be randomly rebooted over the next day or so | 03:25 |
| tchaypo | but neglected to use the appropriate soundtrack | 03:25 |
| tchaypo | in other news, this stack still seems to be UPDATE_IN_PROGRESS... 17 minutes so far | 03:25 |
| tchaypo | but my.cnf has been updated | 03:26 |
| tchaypo | it looks like mysqld hasn't been restarted though so I'm not sure if it will have read the new config | 03:27 |
| *** Chao_Yan has joined #tripleo | 03:28 | |
| lifeless | tchaypo: os-collect-config.log is your friend | 03:30 |
| tchaypo | oh. I think I know what it's going to say | 03:31 |
| tchaypo | Nope, I was wrong. | 03:32 |
| *** bvivek has joined #tripleo | 03:35 | |
| tchaypo | oh goody. | 03:42 |
| tchaypo | heat client installed is 0.2.10 | 03:43 |
| tchaypo | 0.2.11 adds stack-cancel-update, which would be useful right now | 03:43 |
| tchaypo | Is there some other way to make the update go away? | 03:43 |
| stevebaker | I think the server side of that only just landed post j-3, you may not have it yet | 03:44 |
| tchaypo | action-suspend sounds like what I want, until I read the docs and realise that it's something completely different | 03:44 |
| tchaypo | it suspends the stack, not an action | 03:45 |
| tchaypo | stevebaker: do you have a hilight, or just super-powers? | 03:46 |
| *** Chao_Yan has quit IRC | 03:46 | |
| *** Chao_Yan has joined #tripleo | 03:48 | |
| stevebaker | I've been keeping an eye on this all day, its been most interesting | 03:50 |
| tchaypo | right now I think the only thing I can do is wait for the stack-update to time out and then try it again? | 03:52 |
| tchaypo | os-collect-config is barfing in an error in the template so it will never apply.. | 03:52 |
| stevebaker | without cancel-update, that is your option | 03:53 |
| tchaypo | wheeee | 03:53 |
| tchaypo | next time I | 03:53 |
| tchaypo | 'm using a shorter timeout | 03:54 |
| lifeless | tchaypo: whats theerror ? | 03:54 |
| tchaypo | .... the timeout last time was 360 minutes. that's going to take a while. | 03:55 |
| lifeless | tchaypo: (you've summarise but not actuall explained) | 03:55 |
| lifeless | tchaypo: or pastebinned | 03:55 |
| tchaypo | http://paste.openstack.org/show/115170/ | 03:55 |
| tchaypo | I tried to merge hp2_uc_env.json with the existing undercloud_env.json and ended up with conflicting neutron parameters | 03:56 |
| tchaypo | os-collect-config must do something to tell heat the update completed. | 03:56 |
| *** ramishra has joined #tripleo | 03:57 | |
| lifeless | so to work around that you can just signal complete by hand from within the machine | 03:57 |
| lifeless | which is the last script in the post .d dir | 03:58 |
| morganfainberg | lifeless, tchaypo: ok so restored a patch/bug for RC blocker on this | 03:59 |
| morganfainberg | should help a lot to have the columns indexed | 03:59 |
| morganfainberg | thanks for being patient with me, i feel bad for blocking the change earlier in the cycle but couldn't figure out what was causing the spam to that where clause in the select | 03:59 |
| tchaypo | morganfainberg: we're happy to play on your guilt | 04:00 |
| tchaypo | I mean, no problem, it seems like a reasonable thing to do if you can't figure out where the spam came from :) | 04:00 |
| lifeless | morganfainberg: well, whoever put the patch forward could presumablyhave gotten you d=soem debug traces | 04:01 |
| tchaypo | wheee | 04:03 |
| tchaypo | last line of os-collect-config log that I can see is | 04:03 |
| tchaypo | Sep 25 04:01:49 undercloud-undercloud-4ijtx4oh5yh7 os-collect-config: + ifdown eth2 | 04:03 |
| lifeless | tchaypo: you may need the serial console | 04:04 |
| tchaypo | except that we don't have serial consoles | 04:04 |
| * tchaypo sets up vnc port-forward | 04:04 | |
| lifeless | tchaypo: we don't? iLO ? | 04:04 |
| tchaypo | oh yeah | 04:05 |
| tchaypo | real hardware! | 04:05 |
| morganfainberg | lifeless, tchaypo: https://review.openstack.org/#/c/102041/ | 04:06 |
| lifeless | \o/ | 04:07 |
| *** lynxman has quit IRC | 04:10 | |
| *** lynxman has joined #tripleo | 04:10 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/python-tuskarclient: Updated from global requirements https://review.openstack.org/116415 | 04:11 |
| *** ChuckC_ has joined #tripleo | 04:11 | |
| tchaypo | and now, for the first time, I need to know the root password of a node | 04:13 |
| tchaypo | or heat-admin would suffice | 04:13 |
| *** bvivek has quit IRC | 04:15 | |
| *** Azher has joined #tripleo | 04:16 | |
| lifeless | tchaypo: stack:stack | 04:22 |
| tchaypo | of course | 04:22 |
| tchaypo | good old stackuser element | 04:22 |
| tchaypo | br_ctlplane now has an IP in the public net | 04:23 |
| lifeless | messed up :) | 04:24 |
| tchaypo | ho hum. stoppped responding on serial console now | 04:31 |
| tchaypo | i smell a power-cycle coming on | 04:32 |
| *** nikunj2512 has joined #tripleo | 04:34 | |
| nikunj2512 | Please review this patch - https://review.openstack.org/#/c/111980/ | 04:35 |
| *** killer_prince is now known as lazy_prince | 04:37 | |
| *** morgan_remote_ has joined #tripleo | 04:40 | |
| *** marun has joined #tripleo | 04:46 | |
| tchaypo | It's getting let for morganfainberg again, and I need to eat before my head gets so light it floats off, and this server isn't talking on serial any more. | 04:50 |
| tchaypo | I'm going to give up and rebuild the undercloud witht he working heat env, then give it a tiny little update that just updates mysql instead of the other things it just tried to change at the same time, and then we can take another crack tomorrow | 04:51 |
| tchaypo | I'll try another overcloud deploy - with debugging turned on, but with retries set to 3 instead of 50, i think | 04:51 |
| morgan_remote_ | Works for me. Hope to have the index merge tomorrow (pending gate slowness of course) | 04:53 |
| *** lsmola has quit IRC | 04:57 | |
| *** dshulyak_ has quit IRC | 04:58 | |
| *** lsmola has joined #tripleo | 05:11 | |
| *** pradeep has joined #tripleo | 05:18 | |
| *** tzumainn has quit IRC | 05:20 | |
| *** ramishra has quit IRC | 05:25 | |
| *** bvivek has joined #tripleo | 05:34 | |
| *** Ng has quit IRC | 05:47 | |
| *** Ng has joined #tripleo | 05:48 | |
| *** ChanServ sets mode: +v Ng | 05:48 | |
| *** victor_lowther__ has quit IRC | 05:48 | |
| *** victor_lowther__ has joined #tripleo | 05:49 | |
| *** pensu has joined #tripleo | 06:07 | |
| *** vdreamarkitex has quit IRC | 06:08 | |
| *** vdreamarkitex has joined #tripleo | 06:09 | |
| *** mrunge has joined #tripleo | 06:10 | |
| *** ramishra has joined #tripleo | 06:12 | |
| *** ramishra has quit IRC | 06:17 | |
| *** pradeep has quit IRC | 06:22 | |
| *** andreaf has quit IRC | 06:23 | |
| *** pradeep has joined #tripleo | 06:26 | |
| *** ramishra has joined #tripleo | 06:28 | |
| *** k4n0 has joined #tripleo | 06:37 | |
| *** ramishra has quit IRC | 06:38 | |
| *** jprovazn has joined #tripleo | 06:43 | |
| *** jtomasek has joined #tripleo | 06:44 | |
| *** rdopieralski has joined #tripleo | 06:46 | |
| *** dtantsur|afk is now known as dtantsur | 06:56 | |
| *** marun has quit IRC | 06:57 | |
| openstackgerrit | A change was merged to openstack-infra/tripleo-ci: Add TRIPLEO_DEBUG var for enabling debug logging https://review.openstack.org/119214 | 07:02 |
| *** pensu1 has joined #tripleo | 07:07 | |
| *** pensu has quit IRC | 07:09 | |
| *** pensu1 has quit IRC | 07:11 | |
| shadower | lsmola: ping | 07:12 |
| *** ifarkas has joined #tripleo | 07:12 | |
| *** jcoufal has joined #tripleo | 07:12 | |
| *** Chao_Yan has quit IRC | 07:15 | |
| openstackgerrit | A change was merged to openstack/os-cloud-config: Updated from global requirements https://review.openstack.org/122651 | 07:24 |
| *** pensu has joined #tripleo | 07:27 | |
| openstackgerrit | A change was merged to openstack/tuskar-ui: Set pki_setup False https://review.openstack.org/123752 | 07:29 |
| openstackgerrit | A change was merged to openstack/tuskar-ui: Stop using intersphinx https://review.openstack.org/123647 | 07:30 |
| *** gfidente has joined #tripleo | 07:33 | |
| *** saurabhs has left #tripleo | 07:35 | |
| *** pradeep has quit IRC | 07:36 | |
| *** pradeep has joined #tripleo | 07:44 | |
| *** jcoufal has quit IRC | 07:46 | |
| *** jcoufal has joined #tripleo | 07:46 | |
| lsmola | shadower: pong | 07:47 |
| shadower | lsmola: hey, remember how you sent peter the list of elements necessary to build the swift images? | 07:48 |
| shadower | lsmola: do you remember the link? I can't find it | 07:48 |
| shadower | testing the block storage templates now and want to do the object store after | 07:49 |
| pensu | Hi all, I have been trying to work with Ironic in RDO instack tripleo setup....here is the error I am getting.... | 07:49 |
| pensu | $ ironic node-list | 07:49 |
| pensu | publicURL endpoint for baremetal service not found | 07:49 |
| pensu | Any idea which endpoint I need to fix? | 07:49 |
| *** pblaho has joined #tripleo | 07:50 | |
| shadower | dtantsur: ^ any ideas? | 07:50 |
| lsmola | shadower: https://wiki.openstack.org/wiki/Tuskar/Devtest | 07:50 |
| StevenK | pensu: If nova baremetal-node-list works, then it's using nova-baremetal and not ironic | 07:51 |
| shadower | omg am I seriously that daft? | 07:51 |
| shadower | lsmola: I am! Thanks! | 07:52 |
| dtantsur | pensu, did you set endpoint in keystone? | 07:55 |
| dtantsur | pensu, http://docs.openstack.org/developer/ironic/deploy/install-guide.html#configure-identity-service-for-bare-metal | 07:57 |
| StevenK | devtest will do that for you if USE_IRONIC is not 0 | 07:57 |
| StevenK | And its default is 1 | 07:57 |
| dtantsur | right, I didn't notice it's instack | 07:58 |
| dtantsur | I'd still double-check keystone endpoints | 07:59 |
| *** rushiagr_away is now known as rushiagr | 08:00 | |
| *** jp_at_hp has joined #tripleo | 08:01 | |
| d0ugal | The nodes.csv file, is that how the Tuskar UI fines the baremetal vms? | 08:05 |
| *** dkehn has quit IRC | 08:05 | |
| * d0ugal is watching tzumainn's demo | 08:06 | |
| *** dkehn has joined #tripleo | 08:07 | |
| *** yuanying has quit IRC | 08:08 | |
| *** jistr has joined #tripleo | 08:09 | |
| *** morgan_remote_ has quit IRC | 08:10 | |
| pensu | StevenK: yeah, looks like it's using baremetal only. Do we support ironic? | 08:10 |
| tchaypo | who is we? | 08:11 |
| *** athomas has joined #tripleo | 08:11 | |
| *** derekh has joined #tripleo | 08:13 | |
| pensu | tchaypo: sorry, I meant does tripleo support ironic? | 08:13 |
| tchaypo | hrm. 4 hours later and my undercloud is still CREATE_IN_PROGRESS | 08:13 |
| tchaypo | that seems suboptimal | 08:13 |
| tchaypo | pensu: use - as StevenK said, just set USE_IRONIC to non-0 | 08:14 |
| tchaypo | s/use/yes/ | 08:14 |
| *** lucasagomes has joined #tripleo | 08:15 | |
| derekh | tchaypo: your undercloud thats still CREATE_IN_PROGRESS was the image built in the last day or so ? | 08:15 |
| tchaypo | Sep 25 06:19 undercloud.qcow2 | 08:15 |
| tchaypo | yep, 3 hours ago | 08:16 |
| derekh | tchaypo: could this be your problem https://bugs.launchpad.net/tripleo/+bug/1373430 | 08:16 |
| uvirtbot | Launchpad bug 1373430 in horizon "Error while compressing files" [Critical,In progress] | 08:16 |
| tchaypo | derekh: i love you | 08:16 |
| derekh | ;-) | 08:16 |
| tchaypo | I always feel guilty when people hand me answers like this | 08:16 |
| tchaypo | I feel like I chated and mised an opportunity to learn something, or something like that. | 08:16 |
| tchaypo | I think it's protestant-work-ethic guilt | 08:17 |
| tchaypo | s/chated/cheated/ | 08:17 |
| derekh | Should I just let you find the horizon patch yourself so ? | 08:17 |
| tchaypo | oh no | 08:20 |
| tchaypo | a bit of guilt is a wonderfulbit of relaxation | 08:21 |
| tchaypo | 123779 is the patch, right? | 08:21 |
| derekh | tchaypo: yup | 08:23 |
| * tchaypo pulls patch | 08:24 | |
| *** pradeep has quit IRC | 08:25 | |
| * shadower makes sure not to rebuild his images | 08:25 | |
| tchaypo | well, while that's happening, I'm just going to put the laptop down ad watch people playing squash | 08:26 |
| *** bvivek has quit IRC | 08:29 | |
| *** pradeep has joined #tripleo | 08:34 | |
| *** andreaf_ is now known as andreaf | 08:39 | |
| *** bvivek has joined #tripleo | 08:44 | |
| *** openstackgerrit has quit IRC | 08:48 | |
| *** rushiagr is now known as rushiagr_away | 08:52 | |
| *** untriaged-bot has joined #tripleo | 09:00 | |
| untriaged-bot | Untriaged bugs so far: | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1373063 | 09:00 |
| uvirtbot | Launchpad bug 1373063 in tripleo "port not added to bridge after a reboot" [Undecided,In progress] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1369256 | 09:00 |
| uvirtbot | Launchpad bug 1369256 in tripleo "ubuntu: hash sum mismatch when installing packages" [Undecided,New] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1366164 | 09:00 |
| uvirtbot | Launchpad bug 1366164 in tripleo "Add domain roles for admin user" [Undecided,New] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1367742 | 09:00 |
| uvirtbot | Launchpad bug 1367742 in tripleo "keepalived brings the VIP up in multiple places on a controller reboot" [Undecided,New] | 09:00 |
| *** untriaged-bot has quit IRC | 09:00 | |
| *** lazy_prince has quit IRC | 09:08 | |
| *** Dafna has joined #tripleo | 09:19 | |
| *** pradeep has quit IRC | 09:22 | |
| *** pensu has quit IRC | 09:22 | |
| *** pradeep has joined #tripleo | 09:23 | |
| *** pensu has joined #tripleo | 09:23 | |
| *** dsneddon has quit IRC | 09:24 | |
| *** dsneddon has joined #tripleo | 09:25 | |
| *** dtantsur is now known as dtantsur|brb | 09:25 | |
| *** killer_prince has joined #tripleo | 09:26 | |
| *** killer_prince is now known as lazy_prince | 09:27 | |
| *** lazy_prince has quit IRC | 09:48 | |
| *** lazy_prince has joined #tripleo | 09:48 | |
| *** pradeep1 has joined #tripleo | 09:55 | |
| *** pradeep has quit IRC | 09:56 | |
| *** rushiagr_away is now known as rushiagr | 09:56 | |
| *** loki184 has joined #tripleo | 10:07 | |
| *** mrunge has quit IRC | 10:16 | |
| *** yamahata_ has quit IRC | 10:20 | |
| *** akrivoka has joined #tripleo | 10:23 | |
| *** Azher has quit IRC | 10:38 | |
| *** bvivek has quit IRC | 10:39 | |
| *** pradeep1 has quit IRC | 10:42 | |
| *** pradeep has joined #tripleo | 10:42 | |
| *** pradeep has quit IRC | 10:45 | |
| *** dtantsur|brb is now known as dtantsur | 10:48 | |
| *** ramishra has joined #tripleo | 10:49 | |
| *** pensu1 has joined #tripleo | 10:49 | |
| *** pensu has quit IRC | 10:50 | |
| *** pradeep has joined #tripleo | 10:52 | |
| *** pradipta_away has quit IRC | 10:57 | |
| *** bvivek has joined #tripleo | 10:57 | |
| *** pensu1 has quit IRC | 11:13 | |
| *** pradeep has quit IRC | 11:17 | |
| *** pradeep1 has joined #tripleo | 11:17 | |
| *** lucasagomes is now known as lucas-hungry | 11:20 | |
| *** loki184 has quit IRC | 11:35 | |
| *** jang has quit IRC | 11:35 | |
| *** jang has joined #tripleo | 11:39 | |
| *** pradeep1 has quit IRC | 11:41 | |
| *** regebro has quit IRC | 11:44 | |
| *** nikunj2512 has quit IRC | 11:44 | |
| *** pradeep has joined #tripleo | 11:44 | |
| *** denis_makogon has joined #tripleo | 11:45 | |
| *** bvivek has quit IRC | 11:51 | |
| *** denis_makogon has left #tripleo | 11:51 | |
| *** bvivek has joined #tripleo | 11:52 | |
| *** julim has joined #tripleo | 11:55 | |
| *** jang has quit IRC | 11:57 | |
| *** jang has joined #tripleo | 11:58 | |
| *** jang1 has joined #tripleo | 12:01 | |
| *** regebro has joined #tripleo | 12:04 | |
| *** zz_naotok is now known as naotok | 12:07 | |
| *** pradeep has quit IRC | 12:09 | |
| *** morazi has joined #tripleo | 12:12 | |
| *** tzumainn has joined #tripleo | 12:14 | |
| *** rhallisey has joined #tripleo | 12:21 | |
| *** jdob has joined #tripleo | 12:21 | |
| *** rushiagr is now known as rushiagr_away | 12:26 | |
| *** dprince has joined #tripleo | 12:26 | |
| *** rushiagr_away is now known as rushiagr | 12:28 | |
| *** sdake has quit IRC | 12:30 | |
| *** sdake has joined #tripleo | 12:31 | |
| *** lucas-hungry is now known as lucasagomes | 12:37 | |
| *** akrivoka has quit IRC | 12:38 | |
| *** akrivoka has joined #tripleo | 12:47 | |
| *** naotok is now known as zz_naotok | 12:48 | |
| *** regebro has quit IRC | 12:51 | |
| *** pensu has joined #tripleo | 12:51 | |
| *** regebro has joined #tripleo | 12:54 | |
| *** andreaf is now known as andreaf_ | 12:59 | |
| *** noslzzp has joined #tripleo | 12:59 | |
| *** regebro has quit IRC | 13:08 | |
| *** regebro has joined #tripleo | 13:09 | |
| *** pblaho has quit IRC | 13:18 | |
| *** bvivek has quit IRC | 13:33 | |
| *** ramishra has quit IRC | 13:46 | |
| *** ramishra has joined #tripleo | 13:47 | |
| *** sballe_ has joined #tripleo | 13:54 | |
| *** pcrews has joined #tripleo | 13:56 | |
| *** regebro has quit IRC | 13:57 | |
| *** morgan_remote_ has joined #tripleo | 13:59 | |
| *** nosnos has quit IRC | 14:00 | |
| *** nosnos has joined #tripleo | 14:00 | |
| *** nosnos has quit IRC | 14:04 | |
| *** jprovazn has quit IRC | 14:05 | |
| *** ramishra has quit IRC | 14:11 | |
| *** pensu has quit IRC | 14:12 | |
| *** mestery has quit IRC | 14:14 | |
| *** mestery has joined #tripleo | 14:14 | |
| *** openstackgerrit has joined #tripleo | 14:15 | |
| *** ChuckC has joined #tripleo | 14:19 | |
| *** ChuckC_ has quit IRC | 14:20 | |
| *** ramishra has joined #tripleo | 14:20 | |
| *** bvivek has joined #tripleo | 14:22 | |
| *** ChuckC has quit IRC | 14:25 | |
| *** pensu has joined #tripleo | 14:26 | |
| *** sballe_ has quit IRC | 14:30 | |
| *** ramishra has quit IRC | 14:30 | |
| *** ramishra has joined #tripleo | 14:31 | |
| *** openstackgerrit has quit IRC | 14:40 | |
| *** openstackgerrit has joined #tripleo | 14:41 | |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack-infra/tripleo-ci: Update virsh ssh commands to support setting the locale https://review.openstack.org/124072 | 14:43 |
| *** ramishra has quit IRC | 14:46 | |
| *** pensu has quit IRC | 14:51 | |
| *** andreaf_ is now known as andreaf | 14:52 | |
| *** ramishra has joined #tripleo | 14:53 | |
| *** untriaged-bot has joined #tripleo | 15:00 | |
| untriaged-bot | Untriaged bugs so far: | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1373063 | 15:00 |
| uvirtbot | Launchpad bug 1373063 in tripleo "port not added to bridge after a reboot" [Undecided,In progress] | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1369256 | 15:00 |
| *** dtantsur is now known as dtantsur|brb | 15:00 | |
| uvirtbot | Launchpad bug 1369256 in tripleo "ubuntu: hash sum mismatch when installing packages" [Undecided,New] | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1366164 | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1367742 | 15:00 |
| uvirtbot | Launchpad bug 1366164 in tripleo "Add domain roles for admin user" [Undecided,New] | 15:00 |
| uvirtbot | Launchpad bug 1367742 in tripleo "keepalived brings the VIP up in multiple places on a controller reboot" [Undecided,New] | 15:00 |
| *** untriaged-bot has quit IRC | 15:00 | |
| *** andreaf is now known as andreaf_ | 15:01 | |
| *** akrivoka has quit IRC | 15:02 | |
| *** bcrochet has quit IRC | 15:04 | |
| *** beekneemech is now known as bnemec | 15:05 | |
| *** pensu has joined #tripleo | 15:05 | |
| *** bcrochet has joined #tripleo | 15:09 | |
| *** marun has joined #tripleo | 15:10 | |
| *** ramishra has quit IRC | 15:10 | |
| *** jang1 has quit IRC | 15:11 | |
| *** jang has quit IRC | 15:11 | |
| *** ChuckC has joined #tripleo | 15:12 | |
| *** jang has joined #tripleo | 15:14 | |
| *** dshulyak_ has joined #tripleo | 15:20 | |
| *** bvivek has quit IRC | 15:22 | |
| *** rdopieralski has quit IRC | 15:22 | |
| *** yamahata_ has joined #tripleo | 15:23 | |
| *** athomas has quit IRC | 15:24 | |
| *** athomas has joined #tripleo | 15:27 | |
| *** andreaf has joined #tripleo | 15:30 | |
| *** rwsu has quit IRC | 15:33 | |
| *** andreaf has quit IRC | 15:37 | |
| *** andreaf has joined #tripleo | 15:38 | |
| *** k4n0 has quit IRC | 15:43 | |
| *** pensu has quit IRC | 15:45 | |
| *** rwsu has joined #tripleo | 15:45 | |
| *** morazi has quit IRC | 15:45 | |
| *** morazi has joined #tripleo | 15:48 | |
| *** slagle_ has joined #tripleo | 15:52 | |
| *** slagle_ has quit IRC | 15:52 | |
| *** slagle has quit IRC | 15:53 | |
| *** slagle has joined #tripleo | 15:54 | |
| *** jistr has quit IRC | 15:54 | |
| *** cwolferh_ has joined #tripleo | 15:57 | |
| openstackgerrit | Ryan Brady proposed a change to openstack/diskimage-builder: Enable RHEL Registration https://review.openstack.org/123145 | 16:03 |
| *** ChuckC has quit IRC | 16:06 | |
| openstackgerrit | Ryan Brady proposed a change to openstack/diskimage-builder: Enable RHEL Registration https://review.openstack.org/123145 | 16:07 |
| *** dtantsur|brb is now known as dtantsur | 16:08 | |
| openstackgerrit | Ryan Brady proposed a change to openstack/diskimage-builder: Enable RHEL Registration https://review.openstack.org/123145 | 16:09 |
| *** rhallisey has quit IRC | 16:10 | |
| *** morgan_remote_ has quit IRC | 16:10 | |
| *** rushiagr is now known as rushiagr_away | 16:11 | |
| *** lsmola has quit IRC | 16:12 | |
| *** Dafna has quit IRC | 16:14 | |
| *** jp_at_hp has quit IRC | 16:19 | |
| *** jml has quit IRC | 16:22 | |
| *** dshulyak_ has quit IRC | 16:23 | |
| *** jml has joined #tripleo | 16:26 | |
| *** Azher has joined #tripleo | 16:28 | |
| *** pensu has joined #tripleo | 16:28 | |
| *** jml has quit IRC | 16:31 | |
| *** jml has joined #tripleo | 16:33 | |
| *** openstackgerrit has quit IRC | 16:33 | |
| *** cody-somerville has quit IRC | 16:33 | |
| *** dshulyak_ has joined #tripleo | 16:34 | |
| *** cody-somerville has joined #tripleo | 16:36 | |
| *** cody-somerville has joined #tripleo | 16:36 | |
| *** athomas has quit IRC | 16:38 | |
| *** dshulyak_ has quit IRC | 16:39 | |
| *** dshulyak_ has joined #tripleo | 16:41 | |
| *** pradeep has joined #tripleo | 16:42 | |
| *** jml has quit IRC | 16:43 | |
| *** dshulyak_ has quit IRC | 16:45 | |
| *** derekh has quit IRC | 16:48 | |
| *** lucasagomes has quit IRC | 16:48 | |
| *** rushiagr_away is now known as rushiagr | 16:48 | |
| *** jml has joined #tripleo | 16:49 | |
| *** openstackgerrit has joined #tripleo | 16:53 | |
| *** rhallisey has joined #tripleo | 16:54 | |
| *** rhallisey has quit IRC | 16:56 | |
| *** rhallisey has joined #tripleo | 16:56 | |
| *** dshulyak_ has joined #tripleo | 16:59 | |
| *** jdob has quit IRC | 17:00 | |
| *** jdob has joined #tripleo | 17:00 | |
| *** pradeep has quit IRC | 17:01 | |
| *** dshulyak_ has quit IRC | 17:02 | |
| *** dshulyak_ has joined #tripleo | 17:06 | |
| *** michchap has quit IRC | 17:09 | |
| *** pensu has quit IRC | 17:10 | |
| *** marun has quit IRC | 17:10 | |
| *** tzumainn has quit IRC | 17:16 | |
| *** tzumainn has joined #tripleo | 17:23 | |
| *** grauwulf has quit IRC | 17:23 | |
| *** pensu has joined #tripleo | 17:25 | |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-image-elements: Fix RabbitMQ element clustering start and stop https://review.openstack.org/115524 | 17:28 |
| *** jdob has quit IRC | 17:29 | |
| *** jdob has joined #tripleo | 17:30 | |
| *** ChuckC has joined #tripleo | 17:30 | |
| *** pensu has quit IRC | 17:30 | |
| *** grauwulf has joined #tripleo | 17:31 | |
| *** openstackgerrit has quit IRC | 17:33 | |
| *** sballe_ has joined #tripleo | 17:36 | |
| *** pradeep has joined #tripleo | 17:38 | |
| *** jtomasek has quit IRC | 17:38 | |
| *** tzumainn has quit IRC | 17:39 | |
| *** openstackgerrit has joined #tripleo | 17:39 | |
| *** morgan_remote_ has joined #tripleo | 17:40 | |
| *** pcrews has quit IRC | 17:41 | |
| *** tzumainn has joined #tripleo | 17:42 | |
| *** pcrews has joined #tripleo | 17:42 | |
| *** ifarkas has quit IRC | 17:43 | |
| *** pensu has joined #tripleo | 17:45 | |
| *** gfidente has quit IRC | 17:47 | |
| *** jehb has joined #tripleo | 17:48 | |
| *** penick has joined #tripleo | 17:57 | |
| *** Daviey has quit IRC | 18:07 | |
| *** pradeep has quit IRC | 18:08 | |
| *** jang1 has joined #tripleo | 18:09 | |
| *** Daviey has joined #tripleo | 18:19 | |
| *** rushiagr is now known as rushiagr_away | 18:20 | |
| *** dtantsur is now known as dtantsur|afk | 18:22 | |
| *** dprince has quit IRC | 18:24 | |
| *** lazy_prince is now known as killer_prince | 18:25 | |
| openstackgerrit | Erik Colnick proposed a change to openstack/tripleo-specs: TripleO DVR enablement https://review.openstack.org/113983 | 18:28 |
| openstackgerrit | Erik Colnick proposed a change to openstack/tripleo-specs: TripleO DVR enablement https://review.openstack.org/113983 | 18:30 |
| *** pcrews has quit IRC | 18:32 | |
| *** sballe_ has quit IRC | 18:36 | |
| *** sballe_ has joined #tripleo | 18:37 | |
| *** jcoufal has quit IRC | 18:40 | |
| *** dshulyak_ has quit IRC | 18:45 | |
| *** cody-somerville has quit IRC | 18:52 | |
| *** cody-somerville has joined #tripleo | 18:54 | |
| *** andreaf has quit IRC | 18:59 | |
| *** andreaf has joined #tripleo | 18:59 | |
| *** tzumainn has quit IRC | 19:08 | |
| *** tzumainn has joined #tripleo | 19:10 | |
| *** pcrews has joined #tripleo | 19:12 | |
| *** Azher has quit IRC | 19:16 | |
| *** cwolferh_ has quit IRC | 19:25 | |
| *** jehb has quit IRC | 19:26 | |
| *** morgan_remote_ has quit IRC | 19:50 | |
| openstackgerrit | A change was merged to openstack/python-tuskarclient: Stop using intersphinx https://review.openstack.org/123648 | 19:57 |
| openstackgerrit | A change was merged to openstack/tuskar: Stop using intersphinx https://review.openstack.org/123645 | 19:59 |
| *** jml has quit IRC | 20:02 | |
| *** yamahata_ has quit IRC | 20:02 | |
| *** radez has joined #tripleo | 20:05 | |
| *** jml has joined #tripleo | 20:06 | |
| *** jml has joined #tripleo | 20:06 | |
| *** radez is now known as radez_g0n3 | 20:08 | |
| *** radez_g0n3 is now known as radez | 20:10 | |
| *** radez is now known as radez_g0n3 | 20:11 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 20:17 |
| *** andreaf has quit IRC | 20:23 | |
| *** andreaf has joined #tripleo | 20:23 | |
| *** cwolferh has joined #tripleo | 20:27 | |
| *** pcrews has quit IRC | 20:34 | |
| openstackgerrit | Erik Colnick proposed a change to openstack/tripleo-specs: TripleO DVR enablement https://review.openstack.org/113983 | 20:41 |
| *** untriaged-bot has joined #tripleo | 21:00 | |
| untriaged-bot | Untriaged bugs so far: | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1373063 | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1369256 | 21:00 |
| uvirtbot | Launchpad bug 1373063 in tripleo "port not added to bridge after a reboot" [Undecided,In progress] | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1366164 | 21:00 |
| uvirtbot | Launchpad bug 1369256 in tripleo "ubuntu: hash sum mismatch when installing packages" [Undecided,New] | 21:00 |
| uvirtbot | Launchpad bug 1366164 in tripleo "Add domain roles for admin user" [Undecided,New] | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1367742 | 21:00 |
| uvirtbot | Launchpad bug 1367742 in tripleo "keepalived brings the VIP up in multiple places on a controller reboot" [Undecided,New] | 21:00 |
| *** untriaged-bot has quit IRC | 21:00 | |
| *** jdob has quit IRC | 21:02 | |
| *** pensu has quit IRC | 21:04 | |
| *** sdake_ has joined #tripleo | 21:05 | |
| *** pcrews has joined #tripleo | 21:05 | |
| *** jml has quit IRC | 21:19 | |
| *** jml has joined #tripleo | 21:20 | |
| jogo | lifeless: I am happy to +2 https://review.openstack.org/#/c/121155/ if it gets a few +1s from tripleo cores | 21:23 |
| SpamapS | jogo: I +1'd ;) | 21:29 |
| jogo | SpamapS lifeless: thanks | 21:30 |
| *** jang1 has quit IRC | 21:40 | |
| *** sdake_ has quit IRC | 21:41 | |
| *** ccrouch has quit IRC | 21:44 | |
| tchaypo | morganfainberg: lifeless: I need to take the cats to the vet this morning, probably not properly around for another 2.5 hours yet. It looks like the second undercloud build I kicked off last night also got bitten by the horizon bug, so I'll need to fix that when I come back | 21:55 |
| tchaypo | in short I don't expect much progress today | 21:55 |
| *** ccrouch has joined #tripleo | 22:04 | |
| *** morgan_remote_ has joined #tripleo | 22:12 | |
| *** penick has quit IRC | 22:18 | |
| SpamapS | haha.. I can't tell if this is real or not http://uselessd.darknedgy.net/ | 22:19 |
| lifeless | SpamapS: tis real | 22:20 |
| lifeless | tchaypo: ok | 22:21 |
| SpamapS | lifeless: there's so much snark.. even the logo.. the kitchen sink.. it's brilliant | 22:23 |
| lifeless | SpamapS: I know right :0 | 22:27 |
| bkero | It has roughly a useless amount of snark compared to systemd. | 22:30 |
| *** penick has joined #tripleo | 22:31 | |
| morganfainberg | tchaypo, lifeless, sounds good. still battleing the gate for the index update, though you should be able to apply it directly | 22:34 |
| *** andreaf has quit IRC | 22:38 | |
| *** andreaf has joined #tripleo | 22:38 | |
| *** jml has quit IRC | 22:52 | |
| *** penick has quit IRC | 22:53 | |
| *** jml has joined #tripleo | 22:55 | |
| *** sdake_ has joined #tripleo | 23:00 | |
| *** yuanying has joined #tripleo | 23:10 | |
| *** penick has joined #tripleo | 23:11 | |
| *** sdake has quit IRC | 23:12 | |
| *** ccrouch has quit IRC | 23:14 | |
| *** ChuckC has quit IRC | 23:14 | |
| *** ccrouch has joined #tripleo | 23:34 | |
| *** sdake_ has quit IRC | 23:34 | |
| *** sdake has joined #tripleo | 23:36 | |
| *** sballe_ has quit IRC | 23:39 | |
| *** boris-42 has quit IRC | 23:47 | |
| *** pradeep has joined #tripleo | 23:48 | |
| *** boris-42 has joined #tripleo | 23:48 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!