| SpamapS | started tripleo-cd | 00:03 |
|---|---|---|
| *** morazi has joined #tripleo | 00:03 | |
| SpamapS | /opt/stack is now a bind mount to /var/lib/moved/opt/stack | 00:03 |
| SpamapS | and turned off debug logging in neutron-server and nova | 00:03 |
| SpamapS | hopefully we won't use up all 4.9G of space we have now :-P | 00:03 |
| lifeless | ok cool | 00:05 |
| lifeless | so, I'm going to stop tripleo-cd just in case | 00:06 |
| lifeless | they do use the same images, for instance (which should be fine) | 00:06 |
| *** cody-somerville has joined #tripleo | 00:06 | |
| *** cody-somerville has joined #tripleo | 00:06 | |
| StevenK | lifeless: Can we book now? | 00:08 |
| lifeless | cody-somerville: oh hi! | 00:08 |
| lifeless | StevenK: let me find out | 00:08 |
| cody-somerville | lifeless: Hey. | 00:08 |
| lifeless | cody-somerville: had a chance to look at corp mail today ? - I pinged you earlier with a few q's | 00:09 |
| cody-somerville | lifeless: just saw that now, aye. | 00:09 |
| cody-somerville | lifeless: So I'm still working on block room rate (had call today about it) | 00:09 |
| cody-somerville | lifeless: I recommend that you confirm attendee list and then get people to book air travel | 00:09 |
| lifeless | cody-somerville: whats the ETA - I expect plane prices to being skyrocketing nowish for the long haulers | 00:09 |
| lifeless | cody-somerville: can folk choose their own days when the time to book hotels arrives? | 00:10 |
| cody-somerville | lifeless: Did you get e-mail I sent you with link to the budget? | 00:10 |
| lifeless | cody-somerville: I don't see it in my hp mail | 00:10 |
| cody-somerville | sent it to your personal one | 00:10 |
| lifeless | ah, let me check | 00:10 |
| cody-somerville | So room rate thing should def be a go, so people should book flights and not hotel at this point | 00:11 |
| lifeless | cody-somerville: got it. ok. | 00:12 |
| StevenK | I shall book flights to SJC, then | 00:12 |
| lifeless | cody-somerville: and they can then do e.g. leave friday, or for long haulers arrive saturday in flights - and we'll make thehotel line up after that ? | 00:12 |
| cody-somerville | lifeless: so I imagine what will happen is that we'll negotiate a rate for set period of time. If people's bookings are outside of that, they'll pay different rate. But I'll bring this up with our SmartMeeting planner and will let you know when I know. | 00:13 |
| cody-somerville | (different rate for just those nights outside of block room dates that is) | 00:13 |
| lifeless | cody-somerville: ok. great. | 00:13 |
| StevenK | cody-somerville: And if they leave earlier, it's fine too? | 00:14 |
| cody-somerville | Also, we need to look at venues. The SmartMeeting people can help with that. | 00:14 |
| cody-somerville | StevenK: It should be, yup. | 00:14 |
| lifeless | cody-somerville: Yahoo will host us | 00:14 |
| cody-somerville | We need to make sure that'll work. | 00:14 |
| cody-somerville | ie. We'll want someone in Sunnyvale to visit | 00:14 |
| cody-somerville | and to talk with Yahoo people | 00:15 |
| lifeless | cody-somerville: They are also in sunnyvale | 00:15 |
| cody-somerville | Right, hence having someone in Sunnyvale to go check out and verify the venue will meet our needs. | 00:15 |
| lifeless | cody-somerville: what are we wanting to check? Maybe we can just ask harlowja? | 00:15 |
| StevenK | lifeless: Ah, I thought we were in the HP office | 00:15 |
| lifeless | StevenK: we were, monty pointed out the fail networking situation for non-staff | 00:16 |
| SpamapS | StevenK: from our experience, it may not be ideal for hosting lots of outsiders. | 00:16 |
| StevenK | Right | 00:16 |
| cody-somerville | Since we've never been there, I wouldn't want to make assumptions about anything | 00:16 |
| lifeless | StevenK: and Joshua had offered already | 00:16 |
| cody-somerville | also, we can potentially host at Palo Alto | 00:16 |
| cody-somerville | and have switch installed with non-corp visitor internet | 00:16 |
| * SpamapS -> kids | 00:16 | |
| lifeless | cody-somerville: right, but what will some random in sunnyvale know about our needs either ? :) | 00:16 |
| lifeless | cody-somerville: anyhow, I'll let you deal with this - thanks btw! | 00:16 |
| lifeless | cody-somerville: I will loop you in on the y! folks that Joshua mailed. | 00:17 |
| cody-somerville | I think we should think about hosting a local mirror | 00:17 |
| cody-somerville | and we'll want to make sure people have wired access | 00:17 |
| cody-somerville | want to make sure venue is not a distraction and easier for me to make that happen at HP site but very happy to work with Y! if they can accommodate. | 00:17 |
| lifeless | cody-somerville: loop-in-introduction sent | 00:18 |
| cody-somerville | lifeless: Thanks. | 00:18 |
| cody-somerville | lifeless: Did you find link to budget I sent you? | 00:19 |
| cody-somerville | It has current, up to date attendee list for both guests and GP | 00:19 |
| lifeless | cody-somerville: yes - 'got it' above referred to that | 00:19 |
| cody-somerville | *HP | 00:19 |
| cody-somerville | ah, ll | 00:19 |
| cody-somerville | *kk | 00:19 |
| *** matsuhashi has joined #tripleo | 00:19 | |
| lifeless | cody-somerville: clark has said he wants to just be synced with us from seattle | 00:20 |
| lifeless | cody-somerville: so he will miss out on the get together we have one night | 00:20 |
| lifeless | cody-somerville: everything else looks grand | 00:20 |
| cody-somerville | lifeless: Is Joshua planning to attend? | 00:21 |
| lifeless | cody-somerville: I believe he is interested yes | 00:21 |
| lifeless | cody-somerville: I've been telling folk to cc you in their RSVPs | 00:22 |
| lifeless | cody-somerville: I haven't cross-checked to see if they have all done that [yet] | 00:22 |
| cody-somerville | I've been adding folks to the spreadsheet as they do. | 00:22 |
| lifeless | legend | 00:22 |
| lifeless | cody-somerville: StevenK: bbiab, got to drop Lynne and C off at a thing | 00:25 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Stop using all ports option w/ neutron-ovs-cleanup https://review.openstack.org/69937 | 00:29 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Refactor ensure-bridge for persistance https://review.openstack.org/69918 | 00:29 |
| *** CaptTofu has quit IRC | 00:29 | |
| *** CaptTofu has joined #tripleo | 00:30 | |
| * cody-somerville heads to bed. | 00:30 | |
| *** CaptTofu has quit IRC | 00:34 | |
| *** cadenzajon has quit IRC | 00:41 | |
| *** CaptTofu has joined #tripleo | 00:41 | |
| *** spzala has quit IRC | 00:46 | |
| *** UtahDave has quit IRC | 00:49 | |
| peoplemerge | SpamapS: I hadn't noticed this guide before, it's helpful, thought it might be out of date http://docs.openstack.org/developer/tripleo-incubator/deploying.html | 01:06 |
| *** epim has quit IRC | 01:12 | |
| lifeless | jog0: jay had a suggestion on https://review.openstack.org/#/c/69108/ - it needs +2s again | 01:12 |
| jog0 | lifeless: I saww that comment, +2ed | 01:13 |
| *** nosnos has joined #tripleo | 01:28 | |
| *** TravT has quit IRC | 01:30 | |
| *** peoplemerge has quit IRC | 01:33 | |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-incubator: Permit specifying ovs bridge mappings too. https://review.openstack.org/71706 | 01:34 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Add a script to deploy the ci-overcloud. https://review.openstack.org/71154 | 01:35 |
| *** peoplemerge has joined #tripleo | 01:36 | |
| StevenK | mikal: Did you want to +A https://review.openstack.org/#/c/69108/7 ? | 01:44 |
| mikal | StevenK: I dunno... What's in it for me? | 01:47 |
| *** panda_ has joined #tripleo | 01:48 | |
| *** panda has quit IRC | 01:50 | |
| StevenK | mikal: Haha | 01:51 |
| *** newell has quit IRC | 02:02 | |
| lifeless | mikal: satisfaction | 02:11 |
| *** wusWWW has quit IRC | 02:14 | |
| *** wusWWW has joined #tripleo | 02:15 | |
| *** wusWWW has quit IRC | 02:26 | |
| lifeless | SpamapS: around ? | 02:28 |
| lifeless | StevenK: actually, you'll know. When you use Type: JSON can you validate the types of the contents? | 02:28 |
| lifeless | StevenK: and how do you unpick the contents inside a heat template:? Or can you only address bits of it on the client that receives it? | 02:29 |
| jog0 | lifeless: isn't this in progress https://bugs.launchpad.net/nova/+bug/1272623 | 02:32 |
| uvirtbot | Launchpad bug 1272623 in tripleo "nova refuses to start if there are baremetal instances with no associated node" [High,Triaged] | 02:32 |
| lifeless | jog0: thats 69108 isn't it? | 02:32 |
| jog0 | yes | 02:33 |
| jog0 | ahh jenkins will update on merge | 02:33 |
| * jog0 is going through his nova bug report emails | 02:33 | |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-heat-templates: Make it possible to do physical networks for VMs. https://review.openstack.org/71721 | 02:39 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-incubator: Permit specifying ovs bridge mappings too. https://review.openstack.org/71706 | 02:44 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Add a script to deploy the ci-overcloud. https://review.openstack.org/71154 | 02:45 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Add a script to deploy the ci-overcloud. https://review.openstack.org/71154 | 02:48 |
| *** panda_ has quit IRC | 02:48 | |
| lifeless | oh man | 02:50 |
| lifeless | now i'm expected 10x l3 agent fails to start... | 02:50 |
| lifeless | have I filed a bug on that yet? | 02:51 |
| StevenK | lifeless: Will I? The Type: JSON in the heat was pure guesswork on my part thanks to a hint from 'git grep' | 02:51 |
| *** ci-overcloud has joined #tripleo | 02:52 | |
| ci-overcloud | ************** ci-overcloud complete status=130 ************ | 02:52 |
| *** ci-overcloud has quit IRC | 02:52 | |
| StevenK | Hm, 130 | 02:52 |
| lifeless | ctrl-C | 02:52 |
| StevenK | Ah | 02:52 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Add a script to deploy the ci-overcloud. https://review.openstack.org/71154 | 02:52 |
| lifeless | now, to find if there is a string join function in CFN | 02:59 |
| lifeless | Fn::Join I guess | 03:00 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-heat-templates: Allow manual hosts entry exports in overcloud. https://review.openstack.org/71725 | 03:08 |
| lifeless | sorry if anyone goes blind.. | 03:08 |
| *** CaptTofu has quit IRC | 03:11 | |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-incubator: Permit setting a /etc/hosts entry for the API. https://review.openstack.org/71727 | 03:13 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Use rfc compliant example data in hosts element. https://review.openstack.org/71728 | 03:13 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Add a script to deploy the ci-overcloud. https://review.openstack.org/71154 | 03:13 |
| *** ci-overcloud has joined #tripleo | 03:21 | |
| ci-overcloud | ************** ci-overcloud complete status=1 ************ | 03:21 |
| *** ci-overcloud has quit IRC | 03:21 | |
| lifeless | argh, heat stack-list on go-slow. WTF | 03:22 |
| StevenK | lifeless: More COW and btrfs fun? | 03:23 |
| lifeless | StevenK: this is on the cd-undercloud | 03:23 |
| lifeless | so a server with 96G of RAM | 03:24 |
| lifeless | and 24 cores | 03:24 |
| StevenK | lifeless: Spoil my trolling with facts, why don't you :-P | 03:24 |
| lifeless | oops | 03:24 |
| lifeless | DEBUG (connectionpool:374) Setting read timeout to None | 03:25 |
| lifeless | who does that | 03:25 |
| lifeless | really? | 03:25 |
| lifeless | keystone again I guess :( | 03:25 |
| lifeless | 1.4M rows | 03:26 |
| StevenK | lifeless: checkwatches, until I got sick of it hanging around for days and fixed it ... | 03:26 |
| jog0 | lifeless: whao we don't have something to prune that? | 03:31 |
| jog0 | no I know why no one runs this in production ;) | 03:31 |
| *** tchaypo is now known as jamezpolley | 03:32 | |
| *** jamezpolley is now known as tchaypo | 03:32 | |
| lifeless | jog0: we do, its glitching, SpamapS was looking at it | 03:38 |
| jog0 | lifeless: ahh, have you leaned on keystone to get the sane defaults thing on tthis one? I assume you have | 03:40 |
| lifeless | its been discussed | 03:45 |
| lifeless | jog0: TBH it's all novas fault. | 03:45 |
| lifeless | jog0: and partly heats | 03:45 |
| lifeless | | CompletionHandle | ba0432ac-1ebd-4e3f-aa5e-619fc815ac2a | Error: Timed out trying to delete user | DELETE_F | 03:47 |
| jog0 | lifeless: I would like to move the blame off nova and into keystone because they said they would make the clients handle token caching | 03:49 |
| lifeless | jog0: :) | 03:49 |
| jog0 | but yes its nova fault too | 03:49 |
| lifeless | we have 1.4M tokens active atm | 03:49 |
| lifeless | and I bet they are all used like 3 times | 03:49 |
| lifeless | mysql> delete from token limit 10000; | 03:50 |
| lifeless | Query OK, 10000 rows affected (3 min 23.88 sec) | 03:50 |
| StevenK | Blink | 03:51 |
| lifeless | I'm presuming a locking issue of some sort | 03:51 |
| StevenK | How is that anything more than 2 seconds :-( | 03:51 |
| lifeless | I know, right? | 03:51 |
| StevenK | jog0: Feel like another +2 for baremetal file injection? | 03:52 |
| lifeless | mysqld is doing 4M/s of writes, but also -worrying, several hundred k reasd | 03:53 |
| lifeless | I wonder if it needs a bigger buffer size | 03:53 |
| StevenK | Have we tuned it for 96G of RAM? | 03:53 |
| lifeless | stopped keystone to deal with this | 03:53 |
| lifeless | StevenK: ENFI | 03:54 |
| jog0 | StevenK: link | 03:54 |
| StevenK | jog0: https://review.openstack.org/#/c/70252/5 | 03:54 |
| jog0 | StevenK: now that gate is mostly moving again .. I am reviewing nov again | 03:55 |
| StevenK | Hurray | 03:55 |
| lifeless | nothing like breaking it again | 03:55 |
| lifeless | to make the weekend fun | 03:55 |
| StevenK | lifeless: Do I need tests for the spawn changes in 71214? | 03:55 |
| lifeless | StevenK: better to have than not have | 03:56 |
| StevenK | Was trying to work out how to write them when I EOD'd | 03:56 |
| lifeless | the semaphore might be tricky to test with the stock test harness | 03:56 |
| lifeless | that one I think you can make a case for 'we use it and it works' | 03:56 |
| lifeless | uery OK, 10000 rows affected (3 min 45.34 sec) | 03:56 |
| StevenK | MySQL, you a terrible person. | 03:56 |
| StevenK | Take that, mordred | 03:57 |
| StevenK | lifeless: Let's switch to sqlite, because it's web-scale | 03:57 |
| clarkb | drizzle everything | 03:58 |
| *** edmund has quit IRC | 04:00 | |
| lifeless | du -sh /var/lib/mysql/ | 04:01 |
| lifeless | 38G /var/lib/mysql/ | 04:01 |
| *** morazi has quit IRC | 04:01 | |
| jog0 | StevenK: reviewed | 04:01 |
| StevenK | jog0: I saw, thanks. | 04:01 |
| StevenK | jog0: Fixing it right now | 04:02 |
| lifeless | 1G | 04:03 |
| lifeless | lol | 04:03 |
| jog0 | lifeless: how big are our logs | 04:05 |
| jog0 | or do we log rotate | 04:05 |
| lifeless | jog0: thats the data file, not logs | 04:05 |
| lifeless | jog0: and yeah logs have been an issue too, but we're not onto that yet | 04:05 |
| jog0 | lifeless: heh I have seen the logs fill tiny root disk partitions issue too many times | 04:06 |
| jog0 | lifeless: you won't be in utah will you? | 04:06 |
| lifeless | no | 04:06 |
| jog0 | ack, I imagine there will be some gantt discussions there | 04:07 |
| lifeless | good | 04:07 |
| * StevenK peers at jog0 | 04:07 | |
| StevenK | NoSuchGroupError: no such group: True | 04:07 |
| lifeless | I'm just the trigger-man for that :0 | 04:07 |
| jog0 | StevenK: woops got the order wring | 04:07 |
| jog0 | wrong | 04:07 |
| jog0 | I guess group is last | 04:07 |
| jog0 | I was winging it on that code | 04:07 |
| StevenK | 'use_file_injection', True, 'baremetal' ? | 04:08 |
| lifeless | iniset ? group key value | 04:08 |
| jog0 | that would be my guess | 04:08 |
| StevenK | lifeless: CONF.set_override() | 04:08 |
| lifeless | oh | 04:09 |
| lifeless | nfi:P[6~ | 04:09 |
| jog0 | apparently it doesn't work as expected | 04:09 |
| * jog0 calls it a night | 04:09 | |
| lifeless | I wonder if the table is over indexed | 04:10 |
| StevenK | lifeless: Is that a common problem? | 04:11 |
| lifeless | it is in pg land | 04:11 |
| lifeless | to delete a row from 11 indices is 10 times the work vs just the primary index | 04:11 |
| lifeless | delete from token limit 1; | 04:12 |
| lifeless | Query OK, 1 row affected (0.62 sec) | 04:12 |
| lifeless | is pretty slow | 04:12 |
| lifeless | 6 indices | 04:12 |
| StevenK | lifeless: I don't recall that class of problems on LP even for large deletes | 04:13 |
| lifeless | StevenK: stub | 04:13 |
| StevenK | lifeless: Ah, he would sort out that during DB review? | 04:14 |
| lifeless | delete from token where id in (select id from token order by id limit 10000); | 04:14 |
| lifeless | ERROR 1235 (42000): This version of MySQL doesn't yet support 'LIMIT & IN/ALL/ANY/SOME subquery' | 04:14 |
| lifeless | StevenK: yeah consolidating indices, testing etc | 04:14 |
| clarkb | each index is a tree right? and each one would need to be updated when you delete stuff | 04:16 |
| lifeless | yes, so a 1.4M node btree - ln(entriesperpage) high - and that many pages to journal and update on every index altering operation | 04:17 |
| lifeless | innodb recokons its doing 79deletes/second | 04:19 |
| lifeless | which is slllllow IME | 04:19 |
| StevenK | Rather | 04:19 |
| StevenK | 7900 deletes/second is more like it | 04:20 |
| lifeless | one thing we don't have is a battery backed up raid controller | 04:20 |
| StevenK | Really? I'd expect it in the class of hardware we're using | 04:21 |
| lifeless | lol | 04:21 |
| lifeless | waste in a cloud architecture | 04:22 |
| lifeless | anyhow, -> dinner stuff w/C | 04:22 |
| *** UtahDave has joined #tripleo | 04:46 | |
| openstackgerrit | Richard Su proposed a change to openstack/tripleo-incubator: Message broker selection through boot-stack https://review.openstack.org/71736 | 04:48 |
| *** vkozhukalov has joined #tripleo | 04:54 | |
| *** gcha has quit IRC | 04:58 | |
| *** killer_prince has joined #tripleo | 05:06 | |
| *** saju_m has joined #tripleo | 05:08 | |
| *** CaptTofu has joined #tripleo | 05:12 | |
| lifeless | 05:13 | |
| lifeless | 27853 be/4 mysql 0.00 B/s 727.80 K/s 0.00 % 70.67 % mysqld | 05:13 |
| lifeless | so, its write limited | 05:13 |
| lifeless | might be eatmydata time | 05:13 |
| *** jcooley_ has quit IRC | 05:14 | |
| *** jcooley_ has joined #tripleo | 05:15 | |
| *** CaptTofu has quit IRC | 05:16 | |
| StevenK | lifeless: Do I need to mail Cody travel details or magic just happens? | 05:22 |
| lifeless | StevenK: you sorted the CWT fail? | 05:22 |
| StevenK | lifeless: The CWT fail turned out to be "just wait longer" | 05:23 |
| *** UtahDave has quit IRC | 05:24 | |
| lifeless | StevenK: I would assume cody-somerville will mail out instructions when he's got the block bookoing organised | 05:24 |
| StevenK | lifeless: "Your trip has an overnight stay without a hotel reservation", do I choose "I will book a hotel later" or "I booked a hotel outside of this system" ? | 05:26 |
| lifeless | later | 05:27 |
| lifeless | its just reporting | 05:28 |
| tchaypo | that's better than the usual CWT fail | 05:31 |
| tchaypo | which is "you used CWT" | 05:31 |
| StevenK | tchaypo: They are *FAR* better than the C travel agency | 05:32 |
| tchaypo | The agency I used last time I traveled had a website that kept telling me that "you had an unspecified error, please try your booking again" then going back to the homepage | 05:33 |
| tchaypo | after I did this three times I gave up and started poking around and it turned out I had 3 bookings for flights+hotel | 05:33 |
| lifeless | down to 800000 rows | 05:34 |
| StevenK | tchaypo: Haha | 05:38 |
| *** tzumainn has quit IRC | 05:41 | |
| lifeless | I wonder if we should use row format dynamic and the barracuda innodb format | 05:42 |
| lifeless | might help with tokens which are ridiculous in size | 05:42 |
| *** julim has joined #tripleo | 06:01 | |
| peoplemerge | lifeless: thanks for e351911 - make devtest docs work better. This is really great. | 06:02 |
| lifeless | peoplemerge: yw | 06:03 |
| *** akuznetsov has joined #tripleo | 06:08 | |
| lifeless | sadface | 06:15 |
| lifeless | | fault | {u'message': u'No valid host was found. Exceeded max scheduling attempts 3 for instance 50777622-339b-440f-9e30-0593f85c6845', u'code': 500, u'details': u' File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 147, in run_instance | | 06:15 |
| *** saju_m has quit IRC | 06:18 | |
| *** lazy_prince has joined #tripleo | 06:28 | |
| lifeless | heh port quota exceeded | 06:29 |
| lazy_prince | lifeless: got any chance to look at https://blueprints.launchpad.net/diskimage-builder/+spec/build-windows-images blueprint.. | 06:30 |
| lifeless | lazy_prince: sounds like a new project to me | 06:36 |
| lifeless | lazy_prince: what code would be shared with the existing dib code? | 06:36 |
| *** cadenzajon has joined #tripleo | 06:37 | |
| openstackgerrit | Shane Wang proposed a change to openstack/diskimage-builder: Fix misspellings in diskimage-builder https://review.openstack.org/71767 | 06:38 |
| lifeless | hmm wtf File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 179, in _create_port | 06:38 |
| lifeless | raise exception.PortLimitExceeded() | 06:39 |
| lifeless | PortLimitExceeded: Maximum number of ports exceeded | 06:39 |
| lifeless | but | 06:41 |
| lifeless | | port | 50 | | 06:41 |
| lazy_prince | lifeless: there is no code shared but the way the images will be built is similar to how linux images are created without standing up a VM. and so we thought it would be ideal place for this tool. in any case our grand plan is to support provisioning of HyperV using ironic which will help us in tripleo in long term. | 06:41 |
| *** cadenzajon has quit IRC | 06:41 | |
| lazy_prince | we already have a blueprint in ironic for windows deployment on baremetal https://blueprints.launchpad.net/ironic/+spec/windows-disk-image-support | 06:42 |
| *** rsacharya has joined #tripleo | 06:45 | |
| lazy_prince | lifeless: if we still feel that it should be a separate project, we are okay with that too.. but we would prefer to keep image building tools at one place. | 06:45 |
| lifeless | lazy_prince: no code shared == separate codebase. | 06:46 |
| lifeless | lazy_prince: anything else is just maintenance pain | 06:46 |
| lifeless | happy to have it under the tripleo banner | 06:46 |
| *** julim has quit IRC | 06:49 | |
| *** jprovazn has joined #tripleo | 06:50 | |
| lazy_prince | we would love it to be under tripleo banner. and we will work on to get a separate code of its own. however, we need your support in getting it approved. | 06:50 |
| lifeless | sure | 06:51 |
| lazy_prince | lifeless: following this blueprint we have another blueprint https://blueprints.launchpad.net/diskimage-builder/+spec/extensible-windows-images can you please review this one as well.. | 06:51 |
| lazy_prince | lifeless: we plan to target the first blueprint for icehouse and the later one for juno. | 06:59 |
| lifeless | sure | 07:00 |
| lifeless | they look fine, just needs to be a new project | 07:00 |
| lazy_prince | lifeless: so can you approve them please... | 07:01 |
| lazy_prince | lifeless: we will work on getting a seprate codebase for them. | 07:01 |
| lifeless | lazy_prince: they need to be moced to 'tripleo' rather than disk-image-builder - since wrong codebase. | 07:02 |
| lifeless | movwed | 07:03 |
| lifeless | bah | 07:03 |
| lifeless | moved | 07:03 |
| * SpamapS looks around | 07:03 | |
| lazy_prince | lifeless: so does it mean that I will have to start over again and submit new blueprints to tripleo.. | 07:04 |
| *** jcooley_ has quit IRC | 07:05 | |
| *** matsuhashi has quit IRC | 07:05 | |
| *** jcooley_ has joined #tripleo | 07:06 | |
| lazy_prince | lifeless: nevermind, i was able to move them to tripleo. | 07:06 |
| SpamapS | lifeless: regarding deleting users.. I created a user_id index.. speeds up the delete A LOT... | 07:06 |
| StevenK | SpamapS: It was *worse* than 3 and a half minutes? | 07:07 |
| *** jcooley_ has quit IRC | 07:08 | |
| lifeless | StevenK: truncate table works wonders | 07:08 |
| lifeless | bah | 07:08 |
| lifeless | SpamapS: ^ | 07:08 |
| lifeless | but | 07:08 |
| lifeless | the 31GB innodb file worries me | 07:08 |
| StevenK | I love TRUNCATE TABLE | 07:08 |
| * SpamapS reading backscroll | 07:08 | |
| lifeless | now, to find this 'quota | 07:09 |
| SpamapS | lifeless: the 31G innodb file is not what you might think :) | 07:10 |
| lifeless | ahha | 07:10 |
| lifeless | 2014-02-07 06:30:04,447.447 11046 TRACE nova.network.neutronv2.api [instance: 94c56776-2680-429d-b7d6-4e8846dfe832] IpAddressGenerationFailureClient: No more IP addresses available on network e85b44c7-1136-4217-954e-cdf0acdddfe1. | 07:10 |
| lifeless | bad translation | 07:10 |
| lifeless | SpamapS: what do you think I think it is ? | 07:10 |
| SpamapS | lifeless: Data_free: 39582695424 | 07:10 |
| SpamapS | lifeless: untuned mysqld | 07:10 |
| SpamapS | we have a 20G innodb buffer | 07:11 |
| lifeless | SpamapS: I think its the result of having lots of churn in the db + MVCC | 07:11 |
| SpamapS | our working set is well under that | 07:11 |
| SpamapS | lifeless: right, also remember, PKI tokens are ridiculously big | 07:11 |
| lifeless | SpamapS: a large file leaves lots of room for utter fails in data locality | 07:11 |
| SpamapS | Avg_row_length: 20075 | 07:11 |
| lifeless | SpamapS: yes,see my suggesting for changing the row format? | 07:12 |
| SpamapS | lifeless: and agreed on the big file. Only solution is to dump/restore db | 07:12 |
| SpamapS | lifeless: we _might_ be able to set innodb_file_per_table and optimize table. | 07:12 |
| lifeless | SpamapS: only reason we have that data free is I applied truncate table | 07:12 |
| SpamapS | lifeless: we had a lot of data free earlier today too though | 07:12 |
| lazy_prince | lifeless: thanks for approving the blueprints. do you have any pointers on how to start for a new project under tripleo banner. | 07:12 |
| lifeless | that won't free the data but it will give us better locality | 07:13 |
| *** CaptTofu has joined #tripleo | 07:13 | |
| SpamapS | Data_free: 2146435072 | 07:13 |
| SpamapS | that was ~6 hours ago | 07:13 |
| lifeless | lazy_prince: use the cookiecutter facility to get the basic python metadata put together, push that to github, then submit a proposal to openstack-infra/config to clone it into openstack's main repositories. | 07:13 |
| SpamapS | oy wait, that is 2G ... | 07:13 |
| lifeless | SpamapS: yes. | 07:13 |
| SpamapS | Rows: 1394294 | 07:14 |
| SpamapS | Avg_row_length: 26586 | 07:14 |
| SpamapS | Data_length: 37069799424 | 07:14 |
| lifeless | SpamapS: I might not know mysql all that well, but... | 07:14 |
| SpamapS | No I was just not believing what I was seeing | 07:14 |
| lifeless | SpamapS: feel me now ? :) | 07:14 |
| SpamapS | lifeless: ok, so we can do a few tricks... | 07:14 |
| lazy_prince | lifeless: thanks for the help and support.. | 07:14 |
| SpamapS | innodb_file_per_table will at least let us shrink table file sizes. | 07:14 |
| SpamapS | lifeless: perhaps we should make our tokens expire faster. | 07:15 |
| lifeless | how to confuse people | 07:16 |
| SpamapS | lifeless: and perhaps we need to push on the idea of token caching in nova. Is this something where it just isn't on by default? | 07:16 |
| lifeless | https://bugs.launchpad.net/nova/+bug/1277368 | 07:16 |
| uvirtbot | Launchpad bug 1277368 in nova "incorrect 'quota exceeded' error from neutron v2 api when subnet is out of addresses" [Undecided,New] | 07:16 |
| lifeless | SpamapS: and in heat | 07:16 |
| lifeless | SpamapS: and $etc. | 07:16 |
| lifeless | SpamapS: not sure | 07:16 |
| SpamapS | I can't imagine it works well for anybody. | 07:17 |
| lifeless | hmm | 07:17 |
| lifeless | now I have to do DB surgery on neutron's DB | 07:17 |
| lifeless | | 18b96b8d-c989-4820-b330-0c0f7c19c8d5 | | 10.10.16.128/26 | {"start": "10.10.16.171", "end": "10.10.16.188"} | | 07:17 |
| SpamapS | lifeless: so UUID tokens are just that.. a UUID.. that would at least make 1 million tokens only equal 36MB | 07:17 |
| lifeless | since you can't edit the damn subnet allocation pool size | 07:17 |
| *** CaptTofu has quit IRC | 07:18 | |
| SpamapS | It just pounds on keystone | 07:18 |
| SpamapS | I'm almost ready to say that keystone should be compressing tokens in the db | 07:18 |
| lifeless | do you mean returning the same one? That worries me - the potential for utter failure to block someone out... | 07:20 |
| SpamapS | lifeless: also I'm thinking we can live with losing 1s of data from the DB.. so I think we can set this to 2 http://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit | 07:20 |
| *** matsuhashi has joined #tripleo | 07:21 | |
| SpamapS | lifeless: no UUID tokens are just tinier | 07:21 |
| SpamapS | lifeless: they are just a UUID.. there's no PKI.. so nova has to ask keystone if the UUID is valid. | 07:21 |
| SpamapS | and heat does too.. and and and.. | 07:21 |
| SpamapS | it pounds on keystone like mad | 07:21 |
| SpamapS | lifeless: ok, so I've got some optimizations queued up for mysqld but it will need to be restarted. You doing anything important? | 07:24 |
| *** akuznetsov has quit IRC | 07:25 | |
| lifeless | SpamapS: go for it | 07:27 |
| lifeless | ahha | 07:28 |
| lifeless | https://review.openstack.org/#/c/62042/ | 07:28 |
| lifeless | so we can stop running patched nova | 07:29 |
| lifeless | and start running patched neutron | 07:29 |
| *** akuznetsov has joined #tripleo | 07:29 | |
| *** lifeless changes topic to "tripleo-cd running preserve-ephemeral WIP patches and https://review.openstack.org/#/c/62042/ | Using OpenStack to deploy OpenStack;meetings Tuesday 1900 UTC in #openstack-meeting-alt" | 07:30 | |
| lifeless | SpamapS: say when its good again | 07:32 |
| *** saju_m has joined #tripleo | 07:32 | |
| *** jcooley_ has joined #tripleo | 07:34 | |
| SpamapS | lifeless: it's back up but one of the settings was ignored.. checking | 07:35 |
| lifeless | SpamapS: so yeah the 20G buffer was because we had a 30G dataset | 07:37 |
| lifeless | SpamapS: and I figured random freaking btree IO because we had in-tree data. | 07:37 |
| SpamapS | lifeless: ok good | 07:40 |
| openstackgerrit | Shane Wang proposed a change to openstack/os-apply-config: Fix misspellings in os-apply-config https://review.openstack.org/71786 | 07:40 |
| SpamapS | lifeless: I raised buffer to 30G | 07:40 |
| SpamapS | lifeless: because.. why not? ;) | 07:40 |
| lifeless | bad O algorithms in mysql ? :) | 07:40 |
| SpamapS | lifeless: mysql does not rely on the fs buffer | 07:41 |
| SpamapS | at all | 07:41 |
| lifeless | SpamapS: the manual says it has bad O ;) | 07:41 |
| SpamapS | bad O? | 07:41 |
| lifeless | bad order | 07:41 |
| SpamapS | Ok so token is now in its own file | 07:42 |
| lifeless | big O notation etc | 07:42 |
| SpamapS | with the Barracuda format (compacts rows a bit) | 07:42 |
| lifeless | did you set the row format as I suggested? | 07:42 |
| SpamapS | and we're at innodb_flush_log_at_trx_commit=2 .. which means we only flush the logs once per second | 07:42 |
| *** athomas has quit IRC | 07:43 | |
| SpamapS | lifeless: I did not see that suggestion | 07:43 |
| SpamapS | but Row_format is Compact | 07:43 |
| lifeless | 18:42 < lifeless> I wonder if we should use row format dynamic and the barracuda innodb format | 07:44 |
| lifeless | 18:42 < lifeless> might help with tokens which are ridiculous in size | 07:44 |
| SpamapS | lifeless: yeah reading up on dynamic .. it seems like it would keep the indexes more efficient | 07:45 |
| SpamapS | Though I doubt user deleting will be any faster. ;) | 07:45 |
| lifeless | SpamapS: compact stores up to 768 bytes of each token in the tree | 07:45 |
| lifeless | SpamapS: which is pointless | 07:46 |
| SpamapS | yeah | 07:47 |
| SpamapS | I'm looking now at compressed; | 07:47 |
| *** taps has quit IRC | 07:47 | |
| SpamapS | also wondering how we can make keystone set this up or at least try to. | 07:47 |
| *** bauzas has joined #tripleo | 07:48 | |
| lifeless | SpamapS: http://dev.mysql.com/doc/refman/5.5/en/innodb-row-format-dynamic.html | 07:50 |
| lifeless | SpamapS: I'd go dynamic personally | 07:50 |
| lifeless | in the absense of data | 07:50 |
| *** nosnos_ has joined #tripleo | 07:51 | |
| SpamapS | lifeless: I'm just thinking about the data we have... 2 million active tokens was 36G of data or something like that | 07:51 |
| lifeless | yeah | 07:51 |
| lifeless | but the perf hit wasn't the 36G of data | 07:52 |
| lifeless | it was the 768 bytes per row in the indices | 07:52 |
| SpamapS | yeah, and we could cause CPU havoc .. these 24 cores aren't the 24 fastest cores.. ;) | 07:52 |
| SpamapS | ok setting to dynamic | 07:53 |
| SpamapS | too bad MySQL can't be tuned. ;) | 07:53 |
| lifeless | too bad you have to tune it | 07:54 |
| *** nosnos has quit IRC | 07:54 | |
| SpamapS | Make me an auto tuning thing, I'll make you a medicore tuned app running on top of it. ;) | 07:54 |
| lifeless | really, you're going there? | 07:55 |
| SpamapS | no I'm going to bed in about 5 minutes :) | 07:57 |
| SpamapS | lifeless: so our token expiration is 3600 seconds | 07:57 |
| lifeless | SpamapS: can I use things again? | 08:01 |
| SpamapS | lifeless: oh yes I thought I already said that. please do. | 08:01 |
| lifeless | you did and then you said you were changing to dynamic | 08:02 |
| SpamapS | "ok good" wasn't clear enough ;) | 08:02 |
| SpamapS | oh that takes like 5s | 08:02 |
| SpamapS | done online | 08:02 |
| *** rlandy has joined #tripleo | 08:08 | |
| *** gcha has joined #tripleo | 08:08 | |
| openstackgerrit | Shane Wang proposed a change to openstack/tripleo-image-elements: Fix misspellings in tripleo-image-elements https://review.openstack.org/71805 | 08:10 |
| *** matsuhashi has quit IRC | 08:10 | |
| SpamapS | lifeless: so we can make heat cache tokens if we give it a local memcached to cache tokens into | 08:10 |
| SpamapS | we just have to set [keystone_authtoken]\nmemcached_servers=127.0.0.1:6666\n and it will cache everything in memcached for 300 seconds | 08:12 |
| openstackgerrit | Shane Wang proposed a change to openstack/tuskar: Fix misspellings in tuskar https://review.openstack.org/71811 | 08:13 |
| lifeless | SpamapS: wow | 08:14 |
| lifeless | SpamapS: overkill :) | 08:14 |
| SpamapS | there doesn't seem to be a "cache them in /tmp" option :( | 08:15 |
| SpamapS | but we could probably add that to keystoneclient.middleware.auth_token | 08:15 |
| SpamapS | which is what I believe everything uses | 08:15 |
| openstackgerrit | Shane Wang proposed a change to openstack/tuskar-ui: Fix misspellings in tuskar-ui https://review.openstack.org/71813 | 08:18 |
| *** martyntaylor has joined #tripleo | 08:25 | |
| *** bauzas has quit IRC | 08:31 | |
| lifeless | SpamapS: keystone were meant to do that, in fact | 08:32 |
| lifeless | ok, undercloud ip address range updated | 08:32 |
| lifeless | bombs away | 08:32 |
| lifeless | oh nuts, what now - they all when into scheduling and back out :( | 08:33 |
| lifeless | hmm | 08:34 |
| lifeless | well it *claims* it updated it | 08:35 |
| lifeless | but same error from neutron | 08:35 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: os-svc-enable should error when no svcname given. https://review.openstack.org/71155 | 08:38 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Fix os-svc-restart to exit correctly without -n. https://review.openstack.org/71156 | 08:39 |
| *** mrunge has joined #tripleo | 08:48 | |
| lifeless | so here I go, fixing neutron patches :> | 08:52 |
| *** cody-somerville has quit IRC | 08:52 | |
| *** markmc has joined #tripleo | 09:04 | |
| *** panda has joined #tripleo | 09:06 | |
| *** bauzas has joined #tripleo | 09:07 | |
| *** CaptTofu has joined #tripleo | 09:14 | |
| *** rsacharya has quit IRC | 09:15 | |
| *** marun has quit IRC | 09:16 | |
| *** CaptTofu has quit IRC | 09:18 | |
| *** derekh has joined #tripleo | 09:22 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: WIP: build a local pip mirror of python packages being tested https://review.openstack.org/71837 | 09:23 |
| *** lucasagomes has joined #tripleo | 09:30 | |
| mordred | derekh: ^^ hey ho - any reason to not use pypi-mirror? | 09:33 |
| derekh | mordred: not in particular, I can take a look | 09:33 |
| mordred | derekh: it deos a slightly different thing than what you're doing, as I read it - but it might be nice/interesting to see if it's possible to consolidate thought | 09:34 |
| mordred | derekh: OR - it might not :) | 09:34 |
| derekh | mordred: cool, will have a poke at it | 09:34 |
| SpamapS | derekh: I think I'd rather have the global reqs mirror built and available locally than duplicate all of that functionality | 09:37 |
| derekh | SpamapS: you talking about a global mirror available to all ci jobs ? | 09:37 |
| mordred | well, we have one of those in infra - but having one that sits in the tripleo cloud(s) might help for locality | 09:39 |
| SpamapS | derekh: yes.. thats what the pypi-mirror element builds | 09:39 |
| SpamapS | derekh: and maintains | 09:39 |
| SpamapS | derekh: using the pypi-mirror project :) | 09:39 |
| derekh | SpamapS: this is building packages specifically for a certain job, e.g. we may need a specific os-apply-config and python-keystoneclient which will come from local cache then we get everything else from pypi.openstack.org | 09:39 |
| derekh | SpamapS: I hadn't looked at pypi-mirror but will now | 09:40 |
| SpamapS | derekh: not sure what that means. | 09:40 |
| *** cody-somerville has joined #tripleo | 09:40 | |
| SpamapS | derekh: nothing gets to pass tests until its reqs have landed in globa-requirements. ;) | 09:40 |
| SpamapS | derekh: at which point, its' reqs are in pypi.openstack.org | 09:40 |
| SpamapS | derekh: or you're saying.. if we're testing a change for.. python-heatclient .. put it in said cache? | 09:41 |
| derekh | SpamapS: ok so say we have a gale job that is testing a new commit to os-apply-config and also python-keystoneclient we need to pull those in from somewhere | 09:42 |
| derekh | SpamapS: exactly | 09:42 |
| mordred | ah. gotcha. I get intent | 09:42 |
| SpamapS | how does zuul do this for, say, devstack-gate right now? | 09:43 |
| SpamapS | or I should say how does devstack-gate do this | 09:43 |
| mordred | it doesn't | 09:43 |
| mordred | it pulls git repos | 09:43 |
| derekh | SpamapS: so this patch builds packages for the projects that ZUUL says its testing and puts them in cache, everything else comes from pypi.o.o | 09:43 |
| mordred | yah | 09:43 |
| *** nosnos_ has quit IRC | 09:43 | |
| mordred | derekh: I don't think it's _Essential_ but I do know that pypi-mirror has code to take a directory of stuff and pypi-ify it, but it's not split out | 09:43 |
| mordred | derekh: so it might be nice to make a second entry point thing for pypi-mirror that just runs the "here'sa directory, make a mirror" routine | 09:44 |
| mordred | and tehn consume that here | 09:44 |
| derekh | mordred: yup, if it helps and I can use it I will | 09:44 |
| SpamapS | so devstack-gate just pulls git repos.. how does devstack get the change? | 09:45 |
| *** max_lobur_afk is now known as max_lobur | 09:46 | |
| derekh | SpamapS: from what I can see it pulls in git repos and resets master/HEAD to the commit that ZUUL wants tested | 09:46 |
| mordred | derekh: http://git.openstack.org/cgit/openstack-infra/pypi-mirror/tree/pypi_mirror/cmd/run_mirror.py#n346 fwiw | 09:46 |
| lifeless | derekh: reusing pypi-mirror would be ideal | 09:46 |
| mordred | derekh: it will definitely take an upstream patch | 09:46 |
| lifeless | derekh: --no-download might be enough | 09:46 |
| lifeless | derekh: SpamapS: ZUUL_REF ftw | 09:46 |
| SpamapS | I understand how it gets the git tree it needs | 09:47 |
| SpamapS | how does that get used by devstack | 09:47 |
| SpamapS | after installing everything for devstack, does it just go back and pip install -U . ? | 09:48 |
| derekh | mordred: ok, thanks | 09:48 |
| derekh | lifeless: not sure what you mean by "--no-download might be enough" | 09:49 |
| SpamapS | derekh: I think I see where the whole thing is going, and I think I get why it makes sense for image building. :) | 09:52 |
| SpamapS | and now.. I sleep | 09:52 |
| mordred | SpamapS: yes | 09:54 |
| mordred | SpamapS: devstack does pip install -U in each git repo | 09:54 |
| *** saju_m has quit IRC | 09:58 | |
| *** marun has joined #tripleo | 09:59 | |
| lifeless | righty ho, neutron subnet-update fixed. | 10:00 |
| lifeless | now to deploy a m*f* overcloud | 10:01 |
| lifeless | derekh: I don't know why you had the issues you had yesterday | 10:01 |
| lifeless | derekh: there are many other issues you should have had :) | 10:01 |
| lifeless | derekh: like neutron having lost track of ip addresses | 10:02 |
| lifeless | derekh: and keystone contemplating it's navel due to 37G of tokens. | 10:02 |
| derekh | lifeless: I didn't notice any lost IP address, the overcloud came up, I could ssh to it and do a nova list and see a running VM etc... didn't try much more | 10:03 |
| lifeless | derekh: I've put all the /opt/stack/* repos to git review -d <tip of my stack> | 10:03 |
| openstackgerrit | Jan Provaznik proposed a change to openstack/diskimage-builder: Another increase of image padding https://review.openstack.org/71850 | 10:04 |
| lifeless | derekh: yeah, with two hypervisors ..:) | 10:04 |
| derekh | lifeless: yes this is true | 10:04 |
| lifeless | derekh: I just had to fix review 62042 to work to allow us to deploy more machines | 10:04 |
| lifeless | derekh: along the way I noticed we had ips that neutron wasn't even trying to hand out :) | 10:05 |
| lifeless | some edge case in the ipavaialbilityrange merging logic I suspect | 10:05 |
| *** killer_prince has quit IRC | 10:05 | |
| lifeless | derekh: ask mordred what I meant by --no-dowload might be enough | 10:05 |
| lifeless | derekh: I've automated considerably more of the ci-overcloud stuff | 10:06 |
| lifeless | derekh: I haven't added the neutron network definition yet | 10:06 |
| lifeless | but the bridge mappings are define | 10:06 |
| lifeless | derekh: I'm expecting the l3 agents to fall over in a screaming heap with the rootwrap bug | 10:07 |
| lifeless | unless greghaynes fixed it upstream and they miracle of miracles merged it | 10:07 |
| lifeless | derekh: and I haven't automated adding br-untagged *on the control plane* so that needs to be done by hand for now | 10:07 |
| * greghaynes doesnt think he fixed anything | 10:08 | |
| lifeless | derekh: the /etc/hosts stuff should be automated (though you'll go blind looking at the patch) | 10:08 |
| derekh | lifeless: ok, I see its currently building once its up I'll take a look | 10:08 |
| lifeless | greghaynes: didn't you say you added a rootwrap filter to fix neutron trying to kill a process and being denied? | 10:08 |
| greghaynes | Ah, I filed a bug, didnt fic | 10:09 |
| greghaynes | fix | 10:09 |
| derekh | so the hard bit is done | 10:09 |
| lifeless | greghaynes: ah, so yeah - we'll need to do that by hand on the servers or something... | 10:09 |
| greghaynes | https://bugs.launchpad.net/neutron/+bug/1273882 | 10:09 |
| lifeless | derekh: the quota raising for the nodepool tenant will need to be done etc | 10:09 |
| uvirtbot | Launchpad bug 1273882 in tripleo "os-collect-config unable to restart neutron-l3-agent" [Undecided,Confirmed] | 10:09 |
| * derekh nodes | 10:10 | |
| lifeless | derekh: lo and behold we have one up | 10:11 |
| derekh | woot | 10:11 |
| derekh | mordred: ^^ ? <lifeless> derekh: ask mordred what I meant by --no-dowload might be enough | 10:12 |
| * lifeless signs off | 10:12 | |
| derekh | mordred: nevermind, got it | 10:20 |
| openstackgerrit | Dougal Matthews proposed a change to openstack/python-tuskarclient: [WIP] - Add CLI/shell support for Overclouds https://review.openstack.org/70388 | 10:21 |
| *** panda__ has joined #tripleo | 10:26 | |
| *** panda has quit IRC | 10:29 | |
| *** panda__ has quit IRC | 10:37 | |
| *** panda__ has joined #tripleo | 10:37 | |
| *** panda__ has quit IRC | 10:45 | |
| *** jcooley_ has quit IRC | 10:52 | |
| *** jcooley_ has joined #tripleo | 10:52 | |
| *** ci-overcloud has joined #tripleo | 10:56 | |
| ci-overcloud | ************** ci-overcloud complete status=1 ************ | 10:56 |
| *** ci-overcloud has quit IRC | 10:56 | |
| *** jcooley_ has quit IRC | 10:57 | |
| *** e0ne has joined #tripleo | 10:57 | |
| *** panda__ has joined #tripleo | 10:59 | |
| *** mrunge has quit IRC | 11:03 | |
| *** CaptTofu has joined #tripleo | 11:15 | |
| *** gcha has quit IRC | 11:17 | |
| *** CaptTofu has quit IRC | 11:19 | |
| *** boris-42_ has quit IRC | 11:30 | |
| derekh | lifeless: SpamapS was the output from that run piped anywhere ? | 11:33 |
| derekh | the ci-overcloud that is ^^ | 11:33 |
| *** julim has joined #tripleo | 11:36 | |
| *** martyntaylor has left #tripleo | 11:51 | |
| *** boris-42_ has joined #tripleo | 11:53 | |
| *** julim has quit IRC | 12:01 | |
| *** lynxman has quit IRC | 12:12 | |
| *** lynxman has joined #tripleo | 12:13 | |
| *** CaptTofu has joined #tripleo | 12:13 | |
| *** jcooley_ has joined #tripleo | 12:24 | |
| *** hashar has joined #tripleo | 12:26 | |
| *** hashar_ has joined #tripleo | 12:27 | |
| *** lblanchard has joined #tripleo | 12:28 | |
| *** akuznetsov has quit IRC | 12:28 | |
| *** jcooley_ has quit IRC | 12:30 | |
| *** hashar has quit IRC | 12:31 | |
| rbrady | derekh: ping | 12:37 |
| derekh | rbrady: ack | 12:38 |
| rbrady | what is the white space for on line 126?: https://review.openstack.org/#/c/71725/1/overcloud-source.yaml | 12:38 |
| *** lucasagomes is now known as lucas-hungry | 12:40 | |
| derekh | rbrady: its just there to trip me up | 12:41 |
| derekh | rbrady: just missed it | 12:41 |
| rbrady | derekh: I hadn't seen it used in a join before, but wanted to check to see if it was there purposely. thanks. | 12:43 |
| derekh | np | 12:43 |
| derekh | rbrady: actually | 12:44 |
| derekh | maybe it does need to be there | 12:44 |
| derekh | rbrady: see this example from another join | 12:44 |
| derekh | waitcondition_server_url: | 12:45 |
| derekh | Fn::Join: | 12:45 |
| derekh | - '' | 12:45 |
| derekh | - - 'http://' | 12:45 |
| derekh | - Fn::Select: | 12:45 |
| derekh | see the double - - | 12:45 |
| rbrady | yes | 12:45 |
| derekh | I wonder if the white space is equivalent ... no sure | 12:45 |
| derekh | so basically we end up passing in a list to join | 12:45 |
| derekh | if I understand my yaml correctly, | 12:46 |
| derekh | my eyes the goggles do nothing | 12:46 |
| rbrady | I'll see what heat template-validate shows | 12:48 |
| *** ci-overcloud has joined #tripleo | 12:54 | |
| ci-overcloud | ************** ci-overcloud complete status=1 ************ | 12:54 |
| *** ci-overcloud has quit IRC | 12:54 | |
| *** ci-overcloud has joined #tripleo | 12:56 | |
| ci-overcloud | ************** ci-overcloud complete status=1 ************ | 12:56 |
| *** ci-overcloud has quit IRC | 12:56 | |
| rbrady | derekh: results from validate - http://paste.openstack.org/show/63031/ | 12:56 |
| rbrady | derekh: you were right about how it changes it to a list. | 12:56 |
| *** weshay has quit IRC | 12:57 | |
| *** gcha has joined #tripleo | 12:58 | |
| derekh | rbrady: lucky guess, but I think maybe your original comment was valid, we should probably do it the other way to be consistent with the other examples | 12:59 |
| rbrady | derekh: +1 | 12:59 |
| *** ci-overcloud has joined #tripleo | 13:03 | |
| ci-overcloud | ************** ci-overcloud complete status=0 ************ | 13:03 |
| *** ci-overcloud has quit IRC | 13:03 | |
| *** tzumainn has joined #tripleo | 13:14 | |
| *** athomas has joined #tripleo | 13:16 | |
| *** Shrews has joined #tripleo | 13:17 | |
| *** Shrews has quit IRC | 13:18 | |
| *** Shrews has joined #tripleo | 13:18 | |
| *** jcooley_ has joined #tripleo | 13:19 | |
| *** jdob has joined #tripleo | 13:25 | |
| *** jcooley_ has quit IRC | 13:26 | |
| *** CaptTofu has quit IRC | 13:30 | |
| *** weshay has joined #tripleo | 13:33 | |
| openstackgerrit | Ralf Haferkamp proposed a change to openstack/tripleo-image-elements: Install systemd service files into /usr/lib/ https://review.openstack.org/71900 | 13:33 |
| Ng | Morning | 13:43 |
| *** CaptTofu has joined #tripleo | 13:44 | |
| openstackgerrit | Jay Dobies proposed a change to openstack/tuskar: Fix tuskar docs building https://review.openstack.org/71633 | 13:45 |
| openstackgerrit | Jay Dobies proposed a change to openstack/tuskar: Fix tuskar docs building https://review.openstack.org/71633 | 13:47 |
| rlandy | rbrady: ping | 13:52 |
| *** lucas-hungry is now known as lucasagomes | 13:58 | |
| *** akuznetsov has joined #tripleo | 14:00 | |
| *** morazi has joined #tripleo | 14:03 | |
| *** ftcjeff has joined #tripleo | 14:07 | |
| *** dprince has joined #tripleo | 14:10 | |
| *** rlandy has quit IRC | 14:11 | |
| *** jcooley_ has joined #tripleo | 14:14 | |
| *** jdob has quit IRC | 14:17 | |
| *** jdob has joined #tripleo | 14:17 | |
| rbrady | rlandy: ack | 14:17 |
| *** jcooley_ has quit IRC | 14:19 | |
| openstackgerrit | Ralf Haferkamp proposed a change to openstack/tripleo-incubator: Use baremetal element to extract kernel and initrd for undercloud https://review.openstack.org/71912 | 14:32 |
| *** matty_dubs|gone is now known as matty_dubs | 14:32 | |
| openstackgerrit | Ralf Haferkamp proposed a change to openstack/tripleo-incubator: Use baremetal element to extract kernel and initrd for undercloud https://review.openstack.org/71912 | 14:33 |
| openstackgerrit | Ralf Haferkamp proposed a change to openstack/diskimage-builder: Remove unneeded service mapping for mysql on openSUSE https://review.openstack.org/71914 | 14:35 |
| *** edmund has joined #tripleo | 14:36 | |
| *** jayg|g0n3 is now known as jayg | 14:37 | |
| * Ng runs out briefly (hopefully!) to fetch the car from its service | 14:42 | |
| *** martyntaylor has joined #tripleo | 14:42 | |
| *** athomas has quit IRC | 14:44 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Add devtest support for power management https://review.openstack.org/67926 | 14:44 |
| *** jdob_ has joined #tripleo | 14:46 | |
| *** jdob has quit IRC | 14:46 | |
| GheRivero | morning TripleO | 14:53 |
| *** athomas has joined #tripleo | 14:55 | |
| rbrady | morning Ghe | 14:56 |
| *** jcooley_ has joined #tripleo | 15:08 | |
| *** rbrady is now known as rbrady-mtg | 15:08 | |
| *** boris-42_ has quit IRC | 15:12 | |
| * peoplemerge the sun rises on TripleO | 15:12 | |
| *** boris-42_ has joined #tripleo | 15:13 | |
| *** jcooley_ has quit IRC | 15:14 | |
| openstackgerrit | Ralf Haferkamp proposed a change to openstack/tripleo-incubator: Use baremetal element to extract kernel and initrd for undercloud https://review.openstack.org/71912 | 15:30 |
| openstackgerrit | Ralf Haferkamp proposed a change to openstack/tripleo-incubator: Use baremetal element to extract kernel and initrd for undercloud https://review.openstack.org/71912 | 15:32 |
| *** athomas has quit IRC | 15:35 | |
| *** jprovazn is now known as jprovazn_bbl | 15:41 | |
| *** cody-somerville has quit IRC | 15:58 | |
| *** jcooley_ has joined #tripleo | 16:01 | |
| *** athomas has joined #tripleo | 16:02 | |
| *** jcooley_ has quit IRC | 16:08 | |
| *** peoplemerge has quit IRC | 16:14 | |
| *** peoplemerge has joined #tripleo | 16:14 | |
| *** athomas has quit IRC | 16:17 | |
| *** bnemec is now known as beekneemech | 16:23 | |
| *** bauzas has quit IRC | 16:26 | |
| *** cody-somerville has joined #tripleo | 16:34 | |
| *** morazi has quit IRC | 16:36 | |
| *** cody-somerville has quit IRC | 16:39 | |
| *** morazi has joined #tripleo | 16:40 | |
| *** sballe has joined #tripleo | 16:41 | |
| *** CaptTofu has quit IRC | 16:43 | |
| *** CaptTofu has joined #tripleo | 16:43 | |
| *** CaptTofu has quit IRC | 16:48 | |
| *** e0ne has quit IRC | 16:48 | |
| *** CaptTofu has joined #tripleo | 16:49 | |
| *** taps has joined #tripleo | 16:52 | |
| *** vkozhukalov has quit IRC | 16:58 | |
| *** jcooley_ has joined #tripleo | 16:59 | |
| *** lblanchard has quit IRC | 17:01 | |
| *** markmc has quit IRC | 17:01 | |
| *** jcooley_ has quit IRC | 17:03 | |
| *** matty_dubs is now known as matty_dubs|lunch | 17:15 | |
| *** martyntaylor has left #tripleo | 17:17 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: Lets not recommend stackuser is used on testenv images https://review.openstack.org/71954 | 17:19 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: Add script to build and deploy a testenv stack https://review.openstack.org/71955 | 17:20 |
| *** lazy_prince has quit IRC | 17:23 | |
| *** gcha has quit IRC | 17:24 | |
| *** rbrady-mtg is now known as rbrady | 17:24 | |
| *** martyntaylor has joined #tripleo | 17:24 | |
| *** bauzas has joined #tripleo | 17:28 | |
| *** jayg has quit IRC | 17:28 | |
| SpamapS | StevenK: I think you were asking about JSON parameters | 17:29 |
| SpamapS | lifeless: Or you were asking about JSON parameters | 17:29 |
| SpamapS | anyway https://review.openstack.org/#/c/70710/4/doc/source/template_guide/hot_spec.rst | 17:29 |
| SpamapS | About to land in HOT to be able to extract bits of them. :) | 17:29 |
| *** newell has joined #tripleo | 17:35 | |
| *** UtahDave has joined #tripleo | 17:38 | |
| *** akrivoka has joined #tripleo | 17:40 | |
| *** killer_prince has joined #tripleo | 17:46 | |
| *** jcooley_ has joined #tripleo | 17:50 | |
| *** max_lobur is now known as max_lobur_afk | 17:52 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: Add script to build and deploy a testenv stack https://review.openstack.org/71955 | 17:53 |
| *** jcooley_ has quit IRC | 17:56 | |
| *** hashar_ has quit IRC | 17:57 | |
| *** jprovazn_bbl has quit IRC | 17:59 | |
| *** jog0 is now known as flashgordon | 18:03 | |
| *** lblanchard has joined #tripleo | 18:04 | |
| *** cadenzajon has joined #tripleo | 18:05 | |
| *** lucasagomes has quit IRC | 18:11 | |
| *** martyntaylor has left #tripleo | 18:13 | |
| *** matty_dubs|lunch is now known as matty_dubs | 18:13 | |
| *** bauzas has quit IRC | 18:15 | |
| *** jcooley_ has joined #tripleo | 18:16 | |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Add disk-image-edit utility to help edit images. https://review.openstack.org/71958 | 18:16 |
| *** jcooley_ has quit IRC | 18:20 | |
| *** e0ne has joined #tripleo | 18:24 | |
| lifeless | SpamapS: I was asking about taking a single param and referencing different bits in the template | 18:37 |
| *** jcooley_ has joined #tripleo | 18:40 | |
| cadenzajon | looking at your OpenStack image elements, which distribution is being targeted by the existing setup scripts and conf files? | 18:41 |
| cadenzajon | I don't see branches for havana/icehouse/etc... how are these managed? | 18:42 |
| lifeless | cadenzajon: we support rhel,fedora,suse,ubuntu,centos at the moment | 18:45 |
| SpamapS | lifeless: yeah, that is about to land | 18:45 |
| SpamapS | lifeless: get_param [ "json_param_name", "key1", "key2" ] | 18:46 |
| lifeless | cadenzajon: we deploy trunk, and folk interested in static releases are going to take a snapshot at release time | 18:46 |
| *** jcooley_ has quit IRC | 18:47 | |
| cadenzajon | with tripleo do you typically run the *-package-install or *-source-install scripts for your elements? | 18:48 |
| *** jcooley_ has joined #tripleo | 18:48 | |
| lifeless | the package-install stuff is all new; while I support folk that want to do it, I think its a bit of a waste | 18:50 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: ifquery doesn't exist on Fedora https://review.openstack.org/71962 | 18:50 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Support adding DHCP interfaces one at a time. https://review.openstack.org/71963 | 18:50 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Add Fedora DHCP interfaces via udev rules https://review.openstack.org/71964 | 18:50 |
| lifeless | in that the unit of deployment in a cloud is a VM | 18:50 |
| cadenzajon | makes sense | 18:50 |
| *** lblanchard has quit IRC | 18:51 | |
| SpamapS | cadenzajon: the packaging stuff is so that packagers can build images using their desired support model but still take advantage of the other tripleo goodness. | 18:52 |
| openstackgerrit | A change was merged to openstack/tuskar: Added overcloud reload after create https://review.openstack.org/71569 | 18:52 |
| *** jcooley_ has quit IRC | 18:52 | |
| cadenzajon | who is doing this kind of packaging so far? | 18:52 |
| *** jcooley_ has joined #tripleo | 18:53 | |
| SpamapS | cadenzajon: RedHat :) | 18:54 |
| *** lblanchard has joined #tripleo | 18:54 | |
| cadenzajon | have they done it in a product release yet, or all R&D at this point? | 18:56 |
| *** martyntaylor has joined #tripleo | 18:56 | |
| *** martyntaylor has left #tripleo | 18:56 | |
| openstackgerrit | A change was merged to openstack/tuskar: Added missing licence and enabled gating on H102 https://review.openstack.org/71573 | 18:56 |
| slagle | cadenzajon: there have been releases of the packaged openstack projects | 18:57 |
| slagle | cadenzajon: there is also a community version of the packages for fedora | 18:58 |
| slagle | as far as getting tripleo to use those packages, that is a WIP | 18:58 |
| *** wusWWW has joined #tripleo | 19:00 | |
| SpamapS | lifeless: https://blueprints.launchpad.net/heat/+spec/update-hooks <-- graceful stuff.. writing up a spec now | 19:00 |
| *** jayg|g0n` has joined #tripleo | 19:01 | |
| *** epim has joined #tripleo | 19:02 | |
| *** shakayumi has joined #tripleo | 19:02 | |
| *** jcooley_ has quit IRC | 19:06 | |
| *** jcooley_ has joined #tripleo | 19:13 | |
| *** annashen has left #tripleo | 19:16 | |
| hewbrocca | SpamapS: watcha gonna use, cloud-init? | 19:17 |
| openstackgerrit | Ben Nemec proposed a change to openstack/tripleo-incubator: Make load-image check image checksum https://review.openstack.org/71662 | 19:23 |
| Ng | oh how I wish my headless workstation for tripleo testing, had an iLO | 19:27 |
| * Ng trundles upstairs for a monitor to figure out wtf it's doing | 19:27 | |
| lifeless | sadface | 19:44 |
| lifeless | $ nova keypair-show default | 19:44 |
| lifeless | ERROR: can't set attribute | 19:44 |
| lifeless | still | 19:44 |
| lifeless | derekh: hi,nvm on the keypair - I recreated the broker | 19:48 |
| lifeless | derekh: with the prod keys keychain | 19:49 |
| SpamapS | hewbrocca: cloud-init is a first boot tool | 19:49 |
| SpamapS | hewbrocca: so no | 19:49 |
| hewbrocca | ahh so it is | 19:50 |
| *** julim has joined #tripleo | 19:51 | |
| SpamapS | hewbrocca: we already have os-collect-config which scrapes metadata and runs hooks when it changes. | 19:52 |
| hewbrocca | oh, right, yes | 19:52 |
| hewbrocca | I was thinking of that when I said cloud-init even though I know that's not what cloud-init is | 19:53 |
| SpamapS | hewbrocca: though I am a little skeptical that our nice linear workflow of "config changed, -> do all the things" will work in this case. We may need to have a way to tell os-refresh-config to stop without just raising a random error leading to os-refresh-config failing and thus os-collect-config trying again in 30s | 19:54 |
| SpamapS | Like, in the case of a compute node, what we want to do if there is an evacuate/migrate configuration signaled is to evacuate/migrate everything, and then quiesce. We don't want to restart everything. | 19:56 |
| hewbrocca | Right | 19:57 |
| *** dprince has quit IRC | 19:59 | |
| lifeless | SpamapS: I think your logic is flawed. | 19:59 |
| SpamapS | lifeless: Of course you do | 19:59 |
| SpamapS | It wouldn't be Friday if my logic was on target. :) | 20:00 |
| lifeless | SpamapS: we don't know when we see evacuate set that evacuate being set is the only change | 20:00 |
| SpamapS | lifeless: No, we don't. So how about "{evacuate: true, offline: true}" | 20:01 |
| lifeless | SpamapS: so we should apply the config we're given, and in migrations (after everything is done) then evacuate is a migration | 20:01 |
| lifeless | SpamapS: and offline might in fact be represented in the config file to tell the hypervisor to stop advertising itself for incoming deploys | 20:01 |
| SpamapS | oh.. we never did change that order. | 20:02 |
| lifeless | [or it might be a signal, or $whatever] | 20:02 |
| lifeless | SpamapS: no time like the OMG NOW | 20:02 |
| SpamapS | I don't think we have any migrations | 20:02 |
| SpamapS | so.. agreed :) | 20:02 |
| SpamapS | looks like cinder-volume has a migration | 20:07 |
| SpamapS | which assumes things have not been started yet | 20:07 |
| devananda | lifeless: i see you using threading.semaphore in 71219. makes sense to me. why not use https://github.com/openstack/oslo-incubator/blob/master/openstack/common/lockutils.py#L215 ? | 20:08 |
| SpamapS | I think we can move that to "right before restarting cinder-volume" though. | 20:08 |
| *** akrivoka has quit IRC | 20:08 | |
| lifeless | devananda: AFAIK nova compute doesn't run multiple worker processes. | 20:09 |
| lifeless | devananda: that thing is an external file lock utility | 20:09 |
| *** tzumainn has quit IRC | 20:09 | |
| lifeless | devananda: oh, no - its more complex than that | 20:10 |
| lifeless | devananda: but basically, I didn't think to look for non-librarised code for such a trivial application | 20:10 |
| lifeless | devananda: *and*, we may want to make it dialable in future, e.g. up to 3 converts at once on a larger machine | 20:11 |
| devananda | lifeless: i *thought* it was just an external file lock utility... until I read the code again a few minutes ago | 20:11 |
| openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack/tripleo-image-elements: Move cinder-volume creation to post-configure https://review.openstack.org/71975 | 20:11 |
| devananda | it looks like it does the same thing now by default (threading.Semaphore()) | 20:11 |
| lifeless | devananda: so a lock is the wrong conceptual tool | 20:12 |
| lifeless | our nova needed patches are all merged, we could reset nova to trunk. | 20:13 |
| lifeless | do we dare :P | 20:13 |
| devananda | lifeless: diable -- yes. see my comment on the review. I think Ironic should have the same feature | 20:13 |
| Ng | ooi, has anyone with a z420 managed to get any usefulness out of it with IPMI? | 20:13 |
| Ng | I may have to use my wemo switch and write a power driver for it ;) | 20:14 |
| lifeless | devananda: see my reply | 20:16 |
| *** vkozhukalov has joined #tripleo | 20:16 | |
| *** edmund has quit IRC | 20:17 | |
| openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack/os-refresh-config: Move migration to run after post-configure https://review.openstack.org/71977 | 20:17 |
| *** lblanchard has quit IRC | 20:19 | |
| greghaynes | \O/ gerrit downtime right when I wanted to link about a review | 20:21 |
| SpamapS | greghaynes: ... said 1500 people all at once | 20:21 |
| greghaynes | It seems the most recent commit to tripleo-incubator added some required keys for testenv.json that arent added in the setupenv.sh | 20:22 |
| greghaynes | SpamapS: ikr | 20:22 |
| *** taps has quit IRC | 20:22 | |
| lifeless | greghaynes: the undercloud ones are added by the undercloud script | 20:23 |
| lifeless | greghaynes: _testenv only adds the baseline set | 20:23 |
| *** cody-somerville has joined #tripleo | 20:24 | |
| *** cody-somerville has joined #tripleo | 20:24 | |
| greghaynes | hrm, well ./devtest.sh fails for seed due to not having the seed keys | 20:24 |
| *** taps has joined #tripleo | 20:24 | |
| lifeless | greghaynes: which key is missing ? | 20:25 |
| greghaynes | SEED_PM_IPS= .. node-pm-ips | 20:25 |
| greghaynes | Is that supposed to be optional to have? | 20:26 |
| *** lblanchard has joined #tripleo | 20:26 | |
| lifeless | oh, this must have snuck in overnight | 20:26 |
| lifeless | dan had a patch for doing power management stuff that I was pushing back on | 20:27 |
| greghaynes | hehe | 20:27 |
| *** edmund has joined #tripleo | 20:27 | |
| greghaynes | https://review.openstack.org/#/c/67926/ | 20:28 |
| lifeless | also it's added typos | 20:28 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-incubator: Revert "Add devtest support for power management" https://review.openstack.org/71980 | 20:30 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-incubator: Permit specifying ovs bridge mappings too. https://review.openstack.org/71706 | 20:30 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-incubator: Permit setting a /etc/hosts entry for the API. https://review.openstack.org/71727 | 20:30 |
| lifeless | SpamapS: firedrill ^ | 20:30 |
| greghaynes | :) | 20:32 |
| ccrouch | lifeless: have we got one of these for sunnyvale yet? https://etherpad.openstack.org/p/tripleo-havana-sprint | 20:33 |
| lifeless | ccrouch: no | 20:33 |
| ccrouch | no worries | 20:33 |
| ccrouch | just checking i hadnt missed it | 20:33 |
| *** openstackgerrit has quit IRC | 20:34 | |
| *** openstackgerrit has joined #tripleo | 20:34 | |
| *** tzumainn has joined #tripleo | 20:37 | |
| *** akuznetsov has quit IRC | 20:38 | |
| *** lblanchard has quit IRC | 20:44 | |
| lifeless | ccrouch: feel free to make one ;) | 20:46 |
| *** cadenzajon has quit IRC | 20:47 | |
| *** akuznetsov has joined #tripleo | 20:48 | |
| ccrouch | lifeless: be careful what you wish for :-) | 20:49 |
| lifeless | 32G of ram in this laptop? | 20:51 |
| SpamapS | lifeless: all we had to do was add --key-default '' | 20:51 |
| SpamapS | lifeless: still want to revert? | 20:51 |
| ccrouch | ha | 20:51 |
| lifeless | SpamapS: feel free to push a different fix - but be sure to do a proper review of the docs at the same time | 20:51 |
| lifeless | SpamapS: that patch was not properly reviewed AFAICT | 20:52 |
| lifeless | SpamapS: for instance | 20:52 |
| lifeless | -## #. node-macs: A space separated string containing the MAC addresses of eth0 of | 20:52 |
| lifeless | +## #. node-macs: A space separated string containing the MAC addresses of eth0 | 20:52 |
| lifeless | ## each of the emulated bare metal VMs. | 20:52 |
| lifeless | the new docs read 'of eth0 each of' | 20:52 |
| *** akuznetsov has quit IRC | 20:52 | |
| SpamapS | copy/paste fail :-P | 20:52 |
| lifeless | SpamapS: something; I think its better to re-review it properly, my -1 concerns aside. | 20:53 |
| lifeless | (which I still hold) | 20:53 |
| SpamapS | heh it is an awkward sentence that is being constructed | 20:55 |
| SpamapS | the MAC addresses of eth0 of each of the emulated bare metal VMs | 20:55 |
| lifeless | is what it had, which while awkward is correct. The new sentence is not :) | 20:55 |
| SpamapS | lifeless: I am not sure I understand why revert is better than fix. :-P | 20:56 |
| lifeless | SpamapS: *I* don't have time to fix right now. If you do cool. | 20:56 |
| lifeless | I'm here with Cynthia on saturday. | 20:56 |
| SpamapS | yeah I get that :) | 20:56 |
| lifeless | SpamapS: All I'm saying is that if you're going to fix, its not just the devtest error. | 20:56 |
| *** CaptTofu has quit IRC | 20:57 | |
| lifeless | The patch wasn't thoroughly reviewed AFAICT - it sat there with a conceptual -1, then toggled to all-ok with no detailed critique. | 20:57 |
| lifeless | I base this on a) the missing --key-default which review *should* have caught, and b) the grammar error likewise, and I'm speculating that there may be *other* things wrong too. | 20:58 |
| SpamapS | lifeless: I didn't notice the problems.. doesn't mean it wasn't reviewed. :p | 20:58 |
| lifeless | anyhow, up to you | 20:58 |
| SpamapS | We just disagreed on the priority of this fix vs. fixing the json schema. | 20:58 |
| lifeless | I can swing by and do a review if you put fixes up | 20:58 |
| lifeless | SpamapS: I think this patch makes fixing the json schema harder. | 20:59 |
| SpamapS | I'm just about done with a fix | 20:59 |
| SpamapS | lifeless: meh.. only slightly | 20:59 |
| lifeless | SpamapS: because now its a backwards compat problem | 20:59 |
| lifeless | SpamapS: and since fixing the schema in the first place is about 30-50m work... | 20:59 |
| SpamapS | we have that problem for node-macs as much as everything else. | 20:59 |
| lifeless | right | 20:59 |
| lifeless | it should just be a vector of nova bm parameters | 20:59 |
| lifeless | very simple and directly related to the plumbing | 21:00 |
| SpamapS | Yeah thats what I think too. | 21:00 |
| slagle | i don't think backwards compat is a valid argument, given we make those types of changes *all* the time | 21:00 |
| lifeless | jq can pull that out for us trivially | 21:00 |
| lifeless | slagle: if we had CI working (which is what has consumed me this week - yak shaving ---galore--- | 21:00 |
| lifeless | slagle: it is much more of a concern | 21:00 |
| lifeless | slagle: three-step-dance across repositories; not 'cannot do', just 'more to do' | 21:01 |
| lifeless | ok, TE's are registering with the brkoer. | 21:01 |
| lifeless | time to let infra loose | 21:01 |
| openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack/tripleo-incubator: Make recently added node-pm-* parameters optional https://review.openstack.org/71993 | 21:03 |
| SpamapS | ok I'm running a whole new devtest now | 21:03 |
| SpamapS | aaaand just wonderful, brand new saucy images again | 21:10 |
| SpamapS | because we can't have nice things | 21:10 |
| *** cadenzajon has joined #tripleo | 21:13 | |
| SpamapS | whoa.. why are nearly all the boxes on cd-undercloud BUILD/spawning/NOSTATE ? | 21:16 |
| SpamapS | | 4d9dea4d-3a31-4e6a-9c04-f32b93da67c4 | ci-overcloud | CREATE_FAILED | 2014-02-07T10:01:51Z | | 21:16 |
| SpamapS | | f29ad903-a8bc-474b-a92b-686fc1c6a065 | overcloud | CREATE_FAILED | 2014-02-07T20:28:21Z | | 21:16 |
| SpamapS | | 3fc28265-aaaf-44c4-b1c8-f875b6b496b4 | testenv | CREATE_FAILED | 2014-02-07T20:41:07Z | | 21:16 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Seed specific Fedora network configuration https://review.openstack.org/71998 | 21:17 |
| lifeless | derekh: also - nova flavor-create --is-public true h1.large auto 8192 20 1 | 21:19 |
| lifeless | SpamapS: cause shit gets used | 21:19 |
| SpamapS | lifeless: this FAILED state.. this is my retry nightmare. :-/ | 21:19 |
| lifeless | SpamapS: so, ci-overcloud is *fine* don't touch. | 21:19 |
| SpamapS | no I won't touch the others | 21:20 |
| lifeless | SpamapS: its failed because the waitcond can't fire because we had to do stuff by hand | 21:20 |
| SpamapS | I just see it and I'm sad. :-P | 21:20 |
| lifeless | testenv should be fine once it finishes deploying | 21:20 |
| SpamapS | heat is dead at this point. :-P | 21:20 |
| lifeless | but heat has indeed packed a sad | 21:20 |
| lifeless | SpamapS: so, this is probably the 500 I reported a bug on | 21:20 |
| SpamapS | lifeless: ah yeah, bug fix is up and I think may be approved soon | 21:20 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Fix ci-overcloud endpoint for deploy-testenv. https://review.openstack.org/72002 | 21:21 |
| SpamapS | lifeless: actually, it is done | 21:22 |
| SpamapS | merged | 21:22 |
| SpamapS | let's wait until https://review.openstack.org/#/c/63839/ lands and then update Heat | 21:23 |
| SpamapS | (that way we'll get properly ordered event-list again) | 21:23 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Enable stable interface names on Fedora https://review.openstack.org/72003 | 21:24 |
| *** jayg|g0n` has quit IRC | 21:27 | |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Add lsof package to all Fedora images https://review.openstack.org/72005 | 21:30 |
| openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack/tripleo-image-elements: keystone crontab PATH needs /usr/local/bin https://review.openstack.org/72010 | 21:32 |
| lifeless | derekh: so we need to make the te stuff idempotent and handle config updates w/o redeploy, as well as network glitches | 21:33 |
| SpamapS | lifeless: ^^ the PATH fix there is why our keystone tokens got out of hand (it was silently failing) | 21:34 |
| SpamapS | lifeless: already installed on cd-undercloud | 21:34 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: keystone crontab PATH needs /usr/local/bin https://review.openstack.org/72010 | 21:36 |
| beekneemech | Could that have been PATH=/usr/local/bin:$PATH instead? Not that it probably matters, but a wholesale override to add one search directory seems like overkill. :-) | 21:38 |
| SpamapS | beekneemech: Perhaps? I don't recall if crontabs have it exposed to them like that. Probably. :-P | 21:38 |
| beekneemech | SpamapS: Yeah, I'm not sure either. That's part of why I asked. :-) | 21:39 |
| SpamapS | crond is like, that thing that is stuck in 1994 | 21:39 |
| SpamapS | nobody wants to address it | 21:39 |
| SpamapS | nobody cares... it just works and when it works like an old crabby bastard we just feed it some time and it doesn't bother us for another few months. | 21:39 |
| *** cd-undercloud has joined #tripleo | 21:40 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 21:40 |
| *** cd-undercloud has quit IRC | 21:40 | |
| * beekneemech has banged his head against weird cron environment problems more than enough over the years | 21:41 | |
| *** john-n-seattle has quit IRC | 21:44 | |
| lifeless | poor nova bm | 21:45 |
| lifeless | I have the action-sequences from cloudy with a chance of meatballs stuck in my head now | 21:45 |
| lifeless | 'building ipming waiting copying rebooting waiting ..' | 21:46 |
| SpamapS | haha | 21:47 |
| *** john-n-seattle1 has joined #tripleo | 21:48 | |
| *** jcooley_ has quit IRC | 21:48 | |
| *** d0ugal has quit IRC | 21:49 | |
| *** jcooley_ has joined #tripleo | 21:49 | |
| *** panda__ has quit IRC | 21:49 | |
| lifeless | devananda: replied again | 21:51 |
| devananda | lifeless: bmdh is serialized?? | 21:53 |
| devananda | https://github.com/openstack/nova/blob/master/nova/cmd/baremetal_deploy_helper.py#L262 | 21:53 |
| devananda | i thought it was threaded // could dd in parallel | 21:53 |
| devananda | oooh... it's just got one worker thread. lame | 21:54 |
| SpamapS | devananda: <threes company music>We've been waiting for youuuuuuu</threes company music> | 21:54 |
| devananda | hah | 21:55 |
| SpamapS | wow cd-undercloud is reasonably busy | 21:55 |
| SpamapS | we have to do something about token caching though | 21:55 |
| SpamapS | I see 7 openssl's running | 21:55 |
| SpamapS | which means 7 keystone tokens signing | 21:56 |
| lifeless | devananda: experiments with multithreading have so far been a pessimism | 21:56 |
| openstackgerrit | Dirk Mueller proposed a change to openstack/diskimage-builder: Add more package name mappings for openSUSE. https://review.openstack.org/71539 | 21:56 |
| lifeless | devananda: see e.g. slagles reponse on -dev | 21:56 |
| lifeless | devananda: so while you're right that its a limitation, its not a *new* limitation | 21:56 |
| slagle | yes, i tried it, b/c I didn't believe what was being told me :) | 21:57 |
| devananda | lifeless: ack | 21:57 |
| lifeless | and here's why | 21:57 |
| lifeless | say you're deploying from a big beefy machine with lots of memory to regula rmachines with 1 disk | 21:58 |
| lifeless | you'll be limited firstly by network IO (10Gbps say) until you exceed that machines ram | 21:58 |
| lifeless | then you'll be limited by disk IO until it's flushed to disk | 21:58 |
| lifeless | most machines have ram >> image size | 21:58 |
| lifeless | so practically speaking we expect to see network saturation then a pause while the disk catches up | 21:59 |
| lifeless | with the dd approach, ideally we would then have concurrency 1 for bulk transfer (optimise) and then allow concurrent ops while waiting for the fsync() | 22:00 |
| lifeless | now, say its not a big beefy deploy machine | 22:00 |
| lifeless | so you don't have enough ram to hold the source images in RAM at once | 22:00 |
| *** peoplemerge has quit IRC | 22:01 | |
| *** panda__ has joined #tripleo | 22:02 | |
| *** matty_dubs is now known as matty_dubs|gone | 22:03 | |
| lifeless | which means you'll be thrashing on IO if you do concurrent ops, your ideal performance there is linear read from one image, then linear read for the next one etc | 22:04 |
| lifeless | 1Gbps networks are (of course) simply worse | 22:04 |
| lifeless | but the interesting thing there is that even spinning platter disk can usually exceed 1Gbps | 22:05 |
| lifeless | so there is a temptation to use the exces disk capacity to serve many requests, but if you do that, you're immediately context switching on the network | 22:05 |
| lifeless | and thus reducing tcp window size | 22:06 |
| lifeless | -> average time to copy becomes longer | 22:06 |
| *** jdob_ has quit IRC | 22:07 | |
| lifeless | now, one interesting combination is 10Gbps on the deploy server, 1Gbps on the target machine | 22:08 |
| lifeless | there you can genuinely run 9-10 concurrent copies (if the deploy server has sufficient CPU and memory / disk IOPS) and have it be a net win | 22:08 |
| *** peoplemerge has joined #tripleo | 22:09 | |
| lifeless | but even then, given POST times, I don't think the serialisation of qemu-img convert will harm deploy times | 22:09 |
| lifeless | hmm, only 24 te's have come online | 22:13 |
| lifeless | something wrong there, but I can't investigate right now | 22:13 |
| SpamapS | | fault | {u'message': u'No valid host was found. Exceeded max scheduling attempts 3 for instance ec2cda9c-250b-4b94-a29e-74d00e71a629', u'code': 500, u'details': u' File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 147, in run_instance | | 22:13 |
| SpamapS | -rw-rw---- 1 mysql mysql 880M Feb 7 22:17 /var/lib/mysql/keystone/token.ibd | 22:17 |
| SpamapS | less than a day of tokens :-P | 22:17 |
| *** jrist has quit IRC | 22:18 | |
| *** sballe has quit IRC | 22:18 | |
| lifeless | SpamapS: that may be the PXE deploy timeout kicking in | 22:19 |
| *** julim has quit IRC | 22:19 | |
| lifeless | SpamapS: but - its not the problem I was referring to | 22:20 |
| lifeless | SpamapS: because - there are 8 machines online, 24 te's means 3 per machine | 22:20 |
| lifeless | which seems low, 24 cores we should get 4 per machine w/o oversubscription - so 6 of 8 machines have made it all the way up | 22:21 |
| SpamapS | ah | 22:25 |
| SpamapS | lifeless: I'm completely ignorant of the whole system you've setup. I only know the parts I've reviewed... :-P | 22:26 |
| lifeless | SpamapS: well then, you know what to do | 22:26 |
| SpamapS | ignore it some more? Agreed. :) | 22:27 |
| *** epim has quit IRC | 22:28 | |
| * SpamapS tries to write something intelligible to gracefully wait for update hooks | 22:28 | |
| lifeless | have fun | 22:28 |
| lifeless | I need to weekend harder - bai | 22:29 |
| *** jrist has joined #tripleo | 22:30 | |
| *** jcooley_ has quit IRC | 22:31 | |
| *** epim has joined #tripleo | 22:32 | |
| *** panda has joined #tripleo | 22:33 | |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Seed specific Fedora network configuration https://review.openstack.org/71998 | 22:34 |
| *** weshay has quit IRC | 22:36 | |
| *** panda__ has quit IRC | 22:36 | |
| lifeless | SpamapS: is multi-region heat working yet? | 22:36 |
| lifeless | SpamapS: properly that is, not micky mouse | 22:37 |
| SpamapS | lifeless: there is no such thing IIRC | 22:39 |
| SpamapS | lifeless: has not been on my radar at all | 22:40 |
| greghaynes | hrm, now ./devtest fails to make undercloud with 'No valid host found' in schedule_run_instance | 22:48 |
| greghaynes | although nova list showed it building for a while before erroring | 22:48 |
| lifeless | greghaynes: check nova log | 22:49 |
| lifeless | greghaynes: no valid host means 'tried N times and failed each time' | 22:49 |
| lifeless | there can be many actual causes | 22:49 |
| lifeless | nova-compute log specifically | 22:49 |
| greghaynes | ahoy | 22:49 |
| greghaynes | PXE | 22:50 |
| greghaynes | er, PXE deploy failed | 22:50 |
| greghaynes | unexpected error while running dd | 22:51 |
| greghaynes | Gotta love I/O Error | 22:53 |
| *** panda has quit IRC | 22:58 | |
| *** edmund has quit IRC | 22:59 | |
| *** rwsu has quit IRC | 23:01 | |
| *** jcooley_ has joined #tripleo | 23:01 | |
| *** rwsu has joined #tripleo | 23:02 | |
| lifeless | greghaynes: yay | 23:05 |
| greghaynes | AFAICT the iscsi connection gets reset part way through | 23:06 |
| lifeless | suggests a brbm issue | 23:06 |
| *** rwsu has quit IRC | 23:06 | |
| *** jcooley_ has quit IRC | 23:06 | |
| greghaynes | also cooincides with looking for a dhcp lease | 23:06 |
| lifeless | interesting | 23:07 |
| lifeless | ok, gone again | 23:07 |
| openstackgerrit | Dirk Mueller proposed a change to openstack/tripleo-image-elements: Nova-baremetal support for openSUSE https://review.openstack.org/72021 | 23:11 |
| *** panda has joined #tripleo | 23:12 | |
| *** cd-undercloud has joined #tripleo | 23:15 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 23:15 |
| *** cd-undercloud has quit IRC | 23:15 | |
| *** vkozhukalov has quit IRC | 23:16 | |
| *** rwsu has joined #tripleo | 23:19 | |
| flashgordon | SpamapS: ping | 23:21 |
| flashgordon | https://bugs.launchpad.net/nova/+bug/1272600 | 23:21 |
| uvirtbot | Launchpad bug 1272600 in tripleo "object of type 'NoneType' has no len() from neutronclient in get_instance_nwinfo if neutron credentials are missing" [Critical,Fix released] | 23:22 |
| flashgordon | is that still a valid nova bug? | 23:22 |
| *** Lotus907efi has joined #tripleo | 23:26 | |
| SpamapS | flashgordon: looking | 23:26 |
| flashgordon | SpamapS: thanks | 23:26 |
| *** wusWWW has quit IRC | 23:26 | |
| SpamapS | flashgordon: I've done nothing to test or fix it | 23:27 |
| SpamapS | flashgordon: AFAIK, it will happen again if we try the same thing again | 23:27 |
| flashgordon | SpamapS: thanks | 23:28 |
| *** cody-somerville_ has joined #tripleo | 23:30 | |
| *** cody-somerville_ has joined #tripleo | 23:30 | |
| *** cody-somerville has quit IRC | 23:31 | |
| *** morazi has quit IRC | 23:33 | |
| *** epim has quit IRC | 23:34 | |
| SpamapS | cores... https://review.openstack.org/#/c/71993/ needs merge.. soon | 23:40 |
| SpamapS | all is borked.. :( | 23:40 |
| SpamapS | Ng: ^^ | 23:41 |
| SpamapS | slagle: ^^ | 23:43 |
| * SpamapS must be going unfortunately | 23:43 | |
| *** e0ne has quit IRC | 23:44 | |
| *** e0ne has joined #tripleo | 23:46 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!