| *** lblanchard has joined #tripleo | 00:15 | |
| *** weshay has joined #tripleo | 00:16 | |
| *** blamar has quit IRC | 00:17 | |
| *** weshay has quit IRC | 00:21 | |
| *** matsuhashi has joined #tripleo | 00:21 | |
| SpamapS | lifeless: nova failing seems like it may have been a race we haven't identified yet... re-doing the overcloud owrked.. | 00:22 |
|---|---|---|
| SpamapS | dib-run-parts Tue Mar 11 00:21:12 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/74-swift-proxy | 00:22 |
| SpamapS | stop: Unknown instance: | 00:22 |
| SpamapS | start: Job failed to start | 00:22 |
| SpamapS | except it didn't | 00:22 |
| SpamapS | Error: [swift-hash]: both swift_hash_path_suffix and swift_hash_path_prefix are missing from /etc/swift/swift.conf | 00:22 |
| giulivo | guys anyone around using devtest on f20? | 00:23 |
| giulivo | I think I'm facing a problem with the seed where "nova db sync" fails | 00:23 |
| SpamapS | h | 00:23 |
| SpamapS | swift.hash is "" | 00:23 |
| giulivo | with "OperationalError: (OperationalError) (2006, 'MySQL server has gone away') None None" | 00:23 |
| giulivo | anyone has seen that before? | 00:24 |
| SpamapS | giulivo: check your mysql logs.. that would be a disconnect of some kind.. | 00:24 |
| SpamapS | giulivo: either mysqld crashed or something else | 00:24 |
| giulivo | SpamapS, yeah I was checking that and it is actually running but yes | 00:24 |
| giulivo | sometimes I get disconnected during long queries | 00:24 |
| giulivo | any idea what could be going on? | 00:24 |
| SpamapS | giulivo: could be a wait_timeout too long | 00:24 |
| giulivo | I basically reproduced this three times with three attempts to devtest_seed.sh | 00:25 |
| SpamapS | there are a few timeouts :p | 00:25 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: Allow the testenv client to dictate the worker timeout https://review.openstack.org/76171 | 00:25 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: Set the worker timeout https://review.openstack.org/76172 | 00:25 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: Kill commands that outlive test env https://review.openstack.org/75402 | 00:25 |
| SpamapS | lifeless: ok I just forgot to source tripleo-overcloud-passwords before deploying... | 00:27 |
| giulivo | SpamapS, just to make sure, why too long? | 00:28 |
| SpamapS | ... this is where retry would come in really handy :-/ | 00:28 |
| SpamapS | giulivo: strike that. Reverse it. ;) | 00:28 |
| giulivo | ok :) | 00:28 |
| giulivo | SpamapS, but it is set to 28800 so ... :( | 00:29 |
| SpamapS | giulivo: ok so that's one thing down :-P | 00:30 |
| giulivo | SpamapS, interestingly I got the "gone down" even in the interactive shell while doing "show variables" | 00:31 |
| SpamapS | lifeless: side note, this _might_ be causing havoc: https://bugs.launchpad.net/keystone/+bug/1289935 | 00:33 |
| SpamapS | giulivo: yeah are you positive your mysqld isn't dying and being resurrected by systemd or mysqld_safe ? | 00:34 |
| giulivo | SpamapS, nah I'm on your side | 00:34 |
| *** blamar has joined #tripleo | 00:34 | |
| giulivo | it is probably crashing | 00:34 |
| *** CaptTofu has quit IRC | 00:39 | |
| *** lucasagomes has quit IRC | 00:53 | |
| *** spzala has quit IRC | 00:55 | |
| derekh | SpamapS: are you looking into random overcloud jobs taking longer then expected ? | 01:04 |
| derekh | SpamapS: if so, I just hit the same thing here while running other tests | 01:05 |
| derekh | SpamapS: In my case I *think* I know the problem | 01:05 |
| derekh | SpamapS: we're leaving baremetal nodes lieing around running, which may have IP's conflicting with the new testenv being setup | 01:06 |
| derekh | http://paste.fedoraproject.org/84151/99849139/ | 01:06 |
| SpamapS | derekh: ew | 01:06 |
| SpamapS | derekh: don't we use cleanup-env or something like that? | 01:07 |
| derekh | SpamapS: nope, we don't have the access needed for cleanup-env | 01:07 |
| derekh | SpamapS: the domains are setup once when the testenv host comes up | 01:08 |
| SpamapS | ahh | 01:09 |
| SpamapS | right first boot thingy | 01:09 |
| SpamapS | derekh: so, we're racing with still-shutting-down hosts, or forgetting to shut down hosts? | 01:10 |
| derekh | SpamapS: we were only shutting down the seed vm, not the others at all | 01:10 |
| SpamapS | AHHHH | 01:10 |
| SpamapS | doh | 01:10 |
| SpamapS | derekh: that does sound suspicious | 01:11 |
| *** cwolferh_ has quit IRC | 01:13 | |
| derekh | SpamapS: gonna push it so we can see what happens | 01:14 |
| *** rbrady-afk is now known as rbrady | 01:14 | |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/75670 | 01:14 |
| SpamapS | derekh: wheee | 01:14 |
| rbrady | woot! my patch finally passed CI! | 01:14 |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-incubator: Destroy all ci testenv domains https://review.openstack.org/79502 | 01:14 |
| SpamapS | rbrady: feels like the heavens opening up and shining on you doesn't it? | 01:15 |
| lifeless | SpamapS: https://review.openstack.org/#/c/79480/ +A please | 01:15 |
| lifeless | SpamapS: will let me test the faster CI path | 01:15 |
| rbrady | SpamapS: yes! | 01:15 |
| *** CaptTofu has joined #tripleo | 01:15 | |
| derekh | the more I think about it the more I think this would cause all kinds of problems.... | 01:16 |
| lifeless | derekh: what would ? | 01:16 |
| lifeless | derekh: 79502? nova bm / ironic should take care of it already | 01:16 |
| derekh | lifeless: it would only assert state of the node its about to try and boot, wouldn't it? | 01:17 |
| SpamapS | shouldn't we be carefully doing a 'stack-delete overcloud' and then 'stack-delete undercloud' ? | 01:17 |
| lifeless | derekh: nope | 01:17 |
| lifeless | derekh: anything registered with it | 01:17 |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Give the seed and undercloud more IPs to work with https://review.openstack.org/79480 | 01:18 |
| SpamapS | That exercises a ton of really poorly exercised paths in OpenStack and it would turn all the testenv machines off | 01:18 |
| SpamapS | except it might not turn off CREATE_FAILED instances.. so hm | 01:18 |
| derekh | lifeless: ok, well something is wrong with nove-bm then, I've been watching the nodes not get destroyed | 01:18 |
| SpamapS | If we're just ripping down the undercloud and overcloud what is powering off those machines? | 01:19 |
| derekh | SpamapS: but we're not ripping down the undercloud or overcloud, all we do is "virsh destroy seed_x" | 01:20 |
| tchaypo | destroy == power down | 01:20 |
| *** retr0h has joined #tripleo | 01:20 | |
| *** lblanchard has quit IRC | 01:20 | |
| derekh | tchaypo: yup, my point was we only do it for the seed, the rest are left running | 01:21 |
| tchaypo | yeah, I'm not sure what SpamapS' was asking though | 01:21 |
| lifeless | derekh: ok, so - its possible that we have state asserting turned off | 01:22 |
| lifeless | derekh: or maybe its an ironic only feature (but I didn't think so) | 01:22 |
| lifeless | devananda: ^ | 01:22 |
| devananda | ? | 01:22 |
| lifeless | devananda: does nova-bm force off nodes without an instance on them ? | 01:22 |
| lifeless | devananda: as we were discussing for Ironic the other day ? | 01:23 |
| *** rpodolyaka has quit IRC | 01:23 | |
| *** blamar has quit IRC | 01:23 | |
| * SpamapS wonders if this is finally the thing that will drive us onto ironic ;) | 01:23 | |
| devananda | lifeless: ironic certainly does | 01:23 |
| derekh | lifeless: even if it did, the seed only knows about one of the nodes, so would only shut that one down | 01:23 |
| devananda | lifeless: lemme take a look at current nova-bm code | 01:24 |
| lifeless | derekh: yes, but that one is always the undercloud, no ? | 01:24 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Adds nfs-server element https://review.openstack.org/74712 | 01:24 |
| devananda | lifeless: is the context here a newly enrolled node (no prior instance) or something else? | 01:25 |
| derekh | lifeless: ya, thats probably true | 01:25 |
| derekh | lifeless: will rerun my test | 01:25 |
| *** rpodolyaka has joined #tripleo | 01:25 | |
| lifeless | devananda: previous test VM | 01:25 |
| lifeless | devananda: so we deploy new seed, baremetal_N vm may still be running | 01:26 |
| devananda | lifeless: and baremetal_N is enrolled with new seed, I presume | 01:26 |
| devananda | so from seed's POV, that node had no prior instance | 01:26 |
| devananda | lifeless: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5108 | 01:27 |
| devananda | hrm | 01:29 |
| devananda | ignore that, still digging | 01:29 |
| devananda | derekh: do you get this log message? https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5093 | 01:29 |
| jogo | lifeless: ((message:"Build timed out (after" AND message:"minutes). Marking the build as failed.") OR (message:"./safe-devstack-vm-gate-wrap.sh: line" AND message:"Killed timeout -s 9 ${DEVSTACK_GATE_TIMEOUT}m $BASE/new/devstack-gate/devstack-vm-gate.sh"))AND filename:"console.html" AND build_queue:"check-tripleo" | 01:30 |
| derekh | devananda: new seed booting up, will know in a few minutes | 01:30 |
| jogo | bug Bug 1258682 | 01:31 |
| jogo | seeing a lot of hits for tripleo, which makes sense | 01:31 |
| giulivo | SpamapS, in case you're still around for some help | 01:31 |
| giulivo | basically I'm installing using qpidd and qpid.username qpid.password are not defined in os-collect-config/heat_local.json | 01:32 |
| giulivo | so it is failing and executing again and again every 30secs (restarting mysql too) | 01:32 |
| giulivo | now, in the seed, I see heat_local.json is recreated every 30sec with the os-collect-config run | 01:33 |
| giulivo | where does the data comes from? | 01:33 |
| derekh | devananda: no sign of that log message yet | 01:35 |
| devananda | derekh: anything interesting in n-cpu? | 01:35 |
| derekh | devananda: lifeless none of the vm were killed on register bm nodes, only one restarted on heat stack-create overcloud (the rest are left untouched running) | 01:36 |
| derekh | second vm restarted now | 01:37 |
| derekh | devananda: nothing much, only logging on warn level http://paste.openstack.org/show/73092/ | 01:38 |
| devananda | derekh: ok. so a VM restarted ~1 minute later? | 01:39 |
| derekh | and now 3rd | 01:39 |
| devananda | derekh: not stopped -- it restarted? | 01:39 |
| *** nosnos has joined #tripleo | 01:40 | |
| derekh | devananda: actually probably destory/start the virsh id changed | 01:40 |
| derekh | so here is what I see , seed gets 8 bm nodes registered (all running VM's), all are left running, the I do a "heat stack-create overcloud", one of the vm is destroyed/started (based on virsh id changing), then when that nodes becomes active, the second is destroted/started and the 3rd | 01:42 |
| derekh | so we need something to kill those , weither its nova-bm or something like https://review.openstack.org/#/c/79502/1 | 01:44 |
| derekh | brb | 01:44 |
| lifeless | giulivo: is this your seed? | 01:45 |
| lifeless | giulivo: or undercloud/overcloud? | 01:45 |
| giulivo | lifeless, hi thanks for pinging back, my seed yes | 01:45 |
| lifeless | giulivo: are you following devtest?I believe we have examples in there for changing config.json to use qpidd | 01:46 |
| giulivo | I'm following devtest but I might have missed bits :( | 01:46 |
| devananda | derekh: what does 'nova list' and 'nova baremetal-node-list' show? | 01:47 |
| devananda | derekh: and: mysql nova -e "select count(*) from instances" | 01:47 |
| giulivo | lifeless, when you say config.json you mean $TE_DATAFILE ? | 01:47 |
| lifeless | giulivo: no, the seed stack config file | 01:48 |
| lifeless | giulivo: which replaces heat since yuo can't deploy the seed with heat | 01:48 |
| *** tserong has quit IRC | 01:48 | |
| derekh | devananda: http://paste.openstack.org/show/73093/ | 01:49 |
| devananda | derekh: fwiw, spawn should be rebooting them anyway: https://github.com/openstack/nova/blob/master/nova/virt/baremetal/driver.py#L294 | 01:49 |
| devananda | derekh: what exactly is the problem? | 01:49 |
| derekh | devananda: some of the VM are left behind (running) from previous ci runs, so unless they are all killed they will have existing IP's that conflict with a new ci run | 01:50 |
| devananda | ahh | 01:50 |
| devananda | so | 01:51 |
| devananda | you should kill them before tearing down the old seed | 01:51 |
| devananda | nova.compute.manager's sync_power_state logic only works on instances | 01:51 |
| devananda | those left over VMs don't have an instance uuid in the new seed, therefor they aren't synced | 01:52 |
| giulivo | lifeless, so that's the config.json created by the seed-stack-config element, correct? | 01:52 |
| *** rpodolyaka has quit IRC | 01:52 | |
| devananda | and the bm virt driver can't run its own periodic tasks | 01:52 |
| *** rpodolyaka has joined #tripleo | 01:52 | |
| devananda | lifeless: options seem to be: kill VMs before shutdown, add a new periodic task or init hook to nova.compute.manager and push the power state down to the driver, or use ironic | 01:53 |
| devananda | or an external script | 01:53 |
| devananda | that might work too | 01:54 |
| devananda | run just after you enroll the VMs and before heat stack-create | 01:54 |
| devananda | oh, and add a new API hook for nova-bm to expose a set-power-state | 01:54 |
| devananda | i forgot nova doesn't have that | 01:55 |
| lifeless | derekh: I'll +2 your patch given that, if you could please add a note that we don't need it in Ironic ? And perhaps a check for the variable devananda has set for Ironic use... | 01:56 |
| *** tserong has joined #tripleo | 01:56 | |
| *** tserong has joined #tripleo | 01:56 | |
| derekh | lifeless: will do | 01:57 |
| giulivo | lifeless, hacking cfn-init-data worked, maybe we should just add the qpid section here https://github.com/openstack/tripleo-image-elements/blob/master/elements/seed-stack-config/config.json ? | 01:59 |
| derekh | devananda: what can I check to see if ironic is being used? I only see IRONIC_PASSWORD in tripleo-incubator , suppose I could check for ironic in the element list | 02:02 |
| devananda | lifeless: fwiw, https://github.com/openstack/ironic/blob/master/ironic/conductor/manager.py#L95 | 02:02 |
| * StevenK has a sad about people linking github urls | 02:02 | |
| *** Qiming has joined #tripleo | 02:02 | |
| *** antonym has quit IRC | 02:03 | |
| lifeless | derekh: devananda has a patch set up for incubator | 02:03 |
| derekh | lifeless: k | 02:03 |
| lifeless | derekh: I'm suggesting using the same variable name that uses | 02:03 |
| *** antonym has joined #tripleo | 02:03 | |
| lifeless | giulivo: we can add qpid just-in-time rather than statically, was more my point | 02:03 |
| *** Kim-Chi-San has quit IRC | 02:04 | |
| giulivo | lifeless, I'm not there yet, don't know the process very well, need guidance or if you can do it I would check the submission | 02:04 |
| *** spzala has joined #tripleo | 02:04 | |
| *** Kim-Chi-San has joined #tripleo | 02:04 | |
| devananda | StevenK: SpamapS: what's the status of the ironic undercloud work? | 02:05 |
| lifeless | giulivo: I've no idea about qpidd :) | 02:06 |
| StevenK | devananda: The JSON nodes work has landed again, I'll be pushing up a new patchset for toci after lunch | 02:06 |
| devananda | StevenK: awesome | 02:06 |
| giulivo | lifeless, ok so basically this is working fine https://github.com/openstack/tripleo-image-elements/blob/master/elements/qpidd/os-refresh-config/post-configure.d/73-qpidd but in my seed failed because in cfn-init-data I did not have qpid.username and qpid.password | 02:07 |
| devananda | StevenK: did the bits to start using the nova-ironic driver from our tree make it anywhere yet? | 02:07 |
| giulivo | they aren't defined in the seed-stack-config/config.json | 02:07 |
| StevenK | devananda: I don't think so, I think the patch in incubator needs a rework now that nova-ironic is in ironic | 02:07 |
| giulivo | but I don't know how those could be put in place just-in-time | 02:07 |
| lifeless | giulivo: righto, so it looks like the rh folk didn't get through to incubator seed setup with their qpid work | 02:07 |
| StevenK | devananda: I can recall words to that effect in the stream of e-mail I injested this morning. | 02:08 |
| devananda | StevenK: heh :) | 02:08 |
| lifeless | devananda: I -1'd the patch yesterday | 02:08 |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-incubator: Destroy all ci testenv domains https://review.openstack.org/79502 | 02:08 |
| lifeless | devananda: needs fixing | 02:08 |
| lifeless | derekh: reviewed | 02:09 |
| *** rpodolyaka has quit IRC | 02:10 | |
| derekh | lifeless: replied | 02:11 |
| lifeless | derekh: replied :) | 02:18 |
| *** lazy_prince is now known as killer_prince | 02:18 | |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-incubator: Destroy all testenv domains https://review.openstack.org/79502 | 02:24 |
| lifeless | nuts | 02:24 |
| lifeless | I think gear has falled over again | 02:24 |
| derekh | lifeless: updated | 02:25 |
| *** spzala has quit IRC | 02:26 | |
| derekh | lifeless: possibly, I don't think we've fixed the problem yet, this has to make it to testenv hosts https://review.openstack.org/#/c/76546/ | 02:26 |
| lifeless | bouncing it | 02:26 |
| derekh | lifeless: and this will be needed to https://review.openstack.org/#/c/76588/ to avoid a problem we would hit at a later stage | 02:27 |
| lifeless | derekh: wait, what time is it for you ? :) | 02:27 |
| derekh | lifeless: 2:30 AM | 02:27 |
| derekh | lifeless: which makes it sleep time :-) | 02:27 |
| lifeless | geard restarted | 02:27 |
| lifeless | derekh: fairly sure we applied https://review.openstack.org/#/c/76546/ | 02:28 |
| derekh | lifeless: nope, doesn;t look like it on testenv-testenv3-mo3qqxo7373c | 02:29 |
| lifeless | derekh: ack, I've just pinged infra about the gear fix | 02:29 |
| derekh | lifeless: I gotta go sleep but can do that tomorrow if ya want (but we should probably wait for the fix to gear) | 02:29 |
| lifeless | derekh: would be good to test (and deploy) stevenks worker change to add nodes to the JSON | 02:30 |
| lifeless | derekh: cleans up a bunch of stuff | 02:30 |
| *** rpodolyaka has joined #tripleo | 02:30 | |
| lifeless | devananda: whats the status of the Ironic nova driver, usability wise ? | 02:31 |
| derekh | lifeless: ok, I'll test a new local test env tomorrow and if alls good with push to production | 02:31 |
| derekh | if the gear change is in that is, would also be nice to get https://review.openstack.org/#/c/76171/3 in | 02:32 |
| derekh | ok, night, really going now | 02:32 |
| *** derekh has quit IRC | 02:33 | |
| devananda | lifeless: start, stop, you know, the basics | 02:35 |
| devananda | lifeless: console and volume aren't implemented yet. looks like rebuild isn't there yet either | 02:36 |
| killer_prince | slagle: thanks for reviewing https://review.openstack.org/#/c/79289/2 and comments :) | 02:36 |
| lifeless | devananda: so we need boot/rebuild with --preserve-ephemeral and then we should be able to put a CI job up fo rit | 02:38 |
| devananda | ack | 02:39 |
| killer_prince | slagle: i am not sure if it can be done via deploy images as it needs support from images which are being deployed.. as in greub should be present in image.. but i will give it a try and will come back on it | 02:39 |
| lifeless | devananda: (and I'll be thrilled at tht point :)) | 02:40 |
| killer_prince | slagle: could you also review https://review.openstack.org/#/c/79286/ pls.. | 02:40 |
| StevenK | lifeless: That needs fixing | 02:43 |
| lifeless | StevenK: that? | 02:43 |
| StevenK | lifeless: The toci change to populates nodes:[] | 02:43 |
| lifeless | StevenK: right | 02:43 |
| *** killer_prince is now known as lazy_prince | 02:44 | |
| openstackgerrit | Gregory Haynes proposed a change to openstack/diskimage-builder: Add apache2 mod_wsgi pkg map for suse https://review.openstack.org/77285 | 02:44 |
| *** lazy_prince is now known as killer_prince | 02:45 | |
| StevenK | lifeless: I made the toci change before we both took an axe to create-nodes and passed the JSON in. | 02:45 |
| *** rbrady is now known as rbrady-afk | 02:46 | |
| lifeless | ack | 02:49 |
| lifeless | SpamapS: what time is it for you atm ? | 02:50 |
| StevenK | AKA, yay, what the hell has DST done | 02:51 |
| StevenK | python -c 'print open("/root/.ssh/test_env_rsa").read().replace("\n", "\\n")' | 02:52 |
| StevenK | Why can't we use tr like normal people? | 02:53 |
| lifeless | sure, do so | 02:53 |
| openstackgerrit | yangxurong proposed a change to openstack/tuskar: Pin iso8601 logging to WARN https://review.openstack.org/79516 | 02:55 |
| openstackgerrit | lifeless proposed a change to openstack-infra/tripleo-ci: Run overcloud deploy tests against the seed. https://review.openstack.org/79220 | 02:56 |
| openstackgerrit | James Polley proposed a change to openstack/tripleo-incubator: User docs should tell the user to use ./ https://review.openstack.org/79517 | 02:56 |
| *** rpodolyaka has quit IRC | 02:58 | |
| * StevenK grumbles at jq not helpfully writing the data back | 02:59 | |
| tchaypo | StevenK: win | 02:59 |
| StevenK | tchaypo: Do you know where your luggage is? | 03:01 |
| StevenK | lifeless: I'd like to test this toci change. Just git review it and wait for jenkins? | 03:06 |
| *** vkozhukalov has quit IRC | 03:09 | |
| lifeless | StevenK: which toci script are you changing? | 03:09 |
| StevenK | elements/testenv-worker/bin/ensure-test-env | 03:09 |
| tchaypo | my luggage is on my bed | 03:10 |
| lifeless | StevenK: test that by deploying a test environment (script fo rthat is in tie/elements/tripleo-cd/ | 03:10 |
| lifeless | StevenK: and then using it | 03:10 |
| lifeless | StevenK: or you can run it by hand locally | 03:11 |
| StevenK | tchaypo: When did it get there? | 03:11 |
| tchaypo | about an hour ago | 03:12 |
| StevenK | tchaypo: Did you get the TSA luckydip? | 03:13 |
| StevenK | lifeless: Just running ensure-test-env locally doesn't seem like it will work, TBH ... | 03:14 |
| *** CaptTofu has quit IRC | 03:14 | |
| tchaypo | I didn't get a card | 03:15 |
| tchaypo | and I haven't noticed anything missing | 03:15 |
| *** sballe_ has joined #tripleo | 03:22 | |
| *** sballe has quit IRC | 03:24 | |
| *** tzumainn has quit IRC | 03:36 | |
| *** matsuhashi has quit IRC | 03:41 | |
| tchaypo | I'd love a review/approval on https://review.openstack.org/#/c/77048/ (adding a note to devtest.sh to say we test on fedora 20 and ubuntu trusty) if anyone has time | 03:43 |
| StevenK | I thought we were running on Precise | 03:45 |
| tchaypo | we spin up precise nodes in the overclould (and maybe the undercloud?) | 03:50 |
| tchaypo | but devtest.sh is only tested on trusty afaik | 03:50 |
| tchaypo | and F20, but not precise | 03:51 |
| tchaypo | I believe (but haven't checked myself) that precise has an old version of libvirt that lacks features we use | 03:51 |
| StevenK | It might be Saucy | 03:52 |
| greghaynes | Ive also been told its OVS support is lacking | 03:52 |
| greghaynes | lots of heresay about devtest and precise :p | 03:52 |
| tchaypo | but in any case, I've tried to make no claims about what *works*, only what *has CI testing* | 03:53 |
| greghaynes | ugh, for some reason now I get No valid host was found in nova when making undercloud when I devtest | 03:54 |
| StevenK | tchaypo: Trusty isn't released yet, which is why I'm questioning | 03:55 |
| giulivo | lifeless, this isn't working for me https://github.com/openstack/tripleo-image-elements/blob/master/elements/nova-baremetal/os-apply-config/mnt/state/var/lib/nova/virtual-power-key | 03:59 |
| giulivo | the keyfile is correctly populated but because of wrong permissions it can't be used, so virtual power fails | 03:59 |
| giulivo | ever faced it? ideas where the chmod/chown should be placed? | 04:00 |
| giulivo | lifeless, os-config-refresh configure.d ? | 04:01 |
| *** ccrouch1 has quit IRC | 04:05 | |
| *** ccrouch has joined #tripleo | 04:05 | |
| *** victor_lowther_ has quit IRC | 04:05 | |
| greghaynes | eek, getting "unrecognized partition table type\nNo partitions found\nWarning: given size (21504) exceeds max allowable size (21503)\n\nsfdisk: bad input\n'" in nova-baremetal-deploy-helper | 04:05 |
| *** ccrouch has quit IRC | 04:05 | |
| *** ccrouch has joined #tripleo | 04:05 | |
| lifeless | the slaves are precise | 04:06 |
| lifeless | the vm host is saucy I believe | 04:06 |
| *** victor_lowther_ has joined #tripleo | 04:06 | |
| lifeless | the build images are saucy (the default in dib) | 04:06 |
| lifeless | giulivo: we use paramiko, the permissions on the key shouldn't matter | 04:07 |
| * lifeless watches his disk subsystem max out | 04:10 | |
| lifeless | raid rebuildreshapes, FTW | 04:10 |
| *** CaptTofu has joined #tripleo | 04:10 | |
| StevenK | lifeless: I always tend to get worried when I see my fileservers disk light constantly on | 04:13 |
| StevenK | Usually it's softraid going "Time for a resync!" | 04:13 |
| *** CaptTofu has quit IRC | 04:15 | |
| *** killer_prince is now known as lazy_prince | 04:24 | |
| *** sdake has quit IRC | 04:27 | |
| lifeless | just kicked in the final 2 disks from my old server | 04:27 |
| lifeless | dm-0 0.00 0.00 0.40 2.40 1.60 7.20 6.29 6.76 1877.71 902.00 2040.33 357.14 100.00 | 04:27 |
| lifeless | asdg 0.00 10777.00 0.00 94.20 0.00 43481.00 923.16 2.96 31.40 0.00 31.40 4.19 39.44 | 04:28 |
| lifeless | sdh 0.00 10777.00 0.00 94.20 0.00 43481.00 923.16 3.37 35.78 0.00 35.78 4.37 41.20 | 04:28 |
| *** akuznetsov has joined #tripleo | 04:33 | |
| lifeless | [>....................] reshape = 2.5% (75434912/2929556480) finish=1144.5min speed=41559K/sec | 04:34 |
| * StevenK ponders the readable of "sed -e ':a;N;$!ba;s/\n/\\n/g' /root/.ssh/test_env_rsa" versus the python | 04:34 | |
| StevenK | lifeless: Sllloow | 04:35 |
| StevenK | lifeless: My resyncs tend to hit 116000K/sec | 04:35 |
| SpamapS | lifeless: we are now UTC-7 FYI | 04:35 |
| lifeless | StevenK: are you running devtest at the same time ? | 04:35 |
| lifeless | SpamapS: what time is it for you now ? | 04:35 |
| StevenK | Heh | 04:35 |
| StevenK | lifeless: Fair point :-) | 04:35 |
| SpamapS | lifeless: 21:35 | 04:35 |
| lifeless | StevenK: also, raid5 or 6 ? | 04:36 |
| StevenK | Neither, 1 | 04:36 |
| lifeless | StevenK: ahahahaha no comparison then | 04:36 |
| StevenK | I lost a fair bit of data years using softraid 5 when the kernel refused to tell me which drive failed | 04:36 |
| greghaynes | Played a bit of raid5 roulette, eh? | 04:37 |
| StevenK | greghaynes: I knew a drive had failed, since the resync stopped at 7 something %. | 04:37 |
| StevenK | The kernel didn't log which drive it was, and SMART wasn't clever enough to tell me | 04:38 |
| greghaynes | Yep, then you have a nonzero chance of yanking a random drive and it being a correct guess :p | 04:38 |
| lifeless | StevenK: yeah, but raid1 resync is linear reads looking for mismatch | 04:38 |
| lifeless | StevenK: this is raid6 reshape which is moving all the data down 30% on the drives | 04:39 |
| StevenK | lifeless: I ponder moving to RAID 5, since reshape is a thing now | 04:39 |
| lifeless | StevenK: so reading all previous members, writing to /all/ members | 04:39 |
| giulivo | StevenK, anyway, with raid1 you'll have 50% chances to pick the "non working" disk ! | 04:39 |
| StevenK | I've had a few drive failures with RAID 1 and it's always told me | 04:40 |
| StevenK | I was using RAID 5 with like kernel 2.2 | 04:40 |
| giulivo | with raid5 you had what, 66% | 04:40 |
| giulivo | StevenK, only joking about it | 04:40 |
| lifeless | StevenK: buy 4 3TB WD REDs and raid6 em | 04:40 |
| greghaynes | depends on num of drives | 04:40 |
| *** matsuhashi has joined #tripleo | 04:40 | |
| giulivo | greghaynes+1 :) | 04:40 |
| lifeless | StevenK: then even if you guess wrong you'll be ok :) | 04:40 |
| StevenK | Haha | 04:40 |
| StevenK | lifeless: I have 2 2TB and 2 3TB | 04:41 |
| StevenK | 4.5T on /srv | 04:41 |
| greghaynes | Ive just been doing raid0 ssd's for devtest box... | 04:41 |
| lifeless | these drives are reading 70MBps and writing 45MBps simultaneously | 04:41 |
| lifeless | I'm pretty happy with that | 04:41 |
| greghaynes | Is kind of nice actually | 04:41 |
| lifeless | for all that they're spinning rust | 04:41 |
| tchaypo | Use zfs, problem solved | 04:42 |
| greghaynes | lol | 04:42 |
| StevenK | tchaypo: No, then you have two problems | 04:42 |
| lifeless | tchaypo: which problem ? | 04:42 |
| greghaynes | "What could go wrong?" | 04:42 |
| StevenK | greghaynes: What could *possibly* go wrong! | 04:42 |
| StevenK | Just like "How hard can it be" | 04:43 |
| tchaypo | StevenK's problems with the linux kernel | 04:43 |
| StevenK | tchaypo: zfs didn't exist then | 04:43 |
| tchaypo | either he'll be using a BSD kernel or he'll be using the FUSE driver, but either way the linux kernel won't be making him sad | 04:43 |
| StevenK | Like I said, kernel *2.2* | 04:43 |
| tchaypo | and this is why last week was valuable | 04:44 |
| lifeless | tchaypo: because it let you get your trolling skills in order? | 04:44 |
| SpamapS | wtf do you all store on these disks? | 04:44 |
| cody-somerville | my thoughts *exactly* | 04:44 |
| tchaypo | if we hadn't gone to the other side of the world to spend a week together I want have such an accurate mental rendering of StevenK's voice saying "Like I said, kernel *2.2*" | 04:44 |
| lifeless | SpamapS: family DVD and blueray collection | 04:44 |
| SpamapS | oh | 04:45 |
| SpamapS | I have that too | 04:45 |
| giulivo | porns | 04:45 |
| StevenK | Haha | 04:45 |
| SpamapS | its -> [amazon|netflix|googleplay] | 04:45 |
| giulivo | you mean | 04:45 |
| giulivo | sorry guys, see you enjoy the disks | 04:45 |
| StevenK | SpamapS: Move to .au or .nz, and then say that. | 04:45 |
| SpamapS | Zero reason to "buy" content .. lots of reason to actually pay for it and enjoy it at a reasonable price. :) | 04:45 |
| StevenK | I dare you. | 04:45 |
| lifeless | SpamapS: not available, not available, stupidly expensive | 04:45 |
| greghaynes | Fair point | 04:45 |
| SpamapS | So the first two, you all should definitely move away from such a backward country... | 04:46 |
| SpamapS | and what is stupidly expensive? | 04:46 |
| lifeless | SpamapS: seriously, 30NZD for a recent release; buying the DVD / blueray is cheaper | 04:46 |
| lifeless | SpamapS: google play | 04:46 |
| SpamapS | because I just wait for movies to go on special and buy them for $8 or so. | 04:46 |
| *** giulivo has quit IRC | 04:46 | |
| SpamapS | I've found Play to be the more reasonable one actually. | 04:46 |
| lifeless | SpamapS: perhaps in the states | 04:46 |
| SpamapS | Netflix has almost nothing good except for kids shows. Amazon's prices are meh. Play's prices fluctuate madly. | 04:47 |
| SpamapS | lifeless: do you have robber barons there that are willing to pay these prices? It seems counter-intuitive to price oneself out of the market. :-/ | 04:47 |
| clarkb | SpamapS: netflix and amazon have tons of good stuff | 04:47 |
| clarkb | and I get amazon for freeish since I pay for prime for other reasons | 04:47 |
| SpamapS | clarkb: they occasionally have good stuff | 04:47 |
| SpamapS | and we took the plunge and got Amazon Fresh... | 04:47 |
| greghaynes | I do like now nowdays netflix works better than amazon for me in linux | 04:48 |
| clarkb | SpamapS: amazon fresh isn't subscription though | 04:48 |
| clarkb | you just order and stuff arrives the next day | 04:48 |
| SpamapS | in fact I'm ordering ingredients to make goulash tomorrow right now from Fresh. | 04:48 |
| StevenK | tchaypo: So the RAID 5 disaster was ~ 2000, and zfs appeared in 2005 | 04:48 |
| SpamapS | clarkb: it most certainly _IS_ a subscription here in LA. | 04:48 |
| lifeless | SpamapS: indeed | 04:48 |
| StevenK | Mmmm, dinner. I should think about that. | 04:48 |
| lifeless | SpamapS: google play is strictly more expensive than dvd or blueray here | 04:48 |
| clarkb | SpamapS: huh it isn't in seattle | 04:48 |
| SpamapS | clarkb: maybe it is cheaper in Seattle. In LA, it is an upgrade from Prime. | 04:48 |
| clarkb | SpamapS: you literally load up a cart and pick a delivery time | 04:48 |
| *** akuznetsov has quit IRC | 04:49 | |
| StevenK | Takes me a few days to remember that I need to plan meals after coming home from a sprint | 04:49 |
| lifeless | SpamapS: and since we pay for total traffic (vs bandwidth) play is less attractive to consumers except if you can cache it | 04:49 |
| lifeless | SpamapS: which you can't | 04:49 |
| clarkb | SpamapS: pro tip you can order a bunch of normal amazon stuff via amazon fresh whcih means scheduled next day delivery for all the things :) | 04:49 |
| SpamapS | clarkb: $299 for the year. | 04:49 |
| SpamapS | but it includes Prime. | 04:49 |
| clarkb | woah | 04:49 |
| SpamapS | and they're _wicked_ fast. | 04:49 |
| SpamapS | clarkb: It enrages me a bit to hear that you got it for free. | 04:50 |
| clarkb | SpamapS: well I have to pay for delivery | 04:50 |
| clarkb | but I am not required to pay a subscription | 04:50 |
| *** akuznetsov has joined #tripleo | 04:50 | |
| StevenK | SpamapS: So, you're pointing and laughing about movies. Can we do the same about say, health care? :-P | 04:51 |
| SpamapS | But I did the math.. and I use it twice a week.. it saves me about an hour a week. $230 (prime already pays for itself other ways) is well worth 52 hours of my time and wife's time. | 04:51 |
| SpamapS | clarkb: OH.. delivery is free for me. | 04:51 |
| SpamapS | StevenK: let's not | 04:51 |
| StevenK | Hah | 04:51 |
| SpamapS | I'm not laughing actually | 04:51 |
| SpamapS | I was seriously WTF'ing | 04:51 |
| lifeless | StevenK: thats cruel :P | 04:51 |
| SpamapS | just weird to me to think about having any kind of server at home. | 04:51 |
| clarkb | SpamapS: ah so thats not too terrible | 04:51 |
| SpamapS | clarkb: they're A/B'ing us ;) | 04:52 |
| lifeless | SpamapS: its inconcievable to me not to | 04:52 |
| * mordred has no servers at home | 04:52 | |
| * mordred has like 10 cloud accoutns | 04:52 | |
| StevenK | mordred: Lies. | 04:52 |
| lifeless | morganfainberg: other than the drizzle build farm :P | 04:52 |
| lifeless | bah | 04:52 |
| lifeless | mordred: ^ | 04:53 |
| mordred | lifeless: that hasnt' been running since openstack started :) | 04:53 |
| lifeless | see, openstack is too hard to install | 04:53 |
| StevenK | mordred: They may not be powered on, but they're still in your house. | 04:53 |
| lifeless | StevenK: technically they aren't servers if they aren't serving. | 04:53 |
| lifeless | StevenK: OTOH ipads and phones and laptops are servers much of the time :) | 04:54 |
| StevenK | Heh | 04:54 |
| lifeless | https://jenkins07.openstack.org/job/check-tripleo-overcloud-precise/176/console :< | 04:55 |
| SpamapS | So really what's going on is that you need somebody in .nz and/or .au to stand up something like M-GO so Google will have to actually compete. | 04:55 |
| *** akuznetsov has quit IRC | 04:55 | |
| lifeless | that would help | 04:56 |
| mordred | StevenK: they are, in fact, Mac Pro Towers running that would run Windows and OSX if powered on - so I'd say they're barely computers | 04:56 |
| StevenK | Haha | 04:56 |
| lifeless | but honestly, my server is about 1/10th the cost of cloud storage | 04:56 |
| lifeless | and I get compute | 04:56 |
| StevenK | SpamapS: Bandwidth | 04:56 |
| lifeless | offsite backup with a second instance at a friends | 04:57 |
| mordred | lifeless: my cloud accounts are all free | 04:57 |
| lifeless | mordred: for your personal stuff? | 04:57 |
| StevenK | DSL speeds in much of .au and .nz are still pathetic | 04:57 |
| mordred | my cloud operations are uch cheaper than local things :) | 04:57 |
| clarkb | lifeless: my personal cloud account is free | 04:57 |
| clarkb | I opened the cloud | 04:57 |
| mordred | lifeless: yeah man. I opened the cloud - rackspace likes to gift stuff | 04:57 |
| mordred | :) | 04:57 |
| lifeless | mordred: huh; well for you special snowflakes | 04:57 |
| lifeless | but for those of us with 7TB of data to backup | 04:58 |
| StevenK | But then you have to *get* the data, which could take a day | 04:58 |
| lifeless | oh, and don't forget the steam liobrary | 05:00 |
| lifeless | thats a big chunk on its own | 05:00 |
| StevenK | Yeah | 05:00 |
| lifeless | 7GB on S3 is 550USDpm | 05:00 |
| StevenK | I could store my data in Amazon for $210US/month, not counting the charges to actually push more stuff and download existing | 05:00 |
| lifeless | local storage pays for itself in < 1yr | 05:01 |
| lifeless | 440 on RRS | 05:01 |
| lifeless | with no data transfer | 05:01 |
| StevenK | lifeless: Are you counting bandwidth charges (yours, not Amazon or Rackspace) in your calculations? | 05:01 |
| lifeless | 71USDpm i glacier | 05:01 |
| lifeless | StevenK: no, just base storage | 05:02 |
| StevenK | lifeless: And then 4 days notice to actually get anything out | 05:02 |
| lifeless | go https://jenkins07.openstack.org/job/check-tripleo-overcloud-precise/176/console go | 05:06 |
| SpamapS | lifeless: you don't seem to be valuing your time very highly. :p | 05:09 |
| tchaypo | pypi is really making me sad - I keep getting random errors. | 05:10 |
| tchaypo | I may have to stop thinking that greghaynes is over-the-top for running a local mirror | 05:10 |
| SpamapS | tchaypo: -> pypi.openstack.org and/or local pypi-mirror run | 05:10 |
| lifeless | SpamapS: thats the save-1-hour-on-test-runs-test | 05:10 |
| SpamapS | I started a local mirror run earlier today because I want to start playing w/ wheels. :p | 05:10 |
| lifeless | SpamapS: pypi.o.o is terrible on the internet | 05:10 |
| lifeless | SpamapS: you need to stop thinking like an american :) | 05:11 |
| SpamapS | lifeless: it is wonderful through my squid though | 05:11 |
| StevenK | tchaypo: I have a local Ubuntu mirror, is that over-the-top ? | 05:11 |
| lifeless | tchaypo: bandersnatch can mirror all of pypi for you, minus a few bad things | 05:11 |
| tchaypo | i see there's a pypi-openstack element | 05:11 |
| lifeless | tchaypo: yeah, it hardcodes pypi.o.o in | 05:11 |
| tchaypo | I don't really want to mirror all of pypi | 05:12 |
| SpamapS | tchaypo: As lifeless says, pypi.openstack.org will not necessarily improve things for you in .au and .nz .. | 05:12 |
| StevenK | tchaypo: It's only 70G | 05:12 |
| SpamapS | THough just forcing http does improve things if you have a squid. :) | 05:12 |
| tchaypo | just the packages I use | 05:12 |
| StevenK | Oh, that's right. You don't have real computers. | 05:12 |
| SpamapS | I'm running the OpenStack pypi-mirror .. not bandersnatch.. | 05:12 |
| tchaypo | yeah, but if squid checks upstream for if-modified-since and gets a 502, you're out of luck | 05:12 |
| tchaypo | 70G is ~3/16ths of my monthly quota. | 05:13 |
| StevenK | tchaypo: My monthly quota is 60G | 05:14 |
| tchaypo | ouch | 05:14 |
| StevenK | However, it's not quite that bad, since that's only peak (noon-midnight) | 05:14 |
| StevenK | Off peak is unmetered | 05:15 |
| *** akuznetsov has joined #tripleo | 05:15 | |
| StevenK | Ah ha. There is a hotel on my amex | 05:20 |
| StevenK | If that's made it across, I know what I'm doing for the rest of the day | 05:20 |
| SpamapS | StevenK: let me know if you can get eem working with Firefox 27 | 05:21 |
| SpamapS | it has been reporting an OCSP cert problem for me the last 3 weeks | 05:21 |
| SpamapS | (and chromium has never worked) | 05:21 |
| tchaypo | I still have nothing. | 05:22 |
| tchaypo | but plan to file the expense claim just as soon as i get my laptop running properly-encrypted linux | 05:23 |
| StevenK | SpamapS: I can't see the toolbar | 05:23 |
| StevenK | But random clicking around works | 05:23 |
| *** akuznetsov has quit IRC | 05:24 | |
| StevenK | SpamapS: The cert for the header was changed on the 3rd | 05:26 |
| StevenK | And EEM only has two transactions from the amex, so more waiting | 05:26 |
| * StevenK looks for tchaypo's instructions | 05:26 | |
| *** akuznetsov has joined #tripleo | 05:28 | |
| tchaypo | bandersnatch is on the move... i'll let you know how long 70Gb takes to download once I found out | 05:30 |
| *** akuznetsov has quit IRC | 05:30 | |
| SpamapS | StevenK: if you pop up the javascript console you can click the javascript links that paint the toolbar and accept the security exceptions | 05:31 |
| SpamapS | lifeless: 2014-03-11 04:39:54.268 | Waiting for the overcloud stack to be ready | 05:31 |
| SpamapS | 2014-03-11 05:21:58.994 | Build timed out (after 145 minutes). Marking the build as failed. :( | 05:31 |
| StevenK | SpamapS: I'm trying to get firefox to trust HP's CA | 05:31 |
| SpamapS | StevenK: you have HP's ca in /etc/openvpn | 05:31 |
| StevenK | Yes | 05:32 |
| SpamapS | StevenK: trust your feelings.. and your corporate cert | 05:32 |
| StevenK | Trying to work out how to pour into Firefox | 05:32 |
| mikal | tchaypo: what pypi problem are you seeing? I'm hoping it mirrors mine. | 05:32 |
| StevenK | But also trying to see details about it, which means I'm trying to swap back in how the hell to drive openssl | 05:33 |
| SpamapS | # du -hs /opt/stack/venvs/os-*-config/ | 05:33 |
| SpamapS | 15M/opt/stack/venvs/os-apply-config/ | 05:33 |
| SpamapS | 47M/opt/stack/venvs/os-collect-config/ | 05:33 |
| SpamapS | 13M/opt/stack/venvs/os-refresh-config/ | 05:33 |
| SpamapS | One of these things, is nooot like ottherrr | 05:33 |
| tchaypo | StevenK: once you find the cert file, look in settings for the button that lets you manage certificates and then import it | 05:34 |
| SpamapS | babel.. 11M | 05:34 |
| SpamapS | we should consider sharing things like babel among all virtualenvs | 05:34 |
| tchaypo | mikal: i haven't pinned it down - the last one was a 502 trying to download netaddr, but earlier I think I got a 401 for something | 05:35 |
| StevenK | SpamapS: So, yes, EEM works for me with Firefox 27 | 05:35 |
| SpamapS | 152MB of duplicate babel/localedata on our controller image | 05:35 |
| mikal | tchaypo: I've been seeing "connection reset by peer" all day | 05:35 |
| SpamapS | StevenK: thx.. will try the windows solution (reboot!) | 05:35 |
| mikal | tchaypo: I think its the SYD CDN node being broken | 05:35 |
| clarkb | StevenK: I couldn't get it to work in sacuy firefox but chromium was fine | 05:36 |
| mikal | tchaypo: hard coding /etc/hosts to a node in the US seems to have helped | 05:36 |
| tchaypo | mikal: my 502 (proxy error) must have come from squid - mayhap that represents an upstream reset-by-peer | 05:36 |
| clarkb | no idea why but gave up when I found the chromium workaround | 05:36 |
| StevenK | clarkb: Haha. And SpamapS says that chromium has never worked | 05:36 |
| StevenK | clarkb: Works fine for me in current saucy | 05:36 |
| tchaypo | mikal: that would make sense. bandersnatch seems to be chugging away. Having a local mirror probably makes sense anyway. | 05:37 |
| mikal | Tell me more about bandersnatch | 05:38 |
| mikal | I've never used it | 05:38 |
| clarkb | its does a proper mirror of pypi | 05:38 |
| tchaypo | neither had I unto 5 minutes ago | 05:38 |
| mordred | right. but the problem is | 05:38 |
| mikal | All of pypi? | 05:38 |
| clarkb | so you get external linkage breakage and no special wheels | 05:38 |
| mordred | it's a PROPER mirror | 05:38 |
| tchaypo | https://pypi.python.org/pypi/bandersnatch | 05:38 |
| mordred | so it's poop | 05:38 |
| StevenK | mordred: How so? | 05:38 |
| mordred | because it doesn't spider external things | 05:38 |
| SpamapS | one man's poop, is another man's web app | 05:38 |
| clarkb | StevenK: or build platform specific wheels | 05:39 |
| mordred | someone should check out devpi and see if it's good now | 05:39 |
| mordred | last time I checked it out it was almost good | 05:39 |
| StevenK | Mind you, I haven't looked at a local pypi mirror. I was schooling people that local Ubuntu mirrors aren't 0.5T at LCA | 05:40 |
| clarkb | StevenK: the size isn't an issue | 05:40 |
| clarkb | StevenK: the problem is it misses important bits | 05:40 |
| clarkb | the bits that fail most often | 05:40 |
| mikal | So, we've been using the dodgy pypi mirror thingie from the tripleo people | 05:40 |
| mikal | But it broke this morning when our connectivity to pypi.python.org became unreliable | 05:40 |
| mikal | And took turbo hipster with it | 05:41 |
| clarkb | mikal: did pypi.openstack.org not work? | 05:41 |
| tchaypo | so I hear I shouldn't be bothering with bandersnatch... | 05:41 |
| mikal | clarkb: I'd have to double check with jhesketh, but we saw our nodes will poking upstream as well | 05:41 |
| tchaypo | mikal: can you clarify what you mean by "dodgy pypi mirror thingie"? is that the pip-cache element? | 05:41 |
| clarkb | tchaypo: it will probably help the cross pacific problem | 05:41 |
| mikal | clarkb: probably because we test against stable releases as well | 05:41 |
| clarkb | tchaypo: but when external links disappear it won't | 05:41 |
| clarkb | mikal: upstream? | 05:42 |
| mikal | tchaypo: https://git.openstack.org/cgit/openstack-infra/pypi-mirror | 05:42 |
| mikal | clarkb: pypi.python.org | 05:42 |
| mikal | tchaypo: dodgy because nothing that poorly documented can be good | 05:42 |
| tchaypo | I don't think I have a cross-pacific problem - ping times to pypi.python.org are not consistent with trans-pacific traversal | 05:42 |
| StevenK | pypi is on a CDN | 05:43 |
| mikal | tchaypo: yeah | 05:43 |
| mikal | tchaypo: and I have lost faith in the local SYD CDN endpoint | 05:43 |
| StevenK | pypi.python.org is an alias for global.prod.fastly.net. | 05:43 |
| clarkb | mikal: it does fit a particular niche because we would like bandersnatch to not suck | 05:43 |
| mikal | clarkb: the mirror thing? I think I'm mostly tainted because lifeless insisted the documentation was in the tripleo image builder code, and I was jetlagged | 05:44 |
| clarkb | mikal: yeah | 05:44 |
| mikal | clarkb: some documentation wouldn't be terrible | 05:44 |
| mikal | clarkb: do you guys see unicode decode errors when you run it? | 05:45 |
| mikal | clarkb: that's the problem I currently have | 05:45 |
| clarkb | I don't think so | 05:45 |
| mikal | Command python setup.py egg_info failed with error code 1 in /tmp/tmpmmbWtg/venv/build/sphinxcontrib-docbookrestapi | 05:45 |
| clarkb | its easy to check jenkins /me looks | 05:45 |
| tchaypo | clarkb: do we have a standard yaml config for the pypi-mirror? | 05:45 |
| mikal | Traceback (most recent call last): | 05:45 |
| mikal | File "/tmp/tmpmmbWtg/venv/bin/pip", line 11, in <module> | 05:45 |
| mikal | sys.exit(main()) | 05:46 |
| mikal | File "/tmp/tmpmmbWtg/venv/local/lib/python2.7/site-packages/pip/__init__.py", line 185, in main | 05:46 |
| mikal | return command.main(cmd_args) | 05:46 |
| mikal | File "/tmp/tmpmmbWtg/venv/local/lib/python2.7/site-packages/pip/basecommand.py", line 161, in main | 05:46 |
| mikal | text = '\n'.join(complete_log) | 05:46 |
| mikal | UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 62: ordinal not in range(128) | 05:46 |
| clarkb | tchaypo: yes, ts in openstack-infra/config somewhere | 05:46 |
| StevenK | I think we're going to find the entire Internet in openstack-infra/config somewhere soon | 05:46 |
| mikal | tchaypo: I use http://paste.openstack.org/show/73105/ | 05:47 |
| *** akuznetsov has joined #tripleo | 05:47 | |
| StevenK | Since everything else is in there | 05:47 |
| clarkb | mikal: looks happy https://jenkins.openstack.org/job/periodic-mirror-python27/215/consoleFull | 05:47 |
| mikal | clarkb: that makes me wonder what we've done wrong | 05:47 |
| lifeless | mordred: you can layer pypi-mirror for wheels on top of bandersnatch, and the external issue thing is slowly going away | 05:47 |
| clarkb | mikal: we use older pip though | 05:47 |
| mikal | tchaypo: ./modules/openstack_project/files/pypi-mirror.yaml seems like a winner | 05:47 |
| mikal | clarkb: what version of pip are you using? | 05:48 |
| clarkb | lifeless: we also use the mirror for enforcement or something | 05:48 |
| clarkb | but I suppose if bandersnatch was only behind the scnees that would work | 05:48 |
| lifeless | clarkb: refuted several times | 05:48 |
| clarkb | lifeless: o? | 05:48 |
| clarkb | I tried refuting it many times and lost each time :) | 05:49 |
| lifeless | clarkb: check infra logs for phrases like 'and this is why the mirror isn't an enforcement for version constriants' | 05:49 |
| mordred | clarkb: I think we shoud stop using the mirror for enforcement | 05:49 |
| mordred | clarkb: which I believe was your original point | 05:49 |
| clarkb | mordred: it was/is | 05:49 |
| mordred | clarkb: I think now that we've got requirements repo doing that for us | 05:49 |
| mordred | the mirror should be about solving network glitches | 05:49 |
| mordred | the amount of fail that we deal with in the current design is immense | 05:50 |
| tchaypo | https://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/pypi/README.md is most helpful also | 05:50 |
| mordred | lifeless: the mirror IS currently a second layer of enforcement - but it shoudl stop beign so | 05:50 |
| lifeless | SpamapS: so yeah, that build fail sucks | 05:50 |
| openstackgerrit | A change was merged to openstack/tripleo-heat-templates: Add rabbit.cookie attribute https://review.openstack.org/70238 | 05:51 |
| * tchaypo realises why he's tired - it's 21:51 Sunnyvale non-DST | 05:52 | |
| StevenK | I got roughly 11 hours sleep, so I'm okayish | 05:52 |
| StevenK | I'm expecting to hit a wall at about 9:30pm | 05:52 |
| lifeless | SpamapS: 2014-03-11 00:13:41.770 3267 ERROR nova.api.metadata.handler [-] Failed to get metadata for ip: 192.0.2.2 | 05:53 |
| lifeless | 2014-03-11 00:13:41.829 3267 ERROR nova.api.metadata.handler [-] Failed to get metadata for ip: 192.0.2.5 | 05:53 |
| lifeless | 2014-03-11 00:14:11.021 3267 WARNING nova.network.neutronv2 [req-4ed301ee-6f85-467b-bdc6-247aaa634ee9 None None] Using neutron_admin_tenant_name for authentication is deprecated and will be removed in the next release. Use neutron_admin_tenant_id instead. | 05:53 |
| lifeless | 2014-03-11 00:14:11.214 3267 ERROR nova.api.metadata.handler [-] Failed to get metadata for ip: 192.0.2.8 | 05:53 |
| *** akuznetsov has quit IRC | 05:53 | |
| *** rpodolyaka has joined #tripleo | 05:54 | |
| lifeless | jogo: so - this failed to get metadata for X | 05:55 |
| lifeless | jogo: you're saying that when there is an exception trace, we hide it by default ? | 05:55 |
| tchaypo | I hit the wall badly about 9 last night | 05:55 |
| lifeless | ok, C-> bed then some r&r before my 2200 call with the UK | 05:56 |
| lifeless | I'm going to file a bug on this flaky test thing though | 05:56 |
| lifeless | its batshit | 05:56 |
| lifeless | also | 05:56 |
| lifeless | reminder | 05:56 |
| lifeless | DO RRIAGE | 05:56 |
| lifeless | TRIAGE | 05:56 |
| *** CaptTofu has joined #tripleo | 05:58 | |
| *** CaptTofu has quit IRC | 06:02 | |
| lifeless | StevenK: https://bugs.launchpad.net/tripleo/+bug/1290483 might be interesting to you once you finish the nodes arc. | 06:03 |
| *** matsuhashi has quit IRC | 06:05 | |
| *** akuznetsov has joined #tripleo | 06:07 | |
| *** matsuhas_ has joined #tripleo | 06:08 | |
| *** akuznetsov has quit IRC | 06:08 | |
| *** akuznetsov has joined #tripleo | 06:09 | |
| *** akuznetsov has quit IRC | 06:10 | |
| greghaynes | so, has anyone experienced nova bm barfing with ERROR: sector 0 does not have an msdos signature for the undercloud iscsi? | 06:15 |
| lifeless | pretty sure sfdisk outputs that always | 06:17 |
| lifeless | greghaynes: what log dod you see that in ? | 06:17 |
| greghaynes | hrm. Heres /nova-baremetal-deploy-helper.log | 06:19 |
| lifeless | jogo: https://bugs.launchpad.net/tripleo/+bug/1290701 | 06:19 |
| lifeless | greghaynes: ok, I may be wrong :) | 06:19 |
| greghaynes | syslog also shows iscsi detected connection error, not sure how relevant though | 06:19 |
| greghaynes | aaah, forgot link: Heres /nova-baremetal-deploy-helper.log https://gist.githubusercontent.com/anonymous/9480493/raw/467dea61459a80c516e7e3886652ec88215cd12d/gistfile1.txt | 06:21 |
| lifeless | greghaynes: this is your problem | 06:22 |
| lifeless | sfdisk: bad input\n' | 06:22 |
| greghaynes | :/ | 06:22 |
| lifeless | greghaynes: the rest is a distraction | 06:22 |
| greghaynes | ok | 06:22 |
| lifeless | Warning: given size (21504) exceeds max allowable size (21503) | 06:22 |
| lifeless | may be implicated | 06:22 |
| lifeless | greghaynes: does your flavor allocate more disk space than you have ? | 06:23 |
| greghaynes | Shouldnt, shows 21gb for flavor, 211gb avail on disk | 06:24 |
| lifeless | greghaynes: paste nova flavor-list ? just for my satisfaction :) | 06:24 |
| greghaynes | http://paste.ubuntu.com/7071915/ | 06:25 |
| greghaynes | 21. useful units are useful | 06:25 |
| lifeless | greghaynes: and the 211G is the qcow2 VM disk size, or free space on your host ? | 06:26 |
| greghaynes | free space on host | 06:26 |
| lifeless | need the VM block device size :) | 06:27 |
| lifeless | qemu-img info or some such | 06:27 |
| *** vkozhukalov has joined #tripleo | 06:27 | |
| greghaynes | virtual size for qemu being created shows 4.3G | 06:28 |
| lifeless | 20G >> 4.3G | 06:29 |
| greghaynes | indeed | 06:29 |
| greghaynes | hrm | 06:29 |
| lifeless | but try | 06:30 |
| lifeless | sudo qemu-img info /var/lib/libvirt/images/baremetal_0.qcow2 | 06:30 |
| lifeless | for instance | 06:30 |
| greghaynes | ah, virtual size: 21G (22548578304 bytes) | 06:31 |
| lifeless | so 21 and 21 | 06:31 |
| lifeless | should fit | 06:31 |
| lifeless | you might try, just for kicks, a 20G flavor definition instead | 06:31 |
| greghaynes | yes, although that size discrepency was pretty narrow | 06:31 |
| lifeless | one (unit) | 06:32 |
| lifeless | but note that we consume one 'track' for partition table | 06:32 |
| lifeless | so its entirely possible.... | 06:32 |
| lifeless | ok, bbiaw | 06:33 |
| greghaynes | ah, fun | 06:33 |
| *** killer_prince has joined #tripleo | 06:40 | |
| *** rdopieralski has joined #tripleo | 06:40 | |
| *** rdopieralski has quit IRC | 06:41 | |
| *** rdopieralski has joined #tripleo | 06:41 | |
| *** Qiming has quit IRC | 06:44 | |
| openstackgerrit | A change was merged to openstack/tuskar-ui: Error out on too few nodes https://review.openstack.org/77942 | 06:59 |
| *** rpodolyaka has quit IRC | 07:01 | |
| *** rpodolyaka has joined #tripleo | 07:01 | |
| *** rpodolyaka has quit IRC | 07:01 | |
| *** rpodolyaka has joined #tripleo | 07:01 | |
| *** rpodolyaka has quit IRC | 07:02 | |
| *** rpodolyaka has joined #tripleo | 07:07 | |
| *** rsacharya has joined #tripleo | 07:14 | |
| GheRivero | greghaynes: https://review.openstack.org/#/c/79344/ | 07:16 |
| GheRivero | morning TripleO | 07:16 |
| *** rpodolyaka has quit IRC | 07:17 | |
| tchaypo | morning GheRivero | 07:18 |
| *** mrunge has joined #tripleo | 07:18 | |
| *** mrunge has quit IRC | 07:18 | |
| *** mrunge has joined #tripleo | 07:18 | |
| openstackgerrit | A change was merged to openstack/tuskar-ui: Fix task states of nova-baremetal nodes https://review.openstack.org/78688 | 07:18 |
| *** ramishra has joined #tripleo | 07:28 | |
| openstackgerrit | Rohan Kanade proposed a change to openstack/tuskar-ui: Fix task state of nova instance https://review.openstack.org/78685 | 07:39 |
| *** CaptTofu has joined #tripleo | 07:46 | |
| *** vkozhukalov has quit IRC | 07:50 | |
| *** CaptTofu has quit IRC | 07:51 | |
| *** rcarrillocruz1 has joined #tripleo | 07:57 | |
| *** rcarrillocruz has quit IRC | 07:59 | |
| *** akrivoka has joined #tripleo | 08:00 | |
| *** ifarkas has quit IRC | 08:04 | |
| *** pblaho has joined #tripleo | 08:05 | |
| *** bauzas has joined #tripleo | 08:07 | |
| *** rcarrillocruz has joined #tripleo | 08:07 | |
| *** rcarrillocruz1 has quit IRC | 08:08 | |
| *** e0ne has joined #tripleo | 08:10 | |
| *** jprovazn has joined #tripleo | 08:16 | |
| *** ramishra has quit IRC | 08:16 | |
| *** ramishra has joined #tripleo | 08:16 | |
| *** ramishra has quit IRC | 08:21 | |
| *** jcoufal has joined #tripleo | 08:24 | |
| *** giulivo has joined #tripleo | 08:25 | |
| openstackgerrit | stephen-mulcahy proposed a change to openstack/tripleo-incubator: Add basic timing instrumentation to devtest.sh https://review.openstack.org/77564 | 08:25 |
| *** morganfainberg is now known as morganfainberg_Z | 08:27 | |
| *** jistr has joined #tripleo | 08:33 | |
| *** ifarkas has joined #tripleo | 08:34 | |
| *** rlandy has joined #tripleo | 08:41 | |
| *** bogdando has quit IRC | 08:42 | |
| openstackgerrit | Ana Krivokapic proposed a change to openstack/tuskar-ui: Show reason of failure in case of deployment error https://review.openstack.org/78412 | 08:43 |
| *** rcarrillocruz1 has joined #tripleo | 08:46 | |
| *** rcarrillocruz has quit IRC | 08:46 | |
| *** andreaf has joined #tripleo | 08:48 | |
| *** petertoft has joined #tripleo | 08:50 | |
| *** adrian17od has joined #tripleo | 08:51 | |
| *** akrivoka has quit IRC | 08:55 | |
| *** akuznetsov has joined #tripleo | 08:58 | |
| *** jcoufal has quit IRC | 09:00 | |
| *** jcoufal_ has joined #tripleo | 09:00 | |
| *** derekh has joined #tripleo | 09:01 | |
| *** jcoufal_ has quit IRC | 09:02 | |
| *** rsacharya has quit IRC | 09:02 | |
| rlandy | SpamapS: ping | 09:03 |
| lifeless | greghaynes: https://bugs.launchpad.net/tripleo/+bug/1290399 | 09:04 |
| lifeless | greghaynes: same symptoms | 09:04 |
| GheRivero | lifelesss: https://review.openstack.org/#/c/79344/ | 09:04 |
| SpamapS | rlandy: pong (but fading fast ;) | 09:05 |
| lifeless | GheRivero: check-tripleo-overcloud-precise FAILURE in 2h 25m 16s | 09:05 |
| rlandy | SpamapS: I'll be quick :) ... | 09:05 |
| *** pbelanyi has joined #tripleo | 09:05 | |
| rlandy | SpamapS: I posted a question yesterday (before I faded out) about heat stack registering that the overcloud had completed before a cinder node had finished building | 09:06 |
| *** martyntaylor has joined #tripleo | 09:06 | |
| *** rsacharya has joined #tripleo | 09:06 | |
| *** killer_prince has quit IRC | 09:07 | |
| rlandy | SpamapS: what qualifies as a 'completed overcloud'? | 09:07 |
| *** jrist has quit IRC | 09:07 | |
| openstackgerrit | Jan Provaznik proposed a change to openstack/diskimage-builder: Adds "provides" support to element dependencies https://review.openstack.org/79421 | 09:07 |
| *** matsuhas_ has quit IRC | 09:08 | |
| SpamapS | rlandy: the status of the stack would be 'CREATE_COMPLETE' | 09:08 |
| rlandy | SpamapS: http://paste.openstack.org/show/73083/ | 09:08 |
| *** hashar has joined #tripleo | 09:09 | |
| SpamapS | rlandy: right and I asked you for 'heat event-list overcloud' | 09:09 |
| rlandy | copying ... sec | 09:09 |
| *** matsuhashi has joined #tripleo | 09:09 | |
| *** martyntaylor has quit IRC | 09:10 | |
| rlandy | SpamapS: http://paste.openstack.org/show/73118/ | 09:10 |
| *** tomh_ has joined #tripleo | 09:10 | |
| *** akrivoka has joined #tripleo | 09:12 | |
| SpamapS | | BlockStorage0 | aeee778c-543c-4b95-8214-b11c3ff586c5 | state changed | CREATE_COMPLETE | 2014-03-10T22:09:15Z | | 09:12 |
| SpamapS | rlandy: that is likely a bug in Heat | 09:12 |
| SpamapS | and with that, I pass out | 09:13 |
| rlandy | SpamapS: thanks - will log that | 09:13 |
| *** jcoufal has joined #tripleo | 09:15 | |
| lifeless | SpamapS: https://bugs.launchpad.net/tripleo/+bug/1290759 | 09:19 |
| Ng | +1 more logging | 09:19 |
| Ng | MOAR | 09:19 |
| lifeless | night all | 09:19 |
| GheRivero | night | 09:19 |
| StevenK | Ng: Such log | 09:20 |
| lifeless | much debug | 09:21 |
| *** jrist has joined #tripleo | 09:21 | |
| StevenK | Ng: Ursula is a bad person, and shared http://dogeweather.com/ on FB | 09:21 |
| Ng | hah | 09:22 |
| *** lucasagomes has joined #tripleo | 09:24 | |
| *** martyntaylor has joined #tripleo | 09:28 | |
| *** ramishra has joined #tripleo | 09:33 | |
| *** CaptTofu has joined #tripleo | 09:35 | |
| *** athomas has joined #tripleo | 09:35 | |
| *** ramishra has quit IRC | 09:37 | |
| *** CaptTofu has quit IRC | 09:40 | |
| *** jp_at_hp has joined #tripleo | 09:49 | |
| *** martyntaylor has quit IRC | 09:49 | |
| *** akuznetsov has quit IRC | 09:50 | |
| *** martyntaylor has joined #tripleo | 09:53 | |
| *** martyntaylor has quit IRC | 09:56 | |
| *** martyntaylor has joined #tripleo | 09:57 | |
| *** rsacharya has quit IRC | 10:05 | |
| *** martyntaylor has quit IRC | 10:05 | |
| *** saju_m has joined #tripleo | 10:05 | |
| *** killer_prince has joined #tripleo | 10:07 | |
| *** saju_m has quit IRC | 10:18 | |
| *** rcarrillocruz has joined #tripleo | 10:18 | |
| *** martyntaylor has joined #tripleo | 10:19 | |
| *** rcarrillocruz1 has quit IRC | 10:20 | |
| *** matsuhashi has quit IRC | 10:29 | |
| *** rcarrillocruz1 has joined #tripleo | 10:30 | |
| *** saju_m has joined #tripleo | 10:30 | |
| *** rcarrillocruz has quit IRC | 10:31 | |
| *** andreaf has quit IRC | 10:35 | |
| *** markmc has joined #tripleo | 10:48 | |
| openstackgerrit | A change was merged to openstack/tuskar-ui: Fix task state of nova instance https://review.openstack.org/78685 | 10:53 |
| *** saju_m has quit IRC | 10:54 | |
| *** CaptTofu has joined #tripleo | 11:05 | |
| *** hashar has quit IRC | 11:17 | |
| *** rcarrillocruz has joined #tripleo | 11:17 | |
| *** hashar has joined #tripleo | 11:19 | |
| *** rcarrillocruz1 has quit IRC | 11:20 | |
| *** rcarrillocruz1 has joined #tripleo | 11:28 | |
| *** rcarrillocruz has quit IRC | 11:29 | |
| *** morazi has joined #tripleo | 11:30 | |
| *** rcarrillocruz has joined #tripleo | 11:32 | |
| *** andreaf has joined #tripleo | 11:35 | |
| *** rcarrillocruz1 has quit IRC | 11:35 | |
| *** rdopieralski has quit IRC | 11:37 | |
| *** jtomasek has quit IRC | 11:46 | |
| *** hashar has quit IRC | 11:57 | |
| *** jrist has quit IRC | 11:58 | |
| *** jtomasek has joined #tripleo | 12:00 | |
| *** weshay has joined #tripleo | 12:03 | |
| *** crimi has joined #tripleo | 12:03 | |
| *** killer_prince has quit IRC | 12:05 | |
| jp_at_hp | is there a bug for a generic tripleo CI failure? saying recheck no bug seems to be avoiding the fact that the ci wasn't working? Or is that the implied meaning of no bug? | 12:08 |
| Ng | jp_at_hp: I would tend to think that any failure of the CI environment to self-assemble, is a bug somewhere. I'm not sure that we necessarily want a new bug in lp for each failure, but we should be checking over failures to see what was going wrong | 12:15 |
| jp_at_hp | Ng: yes - that's what I would have thought. It makes sense... | 12:18 |
| jp_at_hp | So to that end - I have a test failing because of permissions writing into the seed image it is trying to create. | 12:19 |
| jp_at_hp | 2014-03-11 12:03:30.517 | cp: cannot create regular file `/tmp/image.KcBeq5Bk/mnt/etc/dib-git-manifest-seed': Permission denied | 12:19 |
| slagle | jp_at_hp: b/c you need to use sudo there | 12:20 |
| jp_at_hp | I see other writes to that dir happening, and my testing didn't have a problem :( Although I was testing as root. I guess from this I can assume that the upstream ci run is as a non-root user? But with passwordless sudo? | 12:20 |
| slagle | jp_at_hp: was just looking at your review :). scripts under extra-data.d do no run as root in the chroot | 12:20 |
| jp_at_hp | So is the bug in my change, or in the running of the extra-data.d scripts? | 12:21 |
| slagle | jp_at_hp: in your change. | 12:21 |
| jp_at_hp | What is the case for not running any of those scripts as root, when the sole purpose is to create the image as desired? | 12:21 |
| slagle | just add a sudo before that cp at line 187 | 12:21 |
| jp_at_hp | slagle: yes, I know that will fix it. But what is the reason for not running the extra-data.d scripts as root? | 12:22 |
| slagle | jp_at_hp: some hook scripts run chrooted into the image as root, some don't. have a look at the README | 12:23 |
| slagle | jp_at_hp: it's so that you can manipulate the image to do things *before* chrooting into it | 12:23 |
| *** rbrady-afk is now known as rbrady | 12:23 | |
| slagle | like copy stuff in from the build host, as you're doing here | 12:23 |
| slagle | originally, i *believe* dib had the sudo happening at a higher level, so that an element author didn't have to use sudo. but, that was changed | 12:24 |
| jp_at_hp | I'll change my patch, no probs, but getting insight into *why* thingsd are as they are would be super-helpful. | 12:24 |
| jp_at_hp | This commit seems to be related: 357c2de681c698915ea062de72cd1863067c080b | 12:25 |
| slagle | well, i would guess it's because dib tries to not use sudo/root where possible | 12:26 |
| slagle | so instead of running every extra-data.d with sudo by default, where it may not be needed, it's left up to the element author to determine when it's needed | 12:26 |
| slagle | and for the hook scripts that run chrooted into the image, you don't have to use sudo | 12:27 |
| slagle | i admit it's confusing, i often forget to use sudo in extra-data.d | 12:27 |
| jp_at_hp | slagle: I can appreciate that, just not sure it makes complete sense to me. Any package management system operating outside of a users home directory requires execution as root. It seems that requiring same for diskimage-builder is not inreasonable, and is just a further tightening of the run requirements, as was the case for the last commit affecting sudo-related things that I posted above. | 12:28 |
| slagle | jp_at_hp: sure, though, i would not like running dib as root. even though it uses sudo everywhere as-is | 12:29 |
| jp_at_hp | slagle: what - you don't trust it? | 12:30 |
| jp_at_hp | I suppose there is a danger of a riogue element affecting your local system, but if you need passwordless sudo anyways... | 12:30 |
| slagle | not explicitly | 12:30 |
| *** lucasagomes is now known as lucas-hungry | 12:31 | |
| *** jdob has joined #tripleo | 12:31 | |
| *** e0ne_ has joined #tripleo | 12:39 | |
| openstackgerrit | Jon-Paul Sullivan proposed a change to openstack/diskimage-builder: Create a git manifest from source-repositories https://review.openstack.org/75829 | 12:39 |
| *** e0ne has quit IRC | 12:41 | |
| *** hewbrocca has joined #tripleo | 12:44 | |
| *** nosnos has quit IRC | 12:44 | |
| *** CaptTofu has quit IRC | 12:44 | |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Additional swift storage service mappings https://review.openstack.org/76753 | 12:48 |
| giulivo | guys I restarted my seed and didn't 192.0.2.1 set on the eth1 interface, if I set it manually things get back working but I can't find the element setting it up ... that's because init-neutron-ovs isn't called again after initial configuration, right? | 12:51 |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Replace more then just "-" in REPONAME https://review.openstack.org/76679 | 12:51 |
| *** rcarrillocruz1 has joined #tripleo | 12:53 | |
| *** rcarrillocruz has quit IRC | 12:54 | |
| *** lazy_prince is now known as killer_prince | 12:56 | |
| *** Qiming has joined #tripleo | 13:05 | |
| *** dprince has joined #tripleo | 13:05 | |
| *** Qiming has quit IRC | 13:06 | |
| *** CaptTofu has joined #tripleo | 13:07 | |
| *** lblanchard has joined #tripleo | 13:08 | |
| *** akuznetsov has joined #tripleo | 13:09 | |
| *** CaptTofu has quit IRC | 13:11 | |
| openstackgerrit | Dirk Mueller proposed a change to openstack/tripleo-image-elements: remove keystone-signing dir creation https://review.openstack.org/74989 | 13:13 |
| jprovazn | greghaynes: hi, are you around? | 13:13 |
| openstackgerrit | Jon-Paul Sullivan proposed a change to openstack/diskimage-builder: Fixup all occurrences of REPONAME for replacing '-' https://review.openstack.org/79602 | 13:14 |
| *** hashar has joined #tripleo | 13:14 | |
| openstackgerrit | Dirk Mueller proposed a change to openstack/tripleo-image-elements: Excplicitly install dhcp-agent https://review.openstack.org/74991 | 13:14 |
| *** tzumainn has joined #tripleo | 13:17 | |
| openstackgerrit | Dirk Mueller proposed a change to openstack/tripleo-image-elements: Restrict deletion of packaged files to Fedora like packages https://review.openstack.org/74990 | 13:17 |
| *** julim has joined #tripleo | 13:18 | |
| *** jrist has joined #tripleo | 13:26 | |
| *** rdopieralski has joined #tripleo | 13:31 | |
| *** jprovazn has quit IRC | 13:34 | |
| *** jprovazn has joined #tripleo | 13:37 | |
| *** lucas-hungry is now known as lucasagomes | 13:38 | |
| *** bogdando has joined #tripleo | 13:42 | |
| *** ftcjeff has joined #tripleo | 13:48 | |
| *** spzala has joined #tripleo | 13:49 | |
| *** jcoufal has quit IRC | 13:50 | |
| *** crimi has quit IRC | 13:52 | |
| openstackgerrit | A change was merged to openstack/tuskar-ui: Show reason of failure in case of deployment error https://review.openstack.org/78412 | 13:53 |
| *** Matt2 has joined #tripleo | 13:56 | |
| *** akuznetsov has quit IRC | 13:58 | |
| *** akuznetsov has joined #tripleo | 13:59 | |
| openstackgerrit | Petr Blaho proposed a change to openstack/tuskar: Updates CONTRIBUTING docs sort bug in fedora 20 https://review.openstack.org/79609 | 14:08 |
| *** pblaho has quit IRC | 14:08 | |
| *** ramishra has joined #tripleo | 14:24 | |
| *** matty_dubs|gone is now known as matty_dubs | 14:28 | |
| GheRivero | Is anyone using the pypi-mirror element? The released version is failing due to the external pypi references (fixed upstream) | 14:31 |
| *** mrunge has quit IRC | 14:31 | |
| rbrady | bnemec: https://review.openstack.org/#/c/77283/ | 14:34 |
| bnemec | rbrady: Thanks | 14:37 |
| rbrady | bnemec: np | 14:38 |
| slagle | derekh: https://review.openstack.org/#/c/79344/ | 14:39 |
| derekh | slagle: thanks | 14:39 |
| *** jcoufal has joined #tripleo | 14:40 | |
| *** rpodolyaka has joined #tripleo | 14:40 | |
| *** jprovazn has quit IRC | 14:40 | |
| *** rcarrillocruz1 has quit IRC | 14:42 | |
| *** jprovazn has joined #tripleo | 14:43 | |
| *** rcarrillocruz has joined #tripleo | 14:43 | |
| *** jrist has quit IRC | 14:43 | |
| killer_prince | please review https://review.openstack.org/#/c/79286/ | 14:44 |
| openstackgerrit | A change was merged to openstack/tuskar: Pin iso8601 logging to WARN https://review.openstack.org/79516 | 14:45 |
| *** blamar has joined #tripleo | 15:02 | |
| *** sparkycollier has joined #tripleo | 15:03 | |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-image-elements: List filesystem and swift and known glance stores https://review.openstack.org/79631 | 15:06 |
| *** jcoufal has quit IRC | 15:06 | |
| *** jcoufal has joined #tripleo | 15:07 | |
| derekh | ^^ probably the reason ci overcloud jobs are currently failing (because we don't have enough problems) | 15:07 |
| *** rlandy has quit IRC | 15:07 | |
| *** fungi has quit IRC | 15:07 | |
| *** rlandy has joined #tripleo | 15:08 | |
| *** jistr has quit IRC | 15:11 | |
| *** jcoufal has quit IRC | 15:12 | |
| *** jcoufal has joined #tripleo | 15:13 | |
| *** rcarrillocruz1 has joined #tripleo | 15:19 | |
| jprovazn | I'm getting "Image storage media is full: There is not enough disk space on the image storage media." when loading undercloud image into seed's glance - seed VM is created with incorrect storage (4G instead of 20G defined in NODE_DIST) | 15:20 |
| jprovazn | is this known issue ^? | 15:20 |
| *** rcarrillocruz has quit IRC | 15:21 | |
| jdob | jprovazn: i think I saw a similar problem when changing NODE_DIST, but I assumed it was something I screwed up | 15:22 |
| jdob | er, DISK | 15:23 |
| jdob | (I was looking at your comment when I typed it and got mentally screwed up) | 15:23 |
| jprovazn | jdob: :) ah, sry | 15:23 |
| jprovazn | jdob: thanks, will give it one more try | 15:24 |
| jdob | nah, its my fault. need more coffee. | 15:24 |
| *** jcoufal has quit IRC | 15:25 | |
| openstackgerrit | Jon-Paul Sullivan proposed a change to openstack/diskimage-builder: Require that root runs image manipulation scripts https://review.openstack.org/79636 | 15:26 |
| jprovazn | greghaynes: hi, ping me when you are here, I'd like to discuss one percona issue | 15:27 |
| openstackgerrit | Jon-Paul Sullivan proposed a change to openstack/diskimage-builder: Require that root runs image manipulation scripts https://review.openstack.org/79636 | 15:27 |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-image-elements: List filesystem and swift and known glance stores https://review.openstack.org/79631 | 15:29 |
| *** sparkycollier has quit IRC | 15:30 | |
| *** jistr has joined #tripleo | 15:35 | |
| *** jistr is now known as jistr|afk | 15:36 | |
| derekh | dprince: backing up a copy of the seed host's /root so I can update some repo's to build a new testenv image | 15:39 |
| derekh | dprince: backup is in /var/lib/misc/backups/20140311_1537/ | 15:39 |
| *** rpodolyaka has quit IRC | 15:40 | |
| dprince | derekh: okay. Is there something that needs rebasing? | 15:40 |
| dprince | derekh: that is fine for now... but I would rather keep us in one tree so as to avoid confusion. | 15:41 |
| * dprince runs off for lunch | 15:41 | |
| derekh | dprince: ok, I'll move into a subdir | 15:41 |
| *** noslzzp has quit IRC | 15:42 | |
| *** jcoufal-mobile has joined #tripleo | 15:45 | |
| *** fungi has joined #tripleo | 15:46 | |
| *** noslzzp has joined #tripleo | 15:46 | |
| *** rdopieralski has quit IRC | 15:46 | |
| *** ramishra has quit IRC | 15:48 | |
| *** ramishra has joined #tripleo | 15:48 | |
| killer_prince | i am trying to put common code used to find appropriate kernel to be used for booting instances (in vm and baremetal element and disk-image-get-kernel) but i can see that some scripts are executed using chroot and some are executed without chroot. is there a place where i can put the code and will be accessible by all scripts..? | 15:53 |
| *** bogdando has quit IRC | 15:55 | |
| *** adrian17od has left #tripleo | 15:57 | |
| *** ramishra has quit IRC | 15:57 | |
| *** rpodolyaka has joined #tripleo | 15:57 | |
| *** jprovazn has quit IRC | 15:58 | |
| *** mrunge has joined #tripleo | 15:59 | |
| *** jcoufal has joined #tripleo | 16:00 | |
| openstackgerrit | Ryan Brady proposed a change to openstack/tripleo-heat-templates: Merge Block Storage into Overcloud https://review.openstack.org/79654 | 16:04 |
| *** cwolferh_ has joined #tripleo | 16:04 | |
| *** hewbrocc` has joined #tripleo | 16:08 | |
| rpodolyaka | morning tripleo | 16:09 |
| *** hewbrocca has quit IRC | 16:09 | |
| *** bogdando has joined #tripleo | 16:09 | |
| *** ifarkas has quit IRC | 16:11 | |
| *** rpodolyaka has quit IRC | 16:15 | |
| *** vkozhukalov has joined #tripleo | 16:16 | |
| *** rpodolyaka has joined #tripleo | 16:17 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: Install jq on Test Environment hosts https://review.openstack.org/79663 | 16:19 |
| *** jrist has joined #tripleo | 16:19 | |
| *** matty_dubs is now known as matty_dubs|lunch | 16:22 | |
| *** cadenzajon has joined #tripleo | 16:24 | |
| *** rpodolyaka has quit IRC | 16:27 | |
| *** e0ne has joined #tripleo | 16:27 | |
| *** e0ne_ has quit IRC | 16:30 | |
| *** mandeep has joined #tripleo | 16:31 | |
| *** mandeep has left #tripleo | 16:31 | |
| *** mandeep has joined #tripleo | 16:33 | |
| *** jcoufal-mobile has quit IRC | 16:36 | |
| *** jistr|afk is now known as jistr | 16:36 | |
| *** jcoufal-mobile has joined #tripleo | 16:36 | |
| *** jcoufal-mobile has quit IRC | 16:37 | |
| *** rcarrillocruz has joined #tripleo | 16:38 | |
| *** rcarrillocruz1 has quit IRC | 16:38 | |
| *** rpodolyaka has joined #tripleo | 16:40 | |
| openstackgerrit | Jonathan Brownell proposed a change to openstack/diskimage-builder: Fix "(None)" hostname; provide default to cloud-init https://review.openstack.org/77301 | 16:41 |
| *** killer_prince is now known as lazy_prince | 16:42 | |
| *** CaptTofu has joined #tripleo | 16:43 | |
| *** CaptTofu has quit IRC | 16:43 | |
| *** mandeep_ has joined #tripleo | 16:44 | |
| *** CaptTofu has joined #tripleo | 16:45 | |
| *** mandeep has quit IRC | 16:45 | |
| *** jang1 has joined #tripleo | 16:47 | |
| *** mandeep_ has quit IRC | 16:50 | |
| SpamapS | 'morning | 16:55 |
| rpodolyaka | o/ | 16:56 |
| SpamapS | derekh: perhaps you know this.. have we considered building a local on-disk pypi mirror for the test envs? | 16:56 |
| derekh | SpamapS: for the repositories being tested ? yes https://review.openstack.org/#/c/71837/ | 16:57 |
| derekh | SpamapS: I was hoping to get feedback on it befor proceeding too far down that path | 16:58 |
| *** ifarkas has joined #tripleo | 16:58 | |
| *** rsacharya has joined #tripleo | 16:59 | |
| SpamapS | derekh: so I have a once-per-day-updated mirror on my laptop now and it cut about 20 minutes off my usual run time, even though I have a warm squid and 100Mbit internet.... | 17:00 |
| *** mandeep has joined #tripleo | 17:00 | |
| SpamapS | derekh: seems like we should just have one pypi-mirror that we tar up and push out onto the testenvs. | 17:00 |
| derekh | SpamapS: ahh ok, thats a different case, yes its on the list https://etherpad.openstack.org/p/tripleo-parallel-test | 17:01 |
| SpamapS | derekh: IMO, we shouldn't bother fixing pypi-openstack to work with pypi. Our goal should be to use --offline anyway. | 17:01 |
| *** jcoufal has quit IRC | 17:04 | |
| derekh | SpamapS: makes sense to me, although I think the tarball should be kept update as much as possible | 17:04 |
| *** mandeep has quit IRC | 17:06 | |
| *** mandeep_ has joined #tripleo | 17:06 | |
| *** mandeep_ has quit IRC | 17:10 | |
| SpamapS | derekh: yeah hourly would even make sense. | 17:10 |
| *** e0ne has quit IRC | 17:11 | |
| *** mandeep has joined #tripleo | 17:11 | |
| jogo | lifeless: if CONF.verbose or CONF.debug: | 17:11 |
| jogo | extra['exc_info'] = (exc_type, value, tb) | 17:11 |
| *** sdake has joined #tripleo | 17:11 | |
| *** d0ugal has quit IRC | 17:13 | |
| *** rpodolyaka has quit IRC | 17:16 | |
| *** mandeep has quit IRC | 17:20 | |
| jogo | but that is not it we have a exception and error message that are identical | 17:22 |
| *** rpodolyaka has joined #tripleo | 17:22 | |
| *** mandeep has joined #tripleo | 17:22 | |
| *** matty_dubs|lunch is now known as matty_dubs | 17:24 | |
| *** martyntaylor has quit IRC | 17:25 | |
| *** pbelanyi has quit IRC | 17:26 | |
| *** mandeep has quit IRC | 17:27 | |
| derekh | ci-overcloud is busted, I think will be fixed by this https://review.openstack.org/#/c/79631 but I can't know for sure because we no longer grab enough logs (will fix) | 17:28 |
| derekh | the error that caused that jobs to fail was a PXE deploy error, probably fixed by this https://review.openstack.org/#/c/79502/ | 17:29 |
| *** mrunge has quit IRC | 17:29 | |
| *** mandeep has joined #tripleo | 17:29 | |
| derekh | thats a lot of I thinks and probables but its all I got :-( | 17:30 |
| *** marun has quit IRC | 17:30 | |
| *** marun has joined #tripleo | 17:31 | |
| *** markmc has quit IRC | 17:31 | |
| SpamapS | hmmm... | 17:34 |
| SpamapS | we should be watching for UpgradeImpact commits as a team | 17:34 |
| openstackgerrit | Jenkins proposed a change to openstack/os-collect-config: Updated from global requirements https://review.openstack.org/79690 | 17:35 |
| SpamapS | derekh: damnit | 17:37 |
| *** rupsky has joined #tripleo | 17:38 | |
| *** bauzas has quit IRC | 17:38 | |
| *** rpodolyaka has quit IRC | 17:38 | |
| *** rcarrillocruz1 has joined #tripleo | 17:39 | |
| derekh | damnit indeed | 17:39 |
| *** rcarrillocruz has quit IRC | 17:39 | |
| *** rcarrillocruz has joined #tripleo | 17:41 | |
| *** rupsky has quit IRC | 17:41 | |
| SpamapS | derekh: I'll +A as soon as CI passes. Don't need two +2's | 17:41 |
| *** rupsky has joined #tripleo | 17:41 | |
| *** SpamapS changes topic to "FIREDRILL: CI failing https://review.openstack.org/#/c/79631/ | tripleo-cd running preserve-ephemeral WIP patches and https://review.openstack.org/#/c/62042/ | Using OpenStack to deploy OpenStack;meetings Tuesday 1900 UTC in #openstack-meeting-alt" | 17:42 | |
| derekh | SpamapS: The ci on the swift change is going to fail, and will need to be restarted | 17:42 |
| SpamapS | derekh: right, but the glance fix will pass right? | 17:42 |
| derekh | SpamapS: opps, I meant glance, I took a look at the instance and the heat stack-create has already failed | 17:43 |
| *** rcarrillocruz1 has quit IRC | 17:43 | |
| derekh | SpamapS: because of a PXE deploy failure | 17:44 |
| derekh | SpamapS: I hav't restarted the job becasue I thought we shoudl wait to get the logs | 17:44 |
| derekh | *havn't | 17:44 |
| *** rsacharya has quit IRC | 17:44 | |
| *** athomas has quit IRC | 17:45 | |
| SpamapS | https://bugs.launchpad.net/glance/+bug/1290969 | 17:45 |
| SpamapS | derekh: ah gotchya | 17:45 |
| SpamapS | derekh: let's get a bug ref on it | 17:46 |
| derekh | k, willdo | 17:46 |
| *** petertoft has quit IRC | 17:46 | |
| SpamapS | derekh: https://bugs.launchpad.net/tripleo/+bug/1290969 | 17:46 |
| SpamapS | derekh: filed already | 17:46 |
| SpamapS | and submitting revert to glance | 17:46 |
| derekh | Actually no point in waiting as we no longer get the undercloud logs anyways | 17:47 |
| *** rollerj has joined #tripleo | 17:47 | |
| *** hashar has quit IRC | 17:48 | |
| *** rpodolyaka has joined #tripleo | 17:49 | |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-image-elements: List filesystem and swift as known glance stores https://review.openstack.org/79631 | 17:49 |
| *** rupsky has quit IRC | 17:49 | |
| *** mandeep has quit IRC | 17:50 | |
| SpamapS | derekh: _closes_ bug | 17:50 |
| SpamapS | derekh: So the bot will close tripleo's task on the bug. | 17:51 |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-image-elements: List filesystem and swift as known glance stores https://review.openstack.org/79631 | 17:51 |
| derekh | SpamapS: ahh I was trying to prevent it closing the glace part, didn't know it could be selective | 17:52 |
| *** rupsky has joined #tripleo | 17:53 | |
| SpamapS | derekh: yes it is entirely selective.. one of my favorite parts of our launchpad integration actually | 17:54 |
| *** sgrasley1 has joined #tripleo | 17:57 | |
| derekh | gotta run, fingers crossed that job passes.... | 17:59 |
| SpamapS | derekh: ok, I'll babysit it | 17:59 |
| derekh | lifeless: looked at the new update to json for testenv-workers today, comments added https://review.openstack.org/#/c/79631/ testing a new testenv at the moment with that change | 18:00 |
| *** derekh has quit IRC | 18:00 | |
| *** mrunge has joined #tripleo | 18:00 | |
| *** mandeep has joined #tripleo | 18:01 | |
| *** mandeep has quit IRC | 18:04 | |
| jdob | meeting today? | 18:06 |
| slagle | i assume there will be at the correct time :) | 18:07 |
| slagle | ...DST | 18:07 |
| jdob | OH | 18:07 |
| slagle | i was wonder myself for a moment :) | 18:07 |
| greghaynes | Yep, DST messing up all the things | 18:07 |
| jdob | shit, ok, thanks | 18:07 |
| jrist | haha | 18:08 |
| *** bauzas has joined #tripleo | 18:14 | |
| *** andreaf has quit IRC | 18:17 | |
| SpamapS | I kind of wish I could get Unity's clock to show me Local + UTC at all times | 18:19 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Neutron: don't hard code the venv dir https://review.openstack.org/76626 | 18:23 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Update elements to use os-svc-daemon -i. https://review.openstack.org/76963 | 18:23 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Update elements to use os-svc-install -i. https://review.openstack.org/76964 | 18:23 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Wire in OPENSTACK_VENV_DIR default if set https://review.openstack.org/76965 | 18:23 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Wire in _EXTRA_INSTALL_OPTS... https://review.openstack.org/76966 | 18:23 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Add a new common-venv element https://review.openstack.org/76967 | 18:23 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Glance/Cinder/Keystone/Swift: don't hard code venv https://review.openstack.org/76666 | 18:23 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Stop using the os-svc-install -n -c options. https://review.openstack.org/76604 | 18:23 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Drop the os-svc-install -n -c options. https://review.openstack.org/76605 | 18:23 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Link db sync utilities to /usr/local/bin... https://review.openstack.org/76579 | 18:23 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Heat/Horizon/Tempest: don't hard code venv dir https://review.openstack.org/76888 | 18:23 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Ceilo/Ironic/Tuskar: don't hard code venv dir https://review.openstack.org/76887 | 18:23 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Nova: don't hard code the venv dir https://review.openstack.org/76619 | 18:23 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Add -i options to os-svc-install/os-svc-daemon... https://review.openstack.org/76952 | 18:23 |
| greghaynes | wow | 18:24 |
| *** rpodolyaka has quit IRC | 18:24 | |
| SpamapS | much commit | 18:26 |
| SpamapS | so patch stream | 18:26 |
| SpamapS | dprince: what's it like out there in the weeds? ;-) | 18:26 |
| dprince | SpamapS: quite a nice day actually. I may have to leave and go for a ride | 18:26 |
| *** rpodolyaka has joined #tripleo | 18:27 | |
| SpamapS | dprince: You will have to recheck bug 1290969 all those changes unfortunately | 18:27 |
| dprince | SpamapS: meh. I'll do it later | 18:28 |
| *** rcarrillocruz1 has joined #tripleo | 18:28 | |
| dprince | SpamapS: thanks for the heads up though | 18:28 |
| *** rcarrillocruz has quit IRC | 18:29 | |
| SpamapS | dprince: yeah we're still about 1.5hrs away from the fix commit landing anyway :-P | 18:31 |
| *** cwolferh has joined #tripleo | 18:32 | |
| *** cwolferh_ has quit IRC | 18:32 | |
| dprince | SpamapS: well. if you guys would just land all my code we could be (with a single venv) only an hour away :) | 18:32 |
| dprince | SpamapS: maybe less than that? | 18:32 |
| *** mrunge has quit IRC | 18:32 | |
| dprince | SpamapS: no rush though, I can wait on CI | 18:33 |
| SpamapS | dprince: I have an idea for a compromise on the single venv idea. | 18:34 |
| SpamapS | dprince: it involves using hard links though.. so might be scary for some. ;) | 18:35 |
| dprince | SpamapS: Well... so long as it is optional we could support both. I like that the single venv behaves more like real packages anyway. | 18:36 |
| dprince | SpamapS: in any case like I mentioned before 99 percent of my changes (everything except that last commit for the common-venv element) is just plain good coding practice i.e. DRY | 18:37 |
| *** akrivoka has quit IRC | 18:37 | |
| SpamapS | dprince: do you not see venvs as "real" ? | 18:39 |
| SpamapS | Or just, not real enough for redhat customers? ;-) | 18:40 |
| dprince | SpamapS: I suppose I don't. :) | 18:40 |
| SpamapS | anyway, didn't mean to be confrontational.. :) | 18:41 |
| dprince | SpamapS: They have some utility... but are quite hard to nail down for reproducability | 18:41 |
| SpamapS | dprince: so the single venv thing is an optimization that loses some features.. I'm thinking we can get most of that optimization without losing the isolation feature. | 18:41 |
| dprince | SpamapS: never know what the cat is going to drag into one of them... | 18:41 |
| SpamapS | dprince: well you do know, but sometimes you don't realize that you authorized the cat to drag in crap. ;) | 18:42 |
| *** jprovazn has joined #tripleo | 18:43 | |
| SpamapS | dprince: my idea is simply to comb through all the requirements.txt's before pip installing them, and build a single venv that is a union of the common requirements, and then link (sym or hard, not sure) lib/python2.7/* from there, the same way system-site-packages symlinks, into the new venvs... | 18:43 |
| SpamapS | that will get a huge space savings (babel == 11MB * 20 venvs on controller) and will avoid double-building things. | 18:44 |
| dprince | SpamapS: I'm not against it but it is conceptually more complex than my solution (and could break things). I think the right solution is to make all these things options and then users can choose. | 18:44 |
| SpamapS | boo | 18:44 |
| SpamapS | options just slow users down | 18:44 |
| dprince | SpamapS: not sure how it could break things but it is something more to maintain and I could see us having to fix it | 18:45 |
| SpamapS | If we really don't want the isolation feature anymore, lets just drop it. | 18:45 |
| SpamapS | I don't see any value in having an explosion of options. | 18:45 |
| *** akuznetsov has quit IRC | 18:45 | |
| dprince | SpamapS: well, to be fair I didn't add any new options. Just a new element that sets a simple ENV variable to override the default venv dir. | 18:45 |
| dprince | SpamapS: Although I suppose os-svc-daemon got an extra option. But then I removed two unused ones as well so I think we made out for the better | 18:46 |
| SpamapS | net loss of code == win | 18:46 |
| SpamapS | dprince: I think rather than having an option for something you think sucks, and an option for something I think sucks, we should just pick something that sucks the least. :) | 18:46 |
| SpamapS | I'm really o-k with one venv, as long as we all agree per-project venv isolation is too expensive. | 18:47 |
| *** jprovazn has quit IRC | 18:47 | |
| SpamapS | Because if it is too expensive in CI, it is too expensive in CD environments. | 18:48 |
| dprince | SpamapS: I don't think your solution sucks. It is on the clever side though and perhaps could lead to unintended consequences if we mess w/ hard linked venvs too much after the fact | 18:48 |
| SpamapS | yeah I don't like clever things either | 18:48 |
| *** bauzas has quit IRC | 18:48 | |
| SpamapS | I'm hoping to distill it down to something simpler. | 18:48 |
| SpamapS | Realistically the mechanics are already in virtualenv. | 18:49 |
| dprince | SpamapS: new topic. ensure-bridge. https://review.openstack.org/#/c/69918/. Based on your comment if I spin up an Ubuntu seed VM and test it I'll see your error? | 18:49 |
| SpamapS | I just need it to work like --system-site-packages, but linking to another virtualenv instead of the system's site packages. | 18:49 |
| SpamapS | dprince: yes I would expect that. | 18:50 |
| SpamapS | dprince: I did not dig into the actual files to see what the problem is specifically | 18:51 |
| *** rpodolyaka has quit IRC | 18:51 | |
| *** rpodolyaka has joined #tripleo | 18:52 | |
| dprince | SpamapS: I'll give it a go. Haven't tried Ubuntu yet but Fedora works quite nicely w/ this patch | 18:52 |
| SpamapS | dprince: If you give me a simple test I'm happy to test on Ubuntu as wel | 18:54 |
| SpamapS | reminder: meeting in 6 minutes | 18:54 |
| dprince | SpamapS: the test is does it work! | 18:55 |
| dprince | SpamapS: I need to push another iteration anyway so let me see what I can do | 18:55 |
| SpamapS | dprince: ah, so in theory CI will provide enough information that we can debug? | 18:57 |
| dprince | SpamapS: for this one, yes | 18:58 |
| dprince | SpamapS: there are some finer points though. Previously after ensure bridge ran you would have (I think) a dhclient on both the physical NIC and the OVS bridge. Which isn't what you want. With this you should just end up w/ a dhclient on the bridge alone. | 18:59 |
| dprince | SpamapS: at least that is what I see on Fedora | 18:59 |
| SpamapS | dprince: there are a few things that I feel like we should do that are CI specific.. like asserting that there is only a dhclient on the bridge. | 19:00 |
| dprince | SpamapS: plus it is nice to test a reboot and see if it works after this refactoring (it should) | 19:00 |
| *** lblanchard has quit IRC | 19:00 | |
| SpamapS | -> meeting | 19:00 |
| *** sgrasley1 has left #tripleo | 19:02 | |
| *** rcarrillocruz has joined #tripleo | 19:04 | |
| *** rcarrillocruz1 has quit IRC | 19:06 | |
| *** sballe_ has quit IRC | 19:08 | |
| *** petertoft has joined #tripleo | 19:11 | |
| *** lazy_prince has quit IRC | 19:14 | |
| *** ganetin has joined #tripleo | 19:15 | |
| *** lazy_prince has joined #tripleo | 19:23 | |
| *** lazy_prince is now known as killer_prince | 19:23 | |
| *** ganetin has quit IRC | 19:24 | |
| rbrady | looking for reviews of cinder element backend changes: https://review.openstack.org/#/c/74563/ | 19:24 |
| *** pblaho has joined #tripleo | 19:26 | |
| *** SpamapS has quit IRC | 19:26 | |
| *** SpamapS_ has joined #tripleo | 19:29 | |
| *** rcarrillocruz1 has joined #tripleo | 19:30 | |
| *** SpamapS_ is now known as SpamapS | 19:30 | |
| *** jp_at_hp has quit IRC | 19:30 | |
| *** e0ne has joined #tripleo | 19:31 | |
| *** rcarrillocruz has quit IRC | 19:31 | |
| *** ifarkas has quit IRC | 19:33 | |
| * tchaypo grumbles about smartbuy giving a 404 | 19:34 | |
| lifeless | stupidbuy ? | 19:35 |
| lifeless | ;p | 19:36 |
| *** bauzas has joined #tripleo | 19:36 | |
| tchaypo | so I'd like to start poaching some bugs and smacking them | 19:42 |
| *** voskmaxx has joined #tripleo | 19:43 | |
| tchaypo | but before i start smacking bugs I've been trrying to get a stable test environment so that i don't waste time wondering if the breakage i see is related to the change I jus tmade or not | 19:43 |
| tchaypo | but on my devtest box under my desk, every run produces strange and unusual breakages. | 19:44 |
| giulivo | tchaypo, maybe one of the breakages is that the undercloud doesn't boot as in the seed neutron complains about "no dhcp agents" ? | 19:45 |
| SpamapS | tchaypo: are you cherry picking the already known needed fixes? | 19:45 |
| tchaypo | I may have seen that one last night | 19:45 |
| *** dprince has quit IRC | 19:46 | |
| tchaypo | SpamapS: we have known needed fixes? | 19:46 |
| SpamapS | tchaypo: see topic | 19:46 |
| *** voskmaxx has quit IRC | 19:46 | |
| SpamapS | tchaypo: any time there are known devtest breakers you can expect to see them there (if not, add them. ;) | 19:46 |
| tchaypo | one of the known needed fixes is that I need to buy myself some peripherals to make this hp laptop useable | 19:46 |
| tchaypo | starting with a microsoft keyboard | 19:47 |
| *** e0ne has quit IRC | 19:47 | |
| *** morganfainberg_Z is now known as morganfainberg | 19:48 | |
| tchaypo | ah, you mean the FIREDRILL bit? | 19:49 |
| *** rcarrillocruz has joined #tripleo | 19:49 | |
| tchaypo | I've been ignoring that because i thought it was saying that was a critical bug that needed work done, and I wanted to get a comfortable working environment before I start looking at them. | 19:49 |
| tchaypo | but there's a bit of chicken-and-egg there | 19:49 |
| *** rcarrillocruz1 has quit IRC | 19:50 | |
| SpamapS | tchaypo: that is a patch that, without, you will surely not finish devtest. | 19:51 |
| tchaypo | i believe i figured out yesterday that if I cherry-pick that into ~/.cache/tripleo/tripleo-image-elments, devtest will leave that repo alone when it's doing its update, right? | 19:52 |
| * tchaypo feels like such a noob | 19:53 | |
| SpamapS | tchaypo: correct | 19:53 |
| SpamapS | tchaypo: yes, you noob, you shoul understand all 17 ways we have to get a patch inserted into our 2 hour developer setup process. _SHEESH_ | 19:54 |
| *** petertoft has quit IRC | 19:54 | |
| tchaypo | I alredy know about the DIB_REPO* variables thanks to writing the README.md on derekh's patch, so i tried to use them the other day to make it use a patch for DIB itself | 19:56 |
| tchaypo | much facepalm ensued when I realised the problem with that | 19:57 |
| *** d0ugal has joined #tripleo | 19:58 | |
| *** d0ugal has quit IRC | 19:58 | |
| *** d0ugal has joined #tripleo | 19:58 | |
| *** petertoft has joined #tripleo | 19:59 | |
| *** markmc has joined #tripleo | 19:59 | |
| tchaypo | my other question is about what i need to do to progress https://review.openstack.org/#/c/77048/ | 20:01 |
| *** pblaho has quit IRC | 20:01 | |
| *** jistr has quit IRC | 20:02 | |
| tchaypo | firstly - can i just ignore jenkins? the check that failed doesn't seem to be gating (if I understand the term correctly) so it's got a +1 from jenkins anyway. Sicne this is just a doc change, and given the state of CI, it feels like it's okay to ignore that and charge on | 20:02 |
| tchaypo | but secondly - I'm going to need a +2 in order to make it land | 20:03 |
| devananda | lifeless: woops, ignore the ping in -meeting. meant to ping you here and wave there, ... | 20:04 |
| devananda | lifeless: do you want to continue? or resume after meetings? | 20:04 |
| SpamapS | has anyone else seen the keystone auth fail thing? | 20:04 |
| SpamapS | we have an out of date keystone-paste.ini which I think may be causing it | 20:05 |
| bnemec | SpamapS: I think I saw something similar yesterday, but I was messing around with some cherry-picked patches locally so I wasn't sure if it was my fault. | 20:05 |
| bnemec | And I'm not sure it was exactly the same failure. | 20:06 |
| lifeless | devananda: after meetings | 20:06 |
| lifeless | slagle: SpamapS: thinking we should fast track https://review.openstack.org/#/c/79502/ | 20:06 |
| lifeless | since it may fix the couldn't-get-metadata-for-IP issue, and simultaneously unbreak overcloud jobs | 20:06 |
| *** rpodolyaka has quit IRC | 20:07 | |
| lifeless | it failed due to glance | 20:07 |
| SpamapS | lifeless: you got my +2 | 20:07 |
| SpamapS | lifeless: do you think it might be a reason for the glance job's failure? | 20:07 |
| SpamapS | because I see a commit that landed changing keystone-paste.ini .. | 20:07 |
| SpamapS | related to tokens/revocation.. | 20:07 |
| openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack/tripleo-image-elements: Use upstream/package keystone-paste.ini. https://review.openstack.org/79750 | 20:08 |
| SpamapS | these paste.ini files are ridiculous | 20:08 |
| SpamapS | they're effectively software | 20:08 |
| lifeless | they shouldn't be in /etc then | 20:09 |
| lifeless | file bugs | 20:09 |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Destroy all testenv domains https://review.openstack.org/79502 | 20:09 |
| SpamapS | lifeless: would you think less of me if I said I'm a little tired of filing bugs like that, sort of like I eventually get tired of throwing pebbles back in the sea? | 20:10 |
| lifeless | SpamapS: no, I would sympthasise and tell you to flame the list | 20:10 |
| SpamapS | BURNNNNNN | 20:10 |
| SpamapS | Ok so I need to test 79631 + 79750 ... | 20:11 |
| SpamapS | if I make one dep on the other will CI test them together? | 20:11 |
| SpamapS | dur.. of course it will | 20:11 |
| openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack/tripleo-image-elements: List filesystem and swift as known glance stores https://review.openstack.org/79631 | 20:12 |
| SpamapS | ok.. so that CI run should pass if the keystone thing is in fact what I think it is. | 20:12 |
| *** vkozhukalov has quit IRC | 20:13 | |
| *** rcarrillocruz has quit IRC | 20:13 | |
| * bnemec subscribes to the review | 20:14 | |
| SpamapS | Ok, while it runs I think I need a little TF2 style carnage. | 20:15 |
| *** rcarrillocruz has joined #tripleo | 20:16 | |
| *** hashar has joined #tripleo | 20:17 | |
| bnemec | TF2 has much more fun criticals than we do. :-) | 20:17 |
| *** andreaf has joined #tripleo | 20:20 | |
| tchaypo | So if you're looking for something fare more exciting than TF2 | 20:21 |
| *** andreaf has quit IRC | 20:21 | |
| tchaypo | lifeless was talking to me last week about wanting to have a more robust way of bootstrapping things like galera | 20:22 |
| tchaypo | rather than nominating one node as the master and building it first, using a consensus protocal such as paxos to let the nodes decides on a master for themselves, and if that master dies during the build, just elect another one | 20:23 |
| slagle | SpamapS: were you saying you thought git format-patch -1 bddd2908b3f9758a8c6be9594599200101f5d7bb | 20:23 |
| tchaypo | so this led to me looking at python paxos implementations, particularly concoord, and it made me sad | 20:23 |
| slagle | gah, copy/paste fail | 20:23 |
| * slagle tries again | 20:23 | |
| slagle | SpamapS: were you saying you thought https://review.openstack.org/#/c/79750/ was a fix for the "Authentication Required" bug I filed? | 20:24 |
| tchaypo | concoord doesn't bother using paxos to elect a leader, it just uses the node with the lowest port number. pragmatically speaking that might work fine most of the time... | 20:24 |
| tchaypo | when nodes go away, instead of saying "oh no, i don't have quorum any more", it just shrinks its idea of what the quorum is - so if there's segmentation, both sides will happily elect a leader and keep accepting proposals | 20:25 |
| greghaynes | tchaypo: is that split brain documented somewhere? | 20:25 |
| greghaynes | seems like a big issue | 20:25 |
| tchaypo | greghaynes: no, it's something i noticed from debug logs and then confirmed by looking at the code | 20:26 |
| greghaynes | huh. | 20:26 |
| tchaypo | its idea of quorum is "number of currently live nodes /2 + 1" | 20:26 |
| *** jprovazn has joined #tripleo | 20:26 | |
| greghaynes | thats normal | 20:26 |
| *** weshay has quit IRC | 20:26 | |
| *** rupsky has quit IRC | 20:27 | |
| * greghaynes tries not to get nerdsniped by things on the backburner | 20:27 | |
| tchaypo | it's not what i expected | 20:27 |
| *** rlandy has quit IRC | 20:27 | |
| lifeless | tchaypo: this is very odd :) | 20:27 |
| tchaypo | it leads to the split-brain thing | 20:27 |
| lifeless | tchaypo: perhaps raft is better? | 20:27 |
| lifeless | tchaypo: either way we're going to be a stte machine on top of the substrate | 20:27 |
| lifeless | tchaypo: I think we probably want to write up that state machine into the etherpad | 20:28 |
| tchaypo | because segmentation leads to both sides just discarding all the nodes that aren't talking to them any more and shrinking their quorum... when what I'd like is for the smaller half of the segmentation to realise they don't have a quorum and sit around waiting for a reconnect | 20:28 |
| tchaypo | lifeless: I was hoping to avoid that, but I'm starting to think it's inevitable. | 20:29 |
| greghaynes | yea, that seems silly | 20:29 |
| tchaypo | Writing a consensus protocol on top of a consensus protocol.. | 20:29 |
| greghaynes | lol | 20:29 |
| tchaypo | I found the live chat and asked about smartbuy, they gave me an entry to put in /etc/hosts | 20:29 |
| greghaynes | oh yes | 20:29 |
| tchaypo | if only there was some way of distributing these hostname/ip mappings | 20:30 |
| *** rupsky has joined #tripleo | 20:30 | |
| lifeless | tchaypo: sbesp.houston.hp.com ? yeah, epic DNS fail | 20:30 |
| tchaypo | I swear I poked at py-raft last week, but since I don't remember anything I can only assume that it must have been on sunday | 20:31 |
| *** rcarrillocruz has quit IRC | 20:31 | |
| tchaypo | greghaynes: it sounds like you've got some prior experience with concoord? | 20:32 |
| greghaynes | nope | 20:33 |
| tchaypo | just paxos in general? | 20:33 |
| greghaynes | I do have a couple papers in my backlog before I dive into that, though | 20:33 |
| greghaynes | not much, just have had to interface with a couple services that do similar things before | 20:33 |
| lifeless | tchaypo: whats the etherpad again ? | 20:34 |
| tchaypo | https://etherpad.openstack.org/p/tripleo-robust-bootstrap | 20:35 |
| tchaypo | so if we're building state on top of an underlying consensus mechanism | 20:36 |
| tchaypo | maybe it doesn't matter if the underlying implementations' idea of quorum is a little strange | 20:36 |
| tchaypo | if heat is telling the nodes "There's six of you", that can be fed into our state machine | 20:37 |
| tchaypo | and we can do the quorum checking ourselves | 20:37 |
| *** d0ugal_ has joined #tripleo | 20:38 | |
| lifeless | tchaypo: ugh no lets avoid that sort of insanity :) | 20:39 |
| *** d0ugal has quit IRC | 20:39 | |
| * SpamapS feels better now that he sploded stuff | 20:39 | |
| lifeless | tchaypo: specifically, lets have a simple state machine that has amongst its possible transitions 'new master' for every input state | 20:40 |
| *** julim has quit IRC | 20:40 | |
| SpamapS | slagle: yes I think that may be it. | 20:40 |
| SpamapS | slagle: the undercloud keystone log has some refused API calls in it | 20:40 |
| *** valerii has joined #tripleo | 20:43 | |
| *** d0ugal_ has quit IRC | 20:43 | |
| tchaypo | so my lateset devtest failed because... | 20:44 |
| tchaypo | "given size (21504) exceeds max allowable size" | 20:44 |
| tchaypo | greghaynes: didn't you have this happen yesterday? | 20:44 |
| *** d0ugal_ has joined #tripleo | 20:45 | |
| SpamapS | tchaypo: more context? | 20:45 |
| *** valerii has quit IRC | 20:47 | |
| slagle | tchaypo: https://review.openstack.org/#/c/79344/ should fix that | 20:49 |
| slagle | SpamapS: cool, thx | 20:49 |
| SpamapS | slagle: I'm running up a local overcloud from scratch here to confirm that | 20:50 |
| lifeless | tchaypo: see the bottom of the etherpad | 20:50 |
| greghaynes | I overclouded from scratch and that worked | 20:53 |
| greghaynes | earlier today | 20:53 |
| greghaynes | well, that + the glance fix | 20:54 |
| *** jdob has quit IRC | 20:58 | |
| tchaypo | http://paste.openstack.org/show/73190/ has more context | 20:59 |
| tchaypo | thanks slagle and greghaynes, that looks like it's it | 21:00 |
| *** e0ne has joined #tripleo | 21:00 | |
| *** petertoft has quit IRC | 21:02 | |
| jprovazn | greghaynes, hi, I tried the percona patch today, did you hit any memory issue? On my devtest setup (fedora 64bit) after a short time mysql allocates ~700M memory and any openstack command fails because of not enough memory | 21:03 |
| greghaynes | eek, no I did not | 21:03 |
| jprovazn | greghaynes, it seems that some memory optimization in my.cnf would be required | 21:03 |
| greghaynes | yep | 21:04 |
| lifeless | 700M. yay. | 21:04 |
| jprovazn | greghaynes, will try look at this deeper tomorrow, but if you find anything, let me know please | 21:04 |
| greghaynes | ok, building with that patch now and will mess with it | 21:04 |
| jprovazn | greghaynes, thanks | 21:04 |
| tchaypo | lifeless: i think we also need to handle a couple of transitions - having a replica suddenly elected the new master, and having a master re-connect after a split and get demoted to a replica | 21:06 |
| tchaypo | but I guess the specifics of how to handle that will depend on what it is that we're deploying | 21:07 |
| *** rpodolyaka has joined #tripleo | 21:09 | |
| lifeless | tchaypo: so, walk through the state transition for that. What state are we in, what happens. Perhaps in the etherpad to capture it | 21:15 |
| *** e0ne has quit IRC | 21:16 | |
| *** e0ne has joined #tripleo | 21:16 | |
| *** jprovazn has quit IRC | 21:20 | |
| *** StevenK_ has joined #tripleo | 21:25 | |
| *** noslzzp has quit IRC | 21:28 | |
| *** ftcjeff has quit IRC | 21:28 | |
| *** morazi has quit IRC | 21:28 | |
| *** tomh_ has quit IRC | 21:28 | |
| *** mkerrin has quit IRC | 21:28 | |
| *** sdake_ has quit IRC | 21:28 | |
| *** rwsu has quit IRC | 21:28 | |
| *** ohadlevy has quit IRC | 21:28 | |
| *** d0ugal_ has quit IRC | 21:28 | |
| *** cwolferh has quit IRC | 21:28 | |
| *** rollerj has quit IRC | 21:28 | |
| *** victor_lowther_ has quit IRC | 21:28 | |
| *** tserong has quit IRC | 21:28 | |
| *** dkehn_ has quit IRC | 21:28 | |
| *** dtantsur has quit IRC | 21:28 | |
| *** tteggel has quit IRC | 21:28 | |
| *** EntropyWorks has quit IRC | 21:28 | |
| *** rainya has quit IRC | 21:28 | |
| *** stevebaker has quit IRC | 21:28 | |
| *** funzo has quit IRC | 21:28 | |
| *** bauzas has quit IRC | 21:28 | |
| *** Matt2 has quit IRC | 21:28 | |
| *** giulivo has quit IRC | 21:28 | |
| *** Kim-Chi-San has quit IRC | 21:28 | |
| *** antonym has quit IRC | 21:28 | |
| *** slagle has quit IRC | 21:28 | |
| *** dguerri has quit IRC | 21:28 | |
| *** sdague has quit IRC | 21:28 | |
| *** smulcahy has quit IRC | 21:28 | |
| *** shortstop has quit IRC | 21:28 | |
| *** markmc has quit IRC | 21:28 | |
| *** CaptTofu has quit IRC | 21:28 | |
| *** spzala has quit IRC | 21:28 | |
| *** jtomasek has quit IRC | 21:28 | |
| *** cody-somerville has quit IRC | 21:28 | |
| *** Kiall has quit IRC | 21:28 | |
| *** lynxman has quit IRC | 21:28 | |
| *** vipul has quit IRC | 21:28 | |
| *** sbadia has quit IRC | 21:28 | |
| *** dkehn has quit IRC | 21:28 | |
| *** lsmola has quit IRC | 21:28 | |
| *** ewindisch has quit IRC | 21:28 | |
| *** Shrews has quit IRC | 21:28 | |
| *** clarkb has quit IRC | 21:28 | |
| *** kevinbenton has quit IRC | 21:28 | |
| *** rbrady has quit IRC | 21:28 | |
| *** e0ne has quit IRC | 21:28 | |
| *** rpodolyaka has quit IRC | 21:28 | |
| *** hashar has quit IRC | 21:28 | |
| *** marun has quit IRC | 21:28 | |
| *** jang1 has quit IRC | 21:28 | |
| *** blamar has quit IRC | 21:28 | |
| *** juice has quit IRC | 21:28 | |
| *** davidlenwell has quit IRC | 21:28 | |
| *** AaronGreen has quit IRC | 21:28 | |
| *** sseago has quit IRC | 21:28 | |
| *** Slower_ has quit IRC | 21:28 | |
| *** yamahata_ has quit IRC | 21:28 | |
| *** rha has quit IRC | 21:28 | |
| *** hewbrocc` has quit IRC | 21:28 | |
| *** sgrasley has quit IRC | 21:28 | |
| *** Hefeweizen has quit IRC | 21:28 | |
| *** jeblair has quit IRC | 21:28 | |
| *** openstackgerrit has quit IRC | 21:28 | |
| *** jpeeler has quit IRC | 21:28 | |
| *** sff has quit IRC | 21:28 | |
| *** SloppyJ_ has quit IRC | 21:28 | |
| *** michchap has quit IRC | 21:28 | |
| *** DuncanT- has quit IRC | 21:28 | |
| *** pleia2 has quit IRC | 21:28 | |
| *** shadower has quit IRC | 21:28 | |
| *** spiffxp_ has quit IRC | 21:28 | |
| *** jcooley has quit IRC | 21:28 | |
| *** jogo has quit IRC | 21:28 | |
| *** gilliard has quit IRC | 21:28 | |
| *** matty_dubs has quit IRC | 21:28 | |
| *** lifeless has quit IRC | 21:28 | |
| *** jesusaurus has quit IRC | 21:28 | |
| *** marios has quit IRC | 21:28 | |
| *** retr0h has quit IRC | 21:28 | |
| *** mordred has quit IRC | 21:28 | |
| *** Daviey has quit IRC | 21:28 | |
| *** lucasagomes has quit IRC | 21:28 | |
| *** greghaynes has quit IRC | 21:28 | |
| *** al has quit IRC | 21:28 | |
| *** boris-42 has quit IRC | 21:28 | |
| *** StevenK has quit IRC | 21:28 | |
| *** wendar has quit IRC | 21:29 | |
| *** adam_g has quit IRC | 21:29 | |
| *** mikal has quit IRC | 21:29 | |
| *** ChanServ has quit IRC | 21:29 | |
| *** morganfainberg has quit IRC | 21:29 | |
| *** cadenzajon has quit IRC | 21:29 | |
| *** Ng has quit IRC | 21:29 | |
| *** ekarlso has quit IRC | 21:29 | |
| *** zaro has quit IRC | 21:29 | |
| *** bnemec has quit IRC | 21:29 | |
| *** anteaya has quit IRC | 21:29 | |
| *** tchaypo has quit IRC | 21:29 | |
| *** jomara has quit IRC | 21:29 | |
| *** phschwartz has quit IRC | 21:29 | |
| *** shardy has quit IRC | 21:29 | |
| *** NobodyCam has quit IRC | 21:29 | |
| *** devananda has quit IRC | 21:29 | |
| *** SlickNik has quit IRC | 21:29 | |
| *** cwolferh has joined #tripleo | 21:41 | |
| *** al has joined #tripleo | 21:41 | |
| *** EntropyWorks has joined #tripleo | 21:41 | |
| *** wendar has joined #tripleo | 21:41 | |
| *** tteggel has joined #tripleo | 21:41 | |
| *** victor_lowther_ has joined #tripleo | 21:41 | |
| *** funzo has joined #tripleo | 21:41 | |
| *** dkehn_ has joined #tripleo | 21:41 | |
| *** tserong has joined #tripleo | 21:41 | |
| *** dtantsur has joined #tripleo | 21:41 | |
| *** adam_g has joined #tripleo | 21:41 | |
| *** stevebaker has joined #tripleo | 21:41 | |
| *** greghaynes has joined #tripleo | 21:41 | |
| *** lucasagomes has joined #tripleo | 21:41 | |
| *** derekh has joined #tripleo | 21:41 | |
| *** boris-42 has joined #tripleo | 21:41 | |
| *** mikal has joined #tripleo | 21:41 | |
| *** weshay has joined #tripleo | 21:41 | |
| *** rpodolyaka has joined #tripleo | 21:41 | |
| *** e0ne has joined #tripleo | 21:41 | |
| *** hashar has joined #tripleo | 21:41 | |
| *** markmc has joined #tripleo | 21:41 | |
| *** bauzas has joined #tripleo | 21:41 | |
| *** marun has joined #tripleo | 21:41 | |
| *** jang1 has joined #tripleo | 21:41 | |
| *** CaptTofu has joined #tripleo | 21:41 | |
| *** cadenzajon has joined #tripleo | 21:41 | |
| *** hewbrocc` has joined #tripleo | 21:41 | |
| *** blamar has joined #tripleo | 21:41 | |
| *** Matt2 has joined #tripleo | 21:41 | |
| *** spzala has joined #tripleo | 21:41 | |
| *** jtomasek has joined #tripleo | 21:41 | |
| *** giulivo has joined #tripleo | 21:41 | |
| *** Kim-Chi-San has joined #tripleo | 21:41 | |
| *** antonym has joined #tripleo | 21:41 | |
| *** retr0h has joined #tripleo | 21:41 | |
| *** morganfainberg has joined #tripleo | 21:41 | |
| *** sgrasley has joined #tripleo | 21:41 | |
| *** Hefeweizen has joined #tripleo | 21:41 | |
| *** cody-somerville has joined #tripleo | 21:41 | |
| *** dguerri has joined #tripleo | 21:41 | |
| *** juice has joined #tripleo | 21:41 | |
| *** davidlenwell has joined #tripleo | 21:41 | |
| *** AaronGreen has joined #tripleo | 21:41 | |
| *** sseago has joined #tripleo | 21:41 | |
| *** Slower_ has joined #tripleo | 21:41 | |
| *** jeblair has joined #tripleo | 21:41 | |
| *** dickson.freenode.net sets mode: +v davidlenwell | 21:41 | |
| *** slagle has joined #tripleo | 21:41 | |
| *** ekarlso has joined #tripleo | 21:41 | |
| *** openstackgerrit has joined #tripleo | 21:41 | |
| *** jpeeler has joined #tripleo | 21:41 | |
| *** sdague has joined #tripleo | 21:41 | |
| *** smulcahy has joined #tripleo | 21:41 | |
| *** shortstop has joined #tripleo | 21:41 | |
| *** lynxman has joined #tripleo | 21:41 | |
| *** vipul has joined #tripleo | 21:41 | |
| *** yamahata_ has joined #tripleo | 21:41 | |
| *** sbadia has joined #tripleo | 21:41 | |
| *** dkehn has joined #tripleo | 21:41 | |
| *** sff has joined #tripleo | 21:41 | |
| *** lsmola has joined #tripleo | 21:41 | |
| *** Ng has joined #tripleo | 21:41 | |
| *** SloppyJ_ has joined #tripleo | 21:41 | |
| *** ewindisch has joined #tripleo | 21:41 | |
| *** Shrews has joined #tripleo | 21:41 | |
| *** gilliard has joined #tripleo | 21:41 | |
| *** rha has joined #tripleo | 21:41 | |
| *** michchap has joined #tripleo | 21:41 | |
| *** clarkb has joined #tripleo | 21:41 | |
| *** kevinbenton has joined #tripleo | 21:41 | |
| *** DuncanT- has joined #tripleo | 21:41 | |
| *** rbrady has joined #tripleo | 21:41 | |
| *** zaro has joined #tripleo | 21:41 | |
| *** pleia2 has joined #tripleo | 21:41 | |
| *** bnemec has joined #tripleo | 21:41 | |
| *** shadower has joined #tripleo | 21:41 | |
| *** spiffxp_ has joined #tripleo | 21:41 | |
| *** jcooley has joined #tripleo | 21:41 | |
| *** jogo has joined #tripleo | 21:41 | |
| *** anteaya has joined #tripleo | 21:41 | |
| *** tchaypo has joined #tripleo | 21:41 | |
| *** jomara has joined #tripleo | 21:41 | |
| *** matty_dubs has joined #tripleo | 21:41 | |
| *** lifeless has joined #tripleo | 21:41 | |
| *** jesusaurus has joined #tripleo | 21:41 | |
| *** marios has joined #tripleo | 21:41 | |
| *** mordred has joined #tripleo | 21:41 | |
| *** Daviey has joined #tripleo | 21:41 | |
| *** phschwartz has joined #tripleo | 21:41 | |
| *** ChanServ has joined #tripleo | 21:41 | |
| *** shardy has joined #tripleo | 21:41 | |
| *** NobodyCam has joined #tripleo | 21:41 | |
| *** devananda has joined #tripleo | 21:41 | |
| *** dickson.freenode.net sets mode: +vvo Ng lifeless ChanServ | 21:41 | |
| *** SlickNik has joined #tripleo | 21:41 | |
| *** rainya has joined #tripleo | 21:41 | |
| tchaypo | lifeless: the machine; have it pxe-boot again and start the build fresh. | 21:41 |
| *** kiall has joined #tripleo | 21:42 | |
| lifeless | tchaypo: I believe I have handled that case in the state machine | 21:42 |
| tchaypo | when you say "layering violation" i think you're talking about the fact that the system that formerly was just electing a master is suddenly reaching down a layer or two and telling heat to reset the host it's on? | 21:42 |
| derekh | + os-svc-enable -n rsync | 21:43 |
| derekh | Failed to issue method call: No such file or directory | 21:43 |
| *** rollerj has joined #tripleo | 21:44 | |
| tchaypo | just before my session times out again - if i order an SSD from smartbuy, will it come with a caddy to put it in the upgrade bay? I can't find those listed seperately | 21:45 |
| lifeless | tchaypo: no idea :) | 21:45 |
| *** sdake_ has joined #tripleo | 21:46 | |
| *** petertoft has joined #tripleo | 21:46 | |
| *** rwsu has joined #tripleo | 21:46 | |
| *** giulivo has quit IRC | 21:46 | |
| tchaypo | lifeless: then it seems like I don't understand what you've written. I think states are down the left, transitions across the top? | 21:46 |
| lifeless | tchaypo: yes, so consider what happens if a node being configured as master dies | 21:47 |
| tchaypo | and the entries in the cells indicate that you transition from the state on the left via the transition at the top to the state in the cell? | 21:48 |
| lifeless | right | 21:48 |
| openstackgerrit | Ben Nemec proposed a change to openstack/tuskar: Remove tools/flakes.py https://review.openstack.org/79783 | 21:48 |
| lifeless | code to run on the master (can influence transitions) and slaves in each state is listed below | 21:48 |
| tchaypo | give me a second to draw this up a different way to make sure I have it in my head | 21:49 |
| *** morazi has joined #tripleo | 21:49 | |
| *** petertoft has quit IRC | 21:50 | |
| *** e0ne has quit IRC | 21:57 | |
| *** markmc has quit IRC | 21:57 | |
| *** petertoft has joined #tripleo | 21:59 | |
| rbrady | looking for reviews of https://review.openstack.org/#/c/74563/ | 22:00 |
| *** hashar has quit IRC | 22:01 | |
| tchaypo | lifeless: I've added a graphviz version of the state machine, which I prefer because it makes a pretty picture that my brain can understand | 22:03 |
| *** cadenzajon has quit IRC | 22:03 | |
| derekh | frick, pretty much all of the ci jobs simultaneously failed about 30 minutes ago (network blip??) | 22:05 |
| lifeless | tchaypo: dote ? | 22:08 |
| tchaypo | lifeless: well done, you've now conflated the concepts of "dot" and "doge" in my head. | 22:10 |
| tchaypo | Very digraph, much acyclic, wow | 22:10 |
| *** StevenK_ is now known as StevenK | 22:10 | |
| SpamapS | daaamnnit | 22:10 |
| SpamapS | 2014-03-11 21:33:28.560 | Updating cache of https://github.com/kanaka/noVNC.git in /home/jenkins/.cache/image-create/source-repositories/novnc_eac807d7dcef19baf4dd80e1f38fc67a7720c193 with ref master | 22:10 |
| SpamapS | 2014-03-11 21:33:43.049 | error: The requested URL returned error: 403 while accessing https://github.com/kanaka/noVNC.git/info/refs | 22:11 |
| tchaypo | lifeless: not sure if you mean "what is this dot thing, I have not heard of it" or if you mean "when you say graphviz, do you mean dot?" | 22:11 |
| lifeless | tchaypo: I don't have a dote command, and your prose says to use it | 22:11 |
| tchaypo | SpamapS: that's one i saw yesterday, and I can't figure out why. It's worked for me after that, and it works fine in my browser | 22:11 |
| tchaypo | lifeless: well done me. fixed | 22:12 |
| SpamapS | tchaypo: likely rate limiting or just intermittent github fail | 22:12 |
| tchaypo | okay, my female cat seems to be attempting to mount my male cat. I don't know what to think. | 22:13 |
| *** rupsky_ has joined #tripleo | 22:13 | |
| SpamapS | maybe we should d/l the tarball instead of git cloning it | 22:13 |
| jang1 | as long as it's not "youtube needs to see this", tchaypo | 22:14 |
| *** rupsky has quit IRC | 22:14 | |
| tchaypo | lifeless: *really* fixed, now that etherpad has decided to reconnect | 22:15 |
| *** jang1 has quit IRC | 22:16 | |
| tchaypo | lifeless: what I think is missing from that state machine is some of the initialisation. I think we need to start in a "Waiting for quorum" state, which has an "election" transition leading to the "uninitialised" state (which has seperate code paths for master and replicas) | 22:18 |
| *** matty_dubs is now known as matty_dubs|gone | 22:19 | |
| lifeless | tchaypo: so I'm trying to focus on the programming model we need, not on the consensus tech | 22:19 |
| lifeless | tchaypo: for the avoidance of confusion, do you mean 'wait for consensus quorum' or 'wait for db/etc/thing quorum' ? | 22:20 |
| tchaypo | i mean "wait until we have >1/2 the nodes heat says are coming are alive". I think that whether that is the same thing as "consensus quorum" or whether it has to be implemented on a layer above that is an implementation details | 22:22 |
| bnemec | rbrady: Did your change to fix the duplicate RabbitPassword in the block-storage template get lost? I'm still getting that error with your latest change and I don't see any other pending reviews. | 22:23 |
| lifeless | tchaypo: it doesn't, because you can't start running the replicated state machine until consensus is reached | 22:24 |
| tchaypo | yep; so we assume that the underlying consensus mechanism is handling that, and our state machine won't exist until that's true | 22:28 |
| tchaypo | hence waiting for quorum, electing quorum leader etc is outside the scope of this state machine | 22:28 |
| lifeless | right | 22:28 |
| lifeless | We may well need to hack on that consensus plumbing | 22:28 |
| lifeless | but I would like to be able to map what we come up with into e.g. taskflow | 22:28 |
| lifeless | which suggesets separating concerns carefully | 22:29 |
| tchaypo | makes sense | 22:29 |
| tchaypo | Why is Quorum replicas going to master inited via new master? | 22:29 |
| tchaypo | wouldn't that transition happen because the inited master has gone away, so the new master has to start its initialisation again? ie, shouldn't it be back to uninitialised? | 22:30 |
| *** andreaf has joined #tripleo | 22:31 | |
| lifeless | ok so say we hve 3 nodes | 22:33 |
| lifeless | A B C | 22:33 |
| lifeless | A and B come online ok | 22:33 |
| lifeless | and we get a leader (A) | 22:34 |
| lifeless | B gets a full state vector sync | 22:34 |
| lifeless | A fails | 22:34 |
| lifeless | we're now new leader (and thats B) | 22:34 |
| lifeless | A is down, B is up with a full replica (but isn't quorate), and C is still not up | 22:35 |
| ccrouch | SpamapS: from todays meeting, who has the ball around answering this question? | 22:39 |
| ccrouch | 19:59:19 <SpamapS> - There is a question as to whether TripleO's gate jobs can make use of an incubated but not integrated project, like Ironic, that needs answering. | 22:39 |
| *** openstackstatus has joined #tripleo | 22:44 | |
| SpamapS | ccrouch: good question we ended abruptly w/o actions | 22:46 |
| *** petertoft has quit IRC | 22:49 | |
| *** CaptTofu has quit IRC | 22:51 | |
| lifeless | tchaypo: so - I think we should let this mull and evolve and focus on things that affect more than initial deploy for now | 23:00 |
| lifeless | tchaypo: since its fairly crunch time | 23:00 |
| openstackgerrit | tom-howley proposed a change to openstack/tripleo-incubator: Add standadlone script to build all images. https://review.openstack.org/77462 | 23:07 |
| *** rpodolyaka has quit IRC | 23:09 | |
| openstackgerrit | tom-howley proposed a change to openstack/tripleo-incubator: Add standalone script to build all images. https://review.openstack.org/77462 | 23:09 |
| *** rpodolyaka has joined #tripleo | 23:12 | |
| tchaypo | lifeless: agree | 23:14 |
| tchaypo | I'd say that in the scnario above we don't have a new leader. B can't know if it's really the only survivor, or if A and C are talking but segregated from it | 23:15 |
| *** lucasagomes has quit IRC | 23:16 | |
| tchaypo | If C starts talking to B, *then* we have a new leader, and it makes sense to elect B rather than C (election protocol probably needs to weight "full state vector" highly) | 23:16 |
| lifeless | tchaypo: right, so the state machine pauses | 23:16 |
| lifeless | tchaypo: then if A comes back either A becomes leader, or B becomes leader. | 23:17 |
| lifeless | tchaypo: so there you want to make sure the thing under intiialisation hits quorum itself, and then move forward a state | 23:17 |
| lifeless | tchaypo: but if A doesn't come back and C comes back, thats when B || C becomes leader. | 23:17 |
| lifeless | tchaypo: if B becomes the leader. we have one replica and need to wait for C (slave) to sync from B; if C becomes leader I'm proposing we drop back down to unitialised (so that we don't have to deal with a master that is not leader | 23:18 |
| *** shakayumi has joined #tripleo | 23:19 | |
| lifeless | (see line 103 of the etherpad) | 23:19 |
| * tchaypo is certified ethically compliant and stuff | 23:20 | |
| lifeless | .oO | 23:20 |
| *** andreaf has quit IRC | 23:21 | |
| openstackgerrit | tom-howley proposed a change to openstack/tripleo-incubator: Add standalone script to build all images. https://review.openstack.org/77462 | 23:27 |
| SpamapS | btw we should not use 'slave' and 'master' .. it is insensitive. dom and sub have far less negative cultural context. | 23:29 |
| tchaypo | I swear I saw a van that said "S&M Cleaning" while I was in Sunnyvale | 23:30 |
| SpamapS | haha | 23:30 |
| SpamapS | We don't clean the house, BUT WE WILL MAKE SURE YOU DO | 23:30 |
| SpamapS | MAGGOT | 23:31 |
| tchaypo | ... come to think of it, around SF, that kind of niche is probably big enough to attract one or two businesses | 23:31 |
| SpamapS | who you calling "niche" ;) | 23:31 |
| tchaypo | In other news | 23:33 |
| tchaypo | I justt discovered https://code.google.com/p/canviz/ | 23:33 |
| tchaypo | and now I'm imagining a graphviz-pastebin | 23:33 |
| tchaypo | paste your text, get a url to a rendered graph | 23:33 |
| SpamapS | tchaypo: lovely | 23:35 |
| lifeless | tchaypo: like http://g.jk.gs/ ? | 23:36 |
| tchaypo | almost exactly like that | 23:38 |
| SpamapS | 2014-03-11 23:35:22.803 | Waiting for the overcloud stack to be ready | 23:38 |
| tchaypo | except that rather than a static .png I'd like the graph-source to be an etherpad. | 23:38 |
| SpamapS | so | 23:38 |
| SpamapS | much | 23:38 |
| SpamapS | waiting | 23:38 |
| tchaypo | when i get to that bit, i tend to start opening a second window and watching logs and checking heat/nova output by hand | 23:38 |
| tchaypo | since the scripts don't seem to notice failures | 23:39 |
| tchaypo | https://review.openstack.org/#/c/79344/ seems to be breaking lots of things, mayhap it should be listed as FIREDRILL as well? | 23:41 |
| SpamapS | tchaypo: yes, we need CI to do that | 23:41 |
| *** rupsky_ has quit IRC | 23:41 | |
| SpamapS | tchaypo: what? More fail? I can't take more fail. | 23:42 |
| lifeless | lunch | 23:43 |
| tchaypo | neither can I, but it's been 5 hours since I started work so I'm stepping away. Going to go on a bit of an officeworks binge - need to get myself a new keyboard, a laptop stand, and maybe some kind of desk organizer thing | 23:43 |
| tchaypo | openstack-infra mailing list has someone asking about kanaka/noVNC | 23:45 |
| tchaypo | which mordred has responded to already, very good | 23:48 |
| *** dividehex has joined #tripleo | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!