| NobodyCam | it works | 00:00 |
|---|---|---|
| jlvillal | NobodyCam: I think it is bad :( | 00:02 |
| jlvillal | NobodyCam: http://paste.openstack.org/show/496640/ | 00:03 |
| jlvillal | NobodyCam: Seems to call the function one time and then store that value... | 00:04 |
| jlvillal | NobodyCam: So my gut feeling was wrong. | 00:04 |
| jlvillal | From a StackForge comment: When the function is instantiated (when the declaration is executed), the default parameter expression is evaluated, and the resulting object is bound to the created function object. | 00:05 |
| * jlvillal learned something new :) | 00:06 | |
| NobodyCam | good searching jlvillal :) | 00:06 |
| jlvillal | NobodyCam: Thanks :) | 00:07 |
| jlvillal | NobodyCam: http://stackoverflow.com/questions/1132941/least-astonishment-in-python-the-mutable-default-argument | 00:07 |
| *** spandhe has quit IRC | 00:07 | |
| jlvillal | NobodyCam: Good job knowing (in your gut) that it was wrong! | 00:08 |
| NobodyCam | would you like to add that to the review I was looking at with just that situation | 00:08 |
| jlvillal | NobodyCam: Link? | 00:08 |
| *** rama_y has quit IRC | 00:08 | |
| NobodyCam | https://review.openstack.org/#/c/297727 | 00:10 |
| *** wajdi has joined #openstack-ironic | 00:12 | |
| jlvillal | NobodyCam: Thanks and commented | 00:13 |
| NobodyCam | no no Thank you for the great searching | 00:13 |
| jlvillal | :) | 00:14 |
| *** chlong_ has joined #openstack-ironic | 00:22 | |
| *** chlong_ has quit IRC | 00:22 | |
| *** chlong_ has joined #openstack-ironic | 00:27 | |
| *** chlong_ has quit IRC | 00:28 | |
| *** mbound has quit IRC | 00:45 | |
| *** mtanino has quit IRC | 00:48 | |
| *** rloo has joined #openstack-ironic | 00:50 | |
| *** jaybeale has quit IRC | 00:55 | |
| *** rloo has quit IRC | 00:56 | |
| jlvillal | cinerama, I think it should be back to normal now. I reverted my change about the tinyipa 7 nodes | 01:01 |
| *** jaybeale has joined #openstack-ironic | 01:03 | |
| *** crinkle has joined #openstack-ironic | 01:04 | |
| *** hoangcx has joined #openstack-ironic | 01:06 | |
| *** penick has quit IRC | 01:08 | |
| *** jaybeale has quit IRC | 01:11 | |
| *** rama_y has joined #openstack-ironic | 01:19 | |
| *** ChrisAusten has joined #openstack-ironic | 01:22 | |
| *** aNupoisc has quit IRC | 01:25 | |
| *** piet has joined #openstack-ironic | 01:28 | |
| *** amotoki has quit IRC | 01:30 | |
| *** amotoki has joined #openstack-ironic | 01:41 | |
| *** mbound has joined #openstack-ironic | 01:46 | |
| *** vishwanathj has quit IRC | 01:48 | |
| *** amotoki has quit IRC | 01:50 | |
| *** mbound has quit IRC | 01:51 | |
| *** piet has quit IRC | 01:54 | |
| *** amotoki has joined #openstack-ironic | 01:56 | |
| *** joprovost has joined #openstack-ironic | 02:10 | |
| *** joprovost1 has joined #openstack-ironic | 02:13 | |
| *** joprovost has quit IRC | 02:15 | |
| *** joprovost1 is now known as joprovost | 02:15 | |
| *** amotoki has quit IRC | 02:19 | |
| *** spandhe has joined #openstack-ironic | 02:19 | |
| *** hoangcx has quit IRC | 02:24 | |
| *** absubram has joined #openstack-ironic | 02:29 | |
| *** absubram_ has joined #openstack-ironic | 02:30 | |
| *** amotoki has joined #openstack-ironic | 02:32 | |
| *** absubram has quit IRC | 02:34 | |
| *** absubram_ is now known as absubram | 02:34 | |
| *** hoangcx has joined #openstack-ironic | 02:36 | |
| *** baoli has joined #openstack-ironic | 02:38 | |
| *** piet has joined #openstack-ironic | 02:45 | |
| *** joprovost has quit IRC | 02:48 | |
| *** joprovost has joined #openstack-ironic | 02:49 | |
| openstackgerrit | Tan Lin proposed openstack/ironic: Update the api-ref documentation for Root and Nodes https://review.openstack.org/312795 | 03:03 |
| *** absubram has quit IRC | 03:05 | |
| *** piet has quit IRC | 03:07 | |
| *** absubram has joined #openstack-ironic | 03:08 | |
| *** joprovost has joined #openstack-ironic | 03:08 | |
| *** Nisha has joined #openstack-ironic | 03:12 | |
| *** Nisha_away has joined #openstack-ironic | 03:12 | |
| *** joprovost1 has joined #openstack-ironic | 03:19 | |
| *** joprovost has quit IRC | 03:21 | |
| *** joprovost1 is now known as joprovost | 03:21 | |
| *** links has joined #openstack-ironic | 03:21 | |
| *** joprovost has quit IRC | 03:22 | |
| *** joprovost has joined #openstack-ironic | 03:22 | |
| *** joprovost has quit IRC | 03:23 | |
| *** joprovost has joined #openstack-ironic | 03:24 | |
| *** joprovost has quit IRC | 03:24 | |
| *** joprovost has joined #openstack-ironic | 03:25 | |
| *** joprovost has joined #openstack-ironic | 03:26 | |
| *** joprovost has quit IRC | 03:27 | |
| *** joprovost has joined #openstack-ironic | 03:27 | |
| *** joprovost has quit IRC | 03:27 | |
| *** joprovost has joined #openstack-ironic | 03:28 | |
| *** mbound has joined #openstack-ironic | 03:30 | |
| *** mbound has quit IRC | 03:35 | |
| *** vishwanathj has joined #openstack-ironic | 03:36 | |
| *** irf has joined #openstack-ironic | 03:37 | |
| *** joprovost1 has joined #openstack-ironic | 03:37 | |
| *** joprovost has quit IRC | 03:39 | |
| *** joprovost1 is now known as joprovost | 03:39 | |
| *** joprovost has quit IRC | 03:52 | |
| *** Nisha has quit IRC | 03:54 | |
| *** vishwanathj has quit IRC | 03:54 | |
| *** Nisha_away has quit IRC | 03:54 | |
| *** baoli has quit IRC | 04:01 | |
| *** pcaruana has joined #openstack-ironic | 04:02 | |
| *** jaybeale has joined #openstack-ironic | 04:03 | |
| *** vmud213 has joined #openstack-ironic | 04:04 | |
| *** stendulker has joined #openstack-ironic | 04:09 | |
| *** jaybeale has quit IRC | 04:16 | |
| *** appu103_ has joined #openstack-ironic | 04:29 | |
| *** yuanying has quit IRC | 04:44 | |
| *** jaybeale has joined #openstack-ironic | 04:45 | |
| *** amotoki_ has joined #openstack-ironic | 04:46 | |
| *** amotoki has quit IRC | 04:48 | |
| *** ijw has quit IRC | 04:53 | |
| *** rcernin has joined #openstack-ironic | 04:58 | |
| *** jaybeale has quit IRC | 05:10 | |
| *** Nisha_away has joined #openstack-ironic | 05:16 | |
| *** Nisha has joined #openstack-ironic | 05:16 | |
| *** Nisha_away has quit IRC | 05:17 | |
| *** jaybeale has joined #openstack-ironic | 05:23 | |
| *** ChubYann has quit IRC | 05:26 | |
| *** jaybeale has quit IRC | 05:31 | |
| *** appu103_ is now known as aparnav | 05:33 | |
| *** absubram has quit IRC | 05:40 | |
| *** absubram has joined #openstack-ironic | 05:41 | |
| *** amotoki_ has quit IRC | 05:56 | |
| *** Nisha has quit IRC | 06:01 | |
| *** Nisha has joined #openstack-ironic | 06:01 | |
| *** yuanying has joined #openstack-ironic | 06:01 | |
| *** spandhe has quit IRC | 06:03 | |
| *** piet has joined #openstack-ironic | 06:16 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack/python-ironicclient: Trivial: ignore openstack/common in flake8 exclude list https://review.openstack.org/314877 | 06:23 |
| *** david-lyle has quit IRC | 06:23 | |
| *** rama_y has quit IRC | 06:24 | |
| *** piet has quit IRC | 06:25 | |
| *** david-lyle has joined #openstack-ironic | 06:26 | |
| *** jaybeale has joined #openstack-ironic | 06:32 | |
| *** irf has quit IRC | 06:43 | |
| *** sacharya has quit IRC | 06:56 | |
| *** fragatina has quit IRC | 07:00 | |
| *** fragatina has joined #openstack-ironic | 07:00 | |
| *** amotoki has joined #openstack-ironic | 07:04 | |
| *** jaybeale has quit IRC | 07:04 | |
| *** fragatin_ has joined #openstack-ironic | 07:05 | |
| *** fragatina has quit IRC | 07:06 | |
| -openstackstatus- NOTICE: pip 8.1.2 broke our local python mirror, some jobs will fail with "No matching distribution found". We're investigating. Do not "recheck" until the issue is solved | 07:06 | |
| *** ChanServ changes topic to "pip 8.1.2 broke our local python mirror, some jobs will fail with "No matching distribution found". We're investigating. Do not "recheck" until the issue is solved" | 07:06 | |
| *** rcernin has quit IRC | 07:09 | |
| *** rcernin has joined #openstack-ironic | 07:10 | |
| *** fragatin_ has quit IRC | 07:10 | |
| *** e0ne has joined #openstack-ironic | 07:11 | |
| *** fragatina has joined #openstack-ironic | 07:12 | |
| openstackgerrit | Vasyl Saienko proposed openstack/ironic: TEST Checking Gates https://review.openstack.org/311109 | 07:13 |
| *** fragatina has quit IRC | 07:17 | |
| *** yuanying has quit IRC | 07:18 | |
| *** amotoki has quit IRC | 07:18 | |
| *** athomas has joined #openstack-ironic | 07:21 | |
| openstackgerrit | Om Kumar proposed openstack/ironic: Add support for providing custom Swift endpoint URLs https://review.openstack.org/251277 | 07:22 |
| *** fragatina has joined #openstack-ironic | 07:24 | |
| *** amotoki has joined #openstack-ironic | 07:25 | |
| *** daemontool_ has joined #openstack-ironic | 07:26 | |
| *** gabriel-bezerra has joined #openstack-ironic | 07:26 | |
| *** fragatina has quit IRC | 07:28 | |
| *** fragatina has joined #openstack-ironic | 07:30 | |
| openstackgerrit | Om Kumar proposed openstack/ironic: Add support for providing custom Swift endpoint URLs https://review.openstack.org/251277 | 07:31 |
| *** yolanda has joined #openstack-ironic | 07:37 | |
| *** ifarkas has joined #openstack-ironic | 07:37 | |
| *** ohamada has joined #openstack-ironic | 07:39 | |
| *** chlong has quit IRC | 07:49 | |
| *** amotoki has quit IRC | 07:54 | |
| *** milan has quit IRC | 07:54 | |
| *** amotoki has joined #openstack-ironic | 07:55 | |
| *** sacharya has joined #openstack-ironic | 07:56 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-ironic | 08:00 | |
| *** sacharya has quit IRC | 08:01 | |
| *** amotoki has quit IRC | 08:04 | |
| *** ifarkas has quit IRC | 08:04 | |
| lucasagomes | morning all | 08:07 |
| *** dmk0202 has joined #openstack-ironic | 08:08 | |
| *** yushiro has joined #openstack-ironic | 08:09 | |
| *** kromanenko_ has joined #openstack-ironic | 08:11 | |
| yushiro | ping vsaienko, | 08:11 |
| vsaienko | morning yushiro | 08:13 |
| vsaienko | morning lucasagomes and all | 08:14 |
| yushiro | vsaienko, morning :) Let me confirm about ironic-neutron integration patches. | 08:15 |
| yushiro | vsaienko, Currently, I'm testing these patches in my environment. | 08:15 |
| *** irf has joined #openstack-ironic | 08:15 | |
| yushiro | vsaienko, However, some patch occurs 'merge conflict'. | 08:16 |
| yushiro | vsaienko, So, I'd like to rebase these patches. Is it OK? | 08:16 |
| vsaienko | yushiro, there are some comments that should be resolved | 08:18 |
| *** daemontool_ has quit IRC | 08:18 | |
| *** ifarkas has joined #openstack-ironic | 08:19 | |
| yushiro | vsaienko, Yes. | 08:20 |
| *** daemontool_ has joined #openstack-ironic | 08:20 | |
| *** irf has quit IRC | 08:20 | |
| *** Haomeng has quit IRC | 08:23 | |
| *** irf has joined #openstack-ironic | 08:23 | |
| openstackgerrit | Om Kumar proposed openstack/ironic: Add support for providing custom Swift endpoint URLs https://review.openstack.org/251277 | 08:26 |
| *** irf has quit IRC | 08:32 | |
| *** irf has joined #openstack-ironic | 08:32 | |
| irf | morning Ironic!! | 08:33 |
| *** yuanying has joined #openstack-ironic | 08:34 | |
| *** Nisha has quit IRC | 08:35 | |
| *** Nisha has joined #openstack-ironic | 08:35 | |
| *** yuanying has quit IRC | 08:37 | |
| *** jistr has joined #openstack-ironic | 08:38 | |
| *** yuanying has joined #openstack-ironic | 08:38 | |
| sambetts|afk | Morning all | 08:39 |
| *** sambetts|afk is now known as sambetts | 08:39 | |
| *** yuanying has quit IRC | 08:41 | |
| irf | virsh command is to verify the stats of the hypervisor ? | 08:43 |
| *** yuanying has joined #openstack-ironic | 08:43 | |
| sambetts | virsh is for controlling libvirt | 08:46 |
| *** yuanying has quit IRC | 08:47 | |
| irf | ok | 08:47 |
| irf | ExactRamFilter tries to match a host with the same amount of RAM as the amount of RAM specified in the flavor chosen for the VM ? is this rite ? | 08:48 |
| sambetts | yes | 08:49 |
| *** amotoki has joined #openstack-ironic | 08:50 | |
| *** derekh has joined #openstack-ironic | 08:51 | |
| irf | ok | 08:52 |
| *** yuanying has joined #openstack-ironic | 08:52 | |
| vmud213 | morning all | 08:55 |
| vmud213 | lucasagomes,sambetts: i have a question related to the presence of package in DIB | 08:56 |
| vmud213 | s/package/sudo package | 08:56 |
| vmud213 | I see that this package is in the list of packages to be removed | 08:57 |
| *** yuanying has quit IRC | 08:57 | |
| vmud213 | for fedora | 08:57 |
| *** sacharya has joined #openstack-ironic | 08:59 | |
| vmud213 | But when devuser element is included, we need this package as i see it being added to sudoers | 09:00 |
| vmud213 | What's your take on this? | 09:00 |
| *** yushiro has quit IRC | 09:01 | |
| *** yushiro has joined #openstack-ironic | 09:01 | |
| vmud213 | lucasagomes,sambetts ^^^ | 09:02 |
| *** milan has joined #openstack-ironic | 09:02 | |
| sambetts | vmud213: hmm, interesting, so we strip packages out to help reduce the size of the ramdisk produced, I assume you want the devuser package added for debugging? | 09:03 |
| openstackgerrit | Sharat Sharma proposed openstack/ironic: Added the request parameter maintenance for node.update. https://review.openstack.org/314917 | 09:03 |
| milan | morning all! | 09:03 |
| sambetts | Hey milan | 09:03 |
| vmud213 | yes.But anyone wanting that to use will also expect sudo to run certain operations | 09:03 |
| milan | sambetts, hi! :) | 09:04 |
| *** sacharya has quit IRC | 09:04 | |
| vmud213 | So i think if the image built using the devuser element we should not remove this package. | 09:05 |
| vmud213 | morning milan | 09:05 |
| milan | vmud213, morning :) | 09:05 |
| sambetts | if we can have a conditional to make that possible then sure, but I don't want to add it permenantly because we don't require it for IPA | 09:06 |
| *** mgould|afk is now known as mgould | 09:06 | |
| mgould | morning everyone! | 09:06 |
| sambetts | o/ mgould | 09:06 |
| *** yushiro has quit IRC | 09:06 | |
| lucasagomes | vmud213, ++ we can change the ironic-agent element to not remove it | 09:06 |
| *** electrofelix has joined #openstack-ironic | 09:06 | |
| irf | good morning mgould, milan , vmud213 and all | 09:06 |
| mgould | morning sambetts vmud213 lucasagomes milan | 09:07 |
| lucasagomes | vmud213, mind putting up a patch for dib? | 09:07 |
| lucasagomes | mgould, morning | 09:07 |
| milan | sambetts or ifarkas guys, we've got several +1 and a +2 on this patch: https://review.openstack.org/#/c/276331/7 authored by dtantsur|afk would you mind reviewing? ;) | 09:07 |
| milan | morning irf mgould lucasagomes :) | 09:07 |
| sambetts | milan: I'll take a look in a minute | 09:07 |
| lucasagomes | milan, good morning | 09:07 |
| milan | sambetts, thx! | 09:08 |
| milan | lucasagomes hi :) | 09:08 |
| vmud213 | yes, i already put a patch | 09:08 |
| vmud213 | lucasagomes,sambetts: https://review.openstack.org/#/c/308747/ | 09:08 |
| vmud213 | morning irf | 09:09 |
| vmud213 | mgould morning | 09:09 |
| lucasagomes | vmud213, right, you are removing sudo only if devstack user element is being used | 09:12 |
| lucasagomes | I think I agree with ian, we can just remove it at all times | 09:12 |
| sambetts | vmud213: my question is are there other elements that could be mixed in that require sudo or other packages that we're stripping? I wonder if we should make removing the packages optional using a IRONIC_AGENT_STRIP_PACKAGES=true/false flag so that if someone is mixing in a different element like devuser or whatever that they can disable it all | 09:12 |
| lucasagomes | I mean remove it from the list of packages to be removed heh | 09:12 |
| sambetts | or a IRONIC_AGENT_DONT_STRIP=<list of packages to exclude>, so if you require sudo then you can do IRONIC_AGENT_DONT_STRIP=sudo | 09:13 |
| vmud213 | sambetts, actually that is already included but removed in the final stage to make the ramdisk compact. | 09:14 |
| vmud213 | sambetts +1 | 09:15 |
| vmud213 | lucasagomes, so u think, we don't need to even check for devuser element. instead remove "sudo" from the "list of packages to be removed" | 09:16 |
| vmud213 | lucasagomes: But why do we need it if only "root" user exists? | 09:17 |
| sambetts | I think its a wider problem, e.g. if I wanted to mix in an element that requires genisoimage for example, ironic-agent will strip it | 09:17 |
| vmud213 | sambetts, yes.We need to address this in a generic way not just limiting to this context. | 09:18 |
| openstackgerrit | Sam Betts proposed openstack/ironic-python-agent: TinyIPA: Shave off some file size from tinyipa ramdisk https://review.openstack.org/313675 | 09:20 |
| lucasagomes | vmud213, well there are other elements also stackuser and things like that | 09:20 |
| lucasagomes | other elements that create users* | 09:20 |
| *** tesseract has joined #openstack-ironic | 09:20 | |
| vmud213 | lucasagomes: ok. i got it. thanks | 09:21 |
| *** daemontool_ has quit IRC | 09:21 | |
| vmud213 | sambetts: A separate patch is worth for the idea u have. Will try to work on it if time permits. | 09:22 |
| *** daemontool_ has joined #openstack-ironic | 09:22 | |
| sambetts | :) cool | 09:22 |
| *** fragatina has quit IRC | 09:24 | |
| *** fragatina has joined #openstack-ironic | 09:25 | |
| irf | for the command : "nova flavor-show baremetal", there is key called ram and in the extra_specs option also, there is a key called RAM. are they both different OR same. ? | 09:27 |
| *** dtantsur|afk is now known as dtantsur | 09:30 | |
| dtantsur | Morning Ironic | 09:30 |
| openstackgerrit | Sam Betts proposed openstack/ironic-python-agent: Remove "Experimental" warning from tinyipa README https://review.openstack.org/314929 | 09:31 |
| *** amotoki has quit IRC | 09:35 | |
| sambetts | huh... apprently gerrit bot doesn't output pushes to stable branches | 09:37 |
| dtantsur | correct | 09:39 |
| *** daemontool_ has quit IRC | 09:40 | |
| *** daemontool_ has joined #openstack-ironic | 09:40 | |
| *** daemontool_ has quit IRC | 09:40 | |
| dtantsur | sambetts, thanks for approving the name patch? | 09:47 |
| dtantsur | now, do we need something else to merge before the release? | 09:47 |
| sambetts | dtantsur: https://bugs.launchpad.net/ironic-inspector/+bug/1557979 can we mark this as done? | 09:49 |
| openstack | Launchpad bug 1557979 in Ironic Inspector "Prevent DHCP'ing when no introspection is going on and discovery is disabled" [High,In progress] - Assigned to Dmitry Tantsur (divius) | 09:49 |
| dtantsur | it has one more action item "The next step would be to cache all MAC's from the previous introspection runs, and use them for blacklist in additional to Ironic port list." | 09:49 |
| dtantsur | maybe we should split it away in an RFE of its own | 09:50 |
| dtantsur | sambetts, ^^ | 09:50 |
| sambetts | dtantsur: might be a good idea, because I would say that bug is pretty much fixed, and it would be good to mark it off for this release | 09:51 |
| dtantsur | sambetts, mind doing that please? I remember that was your request :) | 09:51 |
| sambetts | RFE created, I had to create the discription from memory so it might not be super accurate | 09:56 |
| irf | for the command : "nova flavor-show baremetal", there is key called ram and in the extra_specs option also, there is a key called RAM. are they both different OR same. ? | 09:57 |
| sambetts | dtantsur: I only see 3 high bugs in our list right now, 1 we've just marked as commited, 1 is incomplete and 1 is triaged, I would like to see the triaged one fixed as it should be a pretty quick merge as its just docs WDYT? | 10:03 |
| *** athomas has quit IRC | 10:03 | |
| dtantsur | sambetts, for the future: please mark finished bugs as "released". infra automation no longer closes them for us.. | 10:04 |
| dtantsur | especially since this bug is actually released (for liberty and mitaka) | 10:04 |
| sambetts | dtantsur: ah right :/ I thought they still marked them when we cut a release | 10:05 |
| dtantsur | aha, the last one is https://bugs.launchpad.net/ironic-inspector/+bug/1564238 right? sambetts wanna fix it or should I? I'm stuck in downstream bureaucracy right now :( | 10:05 |
| openstack | Launchpad bug 1564238 in Ironic Inspector "Documentation examples are wrong and misleading creation of DSL rules" [High,Triaged] | 10:05 |
| sambetts | dtantsur: yeah, I can knock up a quick patch for it | 10:05 |
| dtantsur | cheers! | 10:05 |
| sambetts | dtantsur: re: removing the experimental from tinyipa, shall I just remove the note entirely because its going to be hard to maintain? or do you think listing tested platforms is useful? | 10:07 |
| dtantsur | sambetts, as all major platforms are covered, I think it can be removed | 10:07 |
| openstackgerrit | Sam Betts proposed openstack/ironic-python-agent: Remove "Experimental" warning from tinyipa README https://review.openstack.org/314929 | 10:08 |
| sambetts | done ^ | 10:08 |
| *** athomas has joined #openstack-ironic | 10:08 | |
| *** hoangcx has quit IRC | 10:11 | |
| *** stevemar has quit IRC | 10:13 | |
| *** stevemar has joined #openstack-ironic | 10:15 | |
| *** daemontool has joined #openstack-ironic | 10:18 | |
| openstackgerrit | Merged openstack/ironic-inspector: Support Ironic node names in our API https://review.openstack.org/276331 | 10:28 |
| openstackgerrit | Sam Betts proposed openstack/ironic-inspector: Ensure rules documentation examples are valid JSON https://review.openstack.org/314953 | 10:29 |
| sambetts | dtantsur: ^^ | 10:29 |
| sambetts | mgould: we actually use gdisk and sgdisk | 10:32 |
| mgould | sambetts, cool | 10:32 |
| mgould | maybe just copy them, then - how large are they? | 10:33 |
| sambetts | less than 200k each | 10:34 |
| mgould | OK, probably not worth bothering :-) | 10:34 |
| sambetts | it was ICU that was 25mb+ that I wanted to nuke | 10:34 |
| mgould | I suppose we're also copying cgdisk and fixparts across, so that's 400k | 10:34 |
| mgould | yeah, not worth the effort | 10:35 |
| mgould | sambetts, what are boot times like now? | 10:35 |
| sambetts | actually didn't check yesterday because I only managed to get my devstack up and running sucessfully at 22:00 :( I was very excited that I managed to get it to run with only 256mb of RAM though! | 10:36 |
| mgould | sambetts, yikes | 10:36 |
| * mgould +1s a sambetts patch for the third time this morning :-) | 10:37 | |
| sambetts | heh | 10:38 |
| *** MattMan has quit IRC | 10:42 | |
| *** MattMan has joined #openstack-ironic | 10:42 | |
| *** daemontool has quit IRC | 10:48 | |
| aarefiev | morning all! | 10:54 |
| sambetts | hey aarefiev | 10:55 |
| aarefiev | sambetts: hey Sam, how was your way back? | 10:56 |
| sambetts | aarefiev: pretty good thanks :D yours? | 10:56 |
| aarefiev | very long ) | 10:56 |
| vsaienko | morning aarefiev | 10:58 |
| aarefiev | vsaienko: morning | 10:59 |
| sambetts | aarefiev: Yeah I bet! How much changes did you have to make? | 10:59 |
| *** jistr is now known as jistr|mtg | 11:00 | |
| sambetts | mgould: oh nice they actually listened on the tinycore foru :-P | 11:00 |
| sambetts | formum | 11:00 |
| sambetts | forum... | 11:00 |
| aarefiev | sambetts: there was 3 planes and then train | 11:02 |
| sambetts | mgould: with the 38mb ramdisk @ 256mb of RAM power on to resume took 46 seconds | 11:02 |
| sambetts | aarefiev: :O wow!! | 11:03 |
| aarefiev | sambetts: yeah, thanks for fixing json mistakes, I was going to do this after PTO | 11:05 |
| *** dprince has joined #openstack-ironic | 11:06 | |
| *** stendulker has quit IRC | 11:06 | |
| *** irf has quit IRC | 11:06 | |
| *** amotoki has joined #openstack-ironic | 11:07 | |
| sambetts | :) thats ok, dtantsur and I want to make a inspector release, and I wanted to get the high priotry bugs done | 11:08 |
| *** baoli has joined #openstack-ironic | 11:10 | |
| jroll | morning y'all | 11:11 |
| sambetts | hey jroll | 11:11 |
| aarefiev | morning jroll | 11:12 |
| *** baoli_ has joined #openstack-ironic | 11:12 | |
| *** baoli has quit IRC | 11:16 | |
| *** baoli_ has quit IRC | 11:21 | |
| *** baoli has joined #openstack-ironic | 11:22 | |
| *** baoli has quit IRC | 11:25 | |
| *** baoli has joined #openstack-ironic | 11:25 | |
| dtantsur | morning jroll, aarefiev | 11:31 |
| dtantsur | aarefiev, 3 planes omg.. it's worth than we us for sure | 11:31 |
| *** sacharya has joined #openstack-ironic | 11:31 | |
| lucasagomes | aarefiev, jroll sambetts good morning | 11:31 |
| dtantsur | jroll, should I say that we remove the bash ramdisk support around right now? | 11:31 |
| dtantsur | or do we want some period after the announcement? | 11:32 |
| aarefiev | morning dtantsur lucasagomes | 11:32 |
| jroll | dtantsur: we already announced the deprecation a long time ago, right? | 11:32 |
| jroll | dtantsur: I'm inclined to say "near future" | 11:32 |
| dtantsur | yeah, just checking | 11:32 |
| *** sacharya has quit IRC | 11:37 | |
| * lucasagomes released ironic-staging-drivers 0.2.0, should I send an mail about it!? | 11:40 | |
| jroll | lucasagomes: do you do the releases repo thing? | 11:41 |
| openstackgerrit | Merged openstack/ironic-inspector: Ensure rules documentation examples are valid JSON https://review.openstack.org/314953 | 11:41 |
| jroll | if not, yeah you should email, not sure if openstack-announce or openstack-dev though | 11:42 |
| lucasagomes | jroll, nop, pushed a tag and uploded to pypi directly | 11:42 |
| *** ChrisAusten has quit IRC | 11:45 | |
| lucasagomes | jroll, but that's an interesting thing, I think I can not use the release project since I'm not in openstack/governance, correct? | 11:46 |
| dtantsur | #openstack-release is a better place to ask | 11:46 |
| jroll | lucasagomes: I think you can, honestly, but yeah what he said :) | 11:46 |
| lucasagomes | doing that | 11:46 |
| dtantsur | also are we ready for an ironic-inspector release now? | 11:47 |
| sambetts | dtantsur: as far as I'm aware, the only other high priority bug is marked incomplete so I say lets go for it | 11:49 |
| mat128 | morning mgould jroll sambetts aarefiev dtantsur lucasagomes | 11:49 |
| lucasagomes | mat128, hello there, good morning | 11:50 |
| dtantsur | morning mat128 | 11:50 |
| mat128 | sambetts: http://forum.tinycorelinux.net/index.php/topic,19951.0.html < the guy fixed it but in 7.x | 11:50 |
| mgould | morning mat128 | 11:50 |
| mgould | mat128, yeah, I left a comment to that effect on the patch | 11:50 |
| dtantsur | time to switch to 7.x? | 11:50 |
| dtantsur | also morning mat128 | 11:50 |
| mgould | +1 for switching | 11:50 |
| mat128 | "why not?" comes to mind :) | 11:50 |
| sambetts | hehe, I'm sure something went wrong when I tried changing it before :/ | 11:51 |
| * lucasagomes goes grab some lunch | 11:52 | |
| *** lucasagomes is now known as lucas-hungry | 11:52 | |
| mat128 | trying a 7.x build locally | 11:52 |
| mat128 | we'll see | 11:52 |
| sambetts | also trying :-P | 11:53 |
| sambetts | mat128: ah missing dependencies in 7.x | 11:54 |
| mat128 | some of the things we require are simply missing? | 11:55 |
| sambetts | hmm, I think it might be a mistake on my end, let me try something | 11:56 |
| *** ChanServ changes topic to "Bare Metal Provisioning | Status: http://bit.ly/ironic-whiteboard | Docs: http://docs.openstack.org/developer/ironic/ | Bugs: https://bugs.launchpad.net/ironic" | 11:57 | |
| -openstackstatus- NOTICE: We have a workaround for our mirrors to attempt to translate package names if a match isn't immediately obvious. A more complete fix is yet to come. It is now safe to 'recheck' any jobs that failed due to "No matching distribution found". Please join #openstack-infra if you discover more problems. | 11:57 | |
| dtantsur | jroll, a friendly reminder about https://review.openstack.org/#/c/314713/ thanks! :) | 11:58 |
| jroll | dtantsur: ah, right :) | 11:58 |
| mat128 | sambetts: 404 on raid-dm-3.16.6-tinycore64.tcz | 11:59 |
| dtantsur | sounds familiar, the last time it was due to a wrong kernel version detected | 11:59 |
| sambetts | mat128: thats were I failed, but thats because its a different kernel version so I have to change the deps 3.16.6 to 4.6.9 | 11:59 |
| sambetts | 4.2.9 * | 12:00 |
| mat128 | ok good | 12:00 |
| mat128 | scsi-3.16.6-tinycore64.tcz will probably also crash | 12:00 |
| jroll | dtantsur: +A | 12:01 |
| dtantsur | \o/ | 12:01 |
| dtantsur | thanks | 12:01 |
| sambetts | mat128: yup I changed both, hmm make build for tgt failed with a segfault :/ | 12:01 |
| dtantsur | wow awesome | 12:02 |
| jroll | dtantsur: wanna land https://review.openstack.org/#/c/314662/1 ? | 12:02 |
| dtantsur | sambetts, do we need tgt? does it support LIO instead? | 12:02 |
| dtantsur | jroll, yeah, yesterday I had a question about the failed tinyipa job. looks weird and unrelated to the patch, just checking with you. | 12:03 |
| jroll | dtantsur: yeah, left a note there as to why | 12:03 |
| jroll | turns out the code to download tinyipa doesn't exist in mitaka | 12:03 |
| sambetts | dtantsur: so in 6.x LIO didn't work for some reason, something to do with missing kernel modules :/ not sure if it'll work in 7.x | 12:04 |
| jroll | so BUILD_RAMDISK=False just goes straight to downloading coreos | 12:04 |
| dtantsur | jroll, I really wonder if we should move our devstack plugins to a separate branchless repo... | 12:04 |
| jroll | mmm | 12:04 |
| jroll | but devstack is branched :| | 12:04 |
| dtantsur | let it be | 12:04 |
| * sambetts was wondering the same this morning, however won't we end up with a bunch of conditionals ? | 12:04 | |
| jroll | now tempest plugin, probably yes, sounds like painful work though | 12:04 |
| jroll | dtantsur: think of all the conditionals for config changes | 12:04 |
| dtantsur | after cloning of the ironic repo we can source a file with more stuff | 12:05 |
| dtantsur | just thinking aloud though | 12:05 |
| sambetts | jroll: BTW https://review.openstack.org/#/c/314933/ | 12:05 |
| jroll | sambetts: why does that depend on a readme change... | 12:06 |
| mat128 | sambetts: didnt we have this in before we cut the mitaka branch? | 12:06 |
| jroll | mat128: nope :( | 12:06 |
| sambetts | jroll: because when that merges it'll trigger the post job to build a version of tinyipa on stable/mitaka | 12:06 |
| jroll | sambetts: I just approved another mitaka change so that should handle it | 12:07 |
| sambetts | jroll: oh nice :) cool, I can remove the deps then | 12:07 |
| jroll | I'm not sure I see the use in backporting a readme change | 12:07 |
| jroll | also, the job is broken until we publish a mitaka image anyway, so might as well just land it | 12:08 |
| sambetts | jroll: makes sense, I've removed the depends | 12:08 |
| *** raildo-afk is now known as raildo | 12:08 | |
| jroll | sambetts: one more comment then lgtm | 12:10 |
| *** rbudden has joined #openstack-ironic | 12:10 | |
| *** jistr|mtg is now known as jistr | 12:11 | |
| mat128 | sambetts: netfilter-3.16.6 because of fakeuname | 12:12 |
| sambetts | done ;) | 12:12 |
| sambetts | mat128: good point... | 12:13 |
| mat128 | that was in the final stage | 12:13 |
| mat128 | I should have a build soon | 12:13 |
| mat128 | it'll go though our internal tests | 12:13 |
| mat128 | basically expecting a lookup + heartbeat and making sure we can talk to IPA | 12:14 |
| mat128 | repo.tinycorelinux.net is slow :( | 12:14 |
| sambetts | mat128: yeah I was going to run the ironic tempest tests against it | 12:14 |
| mat128 | I wonder if we should mirror this somewhere | 12:14 |
| dtantsur | we should, that's the big problem | 12:14 |
| *** rbudden has quit IRC | 12:15 | |
| sambetts | mat128: I tend to use the distro.ibibilo.org mirror | 12:15 |
| mat128 | sambetts: yeah even better :) ours is just a small 5 min gate to ensure binaries that come out of the project are worthy of being tested in our big test | 12:15 |
| dtantsur | everything used in gate should be mirrored. we're not there with the coreos builder though | 12:15 |
| sambetts | mat128: :/ I can't get past building tgt it seg faults on me every time | 12:15 |
| mat128 | sambetts: how do you force tce-load to use a mirror of your choice? | 12:16 |
| mat128 | dtantsur: how does one go and make this mirrored? work with -infra? | 12:16 |
| milan | aarefiev, hi, would you mind me rebasing the tempest testing for inspector? | 12:16 |
| * milan would like to contribute here | 12:16 | |
| dtantsur | mat128, yes | 12:16 |
| sambetts | mat128: ooooh for that not sure, I thought you meant in the browser, I think there is a file that used to list the mirrors or something :/ | 12:16 |
| mat128 | sambetts: I see we're pulling the base kernel/ramdisk from ibiblio.org, but have no control over the rest | 12:17 |
| * milan was known as mkovTABTABTAB before, aarefiev ;) | 12:18 | |
| aarefiev | milan: hey, yeah I get it | 12:19 |
| openstackgerrit | Mathieu Mitchell proposed openstack/ironic-python-agent: Use TinyCore Linux 7.x for TinyIPA https://review.openstack.org/314999 | 12:19 |
| aarefiev | milan: I was going to rebase it today, but I don't mind :) less work for me | 12:19 |
| mat128 | sambetts: ^ we'll see what the gate thinks | 12:19 |
| mat128 | segfault or not, and whether it passes the other gates | 12:20 |
| milan | aarefiev, cool :) | 12:20 |
| * milan will do | 12:20 | |
| mat128 | sambetts: it's gzipping right now on my box | 12:20 |
| dtantsur | jroll, I see that ironic projects don't have stable:follows-policy in governance. should we add one? | 12:21 |
| aarefiev | milan: you made rebranding :) | 12:21 |
| dtantsur | at least for ironic, inspector and both clients? | 12:21 |
| jroll | dtantsur: it's in progress: https://review.openstack.org/#/c/310288/ | 12:21 |
| milan | aarefiev, correct :D | 12:21 |
| aarefiev | milan: cool, thanks a lot | 12:21 |
| dtantsur | awesome | 12:21 |
| milan | aarefiev, well, I've not changed the company, just the nick upstream ;) | 12:22 |
| aarefiev | lol | 12:22 |
| dtantsur | jroll, hmm, does inspector need any kind of periodic jobs for that? | 12:22 |
| jroll | dtantsur: possibly. probably. tonyb is reviewing what we have now and we'll work to improve it if needed | 12:23 |
| mat128 | sambetts: mucho success [exec] 172.27.59.12 - - [11/May/2016:12:23:33 +0000] "POST /v1/nodes/deadbeef-dead-beef-dead-beefdeadbeef/vendor_passthru/heartbeat HTTP/1.1" 404 560 "-" "python-requests/2.10.0" | 12:24 |
| sambetts | mat128: oh nice! | 12:24 |
| dtantsur | jroll, cool, please keep me in the loop | 12:24 |
| jroll | dtantsur: totes | 12:24 |
| sambetts | mat128: no idea why my machine is segfaulting out :'( | 12:24 |
| mat128 | enough memory? | 12:24 |
| sambetts | mat128: 8g should be plenty | 12:25 |
| milan | sambetts, try turning it off and on again :P | 12:25 |
| *** nicodemos has joined #openstack-ironic | 12:26 | |
| sambetts | haha | 12:26 |
| sambetts | mat128: it works fine outside of the chroot, but inside it doesn't like it at all | 12:26 |
| sambetts | and it doesn't give me any reasoning behind it either :/ | 12:28 |
| mat128 | sambetts: maybe compare logs? http://paste.openstack.org/show/496708/ | 12:29 |
| sambetts | hmmmm, it appears that its make thats broken | 12:29 |
| sambetts | if I manaully chroot into the build dir, and just in / run make --version it fails | 12:30 |
| sambetts | Seg fault | 12:30 |
| openstackgerrit | Milan Kováčik proposed openstack/ironic-inspector: [WIP] Tempest: add basic test https://review.openstack.org/293926 | 12:30 |
| *** jjohnson2 has joined #openstack-ironic | 12:30 | |
| mat128 | sambetts: for the record, it built a 53M ramdisk | 12:30 |
| mat128 | sambetts: ouch | 12:31 |
| sambetts | I wonder if its a incompatiblity thing with my host's kernel? on your host whats the uname output? | 12:31 |
| sambetts | uname -a | 12:31 |
| mat128 | stock ubuntu 14.04: Linux default 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | 12:31 |
| sambetts | mat128: I'm running debian testing, Linux ArticalVirtualJessie 4.4.0-1-amd64 #1 SMP Debian 4.4.6-1 (2016-03-17) x86_64 GNU/Linux | 12:31 |
| sambetts | much newer kernel :/ | 12:31 |
| mat128 | I can pretty easily that that on 16.04 | 12:32 |
| mat128 | s/that/try/ | 12:32 |
| *** openstackgerrit has quit IRC | 12:33 | |
| mat128 | it's a one byte change in downstream build script :) | 12:33 |
| sambetts | heh awesome :D | 12:33 |
| mat128 | we use ephemeral instances to build stuff | 12:33 |
| mat128 | here we go | 12:33 |
| *** openstackgerrit has joined #openstack-ironic | 12:34 | |
| *** xavierr_pizza is now known as xavierr | 12:34 | |
| *** ayoung has quit IRC | 12:36 | |
| dtantsur | jroll, please remind me what we agreed on about the DIB job. Just kill it? | 12:38 |
| dtantsur | (I mean, our job on DIB, which is not voting anyway) | 12:38 |
| jroll | dtantsur: yes | 12:39 |
| dtantsur | k | 12:40 |
| *** wajdi has quit IRC | 12:41 | |
| *** irf has joined #openstack-ironic | 12:42 | |
| dtantsur | jroll, do you want to make the last release of ironic before removing things? we're doing it for inspector. | 12:42 |
| *** aparnav has quit IRC | 12:46 | |
| dtantsur | jroll, he's my draft email if you want to make any changes: https://etherpad.openstack.org/p/dPsYfGymFd | 12:46 |
| jroll | dtantsur: I was thinking the other way, drop this code as part of 6.0.0 | 12:46 |
| mat128 | sambetts: I'm also trying the same 14.04 box that built it with success using the Xenial kernel, which is 4.4.0 | 12:46 |
| jroll | dtantsur: because people don't like many major versions in one cycle, apparently | 12:46 |
| mat128 | more data, more science :) | 12:46 |
| sambetts | mat128: :D | 12:46 |
| dtantsur | jroll, so, the current release, right? | 12:46 |
| jroll | dtantsur: right, drop it before release | 12:47 |
| jroll | 2. Remove the gate-tempest-dsvm-ironic-pxe_ssh job from Ironic master only. | 12:47 |
| jroll | is interesting | 12:47 |
| jroll | I think we'll just have to make it a noop on master, we'll need to bug infra on the best route for that | 12:47 |
| dtantsur | jroll, it's possible to remove a job from a branch | 12:48 |
| jroll | orly? nice | 12:48 |
| dtantsur | we do it all the time actually | 12:48 |
| sambetts | I think we can just make it so that it doesn't run on master right? e.g. in the same way we prevent jobs running on older branches | 12:48 |
| dtantsur | e.g. tinyipa jobs on liberty | 12:48 |
| dtantsur | e.g. https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L1281 | 12:49 |
| *** lucas-hungry is now known as lucasagomes | 12:49 | |
| jroll | ah, cool | 12:49 |
| jroll | thanks | 12:49 |
| jroll | dtantsur: email looks fine to me btw | 12:50 |
| mat128 | sambetts: segfaulted on 14.04 running xenial 4.4.0-22 | 12:50 |
| sambetts | mat128: :'( | 12:50 |
| mat128 | worked on 16.04 (!) | 12:50 |
| sambetts | what?! | 12:50 |
| dtantsur | jroll, awesome, sending now. should I include someone but openstack-dev? | 12:50 |
| *** daemontool has joined #openstack-ironic | 12:51 | |
| sambetts | mat128: btw if we want to change the mirror we use for tce-load, you just need to replace the URL in /opt/tcemirror | 12:51 |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack/ironic: Update devstack plugin with new auth options https://review.openstack.org/297312 | 12:51 |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack/ironic: Migrate to using keystoneauth Sessions https://review.openstack.org/236982 | 12:51 |
| mat128 | oh, good | 12:51 |
| mat128 | I'll push a change to use the same mirror for both kernel/ramdisk initial download and build | 12:51 |
| mat128 | then we can bug infra and change it in one place | 12:52 |
| *** links has quit IRC | 12:52 | |
| jroll | dtantsur: openstack-operators@ | 12:52 |
| dtantsur | jroll, I don't think I'm subscribed to it, maybe you could sent? | 12:52 |
| jroll | dtantsur: sure | 12:53 |
| dtantsur | cheers! | 12:53 |
| *** daemontool has quit IRC | 12:53 | |
| *** daemontool has joined #openstack-ironic | 12:53 | |
| sambetts | mat128: https://review.openstack.org/#/c/313675/ btw this is tinycore 6.x with manual build of gdisk and results in a 38mb version | 12:54 |
| jroll | dtantsur: done, thanks for writing it | 12:54 |
| sambetts | mat128: at that size it boots successfully with 256mb of RAM | 12:54 |
| dtantsur | jroll, so, wait a couple of days in case someone complains too much, then I proceed with #1 and #2? | 12:55 |
| mat128 | sambetts: and removing Babel and everything else, good stuff! | 12:56 |
| mat128 | mine was 53M without all of these changes | 12:56 |
| mat128 | but with a smaller gdisk directly from upstream | 12:56 |
| jroll | dtantsur: yeah, and even if someone does complain, probably proceed anyway :) | 12:56 |
| mat128 | do you think 7.x is larger than 6.x? | 12:56 |
| dtantsur | hehe, good | 12:57 |
| jroll | http://jroll.ghost.io/ heh | 12:57 |
| jroll | "Our network is literally on fire" | 12:57 |
| sambetts | mat128: not sure :/ if I can get 7.x working on my local machine then I planned to rebase your 7.x change ontop of those ones | 12:57 |
| lucasagomes | jroll, heh wow | 12:57 |
| sambetts | jroll: hehe | 12:58 |
| lucasagomes | jroll, btw what happened with your old domain? jimrollenhagen.com | 12:58 |
| *** Goneri has joined #openstack-ironic | 12:58 | |
| *** irf has quit IRC | 12:58 | |
| *** mgould has quit IRC | 12:58 | |
| mat128 | sambetts: 16.04 native http://paste.openstack.org/show/496717/ 14.04 with upgraded 16.04 kernel http://paste.openstack.org/show/496718/ | 12:59 |
| jroll | lucasagomes: some HSTS configuration mistakes made the link to their hosting not work right :) | 12:59 |
| jroll | lucasagomes: I'll fix it one day | 12:59 |
| lucasagomes | fair enough | 12:59 |
| jroll | I did choose ghost because you can run it yourself, just need to do it | 12:59 |
| mat128 | not much tools in the chroot | 13:00 |
| lucasagomes | yeah i remember we talked a little about ghost | 13:00 |
| lucasagomes | looks pretty good indeed | 13:00 |
| *** mgould has joined #openstack-ironic | 13:00 | |
| *** milan_ has joined #openstack-ironic | 13:00 | |
| *** milan has quit IRC | 13:03 | |
| sambetts | mat128: just upgrading my machine, and I'll see if it was a bug in that specifc kernel version or something, because its bumping me to 4.5 | 13:04 |
| mat128 | ok, we'll see | 13:05 |
| mat128 | I'll have a hard time reproducing issues with 4.5, but at least I'm getting it with 4.4.0-22-generic (but not 4.4.0-21-generic?!) | 13:05 |
| mat128 | maybe it's not just the kernel | 13:05 |
| mat128 | even make clean segfaults | 13:06 |
| mat128 | but not make --help :( | 13:06 |
| *** amotoki has quit IRC | 13:08 | |
| *** Nakato has quit IRC | 13:08 | |
| *** Nakato has joined #openstack-ironic | 13:08 | |
| sambetts | yeah make --version does too | 13:09 |
| vdrok | morning everyone | 13:10 |
| vdrok | jroll: we have a bug in resource tracker I think. please take a look http://paste.openstack.org/show/496718/. test_create_server setUpClass works fine, and hypervisor details are correct too, next 2 tests setup fails because all hypervisors have 0 vcpus, memory_mb, ramfilter throws all nodes out | 13:12 |
| vdrok | while all the nodes are in available state | 13:12 |
| *** rloo has joined #openstack-ironic | 13:12 | |
| vdrok | jroll: err, http://paste.openstack.org/show/496722 | 13:13 |
| jroll | vdrok: you sure you don't just need to wait for the resource tracker to run again? | 13:15 |
| jroll | I also see: "free_ram_mb": 384 | 13:15 |
| jroll | a few times in both calls | 13:15 |
| vdrok | jroll: idk. I have 7 nodes. In the test_create_server one instance is booted, after it is deleted, all the nodes are updated, with 0s | 13:16 |
| * jroll has to step away for a while | 13:16 | |
| sambetts | myself and mgould saw a something that sounds similar the other day, we had 7 nodes availible and set 1 to maintenace = True and that removed all the nodes from being schedularble | 13:16 |
| jroll | vdrok: wait for it to get to available, then wait for resource tracker to run | 13:16 |
| jroll | sambetts: wat | 13:16 |
| vdrok | sambetts: nope, all are not in maintenance and in available state | 13:17 |
| jroll | anyway, bbiab | 13:17 |
| sambetts | jroll: yeah, :/ | 13:17 |
| *** milan_ has quit IRC | 13:17 | |
| mgould | sambetts, there was something about that in the channel this morning IIRC | 13:17 |
| mgould | some line in ironic_virt_driver that explicitly set everything to zero | 13:17 |
| *** milan_ has joined #openstack-ironic | 13:18 | |
| *** mgould is now known as mgould|brb | 13:18 | |
| sambetts | yeah, it should do it for the node in maintenance not the all of them XD | 13:18 |
| *** mkovacik__ has joined #openstack-ironic | 13:19 | |
| sambetts | mat128: 4.5.0-2-amd64 fails too :'( | 13:20 |
| *** caiobo has joined #openstack-ironic | 13:21 | |
| mat128 | I have a core dump, but I'm not too familiar with troubleshooting this | 13:21 |
| mat128 | sambetts: crap :( | 13:21 |
| *** milan_ has quit IRC | 13:22 | |
| *** mkovacik__ is now known as milan | 13:24 | |
| caiobo | good morning Ironic! :) | 13:25 |
| *** mgould|brb is now known as mgould | 13:26 | |
| * mgould returns from the third office barbecue of the year | 13:26 | |
| mgould | what is this strange country, and what has it done with the real Scotland? | 13:27 |
| *** hamzy has quit IRC | 13:29 | |
| *** hamzy has joined #openstack-ironic | 13:29 | |
| sambetts | mgould: ha! | 13:30 |
| *** vmud213 has quit IRC | 13:31 | |
| *** afaranha has joined #openstack-ironic | 13:33 | |
| *** daemontool has quit IRC | 13:34 | |
| dtantsur | meh, the coreos builder is so slow compared to the tinyipa one... | 13:34 |
| sambetts | mat128: hmmm, there seems to be no consistancy in the v7.x tcz repo, things have been compile for 4.2.9 or 4.2.7 and the linux headers package they provide is 4.2.1 :/ no sure if that might be contributing to the crazyness | 13:35 |
| sambetts | dtantsur: hehe | 13:35 |
| xavierr | good morning caiobo | 13:37 |
| xavierr | and good morning Ironic | 13:38 |
| mgould | morning caiobo xavierr | 13:38 |
| *** chlong has joined #openstack-ironic | 13:38 | |
| mat128 | sambetts: maybe we can switch to 4.2.7 just to check? | 13:39 |
| caiobo | morning xavierr sambetts mgould dtantsur and everybody else too | 13:39 |
| dtantsur | morning xavierr, caiobo | 13:40 |
| sambetts | mat128: giving it a go | 13:40 |
| lucasagomes | xavierr, morning | 13:41 |
| lucasagomes | caiobo, morning dude :-) | 13:41 |
| caiobo | morning lucasagomes! :D how are you dude? | 13:44 |
| sambetts | mat128: switched all the deps etc over to 4.2.7 and it still fails | 13:45 |
| *** absubram has quit IRC | 13:45 | |
| lucasagomes | caiobo, all is well, you? | 13:45 |
| NobodyCam | Good Morning Ironicers | 13:46 |
| mgould | morning NobodyCam | 13:46 |
| xavierr | morning lucasagomes, what's the craic? hahaha | 13:47 |
| xavierr | morning dtantsur NobodyCam :) | 13:47 |
| dtantsur | morning NobodyCam | 13:48 |
| lucasagomes | xavierr, not much craic today, just internal stuff + meetings. Hope to have some now in the afternoon :-) | 13:48 |
| openstackgerrit | Mario Villaplana proposed openstack/ironic: Add notification base classes and docs https://review.openstack.org/298461 | 13:48 |
| openstackgerrit | Dmitry Tantsur proposed openstack/python-ironic-inspector-client: Bump supported API version to 1.5 https://review.openstack.org/315047 | 13:48 |
| NobodyCam | morning mgould xavierr dtantsur lucasagomes sambetts jroll | 13:49 |
| caiobo | morning NobodyCam | 13:49 |
| sambetts | o/ NobodyCam | 13:49 |
| lucasagomes | fwiw craic = fun (in irish) it's not crack! | 13:49 |
| NobodyCam | morning caiobo :) | 13:50 |
| NobodyCam | :) | 13:50 |
| caiobo | lucasagomes same here :D | 13:50 |
| xavierr | lucasagomes, thats good. +1 for the explanation :) | 13:52 |
| openstackgerrit | Mario Villaplana proposed openstack/ironic: Regenerate sample configuration https://review.openstack.org/315052 | 13:55 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/bifrost: Updated from global requirements https://review.openstack.org/313712 | 13:56 |
| *** appu103_ has joined #openstack-ironic | 13:56 | |
| *** ametts has joined #openstack-ironic | 13:56 | |
| *** daemontool has joined #openstack-ironic | 13:58 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ironic: Updated from global requirements https://review.openstack.org/315057 | 13:58 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ironic-python-agent: Updated from global requirements https://review.openstack.org/314354 | 13:58 |
| *** [1]cdearborn has joined #openstack-ironic | 13:59 | |
| mariojv | i wonder if there's a way to get the proposal bot to automatically regenerate the sample config for us | 14:00 |
| *** ayoung has joined #openstack-ironic | 14:01 | |
| *** links has joined #openstack-ironic | 14:01 | |
| *** joprovost has joined #openstack-ironic | 14:04 | |
| *** appu103_ has quit IRC | 14:05 | |
| *** appu103_ has joined #openstack-ironic | 14:07 | |
| *** rloo has quit IRC | 14:07 | |
| *** rloo has joined #openstack-ironic | 14:08 | |
| *** ijw has joined #openstack-ironic | 14:08 | |
| *** wajdi has joined #openstack-ironic | 14:11 | |
| *** daemontool has quit IRC | 14:13 | |
| mariojv | #openstack-infra is saying there's a sphinx extension that does it automatically for you | 14:17 |
| jroll | that's a different thing, and we need to get on oslo's config generator to do that | 14:18 |
| jroll | that won't do anything with the in-tree sample config (but we could drop it once we're publishing) | 14:18 |
| jroll | the configgen thing is here: https://review.openstack.org/#/c/247331/ | 14:19 |
| mariojv | ok, thanks | 14:20 |
| vdrok | jroll: so the problem is the following - I have 7 nodes, I do nova boot (single instance), instance is scheduled to a node, node starts deploying, and ALL the nova hypervisors are updated with vcpu=0, memory_mb=0, local_gb=0. that node becomes active, and after some time (periodic task starts) these values are back to normal. that's not OK or am I missing | 14:23 |
| vdrok | something? | 14:23 |
| *** appu103_ has quit IRC | 14:23 | |
| sambetts | vdrok: no, thats not cool... | 14:23 |
| jroll | vdrok: yeah something is totally wrong there | 14:23 |
| vdrok | that's just the nova boot, no tempest | 14:24 |
| vdrok | and it is the reason everything fails in tempest smoke | 14:24 |
| *** appu103_ has joined #openstack-ironic | 14:24 | |
| jroll | so interestingly enough, smoke passes for me | 14:24 |
| *** joprovost1 has joined #openstack-ironic | 14:24 | |
| vdrok | heh | 14:25 |
| *** joprovost has quit IRC | 14:25 | |
| *** joprovost1 is now known as joprovost | 14:25 | |
| jroll | http://logs.openstack.org/65/311865/3/experimental/gate-grenade-dsvm-ironic/e635dec/logs/grenade.sh.txt.gz | 14:25 |
| jroll | are you using CCM vdrok ? | 14:25 |
| vdrok | jroll: nope, it is the default compute manager | 14:25 |
| jroll | O_o | 14:25 |
| *** joprovost has quit IRC | 14:25 | |
| jroll | maybe something new breaking it then? | 14:25 |
| jroll | new since yesterday | 14:26 |
| sambetts | jroll: this really sounds like its related to the thing me and mgould were seeing :/ | 14:26 |
| *** joprovost has joined #openstack-ironic | 14:26 | |
| sambetts | basically 1 node affecting the state of all the nodes | 14:26 |
| jroll | not much though https://github.com/openstack/nova/commits/master | 14:26 |
| jroll | right | 14:26 |
| jroll | that's... odd | 14:26 |
| dtantsur | there was one patch changing tracking resources | 14:27 |
| dtantsur | can't find it though | 14:27 |
| *** wajdi has quit IRC | 14:27 | |
| jroll | oooo | 14:27 |
| jroll | I know the one | 14:27 |
| *** joprovost has quit IRC | 14:27 | |
| *** wajdi has joined #openstack-ironic | 14:27 | |
| *** joprovost has joined #openstack-ironic | 14:27 | |
| dtantsur | https://github.com/openstack/nova/commit/016b810f675b20e8ce78f4c82dc9c679c0162b7a ? | 14:27 |
| jroll | https://review.openstack.org/#/c/306670/ | 14:27 |
| sambetts | I was seeing this happen last week, I thought it was just a problem with my env though | 14:27 |
| dtantsur | right :) | 14:27 |
| jroll | vdrok: sambetts: try with that reverted just for kicks? | 14:28 |
| jroll | I can't imagine this is it though | 14:28 |
| *** rbudden has joined #openstack-ironic | 14:28 | |
| *** joprovost has quit IRC | 14:28 | |
| jroll | max(0, free) | 14:28 |
| *** joprovost has joined #openstack-ironic | 14:28 | |
| vdrok | jroll: yep, I guess pas-ha mentioned this happening even earlier, before that was merged, but i'll try | 14:29 |
| jroll | vdrok: oh, weird | 14:29 |
| *** amotoki has joined #openstack-ironic | 14:30 | |
| *** xhku has joined #openstack-ironic | 14:30 | |
| *** wajdi_ has joined #openstack-ironic | 14:30 | |
| *** daemontool has joined #openstack-ironic | 14:31 | |
| *** wajdi has quit IRC | 14:33 | |
| lucasagomes | jroll, vdrok I wonder if the tempest test could/should check the hypervisor-stats once the node is being deployed | 14:34 |
| lucasagomes | making sure the right amount of resources was subtracted from there | 14:34 |
| jroll | yeah, it should | 14:34 |
| lucasagomes | that might prevent/capture patches doing weird stuff with the resource tracking in nova | 14:34 |
| * lucasagomes will check it out | 14:36 | |
| vdrok | lucasagomes: the problem is, eg vcpus sometimes switches to 0 and then back to 1, without any actions from me, just noticed that. %) | 14:41 |
| lucasagomes | vdrok, yup, I mean, once that problem is fixed | 14:41 |
| vdrok | ah, ++ | 14:41 |
| *** daemontool has quit IRC | 14:45 | |
| openstackgerrit | Aparna proposed openstack/proliantutils: Adds support in hpssa for SDD interface 'Solid State SAS' https://review.openstack.org/311713 | 14:47 |
| vdrok | jroll: still happens after revert | 14:48 |
| wajdi_ | good morning folks. | 14:48 |
| *** wajdi_ is now known as wajdi | 14:48 | |
| vdrok | morning wajdi | 14:48 |
| jroll | vdrok: cool, good data | 14:49 |
| *** e0ne has quit IRC | 14:49 | |
| *** ijw has quit IRC | 14:52 | |
| *** ijw has joined #openstack-ironic | 14:52 | |
| *** links has quit IRC | 14:54 | |
| rloo | vdrok, lucasagomes, sambetts etc. I doubt this has anything to do with it but thought I'd mention it, this is the ClusteredComputeManager and I don't think you're using it: https://review.openstack.org/#/c/287498/ | 14:54 |
| lucasagomes | rloo, yeah apparently vdrok is not using it | 14:55 |
| lucasagomes | (see jroll's q about CCM there) | 14:55 |
| rloo | lucasagomes: good. wanted to be sure. i don't see how that code could break something but one never knows. | 14:55 |
| lucasagomes | heh ++ | 14:55 |
| *** daemontool has joined #openstack-ironic | 14:55 | |
| vdrok | rloo: morning, yep, I'm on the default one | 14:55 |
| rloo | vdrok and everyone else. Hi and good morning! | 14:56 |
| lucasagomes | morning rloo | 14:56 |
| NobodyCam | morning vdrok, rloo and wajdi | 14:56 |
| vdrok | morning NobodyCam :) | 14:56 |
| NobodyCam | :) | 14:57 |
| *** mtanino has joined #openstack-ironic | 14:58 | |
| pas-ha | vdrok, jroll bug I reported in Nova on memory_mb https://bugs.launchpad.net/nova/+bug/1572555 | 14:58 |
| openstack | Launchpad bug 1572555 in OpenStack Compute (nova) "Nova reports memory_mb=0 for available Ironic node" [Undecided,New] | 14:58 |
| openstackgerrit | Caio Oliveira proposed openstack/python-oneviewclient: Adding dynamic allocation operations https://review.openstack.org/258699 | 15:00 |
| lucasagomes | pas-ha, interesting | 15:00 |
| lucasagomes | wondering if the ironic virt driver is reporting it wrong somehow | 15:00 |
| pas-ha | the fun/strange part is it was enough to bring 2 of 3 nodes to maintenance to bring 0 on actually available node o_O | 15:01 |
| *** electrofelix has quit IRC | 15:01 | |
| lucasagomes | pas-ha, what if you delete the two other nodes? | 15:02 |
| *** absubram has joined #openstack-ironic | 15:03 | |
| vdrok | pas-ha: well, yeah, the problem I see right now, do the cleaning on one node, and all the others report 0 | 15:03 |
| pas-ha | did not tried it, will do | 15:03 |
| jroll | sambetts: morgabra: I need your eyes on this as far as ironic impact: https://review.openstack.org/#/c/313001/1/specs/newton/approved/prep-for-network-aware-scheduling.rst | 15:03 |
| pas-ha | vdrok: hmm.. seems that virt driver somehow takes a combined view on its resources | 15:04 |
| vdrok | pas-ha: and I fail to find this place :) everywhere I look in resource tracker, all the operations happen on compute_node, which should be ok | 15:04 |
| sambetts | jroll: I think that its going to make an ironic attach_vif API even more nessesary after a quick scan | 15:05 |
| jroll | sambetts: yeah, I'm concerned about mac selection and such | 15:05 |
| jroll | sambetts: if you could give it a good think and review this week, that would be awesome | 15:05 |
| *** electrofelix has joined #openstack-ironic | 15:06 | |
| *** spandhe has joined #openstack-ironic | 15:08 | |
| *** xavierr is now known as xavierr_afk | 15:09 | |
| *** rama_y has joined #openstack-ironic | 15:11 | |
| sambetts | jroll: actually this fails well into a conversation myself and sukdev had about mac addresses, and making it so that when we create the port we don't give it a mac address, and we make baremetal vnics not generate one either by default, so when the port is created it has no mac address until we do our port-update call | 15:11 |
| dtantsur | inspector folks, if you review https://review.openstack.org/315047 I can create an inspector-client release today/tomorrow as well. Deal? :D | 15:12 |
| jroll | sambetts: ++ that was the thought when talking to johnthetubaguy | 15:12 |
| wajdi | general question. Should minor typos be commented on in reviews? Or would they be considered a bit of a nuisance when trying to push something through? | 15:12 |
| jroll | wajdi: I try to prefix them with "nit: " and not -1 for them | 15:12 |
| dtantsur | wajdi, should. but you should not -1 only for minor typos | 15:12 |
| jroll | (unless it's docs, I don't like typos in docs) | 15:12 |
| dtantsur | yeah, docs and release notes are special | 15:12 |
| wajdi | great. Thanks for the input | 15:12 |
| jroll | np | 15:13 |
| johnthetubaguy | sambetts: if it helps the routed network stuff means you create a port without an IP to start with, its not added till you bind the port to a segment | 15:13 |
| *** jistr has quit IRC | 15:13 | |
| johnthetubaguy | sambetts: the problem is what to do for end users when they create a port, I actually quite like not getting a mac address and IP until a later point all the time | 15:14 |
| johnthetubaguy | sambetts: I am curious how ironic deals with port re-use, so the mac address is defined, I guess some hardware might be able to deal with that, but many probably just fails, etc. | 15:14 |
| johnthetubaguy | sambetts: anyways, appreciate you taking a look at that, thank you! | 15:15 |
| jroll | johnthetubaguy: yeah, we don't support that today | 15:15 |
| sambetts | johnthetubaguy: yeah, we were saying that we think if make it so if they define the port as a baremetal vnic type then it doesn't generate a mac address or if it does we overide it on port update | 15:15 |
| sambetts | called from ironci | 15:16 |
| jroll | which we could do today, before this lands yeah? | 15:16 |
| sambetts | technically yes | 15:16 |
| sambetts | tbh I'd alway like to campain for separating neutron port macs from physical mac, but JayF informs me that some nics don't support sub interfaces with differnt macs to their parent :'( | 15:17 |
| jroll | yaaaa | 15:17 |
| johnthetubaguy | yeah, its never that easy I guess :( | 15:18 |
| sambetts | IKR ... | 15:18 |
| johnthetubaguy | I am OK updating the mac address when its "bound" to a host, but no idea what mess that causes inside neutron, DVR, etc, etc | 15:18 |
| sambetts | johnthetubaguy: as far as I know because the port isn't bound yet updating the mac is more or less just a db change | 15:19 |
| sambetts | johnthetubaguy: the dhcp service etc will pick that up and update itself etc because for some reason that operates even on unbound ports | 15:20 |
| johnthetubaguy | sambetts: that sounds possible, its just with port re-use I am not sure if unbinding the port does the required clean up, I guess it has to | 15:20 |
| johnthetubaguy | oh, fun | 15:20 |
| sambetts | yeah ... | 15:21 |
| JayF | sambetts: well, it's like handled in the driver. and linux nic drivers are notoriously awful | 15:21 |
| JayF | sambetts: I don't personally have a negative example at hand; but I bet I could find one | 15:21 |
| sambetts | JayF: heh :-p | 15:21 |
| *** milan has quit IRC | 15:21 | |
| mat128 | sambetts: made progress so far on the segfault: it seems to work in non-interactive mode only, once you're in interactive, make segfaults | 15:23 |
| sambetts | mat128: woah?! | 15:23 |
| mat128 | sambetts: I get segfaults on 16.04 running stock kernel (which didnt segfault before) when doing it interactively | 15:23 |
| *** spandhe has quit IRC | 15:23 | |
| mat128 | yup, a colleague's hypothesis revealed to be true | 15:23 |
| mat128 | really surprised | 15:23 |
| *** appu103_ has quit IRC | 15:23 | |
| mgould | segfaulting make? whoa, I've never seen that before | 15:24 |
| mgould | is it make itself, or something it calls? | 15:24 |
| JayF | mat128: do you have a bug or gist up with the deets? | 15:24 |
| *** spandhe has joined #openstack-ironic | 15:24 | |
| dtantsur | as I said 2 minutes ago in another channel, computers were a mistake >_< | 15:24 |
| sambetts | mat128: make itself | 15:24 |
| sambetts | mgould: ^ | 15:24 |
| jroll | dtantsur: ++ | 15:24 |
| mgould | whoa | 15:24 |
| mat128 | JayF: get this https://review.openstack.org/#/c/314999/ | 15:25 |
| * jroll would love to see the log | 15:25 | |
| JayF | lol | 15:25 |
| dtantsur | mat128, wow, great! | 15:25 |
| mat128 | incoming logs | 15:25 |
| JayF | ignore my question, throw a review at me | 15:25 |
| JayF | :P | 15:25 |
| jroll | "here, go segfault your own machine" :P | 15:25 |
| mat128 | JayF: LOL, I have a few logs of successful/crashing logs | 15:25 |
| mat128 | gate is happy with it, downstream build here is happy because it's done through non-interactive mode | 15:26 |
| JayF | oh make is segfaulting on this patch | 15:26 |
| JayF | got it | 15:26 |
| mat128 | no not even | 15:26 |
| mat128 | tinyIPA's make works, but then it tries to compile tgt | 15:26 |
| mat128 | using make, which segfaults | 15:26 |
| JayF | have you tried to repro on anything other than xenial? | 15:26 |
| betherly | devananda: jroll: krotscheck: rloo: as per discussion yesterday, patch submitted to sort the infra side of the ironic-ui docs. will work on updating the docs themselves for the various pages and have a patches up soon. https://review.openstack.org/#/c/315107/ | 15:26 |
| mat128 | this morning I had 14.04 A1, 16.04 A1 but 14.04 on xenial's kernel crashing | 15:27 |
| mat128 | caused by bad methodology: I was in interactive mode testing the latter | 15:27 |
| jroll | betherly: woot, thanks | 15:27 |
| mat128 | I get 100% success in non-interactive | 15:27 |
| rloo | thx betherly | 15:27 |
| mat128 | trying to provide you with easy steps to reproduce | 15:27 |
| mgould | mat128, but the segfault only happens when run interactively? | 15:27 |
| mat128 | thats correct | 15:27 |
| mgould | weeeeiiiirrrrd | 15:27 |
| mat128 | yup | 15:28 |
| mgould | uname -c unlimited; gdb the coredump, backtrace? | 15:28 |
| mat128 | having a hard time with this (pretty much first hands on experience troubleshooting a segfault), tried installing gdb in the chroot but it crashes because of missing symbols | 15:29 |
| jlvillal | sambetts, mgould: Just an FYI. When I tried to use 384MB for the tinyipa memory size it failed for me. In stable/mitaka | 15:29 |
| mat128 | tried outside (in my host) but cant seem to get anywhere | 15:29 |
| jlvillal | Took me most of the day to figure that out :( | 15:29 |
| JayF | mat128: mgould: Environment, perhaps? | 15:29 |
| jlvillal | mgould: sambetts: 512MB is working | 15:29 |
| mgould | jlvillal, thanks! | 15:29 |
| jlvillal | mgould: I am now creating 3 tinyipa VMs. Still seems to be failing in the same area for resource phase create :( | 15:29 |
| mgould | jlvillal, :-( | 15:30 |
| jroll | jlvillal: see my notes from yesterday? | 15:30 |
| * mgould didn't get any further | 15:30 | |
| jroll | slash irc pings | 15:30 |
| *** jaybeale has joined #openstack-ironic | 15:31 | |
| mgould | jroll, on the ironic-grenade etherpad? | 15:31 |
| sambetts | jlvillal: have you got IRONIC_BUILD_DEPLOY_RAMDISK=true or false? | 15:31 |
| jroll | mgould: yes, at the bottom | 15:31 |
| jlvillal | sambetts: Not sure. Checking... | 15:31 |
| *** jistr has joined #openstack-ironic | 15:31 | |
| sambetts | jlvillal: if youve got it false for stable/mitaka it'll end up running coreos by accident | 15:31 |
| mgould | ETOOMANYBROWSERTABS | 15:32 |
| jlvillal | accident? :) | 15:32 |
| sambetts | jlvillal: https://review.openstack.org/#/c/314933/ | 15:32 |
| mat128 | mgould: I get those way too often :) | 15:32 |
| openstackgerrit | Mario Villaplana proposed openstack/ironic: Allow vendor drivers to acquire shared locks https://review.openstack.org/226058 | 15:32 |
| jlvillal | sambetts: Not set at all | 15:32 |
| *** sacharya has joined #openstack-ironic | 15:33 | |
| mat128 | Instructions for reproducing: checkout https://review.openstack.org/#/c/314999/ then cd imagebuild/tinyipa; make | 15:33 |
| *** jistr has quit IRC | 15:33 | |
| mgould | jroll, cool | 15:33 |
| *** jistr has joined #openstack-ironic | 15:33 | |
| sambetts | jlvillal: it defaults to True so there goes my hypotheis | 15:33 |
| jroll | it does? I didn't think it did | 15:34 |
| mat128 | it didnt when we merged it | 15:34 |
| mat128 | maybe it changed | 15:34 |
| jlvillal | sambetts: Should I cherry pick that patch? | 15:34 |
| * mgould will set it to true explicitly and try again | 15:34 | |
| mat128 | we set it to true here to accelerate things | 15:34 |
| jlvillal | sambetts: I'm pretty sure I saw in the log it building tinyipa | 15:34 |
| *** vishwanathj has joined #openstack-ironic | 15:34 | |
| sambetts | jlvillal: yeah, I think you're building it | 15:35 |
| *** ifarkas has quit IRC | 15:35 | |
| jlvillal | sambetts: Any change from stable/mitaka to master to make it work in less memory? Since 384MB seems to be failing. But 512MB works. | 15:36 |
| jlvillal | I am pretty sure that was the issue I was seeing yesterday. | 15:36 |
| mat128 | sambetts, mgould, JayF : all the logs http://paste.openstack.org/show/496763/ | 15:36 |
| * jlvillal won't guarantee he didn't screw up somewhere else | 15:37 | |
| mat128 | interactively vs non-interactively, hypothesis proven | 15:37 |
| sambetts | jlvillal: no, it was working at 384mb back when I merged it, unless we've merged some giant dep into IPA :/ | 15:37 |
| jroll | wfm yesterday at 384 | 15:37 |
| sambetts | on stable/mitaka? | 15:37 |
| jroll | yes | 15:38 |
| jroll | http://logs.openstack.org/65/311865/3/experimental/gate-grenade-dsvm-ironic/e635dec/logs/grenade.sh.txt.gz | 15:38 |
| sambetts | :/ | 15:38 |
| *** sacharya has quit IRC | 15:38 | |
| jroll | smoke tests passing there | 15:38 |
| *** spandhe has quit IRC | 15:38 | |
| jlvillal | sambetts: jroll: Okay. I will try again. See if I can better identify the failure. | 15:38 |
| JayF | mat128: can you post the actual segfaults from your kernel logs? | 15:38 |
| mat128 | core dumps? | 15:39 |
| jroll | jlvillal: and I figured out why resource create is failing; the server build fails because ironic and IPA can't talk, because resource create makes a new neutron network and this code doesn't get run on it: https://github.com/openstack/ironic/blob/master/devstack/lib/ironic#L653 | 15:39 |
| mat128 | AHH | 15:39 |
| mat128 | I'm an idiot | 15:39 |
| jlvillal | jroll: Oh awesome! | 15:39 |
| mat128 | [10116.380127] make[19006]: segfault at 0 ip 00002ba76f2c4d62 sp 00007ffd2c176370 error 4 in libc-2.22.so[2ba76f256000+163000] | 15:39 |
| mgould | jroll++ | 15:39 |
| jroll | jlvillal: so I bugged sean about it, he's trying to focus on other things and asked me to hit the ML | 15:40 |
| jroll | so I'll do that at some point today | 15:40 |
| mgould | mat128, oh, of course! how didn't I think of that! | 15:40 |
| jroll | need to gather my references and such | 15:40 |
| mgould | but perhaps you could expand for the rest of the channel? :-) | 15:40 |
| JayF | mat128: I'm curious if 14.04 hardware enablement installs anew libc | 15:40 |
| mat128 | JayF: but it works in non-interactive mode :( | 15:40 |
| JayF | mat128: sure, that doesn't matter neccessarily | 15:40 |
| *** sacharya has joined #openstack-ironic | 15:41 | |
| mgould | mat128, is it a libc version-skew error? | 15:41 |
| mat128 | mgould: between my host and tinycore? | 15:41 |
| JayF | mat128: https://sourceware.org/bugzilla/show_bug.cgi?id=19861 | 15:42 |
| openstack | sourceware.org bug 19861 in nptl "pthread library fork invocation leads to crash" [Normal,New] - Assigned to unassigned | 15:42 |
| JayF | mat128: unsure if it's related, thinking not the more I read | 15:42 |
| mat128 | JayF: how did you find this? looking for libc 2.22 ? | 15:42 |
| sambetts | mat128: in none of the logs you posted do I see the seg fault I see | 15:43 |
| JayF | google://segfault in libc-2.22.so ubuntu 16.04 | 15:43 |
| *** aNupoisc has joined #openstack-ironic | 15:43 | |
| JayF | sambetts: look above ^ he pasted the segfault from kernel messages into chat | 15:43 |
| mat128 | JayF: I can try again with a fresh 14.04 | 15:43 |
| mat128 | sambetts: [10116.380127] make[19006]: segfault at 0 ip 00002ba76f2c4d62 sp 00007ffd2c176370 error 4 in libc-2.22.so[2ba76f256000+163000] | 15:43 |
| JayF | mat128: do you know if ubu hardware enablement upgrades libc too? | 15:43 |
| mat128 | JayF: don't know sorry, I'll check which libc I have | 15:44 |
| sambetts | mat128: it should be the version installed in the chroot right? | 15:44 |
| openstackgerrit | Kyrylo Romanenko proposed openstack/python-ironicclient: Functional tests for portgroups in ironicclient https://review.openstack.org/297727 | 15:44 |
| JayF | looks like 2.19 is what is in trusty, 2.23 is in xenial | 15:44 |
| JayF | mat128: ^ so what version is installed on your 14.04 w/16.04 kernel (hardware enablement) | 15:45 |
| mat128 | sambetts: yes 2.22 comes from TinyCore | 15:45 |
| sambetts | JayF: http://paste.openstack.org/show/496766/ its inside the chroot its blowing up though | 15:45 |
| mat128 | JayF: 2.19 | 15:45 |
| JayF | oooooh | 15:45 |
| JayF | inside the chroot | 15:45 |
| sambetts | make outside the chroot for me is fine | 15:45 |
| JayF | so you're using libc from tinycorelinux | 15:45 |
| mat128 | JayF: that's correct | 15:45 |
| mat128 | JayF: ./tinyipabuild/lib/libc-2.22.so | 15:45 |
| mgould | mat128, does the make binary also come from the chroot? | 15:45 |
| JayF | have you just like posted this upstream to their mailing list? | 15:46 |
| mat128 | mgould: yes | 15:46 |
| mat128 | JayF: I haven't so far | 15:46 |
| JayF | I would, especially with the interactive vs noninteractive | 15:46 |
| JayF | it's going to be an interesting bug | 15:46 |
| JayF | I suspect it'll have SOMETHING to do with the environment though, that's really all that's different, right? | 15:46 |
| sambetts | mat128: how are you switching between interactive and non-interactive? | 15:46 |
| JayF | bceause even with noninteractive, you have stdin/stdout/stderr hooked up in similar ways | 15:46 |
| JayF | so it shouldn't make a diff | 15:46 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/virtualbmc: Add unittests for the manager.py module https://review.openstack.org/314681 | 15:47 |
| sambetts | mat128: i'd like to try on mine | 15:47 |
| mat128 | sambetts, JayF: our local build script makes use of vagrant and that makes it non-interactive | 15:47 |
| mat128 | sambetts: let me check if I can repro with simple ssh -c | 15:47 |
| *** rcernin has quit IRC | 15:47 | |
| sambetts | I tried doing a bash -c "make" on mine to see if that worked but I got the same seg fault | 15:48 |
| mgould | JayF, I'm wondering if make uses pthreads when run interactively | 15:48 |
| JayF | mgould: that's exactly what I'm thinking, and I think that bug I linked above is being exposed | 15:48 |
| JayF | https://sourceware.org/bugzilla/show_bug.cgi?id=19861 | 15:48 |
| openstack | sourceware.org bug 19861 in nptl "pthread library fork invocation leads to crash" [Normal,New] - Assigned to unassigned | 15:48 |
| *** e0ne has joined #openstack-ironic | 15:49 | |
| * mgould clones the make repo and takes a look | 15:49 | |
| mat128 | mgould, JayF: in interactive, doing make --version crashes while make --help doesn't | 15:50 |
| mat128 | might help while statically looking at code | 15:50 |
| mgould | wow | 15:50 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-inspector: Always convert the size root device hints to an integer https://review.openstack.org/315140 | 15:52 |
| mgould | huh, looks like GNU Make only uses threads under win32 | 15:53 |
| mgould | so I don't think it is the pthread issue | 15:54 |
| *** Sukhdev has joined #openstack-ironic | 15:54 | |
| mgould | mat128, do you have a core dump? | 15:55 |
| mat128 | I certainly could, but I dont know how to read it | 15:55 |
| mgould | load it into gdb | 15:55 |
| mgould | one moment... | 15:55 |
| mat128 | gdb in the chroot or on the host? | 15:55 |
| *** rloo has quit IRC | 15:55 | |
| mgould | gdb /path/to/make coredump_file | 15:55 |
| mat128 | mgould, JayF, sambetts: running non-interactively http://paste.openstack.org/show/496767/ | 15:56 |
| mgould | shouldn't matter, I think | 15:56 |
| dtantsur | have to run now, see you tomorrow | 15:56 |
| *** dtantsur is now known as dtantsur|afk | 15:56 | |
| mgould | dtantsur, see you tomorrow! | 15:56 |
| mat128 | ok let me set ulimit and try again | 15:56 |
| mgould | mat128, cool | 15:56 |
| JayF | mat128: you running it with env -i when doing it 'interactively'? | 15:56 |
| mat128 | JayF: Interactively is here http://paste.openstack.org/show/496768/ | 15:57 |
| mat128 | exact copy paste of the command | 15:57 |
| JayF | I think I got it | 15:58 |
| JayF | when running non-interactively; you execute bash from inside the chroot, right? | 15:58 |
| JayF | but when doing it interactively, you're still inside the bash from ubuntu executing those scripts | 15:58 |
| mat128 | non-interactive: bash -e of sudo chroot /bin/sh -c "make --version" | 15:59 |
| mat128 | interactive: simple ssh (so shell of my user, guess bash) then sudo chroot /bin/sh -c "make --version" | 15:59 |
| mat128 | JayF: I can try anything | 15:59 |
| JayF | so in both cases you're exec'ing sh from within the chroot | 15:59 |
| JayF | so scratch that idea | 15:59 |
| JayF | this is bonkers | 15:59 |
| mat128 | JayF: thats correct, sh from within the chroot | 16:00 |
| JayF | can you attach gdb to it? | 16:00 |
| jlvillal | All your PyCharm are belong to us ;) | 16:00 |
| mat128 | mgould, JayF: http://paste.openstack.org/show/496769/ | 16:00 |
| jlvillal | Looks like a security issue for all JetBrains IDEs | 16:00 |
| mat128 | jlvillal: ouch, where? (IntelliJ here) | 16:01 |
| *** nicodemos has quit IRC | 16:01 | |
| *** appu103_ has joined #openstack-ironic | 16:01 | |
| mat128 | JayF: gdb from within the chroot seems broken | 16:01 |
| jlvillal | mat128: I got an email about it | 16:01 |
| *** rloo has joined #openstack-ironic | 16:01 | |
| mat128 | jlvillal: Oh, same | 16:01 |
| JayF | hrmph | 16:01 |
| mgould | mat128, set sysroot [chroot dir] at the gdb prompt | 16:02 |
| * mgould has never actually used gdb with chroots before, so this is guesswork | 16:02 | |
| *** fellypefca has joined #openstack-ironic | 16:03 | |
| mat128 | mgould: how do I "reload" the coredump after I do the set sysroot? | 16:03 |
| *** nicodemos has joined #openstack-ironic | 16:03 | |
| *** appu103_ has quit IRC | 16:03 | |
| mgould | mat128, you've got ironic-python-agent inside a Vagrant box, and you're running `make` within the tinyipa directory, right? | 16:03 |
| *** appu103_ has joined #openstack-ironic | 16:04 | |
| mat128 | but vagrant is hooked onto a privatecloud, so it could as well just be ssh | 16:04 |
| mgould | OK | 16:04 |
| mat128 | mgould: http://paste.openstack.org/show/496767/ < running non-interactively | 16:04 |
| mat128 | simple ssh | 16:04 |
| mgould | so I probably shouldn't have run make on my actual laptop, then? | 16:04 |
| mat128 | worst case, ssh onto your own box | 16:04 |
| mgould | OK, cool | 16:04 |
| mat128 | OS X here so I can't run most of this stuff directly :) | 16:04 |
| JayF | mat128: wait, that's an important consideration: when you run vagrant you're actually working from *different hardware*? | 16:05 |
| mgould | I'm just trying to get to a core dump of my very own | 16:05 |
| mat128 | mgould: I could always send it to you | 16:05 |
| mgould | mat128, I don't know if you need to reload the core dump; try `bt` | 16:05 |
| mat128 | mgould: given how everything should be the same | 16:05 |
| mgould | mat128, yes please - mgould at redhat.com if it's not too large :-) | 16:06 |
| *** appu103_ has quit IRC | 16:06 | |
| mat128 | mgould: backtrace http://paste.openstack.org/show/496771/ | 16:06 |
| JayF | I mean, can we put it in cloud files or something? | 16:06 |
| JayF | perfect | 16:06 |
| mat128 | cloudfiles? | 16:06 |
| *** appu103_ has joined #openstack-ironic | 16:06 | |
| mat128 | oh yeah | 16:06 |
| mat128 | your thing :) | 16:06 |
| JayF | or s3, or whatever | 16:06 |
| JayF | cloud files usually works as generic enough that folks know what I'm taking about | 16:06 |
| JayF | lol | 16:07 |
| *** dmk0202 has quit IRC | 16:07 | |
| mgould | strdup? huh | 16:07 |
| JayF | so a crazy thought | 16:09 |
| JayF | has anyone taken one of the successful versions of these builds | 16:09 |
| JayF | and made sure the ramdisk produced is sane and works? | 16:09 |
| mat128 | JayF: I did | 16:09 |
| mat128 | and the gate did | 16:09 |
| JayF | okay, cool | 16:09 |
| mat128 | mgould: Your mail server is apparently rejecting my mail: "Rejecting because of virus Heuristics.Broken.Executable" | 16:10 |
| mat128 | lmao | 16:10 |
| mat128 | Now to get access to share files :( | 16:11 |
| mgould | mat128, sorry! | 16:11 |
| mat128 | mgould: I know it's not something you can control | 16:11 |
| mat128 | lol | 16:11 |
| mat128 | mgould, JayF: https://www.dropbox.com/s/00c6ysapmqg9qqt/core.gz?dl=0 | 16:11 |
| mat128 | sorry about that, easiest thing I could find | 16:11 |
| mat128 | dropbox | 16:11 |
| JayF | that's perfect | 16:11 |
| mgould | mat128, thanks! | 16:12 |
| JayF | although not sure what I'd do with it other than pull a backtrace so I'll let those more talented tear the core file apart :P | 16:12 |
| mat128 | we're in the same boat lol | 16:12 |
| mgould | JayF, my plan was frame and list | 16:12 |
| mat128 | JayF: Have you seen the backtrace? | 16:12 |
| JayF | mat128: I looked at the one you linked | 16:12 |
| afaranha | Hey guys, I got an error when trying the ironic-neutron integration code: http://paste.openstack.org/show/496772/ | 16:15 |
| * mgould loads it up in gdb and discovers the debugging symbols have all been stripped | 16:15 | |
| afaranha | more specifically, I found this | 16:15 |
| *** athomas has quit IRC | 16:15 | |
| mgould | that seemed like *such* a good idea yesterday... | 16:15 |
| afaranha | ... --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file etc/neutron/plugins/ml2/ml2_conf_genericswitch.ini ... | 16:15 |
| mat128 | mgould: are you sure it merged? | 16:15 |
| afaranha | devstack is trying to read the genericswitch configuration on etc/neutron/... and not on /etc/neutron/.... | 16:15 |
| afaranha | The error was that: oslo_config.cfg.ConfigFilesNotFoundError: Failed to find some config files: /home/ubuntu/devstack/etc/neutron/plugins/ml2/ml2_conf_genericswitch.ini | 16:16 |
| mgould | mat128, not sure, will check | 16:16 |
| mat128 | I was thinking it was still in the in-flight change | 16:16 |
| mat128 | the find | strip | 16:16 |
| afaranha | As I'm not used to change devstack, does someone knows how can I tell devstack to look for the conf file in '/etc/' instead of 'etc/'? | 16:16 |
| JayF | I think it's intentional to not have the leading / | 16:17 |
| JayF | as most config files are in /opt/stack/etc rather than /etc, iirc | 16:17 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/virtualbmc: Add unittests for the manager.py module https://review.openstack.org/314681 | 16:17 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/virtualbmc: Raise an exception for domains already registered https://review.openstack.org/315149 | 16:17 |
| afaranha | JayF: Can I just copy the ml2_conf_genericswitch.ini to ~/devstack? | 16:18 |
| mgould | mat128, you're right, looks like that hasn't landed yet | 16:18 |
| mat128 | mgould: so how come it can't find symbols? | 16:18 |
| mat128 | mgould: we need symbols for libc, right? | 16:19 |
| JayF | afaranha: you can stick it in the right place, yes, I don't think that's ~/devstack, but it depends on how your local config is | 16:19 |
| mat128 | mgould: or make? | 16:19 |
| mgould | mat128, for make | 16:19 |
| mgould | brb | 16:19 |
| afaranha | JayF: Thanks, gonna try copy it :) | 16:20 |
| mgould | back | 16:21 |
| *** athomas has joined #openstack-ironic | 16:22 | |
| lucasagomes | Coverage report: 88% (for virtualbmc) | 16:23 |
| lucasagomes | not bad | 16:23 |
| mat128 | mgould, JayF: afk for a while, let me know if you find something or need anything | 16:24 |
| mgould | lucasagomes, nice | 16:24 |
| lucasagomes | mgould, that's the result with the 2 patches I have in queue yet | 16:24 |
| mgould | lucasagomes, yep, I'll take a look at them in a minute | 16:25 |
| lucasagomes | cool, ty, no rush! | 16:25 |
| JayF | So here's my question: Should we wait to merge those TCL things into tinyipa until we get to the bottom of the segfault | 16:25 |
| mat128 | JayF: The push for 7.x was mainly to leverage the work they did upstream in slimming down gpart | 16:26 |
| JayF | I realize that; my question is still valid | 16:26 |
| pas-ha | soo, it looks vsaienko dug the offending commit in Nova https://review.openstack.org/#/c/279313/ | 16:26 |
| mat128 | we can either: a) ask for a backport to 6.x; or b) use sambetts' patch where he rebuilds the package to avoid libicu | 16:26 |
| pas-ha | without it, hypervisor stats are owrking as expected | 16:26 |
| mat128 | JayF: we can go ahead with optimizations and slimming down without handling the segfault or switching to 7.x | 16:27 |
| mat128 | JayF: We could even get on 7.x, considering all builds are done in a non-interactive mode :) | 16:27 |
| mgould | mat128, oh, you're on 7.x | 16:27 |
| mat128 | but that's going to be a pain for people working on tinyIPA manually | 16:27 |
| mgould | do you have a branch I can check out? | 16:27 |
| mat128 | mgould: https://review.openstack.org/#/c/314999/ | 16:27 |
| JayF | mat128: yeah, that's why I asked; no reason gate wise or automation wise not to | 16:27 |
| openstackgerrit | Merged openstack/ironic-python-agent: Updated from global requirements https://review.openstack.org/314354 | 16:27 |
| mgould | mat128, thanks! | 16:27 |
| JayF | but it could make troubleshooting fun | 16:27 |
| JayF | I should also see if I can repro on my dev vm | 16:27 |
| JayF | which is fedora 23 | 16:28 |
| JayF | brb | 16:28 |
| sambetts | JayF: I'm running debian testing on kernel 4.5.0 and I'm seeing it :'( | 16:28 |
| mat128 | sambetts: do you get it in non-interactive? | 16:28 |
| * mgould is running it natively on his fedora 22 machine, because he likes to live dangerously | 16:28 | |
| vdrok | jroll: https://review.openstack.org/#/c/279313/ this might be it | 16:29 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/virtualbmc: Add "cover" to .gitignore https://review.openstack.org/315154 | 16:33 |
| *** jaybeale has quit IRC | 16:33 | |
| *** aNupoisc has quit IRC | 16:35 | |
| *** thrash is now known as thrash|biab | 16:41 | |
| sambetts | mat128: ... just run ssh sam@127.0.0.1 bash -e <<EOF cd /home/sam/Work/ironic-python-agent/imagebuild/tinyipa && echo "password" | sudo -S make build EOF and it worked just fine... WTF computers?! | 16:41 |
| mat128 | sambetts: thats it, non interactive mode makes it work | 16:42 |
| sambetts | mat128: how?! and why?! sigh... | 16:42 |
| JayF | sambetts: ssh -t | 16:42 |
| JayF | sambetts: give it a tty and watch it segfault, I bet | 16:42 |
| *** jaybeale has joined #openstack-ironic | 16:44 | |
| jlvillal | jroll: Hopefully I captured correctly what you said. I did mostly cut-n-paste :) https://etherpad.openstack.org/p/ironic-newton-grenade-whiteboard lines 11-14 | 16:44 |
| mat128 | I'll quote Dimtry on this: "computers were a mistake" | 16:44 |
| *** Nisha has quit IRC | 16:44 | |
| JayF | sambetts: that's the only thing I can think of that would be diff about executing over ssh vs not, would be presence of a tty | 16:44 |
| mat128 | JayF: with -t I get a "Pseudo-terminal will not be allocated because stdin is not a terminal." and it still works | 16:45 |
| sambetts | same | 16:45 |
| JayF | so -t is just ineffective | 16:45 |
| mat128 | tahts it | 16:46 |
| sambetts | mat128: could switch it out for bash -i -e to make it an interactive terminal | 16:46 |
| mgould | mat128, `make build` fails for me on the 7.x branch :-( | 16:47 |
| lucasagomes | mat128, mgould not sure if I have said that, but thanks a lot for all the reviews on vbmc | 16:47 |
| mat128 | mgould: because of the segfault? | 16:47 |
| lucasagomes | very appreciated! | 16:47 |
| mat128 | lucasagomes: always a pleasure :) | 16:47 |
| mgould | mat128, it's not even getting that far, afaict | 16:48 |
| * mgould tries again in case it was an intermittent error | 16:48 | |
| mat128 | JayF, sambetts: Multiple -t options force tty allocation, even if ssh has no local tty. | 16:48 |
| mat128 | watch me force it :) | 16:48 |
| mat128 | mgould: logs? | 16:49 |
| mgould | this is on fedora 22, I'll try in an ubuntu VM if it fails this time | 16:49 |
| *** irf has joined #openstack-ironic | 16:49 | |
| mgould | mat128, sure, will post the result of this run | 16:49 |
| mat128 | GOT IT! | 16:50 |
| mgould | ? | 16:50 |
| mat128 | ssh -t -t ubuntu@172.27.59.12 bash -e <<EOF | 16:50 |
| mat128 | reproduces it | 16:50 |
| mat128 | 100% | 16:50 |
| mgould | awesome | 16:50 |
| mat128 | you can easily switch interactive/non-interactive | 16:50 |
| JayF | so if you have a tty, it breaks | 16:50 |
| JayF | it sounds like we have a perfect bug to file against TCL | 16:50 |
| mat128 | JayF: That's correct | 16:50 |
| mgould | mat128, just hit the segfault! | 16:50 |
| JayF | and should probably stop rabbitholling here | 16:50 |
| afaranha | About the ironic-neutron integration, is the line 39 correct? https://review.openstack.org/#/c/258596/64/doc/source/dev/ironic-neutron-integration.rst | 16:51 |
| sambetts | mgould, JayF, mat128: http://patches.openembedded.org/patch/117921/ maybe? | 16:51 |
| JayF | bingo | 16:52 |
| JayF | mount your /dev/pts into the chroot | 16:52 |
| JayF | rather than applying that patch | 16:52 |
| sambetts | yeah, going to try | 16:52 |
| JayF | group troubleshooting ftw | 16:52 |
| mat128 | :D | 16:52 |
| sambetts | :D | 16:52 |
| *** amotoki has quit IRC | 16:52 | |
| mat128 | bingo | 16:53 |
| mat128 | works by mounting bind /dev/pts | 16:54 |
| mat128 | http://paste.openstack.org/show/496778/ | 16:54 |
| *** derekh has quit IRC | 16:55 | |
| mgould | cool, so we can use that as a workaround for now? | 16:55 |
| mat128 | updating the review as we speak | 16:56 |
| sambetts | mgould: just testing it in the main script now | 16:56 |
| openstackgerrit | Mathieu Mitchell proposed openstack/ironic-python-agent: Use TinyCore Linux 7.x for TinyIPA https://review.openstack.org/314999 | 16:56 |
| mat128 | ^updated | 16:56 |
| mat128 | mount bind just below /proc | 16:57 |
| mat128 | off to lunch now :) | 16:57 |
| mgould | hah, just as I got a core file open in gdb :-) | 16:57 |
| *** mat128 is now known as mat128|lunch | 16:57 | |
| mgould | not that it's actually helping, because of the lack of symbols | 16:58 |
| mat128|lunch | yeah | 16:58 |
| mgould | oh well | 16:58 |
| mat128|lunch | sambetts found it | 16:58 |
| mgould | cool | 16:58 |
| *** rpioso has joined #openstack-ironic | 17:00 | |
| *** ohamada has quit IRC | 17:00 | |
| sambetts | we all found it really by realising that we couldn't make with a tty | 17:02 |
| *** jistr has quit IRC | 17:02 | |
| *** keedya has joined #openstack-ironic | 17:02 | |
| *** jaybeale has quit IRC | 17:03 | |
| lucasagomes | sambetts, https://review.openstack.org/#/c/314933/ seems to be raising a 404 for the pre-built images | 17:05 |
| lucasagomes | logs.openstack.org/33/314933/3/check/gate-tempest-dsvm-ironic-pxe_ipa-tinyipa-nv/eddfb89/logs/devstacklog.txt.gz#_2016-05-11_12_39_24_970 | 17:05 |
| sambetts | lucasagomes: yup, thats because there hasn't been a merge into IPA stable/mitaka to trigger the post job | 17:05 |
| lucasagomes | sambetts, oh *sigh* | 17:06 |
| JayF | sambetts: did you post a docs update to mitaka + master updating that readme for tinyipa to not say experimental? | 17:08 |
| *** thrash|biab is now known as thrash | 17:08 | |
| JayF | sambetts: that would be easy as hell to get in and trigger that job | 17:08 |
| *** kromanenko_ has quit IRC | 17:09 | |
| lucasagomes | yeah I was thinking also if someone can do a -infra-fu and trigger that job manually | 17:10 |
| lucasagomes | it's a chicken and egg problem | 17:10 |
| sambetts | JayF: So I'm seeing a couple of patches merged and I'm looking to see why there isn't a build for tinyipa for stable/mitaka | 17:10 |
| JayF | ah | 17:10 |
| JayF | sambetts: to IPA? | 17:10 |
| JayF | sambetts: we haven't approved much to stable/mitaka in ipa, if anything | 17:11 |
| sambetts | yeah | 17:11 |
| *** aNupoisc has joined #openstack-ironic | 17:11 | |
| sambetts | this landed this afternoon https://review.openstack.org/#/c/314713/ | 17:11 |
| JayF | https://review.openstack.org/#/c/314713/ | 17:11 |
| JayF | yep, lol | 17:11 |
| lucasagomes | should we recheck that other patch then? | 17:12 |
| sambetts | ah the post job is still in the queue | 17:12 |
| lucasagomes | oh :-/ | 17:12 |
| * lucasagomes waits | 17:12 | |
| JayF | sambetts: I didn't see it there? | 17:12 |
| JayF | oooh | 17:12 |
| sambetts | in the post queue it the third one down | 17:12 |
| JayF | post jobs indexed by sha | 17:12 |
| JayF | not by patch # | 17:13 |
| lucasagomes | yeah, the post queue is quite slow since it's low prio | 17:13 |
| *** xavierr_afk is now known as xavierr | 17:13 | |
| *** ChubYann has joined #openstack-ironic | 17:16 | |
| lucasagomes | calling it a day, have a great evening all! | 17:21 |
| lucasagomes | see you tomorrow | 17:21 |
| *** electrofelix has quit IRC | 17:25 | |
| mgould | lucasagomes, good night | 17:26 |
| openstackgerrit | Jarrod Johnson proposed openstack/pyghmi: Extend session keepalive https://review.openstack.org/315174 | 17:26 |
| *** lucasagomes is now known as lucas-dinner | 17:27 | |
| lucas-dinner | see ya | 17:27 |
| *** irf has quit IRC | 17:28 | |
| *** irf has joined #openstack-ironic | 17:28 | |
| * mgould -> home; good night! | 17:31 | |
| *** e0ne has quit IRC | 17:31 | |
| *** mgould has quit IRC | 17:31 | |
| vdrok | good night lucas-dinner and mgould | 17:31 |
| openstackgerrit | Sam Betts proposed openstack/ironic-python-agent: Use TinyCore Linux 7.x for TinyIPA https://review.openstack.org/314999 | 17:32 |
| mat128|lunch | sambetts: nice catch :) | 17:33 |
| *** openstackgerrit has quit IRC | 17:33 | |
| sambetts | heh :-P | 17:33 |
| *** mat128|lunch is now known as mat128 | 17:33 | |
| mat128 | I wonder if we should have that in a trap instead | 17:33 |
| vdrok | one general question - I see in our gate jobs we had ceilometer enabled, by adding it to enabled_services, now it seems to be moved to plugin, so it's not being run, do you think we should reenable it? | 17:33 |
| *** openstackgerrit has joined #openstack-ironic | 17:34 | |
| *** milan has joined #openstack-ironic | 17:36 | |
| vdrok | anyway, good night everyone :) | 17:38 |
| *** fellypefca has quit IRC | 17:38 | |
| sambetts | goodnight vdrok | 17:38 |
| mat128 | good night vdrok, unfortunately I dont know the answer to your question :P | 17:39 |
| vdrok | yeah, np, just asking to not to forget :) | 17:39 |
| sambetts | vdrok: apprently grenade can do an upgrade of ceilometer according to their README | 17:40 |
| sambetts | vdrok: I guess in the future we will have tests to check if the stats we send to ceiometer actually work, so I think its probably worth reenabling | 17:42 |
| openstackgerrit | Caio Oliveira proposed openstack/python-oneviewclient: Adding dynamic allocation operations https://review.openstack.org/258699 | 17:42 |
| JayF | sambetts: commented on that TCL7.x patch | 17:44 |
| sambetts | JayF: we do clean it up, just further up the script | 17:45 |
| JayF | why would you unmount proc while still doing stuff in the chroot? | 17:45 |
| JayF | that seems wrongish | 17:45 |
| JayF | inconsistent environment for building at worse | 17:45 |
| sambetts | not sure XD but it worked haha | 17:46 |
| *** e0ne has joined #openstack-ironic | 17:46 | |
| openstackgerrit | Mathieu Mitchell proposed openstack/ironic-python-agent: Use TinyCore Linux 7.x for TinyIPA https://review.openstack.org/314999 | 17:46 |
| openstackgerrit | Merged openstack/ironic: Allow configuring shred's final overwrite with zeros https://review.openstack.org/304101 | 17:46 |
| sambetts | JayF: tbh I really want to re-factor the whole script to have a more try: except: finally: style format, because right now if something breaks during building then it leaves things mounted and you have to manually umount :( | 17:47 |
| mat128 | JayF, sambetts: Moved the umount to a trap that will always be run | 17:47 |
| mat128 | on "exit" so no matter how you exit (compilation failed with a segfault ;), or exited successfully) | 17:47 |
| JayF | I haven't learned a new bash trick in months | 17:47 |
| JayF | and TIL trap 'command' EXIT | 17:48 |
| JayF | that's awesome | 17:48 |
| mat128 | :D | 17:48 |
| mat128 | Awesome! | 17:48 |
| mat128 | you can also handle specific signals | 17:48 |
| mat128 | int, kill, etc | 17:48 |
| sambetts | thats exactly what I wanted!!! | 17:48 |
| *** ChrisAusten has joined #openstack-ironic | 17:48 | |
| JayF | I've used trap to catch signals before | 17:49 |
| JayF | to give like a clean exit for a bash daemon | 17:49 |
| mat128 | exit is a magic one | 17:49 |
| JayF | yeah, that's the cool bit | 17:49 |
| *** appu103_ has quit IRC | 17:49 | |
| JayF | Assuming this was tested locally to confirm it works? | 17:50 |
| *** appu103_ has joined #openstack-ironic | 17:50 | |
| JayF | Not sure gate would catch if it didn't | 17:50 |
| mat128 | tested | 17:50 |
| mat128 | can show you logs if you want | 17:50 |
| mat128 | I intentionally inserted a typo in the middle of the thing | 17:50 |
| JayF | I mean, I take your word for it in this case | 17:50 |
| mat128 | :) | 17:51 |
| mat128 | http://paste.openstack.org/show/496787/ | 17:51 |
| mat128 | just cause I still had it open | 17:51 |
| JayF | Not many folks in the habit of promising to have tested stuff and not having done it | 17:51 |
| JayF | lol | 17:51 |
| JayF | fwiw that log doesn't prove it was unmounted | 17:51 |
| mat128 | how come I always end up with half pasted logs | 17:51 |
| JayF | lol | 17:51 |
| JayF | it's fine though | 17:51 |
| mat128 | there, http://paste.openstack.org/show/496788/ | 17:51 |
| sambetts | filling your clipboard buffer? | 17:51 |
| mat128 | sambetts: that must be it :( | 17:51 |
| JayF | +2'd, going to wait for gate to land it | 17:52 |
| mat128 | http://paste.openstack.org/show/496789/ | 17:52 |
| mat128 | there | 17:52 |
| * sambetts will rebase the other slimming patches when that lands, so that we can get back to 38mb | 17:52 | |
| JayF | I thought we were down to 256 | 17:53 |
| JayF | what bumped it back up? | 17:53 |
| *** ijw has quit IRC | 17:53 | |
| mat128 | 38 was using a change that hasn't landed yet | 17:53 |
| mat128 | https://review.openstack.org/#/c/313675 | 17:53 |
| JayF | no, that's def not true | 17:53 |
| krtaylor | jlvillal, thanks for the comments, good stuff, got me thinking about what we can do in the ci tools repo | 17:53 |
| sambetts | my patches that were in flight brought initrd to 38mb, and that run at 256mb | 17:53 |
| JayF | jroll had those ramdisks running at 384 in the gate | 17:54 |
| JayF | oh you're talking about disk size | 17:54 |
| JayF | not about ram needed to run | 17:54 |
| mat128 | ooh | 17:54 |
| sambetts | yeah | 17:54 |
| mat128 | 38 looks like 384 | 17:54 |
| sambetts | XD | 17:54 |
| mat128 | much confusion :) | 17:54 |
| sambetts | 38mb initrd == fast(ish) transfers | 17:54 |
| sambetts | :d | 17:54 |
| sambetts | :D | 17:54 |
| mat128 | IIRC, I had a 52 or 54MB ramdisk on 7.x | 17:55 |
| sambetts | and smaller initrd to unpack == less ram | 17:55 |
| mat128 | without icu | 17:55 |
| sambetts | mat128: yeah, I just rebased my patch ontop of yours and removed the manual build of gdisk and its back down to 38mb after stripping things like babel out | 17:55 |
| mat128 | sambetts: excellent! | 17:55 |
| JayF | sambetts: you shuld push that up so we get gates running on it sooner rather than later | 17:56 |
| JayF | if we play our cards right by as early as EOW we could have tinyipa in the gate at 256MB size | 17:56 |
| *** afaranha has quit IRC | 17:56 | |
| mat128 | JayF: that'd be great | 17:56 |
| JayF | I'm going to fix my project-config patch | 17:57 |
| mat128 | TIL we have someone called "early" in this room | 17:57 |
| JayF | been a bear getting that one right | 17:57 |
| * sambetts is wondering if backporting optimisations to the tinyipa build scripts is seen as wrong ... | 17:57 | |
| JayF | sambetts: I mean, it's a grey area for sure, but given it would go towards making grenade work I think we've been bending those rules | 17:57 |
| JayF | sambetts: all you can do is push it up and see; I find a PR someone disagrees with is a faster path to getting something fixed than no pr at all | 17:58 |
| *** e0ne has quit IRC | 17:59 | |
| * mat128 feels like getting a good grade on a test when I get a +2 +A on a change :) | 18:00 | |
| *** ChrisAusten has quit IRC | 18:01 | |
| sambetts | hehe | 18:02 |
| JayF | Honestly, I always get weirded out if I get +2's without any back and forth | 18:04 |
| openstackgerrit | Merged openstack/ironic: Regenerate sample configuration https://review.openstack.org/315052 | 18:05 |
| JayF | I don't have the confidence that I would get something right on the first trry, lol | 18:05 |
| sambetts | if something works first time you've done it wrong XD | 18:05 |
| mat128 | back and forth makes it better, exposes an idea to more pairs of eyes | 18:05 |
| sambetts | shares the blame ;) | 18:05 |
| openstackgerrit | Sam Betts proposed openstack/ironic-python-agent: TinyIPA: Shave off some file size from tinyipa ramdisk https://review.openstack.org/313675 | 18:06 |
| openstackgerrit | Sam Betts proposed openstack/ironic-python-agent: TinyIPA: Precompile python code for faster load https://review.openstack.org/313103 | 18:06 |
| sambetts | btw ^^ | 18:07 |
| JayF | sambetts: -1 on 313675, I think your commit message is wrong | 18:09 |
| JayF | sambetts: because we don't poke at gdisk anymore i nthat | 18:09 |
| *** keedya has quit IRC | 18:09 | |
| openstackgerrit | Sam Betts proposed openstack/ironic-python-agent: TinyIPA: Shave off some file size from tinyipa ramdisk https://review.openstack.org/313675 | 18:09 |
| sambetts | fixxed ^^^ | 18:09 |
| JayF | k, +2 on whole series now | 18:10 |
| JayF | will check back later after jobs run to land them if someone else gets a vote on 'em | 18:10 |
| *** ChubYann has quit IRC | 18:11 | |
| *** ChubYann has joined #openstack-ironic | 18:23 | |
| *** spandhe has joined #openstack-ironic | 18:25 | |
| *** Sukhdev has quit IRC | 18:29 | |
| *** krotscheck_ has joined #openstack-ironic | 18:30 | |
| *** Sukhdev has joined #openstack-ironic | 18:31 | |
| *** krotscheck has quit IRC | 18:31 | |
| *** krotscheck_ is now known as krotscheck | 18:33 | |
| *** spandhe has quit IRC | 18:36 | |
| *** afaranha has joined #openstack-ironic | 18:41 | |
| *** jjohnson2 has quit IRC | 18:41 | |
| *** e0ne has joined #openstack-ironic | 18:44 | |
| *** e0ne has quit IRC | 18:45 | |
| *** tesseract has quit IRC | 18:51 | |
| *** afaranha has quit IRC | 18:53 | |
| openstackgerrit | Stephanie Miller proposed openstack/bifrost: Add pycrypto to requirements https://review.openstack.org/315213 | 18:53 |
| *** irf has quit IRC | 18:56 | |
| *** fragatina has quit IRC | 19:01 | |
| * sambetts -> home | 19:04 | |
| *** keedya has joined #openstack-ironic | 19:04 | |
| *** sambetts is now known as sambetts|afk | 19:04 | |
| openstackgerrit | Merged openstack/pyghmi: Extend session keepalive https://review.openstack.org/315174 | 19:07 |
| *** davideagnello has quit IRC | 19:21 | |
| *** fragatina has joined #openstack-ironic | 19:26 | |
| *** aNupoisc has quit IRC | 19:28 | |
| NobodyCam | sambetts|afk: WOW 38 Mb +++ awesome work! | 19:29 |
| mat128 | :D | 19:30 |
| *** appu103_ has quit IRC | 19:31 | |
| *** joprovost has quit IRC | 19:34 | |
| *** aNupoisc has joined #openstack-ironic | 19:36 | |
| *** bnemec has quit IRC | 19:36 | |
| *** dmk0202 has joined #openstack-ironic | 19:38 | |
| *** e0ne has joined #openstack-ironic | 19:39 | |
| *** e0ne has quit IRC | 19:41 | |
| *** milan_ has joined #openstack-ironic | 19:43 | |
| *** milan has quit IRC | 19:44 | |
| *** bnemec has joined #openstack-ironic | 19:44 | |
| *** aNupoisc has quit IRC | 19:45 | |
| *** r-daneel has joined #openstack-ironic | 19:49 | |
| mat128 | JayF: https://review.openstack.org/#/c/314999/ poke =) gates are green | 19:52 |
| JayF | mat128: gotta get another core review on it first | 19:59 |
| * JayF goes back to lunching | 19:59 | |
| *** athomas has quit IRC | 19:59 | |
| *** ijw has joined #openstack-ironic | 20:04 | |
| NobodyCam | JayF: +2/+a on https://review.openstack.org/#/c/313675 | 20:08 |
| *** daemontool has quit IRC | 20:09 | |
| *** e0ne has joined #openstack-ironic | 20:11 | |
| *** ijw has quit IRC | 20:11 | |
| *** e0ne has quit IRC | 20:13 | |
| *** daemontool has joined #openstack-ironic | 20:14 | |
| klindgren | Hello all - looking for some direction here. I am trying to use ironic-discoveryd on kilo to inspect nodes. But keep running into an error where ironic_discoverd can't find the node once the json blob is reurned. | 20:20 |
| klindgren | http://paste.ubuntu.com/16369186/ | 20:20 |
| klindgren | when I do ironic node-set-provisionin-state inspect. Is something suppose to be created within the inspection database? | 20:21 |
| klindgren | As I understand the process its roughly: enroll bmc into ironic, manage node, inspect node. server is booted onto a network where the ipa image is pxe booted on the server, the inspection is done and posted to the inspection url. THe inspector db is updated. | 20:22 |
| klindgren | mean while ironic is polling the inspection api to pull the appropriate data from inspector once inspection happens | 20:23 |
| klindgren | but I keep getting: DEBUG:ironic_discoverd.node_cache:Empty value for attribute bmc_address | 20:23 |
| klindgren | DEBUG:ironic_discoverd.node_cache:Trying to use mac of value [u'00:8c:fa:01:14:00'] for node look up | 20:23 |
| klindgren | ERROR:ironic_discoverd.utils:Could not find a node for attributes {'bmc_address': u'', 'mac': [u'00:8c:fa:01:14:00']} | 20:23 |
| klindgren | and feel like I am missing something stupidly obvious | 20:23 |
| *** jaybeale has joined #openstack-ironic | 20:24 | |
| openstackgerrit | Caio Oliveira proposed openstack/python-oneviewclient: Adding dynamic allocation operations https://review.openstack.org/258699 | 20:25 |
| devananda | klindgren: does that server's BMC have an IP address? | 20:25 |
| devananda | that ERROR line seems to indicate that the ramdisk wasn't able to determine the IP address of the BMC using ipmitool locally | 20:26 |
| devananda | which is why it isn't able to correlate the hardware inspection data back to a Node | 20:26 |
| devananda | *which would explain why .. | 20:26 |
| klindgren | devananda, it does have an ip adress | 20:27 |
| *** absubram has quit IRC | 20:38 | |
| JayF | NobodyCam: https://review.openstack.org/#/c/314999/4 is the one that needed it | 20:41 |
| JayF | NobodyCam: you +2A'd a dependant patch | 20:41 |
| *** ayoung has quit IRC | 20:41 | |
| *** pcaruana has quit IRC | 20:42 | |
| *** rcernin has joined #openstack-ironic | 20:44 | |
| NobodyCam | JayF: will review 314999 now... also /rant I have I said how much I dislike the new gerrit's displaying of dependencies | 20:45 |
| *** aNupoisc has joined #openstack-ironic | 20:45 | |
| *** xhku has quit IRC | 20:45 | |
| JayF | NobodyCam: yeah; if you can +2 that I can land the whole patch chain pending tests | 20:45 |
| openstackgerrit | Stephanie Miller proposed openstack/bifrost: Add pycrypto to requirements https://review.openstack.org/315213 | 20:45 |
| JayF | looks like the whole chain passed tests | 20:45 |
| NobodyCam | while I review.. have you ever seen anything like this: | 20:46 |
| NobodyCam | stack@localhost:~$ ironic --os-endpoint-type=internalURL node-list | 20:46 |
| NobodyCam | public endpoint for baremetal service not found | 20:46 |
| devananda | NobodyCam: trying out openstackclient? | 20:46 |
| JayF | https://review.openstack.org/#/c/313799/ (allows us to change iPXE default in devstack w/o affecting jobs) and https://review.openstack.org/#/c/313800/ (changes most jobs to ipxe+tinyipa) are both in project-config and could use reviews | 20:46 |
| JayF | NobodyCam: run with --debug, and look at your service catalog | 20:46 |
| JayF | NobodyCam: maybe something incorrect about your os-endpoint-type option? | 20:47 |
| NobodyCam | devananda: its a HOS thing where public url are not set in keystone | 20:47 |
| klindgren | devananda, so looking at the IPA image it looks like none of the ipmi kernel modules are loaded - which would probably explain why ipmitool couldn't get information | 20:49 |
| cinerama | can i get a core to throw a +2A on the above bifrost change (modulo objections) so we can get tests passing again? | 20:52 |
| jlvillal | krtaylor: Your welcome. Sorry I went to lunch and then an almost 2 hour meeting. Just saw your message. | 20:52 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ironic: Updated from global requirements https://review.openstack.org/315057 | 20:52 |
| *** jaypipes has joined #openstack-ironic | 20:54 | |
| jaypipes | jroll: does Ironic support the Compute API "rebuild" operation? | 20:55 |
| jroll | jaypipes: yep, and preserve ephemeral | 20:55 |
| JayF | NobodyCam: jroll took care of it, thanks | 20:55 |
| JayF | mat128: sambetts|afk: Those 3 patches are on the way to the gate | 20:56 |
| NobodyCam | lol and I was just reviewing | 20:56 |
| *** caiobo has quit IRC | 20:59 | |
| *** raildo is now known as raildo-afk | 21:00 | |
| jaypipes | jroll: what about resize? | 21:00 |
| jroll | jaypipes: nope | 21:00 |
| jaypipes | k, thank the lord. | 21:00 |
| jroll | lol | 21:00 |
| jroll | jaypipes: side note, some folks tracked this fun one down today https://bugs.launchpad.net/nova/+bug/1572555 | 21:00 |
| openstack | Launchpad bug 1572555 in OpenStack Compute (nova) "Nova reports memory_mb=0 for available Ironic node" [Undecided,New] | 21:00 |
| JayF | jroll: we should make sure that's not in the downstream release for rackspace in testing now | 21:01 |
| JayF | re: resize, some machine with dynamically configurable and hot swappable ram | 21:01 |
| JayF | you could totes do a nova resize | 21:01 |
| jaypipes | jroll: yeah, saw that earlier. | 21:01 |
| jroll | jcook: ^ | 21:01 |
| jaypipes | the bug... | 21:01 |
| JayF | even if it wasn't hot swappable, power off the node, call hardware API to add ram, boot machine up | 21:01 |
| JayF | resize done | 21:01 |
| jroll | ya :| | 21:02 |
| *** davideagnello has joined #openstack-ironic | 21:02 | |
| *** ijw has joined #openstack-ironic | 21:07 | |
| *** ekarlso has quit IRC | 21:11 | |
| *** ijw has quit IRC | 21:13 | |
| *** davideagnello has quit IRC | 21:14 | |
| *** ekarlso has joined #openstack-ironic | 21:14 | |
| *** ayoung has joined #openstack-ironic | 21:15 | |
| *** davideagnello has joined #openstack-ironic | 21:16 | |
| *** ijw has joined #openstack-ironic | 21:21 | |
| *** keedya has quit IRC | 21:22 | |
| rloo | are we sure we want to make iPXE/TinyIPA the devstack defaults? Is that something we should announce to the community first before it happens? https://review.openstack.org/#/c/313035/ | 21:29 |
| rloo | jroll, JayF ^^ | 21:29 |
| jroll | rloo: I'm sure. is there a particular reason you'd want to announce it? | 21:30 |
| rloo | jroll: not really, except that it is changing the default behaviour. | 21:30 |
| jroll | (I do see some good reasons to email about it, but not sure it needs to be before or after) | 21:30 |
| JayF | My motivation for changing the defaults in devstack was because it's been discussed multiple times we want to change the gate to tinyipa, and it seems like we want folks testing on gate-like environments | 21:30 |
| rloo | as a heads up really. | 21:30 |
| *** baoli has quit IRC | 21:31 | |
| jroll | yeah, the heads up I think is fine before or after | 21:31 |
| JayF | I mean, in that case, kinda wanna JFDI | 21:31 |
| JayF | because otherwise I'll forget it | 21:31 |
| jroll | JayF: yeah, tear it up :) | 21:31 |
| JayF | want me to email about it now? If so you think I should include the project-config PRs to change the gate? | 21:31 |
| rloo | ah, ok, JayF has volunteered. thx JayF! | 21:31 |
| JayF | It's my patch :P | 21:31 |
| * JayF doing all the 'enable all the faster stuff others are doing in devstack/gate' work | 21:31 | |
| jroll | JayF: yes and yes | 21:31 |
| rloo | I'll +A it. that one anyway. not the dependent one :) | 21:32 |
| jroll | idk why we need a "press release" but okay :D | 21:32 |
| JayF | rloo: -1 | 21:32 |
| JayF | rloo: do not land that until the project-config thing it depends on lands | 21:32 |
| rloo | OH, the depends-on doesn't work? | 21:32 |
| jroll | not with project-config | 21:32 |
| JayF | rloo: I know gerrit will sometimes enforce that but I wanna be sure they don't race, because project-config merges don't take effect immediately | 21:32 |
| jroll | JayF: wip it then if you want to take extra caution | 21:32 |
| rloo | JayF: then you need to put a -2 or WIP thingy. | 21:33 |
| jroll | or I can -2 for you | 21:33 |
| JayF | going to W-1 it | 21:33 |
| jroll | cool | 21:33 |
| rloo | thx JayF | 21:33 |
| JayF | https://review.openstack.org/#/c/313799/ btw is all we need to land | 21:33 |
| JayF | it just needs one more infra review | 21:34 |
| JayF | I've asked twice if someone else wants to spam it :) | 21:34 |
| *** ChrisAusten has joined #openstack-ironic | 21:34 | |
| JayF | I even split it off from the bigger patch so it would land more quickly | 21:34 |
| jroll | mordred: wanna help? :) | 21:34 |
| openstackgerrit | Merged openstack/ironic-python-agent: Use TinyCore Linux 7.x for TinyIPA https://review.openstack.org/314999 | 21:34 |
| JayF | I just realized something | 21:36 |
| JayF | the way our jobs work | 21:36 |
| JayF | it's be possible for incompatible Ironic and IPA changes to merge together | 21:37 |
| JayF | because right now, the IPA we're running our gate against is 8h+ old | 21:37 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ironic: Updated from global requirements https://review.openstack.org/315057 | 21:37 |
| JayF | so you merge a change into IPA that works in today-ironic, merge a change into Ironic that breaks that, when the new IPA gets uploaded (at some much later time), it explodes | 21:37 |
| JayF | unless we run a -src job against ironic proper | 21:37 |
| JayF | probably doesn't matter but not a case I had thought about until today, waiting for the post job to run on the stable/mitaka IPA change | 21:38 |
| *** Sukhdev has quit IRC | 21:38 | |
| NobodyCam | any cores want to take a look at a one line gate fix for bifrost? https://review.openstack.org/#/c/315213 | 21:39 |
| JayF | NobodyCam: I actually had a question abbout that | 21:39 |
| NobodyCam | :) | 21:39 |
| JayF | NobodyCam: given ansible isn't in requriements, it seems wrong to put an ansible requirement in there | 21:39 |
| NobodyCam | sure | 21:39 |
| *** ChrisAusten has quit IRC | 21:39 | |
| JayF | like if it depends on ansible, why isn' | 21:39 |
| JayF | *isn't ansible in the requirements | 21:39 |
| JayF | and if using system ansible, why would we install pycrypto via pip? | 21:39 |
| * JayF actually puts that on the patch | 21:40 | |
| NobodyCam | cinerama: ^^^^^ | 21:40 |
| cinerama | just saw that | 21:40 |
| cinerama | so we're typically installing ansible separately and give the user a few choices about it | 21:42 |
| devananda | JayF: I agree with your assessment. I think it's fine - we shouldn't be making the kinds of changes that would break that interaction without a deprecation period | 21:42 |
| mrda | Morning Ironic | 21:42 |
| devananda | JayF: and if we do -- and it stops our gate the next day -- then we'll have all the more reason to fix/revert it right away, and minimize impacts downstream | 21:42 |
| JayF | devananda: just a good thing to have thought about before it inevitably happens by accident at some future point | 21:43 |
| cinerama | if they don't have an ansible on the system already we grab it from git | 21:43 |
| cinerama | also they can choose to install ansible in a python venv | 21:43 |
| devananda | JayF: the one piece I'd want to know, and don't: could we trigger a rebuild/reupload of hte IPA image quickly? or would our gate be inevitably down for a day while the mirrors update? | 21:43 |
| JayF | cinerama: then why is it ever right to install pycrypto alongside bifrost, given ansible could be in a different python environment altogether? | 21:43 |
| JayF | devananda: I honestly don't know either. | 21:43 |
| JayF | devananda: As soon as the post job ran on IPA, we'd be good, but unwedging it would probably require bypassing tests or reverting in Ironic first | 21:44 |
| devananda | hmm. yea. | 21:44 |
| JayF | Well no | 21:44 |
| devananda | wait. we have an ipa-src job on ipa, right? | 21:44 |
| JayF | because IPA jobs run -src | 21:44 |
| JayF | yep | 21:45 |
| devananda | cool | 21:45 |
| JayF | lol like we got the realization simultaneously | 21:45 |
| JayF | I'm actually improving that in a patch | 21:45 |
| JayF | making sure we have voting jobs on both tinyipa + coreos ramdisk | 21:45 |
| jroll | cinerama: agree that pycrypto install should happen wherever you install ansible | 21:45 |
| JayF | running -src | 21:45 |
| openstackgerrit | Merged openstack/ironic-python-agent: TinyIPA: Precompile python code for faster load https://review.openstack.org/313103 | 21:45 |
| openstackgerrit | Merged openstack/ironic-python-agent: TinyIPA: Shave off some file size from tinyipa ramdisk https://review.openstack.org/313675 | 21:45 |
| openstackgerrit | Merged openstack/ironic-python-agent: Remove "Experimental" warning from tinyipa README https://review.openstack.org/314929 | 21:45 |
| devananda | so my concern is that if we landed the fix in IPA, and we know it's a fix because it passes ipa-src, we'd still be left with a small window (until any caches update) that the jobs are down | 21:45 |
| JayF | I don't think tarballs.openstack.org is cached, is it? | 21:46 |
| devananda | JayF: but I think that's acceptable | 21:46 |
| cinerama | i think i see what's happening here | 21:46 |
| *** ChrisAusten has joined #openstack-ironic | 21:47 | |
| devananda | JayF: oh right. pip's take a while to update, because we publish those upstream and then pull down to mirror it | 21:47 |
| cinerama | let me confirm what i suspect and i'll get back to everyone | 21:47 |
| JayF | ayup, so really the question is "can we ask infra, if ever needed, to prioritize a post job" | 21:47 |
| jroll | devananda: JayF: the post queue is usually at 1-8 hours, so it is a problematic thing. that said, I think this is a rare problem. if needed, we can ask them to promote a thing. | 21:47 |
| JayF | otherwise a ton of going-to-fail ironic jobs would be in front of it, which is just silly | 21:48 |
| devananda | if this were to happen during a cruch period (like leading up to release week) -- yea, we can ask infra to promote that job | 21:48 |
| *** jaybeale has quit IRC | 21:48 | |
| *** Goneri has quit IRC | 21:48 | |
| devananda | I have had to do that to unwedge things in the (distant) past | 21:49 |
| jroll | nova still does that occassionally | 21:49 |
| jroll | because the fallout is so huge | 21:49 |
| *** [1]cdearborn has quit IRC | 21:50 | |
| jroll | jlvillal: have you explored why grenade jobs don't publish ironic-bm-logs? | 21:52 |
| jroll | devananda: you're gonna love this grenade problem I found | 21:53 |
| jroll | just sent a mail | 21:53 |
| devananda | ....? | 21:54 |
| jroll | devananda: tl;dr grenade makes a new network | 21:55 |
| devananda | .... | 21:55 |
| jroll | can't find a place to do the crazy ovs tapping we do, before nova boots a thing | 21:55 |
| jroll | so no control plane access for agent | 21:55 |
| jroll | I just thought of one other thing that could be doing it, but pretty sure it's the missing tap | 21:56 |
| devananda | new provider network? or new neutron network? | 21:57 |
| openstackgerrit | Merged openstack/python-ironicclient: Trivial: ignore openstack/common in flake8 exclude list https://review.openstack.org/314877 | 21:57 |
| jroll | tenant (private) network | 21:57 |
| jroll | today in dsvm we boot machines on private network, which we tap into to give control plane / agent access | 21:58 |
| openstackgerrit | Jim Rollenhagen proposed openstack/ironic: Test post don't upvote https://review.openstack.org/311865 | 21:58 |
| devananda | re-attach the tap as grenade brings the services up? | 22:00 |
| jroll | this is before that | 22:01 |
| jroll | so | 22:01 |
| jroll | grenade has a few phases | 22:01 |
| jroll | bring up old, run smoke tests | 22:01 |
| jroll | create some resources (servers, networks, etc) to persist through the upgrade | 22:01 |
| jroll | run upgrades | 22:01 |
| jroll | run more tests | 22:01 |
| jroll | the "create resources" phase is where this happens | 22:01 |
| jroll | we plug into that, but the neutron part creates a network, and the nova part boots a thing, before our plugin runs | 22:02 |
| devananda | oooh right | 22:02 |
| jroll | our plugin runs after | 22:02 |
| devananda | this is happening on the "old" services? | 22:02 |
| jroll | yep | 22:02 |
| devananda | because grenade is not testing in the same way that devstack sets things up | 22:02 |
| devananda | the ML2 integration would address this | 22:03 |
| jroll | yeah, I think so | 22:03 |
| JayF | lifeless: A++++ email on the go thread. That's my thoughts on it too but didn't want to wade into the fray | 22:03 |
| devananda | because ironic could put the node into what ever tenant network is defined -- and back into the prov net | 22:03 |
| devananda | jroll: what about flat network? | 22:03 |
| jcook | jroll https://bugs.launchpad.net/nova/+bug/1572555 neat... | 22:03 |
| openstack | Launchpad bug 1572555 in OpenStack Compute (nova) "Nova reports memory_mb=0 for available Ironic node" [Undecided,New] | 22:03 |
| JayF | jcook: I suggested you might wanna get someone to make sure that's not in i-whatever is being tested now | 22:04 |
| jroll | devananda: well, flat network is the current case, and where we're falling apart | 22:04 |
| jcook | yeah, was just catching up and saw that | 22:04 |
| JayF | jcook: the breaking commit is in the bug | 22:04 |
| devananda | jroll: I mean single flat network. But I guess that would prevent grenade from creating other tenant nets | 22:04 |
| jcook | I'll check | 22:04 |
| jroll | devananda: yeah, and that isn't how we run devstack today. we could totally refactor how we test but... | 22:05 |
| jroll | I guess I also don't know why we do test it this way | 22:05 |
| devananda | jroll: I don't know if there is a reason for it, either. probably that we tried to minimize changes to devstack, before the libification happened | 22:06 |
| devananda | but I'm guessing. it's been a while | 22:06 |
| jroll | hm | 22:06 |
| jroll | devananda: anyway, fun wedge to think about | 22:10 |
| * jroll out for the night, see y'all tomorrow | 22:11 | |
| NobodyCam | night jroll | 22:11 |
| klindgren | So unless I am just off my rocker here.... why is the default CoreOS IPA image that is on tarballs. based on CoreOS 367.1.0? | 22:13 |
| JayF | klindgren: because when it got upgraded to a newer version, it bloated the image so much it caused significant gate failures | 22:14 |
| JayF | klindgren: I want to re-approach that once our gates are using tinyipa by default, which reviews are up for now | 22:15 |
| klindgren | hence the move to tinyipa | 22:15 |
| *** ijw has quit IRC | 22:15 | |
| JayF | exactly | 22:15 |
| klindgren | so for someone starting out with ironic.... ditch the coreos IPA image | 22:15 |
| klindgren | and just use the tiny? | 22:15 |
| klindgren | would be the recommendation from you? | 22:16 |
| jroll | coreos image is great, and better for debugging imo, but takes longer to boot | 22:16 |
| jroll | and is larger in terms of network transfer | 22:16 |
| JayF | I say it really depends | 22:16 |
| jroll | if you're playing with devstack and virtual bare metal, tinyipa all the way | 22:16 |
| JayF | do you want your ramdisk to just deploy, and do nothing else whatsoever? use dib or tinyipa | 22:16 |
| JayF | if you want to do extensive cleaning and have more troubleshooting tools, use coreos | 22:16 |
| * jroll disappears again | 22:17 | |
| JayF | we use coreos downstream for OnMetal today | 22:17 |
| jcook | JayF jroll it does not contain that commit | 22:17 |
| *** davideagnello has quit IRC | 22:17 | |
| JayF | jcook: ++ awesome, we should get it fixed upstream before another pull | 22:17 |
| klindgren | ok - I was hoping that I could avoid building my own ipa for now. But going to fire that up... need the ipmi kernel modules loaded so inspection works | 22:18 |
| jcook | JayF assuming this is right one: git branch --contains f9b62dffe092eaacf04e2bb38ab8ac0dc830ea5f | 22:18 |
| JayF | jcook: I have no idea, lol | 22:18 |
| klindgren | thanks | 22:18 |
| JayF | jcook: I'd probably go something more like git checkout SHA1 and see if it works | 22:18 |
| jcook | JayF https://review.openstack.org/#/c/279313/ | 22:19 |
| JayF | yep | 22:19 |
| jcook | then we good | 22:19 |
| JayF | so uh, sadness | 22:21 |
| JayF | jroll: sambetts|afk: post job finished against stable/mitaka; no tinyipa files uploaded | 22:22 |
| *** rbudden has quit IRC | 22:24 | |
| jroll | JayF: http://logs.openstack.org/ed/ed978f312e1079c6eb7166947253007d141eb82d/post/ironic-python-agent-buildimage-tinyipa/9e0eac7/console.html | 22:26 |
| JayF | 2016-05-11 21:43:14.405 | ./build-tinyipa.sh: line 89: mksquashfs: command not found | 22:26 |
| JayF | would be why it failed | 22:26 |
| jroll | didn't we go through this once already | 22:27 |
| JayF | I didn't help with the first set of building for tinyipa, so idk | 22:27 |
| jroll | https://github.com/openstack/ironic-python-agent/blob/189827e2b2b5f16713e46ba0f49af86f79baa81f/imagebuild/tinyipa/install-deps.sh | 22:28 |
| jroll | some backporting needed there | 22:28 |
| JayF | I'll find it | 22:28 |
| jroll | https://github.com/openstack/ironic-python-agent/commit/4230a3ba6e4360371e259f2b102bf4518134344e | 22:28 |
| JayF | or you will, that's fine | 22:28 |
| JayF | I'll just wait until you're done, lol | 22:28 |
| jroll | all I did was click last commit :P | 22:29 |
| *** dprince has quit IRC | 22:29 | |
| *** Goneri has joined #openstack-ironic | 22:29 | |
| JayF | jroll: so you can still +2 it -> https://review.openstack.org/315276 | 22:30 |
| JayF | jroll: for some reason it didn't notify i nhere | 22:30 |
| jroll | yeah, stable doesn't | 22:31 |
| jroll | JayF: cherry-pick -x | 22:31 |
| JayF | why -x? | 22:31 |
| jroll | to give the "cherry-picked from commit xyz" | 22:31 |
| JayF | so you need me to redo it? i can | 22:32 |
| jroll | yes pls | 22:32 |
| JayF | jroll: fixed https://review.openstack.org/#/c/315276/ | 22:34 |
| jroll | +2 | 22:36 |
| jroll | thanks | 22:36 |
| *** Fdaisuke has quit IRC | 22:39 | |
| *** rpioso has quit IRC | 22:41 | |
| *** mbound has joined #openstack-ironic | 22:41 | |
| JayF | who else has core on stable? | 22:43 |
| jlvillal | JayF: dtantsur|afk and devananda I think | 22:44 |
| JayF | dtantsur|afk: devananda: https://review.openstack.org/#/c/315276/ is needed to get tinyipa post job working on stable/mitaka | 22:44 |
| JayF | jlvillal: ty | 22:44 |
| jlvillal | You're welcome | 22:44 |
| *** Fdaisuke has joined #openstack-ironic | 22:46 | |
| *** xek has quit IRC | 22:46 | |
| *** xek has joined #openstack-ironic | 22:47 | |
| *** dmk0202 has quit IRC | 22:47 | |
| *** Fdaisuke has quit IRC | 22:52 | |
| *** aNupoisc has quit IRC | 22:53 | |
| mrda | JayF: You can bug tonyb and ask for help | 22:54 |
| JayF | mrda: oh hey, I have things you want | 22:55 |
| JayF | mrda: mainly pixie boots stickers | 22:55 |
| mrda | \op/ | 22:55 |
| JayF | mrda: assuming i need to send a few to you all down under, but you're all separte right? I can't send to you and have you be the deliveryman, right? | 22:55 |
| mrda | JayF: Yep, I can be the mailman | 22:56 |
| JayF | so how many total and pm me the address | 22:56 |
| *** rloo has quit IRC | 22:57 | |
| *** Fdaisuke has joined #openstack-ironic | 22:57 | |
| *** aNupoisc has joined #openstack-ironic | 22:59 | |
| JayF | tonyb: https://review.openstack.org/#/c/315276/ is needed to get our tinyipa post job working on stable/mitaka, which is a prereq for grenade tests (tinyipa is the only ramdisk small enough to run testing that significant). It's a simple cherry pick and would love to get it merged in and post job run before dtantsur|afk and sambetts|afk get back in | 23:00 |
| JayF | tonyb: mrda is the one who threw you under the jay-bus, btw :P | 23:01 |
| JayF | beeep beeeeeeep | 23:01 |
| mrda | lol, glad I could help! | 23:01 |
| *** r-daneel has quit IRC | 23:08 | |
| NobodyCam | lol | 23:09 |
| *** yuanying has joined #openstack-ironic | 23:10 | |
| *** sambetts|afk has quit IRC | 23:13 | |
| *** [1]cdearborn has joined #openstack-ironic | 23:13 | |
| *** zigo has quit IRC | 23:14 | |
| *** mrda has quit IRC | 23:15 | |
| *** mikal has quit IRC | 23:15 | |
| *** mikal has joined #openstack-ironic | 23:15 | |
| *** mrda has joined #openstack-ironic | 23:15 | |
| *** zigo has joined #openstack-ironic | 23:15 | |
| *** sambetts has joined #openstack-ironic | 23:15 | |
| tonyb | JayF: thanks, sorry I was dropping the kids at school | 23:21 |
| JayF | tonyb: no problem, I consider IRC async generally anyway :) | 23:21 |
| * tonyb will now commence to lying in the road for the bus to back over me ;P | 23:22 | |
| *** wajdi has quit IRC | 23:24 | |
| JayF | tonyb: it's an easy review ;) just would like to land it so the post job can run again before UK folks get here. Best case; it works; worst case is they can keep troubleshooting | 23:24 |
| tonyb | JayF: +W'd | 23:27 |
| JayF | tyvm | 23:28 |
| tonyb | JayF: it looks pretty low risk to me. | 23:28 |
| JayF | yeah; we don't even gate voting on tinyipa currently on stable | 23:29 |
| JayF | we just want the post job to work so it'll publish a stable/mitaka version of tinyipa | 23:29 |
| *** joprovost has joined #openstack-ironic | 23:29 | |
| clif_h | JayF: what do you think should happen if an image is cached to a node, but then a provision request for another image comes in? Currently IPA just overwrites the previous image. | 23:29 |
| JayF | clif_h: I don't think we should do that at all ever in the upstream case. | 23:30 |
| clif_h | IPA does that today! | 23:30 |
| JayF | clif_h: or we need some kind of "destroy metadata" call in IPA, like is used on iscsi deploys, to ensure all the bullshit is removed from that | 23:30 |
| JayF | clif_h: I know, and I hated it from the beginning if you remember, lol | 23:30 |
| *** [1]cdearborn has quit IRC | 23:31 | |
| clif_h | I remember you hated the idea of an image being written over another, and I understand why it could potentially be very bad. | 23:31 |
| JayF | yeah | 23:31 |
| clif_h | so, destroy metadata? | 23:32 |
| *** Sukhdev has joined #openstack-ironic | 23:32 | |
| JayF | if IPA is going to silently do that, it needs to call something to kill the metadata | 23:32 |
| clif_h | which metadata? where? | 23:32 |
| JayF | I think gdisk is doing that today in ironic-lib, there might be a method there to utilize | 23:32 |
| JayF | like on the disk | 23:32 |
| JayF | purge partition tables | 23:32 |
| JayF | raid metadata | 23:32 |
| JayF | etc | 23:32 |
| clif_h | but no, like, secure erase first? | 23:32 |
| JayF | I mean, that'd be good in a perfect world | 23:32 |
| JayF | but to enforce that in IPA, you'd essentially have to reclean the node | 23:33 |
| clif_h | but you're saying there's code available which kills the metadata? | 23:33 |
| clif_h | and that may be a 'good enough' solution? | 23:33 |
| JayF | mrda: It would take 12 stamps to ship you this envelope, I think I'm going to go by the post office instead, lol | 23:37 |
| JayF | Yeah; I mean the real question about this | 23:37 |
| JayF | would be if ever an image would cache with private data | 23:37 |
| JayF | if so; you have to reclean | 23:37 |
| JayF | if not, then just killing the metadata is secure enough | 23:37 |
| clif_h | right | 23:37 |
| JayF | IDK what assumptions OpenStack generally makes about images in those cases | 23:37 |
| clif_h | I mean, if you've got private data in an image, it should be encrypted right? | 23:38 |
| JayF | I mean, IDK | 23:38 |
| mrda | JayF: That's a lot of stamps | 23:39 |
| mrda | thanks tonyb! | 23:39 |
| JayF | yeah and apparently if they deem it too lumpy it would be $10 more to ship, and my stamps would be wasted. So I'll take it to the post office, no problem as I have Friday off anyway | 23:40 |
| tonyb | mrda: happy to help | 23:40 |
| mrda | "too lumpy" <-- is that a technical term? | 23:40 |
| JayF | Envelope contains items that cause more than 1/4 inch variation in thickness | 23:40 |
| JayF | is the exact way they put in | 23:40 |
| JayF | *it | 23:40 |
| mrda | :) | 23:40 |
| JayF | clif_h: did you want a pixie boots sticker? I have envelopes and stamps out now if you do | 23:42 |
| NobodyCam | mrda: "too lumpy" is a technical term for those who make gravy.. lol | 23:42 |
| * NobodyCam *ducks* | 23:43 | |
| clif_h | JayF: sure, I'll take one | 23:44 |
| mrda | :) | 23:46 |
| openstackgerrit | Clif Houck proposed openstack/ironic-specs: Add spec for image caching to nodes via agent driver https://review.openstack.org/310594 | 23:47 |
| clif_h | JayF: if you have time I'd like to get your opinion on the point Shivanand brought up about if a node should stay in the 'AVAILABLE' state while its caching or not | 23:52 |
| JayF | clif_h: It depends on the answer to the question of if arbitrary images can still be deployed to it | 23:52 |
| JayF | if so, then probably yes | 23:52 |
| JayF | if we decide to clean between images, then it should probably be a new state | 23:53 |
| JayF | well also | 23:53 |
| JayF | you want the node to go to another state while it's caching so... hmm | 23:53 |
| JayF | hard problem | 23:53 |
| clif_h | indeed | 23:53 |
| JayF | we def don't want to just leave it deployable and have the reschedule-while-caching problem we do downstream | 23:53 |
| clif_h | I believe he suggested a manual clean step? | 23:54 |
| JayF | Manual clean steps don't go back to AVAILABLE, iirc | 23:54 |
| JayF | they go managable -> clean -> manageable | 23:54 |
| JayF | http://docs.openstack.org/developer/ironic/deploy/cleaning.html#starting-manual-cleaning-via-api yep | 23:55 |
| JayF | you'd have to do something like AVAILABLE -> DEPLOYING -> AVAILABLE which would be weird | 23:56 |
| JayF | or add a new state | 23:56 |
| JayF | this is the sort of conversation devananda is pretty good at, tbh, I'd be curious what he'd think | 23:56 |
| JayF | I'm better at the technical details and less good at the api design :) | 23:56 |
| clif_h | yeah same | 23:56 |
| JayF | Also if you can't get traction on designing it in spec or IRC, you can put it on a meeting agenda | 23:57 |
| clif_h | hrm meetings | 23:58 |
| JayF | mrda: ^ if he has time might have an opinion too | 23:58 |
| JayF | and he has the freshest brain in the room, lol | 23:58 |
| clif_h | how do I put it on a meeting agenda? | 23:59 |
| clif_h | I mean, I see this https://wiki.openstack.org/wiki/Meetings/Ironic | 23:59 |
| JayF | edit the wiki page. but I'd maybe wait and see if this irc chat spurns any interest :) | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!