Thursday, 2019-01-03

*** mjturek has quit IRC00:19
*** rh-jelabarre has quit IRC00:47
openstackgerritJulia Kreger proposed openstack/ironic master: Fix IPv6 iPXE support  https://review.openstack.org/62800800:52
openstackgerritJulia Kreger proposed openstack/ironic master: WIP ipv6 job  https://review.openstack.org/61316901:02
*** sthussey has quit IRC01:05
TheJuliarpioso: fyi https://storyboard.openstack.org/#!/story/200451901:11
rpiosoTheJulia: y, I’m well aware of that.01:14
TheJuliarpioso: okay, thanks!01:14
rpiosoTheJulia: Thx for checkin :)01:17
rpiosoOn a different topic, I may have found you a system, as we discussed previously. I expect to connect you with the POC tomorrow.01:21
TheJuliarpioso: oh, awesome! Thanks!01:28
TheJuliaI'm going to call it a night shortly, have a wonderful evening01:28
*** rloo has quit IRC01:56
rpiosoU2!02:17
openstackgerritMerged openstack/ironic master: Allow disabling TFTP image cache  https://review.openstack.org/62512002:21
*** jrist has quit IRC03:07
openstackgerritDebayan Ray proposed openstack/ironic master: Fix SushyError namespacing in Redfish inspection  https://review.openstack.org/62809503:21
openstackgerritDebayan Ray proposed openstack/ironic master: Allow non-deterministic ``local_gb`` property  https://review.openstack.org/62809603:43
openstackgerritDebayan Ray proposed openstack/ironic master: Fix port creation post Redfish inspect  https://review.openstack.org/62809704:05
*** codyhill has joined #openstack-ironic05:29
codyhillHey folks. I’m coming along pretty well with Ironic (with nova). And now I’m trying to tackle using swap. I have added 64GB of Swap to my nova flavor. I then subtracted that space from the disk option. My node gets chosen and tries to boot. But I never get to the point in the boot process where it requests the config drive. And the system just hangs a the login prompt from the ramdisk. I have tested with Flavo05:39
codyhillwith 1GB larger and 1GB smaller disks. And The schedulure won’t select the node. So I think I have the flavor correct. I see documentation a lot of places on how to debug the inspector ram disk by passing a root password hash or ssh key. But how can I do that for normal system provisions? I’d like to login and see where this thing is hung up. Any help would be appricated.05:39
codyhillHa! Well… I about the 10th time I tried this… I decided to just work. (I tried again to collect logs to post a link to here) Hmm…06:08
codyhillAnd it’s late so I can’t type properly… It* decided to work.06:09
*** bfournie has quit IRC06:15
codyhillWell… Trying again failed. Here are the conductor logs: https://ghostbin.com/paste/mxhq806:40
codyhillA little more info about the flavor, node, & instance… https://ghostbin.com/paste/edbzn06:46
codyhillAlso this is OpenStack Newton06:46
*** jtomasek has joined #openstack-ironic06:49
*** rcernin has quit IRC07:09
iurygregorygood morning all =D07:23
etingofiurygregory, o/07:25
iurygregoryetingof, o/ cold day yay07:26
* etingof is not sure if the weather is good for biking today07:26
iurygregory-4 feels like -9 lol07:27
*** hwoarang has quit IRC07:47
*** hwoarang has joined #openstack-ironic07:50
*** hjensas has quit IRC07:54
*** moshele has joined #openstack-ironic08:05
*** codyhill has quit IRC08:07
openstackgerritIury Gregory Melo Ferreira proposed openstack/networking-baremetal master: [DNM] Tests  https://review.openstack.org/62467108:29
openstackgerritDigambar proposed openstack/ironic stable/queens: Fix OOB introspection to use pxe_enabled flag in idrac driver  https://review.openstack.org/62438808:35
*** dtantsur|afk is now known as dtantsur08:59
*** robbbe has joined #openstack-ironic09:01
openstackgerritparesh sao proposed openstack/ironic master: OOB RAID implementation for ilo5 based HPE Proliant servers.  https://review.openstack.org/55566409:04
dtantsurmorning ironic09:04
iurygregorymorning dtantsur o/09:05
*** derekh has joined #openstack-ironic09:18
*** hjensas has joined #openstack-ironic09:27
*** e0ne has joined #openstack-ironic09:42
*** dkehn has quit IRC11:04
*** bfournie has joined #openstack-ironic12:03
*** S4ren has joined #openstack-ironic12:18
openstackgerritIlya Etingof proposed openstack/sushy-tools master: Add configuration file to sushy-emulator  https://review.openstack.org/61617812:18
openstackgerritIlya Etingof proposed openstack/sushy-tools master: Add memoization to expensive emulator calls  https://review.openstack.org/61275812:20
openstackgerritIlya Etingof proposed openstack/sushy-tools master: Limit instances exposure  https://review.openstack.org/61651612:22
openstackgerritIlya Etingof proposed openstack/sushy-tools master: Add configurable libvirt firmware  https://review.openstack.org/62060512:23
*** rh-jelabarre has joined #openstack-ironic12:49
*** jcoufal has joined #openstack-ironic13:11
*** trown|outtypewww is now known as trown13:11
*** ianychoi has joined #openstack-ironic13:17
*** derekh has quit IRC13:24
TheJuliagood morning13:42
iurygregoryTheJulia, morning o/13:43
iurygregoryi have news about networking-baremetal for you \o/13:44
iurygregoryi made it work in a devstack env and tempest is green \o/13:44
etingofiurygregory++13:44
iurygregory=D13:44
etingofgood morning, TheJulia o/13:44
iurygregorysnow helped today lol13:45
* etingof is interested in the details of the fix (though has to run now)13:47
iurygregoryvariables missing in the local.conf13:48
iurygregoryjsut double checking if all i added are really necessary now XD13:48
dtantsurTheJulia: morning! would you be fine with approving https://review.openstack.org/#/c/624119/ as it is?13:49
patchbotpatch 624119 - ironic - Allocation API: database and RPC - 6 patch sets13:49
TheJuliaI think so...13:52
openstackgerritJulia Kreger proposed openstack/ironic master: Fix IPv6 iPXE support  https://review.openstack.org/62800813:54
openstackgerritJulia Kreger proposed openstack/ironic master: WIP ipv6 job  https://review.openstack.org/61316913:54
openstackgerritHamdy Khader proposed openstack/ironic master: Add support for Smart NICs  https://review.openstack.org/58357313:54
TheJuliadtantsur: I got ipxe to get the proper next filename on ipv6 :) \o/13:55
dtantsur\o/13:55
TheJuliav6 routing seems to be... problematic though, but one step closer13:56
TheJuliaI think uefi boot from url is largely going to be us trying to figure out how we're going to test it more than anything else....13:57
dtantsurit has always been the case with UEFI, indeed13:57
TheJuliayeah, I looked at the firmware image out there and it appears not to support it by default, and of course it doesn't have the best documentation :(14:02
TheJuliabut, we've got the key elements, delineation and identification is the key next14:02
*** mmethot has joined #openstack-ironic14:03
openstackgerritDmitry Tantsur proposed openstack/ironic master: Allocation API: minor fixes to DB and RPC  https://review.openstack.org/62818714:03
dtantsurTheJulia, kaifeng: follow-up posted ^^14:03
*** jistr is now known as jistr|mtg14:04
*** derekh has joined #openstack-ironic14:04
*** sthussey has joined #openstack-ironic14:09
*** rloo has joined #openstack-ironic14:11
openstackgerritIury Gregory Melo Ferreira proposed openstack/networking-baremetal master: [DNM] Tests  https://review.openstack.org/62467114:13
TheJuliadtantsur: parent patch wf+114:16
dtantsurthnx14:16
*** dkehn has joined #openstack-ironic14:16
openstackgerritHamdy Khader proposed openstack/ironic master: Add support for Smart NICs  https://review.openstack.org/58357314:17
openstackgerritMark Goddard proposed openstack/bifrost master: Always use a virtualenv in CI testing  https://review.openstack.org/62819614:51
TheJuliadtantsur: can you run the meeting on the 14th?14:53
TheJuliaI will be in metal tube on approach to an airport during the meeting14:53
TheJuliawhich I've had problems with the wifi on the aircraft before14:53
*** mjturek has joined #openstack-ironic14:54
*** jistr|mtg is now known as jistr14:55
dtantsurTheJulia: I think I can, yes14:56
TheJuliaI will likely also need someone to run the 28th14:56
TheJuliasince I'll be in Brno14:57
*** Goneri has joined #openstack-ironic14:57
*** hjensas has quit IRC14:59
dtantsurTheJulia: I'll probably be back (returning on the 27th)15:05
*** moshele has quit IRC15:06
*** Bhujay has joined #openstack-ironic15:15
*** Bhujay has quit IRC15:16
iurygregoryCI seems happy yay https://review.openstack.org/#/c/624671/ \o/15:22
patchbotpatch 624671 - networking-baremetal - [DNM] Tests - 10 patch sets15:22
openstackgerritRuby Loo proposed openstack/ironic master: Fix typo in release note  https://review.openstack.org/62820315:26
*** e0ne has quit IRC15:28
openstackgerritIury Gregory Melo Ferreira proposed openstack/networking-baremetal master: Change networking-baremetal to zuulv3/python3  https://review.openstack.org/61402615:28
openstackgerritIury Gregory Melo Ferreira proposed openstack/networking-baremetal master: Change networking-baremetal to zuulv3/python3  https://review.openstack.org/61402615:38
openstackgerritMark Goddard proposed openstack/bifrost master: Use a virtualenv in openSUSE CI testing  https://review.openstack.org/62819615:43
*** hjensas has joined #openstack-ironic15:44
openstackgerritMark Goddard proposed openstack/bifrost master: Use a virtualenv in openSUSE CI testing  https://review.openstack.org/62819615:45
openstackgerritMark Goddard proposed openstack/bifrost master: Use a virtualenv in openSUSE CI testing  https://review.openstack.org/62819615:48
openstackgerritMerged openstack/ironic master: Fix typo in release note  https://review.openstack.org/62820316:07
openstackgerritMark Goddard proposed openstack/ironic master: WIP: Deploy templates: data model, DB API & objects  https://review.openstack.org/62766316:17
*** codyhill has joined #openstack-ironic16:21
codyhillGood Morning.16:21
codyhillI posted an issue last night. But my client quite due to a Mac update… So I’m not sure if I got a response. Is this channel archied anywhere that would allow me to see history?16:23
codyhillAnyhow I’ll ask my question again.16:25
codyhillI’m trying to get the swap partion created using Ironic and Nova. I’ve created the nova flavor with the correct swap. And subtracted that swap from the Disk in the flavor. The swap and disk add up to local_gb. When I boot this system it never gets past booting into the ram disk and getting an IP. The next action I’d expect to see is the config-drive get mounted and copied over… But that doesn’t happen.16:32
codyhill(Although I did get this to happen randomly when I was trying to reporduce the issue. But I couldn’t get it to work again.) Here are the logs: https://ghostbin.com/paste/7pcn3 Here are is some info on the flavor, node, & instance: https://ghostbin.com/paste/jy5wd And this is OpenStack Newton by the way.16:32
TheJuliagood morning codyhill16:33
TheJuliacodyhill: eavesdrop.openstack.org16:33
codyhillWhat I’d really like to know how to do is log into the RAM disk while this thing is hung so that I can debug. I see a way to do this for the ironic inspector by modifying the pxe command line flags. But not during a provision.16:33
codyhillGood Moring TheJulia!16:34
TheJuliaIt is just never network booting for the deployment?16:34
codyhillWe are copying to disk and then booting local after. But it isn’t getting that far.16:35
TheJuliayeah, looks like the ramdisk was never able to reach the API to heartbeat based upon the logs you provided.16:36
openstackgerritMark Goddard proposed openstack/bifrost master: Use a virtualenv in openSUSE CI testing  https://review.openstack.org/62819616:36
codyhillIt is PXE booting. And hangs at the screen that show the login prompt and IPs16:36
TheJuliaso yeah, sounds like purely networking. Your using flat networking right?16:36
codyhillYes. And if i change the flavor to not include swap...16:36
codyhillIt works great.16:36
codyhillThis is Ubuntu-16.04 BTW.16:37
codyhillLet me see If I can ping the instance when it’s in this state. Normally I can.16:39
TheJuliacodyhill: do you see anything reported to the console by IPA?16:39
codyhillIt only shows the IPs that the interfaces get and the login prompt.16:40
codyhillNormally I see a bunch of things flashing.16:40
codyhillANd it prints things.16:40
codyhillBut not when I hit this.16:40
TheJuliaFor some reason I thought swap was GB. I could only guess that maybe IPA is booting up, performing its lookup, and there maybe an exception in the properties.16:44
TheJuliasomehow, your going to need to troubleshoot from with-in the ramdisk16:44
codyhillYes. How do I troubleshoot within the Ramdisk? I see how to do this via inspector. But I couldn’t find documentation on debugging this with a normal provision.16:45
iurygregoryTheJulia, newbie question, tempest tests in python3 take more time than in python2?16:46
iurygregorynetworking-baremetal-multitenant-vlans-python2 is green in https://review.openstack.org/#/c/614026/ but python3 i got a timeout =(16:46
patchbotpatch 614026 - networking-baremetal - Change networking-baremetal to zuulv3/python3 - 16 patch sets16:46
TheJuliaits mb16:47
TheJuliacodyhill: typically you make a special ramdisk with a user built in, and possibly even perform a manual execution of ironic-python-agent starting up in the ramdisk so you can see on a terminal what it is doing. Sure looks like it is on a short deployment timeout before it gave up and powered off the node. You might want to also check your API log to see what, if anything is occurring ironic side, but as far as I can16:50
TheJuliasee from what you provided, IPA is simply never heartbeating... and there doesn't appear to be any reason that would cause that centered around swap in the newton-eol tagged code16:50
codyhillTheJulia: The newton documentation doens’t specify MB or GB. But Pike specifies MB.16:50
TheJuliacodyhill: yeah, I just confirmed it was MB16:50
TheJuliaiurygregory: it really shouldn't take longer. There may be something happening out of immediate view which is taking longer though :\16:51
iurygregory20min =( and boom timeout hehe16:52
iurygregorymaybe the host was slow i dont know16:52
TheJuliarepeatedly, it shouldn't be16:53
codyhillThe Julia: I’m using: `DIB_CLOUD_INIT_DATASOURCES="Ec2, ConfigDrive, OpenStack" DIB_RELEASE=xenial disk-image-create baremetal grub2 ubuntu dhcp-all-interfaces cloud-init-datasources -o ubuntu-16.04` to create my image for this. Do I need to mount one of these disk and inject a Key? Or can I pass an ENV variable to get one inected for the custom ram disk?16:53
TheJuliawhere is it timing out, same place? repeatable?16:53
TheJuliacodyhill: That would be for the image your deploying, not the deployment ramdisk that performs the deployment16:54
codyhillThis generates a ram disk as well.16:54
iurygregoryin python2 didnt timeout with the fix in local.conf maybe python3 will need more time or not16:54
codyhillI then link them together in Glance.16:55
TheJuliayou'll need something like DIB_RELEASE=xenial disk-image-create ubuntu ironic-agent devuser -o ipa16:55
TheJuliayou could edit your ironic-agent element to disable ipa start-up upon boot, and there are other devuser element flags you'll want to set to give yourself a user to login to16:55
iurygregoryhttp://paste.openstack.org/show/739878/16:55
codyhillOk. My cleaning and ironic inspector image was built like that...16:55
codyhillTheJulia: Thanks. I’ll look at docs and see what I can figure out for those flags.16:56
NobodyCamGood Morning Ironic'ers17:00
iurygregorymorning NobodyCam17:00
NobodyCammorning iurygregory :)17:03
iurygregoryTheJulia, do you think i should try a recheck since only the BasicOps failed and was timeout? before was BasicOps and Multienancy failling =) http://logs.openstack.org/26/614026/16/check/networking-baremetal-multitenant-vlans/6b14293/testr_results.html.gz17:08
openstackgerritDmitry Tantsur proposed openstack/ironic master: [WIP] Allocation API: implementation without HA and take over  https://review.openstack.org/62822317:09
iurygregoryand python2 all green \o/ we almost there yay17:09
*** robbbe has quit IRC17:17
*** gyee has joined #openstack-ironic17:25
TheJuliaiurygregory: I'd prefer that you try to understand why it timed out, what was state things were at when it timed out. Was it writing the image, or just bad timing between a status check and the test and deployment17:27
* TheJulia steps away to deal with some family drama for a few minutes17:27
* dtantsur steps away for some exercising17:32
*** dkehn has left #openstack-ironic17:33
openstackgerritRuby Loo proposed openstack/ironic stable/rocky: Allow disabling instance image cache  https://review.openstack.org/62822917:33
TheJuliaexercise+++++++17:37
*** trown is now known as trown|lunch17:41
TheJulialunch, migraine meds, and some exercise sound like a perfect idea17:42
openstackgerritMark Goddard proposed openstack/bifrost master: Perform ironic online data migrations  https://review.openstack.org/62751317:53
*** S4ren has quit IRC18:03
openstackgerritDerek Higgins proposed openstack/ironic master: Testing UEFI job on bionic  https://review.openstack.org/62675518:03
*** derekh has quit IRC18:03
*** robbbe has joined #openstack-ironic18:08
*** MattMan has quit IRC18:09
*** MattMan has joined #openstack-ironic18:09
*** bnemec has quit IRC18:13
*** bnemec has joined #openstack-ironic18:26
*** bnemec has quit IRC18:30
*** bnemec has joined #openstack-ironic18:34
*** bnemec has quit IRC18:39
openstackgerritDmitry Tantsur proposed openstack/ironic master: [WIP] Allocation API: implementation without HA and take over  https://review.openstack.org/62822318:43
*** dtantsur is now known as dtantsur|afk18:43
dtantsur|afksee you tomorrow18:43
*** bnemec has joined #openstack-ironic18:52
*** bnemec has quit IRC18:58
*** trown|lunch is now known as trown19:00
openstackgerritJulia Kreger proposed openstack/ironic master: WIP ipv6 job  https://review.openstack.org/61316919:00
TheJuliahopefully that solves it. Better aligns with my local config19:01
*** bnemec has joined #openstack-ironic19:14
codyhillTheJulia: The issue looks to have been caused by the second Nic in the server getting a DHCP address after the primary NIC. Which lead the system to not beable to route to the ironic-api. I’ve reconfigured the switches to disable the PVID on NIC2 and only have it on NIC1 and that seems to have solved the issue. {fingers crossed} I’m gonig to test a few more times becuase it randomly worked a few times as wellâ19:24
*** irclogbot_1 has quit IRC19:24
codyhillAnd it always worked without swap. I’m not sure if adding a swap partion allowed the race condition to happen? But looks like this is resolved.19:24
*** irclogbot_1 has joined #openstack-ironic19:26
TheJuliacodyhill: ahh, yeah that is a fun nice race condition because the default gw would be changed to who knows what19:30
codyhillYep!19:30
TheJuliait shouldn't of, but race conditions can sometimes be like that19:30
codyhillThanks for the help! I wouldn’t have figured it out without the new ramdisk taht I could loginto.19:36
TheJulia:)19:51
TheJuliacodyhill: happy to help19:51
*** e0ne has joined #openstack-ironic20:21
*** robbbe has quit IRC20:34
*** e0ne has quit IRC20:37
*** e0ne has joined #openstack-ironic20:54
*** moshele has joined #openstack-ironic21:01
rpiosoGood late morning :)21:05
mjturekTheJulia: fyi looked a bit at the job today, and from  a deployment standpoint it looks good..21:14
mjturekpkvmci job that is21:14
mjturekso, need to log into the dsvm and see if something's going wrong there21:14
TheJuliamjturek: ack. It looked like it might have been devstack-gate's doing21:18
rpiosomjturek: Congratulations!21:33
mjturekthank you rpioso!!21:33
rpioso:)21:33
mjturekTheJulia: hmmm didn't notice that21:33
TheJuliamjturek: entirely possible that was one of the other ci systems21:50
mjturekfair enough!21:50
*** e0ne has quit IRC21:50
TheJuliarpioso: Got your email, thanks! I don't have the mental spoons to reply today, but will tomorrow21:52
*** e0ne has joined #openstack-ironic21:52
*** e0ne has quit IRC21:57
rpiosoTheJulia: yw. Hope it helps.21:58
*** rloo has quit IRC22:07
*** rloo has joined #openstack-ironic22:10
*** trown is now known as trown|outtypewww22:13
*** hjensas is now known as hjensas|brb22:16
*** rcernin has joined #openstack-ironic22:42
*** rloo has quit IRC22:43
openstackgerritJulia Kreger proposed openstack/ironic master: WIP ipv6 job  https://review.openstack.org/61316922:43
*** rloo has joined #openstack-ironic22:44
TheJuliait occurs to me that I think derek is kind of fighting the same issue in the neutron plugin init stuffs :(22:48
*** jcoufal has quit IRC22:49
TheJuliaMy brain has reached friend. I guess I'll keep trying on that later so hopefully we can get a successful CI run of it :\22:51
TheJuliafried22:54
TheJuliareached fried22:54
* TheJulia just gets a laugh out of it22:54
openstackgerritIlya Etingof proposed openstack/sushy master: Add System<->Manager linkage  https://review.openstack.org/62166323:08
*** moshele has quit IRC23:17
*** mmethot has quit IRC23:22
openstackgerritIlya Etingof proposed openstack/sushy master: Add System<->Manager linkage  https://review.openstack.org/62166323:23
*** jcoufal has joined #openstack-ironic23:47

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!