Monday, 2018-02-19

*** hshiina has joined #openstack-ironic00:46
*** ptx0 has quit IRC01:13
*** ptx0 has joined #openstack-ironic01:18
*** zshi has joined #openstack-ironic01:50
*** links has joined #openstack-ironic02:11
*** r-daneel has joined #openstack-ironic02:24
*** yangyapeng has quit IRC02:30
*** r-daneel has quit IRC02:32
*** r-daneel has joined #openstack-ironic03:40
*** yangyapeng has joined #openstack-ironic03:44
*** dpawar has joined #openstack-ironic03:49
*** dpawar_ has joined #openstack-ironic03:58
*** dpawar has quit IRC04:02
*** dpawar_ has quit IRC04:09
*** khyr0n has joined #openstack-ironic04:37
*** olivierbourdon38 has joined #openstack-ironic05:04
*** dpawar has joined #openstack-ironic05:11
*** dpawar has quit IRC05:12
*** dpawar has joined #openstack-ironic05:12
openstackgerritMerged openstack/ironic master: Allow sqalchemy filtering by id and uuid  https://review.openstack.org/54504105:24
*** dpawar has quit IRC05:31
*** rbartal has joined #openstack-ironic06:02
*** threestrands has quit IRC06:10
*** rbartal has quit IRC06:18
*** threestrands has joined #openstack-ironic06:23
*** jaganathan has joined #openstack-ironic06:46
*** actatux has quit IRC06:46
*** rbartal has joined #openstack-ironic06:48
*** dpawar has joined #openstack-ironic06:52
*** threestrands has quit IRC07:00
*** mjura has joined #openstack-ironic07:02
*** rcernin has quit IRC07:11
*** e0ne has joined #openstack-ironic07:11
*** vsaienk0 has joined #openstack-ironic07:18
*** e0ne has quit IRC07:29
*** hoonetorg has quit IRC07:30
*** vsaienk0 has quit IRC07:32
*** vsaienk0 has joined #openstack-ironic07:34
*** e0ne has joined #openstack-ironic07:35
*** hoonetorg has joined #openstack-ironic07:46
*** pcaruana has joined #openstack-ironic07:47
*** e0ne has quit IRC07:47
olivierbourdon38Hello everyone. Any known regression lately ? I was configuring ironic with keystone using Bifrost and everything was fine end of last week and today I get 2018-02-19 08:45:17.718 22458 ERROR keystone ImportError: (u'Unable to find %(name)r driver in %(namespace)r.', {'namespace': 'keystone.token.provider', 'name': 'uuid'})07:48
*** vsaienk0 has quit IRC07:51
*** links has quit IRC07:55
*** dpawar has quit IRC07:56
*** AlexeyAbashkin has joined #openstack-ironic07:56
*** dpawar has joined #openstack-ironic07:56
pas-haolivierbourdon38: I suspect it might have been removed from keystone, together with v2 API. lemme check07:58
olivierbourdon38pas-ha thx for your help and time07:58
*** dpawar_ has joined #openstack-ironic07:59
*** dpawar has quit IRC07:59
pas-hayep, it is gone https://review.openstack.org/#/c/543060/08:00
patchbotpatch 543060 - keystone - Remove the sql token driver and uuid token provider (MERGED)08:00
pas-haolivierbourdon38: ^08:00
*** MattMan has quit IRC08:02
*** MattMan has joined #openstack-ironic08:03
*** vsaienk0 has joined #openstack-ironic08:04
mkraiHi, I setup a latest devstack setup for Ironic. I tried to boot from Nova but it failed with "No valid host found" error08:06
mkraiLogs http://paste.openstack.org/show/677483/08:06
mkraiI am not sure how the placement api works.08:06
*** e0ne has joined #openstack-ironic08:06
mkraiCan anyone help me with this?08:06
olivierbourdon38pas-ha08:07
*** links has joined #openstack-ironic08:07
olivierbourdon38pas-ha got it but this is still strange as I am pretty sure I was able to deploy stuff after Feb 13th ....08:08
*** e0ne has quit IRC08:08
*** tesseract has joined #openstack-ironic08:15
*** racedo has joined #openstack-ironic08:17
*** mrunge has quit IRC08:17
mkraipas-ha: TheJulia ^^08:21
olivierbourdon38currently digging out what could be responsible for still calling this removed driver08:24
pas-haolivierbourdon38: I think bifrost sets up keystone using uuid tokens08:30
pas-haolivierbourdon38: https://github.com/openstack/bifrost/blob/master/playbooks/roles/bifrost-keystone-install/templates/keystone.conf.j2#L2608:31
*** mrunge has joined #openstack-ironic08:31
pas-haI suspect this is due to the fact that fernets are a bit more envolved to set up correctly, but now it seems there's no other way :-/08:32
pas-hainvolved08:32
olivierbourdon38thx, you were quicker than I was on this ;-)08:33
*** stendulker has joined #openstack-ironic08:33
*** krtaylor has quit IRC08:38
olivierbourdon38currently trying a fix for this issue in bifrost. Thx again for the help08:44
*** dpawar_ has quit IRC08:52
*** dpawar has joined #openstack-ironic08:53
mkraiHi pas-ha08:53
pas-hamkrai: me neither (sure how placement API works)08:56
mkraipas-ha: oh ok no problem. Who can help with this?09:01
*** jtomasek has joined #openstack-ironic09:03
olivierbourdon38so trying my patch for using fernets instead of uuid, I ended up in another issue09:11
olivierbourdon38        "2018-02-19 08:37:33.325 21884 ERROR keystone.notifications [-] Failed to construct notifier: ImportError: No module named pyngus",09:11
olivierbourdon38        "2018-02-19 08:37:33.325 21884 ERROR keystone.notifications     import pyngus",09:11
olivierbourdon38        "2018-02-19 08:37:33.325 21884 ERROR keystone.notifications ImportError: No module named pyngus",09:11
olivierbourdon38which I solved by adding pyngus to the keystone requirements.txt file but I do not understand why it is not there by default09:12
*** derekh has joined #openstack-ironic09:21
*** mrch has joined #openstack-ironic09:22
*** mgoddard_ has joined #openstack-ironic09:23
*** vsaienk0 has quit IRC09:23
*** r-daneel has quit IRC09:28
*** links has quit IRC09:32
*** vsaienk0 has joined #openstack-ironic09:39
openstackgerritOlivier Bourdon proposed openstack/bifrost master: WIP: Add check for (i)PXE and Ironic location coherence  https://review.openstack.org/54571109:40
*** dpawar has quit IRC09:42
*** links has joined #openstack-ironic09:45
*** r-daneel has joined #openstack-ironic09:46
*** mrch has quit IRC09:51
*** e0ne has joined #openstack-ironic09:54
*** vsaienk0 has quit IRC09:55
*** vsaienk0 has joined #openstack-ironic09:57
*** serlex has joined #openstack-ironic10:05
*** serlex has quit IRC10:05
*** r-daneel has quit IRC10:06
*** vsaienk0 has quit IRC10:07
*** vsaienk0 has joined #openstack-ironic10:08
*** r-daneel has joined #openstack-ironic10:22
*** mrch has joined #openstack-ironic10:26
*** vsaienk0 has quit IRC10:34
openstackgerritOlivier Bourdon proposed openstack/bifrost master: WIP: Add check for (i)PXE and Ironic location coherence  https://review.openstack.org/54571110:35
*** hshiina has quit IRC10:50
*** mrch has quit IRC10:54
*** vsaienk0 has joined #openstack-ironic10:57
*** zshi has quit IRC10:59
*** AlexeyAbashkin has quit IRC11:07
*** stendulker has quit IRC11:18
*** lucas-afk is now known as lucasagomes11:18
*** priteau has joined #openstack-ironic11:19
*** krtaylor has joined #openstack-ironic11:23
*** dtantsur|afk is now known as dtantsur11:23
pas-haolivierbourdon38: looks like pungus is a test requirement for keystone http://codesearch.openstack.org/?q=pyngus&i=nope&files=requirements.txt&repos=11:23
pas-has/keystone/oslo.messaging11:24
pas-haso probably keystone is set up with this backend (as default for testing) for notifications11:24
dtantsurmorning ironic11:24
pas-hamorning dtantsur11:25
pas-haQueens is definitely a fun release for Keystone - no more V2 API, no more UUID tokens... slim and clean :)11:26
*** dpawar has joined #openstack-ironic11:26
*** AlexeyAbashkin has joined #openstack-ironic11:32
openstackgerritDmitry Tantsur proposed openstack/ironic stable/queens: Allow sqalchemy filtering by id and uuid  https://review.openstack.org/54580611:34
dtantsurpas-ha: oh cool :)11:36
*** bnemec is now known as bnemec-pto11:41
openstackgerritDmitry Tantsur proposed openstack/ironic stable/queens: Rework exception handling on deploy failures in conductor  https://review.openstack.org/54580711:44
olivierbourdon38pas-ha however when bifrost deployment calls https://github.com/openstack/bifrost/blob/12ef3cbbadb638e363a73f3c269b3138de740ef0/playbooks/roles/bifrost-keystone-install/tasks/bootstrap.yml#L138 it "fails" even though the deployment just goes on11:47
*** vsaienk0 has quit IRC11:53
*** vsaienk0 has joined #openstack-ironic11:57
*** masber has joined #openstack-ironic11:58
*** mrch has joined #openstack-ironic12:01
openstackgerritDmitry Tantsur proposed openstack/ironic master: Build instance PXE options for unrescue  https://review.openstack.org/54427812:05
*** slagle has joined #openstack-ironic12:20
*** dmellado has quit IRC12:26
*** vsaienk0 has quit IRC12:28
*** mrch has quit IRC12:28
*** ricardoas has quit IRC12:33
*** vsaienk0 has joined #openstack-ironic12:33
*** nicodemos has quit IRC12:35
*** links has quit IRC12:37
*** milan_ has joined #openstack-ironic12:50
*** rbartal has quit IRC12:53
*** dprince has joined #openstack-ironic12:57
*** jlabarre has joined #openstack-ironic13:03
*** deadnull has joined #openstack-ironic13:04
*** dmellado has joined #openstack-ironic13:07
*** deadnull has quit IRC13:10
*** bfournie has quit IRC13:17
*** bfournie has joined #openstack-ironic13:18
*** mrch has joined #openstack-ironic13:21
*** bfournie has quit IRC13:22
*** skazi has quit IRC13:23
*** skazi has joined #openstack-ironic13:24
openstackgerritJulia Kreger proposed openstack/ironic stable/queens: DNM Test disabling .pyc files for Grenade  https://review.openstack.org/54508913:24
*** dpawar has quit IRC13:25
TheJuliaGood morning ironic13:25
dtantsurmorning TheJulia13:28
*** mrch has quit IRC13:32
openstackgerritMark Goddard proposed openstack/ironic master: Register traits on nodes in devstack  https://review.openstack.org/54537013:32
openstackgerritMark Goddard proposed openstack/ironic master: Support nested objects and object lists in as_dict  https://review.openstack.org/54583013:32
openstackgerritMark Goddard proposed openstack/ironic-tempest-plugin master: Validate scheduling fields in basic ops scenario  https://review.openstack.org/54529713:33
*** dmellado has quit IRC13:38
*** mgoddard_ has quit IRC13:39
*** trown|outtypewww is now known as trown13:42
*** ricardoas has joined #openstack-ironic13:42
*** jcoufal has joined #openstack-ironic13:44
* dtantsur reached 0 emails in his inbox, w00t!13:46
dtantsurlucasagomes: hey! I guess you won't mind if I take over https://bugs.launchpad.net/ironic/+bug/1614094 will you?13:48
openstackLaunchpad bug 1614094 in Ironic "Updating the port address for an active instance does not regenerate the PXE configuration" [High,Triaged] - Assigned to Lucas Alvares Gomes (lucasagomes)13:48
dtantsurlucasagomes: ditt for https://bugs.launchpad.net/ironic/+bug/158890113:48
openstackLaunchpad bug 1588901 in Ironic "Nodes stuck on power state transitions" [High,Triaged] - Assigned to Lucas Alvares Gomes (lucasagomes)13:48
*** rbartal has joined #openstack-ironic13:48
lucasagomesdtantsur, not all all, go for it :D13:49
dtantsurcool13:49
*** bfournie has joined #openstack-ironic13:51
jrollmorning y'all \o13:53
lucasagomesjroll, morning13:53
*** masuberu has joined #openstack-ironic13:54
*** vsaienk0 has quit IRC13:56
*** masber has quit IRC13:58
dtantsurmorning jroll14:00
*** vsaienk0 has joined #openstack-ironic14:01
jlvillalGood morning Ironic14:02
jlvillaldtantsur, If you have time to look at: https://review.openstack.org/#/c/524321/14:02
patchbotpatch 524321 - ironic - Fix issue with double mocking of utils.execute fun...14:02
jlvillalAn old patch...14:02
dtantsurmorning jlvillal, putting on my list14:03
jlvillaldtantsur, Thanks :)14:03
jlvillaldtantsur, I'll give you my secret ~/.gitconfig alias14:05
jlvillal    frbi = !git rebase -i $(git log --pretty=oneline --color=always | fzf --ansi | cut -d ' ' -f1)^14:05
jlvillal:)14:05
*** rbartal has quit IRC14:05
*** etingof has quit IRC14:12
*** mrch has joined #openstack-ironic14:12
*** mrch has left #openstack-ironic14:13
* dtantsur tries14:14
dtantsurheh nice!14:15
jlvillaldtantsur, :)14:15
jlvillaldtantsur, 'rbi' = rebase --interactive14:16
jlvillaldtantsur, so 'frbi' is with fzf14:16
jlvillalNow I rarely use 'rbi' since so easy with 'frbi'14:17
dtantsuryeah, that's cool. especially since I always do interactive rebases14:17
dtantsurunrelated, but can someone please review https://review.openstack.org/545069 ?14:17
patchbotpatch 545069 - ironic - Fix rare HTTP 400 from port list API14:17
jlvillaldtantsur, Done. I reviewed it over the weekend and left a comment14:18
jlvillaldtantsur, But not that important of a comment14:18
dtantsurjlvillal: yeah, but good to know, thanks14:18
jlvillalYou're welcome14:18
*** rbartal has joined #openstack-ironic14:19
*** Goneri has joined #openstack-ironic14:20
*** mjturek has joined #openstack-ironic14:24
*** etingof has joined #openstack-ironic14:26
*** mgoddard_ has joined #openstack-ironic14:26
*** jistr is now known as jistr|mtg14:31
*** baha has joined #openstack-ironic14:33
*** bfournie1 has joined #openstack-ironic14:38
*** bfournie has quit IRC14:39
*** links has joined #openstack-ironic14:41
*** dmellado has joined #openstack-ironic14:44
*** jroll has quit IRC14:47
*** jroll has joined #openstack-ironic14:48
*** links has quit IRC14:49
*** hjensas has quit IRC14:50
*** dtantsur is now known as dtantsur|brb15:01
*** links has joined #openstack-ironic15:02
*** dmellado has quit IRC15:02
*** jistr|mtg is now known as jistr15:02
*** rbartal has quit IRC15:06
-openstackstatus- NOTICE: Zuul has been restarted to pick up latest memory fixes. Queues were saved however patches uploaded after 14:40UTC may have been missed. Please recheck if needed.15:15
*** sovsianikov has joined #openstack-ironic15:17
openstackgerritJim Rollenhagen proposed openstack/ironic stable/queens: Allow sqalchemy filtering by id and uuid  https://review.openstack.org/54580615:31
openstackgerritJim Rollenhagen proposed openstack/ironic stable/pike: Allow sqalchemy filtering by id and uuid  https://review.openstack.org/54585415:32
openstackgerritJim Rollenhagen proposed openstack/ironic stable/ocata: Allow sqalchemy filtering by id and uuid  https://review.openstack.org/54585515:32
jrollwoo, three clean cherry picks15:33
jrolldtantsur|brb: ^ probably want to at least hit queens when you have a minute15:33
*** lucasagomes is now known as lucas-hungry15:34
*** devananda has joined #openstack-ironic15:46
*** deadnull has joined #openstack-ironic15:47
sovsianikovHi there, Guys, I need your help/advice.15:47
sovsianikovI deploy Overcloud on my test environment and specify the following serail number of the disk: "serial": "WD-WMC1U5346656", but probably Nova can't use the disk and I have the Error No suitable device was found for deployment using these hints {u'serial': u'WD-WMC1U5346656'}", u'code': 404, u'type': u'DeviceNotFound', u'details':15:47
sovsianikovu"No suitable device was found for deployment using these hints {u'serial': u'WD-WMC1U5346656'}"}15:47
sovsianikovMore logs can be found here http://paste.openstack.org/show/677599/15:47
sovsianikovCould you please take a look at the Error and let me know which parameter should I use instead of serial number of the disk in order to deploy Overcloud?15:48
openstackgerritMarkos Chandras (hwoarang) proposed openstack/bifrost master: Test OPNFV CI verify job. Please ignore.  https://review.openstack.org/36760815:50
*** dtantsur|brb is now known as dtantsur15:54
dtantsurjroll: oh, I also proposed queens :) anyway, I'm fine with merging yours15:54
dtantsursovsianikov: hi! which version of openstack is it? I remember the way some problem (/cc derekh) with lower vs upper case of serial15:55
*** hjensas has joined #openstack-ironic15:55
sovsianikovdtantsur: Hi, it's Newton15:56
dtantsursovsianikov: wild case: try lower case serial in the root hints15:56
dtantsurs/case/guess/15:56
dtantsurmeh, I'm too sleepy today :(15:56
sovsianikovOk, got it. Thanks a lot for your help!15:56
* TheJulia hands dtantsur coffee15:56
dtantsurthnx <315:57
derekhdtantsur: sovsianikov iirc the problem with case was fixed when the device hints matching logic was switched to ironic-lib, what ever version that was15:58
sovsianikovpython-ironic-lib.noarch 2.1.3-1.el7ost15:59
dtantsurderekh: I recall is patching some version downstream, do you remember what it was?15:59
derekhdtantsur: I'm looking for it now15:59
dtantsurs/is/us/ (just pardon all typos from now on)15:59
derekhdtantsur: sovsianikov it was OSP 10 (Newton)16:00
sovsianikovthis one? https://bugs.launchpad.net/ironic-python-agent/+bug/167972616:01
openstackLaunchpad bug 1679726 in ironic-python-agent "discovery does not include /dev/disk/by-path device name" [Wishlist,Fix released] - Assigned to Ilya Etingof (etingof)16:01
dtantsurno, there was another one16:03
*** deadnull has quit IRC16:05
derekhsovsianikov: this was the downstream bug, iirc I didn't send the fix upstream as the newton branch is gone  https://bugzilla.redhat.com/show_bug.cgi?id=145222616:05
sovsianikovThank you derekh and dtantsur fro your help!16:06
derekhnp16:06
*** pcaruana has quit IRC16:07
openstackgerritJohn L. Villalovos proposed openstack/ironic master: Update comment and mock about autospec not working on staticmethods  https://review.openstack.org/54586616:11
jlvillaldtantsur, ^^^ related to your HTTP 400 patch16:11
dtantsurack thanks16:12
openstackgerritJohn L. Villalovos proposed openstack/ironic master: Update comment and mock about autospec not working on staticmethods  https://review.openstack.org/54586616:14
jlvillaldtantsur, ^^ I put the 'import types' in the wrong section the first time :(16:14
*** mjura has quit IRC16:17
jlvillalmgoddard, FYI: https://review.openstack.org/545370  seems to be failing py35 unit tests16:18
patchbotpatch 545370 - ironic - Register traits on nodes in devstack16:18
*** masuberu has quit IRC16:21
jrolldtantsur: oops! I didn't even realize it, even with the second patchset :|16:21
dtantsurno problem16:21
*** r-daneel has quit IRC16:23
openstackgerritMichael Bayer proposed openstack/ironic master: Use oslo_db.sqlalchemy.test_fixtures  https://review.openstack.org/54587416:27
*** lucas-hungry is now known as lucasagomes16:30
*** sovsianikov has quit IRC16:34
mgoddard_jlvillal: hmm, thanks16:36
*** stendulker has joined #openstack-ironic16:36
mgoddard_jlvillal: not seeing any py35 unit test failures on that change16:40
jlvillalmgoddard_, I looked at zuul.o.o16:40
jlvillalmgoddard_, http://logs.openstack.org/70/545370/2/check/openstack-tox-py35/b8c8957/16:41
mgoddard_ah, I'll check there for the preview16:41
*** vsaienk0 has quit IRC16:47
* TheJulia hopes http://zuul.openstack.org/stream.html?uuid=b79bd5917ee14a999c6fb9bbed4fb058&logfile=console.log16:50
*** rpioso|afk is now known as rpioso16:50
rpiosoGood morning16:50
*** AlexeyAbashkin has quit IRC16:50
*** e0ne has quit IRC16:52
*** hshiina has joined #openstack-ironic16:53
TheJuliagood morning16:56
rpioso:)16:57
*** hjensas has quit IRC16:59
TheJulia#startmeeting ironic17:00
openstackMeeting started Mon Feb 19 17:00:10 2018 UTC and is due to finish in 60 minutes.  The chair is TheJulia. Information about MeetBot at http://wiki.debian.org/MeetBot.17:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.17:00
TheJuliao/17:00
dtantsuro/17:00
*** openstack changes topic to " (Meeting topic: ironic)"17:00
openstackThe meeting name has been set to 'ironic'17:00
jrollohai17:00
bahao/17:00
rpiosoo/17:00
etingofo/17:00
stendulkero/17:00
hshiinao/17:00
mgoddard_ /o\17:01
TheJuliaOur agenda is on the wiki as always17:01
TheJuliaHopefully this meeting will be relatively quick given that next week is the PTG.17:01
TheJulia#link https://wiki.openstack.org/wiki/Meetings/Ironic17:02
TheJulia#topic Announcements/Reminders17:02
*** openstack changes topic to "Announcements/Reminders (Meeting topic: ironic)"17:02
TheJulia#info Rocky PTG is next week, and as such next week's IRC meeting is cancelled.17:02
TheJulia#info An initial schedule is up on the PTG planning etherpad. https://etherpad.openstack.org/p/ironic-rocky-ptg17:03
*** khyr0n has quit IRC17:03
TheJulia#info The at PTG evening gathering will be on Tuesday at 7 PM. Information is inside the planning etherpad.17:03
TheJuliaAlso dtantsur, I believe we will be doing our final queens release for this cycle later this week?17:04
dtantsurTheJulia: yep, by Thu17:04
TheJulia#info Final queens release this week.17:04
TheJuliaAnyone have anything else?17:04
mjtureko/17:05
TheJuliaOn, on a side note, expect me to be sporadically available Wednesday-Friday. dtantsur will be likely be the one submitting the final queens release.17:05
dtantsura mere shadow of former powers :D17:06
TheJuliaAnd less stress :)17:07
TheJuliaAnyway, I guess we can move on!17:07
ricardoaso/17:07
dtantsur++ move on17:08
TheJulia#topic Review Action Items17:08
*** openstack changes topic to "Review Action Items (Meeting topic: ironic)"17:08
TheJuliaLast week, we had two action items. (1) Everyone triage bugs and (2) For myself to try and reproduce the crash of nova-conductor locally.17:08
TheJuliaI was unable to do the latter, but we determined that disabling bytecode from being stored on disk prevents the crash created by library incompatibilities in grenade.17:09
*** links has quit IRC17:09
* dtantsur triaged a few things and revives some older bugs17:10
TheJuliaMoving on!17:10
TheJuliadtantsur: thanks!17:10
dtantsur:)17:10
*** dmellado has joined #openstack-ironic17:10
TheJulia#topic Review sub-team status reports17:10
*** openstack changes topic to "Review sub-team status reports (Meeting topic: ironic)"17:10
TheJulia#link https://etherpad.openstack.org/p/IronicWhiteBoard17:10
TheJuliaI doubt there are any updates worth discussing, since last week was a fairly quiet week.17:11
dtantsur#action dtantsur add networking-baremetal and networking-generic-switch to his bug dashboard17:12
* dtantsur dunno if #action works for non-chairs17:12
TheJuliaI don't believe so17:13
TheJulia#action dtantsur add networking-baremetal and networking-generic-switch to his bug dashboard17:13
TheJuliaLooks like there is nothing to really discuss with regards to status, anyone have anything to bring up before we move on?17:14
dtantsurit seems like we have 2 bugs in rescue17:14
dtantsurto finish and backport17:14
dtantsurhttps://review.openstack.org/#/c/544278/ and https://review.openstack.org/#/c/545186/17:15
patchbotpatch 544278 - ironic - Build instance PXE options for unrescue17:15
patchbotpatch 545186 - ironic - Set 'initrd' to 'rescue_ramdisk' for rescue with iPXE17:15
* dtantsur is ready to move on17:15
TheJuliadtantsur: please add those to the priorities list so we're tracking them (if they are not aleady there)17:15
TheJuliaalready17:15
dtantsurI think I did17:16
TheJulia#topic Priorities for the coming week17:16
*** openstack changes topic to "Priorities for the coming week (Meeting topic: ironic)"17:16
TheJulia\o/17:16
mgoddard_There is a bug when using cleaning with traits set: https://bugs.launchpad.net/ironic/+bug/175002717:17
openstackLaunchpad bug 1750027 in Ironic "Traits & cleaning issue" [High,In progress] - Assigned to Mark Goddard (mgoddard)17:17
openstackgerritMerged openstack/ironic master: Conductor version cannot be null in Rocky  https://review.openstack.org/54438217:17
openstackgerritMerged openstack/ironic master: Clean nodes stuck in CLEANING state when ir-cond restarts  https://review.openstack.org/34997117:17
*** trown is now known as trown|lunch17:17
openstackgerritMerged openstack/ironic master: Fix rare HTTP 400 from port list API  https://review.openstack.org/54506917:18
* dtantsur cherry-picks ^^^17:18
mgoddard_Would be nice to get the fix merged before the queens release17:18
openstackgerritDmitry Tantsur proposed openstack/ironic stable/queens: Fix rare HTTP 400 from port list API  https://review.openstack.org/54589217:19
openstackgerritDmitry Tantsur proposed openstack/ironic stable/queens: Clean nodes stuck in CLEANING state when ir-cond restarts  https://review.openstack.org/54589317:19
TheJuliaI think that is good, I cleaned up the lists and removed prior items17:19
dtantsurmgoddard_: I keep an eye on it, just waiting for the CI17:19
mgoddard_dtantsur: thanks. jlvillal pointed out there is a py3.5 unit test failure so there will be at least another revision17:20
TheJuliaI agree we should likely try and merge/backport17:20
jrollthere's also that bug we're seeing in grenade that I'm hacking on - tl;dr explosive problems with placement if ironic goes down for too long17:21
dtantsuroh17:21
TheJuliajroll: has a bug been opened for that?17:21
jrollhere's the wip patch https://review.openstack.org/#/c/545479/17:21
patchbotpatch 545479 - nova - ironic: stop lying to the RT when ironic is down17:21
jrollTheJulia: I should open a separate bug if there's not yet17:22
* jroll will do today17:22
TheJuliajroll: thanks, please add to the priorites list for the week since we will need ironic eyes on it17:22
TheJuliaerr17:22
TheJuliawip patch, I'll add that17:22
jrollthanks17:22
TheJuliaOkay, added17:23
TheJuliaAnything else?17:23
dtantsurnot from me17:24
TheJuliaEveryone happy with the list?17:24
* jroll is good17:24
TheJuliaMoving to Open Discussion since we have no discussion items on the agenda...17:24
TheJulia#topic Open discussion17:24
*** openstack changes topic to "Open discussion (Meeting topic: ironic)"17:24
* jroll thanks derekh for putting together the thing17:25
TheJuliaEveryone going to remember to bring an ironic t-shirt for the team photo?17:25
TheJuliaYes, thanks derekh!17:25
derekhno prob17:25
* jroll has one17:25
jrollteam photo is on thursday, for the curious17:26
TheJulia\o/17:27
TheJuliaDoes anyone have anything else to discuss?17:27
TheJuliaOtherwise, I think we can wrap this short meeting up for today.17:27
dtantsurcri...17:28
devanandaI'd be ever so thrilled to join some of the shenanigans remotely, maybe do a video call in at some point17:28
dtantsurdevananda: we'll call you from the bar ;)17:28
TheJulia++++17:28
devanandadavidlenwell offered to pin a phone to his shirt while we use hangouts ;)17:28
dtantsur... or when we discuss converting to a CMDB finally?17:28
jrollLOL17:28
devanandadtantsur: yes pls! I've got a nice bottle of whisky here - I would love to raise a glass with ya'll17:28
dtantsur:D17:29
devanandadtantsur: bwahaha!17:29
TheJuliaSo sounds like we have some plans!17:29
TheJulia:)17:29
* dtantsur seems to hear crickets again17:30
TheJuliaAnyway, the crickets as a service have returned with a beautiful cricket sound. I think we can call this meeting a wrap17:30
TheJuliaThanks everyone!17:30
dtantsur++ thanks TheJulia17:30
TheJuliaOh, one final thing, I'll break the etherpad up after I have lunch in a little bit, so expect to see text moving around in relation to the schedule17:31
TheJulia#endmeeting17:31
*** openstack changes topic to "Bare Metal Provisioning | Status: http://bit.ly/ironic-whiteboard | Docs: http://docs.openstack.org/developer/ironic/ | Bugs: https://bugs.launchpad.net/ironic"17:31
openstackMeeting ended Mon Feb 19 17:31:43 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:31
openstackMinutes:        http://eavesdrop.openstack.org/meetings/ironic/2018/ironic.2018-02-19-17.00.html17:31
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/ironic/2018/ironic.2018-02-19-17.00.txt17:31
openstackLog:            http://eavesdrop.openstack.org/meetings/ironic/2018/ironic.2018-02-19-17.00.log.html17:31
*** etingof has quit IRC17:32
dtantsurTheJulia: where we ended up with https://bugs.launchpad.net/ironic/+bug/1596421 do you remember?17:32
openstackLaunchpad bug 1596421 in Ironic "[RFE] Stop storing configdrive in the database" [Wishlist,In progress] - Assigned to Dmitry Tantsur (divius)17:32
TheJuliaugh17:33
TheJuliathat one17:33
dtantsuryes, got a customer complaining about that17:34
dtantsurgiven that providing configdrive in rebuild is microversioned.. I'm not sure we can just stop saving it17:34
dtantsurthough maybe we should strip it if we get an error from the database?17:35
dtantsurlike, it's still better than failing completely? or not?17:35
* dtantsur thinks17:35
TheJuliaI think the solution that we came to that would kind of be ideal... was to add a table for it specifically and store it in there17:35
TheJuliabut... that presents other issues17:35
TheJuliaI mean... maybe not if we also begin wiping it out...17:36
* TheJulia thinks that would be a fair microversion change to have17:36
dtantsurwell, in theory we should not store blobs in database, but rather use something like swift17:36
dtantsurnone of this is backportable17:36
TheJuliano, none of it would be17:36
TheJuliaswift would be ideal, but then all deployments require swift. :\17:37
dtantsurfor reference, people asking about it on Newton: https://bugzilla.redhat.com/show_bug.cgi?id=143308817:37
pas-hadtantsur: what about take over happening during deploy/rebuild when config drive is not in the DB?17:37
*** ijw has joined #openstack-ironic17:37
pas-haand w/o swift I mean17:38
*** stendulker has quit IRC17:38
TheJuliaWell, we could store blobs in the db as long as their content is not indexed and the table is kept clean17:38
dtantsurpas-ha: I'm not sure take over is affected here. if config drive is not in DB and is not supplied to rebuild, we won't have a config drive..17:38
TheJuliawhich would be "new table" territory17:38
*** ijw has quit IRC17:38
dtantsuror we should merge that change that makes the column larger on mysql17:39
*** ijw has joined #openstack-ironic17:39
* TheJulia mulls that17:39
pas-haor the bug is about 'not storing configdrive after deploy is done for active node'? I may be confused..17:39
TheJuliaNot great... but we also know some folks have done just that in their deployments17:40
pas-haI mean we still need it during deploy/rebuild as part if instance_info in the DB so that takeover may happen before the configdrive is written17:40
dtantsurpas-ha: we're trying to git the config drive as part of a JSON into a text column. which is something just too small.17:40
*** baha has quit IRC17:40
dtantsurI mean, putting it in instance_info simply does not always work17:40
dtantsurif you have a lot of metadata - ooops17:41
pas-hayeah, that's not great...17:41
dtantsurit's not even about rebuild, you won't deploy in the first place17:41
pas-habut we do need it either in DB or in the external source.17:41
dtantsurpas-ha: we don't strictly need it, since starting with queens nova passes it to us IIRC17:42
dtantsurbut we cannot probably rely on it17:42
dtantsure.g. standalone people might not do it still17:42
TheJuliayeah17:42
dtantsurbut standalone people don't have many options as well: they don't have swift17:42
TheJuliaI think the "cleanest" thing to do is likely expand the column width, and if we need to migrate it out to another table, then we actually have some advantages there to try and keep that quick for most deployments17:43
dtantsurso, https://review.openstack.org/33496717:43
patchbotpatch 334967 - ironic - Fix non-deployed nodes due to short nodes.instance...17:43
TheJuliayeah17:43
dtantsurwhich we won't be able to backport past queens?17:43
TheJuliahey, jroll, any thoughts on ^^17:43
pas-hastill not get it why we do not need to store it w/o swift being available. nova posts it to instance_info -> ironic starts deploy -> ironic-conductor fails even before first heartbeat received -> another conductor takes over -> it needs to recreate the node's instance_info from DB17:44
dtantsurpas-ha: if we have swift, we can try storing it in swift instead17:44
jrollTheJulia: on backporting db changes or the configdrive thing?17:44
TheJuliajroll: in general17:44
TheJuliathe configdrive thing17:44
jrollah, give me a few to finish a nova conversation?17:44
dtantsurTheJulia: well, we can try backporting it further, but we'll have to carefully adjust the migration17:45
pas-hayeah, but that would the require swift, which I thought we were trying to avoid (and me personally too).17:45
TheJuliajroll: sure17:45
dtantsurpas-ha: yes, this is my point: swift could save us, but we cannot rely on it :)17:45
* dtantsur updates the patch17:46
openstackgerritHironori Shiina proposed openstack/ironic master: Set 'initrd' to 'rescue_ramdisk' for rescue with iPXE  https://review.openstack.org/54518617:46
* dtantsur responds to himself: no, we cannot backport it past queens, it will break later updates17:46
pas-haa bitter side note - if we currently start to require swift, we loose Py3 support as we won't be able to test it on gates...17:49
TheJuliadtantsur: that patch as is?17:49
dtantsuryep17:49
dtantsurpas-ha: ugh, true17:49
dtantsuranyway, we're not giving up standalone mode17:49
* TheJulia hasn't even gotten to that point mentally17:50
* TheJulia needs to eat something17:50
dtantsurmmm, should we use MEDIUMTEXT or LONGTEXT? a configdrive can be up to 64MiB IIRC?17:52
dtantsurwhile MEDIUMTEXT is 16 MiB17:52
* dtantsur votes for LONGTEXT and API check on len(instance_info['configdrive']) <= 64 MiB17:53
TheJuliaI kind of like that idea17:54
jrollnote the configdrive is gzipped and b64 encoded, so that length check may not work17:54
jroll(as expected)17:55
TheJuliayeah...17:55
*** racedo has quit IRC17:59
dtantsurhmm, should we even check actually?18:00
*** derekh has quit IRC18:00
*** e0ne has joined #openstack-ironic18:01
jrollidk18:01
* jroll reads back for context18:01
*** mgoddard_ has quit IRC18:02
*** hshiina has quit IRC18:02
dtantsurwell, in PostreSQL TEXT is unlimited, so for consistency we probably should not check18:02
openstackgerritDmitry Tantsur proposed openstack/ironic master: Increase the instance_info column size to LONGTEXT on MySQL/MariaDB  https://review.openstack.org/33496718:03
dtantsurhere goes the patch ^^^18:03
jrolldtantsur: TheJulia: so we're storing it simply so that we can do take_over properly during a build?18:14
* jroll would rather just not store it, people love to put secrets there18:14
dtantsurjroll: for rebuilds as well18:14
jrolldtantsur: rebuilds get a new configdrive now, though, right18:15
*** AlexeyAbashkin has joined #openstack-ironic18:15
* jroll sees we decided a long time ago to pass configdrive on rebuild, then stop storing it18:17
*** rwsu has quit IRC18:17
*** openstackgerrit has quit IRC18:18
jrollI would be fine with failing builds where a conductor dies during the build, btw, to be able to keep the configdrive out of the db18:18
*** AlexeyAbashkin has quit IRC18:20
*** e0ne has quit IRC18:21
*** hjensas has joined #openstack-ironic18:23
TheJuliajroll: I know of at least one group that expected it to continue to be able to deploy with a conductor failure and was in their test plan as such..... :\18:23
TheJuliaAlthough, we've not explicitly stated that is something we do, it is by side-effect18:24
TheJuliaI would really prefer to keep it out of the db, but where do we house it while waiting for the machine?18:24
*** hjensas has quit IRC18:27
jrollum, uh, good question.18:28
jrolldang.18:28
*** jrist is now known as jrist-afk18:28
*** jrist-afk is now known as jrist18:28
*** rwsu has joined #openstack-ironic18:29
* jroll wonders wtf we were planning to do when we decided that18:30
*** tesseract has quit IRC18:30
dtantsurswift when it's present otherwise a separate blob field...18:31
* dtantsur does not remember18:31
jrollyeah, we must have been assuming we'd keep it in ram18:31
* jroll wishes we just had a thread-ish thing that polled the agent, rather than waiting for the callback18:32
TheJuliaI really think it was another table18:33
TheJuliajroll: soonish? ;)18:33
jrollI don't think so: "since bug #1575935 is fixed, we no longer need to store the configdrive in the database at all."18:33
openstackbug 1575935 in python-ironicclient "Rebuild should also accept a configdrive" [Medium,Fix released] https://launchpad.net/bugs/1575935 - Assigned to Mathieu Gagné (mgagne)18:33
dtantsurjroll: even with polling the agent, you have to be able to survive a conductor restart18:35
jrolldtantsur: I'd be willing to let builds fail if the conductor restarts, to keep user secrets out of the db18:35
dtantsurjroll: this is not what everyone would prefer, I'd suspect..18:35
jrolldtantsur: --i-am-a-dummy, then :(18:36
dtantsuron a busy environment it make it essentially impossible to restart conductor without breaking a few (dozens?) of deployments18:36
jrollidk, even though people shouldn't pass secrets in nova metadata, I feel like we should be secure about user data always (or at least by default)18:36
jrolldtantsur: we have soft shutdown for that18:37
dtantsurI'm on the "people should not pass secrets" side18:37
jrollstop taking new builds, finish the existing18:37
dtantsurehmm, how will soft shutdown help here?18:37
dtantsurah, I think I see it18:37
jrollit solves "on a busy environment it make it essentially impossible to restart conductor without breaking a few (dozens?) of deployments"18:37
jrolljust slows down the rolling restart18:37
dtantsuryep, substantially18:37
dtantsurI think currently we don't wait for all nodes in DEPLOY WAIT18:38
jrollthis is true, good point18:38
* dtantsur feels like we've found another topic for TheJulia to squeeze into the PTG schedule18:38
jrollanyway, I don't have the time to fight against storing it at all18:38
*** r-daneel has joined #openstack-ironic18:39
jrollwe're okay with continuing to use instance_info, rather than a new column?18:39
*** trown|lunch is now known as trown18:39
*** e0ne has joined #openstack-ironic18:40
* jroll wonders how drastically a large configdrive slows down un/serialization there18:40
dtantsurokay short-term18:40
dtantsurlike a fix for Queens18:41
jrollah yes, we can backport to queens, but only if we do it quickly18:41
dtantsuryep18:41
jrollwe can't ever go back to MEDIUMTEXT thoug :(18:41
*** khyr0n has joined #openstack-ironic18:42
jrollor TEXT, sorry18:42
dtantsurwell, I'm not sure how crazy mysql is, but I hope it does not allocate too much for LONGTEXT18:42
dtantsurlike, postresql only has LONGTEXT18:42
* dtantsur wishes we used a more serious database, sigh...18:42
jrollheh18:42
* jroll throws mongodb in and runs far far away18:43
dtantsurhaha18:43
dtantsurdevnull as a service18:43
persiaThat's an excellent place to store config drives :)18:43
*** mgoddard_ has joined #openstack-ironic18:44
jrollheh18:44
dtantsurjroll: okay, so the biggest problem is that a new node field will require a new RPC version18:44
jrolldtantsur: yeah, that also isn't awesome for backporting18:44
dtantsurwhich, if I get it right, we cannot backport18:44
jrollI'm good with changing the size for now, let's do it18:44
*** hjensas has joined #openstack-ironic18:44
* jroll +1'd18:44
dtantsuryep, let's do it..18:44
*** rwsu has quit IRC18:46
jroll[openstack-dev] [oslo.db] [all] please DO NOT IMPORT from oslo_db.tests.* ! projects doing this need to revert ASAP18:47
jrollouch18:47
jrollah, he fixed it for us, cool18:47
*** lucasagomes is now known as lucas-brb18:57
*** baha has joined #openstack-ironic18:58
*** baha1 has joined #openstack-ironic18:58
* TheJulia is +1 for size and having a real fix this cycle18:59
*** baha has quit IRC19:01
*** rwsu has joined #openstack-ironic19:01
*** harlowja has joined #openstack-ironic19:03
*** baha1 is now known as baha19:04
* TheJulia watches her first sand storm move in19:07
*** dtantsur is now known as dtantsur|afk19:10
dtantsur|afko/19:10
*** openstackgerrit has joined #openstack-ironic19:10
openstackgerritOlivier Bourdon proposed openstack/bifrost master: WIP: Add check for (i)PXE and Ironic location coherence  https://review.openstack.org/54571119:10
*** ijw has quit IRC19:23
*** mgoddard_ has quit IRC19:25
*** mgoddard_ has joined #openstack-ironic19:56
*** Goneri has quit IRC19:57
*** lucas-brb is now known as lucasagomes19:58
TheJuliaOkay folks, I think I have the etherpad split up20:02
TheJuliafor the PTG that is20:02
openstackgerritMark Goddard proposed openstack/ironic master: Support nested objects and object lists in as_dict  https://review.openstack.org/54583020:10
openstackgerritMark Goddard proposed openstack/ironic master: Register traits on nodes in devstack  https://review.openstack.org/54537020:10
*** lucasagomes is now known as lucas-afk20:14
*** etingof has joined #openstack-ironic20:31
openstackgerritMark Goddard proposed openstack/ironic master: Validate instance_info.traits against node traits  https://review.openstack.org/54346120:34
*** mgoddard has quit IRC20:35
*** devananda has quit IRC20:37
*** mgoddard has joined #openstack-ironic20:42
openstackgerritJulia Kreger proposed openstack/ironic stable/queens: DNM Test disabling .pyc files for Grenade  https://review.openstack.org/54508920:44
openstackgerritOlivier Bourdon proposed openstack/bifrost master: Add check for (i)PXE and Ironic location coherence  https://review.openstack.org/54571120:45
*** mgoddard_ has quit IRC20:53
openstackgerritJulia Kreger proposed openstack/ironic stable/queens: DNM Test disabling .pyc files for Grenade  https://review.openstack.org/54508920:59
*** AlexeyAbashkin has joined #openstack-ironic21:26
*** AlexeyAbashkin has quit IRC21:31
*** dprince has quit IRC21:33
*** rcernin has joined #openstack-ironic21:36
*** threestrands has joined #openstack-ironic21:37
jrollTheJulia: how do we feel about nova-compute failing to start if ironic-api is not up?21:40
* TheJulia ponders21:40
jrollmy self-healing hat says no, but my don't-want-to-get-into-the-guts-of-nova hat says it's fine21:40
TheJuliaI think it is fine in that the compute service has then visibily failed21:41
TheJuliaif we say we're good and not really working, then that in it's self is a disservice to an operator21:41
TheJuliaSo on start-up, I think it is fine for us to fail, the conundrum might be grenade... but that might be the motivation to retool finally if we can't restart the process due to library compatability issues across releases21:42
jrollgood point21:42
jrollit'll keep running if it's already up, fwiw21:42
*** Goneri has joined #openstack-ironic21:42
jrolljust with outdated resource tracking21:42
TheJuliawhich it... ideally... shoudl deal with eventually21:43
TheJuliashould()21:43
TheJuliaI think that returns false21:43
jrollheh21:43
jrollor it hangs forever21:44
TheJuliawoot!21:44
TheJuliawill it stop heartbeating inside nova for service status?21:44
* jroll adds a release note about it21:44
jrollthat's a good question, I don't think so?21:44
TheJuliaWe should put a release note on each side21:44
TheJuliaohhh... hmmmmmm21:44
jrollI mean, we could21:45
TheJuliaknown issue I guess?  We have a solid hour without topics with nova for the schedule right now21:45
jrollit'd be nice to sit down and think this through fully (as there's lots of hacks around this right now), but would also be nice to get this bug fixed now21:45
TheJuliaAgreed21:46
jrollit'd be trivial to mark self._ironic_down = True in the driver21:46
TheJuliaHence why I was thinking about the schedule21:46
jrollyeah21:46
TheJuliaor _compute_down21:46
jrollyeah21:46
* jroll has no clue how the heartbeat works O_o21:48
jrollwell, still not sure where the heartbeat is21:52
jrollbut we can do this: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1707-L171121:52
TheJuliaperfect21:53
*** priteau has quit IRC21:53
*** priteau has joined #openstack-ironic21:54
*** jcoufal has quit IRC21:55
*** trown is now known as trown|outtypewww21:56
* jroll will leave this for another day21:57
TheJuliaOkay21:58
*** priteau has quit IRC21:59
*** e0ne has quit IRC22:06
*** hamzy has quit IRC22:07
* jroll files https://bugs.launchpad.net/nova/+bug/1750450 in the meantime22:09
openstackLaunchpad bug 1750450 in OpenStack Compute (nova) "ironic: n-cpu fails to recover after losing connection to ironic-api and placement-api" [Undecided,New]22:09
*** hamzy has joined #openstack-ironic22:16
*** openstackgerrit has quit IRC22:18
*** openstackgerrit has joined #openstack-ironic22:20
openstackgerritJulia Kreger proposed openstack/ironic stable/queens: DNM Test disabling .pyc files for Grenade  https://review.openstack.org/54508922:20
*** priteau has joined #openstack-ironic22:22
*** masber has joined #openstack-ironic22:23
TheJuliajroll: thanks!22:24
jroll:)22:24
*** baha has quit IRC22:27
openstackgerritJulia Kreger proposed openstack/ironic master: Disable .pyc files for grenade multinode  https://review.openstack.org/54475022:32
*** priteau has quit IRC22:38
*** bfournie1 has quit IRC22:46
*** bfournie has joined #openstack-ironic22:46
openstackgerritJulia Kreger proposed openstack/ironic master: Add docs for ansible deploy interface  https://review.openstack.org/52550122:48
*** bfournie has quit IRC22:53
openstackgerritJulia Kreger proposed openstack/ironic stable/queens: DNM Test disabling .pyc files for Grenade  https://review.openstack.org/54508922:59
*** AlexeyAbashkin has joined #openstack-ironic23:06
*** AlexeyAbashkin has quit IRC23:10
openstackgerritMerged openstack/networking-baremetal master: Update reno for stable/queens  https://review.openstack.org/54233123:13
jlvillalTheJulia, I replied to your comment: https://review.openstack.org/#/c/528303/2/playbooks/legacy/tempest-dsvm-ironic-inspector/run.yaml@a9723:18
patchbotpatch 528303 - ironic - zuul: Remove duplicated TEMPEST_PLUGIN entry23:18
jlvillalAnd thanks for reviewing :)23:18
*** pmannidi has joined #openstack-ironic23:19
*** masber has quit IRC23:34
*** r-daneel has quit IRC23:36
*** masber has joined #openstack-ironic23:37
*** bfournie has joined #openstack-ironic23:52
*** masber has quit IRC23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!