Thursday, 2019-05-16

*** dsneddon has joined #openstack-ironic00:09
*** dsneddon has quit IRC00:16
*** goldyfruit has joined #openstack-ironic00:23
*** gyee has quit IRC00:36
*** rloo has quit IRC00:38
*** dsneddon has joined #openstack-ironic00:52
*** dsneddon has quit IRC01:05
*** oyrogerg has joined #openstack-ironic01:11
*** dsneddon has joined #openstack-ironic01:37
*** rh-jelabarre has quit IRC01:39
*** dsneddon has quit IRC01:43
*** ijw has quit IRC01:55
*** whoami-rajat has joined #openstack-ironic02:06
*** gryf has joined #openstack-ironic02:10
*** dsneddon has joined #openstack-ironic02:17
*** dsneddon has quit IRC02:22
*** dhellmann has quit IRC02:23
*** dhellmann has joined #openstack-ironic02:23
*** dsneddon has joined #openstack-ironic02:24
*** dsneddon has quit IRC02:28
*** oyrogerg has quit IRC02:53
*** oyrogerg has joined #openstack-ironic02:54
*** oyrogerg has quit IRC02:55
*** dsneddon has joined #openstack-ironic02:57
*** oyrogerg has joined #openstack-ironic03:06
*** dsneddon has quit IRC03:10
*** dsneddon has joined #openstack-ironic03:46
*** dsneddon has quit IRC03:59
*** oyrogerg has quit IRC04:12
*** tiendc has joined #openstack-ironic04:12
*** oyrogerg has joined #openstack-ironic04:17
*** dsneddon has joined #openstack-ironic04:35
openstackgerritShivanand Tendulker proposed openstack/ironic master: Remove deprecated option [ilo]power_retry  https://review.opendev.org/65921904:38
*** dsneddon has quit IRC04:41
*** oyrogerg has quit IRC04:44
*** oyrogerg has joined #openstack-ironic04:50
mkraigm ironic04:59
*** dsneddon has joined #openstack-ironic05:10
*** dsneddon has quit IRC05:15
*** dsneddon has joined #openstack-ironic05:43
*** hamzy has quit IRC05:44
*** gkadam has quit IRC05:49
*** dsneddon has quit IRC05:50
*** gkadam has joined #openstack-ironic05:51
*** e0ne has joined #openstack-ironic05:53
*** dsneddon has joined #openstack-ironic05:53
*** tiendc has quit IRC06:10
*** oyrogerg has quit IRC06:16
*** hamzy has joined #openstack-ironic06:21
*** oyrogerg has joined #openstack-ironic06:29
*** pmannidi has joined #openstack-ironic06:34
*** e0ne has quit IRC06:36
*** rachit7 has joined #openstack-ironic06:37
arne_wiebalckgood morning, mkrai and ironic o/06:43
mkraiHi arne_wiebalck06:59
mkraiDo you know what does ironic do with the ports which are pxe_disabled?07:00
*** rcernin has quit IRC07:01
*** openstackgerrit has quit IRC07:03
*** tesseract has joined #openstack-ironic07:05
*** pcaruana has joined #openstack-ironic07:13
*** rpittau|afk is now known as rpittau07:17
rpittaugood morning ironic! o/07:17
*** oyrogerg has quit IRC07:27
rpittaudtruong: I think 'cacert' is not supported by keystone, relatede correct kwargs would be 'cafile' for CA, 'certfile' for the certificate file, and 'keyfile' for the related key file07:27
rpittauhttps://github.com/openstack/keystoneauth/blob/master/keystoneauth1/loading/session.py#L13707:27
*** tssurya has joined #openstack-ironic07:32
*** S4ren has joined #openstack-ironic07:35
*** iurygregory has joined #openstack-ironic07:35
iurygregorygood morning07:35
rpittauhey iurygregory  :)07:36
rpittauwell seems like master is also broken now :/07:50
rpittauat least it's consistent07:50
*** tesseract has quit IRC07:52
*** tesseract has joined #openstack-ironic07:52
mkrairpittau: iurygregory Hi, good morning!07:55
*** gkadam has quit IRC07:55
rpittauhi mkrai :)07:55
iurygregorymorning mkrai08:02
*** S4ren has quit IRC08:05
rpittauI think this is what causing issues: https://opendev.org/openstack/python-ironicclient/src/branch/master/ironicclient/common/http.py#L25808:05
*** stendulker has joined #openstack-ironic08:06
*** e0ne has joined #openstack-ironic08:08
*** rachit7 has quit IRC08:12
*** lucasagomes has joined #openstack-ironic08:12
*** stendulker has left #openstack-ironic08:12
*** stendulker has joined #openstack-ironic08:13
*** S4ren has joined #openstack-ironic08:15
*** dsneddon has quit IRC08:15
*** chihhsin has quit IRC08:32
*** chihhsin has joined #openstack-ironic08:32
*** derekh has joined #openstack-ironic08:38
*** dsneddon has joined #openstack-ironic08:41
*** gkadam has joined #openstack-ironic08:45
*** dsneddon has quit IRC08:49
*** dtantsur|afk is now known as dtantsur08:50
dtantsurmorning ironic08:51
rpittauhi dtantsur :)08:52
rpittaudtantsur: for the issues in the CI, I believe there's something wrong in the api endpoint when trying to get the list of nodes:08:55
rpittauhttp://logs.openstack.org/19/659219/2/check/ironic-grenade-dsvm/7c9df62/logs/screen-ir-api.txt.gz#_May_16_05_22_21_00034108:55
dtantsureew09:03
dtantsurrpittau: this looks like broken endpoint detection :(09:04
rpittauyeah09:04
dtantsurwow, have we broken ironicclient? :(09:04
dtantsurI don't believe so, its CI would not pass09:04
rpittaumaybe, I'm still trying to figure that out :D09:04
rpittautrue09:04
dtantsurso maybe some combination of nova and ironicclient changes?09:04
rpittauehhh I'm checking both, so far haven't seen any changes that might be related, except the one you pointed to, which I don't get how could be related :/09:05
iurygregoryblack magic in CI =(09:14
*** dsneddon has joined #openstack-ironic09:18
*** dsneddon has quit IRC09:23
*** openstackgerrit has joined #openstack-ironic09:30
openstackgerritDmitry Tantsur proposed openstack/metalsmith master: Build configdrives on ironic side  https://review.opendev.org/65950609:30
*** dsneddon has joined #openstack-ironic09:45
openstackgerritDmitry Tantsur proposed openstack/metalsmith master: Expose Allocation objects on Instance  https://review.opendev.org/65951209:49
openstackgerritDmitry Tantsur proposed openstack/metalsmith master: Expose Allocation objects on Instance  https://review.opendev.org/65951209:56
openstackgerritDmitry Tantsur proposed openstack/metalsmith master: Build configdrives on ironic side  https://review.opendev.org/65950609:57
*** pmannidi has quit IRC10:05
openstackgerritDmitry Tantsur proposed openstack/metalsmith master: Update Sphinx requirements to match global-requirements  https://review.opendev.org/65951910:14
openstackgerritDmitry Tantsur proposed openstack/metalsmith master: Expose Allocation objects on Instance  https://review.opendev.org/65951210:16
*** pcaruana has quit IRC10:27
*** mmethot has quit IRC10:29
openstackgerritMerged openstack/metalsmith master: Update Sphinx requirements to match global-requirements  https://review.opendev.org/65951910:37
*** dsneddon has quit IRC10:48
openstackgerritIlya Etingof proposed openstack/ironic master: WIP: Add Redfish Virtual Media support  https://review.opendev.org/63845310:54
johnsagI am following the instructions found: https://docs.openstack.org/bifrost/latest/install/index.html to install bifrost and ironic.  I am at the step where i run the playbook.  I get an error while running this task: TASK [bifrost-ironic-install : Upgrade ironic DB Schema]11:00
johnsagThere seems to be communication with the DB, so I don't think that is the issue.  Not sure how to fix.11:01
johnsag        "DBError: (pymysql.err.InternalError) (1280, u\"Incorrect index name 'deploy_template_id'\") [SQL: u'CREATE INDEX deploy_template_id ON deploy_template_steps (deploy_template_id)'] (Background on this error at: http://sqlalche.me/e/2j85)"11:01
*** mmethot has joined #openstack-ironic11:19
*** dsneddon has joined #openstack-ironic11:21
*** dsneddon has quit IRC11:26
*** johnsag has quit IRC11:28
*** pcaruana has joined #openstack-ironic11:37
*** stendulker has quit IRC11:37
openstackgerritDmitry Tantsur proposed openstack/metalsmith master: Start creating allocations in reserve_node  https://review.opendev.org/65953511:43
*** dtantsur is now known as dtantsur|brb11:46
*** dsneddon has joined #openstack-ironic11:54
*** dsneddon has quit IRC12:00
*** goldyfruit has quit IRC12:02
*** rh-jelabarre has joined #openstack-ironic12:21
*** dsneddon has joined #openstack-ironic12:21
*** dsneddon has quit IRC12:26
*** iurygregory has quit IRC12:27
*** rloo has joined #openstack-ironic12:37
openstackgerritMadhuri Kumari proposed openstack/ironic master: Add IntelIPMIHardware  https://review.opendev.org/65954912:38
*** rh-jelabarre has quit IRC12:39
openstackgerritMadhuri Kumari proposed openstack/ironic master: Add IntelIPMIHardware  https://review.opendev.org/65955012:41
*** dsneddon has joined #openstack-ironic12:53
*** dsneddon has quit IRC12:59
*** jcoufal has joined #openstack-ironic13:07
rpittaudtantsur|brb: I went through a lot of comparisons and what I can see is that the jobs started failing after this change to uppper-constraints that updated python-ironicclient to 2.7.1 from 2.7.013:08
rpittauhttps://github.com/openstack/requirements/commit/2b6e70b464c8b2c646ab45c1bc98ac9005d61a90#diff-0bdd949ed8a7fdd4f95240bd951779c813:08
rpittauchecking the failing jobs, they all seem to have the new 2.7.1 version13:08
rpittaue.g. http://logs.openstack.org/70/658770/2/check/ipa-tempest-partition-bios-ipmi-direct-tinyipa-src/1efb1b6/controller/logs/devstacklog.txt.gz#_2019-05-15_15_23_13_97113:08
rpittauthere is actually a change that puzzles me in that version:13:08
rpittauhttps://github.com/openstack/python-ironicclient/commit/b40630f3b3e747ae4a26d31e8e292554b99713f213:08
*** itlinux has quit IRC13:10
*** goldyfruit has joined #openstack-ironic13:10
*** dtantsur|brb is now known as dtantsur13:18
*** goldyfruit has quit IRC13:18
*** goldyfruit has joined #openstack-ironic13:18
dtantsurrpittau: this is a suspect, yes. I wonder why the CI swallowed it initially..13:18
dtantsurshould we exclude 2.7.1 from upper-constrains while we're figuring out?13:19
rpittauI'm not sure, what I've noticed is that the client version was not propagated at the same time, probably some timing in the job execution13:20
rpittaudtantsur: yeah, we should at least try a job with 2.7.0 and see how it goes13:20
openstackgerritDmitry Tantsur proposed openstack/metalsmith master: Start creating allocations in reserve_node  https://review.opendev.org/65953513:21
*** Goneri has joined #openstack-ironic13:21
dtantsurrpittau: I mean, ironicclient CI runs a job, right? it was supposed to catch the problem..13:21
rpittaudtantsur: that's one of the puzzling things :)13:23
dtantsurrpittau: WUT http://logs.openstack.org/79/653279/1/check/ironicclient-tempest-dsvm-src/224a921/logs/devstacklog.txt.gz#_2019-04-17_04_50_51_41813:24
rpittaudtantsur: that's rocky right ?13:24
dtantsurI mean, why is it pip installing? although.. maybe it's fine13:25
rpittauit' supposed to be installed by pip as it's not in the required projects13:25
dtantsurit is, that's ironicclient CI13:25
dtantsurand it's actually installed from git, but later13:25
dtantsurand I wonder if we restart nova to catch that13:25
dtantsurjesus13:26
rpittaulol thought it was IPA13:26
rpittauoh btw the same thing goes for version 2.5.213:26
rpittauqueens is immune because there was no release after 2.2.213:26
*** ianychoi has joined #openstack-ironic13:26
dtantsurhmm, we should be restarting nova-compute indeed13:29
rpittaucaching?13:29
rpittaulol -> https://review.opendev.org/#/c/653338/13:30
patchbotpatch 653338 - python-ironicclient (stable/queens) - Use endpoint_override in version negotiation - 1 patch set13:30
rpittauhere we go :D13:30
openstackgerritDmitry Tantsur proposed openstack/python-ironicclient master: [DNM] Test endpoint_override  https://review.opendev.org/65956313:32
dtantsurtrying to figure out ^^13:32
rpittauyeah13:33
dtantsurrpittau: well, I know this patch :) it doesn't explain much..13:33
rpittauyep, my point was that it was not merged to queens :)13:33
*** dsneddon has joined #openstack-ironic13:33
*** gkadam has quit IRC13:33
dtantsurright, right13:33
openstackgerritDmitry Tantsur proposed openstack/python-ironicclient master: [DNM] Test endpoint_override  https://review.opendev.org/65956313:34
*** dsneddon has quit IRC13:38
*** rh-jelabarre has joined #openstack-ironic13:54
openstackgerritIlya Etingof proposed openstack/sushy-tools master: Add `get/set_boot_image` to libvirt driver  https://review.opendev.org/65693813:55
*** pcaruana has quit IRC13:56
openstackgerritIlya Etingof proposed openstack/sushy-tools master: Add `get/set_boot_image` to libvirt driver  https://review.opendev.org/65693814:00
*** iurygregory has joined #openstack-ironic14:04
*** dsneddon has joined #openstack-ironic14:11
*** dsneddon has quit IRC14:17
*** itlinux has joined #openstack-ironic14:27
*** itlinux has quit IRC14:31
openstackgerritDmitry Tantsur proposed openstack/metalsmith master: Start creating allocations in reserve_node  https://review.opendev.org/65953514:32
*** sziviani has quit IRC14:37
*** pcaruana has joined #openstack-ironic14:37
*** Lucas_Gray has joined #openstack-ironic14:37
dtantsurrpittau: do we have a bug about this ironicclient situation?14:38
rpittaudtantsur: nope14:38
dtantsurok. I think the /v1/v1 guess is right, and I think I have a cure14:38
rpittaudtantsur: oh great!14:39
* dtantsur facepalms from str.rstrip('/v1')14:41
dtantsurI hope it wasn't me who did it :D14:41
rpittauI'm dying of curiosity now :D14:45
*** dsneddon has joined #openstack-ironic14:47
*** dsneddon has quit IRC14:51
dtantsurrpittau: to avoid dying, could you file a bug? :D14:53
rpittaudtantsur: on it14:53
*** jcoufal_ has joined #openstack-ironic14:57
*** dsneddon has joined #openstack-ironic14:57
*** jcoufal has quit IRC14:59
rpittaudtantsur: https://storyboard.openstack.org/#!/story/200572315:01
dtantsurmany thanks!15:01
openstackgerritDmitry Tantsur proposed openstack/python-ironicclient master: Do not try to use /v1/v1 when endpoint_override is used  https://review.opendev.org/65959215:02
dtantsurrpittau: let's try this ^^15:02
*** devx has quit IRC15:03
openstackgerritDmitry Tantsur proposed openstack/python-ironicclient master: [DNM] Test endpoint_override  https://review.opendev.org/65956315:03
*** devx has joined #openstack-ironic15:04
*** efried has joined #openstack-ironic15:21
*** dsneddon has quit IRC15:23
*** e0ne has quit IRC15:38
*** pcaruana has quit IRC15:47
*** iurygregory has quit IRC15:55
rpittaugood night! o/15:58
*** rpittau is now known as rpittau|afk15:58
*** lucasagomes has quit IRC15:58
openstackgerritDmitry Tantsur proposed openstack/metalsmith master: Build configdrives on ironic side  https://review.opendev.org/65950616:00
dtantsurfolks, https://review.opendev.org/#/c/659592/ seems to fix the CI issue16:01
patchbotpatch 659592 - python-ironicclient - Do not try to use /v1/v1 when endpoint_override is... - 1 patch set16:01
*** dims has quit IRC16:05
*** tesseract has quit IRC16:05
openstackgerritIlya Etingof proposed openstack/sushy-tools master: Add `get/set_boot_image` to libvirt driver  https://review.opendev.org/65693816:07
openstackgerritMatt Riedemann proposed openstack/ironic master: Fix trait check in devstack  https://review.opendev.org/65961116:10
arne_wiebalckbye everyone o/16:19
dtantsur\o16:19
*** dtantsur is now known as dtantsur|afk16:19
*** derekh has quit IRC17:00
*** dims has joined #openstack-ironic17:03
*** jcoufal_ has quit IRC17:05
*** jcoufal has joined #openstack-ironic17:06
*** tssurya has quit IRC17:10
*** dims has quit IRC17:12
*** dims has joined #openstack-ironic17:13
*** Lucas_Gray has quit IRC17:18
*** S4ren has quit IRC17:27
openstackgerritDmitry Tantsur proposed openstack/metalsmith master: Build configdrives on ironic side  https://review.opendev.org/65950617:33
*** Goneri has quit IRC17:35
-openstackstatus- NOTICE: Gerrit is being restarted to add gitweb links back to Gerrit. Sorry for the noise.17:36
*** pcaruana has joined #openstack-ironic17:39
*** Goneri has joined #openstack-ironic17:48
*** jcoufal has quit IRC17:55
*** dsneddon has joined #openstack-ironic18:04
*** jcoufal has joined #openstack-ironic18:06
*** rachit7 has joined #openstack-ironic18:08
*** ianychoi has quit IRC18:15
mgoddardhi jroll18:15
mgoddardgot a minute?18:15
*** michchap has joined #openstack-ironic18:19
jrollmgoddard: sure, what's up?18:26
mgoddardjroll: just in time, I just wrote a bug summary18:27
mgoddardIronic instance tear down failure due to state transition blocking instance info removal18:27
mgoddard(in nova)18:27
mgoddardAFAICT, its introduced in rocky by https://review.opendev.org/#/c/563722/18:28
patchbotpatch 563722 - nova - ironic: add instance_uuid before any other spawn a... (MERGED) - 11 patch sets18:28
jrollmgoddard: this sounds very familiar, is this on master?18:28
jrollheh18:28
jrollI feel like that patch was to resolve the issue you just described :(18:28
mgoddardwhat seems to be happening is that if auto cleaning is enabled, it prevents removal of instance_info18:28
mgoddardnova triggers teardown, waits for completion, then tries to remove instance info18:29
jrollyes, this sounds very familiar18:29
mgoddardif the node has started cleaning by that point, we get a 409 due to the state transition18:29
mgoddardfamiliar as in 'let me dig out the fix'?18:29
mgoddardor 'yes we seem to hit this too'18:30
jrollfamiliar as in "I seem to recall some IRC chatter about this"18:30
jrollmeaning I hope there's a bug filed :)18:30
mgoddardhmm. I should pay more attention :)18:30
jrollanyway, let me look around18:31
mgoddardthanks18:32
mgoddardI couldn't see any relevant nova bugs18:32
mgoddardand from what I can tell I don't think its fixed in nova master18:32
jrollhm, site:bugs.launchpad.net/nova doesn't work as well in google as it used to18:34
jrollaha!18:41
jrollmgoddard: https://review.opendev.org/#/c/643484/18:41
patchbotpatch 643484 - nova - [ironic] ignore conflict upon cleaning up node - 4 patch sets18:41
jrollbug is there too18:42
mgoddardgreat find, thanks!18:42
mgoddardjroll: amazing we haven't seen this sooner18:44
jrollmgoddard: right?18:44
mgoddardit was pretty obvious18:45
jrollI kind of wonder if ironic is locking more/longer these days18:45
mgoddardmaybe people leave their nodes dirty?18:45
mgoddardit's not a lock in this case, the message is confusing18:45
jrolloh that too, people say cleaning takes too long18:45
mgoddardit's just because a state transition is in progress the update is blocked18:45
jrollah right18:46
jrollI think CLEANWAIT is legal for updating though, but maybe not18:46
mgoddardjroll: nope18:47
*** trident has quit IRC18:51
mgoddardthanks for digging that up jroll - heading off now o/18:51
jrollmgoddard: cheers, you're welcome18:52
*** trident has joined #openstack-ironic18:52
*** rachit7 has quit IRC19:03
*** goldyfruit has quit IRC19:21
*** Goneri has quit IRC19:32
*** Goneri has joined #openstack-ironic19:33
*** Goneri has quit IRC19:50
*** pcaruana has quit IRC19:52
*** whoami-rajat has quit IRC20:05
*** e0ne has joined #openstack-ironic20:10
*** goldyfruit has joined #openstack-ironic20:12
*** e0ne has quit IRC20:12
*** Lucas_Gray has joined #openstack-ironic20:19
*** Goneri has joined #openstack-ironic20:27
*** jcoufal has quit IRC20:38
*** ijw has joined #openstack-ironic20:46
*** trident has quit IRC20:59
*** trident has joined #openstack-ironic21:00
stevebakermorning21:08
*** rh-jelabarre has quit IRC21:09
*** Goneri has quit IRC21:42
*** jtomasek has quit IRC22:33
*** ijw has quit IRC22:52
*** rcernin has joined #openstack-ironic23:04
*** Lucas_Gray has quit IRC23:14
*** Lucas_Gray has joined #openstack-ironic23:15
*** Lucas_Gray has quit IRC23:43
*** goldyfruit_ has joined #openstack-ironic23:48
*** goldyfruit_ has quit IRC23:50
*** goldyfruit_ has joined #openstack-ironic23:50
*** goldyfruit has quit IRC23:52
*** goldyfruit_ has quit IRC23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!