Tuesday, 2020-09-15

*** bdodd has quit IRC00:00
*** rcernin has joined #openstack-ironic00:05
*** rcernin has quit IRC00:06
*** bdodd has joined #openstack-ironic00:12
stevebakergood afternoon00:41
openstackgerritChristopher Dearborn proposed openstack/ironic master: Redfish driver firmware update  https://review.opendev.org/74961900:48
*** rloo has quit IRC01:24
*** tzumainn has quit IRC01:26
*** ianychoi_ has joined #openstack-ironic01:46
*** ianychoi has quit IRC01:48
*** iurygregory has quit IRC02:08
*** rh-jelabarre has quit IRC03:12
*** rh-jelabarre has joined #openstack-ironic03:13
*** cdearborn has quit IRC03:17
*** rh-jelabarre has quit IRC03:18
*** Lucas_Gray has joined #openstack-ironic03:31
*** JohnMice has joined #openstack-ironic03:45
*** mkrai has joined #openstack-ironic03:52
*** Qianbiao has joined #openstack-ironic04:00
*** JamesBenson has quit IRC04:09
*** JamesBenson has joined #openstack-ironic04:11
*** JamesBenson has quit IRC04:50
*** uzumaki has joined #openstack-ironic05:06
uzumakigood morning ironic! o/05:08
openstackgerritSteve Baker proposed openstack/ironic master: Utility functions for REST API JSON handling  https://review.opendev.org/75116005:19
openstackgerritSteve Baker proposed openstack/ironic master: Convert allocations endpoint to plain JSON  https://review.opendev.org/74367905:19
openstackgerritSteve Baker proposed openstack/ironic master: Convert bios endpoint to plain JSON  https://review.opendev.org/74547505:19
openstackgerritSteve Baker proposed openstack/ironic master: Convert chassis endpoint to plain JSON  https://review.opendev.org/74547605:19
openstackgerritSteve Baker proposed openstack/ironic master: Convert conductors endpoint to plain JSON  https://review.opendev.org/74577905:19
openstackgerritSteve Baker proposed openstack/ironic master: Convert deploy_templates endpoint to plain JSON  https://review.opendev.org/74578005:19
openstackgerritSteve Baker proposed openstack/ironic master: Convert drivers endpoint to plain JSON  https://review.opendev.org/74186505:19
openstackgerritSteve Baker proposed openstack/ironic master: Convert event endpoint to plain JSON  https://review.opendev.org/74662305:19
openstackgerritSteve Baker proposed openstack/ironic master: Convert nodes endpoint to plain JSON  https://review.opendev.org/74782005:19
openstackgerritSteve Baker proposed openstack/ironic master: Convert ports endpoint to plain JSON  https://review.opendev.org/75012005:19
openstackgerritSteve Baker proposed openstack/ironic master: Convert portgroups endpoint to plain JSON  https://review.opendev.org/75086405:19
*** zzzeek has quit IRC05:19
openstackgerritSteve Baker proposed openstack/ironic master: Convert ramdisk endpoint to plain JSON  https://review.opendev.org/75085705:19
openstackgerritSteve Baker proposed openstack/ironic master: Convert volume endpoint to plain JSON  https://review.opendev.org/75085805:19
openstackgerritSteve Baker proposed openstack/ironic master: Convert volume/connectors endpoint to plain JSON  https://review.opendev.org/75085905:19
openstackgerritSteve Baker proposed openstack/ironic master: Convert volume/targets endpoint to plain JSON  https://review.opendev.org/75086005:19
openstackgerritSteve Baker proposed openstack/ironic master: Expunge the internal version of WSME  https://review.opendev.org/75086105:20
*** abdysn has joined #openstack-ironic05:21
*** zzzeek has joined #openstack-ironic05:22
*** JamesBenson has joined #openstack-ironic05:31
*** JamesBenson has quit IRC05:36
*** Qianbiao has quit IRC05:42
*** stevebaker has quit IRC05:45
*** stevebaker has joined #openstack-ironic05:46
*** jtomasek has joined #openstack-ironic06:03
*** dking has quit IRC06:06
*** dking has joined #openstack-ironic06:09
*** Lucas_Gray has quit IRC06:15
*** Qianbiao has joined #openstack-ironic06:27
arne_wiebalckGood morning, ironic!06:33
*** dking has quit IRC06:42
*** zzzeek has quit IRC06:45
*** zzzeek has joined #openstack-ironic06:47
QianbiaoGood morning ironic.06:52
QianbiaoMorning arne_wiebalck06:52
arne_wiebalckGood morning, Qianbiao o/06:53
Qianbiaohey o/  you are earlier today.06:53
*** mkrai has quit IRC06:53
QianbiaoDo u have a minute look at this story: https://storyboard.openstack.org/#!/story/2008142, am I correct?06:53
* arne_wiebalck is reading ...06:56
*** eandersson has quit IRC06:58
*** iurygregory has joined #openstack-ironic06:58
iurygregorygood morning Ironic!06:59
*** dking has joined #openstack-ironic06:59
*** eandersson has joined #openstack-ironic06:59
arne_wiebalckFrom what I understand, hardware managers are not invoked based on the found hardware. They are detected/found, and when functions need to be called, they are called in the order of the priority of the hardware managers.07:00
*** zzzeek has quit IRC07:01
*** eandersson has quit IRC07:01
arne_wiebalckSo, if your IPA has a special hardware manager built in, it will check it first for any function to be called before it goes to the generic h/w manager.07:01
Qianbiaoarne_wiebalck it seems the generic hardware is run before mlnx hardware.07:02
*** eandersson has joined #openstack-ironic07:02
arne_wiebalckThis can happen if a function is not overwritten in the mlnx one.07:02
arne_wiebalckBut again, this is how I understand things work.07:03
*** zzzeek has joined #openstack-ironic07:03
arne_wiebalckWe have a downstream hw manager and this is how I used it.07:03
Qianbiaoarne_wiebalck look at this: https://github.com/openstack/ironic-python-agent/blob/master/ironic_python_agent/hardware.py#L209507:04
Qianbiaothe hardware call order is restrict to this compare functino.07:04
Qianbiaois it wrong?07:05
arne_wiebalckNo, that is correct. But imagine you have call a function foo() which is not in the higher prio manager. It will then be called in the lower priority one.07:06
arne_wiebalckYou see what I mean.07:06
arne_wiebalck?07:06
arne_wiebalckThe managers are ordered, but if some of them do not implement a function, we fall back to the lower prio one. This may look like they are called out of order.07:07
arne_wiebalckBut they are not.07:07
Qianbiaoyes, you are right.07:08
arne_wiebalckYou may want to switch on debugging to see which functions are called in which order.07:09
arne_wiebalckhttps://docs.openstack.org/ironic-python-agent/latest/contributor/hardware_managers.html07:10
arne_wiebalcksection "How are methods executed on HardwareManagers?"07:10
arne_wiebalckThe other thing to check is whether your IPA really uses the mlnx hardware manager.07:11
arne_wiebalckOr only the generic one.07:11
arne_wiebalckNot sure if manager leave a log message automatically ... our downstream one does, so we know it is executed. I can check where we do this ...07:12
Qianbiao<arne_wiebalck> a big problem to me now is: inspect ramdisk log is truncated,07:13
Qianbiaoi do not know why, everytime it only send a small part of log back.07:13
QianbiaoMorning iurygregory07:14
arne_wiebalckQianbiao: in a meeting now, will get back to you after07:14
Qianbiaothanks !!07:14
iurygregoryhey Qianbiao o/07:20
QianbiaoHey o/07:21
Qianbiaoarne_wiebalck I find the issue, the first run hardware is IntelCnaHardwareManager, and it implements GenericHardwareManager, then mlnx hardware_mgr will never run, if hw supports IntelCnaHardwareManager07:22
*** ricolin has quit IRC07:32
QianbiaoI update the story, it maybe more clear now.07:32
*** JamesBenson has joined #openstack-ironic07:32
*** tosky has joined #openstack-ironic07:34
*** JamesBenson has quit IRC07:36
*** priteau has joined #openstack-ironic07:39
*** ricolin has joined #openstack-ironic07:42
openstackgerritVerification of a change to openstack/ironic failed: Add release note for dhcp-less deploy  https://review.opendev.org/75113907:49
Qianbiaohello iurygregory, if i want to update an old branch.07:53
openstackgerritzhufl proposed openstack/ironic master: Fix invalid assert_has_calls  https://review.opendev.org/75091007:54
QianbiaoI just git checkout stable/train; git checkout -b my-topic.07:54
Qianbiaothen update code and git review?07:54
Qianbiaoit will auto match to the stable branch?07:54
iurygregoryQianbiao, if the change only needs to go to stable/branch yes07:54
Qianbiaook thanks.07:55
iurygregoryif is something you want on master first you send the patch to master, than you do the backport07:55
Qianbiaosure, got.07:56
*** Abdallahyas has joined #openstack-ironic07:58
*** Abdallahyas has quit IRC08:00
*** abdysn has quit IRC08:01
*** mkrai has joined #openstack-ironic08:02
*** sshnaidm|afk is now known as sshnaidm08:04
openstackgerritVerification of a change to openstack/ironic failed: Remove install unnecessary packages  https://review.opendev.org/75163608:11
*** lucasagomes has joined #openstack-ironic08:13
arne_wiebalckQianbiao: I guess you can have more than 2 managers even.08:22
Qianbiaoarne_wiebalck not understand.08:23
QianbiaoI think it's clear that there is a bug in ipa, right?08:24
arne_wiebalckQianbiao: the managers are ordered by priority, so even if you have the intel one it does not mean you cannot have the mlnx one.08:24
Qianbiao<arne_wiebalck> the intel one will do everything.08:25
Qianbiaothen, the mlnx one will never run.08:25
arne_wiebalckcorrect08:25
Qianbiaobecause it has all interface method.08:26
arne_wiebalckyes08:26
arne_wiebalckthat is my understanding as well08:26
Qianbiaowhat is your suggestion to fix this.08:26
QianbiaoI am not quite understand "I guess you can have more than 2 managers even."08:26
arne_wiebalckwhat do you have the intel hw manager for?08:26
arne_wiebalckI mean, why do you use it?08:27
Qianbiaonot sure, our stack hardware all have it.08:27
arne_wiebalckRight. So, my understanding is that a hw manager is supposed to manage specific hardware and it seems the intel one is not the correct one for the nodes you're trying to manage.08:28
QianbiaoI think it's bug, the intel one should not inherit generic hw mgr.08:29
Qianbiaoif it only different in "collect_lldp_data", it should only implement "collect_lldp_data" function.08:30
arne_wiebalckI think all hardware managers need to implement evaluate_hardware_support(), no?08:31
arne_wiebalckYes.08:31
Qianbiaoyes08:31
Qianbiaoit must08:31
arne_wiebalckWhat ours does is to also call the evaluate_hardware_support() of the parent class.08:32
QianbiaoThis issue is about supporting multiple kind hardware in same server.08:34
arne_wiebalckCorrect.08:34
arne_wiebalckThis is what I meant with "a hw manager is supposed to manage specific hardware and it seems the intel one is not the correct one for the nodes you're trying to manage".08:34
QianbiaoYes08:34
arne_wiebalckDoes your hardware have a CNA network card?08:35
QianbiaoYes08:35
Qianbiaoand it has mlnx ib card too..08:35
QianbiaoThen bug come out.08:36
arne_wiebalck:)08:36
arne_wiebalckI am not sure this is a bug. You simply do not have a hardware manager which is able to manage your hardware.08:36
arne_wiebalckIt is a combination of two, actually.08:36
Qianbiaoarne_wiebalck do not agree with u. InterCna mgr should not inherit generic hw mgr.08:38
Qianbiaocurrently, it makes cna manager a high priority "generic" hardware manager.08:38
Qianbiaonone other hardware manager will work in this situation.08:39
arne_wiebalckthis is how it is supposed to be I think08:39
arne_wiebalckthe generic one is the most generic, it has the lowest priority08:39
Qianbiaoa spec hardware manager should only care about its special hardware part. but not all.08:39
arne_wiebalckcorrect08:40
QianbiaoCurrent, intel cna handles all hardware.08:40
*** derekh has joined #openstack-ironic08:40
Qianbiaochange "class IntelCnaHardwareManager(hardware.GenericHardwareManager):" -> "class IntelCnaHardwareManager(hardware.HardwareManager):"08:41
Qianbiaoeverything fixes.08:41
arne_wiebalckunless you want to run sth from the generic hardware manager08:41
Qianbiaoyes, all none spec hardware should be handled by generic one.08:42
arne_wiebalckright, but will it if you don't inherit from it?08:42
Qianbiaoso, if cna can not handle something(not implement function), it will auto use generic finally.08:42
Qianbiaoyes08:42
Qianbiaoif cna can not handle it, it will try low priority manager08:43
Qianbiaountil if find one can handle it.08:43
QianbiaoThis makes it support multiple hardware.08:43
arne_wiebalckoh, I think I see what you mean08:44
arne_wiebalck"Custom HardwareManagers should subclass hardware.HardwareManager or hardware.GenericHardwareManager. "08:45
Qianbiao:)08:45
arne_wiebalckSo, the inheritance from the generic one brings functionality to the cna one which should be taken by the mlnx one.08:46
Qianbiaoshould never inherit GenericHardwareMgr. It will make the whole mechanism fail.08:46
QianbiaoCustomer MGR08:46
arne_wiebalckOk ... but why would you ever inherit from the generic one?08:47
QianbiaoThe Intel CNA did this.. and it break things.08:47
Qianbiaomlnx one did the right thing.08:48
arne_wiebalckThe docs say, one can do this. If our understanding is correct, why would you ever inherit from the generic one?08:48
arne_wiebalckI think we need someone like dtantsur|afk to shed some light :)08:49
Qianbiao:)08:49
Qianbiaoyes08:49
QianbiaoI do not why, IMHO, one should never inherit generic mgr.08:49
arne_wiebalckexactly08:51
arne_wiebalckit works in almost all cases as few people have a hierarchy of manageres08:52
arne_wiebalckmanagers08:52
arne_wiebalckbut, yeah, with our current understanding I agree with you08:52
Qianbiaothis bug only happens when mlnx and cna card both appear in one server.08:53
*** hjensas|afk is now known as hjensas08:54
arne_wiebalckyes, but will appear with any combination of the cna manager and some other hardware08:55
*** JohnMice has quit IRC08:56
Qianbiaoright.08:59
QianbiaoHi, iurygregory. will this kind of patch will be accept on stable branches?08:59
Qianbiaohttps://storyboard.openstack.org/#!/story/200814408:59
*** dtantsur|afk is now known as dtantsur09:00
dtantsurmorning ironic09:00
Qianbiaohey dtantsur09:01
dtantsurQianbiao: as long as you don't *require* a new version, it's fine in my book09:01
dtantsurand hi :)09:01
Qianbiaoo/09:01
dtantsurof course assuming you won't rewrite half of the driver to support newer versions or introduce new user-visible features09:02
Qianbiao:) sure09:02
dtantsurQianbiao: oh, good catch re inheritance of hardware managers09:02
QianbiaoI still remember the issue cause by me.09:03
Qianbiaodtantsur You agree? change the inherit class of intel cna09:03
dtantsurQianbiao: I'm reading the code, it's a bit more subtle. gimme a minute09:04
Qianbiaosure09:04
dtantsurQianbiao: yes, you're right, it has to be changed. but it's not a simple one-line change because their collect_lldp_data method calls "super", which will break09:05
dtantsurso GenericHardwareManager.collect_lldp_data will have to be extracted to a common place09:06
Qianbiaook09:06
QianbiaoI will submit a patch to fix it.09:07
dtantsurthank you!09:07
Qianbiaoyw09:07
arne_wiebalckdtantsur: we were wondering if there is a situation where one would ever inherit from the generic one ...?09:08
arne_wiebalckdtantsur: it works if you have only one more specific one, but with multiple managers?09:10
dtantsurarne_wiebalck: the problem is this line https://opendev.org/openstack/ironic-python-agent/src/branch/master/ironic_python_agent/hardware_managers/cna.py#L7909:10
dtantsurit results in a copy of GenericHardwareManager with elevated priority09:11
dtantsur(MAINLINE vs GENERIC)09:11
dtantsurso it overrides everything of the same priority, including the mlnx hardware manager09:11
dtantsur(because GenericHardwareManager implements all methods)09:11
arne_wiebalckright09:12
arne_wiebalckbut09:12
arne_wiebalckif you have a custom hw manager with whatever prio and you inherit from the generic one you will always have this situation09:12
dtantsurthat's why you don't inherit unless you literally intend to raise priorities of all methods09:12
Qianbiaoarne_wiebalck IMHO, i do not think any mgr should inherit generic one. If it inherit generic, it will make itself the generic one.09:12
arne_wiebalckdtantsur: yes09:13
arne_wiebalckQianbiao: yes09:13
arne_wiebalckbut the docs should probably explain this point09:13
arne_wiebalck"Custom HardwareManagers should subclass hardware.HardwareManager or hardware.GenericHardwareManager."09:13
dtantsurQianbiao: I have a thought of a slightly different implementation09:14
arne_wiebalckAlternatively, should custom hw manager *ever* inherit from the generic one?09:14
dtantsurmaybe their collect_lldp_data should call _disable_embedded_lldp_agent_in_cna_card then raise IncompatibleHardwareMethodError to fall through09:14
dtantsurarne_wiebalck: unlikely09:15
dtantsurQianbiao: this would be an easier implementation and it would work correctly if another hardware manager also implements collect_lldp_info09:15
Qianbiaodtantsur this can do what we want. but i think it's not so directly.09:15
Qianbiaoa bit wierd to me.09:16
dtantsurit is weird, but that's how hardware managers are design09:16
Qianbiaothink about if there are another mgr which implement collect_lldp_info09:16
dtantsurIncompatbleHardwareMethodError means "go find another hardware manager to do this job"09:16
dtantsur(and we do have one - GenericHardwareManager)09:17
Qianbiaoand has a priority lower than intel but great than generic09:17
Qianbiaothen thing will go mess09:17
dtantsurwhy?09:17
dtantsur(btw there is no priorities in between)09:17
Qianbiaoi see, i mean assume.09:18
Qianbiaothen generic mgr's collect_lldp_info may not be called, but the one between intel and generic09:18
Qianbiaothis is not what we want09:18
dtantsurwhy not?09:18
dtantsurI think this is exactly what we want, because it may be able to handle some other network adapters09:19
dtantsur(collect_lldp_info interface is misdesigned, but that's another topic)09:19
Qianbiaohmm. I do not know how to explain. ~~09:20
Qianbiaobecause intel cna collect lldp exactly wants "1. disable xxx, 2. collect lldp exactly as the generic one did now"09:20
Qianbiaoif we raise error, the part2 is not so sure.09:21
dtantsurI don't think part 2 is correct, I think it's an oversight from whoever wrote that09:21
dtantsurbecause there may be other vendors on the same machine09:21
QianbiaoIN this sight, you are right.09:21
Qianbiaothis depends on how intel cna want the "collect lldp" looks like.09:22
dtantsurit seems that all they want is to disable some built-in LLDP facility beforehand09:23
dtantsurI'm not sure why they don't do it on start-up, to be honest09:23
Qianbiaook, will patch it as you said.09:24
*** Lucas_Gray has joined #openstack-ironic09:25
QianbiaoBut if so, every spec hardware mgr need to consider whether there are other hardware mgr will effect current one?09:26
QianbiaoThe 3rd hardware mgr should finish all it want to do, but not leave it to generic one or others.09:26
openstackgerritAnkit Kumar proposed openstack/ironic master: Adding changes for iso less vmedia support  https://review.opendev.org/75200109:27
dtantsurQianbiao: that's a common problem with hardware managers: they don't operate in isolation09:28
Qianbiaook make sense too.09:31
iurygregoryyay IPE installed with bifrost \o/ now I just need to check if I can collect data =D09:32
*** JamesBenson has joined #openstack-ironic09:33
openstackgerritArne Wiebalck proposed openstack/ironic-python-agent master: [doc] Clarify custom hardware manager inheritance  https://review.opendev.org/75200209:35
*** JamesBenson has quit IRC09:37
*** yolanda has joined #openstack-ironic09:37
openstackgerritQianBiao Ng proposed openstack/ironic stable/ussuri: opt: Enhance old stable branches to use latest python-ibmcclient  https://review.opendev.org/75200609:40
*** tkajinam has quit IRC09:49
Qianbiaodtantsur i got another issue. The inspect ramdisk log is always truncated.10:18
Qianbiaoany idea what may cause this.10:19
dtantsurI remember solving a similar problem in RDO ramdisks, I can take a look after lunch10:29
openstackgerritQianBiao Ng proposed openstack/ironic stable/ussuri: opt: Enhance old stable branches to use latest python-ibmcclient  https://review.opendev.org/75200610:37
uzumakiquick question, if I have a stand alone ironic setup, and want to use the openstack CLI to interact with it, how do I setup the RC file?10:37
iurygregoryuzumaki, you can setup a clouds.yaml file fore it10:38
iurygregoryfor*10:38
uzumakiand I'll have to mount that file to the ironic pod then?10:39
iurygregoryare you talking about metal3?10:40
uzumakii am, yes10:41
iurygregoryif you are deployment with the dev-env it should create the file for you10:41
uzumakiI got a messed up clean failed, and I don't want to scroll through gazillion lines of conductor logs, what I see in the node as "last_error" is timeout reached while waiting for ramdisk, I supposed I'll clear the maintenance flag and get rid of the node10:41
iurygregoryyou just need to say export OS_CLOUD=metal310:41
uzumakiI setup the environment, trying to be smart SMH xD10:42
iurygregoryhttp://paste.openstack.org/show/797873/10:42
iurygregoryyou can put a file like this in the dir you need and export OS_CLOUD=<name>10:43
iurygregoryand you should be able to use the openstack cli10:43
uzumakimucho gracias!10:43
*** ociuhandu has joined #openstack-ironic10:58
*** ociuhandu_ has joined #openstack-ironic11:00
*** JamesBenson has joined #openstack-ironic11:02
*** ociuhandu has quit IRC11:03
uzumakino connection adapters were found? what's that supposed to mean? iurygregory11:07
iurygregoryuzumaki, depends your clouds.yml is correct and did you set the env OS_CLOUD?11:07
uzumakiI did11:07
uzumakidoes it need a /v1/ or a /v1 only?11:08
uzumakithe endpoints11:08
iurygregoryhttp://<ip>:638511:09
iurygregoryhttp://<ip>:505011:09
uzumakiah! it worked11:09
uzumakiI was missing the http:// SMH11:09
uzumakimerci11:09
iurygregory=) glad it worked11:11
uzumaki\o/ what's sept 28? I recall yesterday you talked about an interesting CZ holiday11:11
uzumakiI'm curious11:11
iurygregoryits a holiday thats all I know lol11:11
uzumakibecause that's all that mtters? xD11:11
iurygregoryand I will be moving my things to a new flat11:11
uzumakiOhhh une apartment nouvelle? felicitations!11:12
iurygregorywhat? O.o11:12
uzumakiSorry, I got so excited I went French :D11:12
iurygregoryfrench?11:12
iurygregoryoh ok11:12
iurygregory=)11:12
uzumaki:D congratulations! \o/11:13
iurygregoryty11:13
uzumakihow much stuff do you have to carry?11:13
*** zzzeek has quit IRC11:24
iurygregorya lot hehehe11:24
iurygregorygoing to rent a van11:24
*** zzzeek has joined #openstack-ironic11:24
* iurygregory lunch time11:25
Qianbiaodtantsur thanks11:38
*** Goneri has joined #openstack-ironic11:44
*** rh-jelabarre has joined #openstack-ironic11:52
*** mkrai has quit IRC11:52
*** mkrai_ has joined #openstack-ironic11:53
*** rh-jelabarre has quit IRC11:54
*** jtomasek has quit IRC11:54
*** rh-jelabarre has joined #openstack-ironic11:54
openstackgerritAnkit Kumar proposed openstack/ironic master: Adding changes for iso less vmedia support  https://review.opendev.org/75200111:56
openstackgerritQianBiao Ng proposed openstack/ironic-python-agent master: Fix: make Intel CNA hardware manager none generic  https://review.opendev.org/75202411:56
*** ociuhandu_ has quit IRC12:01
*** ociuhandu has joined #openstack-ironic12:01
dtantsurhas anyone looked into what is happening with the CI? I see quite a few failures12:12
iurygregorydtantsur, I saw failures related to oslo.serialization version not found12:13
iurygregoryprobably some mirror problems..12:14
dtantsuryeah, that's likely a mirror lag12:14
iurygregorymaybe they have something on #opendev12:14
*** Lucas_Gray has quit IRC12:14
dtantsuron #openstack-infra12:14
dtantsurit seems to be an upstream pypi mirror problem12:14
dtantsurgreat timing, sigh12:15
iurygregoryouch =(12:15
iurygregorysometimes I just hate systemd...12:15
*** Lucas_Gray has joined #openstack-ironic12:16
iurygregorythe app run if put the commands on shell, on systemd  Active: inactive (dead) -.-'12:16
*** Goneri has quit IRC12:23
*** sshnaidm has quit IRC12:25
dtantsurmake sure you run in foreground12:26
iurygregorydtantsur, http://paste.openstack.org/show/797876/ my service looks like this (In devstack it works)..12:28
*** ociuhandu has quit IRC12:28
dtantsuriurygregory: other than FLASK_DEBUG=1, looks okay to me..12:29
dtantsuriurygregory: oh, and please don't use log files, log to journald instead12:30
dtantsur(that's maybe why you don't see any errors)12:30
iurygregorythe logs are empty .-. thats the strange part12:31
iurygregory+  Active: inactive (dead)12:31
iurygregoryhttp://paste.openstack.org/show/797877/12:31
dtantsurwhat exactly does the -D flag do?12:31
dtantsurI don't see it used on sushy-tools12:32
iurygregorydaemon12:32
iurygregoryin the devstack setup we use like this (for ipe)12:32
iurygregoryhttps://github.com/openstack/ironic-prometheus-exporter/blob/master/devstack/plugin.sh#L3612:32
dtantsuriurygregory: yep, exactly this thing you must not use with systemd :)12:32
dtantsursystemd managed processes should run in foreground, unless specially configured to talk to systemd12:33
dtantsurI think it just forks away and systemd loses track of it12:33
iurygregoryhummm12:34
* iurygregory removes -D12:34
jandershttps://i.imgur.com/D1XK8nk.gif12:34
iurygregoryActive: active (running)  =O12:35
iurygregoryoh yes!12:35
iurygregoryhttps://media.giphy.com/media/lOa0tPKiMLdqVdFiS8/giphy.gif12:35
dtantsurwe should probably use `--access-logfile -` to make it log to stderr12:35
dtantsur(--error-logfile defaults to - as it seems)12:35
openstackgerritDmitry Tantsur proposed openstack/bifrost master: redfish-emulator: log handled requests  https://review.opendev.org/75203612:38
*** priteau has quit IRC12:42
*** Goneri has joined #openstack-ironic12:49
*** sshnaidm has joined #openstack-ironic12:50
*** priteau has joined #openstack-ironic12:55
*** uzumaki has quit IRC12:59
janderssee you tomorrow Ironic o/13:03
*** ociuhandu has joined #openstack-ironic13:12
*** priteau has quit IRC13:15
TheJuliagood morning13:17
openstackgerritIury Gregory Melo Ferreira proposed openstack/bifrost master: Add ipe installation to bifrost  https://review.opendev.org/75204213:19
TheJuliais CI any happier today?13:21
TheJulianope13:24
*** ociuhandu has quit IRC13:29
dtantsurmorning TheJulia13:32
dtantsurno, the CI is in a terrible state13:32
TheJulia:(13:35
TheJuliaall the fastly proxies?13:35
dtantsurTheJulia: seems to boil down to them, yes13:36
openstackgerritvinay kumar muddu proposed openstack/ironic master: Adds ilo-uefi-https boot interface to ilo5  https://review.opendev.org/74550113:38
openstackgerritYogesh Ramachandra proposed openstack/ironic master: Fix idrac-wsman bios factory-reset setting  https://review.opendev.org/74869613:39
*** ociuhandu has joined #openstack-ironic13:46
*** tzumainn has joined #openstack-ironic13:47
TheJuliawell, code reviews could be a thing instead!13:48
*** uzumaki has joined #openstack-ironic13:48
*** yolanda has quit IRC13:53
*** yolanda has joined #openstack-ironic13:54
*** hjensas is now known as hjensas|afk13:56
*** bbezak_ has joined #openstack-ironic13:57
*** sri___ has joined #openstack-ironic13:57
*** arne_wiebalck_ has joined #openstack-ironic13:57
*** andrein_ has joined #openstack-ironic13:57
*** phrobb_ has joined #openstack-ironic13:57
*** v1k0d3n_ has joined #openstack-ironic13:57
*** gmann_ has joined #openstack-ironic13:57
*** guilhermesp_ has joined #openstack-ironic13:57
*** priteau has joined #openstack-ironic13:57
*** sri_ has quit IRC14:00
*** guilhermesp has quit IRC14:00
*** bbezak has quit IRC14:00
*** v1k0d3n has quit IRC14:00
*** fungi has quit IRC14:00
*** arne_wiebalck has quit IRC14:00
*** andrein has quit IRC14:00
*** seba has quit IRC14:00
*** logan- has quit IRC14:00
*** gmann has quit IRC14:00
*** phrobb has quit IRC14:00
*** jroll has quit IRC14:00
*** dtantsur has quit IRC14:00
*** antonym has quit IRC14:00
*** sri___ is now known as sri_14:00
*** bbezak_ is now known as bbezak14:00
*** arne_wiebalck_ is now known as arne_wiebalck14:00
*** guilhermesp_ is now known as guilhermesp14:00
*** v1k0d3n_ is now known as v1k0d3n14:00
*** logan_ has joined #openstack-ironic14:00
*** phrobb_ is now known as phrobb14:00
*** gmann_ is now known as gmann14:00
*** andrein_ is now known as andrein14:00
*** logan_ is now known as logan-14:00
*** jamesdenton has quit IRC14:03
*** jamesdenton has joined #openstack-ironic14:05
*** antonym has joined #openstack-ironic14:07
*** jroll has joined #openstack-ironic14:07
*** fungi has joined #openstack-ironic14:08
*** dtantsur has joined #openstack-ironic14:08
*** rloo has joined #openstack-ironic14:15
dtantsurTheJulia: do you know whom to ask to allow changing timezone for the openstack summit schedule?14:15
dtantsurit's not exactly convenient to do the math on fly14:15
dtantsur(although with a virtual summit I can watch everything later)14:17
TheJuliadtantsur: jamesmcarthur (jimmy) would be the person to talk to14:21
dtantsurTheJulia: thanks! I will unless this thought will get buried under dozens of other thoughts..14:22
TheJulia:)14:28
TheJuliaawesome14:28
TheJuliaiurygregory: w/r/t https://review.opendev.org/#/c/705030/ i think we need to make it non-voting for the multinode job until we sort out the issues there.14:29
patchbotpatch 705030 - ironic - Native zuulv3 grenade multinode multitenant - 33 patch sets14:29
TheJuliait presently votes in the check run to fail which will prevent it from merging.14:29
openstackgerritQianBiao Ng proposed openstack/ironic-python-agent master: Fix: make Intel CNA hardware manager none generic  https://review.opendev.org/75202414:31
openstackgerritvinay kumar muddu proposed openstack/ironic master: Adds ilo-uefi-https boot interface to ilo5  https://review.opendev.org/74550114:31
openstackgerritDmitry Tantsur proposed openstack/ironic master: Deprecate the iscsi deploy interface  https://review.opendev.org/75020415:02
openstackgerritQianBiao Ng proposed openstack/ironic-python-agent stable/train: Fix: make Intel CNA hardware manager none generic  https://review.opendev.org/75206515:07
TheJuliaQianbiao: typically we only backport once the change has merged to master15:09
TheJuliaQianbiao: but then we need to backport that to... multiple branches :)15:09
QianbiaoTheJulia I just need to click cherry-pick and choose branches, right?15:10
Qianbiaoall branch?15:10
*** dhellmann_ has joined #openstack-ironic15:10
*** dhellmann has quit IRC15:11
*** dhellmann_ is now known as dhellmann15:12
TheJuliano. In order typically master -> the bugfix branches -> stable/ussuri, stable/train, stable/stein, stable/rocky (optional, but preferred), stable/queens (optional, but preferred)15:12
Qianbiaook15:13
toskynot sure whether clicking from the UI does the same as git cherry-pick -x15:13
QianbiaoI need to wait it merged to master? after that i can cherry pick.15:14
TheJuliaif merged, the ui does the proper tagging of the commit message15:14
TheJuliabut *shrug*15:14
TheJuliaas long as we know where it came from in master I think we're good15:14
TheJuliaQianbiao: basically yes15:15
Qianbiaook, np, i will do that.15:15
TheJuliaawesome, thanks!15:15
*** ociuhandu_ has joined #openstack-ironic15:17
Qianbiaoyw :)15:18
*** ociuhandu has quit IRC15:21
*** ociuhandu_ has quit IRC15:22
*** mkrai_ has quit IRC15:23
Qianbiaodtantsur the truncated inspect ramdisk log issue, should i create a story to track it15:27
dtantsuryes please (against IPA-builder)15:27
QianbiaoIt can be reproducted in my env. Not sure why.15:27
Qianbiaook15:27
Qianbiaodtantsur https://storyboard.openstack.org/#!/story/200814815:35
*** hjensas|afk is now known as hjensas15:35
*** ianychoi__ has joined #openstack-ironic15:41
*** ianychoi_ has quit IRC15:45
TheJuliaJayF: Yeah, the change from poll to eventlet for the heartbeat is definitely breaking :\15:47
TheJuliaerr, select15:48
*** uzumaki has quit IRC15:49
TheJulianot sure why but requests may not work it seems15:50
dtantsurmaybe the sleep change?15:53
TheJuliaIf I'm remembering how requests calls all the way into urllib... I don't think it should but maybe it is a timing thing at that point15:54
TheJuliaand then becomes random hit/miss15:54
TheJuliawhich is actually kind of like what we're seeing15:54
TheJuliabut... yeah.15:54
JayFYeah; I'm not sure, and bluntly I'm not sure how much value it brings to troubleshoot that.15:56
JayFI'd be happy to abandon it if you all agree.15:56
TheJuliaI was kind of wondering the same question15:56
TheJuliaI mean... I love how much cleaner it is15:56
dtantsurcan we try with reverting to sleep(0.1) first?15:56
dtantsurat least it would give us a hint15:57
openstackgerritJay Faulkner proposed openstack/ironic-python-agent master: Simplify heartbeating by removing use of select()  https://review.opendev.org/74677415:58
JayFI mean, I'm skeptical given sleep(0) wasn't breaking before15:58
JayFbut it's a simple enough edit15:58
dtantsuryeah, I doubt as well.. but to rule out for sure15:58
JayFI suspect this is yet-another-case of needing real threading OR using eventlet everywhere, instead of this wacky middle ground15:58
dtantsuragree15:58
dtantsurtime for futurist and its periodic tasks?15:59
TheJuliadefinitely15:59
* TheJulia might try to convince someone on that16:00
dtantsurJayF: another thought: what if we import eventlet.event.Event explicitly? instead of threading.Event?16:00
JayFI like that idea, a lot actually16:01
JayFlets see how the first change tests out16:01
TheJulia++16:01
JayFI'll try to remember to try that after16:01
JayFbut if I forget feel free to edit in (your) tomorrow morning16:01
TheJuliaif it is the sleep in general, then importing and using directly will cause the heartbeater to most likely freezze16:01
*** mkrai has joined #openstack-ironic16:01
TheJuliaI _bet_ this is exactly what is happening with requests but without injecting a bunch of debugging in to requests I don't think we would see it16:01
*** lucasagomes has quit IRC16:01
dtantsurand probably use eventlet.spawn instead of threading.Thread16:02
dtantsur(we can play with combinations of these solutions)16:02
JayF...or just monkey patch thread, too16:02
JayFif it didn't kaboom, it'd get us closer to "eventlet everything"16:02
dtantsurthat's probably the first option to try, really16:04
TheJuliaI have indecisive thoughts on this matter that request tequila to be delivered16:06
*** gyee has joined #openstack-ironic16:06
dtantsurpython-eventlet.spec: Requires: tequila>=216:07
* dtantsur wonders if people understand rpm jokes16:07
openstackgerritVerification of a change to openstack/ironic failed: Also wipe agent token on manual power off or reboot  https://review.opendev.org/75181316:08
TheJuliaI wonder if stevebaker would be interested in going down the rabbit hole of looking at evenleting it completely16:09
TheJuliadtantsur: oh yes, I got it :)16:09
JayFdtantsur: they make my head spin16:09
JayF(bam, double-rpm joke)16:09
dtantsur:D16:10
-openstackstatus- NOTICE: Our PyPI caching proxies are serving stale package indexes for some packages. We think because PyPI's CDN is serving stale package indexes. We are sorting out how we can either fix or workaround that. In the meantime updating requirements is likely the wrong option.16:10
*** mkrai has quit IRC16:10
TheJuliaall my concern and worry is largely around requests timeouts since we've seen the agent tends to get fired up in super grumpy networks16:11
TheJuliaspeaking of which... inspection connection retries!16:11
Qianbiaohello, when package ipa with dib, where ipa code will be put?16:19
Qianbiaowhich path16:19
dtantsurQianbiao: installed into /opt/ironic-python-agent, I think16:20
dtantsurhttps://opendev.org/openstack/ironic-python-agent-builder/src/branch/master/dib/ironic-python-agent-ramdisk/install.d/ironic-python-agent-ramdisk-source-install/60-ironic-python-agent-ramdisk-install#L12-L1416:21
Qianbiao<dtantsur> thanks16:22
Qianbiao.... it seems it does not package latest code.16:23
TheJuliaWhat version is getting packaged?16:23
QianbiaoTheJulia I am using this script: https://storyboard.openstack.org/#!/story/200814816:24
Qianbiaoit should use my local repo fold HEAD commit16:24
Qianbiaobut it does not.16:24
dtantsurQianbiao: how exactly do you configure that?16:25
Qianbiaocheck the story: export DIB_REPOLOCATION_ironic_python_agent="/opt/stack/ironic-python-agent"16:25
Qianbiaodtantsur ^^16:26
dtantsurQianbiao: do you set DIB_REPOREF_ironic_python_agent=HEAD? I'm not 100% sure it's needed, but we do it in the CI.16:27
Qianbiaoyes16:27
Qianbiaoi set that.16:27
dtantsuraha, I see now16:27
dtantsurhmmm, the instruction on the story looks correct16:29
TheJuliaI'm thinking the same16:29
TheJuliado you have the build log from the ramdisk creation?16:29
TheJuliaI ask because that version of the agent is not logging anything upon startup16:29
TheJuliait could be your local branch, but we've had that in the agent for close to a year I think.16:29
Qianbiao<TheJulia> will rebuild for now16:30
Qianbiao* --for--16:30
dtantsuras to trimmed logging, I think we need something like https://review.rdoproject.org/r/#/c/21142/16:31
Qianbiaohow can i get the ipa versin in built initramfs.16:32
Qianbiaoi have uncompress it.16:32
*** jlvillal has quit IRC16:32
*** jlvillal has joined #openstack-ironic16:32
Qianbiao<dtantsur> you mean log to systemd, and then journal get it, then send it back.16:33
dtantsurwell, yeah. currently we log to stdout, which gets to journald, which we send back16:33
dtantsurthis stdout -> journald process is buffered, so something is usually trimmed16:33
dtantsurthe RDO change I made some time ago makes IPA (or rather oslo.log) talk to journald directly16:34
dtantsurwhich is unbuffered and real-time16:34
Qianbiaonice16:35
Qianbiaothe version is: ironic_python_agent-5.0.4.dev8-py3.6.egg-info16:36
dtantsurbfournie: could you check if https://review.opendev.org/751813 fixes https://storyboard.openstack.org/#!/story/2008097 ?16:37
patchbotpatch 751813 - ironic - Also wipe agent token on manual power off or reboot - 2 patch sets16:37
TheJuliafrom with-in the runtime is there not a way to turn off python's buffered io?16:38
dtantsurTheJulia: I don't remember, I think I tried. It's not just about Python, also about glibc, I'm afraid..16:40
TheJuliathat would explain why only the env-var works afaik16:40
*** dtantsur is now known as dtantsur|afk16:46
dtantsur|afko/16:46
dtantsur|afksee you tomorrow, hopefully with the CI working16:46
TheJuliagoodnight!16:46
Qianbiaosee you16:47
QianbiaoTheJulia http://paste.openstack.org/show/797894/16:47
Qianbiaoit's wierd, this head is not my HEAD local16:47
QianbiaoMy git log is: http://paste.openstack.org/show/797895/16:49
TheJuliatry setting the ref to ab639cc672d984f897ad1117c0e98d512dc0c7e616:49
TheJuliaI suspect this might be a mechanics interaction with the internal clone16:50
Qianbiaoagree16:51
Qianbiaowill try16:51
QianbiaoTheJulia http://paste.openstack.org/show/797896/16:54
Qianbiaoit seems it must get ref from origin?16:55
Qianbiaoor i should use a short commit id16:55
TheJuliagit reflog16:55
TheJuliayeah, it really wants the ref :\16:55
TheJuliabut reflog should give you visibility on what the tag is to use16:56
*** derekh has quit IRC16:56
*** jtomasek has joined #openstack-ironic16:59
QianbiaoTheJulia it did not support local ref from not origin. http://paste.openstack.org/show/797897/17:01
Qianbiao~~17:01
TheJuliaugh17:01
openstackgerritQianBiao Ng proposed openstack/ironic-python-agent master: Fix: make Intel CNA hardware manager none generic  https://review.opendev.org/75202417:32
Qianbiaosorry, TheJulia, did not run tox testcase locally.17:32
Qianbiao(network here is really hard to download packages with tox)17:33
*** tosky has quit IRC17:40
*** Qianbiao is now known as Qianbiao|afk17:44
*** Qianbiao|afk has quit IRC17:44
openstackgerritQianBiao Ng proposed openstack/ironic stable/ussuri: opt: Enhance old stable branches to use latest python-ibmcclient  https://review.opendev.org/75200617:46
*** Lucas_Gray has quit IRC17:50
*** dsneddon has quit IRC17:53
*** cdearborn has joined #openstack-ironic18:20
*** ociuhandu has joined #openstack-ironic18:29
*** ociuhandu has quit IRC18:34
*** VisBits has joined #openstack-ironic18:35
VisBits Hi,  I am curious how the driver implementation works for adding the PXE options to neutron-dhcp-agent.. i cannot figure how how that works.  My issue is i have a neutron network (not discovery or cleaning) that is adding the PXE entries for the ironic pxe server and my baremetal host are rebooting into discovery upon provision..18:35
*** zzzeek has quit IRC18:39
*** zzzeek has joined #openstack-ironic18:41
TheJuliao/18:44
TheJuliaVisBits: so that sounds like your ironic-inspector service is not filtering its configuration or firewalling properly18:45
TheJuliawell18:46
TheJuliaare the PXE entries on the port themselves?18:46
TheJuliathe settings would be set if the machine is set to network boot. We recommend local booting machines.18:46
*** VisBits has quit IRC18:48
TheJulia\o/18:49
*** zzzeek has quit IRC18:51
*** VisBits has joined #openstack-ironic18:52
*** Lucas_Gray has joined #openstack-ironic18:52
*** zzzeek has joined #openstack-ironic18:53
openstackgerritJulia Kreger proposed openstack/ironic-python-agent master: Fix: make Intel CNA hardware manager none generic  https://review.opendev.org/75202418:53
*** zzzeek has quit IRC19:00
*** zzzeek has joined #openstack-ironic19:03
*** zzzeek has quit IRC19:10
*** zzzeek has joined #openstack-ironic19:12
*** priteau has quit IRC19:16
*** dsneddon has joined #openstack-ironic19:26
openstackgerritChristopher Dearborn proposed openstack/ironic master: Redfish driver firmware update  https://review.opendev.org/74961919:27
*** rloo has quit IRC19:40
*** rloo has joined #openstack-ironic19:40
openstackgerritVerification of a change to openstack/ironic-inspector failed: Add NVIDIA V100 to known accelerator devices  https://review.opendev.org/75166020:29
*** stevebaker has quit IRC21:00
openstackgerritChristopher Dearborn proposed openstack/ironic master: Redfish driver firmware update  https://review.opendev.org/74961921:10
*** stevebaker has joined #openstack-ironic21:23
*** JamesBenson has quit IRC21:24
*** Lucas_Gray has quit IRC21:50
*** Lucas_Gray has joined #openstack-ironic21:51
*** k_mouza has joined #openstack-ironic21:53
iurygregoryTheJulia, yeah I will update the patch to put non-voting =)22:09
stevebakerTheJulia: what is still not eventletted?22:17
* stevebaker looks for where to start in the backscroll22:19
*** rcernin has joined #openstack-ironic22:25
TheJuliaiurygregory: thanks22:27
TheJuliastevebaker: so the tl;dr is that the agent has some... interesting code22:28
TheJuliastevebaker: specifcially for the heartbeater which we launch as a separate thread and only kill it off if we need to exit.  That is basically unchanged and this continues to kind of bite us in other areas where we end up seeing weird things with eventlet/threading in other parts of the app22:29
stevebakerTheJulia: I see. When I have a gap I'll see if I can get my head around how it works currently22:32
TheJuliahappy to spend some time walking through it with you22:33
TheJuliabut yeah, someone who has never really looked deeply into is not indoctrinated into it either :)22:33
TheJuliawoot, looks like CI issues may soon be resoled22:35
*** k_mouza has quit IRC22:51
*** tkajinam has joined #openstack-ironic22:51
JayFstevebaker: essentially: in the __init__.py where we monkey patch, change monkey patch to all=True, and image streaming breaks. That's the first thing. You'll probably also need https://review.opendev.org/#/c/746774/22:55
patchbotpatch 746774 - ironic-python-agent - Simplify heartbeating by removing use of select() - 25 patch sets22:55
JayFstevebaker: (in the gate; image streaming is the metalsmith job)22:56
openstackgerritJay Faulkner proposed openstack/ironic-python-agent master: Simplify heartbeating by removing use of select()  https://review.opendev.org/74677422:58
JayF^^ sleep(0.1) helped nothing; trying the monkey_patch of threading as discussed this morning22:58
*** k_mouza has joined #openstack-ironic23:00
*** Goneri has quit IRC23:03
*** k_mouza has quit IRC23:05
TheJuliablarg, ci still not happy23:10
TheJuliaJayF: ack :(23:11
VisBits Hi,  I am curious how the driver implementation works for adding the PXE options to neutron-dhcp-agent.. i cannot figure how how that works.  My issue is i have a neutron network (not discovery or cleaning) that is adding the PXE entries for the ironic pxe server and my baremetal host are rebooting into discovery upon provision..23:22
*** k_mouza has joined #openstack-ironic23:46
*** k_mouza has quit IRC23:50
*** tzumainn has quit IRC23:52
*** Lucas_Gray has quit IRC23:55

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!