Tuesday, 2020-01-07

*** TxGirlGeek has joined #openstack-ironic00:09
*** bobmel has joined #openstack-ironic00:13
*** TxGirlGeek has quit IRC00:15
*** bobmel has quit IRC00:17
*** rloo has quit IRC00:40
*** ricolin_ has joined #openstack-ironic00:44
*** ricolin_ has quit IRC00:45
*** cdearborn has quit IRC01:13
*** yedongcan has joined #openstack-ironic01:28
openstackgerritMerged openstack/ironic master: Refactor glance retry code to use retrying lib  https://review.opendev.org/70107801:32
openstackgerritKaifeng Wang proposed openstack/ironic master: Make qemu hook running with python3  https://review.opendev.org/70099401:35
*** igordc has joined #openstack-ironic01:49
*** Goneri has quit IRC02:19
*** goldyfruit_ has joined #openstack-ironic02:35
*** goldyfruit_ has quit IRC02:36
*** goldyfruit_ has joined #openstack-ironic02:36
*** igordc has quit IRC02:43
*** bobmel has joined #openstack-ironic02:56
*** bobmel has quit IRC03:00
*** rcernin_ has joined #openstack-ironic03:13
*** rcernin has quit IRC03:13
*** mkrai has joined #openstack-ironic03:28
*** rcernin_ has quit IRC03:56
*** rcernin has joined #openstack-ironic04:00
openstackgerritJulia Kreger proposed openstack/bifrost master: Skip status code check and make vagrant install quicker  https://review.opendev.org/70106704:22
TheJuliamgoddard: re ^^^ so, when I run test-bifrost.sh locally, It puts my local test vm into a not great state.. but install.yaml works just fine and gets a working ironic deployment. :\04:23
*** goldyfruit_ has quit IRC04:24
*** goldyfruit_ has joined #openstack-ironic04:24
*** mkrai has quit IRC04:29
*** mkrai_ has joined #openstack-ironic04:29
*** bobmel has joined #openstack-ironic04:44
*** goldyfruit_ has quit IRC04:48
*** bobmel has quit IRC04:49
*** tzumainn has quit IRC04:53
*** gyee has quit IRC05:00
*** goldyfruit_ has joined #openstack-ironic05:13
*** ociuhandu has joined #openstack-ironic05:30
*** ociuhandu has quit IRC05:35
*** pcaruana has joined #openstack-ironic06:00
*** pcaruana has quit IRC06:14
*** ileixe has joined #openstack-ironic06:53
ileixeHello Ironic /06:53
ileixeI have some question about ironic network06:53
*** rcernin has quit IRC06:54
ileixeis there any reference architecture to integrate Neutron vlan, flat network?06:54
ileixeI tried to use flat provider network but it's quite difficult to scale06:54
ileixeany advice?06:54
*** henriqueof has quit IRC07:02
*** henriqueof1 has joined #openstack-ironic07:02
*** jawad_axd has joined #openstack-ironic07:18
*** bobmel has joined #openstack-ironic07:26
*** bobmel has quit IRC07:31
mkrai_ileixe, Hi, you can refer https://docs.openstack.org/ironic/latest/admin/multitenancy.html07:35
kaifenghi ileixe, we have some reference architectures at https://docs.openstack.org/ironic/latest/install/refarch/index.html07:35
mkrai_kaifeng, Hi good morning07:36
kaifengit's not complete though.07:36
mkrai_And a very happy new year to all Ironicers :)07:36
kaifengGood morning mkrai_ o/07:36
mkrai_kaifeng, I have question about sushy. Sushy sends Redfish APIs to BMC or some agent?07:37
arne_wiebalckGood morning, ironic!07:41
kaifengmkrai_: I have bare knowledge on sushy, as I understand it, sushy talks to BMCs that support redfish APIs07:42
kaifengGood morning, arne_wiebalck o/07:43
arne_wiebalckHey kaifeng o/07:43
mkrai_arne_wiebalck, Hi, good morning o/07:43
arne_wiebalckHey mkrai o/07:43
mkrai_kaifeng, Thanks!07:46
kaifengmkrai_: sushy itself is not a service, it's a library consumed by services like ironic.07:47
mkrai_kaifeng, Yes I know that. I am just searching if there's a redfish agent in openstack07:48
kaifengmkrai_: no idea on redfish agent, what do we expect it do?07:50
ileixemkrai_, kaifeng: Thanks! I will check that07:52
arne_wiebalckmkrai_: I understand it the same way kaifeng does. For specific redfish questions etingof is probably the best to talk to.07:54
ileixehttps://docs.openstack.org/ironic/latest/admin/multitenancy.html from the reference07:55
ileixedoes it saying I need special ML2 implmentation ? (Cisco/FUJITSU..)07:55
mkrai_kaifeng, configure the settings on the node, in simple terms the same redfish does with BMC07:56
mkrai_arne_wiebalck, thanks my understanding is same07:56
mkrai_ileixe, yes there are different ML2 drivers for different type of switch hardware07:57
ileixeOh.. is there any H/W list to support?07:57
kaifengileixe: in case of vlan network you'll need some ml2 driver to apply switch config for you.07:58
ileixeusually i never touched ToR, so I could not find the right way07:58
kaifengileixe: I haven't used others, but networking-generic-switch works for us, it supports many vendor switches.07:58
ileixe:) Thanks07:58
kaifengmkrai_: I am not sure I understand, redfish is a protocol..07:59
kaifengsounds like firware management tool in the host machine?08:00
kaifengs/firware/firmware/08:00
* kaifeng runs into a meeting08:01
mkrai_kaifeng, yes something similar.08:02
*** iurygregory has joined #openstack-ironic08:02
iurygregorygood morning Ironic08:03
kaifenghi iurygregory o/08:04
iurygregorykaifeng, o/08:04
kaifengmkrai_: etingof might know :)08:04
mkrai_kaifeng, Ok will check with him, thanks :)08:05
mkrai_etingof, hi o/08:05
*** tesseract has joined #openstack-ironic08:17
*** pcaruana has joined #openstack-ironic08:29
*** FlorianFa has quit IRC08:30
*** aedc_ has quit IRC08:34
*** rpittau|afk is now known as rpittau08:36
rpittaugood morning ironic! o/08:36
*** bobmel has joined #openstack-ironic08:40
etingofmkrai_, o/08:41
etingofmkrai_, redfish agent normally lives inside BMC. we have sushy-emulator that effectively implements kind of redfish BMC for testing purposes08:42
etingofmkrai_, sushy is a redfish client that abstracts away some redfish protocol details into higher-level operations08:43
mkrai_etingof, Hi o/08:43
etingofmkrai_, ironic does all its red/fishy business via sushy08:43
mkrai_etingof, Ok so there's no redfish agent that runs on the baremetal nodes, right?08:44
etingofmkrai_, it is! but this spaghetti monster is hiding inside BMC firmware08:45
etingofmkrai_, redfish agent is not running on the server itself08:46
mkrai_etingof, Ok got it, thanks!08:46
*** priteau has joined #openstack-ironic08:47
ileixeFor networking-generic-switch, I think it needs permission of switch control. How do you guys handle of this issue? I assume that cloud does not touch underlay network ..08:48
ileixeWe have strictly seperated role hm..08:49
iurygregorymorning rpittau o/08:52
rpittauhey iurygregory :)08:52
*** tridde has quit IRC08:53
*** trident has joined #openstack-ironic08:55
*** jawad_axd has quit IRC08:57
*** jawad_axd has joined #openstack-ironic09:03
*** jawad_axd has quit IRC09:03
*** dougsz has joined #openstack-ironic09:04
*** dmellado has quit IRC09:07
*** dmellado has joined #openstack-ironic09:08
*** dougsz has quit IRC09:09
*** lucasagomes has joined #openstack-ironic09:10
kaifengilexi: right, you need to provide credentials in the driver configuration file09:14
kaifenghey rpittau o/09:14
rpittauhey kaifeng :)09:14
*** dougsz has joined #openstack-ironic09:29
*** derekh has joined #openstack-ironic09:36
*** alexmcleod has joined #openstack-ironic09:38
*** dtantsur|afk is now known as dtantsur09:40
dtantsurTheJulia: sorry, I forgot that I have a holiday yesterday. I hope you get better today!09:41
dtantsurmorning ironic09:41
iurygregorymorning dtantsur09:42
*** khansa has joined #openstack-ironic09:51
*** aedc has joined #openstack-ironic09:54
*** ociuhandu has joined #openstack-ironic10:00
*** khansa has quit IRC10:01
*** mkrai_ has quit IRC10:03
*** dougsz has quit IRC10:05
*** ociuhandu has quit IRC10:08
*** afasano has joined #openstack-ironic10:09
*** dougsz has joined #openstack-ironic10:22
kaifengmorning dtantsur o/10:29
*** priteau has quit IRC10:37
*** ociuhandu has joined #openstack-ironic10:39
*** dougsz has quit IRC10:54
*** ociuhandu has quit IRC10:54
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic-python-agent master: Avoid grub2-install when on UEFI boot mode  https://review.opendev.org/69691410:55
*** ociuhandu has joined #openstack-ironic10:55
*** ociuhandu has quit IRC10:56
*** ociuhandu has joined #openstack-ironic10:56
*** mkrai_ has joined #openstack-ironic11:01
*** ociuhandu has quit IRC11:02
*** ociuhandu has joined #openstack-ironic11:05
*** ociuhandu has quit IRC11:06
*** ociuhandu has joined #openstack-ironic11:06
*** dougsz has joined #openstack-ironic11:10
*** ociuhandu has quit IRC11:11
*** rpittau is now known as rpittau|bbl11:16
*** khansa has joined #openstack-ironic11:18
*** khansa has quit IRC11:21
*** khansa has joined #openstack-ironic11:31
*** yedongcan has left #openstack-ironic11:33
*** khansa has quit IRC11:35
*** khansa has joined #openstack-ironic11:36
*** Lucas_Gray has joined #openstack-ironic11:40
*** khansa has quit IRC11:42
*** bobmel has quit IRC11:54
*** aedc has quit IRC12:03
*** hjensas|afk is now known as hjensas12:14
openstackgerritDmitry Tantsur proposed openstack/ironic-tempest-plugin master: Explicitly tear down software RAID after testing it  https://review.opendev.org/68392112:24
*** bfournie has quit IRC12:41
*** khansa has joined #openstack-ironic12:43
*** khansa has quit IRC12:47
*** rachit7 has joined #openstack-ironic12:52
*** sziviani has joined #openstack-ironic12:55
*** khansa has joined #openstack-ironic12:56
*** jawad_axd has joined #openstack-ironic12:58
*** khansa has quit IRC13:02
*** rh-jelabarre has joined #openstack-ironic13:05
*** mkrai_ has quit IRC13:10
*** goldyfruit_ has quit IRC13:20
*** rpittau|bbl is now known as rpittau13:26
*** rloo has joined #openstack-ironic13:26
*** Lucas_Gray has quit IRC13:30
*** bfournie has joined #openstack-ironic13:33
openstackgerritMerged openstack/ironic master: Increasing BUILD_TIMEOUT value for multinode job  https://review.opendev.org/69872013:41
*** Goneri has joined #openstack-ironic13:42
*** Lucas_Gray has joined #openstack-ironic13:42
dtantsuretingof, TheJulia, hey, can we setup something (a public call maybe?) to figure out the L3 spec directions? I feel like my comments are not getting addressed, and I think at this point the update has drifted too far.13:45
etingof+1 for a call13:45
dking_desktopTheJulia: Thank you very much! I was second guessing everything. Do you know what the best stable branch or commit is?13:45
*** Lucas_Gray has quit IRC13:46
etingofI am not quite done addressing your comments (with the latest spec revision some of them feel obsolete)13:46
etingofbut I am still at them13:46
*** priteau has joined #openstack-ironic13:46
dtantsuretingof: well, before you get too far with changing the spec, let's try to see where we want it to be13:47
dtantsurI'm not entirely fond of the approach of injecting the file as is13:47
dtantsurwhile it Gets The Job Done, it's not API, and I have a feeling we've been asked for an API13:48
etingofdtantsur, yeah, I like second config-drive idea better13:48
dtantsurmy favourite approach is still to create a slim API. but I think the configdrive approach is at least easier (and does allow using cloud-init if desired).13:49
*** Lucas_Gray has joined #openstack-ironic13:49
dtantsurwhat I doubt, however, is the ramdisk side implementation13:49
dtantsurif we declare we support whatever cloud-init supports, we're essentially locked into using cloud-init..13:49
etingofexactly13:49
etingofthat's why I try to keep ironic's hands out of network_data.json as much as possible...13:50
dtantsur(to be clear: this problem affects both the current proposal and the configdrive proposal)13:50
dtantsuretingof: the other way around: if you keep ironic's hands, you're locked into this problem13:50
dtantsur* hands out13:50
dtantsurbecause you have no way to say "oh, wait, we cannot do this here"13:50
dtantsurand you have to support port bonding and whatever the current format does (or will) support13:51
dtantsur(and you cannot validate for stupid mistakes like non-matching MAC addresses. people do this all the time.)13:51
openstackgerritMerged openstack/ironic master: Add logic to determine Ironic node is HW or not into configure_ironic_dirs  https://review.opendev.org/67195713:51
etingofdtantsur, how a validating API would work for stand-alone ironic? perhaps all network info should come from the operator in this case13:53
etingofif there is no neutron, then ironic's API would need to obtain MAC+IP+network+dns from the operator, then format network_data.json out of it?13:54
dtantsuretingof: I mean sanity checks. More specifically that network_data: 1. is consistent with information in ironic (ports, etc), 2. is implemented by us (which has a not well defined meaning here)13:54
*** ociuhandu has joined #openstack-ironic13:54
dtantsurand then, yes, we have neutron, so we need to somehow (?) converge the operator-provided and neutron-provided information13:55
*** Lucas_Gray has quit IRC13:55
etingofwith stand-alone ironic, perhaps there's a little that ironic could possibly validate13:55
dtantsuretingof: I disagree completely, I think the two points I mentioned are specifically important for standalone ironic13:56
dtantsurbecause the operators can (and thus eventually will) provide incorrect information, while we assume that neutron is sane13:56
etingofif neutron is in place, what kind of network info do we need from the operator?13:56
dtantsuretingof: probably none? which doesn't prevent them from providing one though..13:56
etingofmy understanding is that for neutron case the operator only needs to bind MAC to ironic port13:57
etingoffor non-neutron case, everything should come from the operator13:57
dtantsurlet's leave the neutron case alone for a while. it's pretty simple and on its own it does not require any API modifications.13:57
dtantsurwe're adding the new API specifically to target the standalone case13:58
dtantsurand this is where we're contentious on 1. how to present this API, 2. which feature set to support13:58
*** Lucas_Gray has joined #openstack-ironic13:59
etingofright, woth the design I am proposing, ironic is not involved at all - the operator prepares network_data.json and ironic blindly passes it over to ramdisk/cloud-init13:59
dtantsur(honestly, my favourite UX would be $ openstack baremetal port set <port> --deploy-ip-address 1.2.3.4)13:59
dtantsuretingof: ramdisk IS a part of ironic. ironic IS involved.13:59
dtantsurand the API is part of ironic, so again, ironic IS involved.14:00
etingofI am not sure about that ^14:00
dtantsurabout what?14:00
*** dougsz has quit IRC14:01
dtantsura decision to pass data in unverified is an API design decision, and we'll be responsible for it14:01
etingofI am not sure that ramdisk is part of ironic because IPA is indeed part of ironic, however ramdisk OS bootstrapping is somewhat independent of ironic14:01
*** ociuhandu has quit IRC14:01
dtantsuretingof: IPA and the way to build ironic are part of ironic, both logically (who else?) and officially (through IPA-builder)14:01
dtantsurthere is absolutely no question in this matter. when we call this feature delivered, we'll deliver both the ironic and the ramdisk sides.14:02
dtantsurand trust me, our customers see them as one feature14:02
dtantsuror even: our customers don't care that there is a ramdisk side as long as it works. the question is now: how exactly will we make it work?14:02
* etingof is afk briefly14:06
*** xXraphXx has quit IRC14:11
*** ociuhandu has joined #openstack-ironic14:11
*** dougsz has joined #openstack-ironic14:20
*** pcaruana has quit IRC14:20
etingofdtantsur, then the way to go would be to have stand-alone ironic accepting the entire IP stack config (IP+network+gw+dns...)?14:21
*** etingof is now known as etingof|afk14:21
dtantsuretingof|afk: a good question that I want us to solve before we rush into a decision :)14:21
dtantsurone thing is to support IPs+routes (the MVP we've been asked IIUC)14:22
dtantsuranother thing - to support everything that cloud-init supports14:22
dtantsurI guess you're voting for the latter?14:22
*** ociuhandu has quit IRC14:22
* TheJulia yawsn14:23
dtantsurmorning TheJulia14:25
*** pcaruana has joined #openstack-ironic14:32
*** ociuhandu has joined #openstack-ironic14:33
openstackgerritDmitry Tantsur proposed openstack/ironic-tempest-plugin master: DNM Explicitly tear down software RAID after testing it  https://review.opendev.org/68392114:33
*** Lucas_Gray has quit IRC14:36
*** ociuhandu has quit IRC14:42
*** tzumainn has joined #openstack-ironic14:44
*** etingof|afk is now known as etingof14:45
* etingof is not voting to support the open-ended feature set of cloud-init14:45
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic-python-agent-builder master: Add efivar  https://review.opendev.org/70137414:49
*** jawad_axd has quit IRC14:49
dtantsuretingof: then we need to 1. define the feature set, 2. define what happens if an operator requests something outside of it14:49
etingofeven seemingly simple IP+routes config can get complicated once it invokes sophisticated iproute2 features...14:49
*** jawad_axd has joined #openstack-ironic14:50
dtantsurexactly. which to me seems an argument for limiting the feature set to the bare minimum.14:50
TheJuliawould it be helpful to have a higher bandwidth call on this topic?14:51
openstackgerritJulia Kreger proposed openstack/bifrost master: Skip status code check and make vagrant install quicker  https://review.opendev.org/70106714:52
dtantsurTheJulia: that's the proposal I started this discussion with :)14:52
etingofstepping back in our discussion, how badly do we want ironic to process/verify IP stack config? I understand that catching config bugs earlier improves overall usability. but there are incurred costs...14:52
dtantsuretingof: which costs do you mean? It's certainly more coding, but this pays off in user experience.14:53
TheJuliadtantsur: oh good :)14:53
dtantsurespecially since any failure during IP configuration means deploy timeout14:53
*** jawad_ax_ has joined #openstack-ironic14:53
TheJuliaI would prefer we have to avoid explicitly blessing and testing every single configuration possibility14:53
TheJuliawhich is why I'm all for a pass-through14:54
dtantsurTheJulia: there is no way to avoid it: we have to write the ramdisk side.14:54
etingofI mean, ironic would have to align with some third party tool such as cloud-init14:54
dtantsurunless you suggest to rely on cloud-init14:54
etingofthe alternative (I have been proposing) is the pass-through14:54
dtantsuretingof: it's up to us whether to rely on it or not14:54
TheJuliadtantsur: I was honestly originally thinking glean since it is lightweight and we limit it to that14:54
*** Lucas_Gray has joined #openstack-ironic14:54
dtantsurTheJulia: we need to check if glean has the features we need14:55
*** jawad_axd has quit IRC14:55
TheJuliathis is a good point14:55
TheJuliaput IP on an interface it definitely has14:55
dtantsurand.. I still dislike the mode of operation where any error in a free-form JSON results in a deploy timeout with no explanation :(14:55
TheJuliafancy networking with bridging and vlans, it likely does not14:55
TheJuliaif someone wants those features, they are welcome to propose such code, we shouldn't be trying to have an exhaustive list supported day 114:56
dtantsursure, I don't disagree with that14:56
etingofmy experience with ironic ramdisk is that I frequently have to fire up the console to troubleshoot14:56
etingofI am trying to say that while early validation is indeed a good thing to have, it does not radically change anything14:57
dtantsurTheJulia: looking at https://opendev.org/opendev/glean I cannot find its feature set. but the first glance looks promising.14:57
dtantsuretingof: you're not the target customer ;)14:57
TheJuliadtantsur: it is what infra uses for config drive processing14:58
*** jawad_ax_ has quit IRC14:58
TheJuliasupers simple14:58
TheJuliaand we can blame mordred for all bugs14:58
TheJulias/supers/super/14:59
dtantsurOH, I'm sold now!!14:59
dtantsur:)14:59
TheJuliaif that seriously sold you, I'm alarmed :)14:59
* etingof includes mordred dependency in the spec 14:59
dtantsurTheJulia: I'm an openstacksdk core, I'm used to blaming mordred for things :D14:59
dtantsurseriously though, glean seems to support a lot of features. not clear if it supports RHEL 8 though14:59
TheJuliaetingof: this is a true potential issue, but I also hear of grumbling about getting fixes or anything into cloud-init and it is so heavy weight15:00
dtantsurI'm still strongly on the side of providing a schema on the ironic API side. it is our API, we're responsible for it (and for it to work).15:00
* etingof is not proposing cloud-init in ramdisk15:00
TheJuliadtantsur: they would surely accept patches and we have plenty of people they can trust to review the code... if they are willing to trust15:00
dtantsurso, glean seems promising, we need to: 1. figure out the feature set, 2. figure out RHEL 8 status, 3. patch it to understand our CD format.15:01
etingof+ keep linking up ironic API with potentially expanding glean feature set15:02
etingofs/linking/leveling/15:02
TheJulia++15:02
dtantsurit's annoying, but it's less annoying that debugging a deploy timeout when the customer does not know what a virtual console is :)15:03
etingofif it's an operator-managed network (i.e. not neutron-maneged), it's probably more fragile in its own15:03
TheJuliaodds are, in the edge cases, neutron will know exactly nothing about it15:04
etingofeven if glean config looks clean, the ramdisk may still not show up on the network because of operator mistake in part of network config15:04
TheJuliathis is entirely possible15:04
dtantsursure, we cannot protect from anything. it doesn't mean we have to protect from nothing.15:04
TheJuliaanything and everything15:05
TheJuliaor just everything15:05
dtantsur* everything15:05
etingofright, but the cost of building this protection - is in justified in this case...?15:05
dtantsuretingof: which cost again?15:05
dtantsurit's like 1 JSON schema and a few dozens of lines of sanity check code15:05
etingofironic network config API maintenance15:05
TheJuliawhat would the code really cost to perform some basic strcutural validations15:05
etingofcode is cheap, maintenance could be expensive15:06
TheJuliawrite anything in flask and be ahead of the techdebt? *ducks*15:06
dtantsuretingof: which maintenance?15:06
dtantsurwe're literally talking about 100-300 lines of Python code doing string manipulations15:07
etingofleveling up ironic network config API with glean changes/bugs/features/deprecations15:07
dtantsurwell, that's what API means15:07
dtantsurwe can say "we support glean >= 3.5.6,<5.0.0"15:07
etingofyeah, but it's not self-contained in this case15:07
dtantsurif API is not defined, it's not API :) and we're working on bare metal API here.15:08
dtantsurjust imagine yourself on the customer side of this feature15:08
dtantsur"we build an API that uses glean, go figure what it support. no, we don't know how you've build the ramdisk."15:08
*** ociuhandu has joined #openstack-ironic15:08
etingofwould it make sense to have some sort of config builder for glean that ensures valid outcome?15:09
etingofas an alternative to user-built config validation15:10
TheJuliawhat is minimally viable?15:10
*** rachit7 has quit IRC15:10
dtantsuretingof: realistically, we don't expect the part we need to ever change15:10
dtantsurso the maintenance discussion here is a bit theoretic15:10
dtantsurchances are high, we'll write it once and forever15:11
TheJuliaI think where maintenance idea is coming from continuing to carry forth and add more and I don't think we need to do that. We only need to do the initial lift15:11
dtantsur(unless we expect the IP protocol family to get deprecated)15:11
TheJuliaagain, bringing us to what is minimally viable for a user to be able to leverage the feature and not have to jump into IRC to ask for help or file support tickets15:11
dtantsur++15:12
TheJuliaI heard global v6 traffic has gone down to liek 12%... so I find V4 being deprecated a bit of a funny idea15:12
dtantsurwe'll retire by the time the potential deprecation reaches enterprises15:13
TheJulia+++++++++++++15:13
jrollwe know as well as anyone that deprecation never ensures death :)15:13
TheJuliaor have become goat farmers or something15:13
dtantsuryeah, what's the new python 2 death date? :)15:13
dtantsurApril 1st?15:13
TheJulialol15:13
dtantsurmorning jroll15:13
TheJuliagood morning jroll15:13
jrolllol15:13
jrolloh, morning everyone :)15:13
dtantsurTheJulia: I'd make cheese in Swiss Alps. that's my dream.15:13
jroll++15:14
etingofdtantsur, what's so special about that?15:14
TheJuliadtantsur: We're actually pondering getting a laser and plasma cutter and starting a fabrication shop on the side....15:14
TheJuliawell, I want the plasma cutter, summer just wants the laser cutter15:14
dtantsuretingof: about what? I'm merely saying that the data format we're planning to use is quite stable.15:14
dtantsurTheJulia: sounds like a lot of fun! :)15:15
TheJulia(which is the same premise glean took)15:15
etingofdtantsur, I've been wondering about your prospective farming career ;)15:15
dtantsuraaaah15:16
dtantsur:)15:16
rpittau+1 to goat farming, sheep would be good too15:16
dtantsuretingof: have you been to Swiss Alps? It's fantastic! And fresh cheese, mmmmm... :)15:16
TheJuliaTree farming too...15:16
* dtantsur has just lost his (already negligible) desire to work today15:16
rpittauI expect glean working on rhel/centos 8 since it works on systemd envs15:17
etingofdtantsur, yes, I got stuck in piles of manure there15:17
TheJuliasystemd !=  systemd-network15:17
dtantsurheh15:17
TheJuliaor NetworkManager15:17
dtantsurit uses NM, I think15:17
rpittauoh it works on NetworkManager15:17
TheJuliaoh, goodie15:17
dtantsuradapting glean to RHEL 8 may be a matter of a few switches to detect it15:18
dtantsuror maybe someone can just try it on a VM?15:18
TheJulia(or it maybe already does correctly)15:18
TheJulia(all vms in use at the moment)15:18
dtantsurI can fire up a VM after I get my afternoon tea15:18
* dtantsur goes afk exactly for that15:18
rpittaummm tea15:19
TheJulia++15:19
etingofit used to look much simpler with just iproute2 dependency of ramdisk15:20
TheJuliain my effort to fix bifrost to fix kolla... I ran into an issue where bionic basically keeps overriding route settings on a long lease where the route is changed. That wouldn't be a simple thing... so we have to kind of go above and beyond just interface settings becasue people have coded tools that disregard and discourage their use15:22
*** ociuhandu has quit IRC15:22
dtantsuretingof: the problem with API validation was there just as well15:33
dtantsurthe in-band deploy steps spec has got 2x +2, it's the right time to check it: https://review.opendev.org/#/c/696619/15:35
patchbotpatch 696619 - ironic-specs - Add in-band deploy steps spec - 4 patch sets15:35
etingofyes, but virtually no dependencies...15:36
dtantsuris it good or bad? ;)15:36
dtantsura nice (or terrible) thing about glean: it seems to replace dhcp-all-interfaces15:37
*** TxGirlGeek has joined #openstack-ironic15:41
dtantsurTheJulia, etingof, fungi thinks that glean works with CentOS/RHEL 8. so we're good here.15:42
fungiall our nodepool nodes in opendev use glean instead of cloud-init, mainly to avoid the giant mass of python dependencies cloud-init drags in15:44
dtantsurthat's what makes it appealing to us as well15:45
fungiso any node type you see jobs use in opendev at least have our cloud provider use cases covered in glean15:45
fungi(and that includes centos-8)15:46
* etingof noted that glean supports a subset of cloud-init features15:52
*** Lucas_Gray has quit IRC15:54
etingof...but what's supported is not seemingly documented15:55
dtantsuryep15:56
* dtantsur ponders supporting the SSH key functionality15:56
etingofstill, why would not we approach the problem from the different end - generate valid network_data.json from operator-supplied input?15:58
etingof...instead of building glean validator into ironic15:59
dtantsuretingof: I don't mind it either, as long as we have a clearly defined API15:59
dtantsurs/glean validator/openstack metadata validator/15:59
etingofdo we still need any API in ironic if network_data.json is guaranteed to be sane?15:59
etingofI'd say it's still a glean validator, or even a glean syntax subset validator16:00
dtantsuretingof: we're working on a feature: "Create API to provide IP addresses to the ramdisk in DHCP-less environment"16:01
dtantsurdoes it answer your question?16:01
dtantsuror do you suggest concentrating only on the neutron case?16:02
etingofis the API part also a requirement? could we describe the feature as "Make ironic ramdisk running in DHCP-less environment"?16:03
*** jawad_axd has joined #openstack-ironic16:04
TheJulianeutron would be a good intermediate stepping stone, but won't really be used in edge scenarios unless it is a far flung completely openstack managed which I can only see in the likes of Oath16:04
etingofno, I am not looking at neutron within the context of this conversation16:04
dtantsuretingof: how do you provide IP addresses if you don't have an API for that?16:05
dtantsurI see two options in the discussion: from neutron and from the API. do you see another option?16:05
*** ociuhandu has joined #openstack-ironic16:05
etingofso the thought I am trying to convey is this: (1) operator obtains network config for a node somehow (2) operator invokes a glean config builder that consumes operator-supplied network config and produces valid network_data.json file (3) ironic passes-through network_data.json to glean running inside ramdisk (w/o any validation or interpretation)16:07
dtantsuretingof: (3) ironic passes through <-- this is API. API requires validation.16:07
dtantsuror you can suggest passing the complete configdrive, which pushes the problem down the stream.16:07
dtantsur(for the reasons that are still unclear to me)16:08
TheJuliaI feel like bifrost is soooo close...16:08
TheJuliahttps://www.irccloud.com/pastebin/TRzoTfpL/16:08
etingofmy thought is that if network_data.json is guaranteed to be well-formed (because it's validated by the building tool), then we arguably do not need another validation in ironic16:08
openstackgerritMerged openstack/ironic master: Make qemu hook running with python3  https://review.opendev.org/70099416:09
openstackgerritMerged openstack/ironic master: Explicitly use ipxe as boot interface for iPXE testing  https://review.opendev.org/69814616:09
dtantsuretingof: guaranteed - by whom to whom? how do we know that?16:09
dtantsurcan we remove all validation from our API under the assumption that ironicclient/openstacksdk are written correctly?16:09
dtantsuralso, are you actually suggesting that we write a new tool to avoid writing 100-200 lines of string manipulations?16:10
etingofwell, if we code up a tool that would produce network_data.json under the same constraints as ironic network config API would impose on human-generated network_data.json...16:10
dtantsur... and users won't use it?16:11
dtantsurmore importantly, what are you saving here? you still have to do the same job, but now the validation is optional, because.. why?16:11
etingofI am not counting in lines of code here. I am more looking to avoid ironic depending on glean16:11
dtantsurwhy?16:11
dtantsurwhat's wrong with using a tool that gets the job done?16:12
dtantsurokay, you can get back to the iproute2 proposal, but you'll still need to validate the input16:12
TheJuliaIs the desire to expand the scope beyond purely deployment to instance operation?16:12
etingofI am not against using glean in ramdisk16:12
dtantsurTheJulia: we already have it...16:13
etingofI am worrying that pulling (undocumented) glean's details into ironic code16:13
TheJuliawell, tehcnically as long as someone attached a configuration drive....16:13
etingof...may be expensive in the long run16:13
dtantsuretingof: let's ask them document it! or help them document it16:13
TheJuliabut we shouldn't be engineering the same thing again16:13
*** ociuhandu has quit IRC16:13
TheJuliaetingof: bifrost was a super early user of it as well, so the bonds with the ironic community are strong :)16:14
*** ociuhandu has joined #openstack-ironic16:15
etingofso we are basically saying to ironic users that they need to produce network_data.json that is compliant to a subset of glean syntax?16:15
*** gyee has joined #openstack-ironic16:16
etingofthen, as dtantsur said, we need to (1) establish what glean syntax is and (2) strip it to a bare minimum16:16
dtantsuretingof: right. and we'll have to document it somewhere16:16
dtantsur(or ask glean people to document, or document it for them)16:17
dtantsuretingof: chances are non-zero, glean supports the whole network_data.json16:17
dtantsurfungi: do you know ^^ or somebody who knows?16:17
etingofdtantsur, they say it is not16:17
dtantsurit might be outdated. the code looks quite sophisticated.16:17
etingof> Please note that glean does not implement every feature listed.16:18
dtantsurfrom configdrive - sure16:18
dtantsurit doesn't seem to bother with vendor_data or all of metadata16:18
dtantsurin any case, whether we use glean, cloud-init or iproute2, we'll need to define what API we support16:18
TheJuliaI suspect dtantsur's point is "what is the mechanics and use path (in terms of a human programming/interaction interface) to get from A to booted ramdisk16:19
etingofI take it purely as a stripped down network_data.json schema (which is not documented as well, afaik)16:20
openstackgerritDmitry Tantsur proposed openstack/ironic master: Add a missing versionadded for configdrive[vendor_data]  https://review.opendev.org/70140016:20
dtantsuretingof: it may even be complete, let's see if we can get someone to confirm16:21
funginot entirely certain what the glean question is, but glean is basically focused on setting up network interface configuration at boot time using information provided in a configdrive and/or dhcp16:22
fungi(it doesn't implement dhcp itself, but can configure the interface to use whatever the platform's provided dhcp solution is)16:22
fungioh, also ipv6 autoconfig16:23
TheJuliaThat is good to know about v6 autoconfig16:23
etingofdtantsur, so regarding the ironic network config API - is it going to leak all the details (IP, gw, dns...) to ironic REST API? or ironic osc should build network_data.json out of user-supplied options and pass it over to ironic as a blob?16:23
fungias for dhcp6, i'm not sure how much is supported (stateless vs stateful), in most cases it's been implemented piecemeal as we or other glean users encountered provider environments providing particular features16:24
dtantsurfungi: does it support static IPv6? it's not clear from the code.16:24
TheJuliaetingof: no matter what we do, the data would need to be transmitted to the API. OSC is not our only API consumer.16:24
TheJuliadtantsur: that might be a good feature to add if not...16:25
dtantsuryep, especially for people here who work on metal3 ;)16:25
TheJuliaI think they can live with just slaac, but you never know.16:25
dtantsurfungi: this looks suspicious: https://opendev.org/opendev/glean/src/branch/master/glean/cmd.py#L270-L27116:25
fungithere are also some nasty inetractions between networkmanager and kernel ipv6 slaac autoconf where nm can refuse to set up v4 addresses if the kernel has already configured a v6 addy, i think we finally got that worked out but some of it may still be timing sensitive16:25
etingofso perhaps leaking IP config details to ironic API is a way to go then16:25
fungidtantsur: not sure about static v6 either, would need to look in the docs/source. if it doesn't, i'm sure we'd be happy for that to get added16:26
dtantsurokay, so this ^^ is another potential TODO16:27
dtantsurfungi: for the context: for Edge deployments we're working on a way to configure IP addresses for our ramdisk without DHCP16:27
*** mbeierl has quit IRC16:27
dtantsurwe suspect glean may help us on the ramdisk side. we'll need mostly static configuration + routes.16:27
etingof+dns16:28
fungilike i said, it's mostly grown support for situations as we've encounetred them in the wild, and i don't think we have any providers for opendev who are using static v6 in configdrive (nova might not even put that in the configdrive data, i'm not certain)16:28
*** ociuhandu has quit IRC16:28
*** mbeierl has joined #openstack-ironic16:28
dtantsuretingof: yep. of all these, only static IPv6 is questionable.16:28
dtantsurfungi: makes sense, thanks!16:29
clarkbit depends on the platform16:29
clarkbwe've added things that we've needed as we've added platforms16:30
TheJuliaa support matrix might be needed then, but that shouldn't be a big deal16:30
* TheJulia is on a matrix kick for some reason16:30
JayFProbably good to document that for glean in any case.16:30
dtantsurexactly16:30
clarkbI think part of the struggle there is it is nebulous16:31
dtantsuryeah, from code it seems that static IPv6 is supported for networkd and debian, not supported for RH and gentoo16:31
clarkbyou add config to systemd-networkd16:31
clarkbthen see what comes out the other end16:31
clarkb(similar with networkmanager)16:31
TheJuliaThis is true, but if we at least point out high level cases, I think we're better off than nothing16:31
* dtantsur finds glean code quite readable btw16:31
clarkbI guess this has mostly been an issue with networkmanager16:32
clarkbwe configure ipv4 and ipv6 and then only get one or the other out the other end due to ancient bugs in network manager16:32
TheJuliaclarkb: ouch16:32
clarkb(which we hope we've worked around at this point)16:32
fungiright, working around the quirks of various network interface configuration frameworks across distros has been the biggest challenge16:33
etingofre the prospective ironic api - I understand we are adding a select of IP config details as node/port properties, on per-interface basis for ramdisk and instance. is that about right?16:33
fungis/quirks/long unfixed bugs/16:33
dtantsuretingof: I'm quite against doing anything for the instance, we already have an API for that.16:33
etingofthat would be a bit inconsistent for the stand-alone ironic operator...16:34
dtantsuretingof: I don't have a strong opinion on node vs ports as long as API is meaningful16:34
dtantsuretingof: not really. we still accept a configdrive.16:34
TheJuliaper interface is what it should be because we need a MAC for lookup (unless we merge agent token and then add in an embedding "hey, this is your uuid, have a nice day"16:34
dtantsurit's just that nova generates it behind the scenes, while in the standalone case the operator is responsible16:34
etingofright, but to deploy a node entirely dhcpless, the operator will have to (1) configure ramdisk via api and (2) produce network_data.json somehow, wrap it into a config-drive and pass to ironic16:35
dtantsuretingof: or just pass it to ironic to two places if it's completely the same16:35
TheJuliaSo dmitry actually raises a very very very very good point16:36
TheJuliaon an edge site, your not going to have a different network... your likely going to provide the exact same configuration drive data for the deployment16:36
TheJuliayour internet or internal network or outbound path should be the same16:36
JayFShould that be a configuration we make easy by default, given it's less-than-ideal security implications?16:36
TheJulia(which is why the agent token code is so important16:36
TheJulia)16:36
*** ociuhandu has joined #openstack-ironic16:37
dtantsurJayF: you mean, the complete feature? or using the same networking configuration?16:38
TheJuliaJayF: in this case is for virtual media usage, we would just embed the data in the iso image being attached16:38
etingofdtantsur, these two places - what are they?16:38
JayFdtantsur: using the same network configuration16:38
dtantsuretingof: for deployment - the one we're designing. for instances - https://docs.openstack.org/api-ref/baremetal/?expanded=change-node-provision-state-detail#change-node-provision-state (see network_data).16:38
TheJuliaJayF: we can't orchustrate changing network configuration around in the edge case. It is literally an active/standby node pair on a telephone pole with embedded radios doing magical stuff16:39
etingofdtantsur, "pass it to ironic" - "it" stands for config-drive here?16:39
etingofor network-data?16:40
dtantsuretingof: for the final instance you can do either of these16:40
JayFTheJulia: I must be missing something about this discussion then, because I didn't realize that was a requirement? I'll bow outta the conversation then...16:40
dtantsurstarting with Train you can pass only network_data itself16:40
JayFThat sounds like an awful requirement to have to meet :-|16:40
etingofright, but for deployment we are feeding all config details to ironic via dedicated API node/port properties, right?16:41
TheJuliaJayF: not terribly awful, just the reaching consensus part has been... not fun16:41
dtantsuretingof: via the provision state API (please see the link)16:41
JayFI mean, if I were still closely involved and more well-read, I might ask the "should we" question... but I'm not, so I'm heading off IRC to go to code-work16:41
etingofdtantsur, right16:42
dtantsur(thinking aloud) in theory nothing prevents using Far Edge deployments with the 'neutron' networking for network separation..16:42
TheJuliaJayF: nobody wants to roll a truck to re-image nodes, so from a capability standpoint16:42
*** xXraphXx has joined #openstack-ironic16:43
TheJuliadtantsur: for an envrionment with a managed switch fabric, sure and I think in the neutron case that can be supported, the operator "i need to reimage this server in the field that doesn't really have a real switch... is where the original spec came from16:43
* etingof still worries that expanding provision state API with glean options is a bit more intrusive compared to an external network_data generation tool...16:45
dtantsurTheJulia: I would expect even a cheap switch from eBay to be programmable nowadays, but I may be terribly wrong..16:45
JayFdtantsur: that is pretty wrong :)16:45
etingof...and internally inconsistent ramdisk vs instance config16:45
dtantsuretingof: wait, wait, we don't do that.16:45
dtantsurJayF: #sadpanda16:45
JayFdtantsur: I have a box of dumb switches in my closet, all purchased on amazon for <$5016:45
*** ociuhandu has quit IRC16:46
JayFand you can absolutely buy switches marketed to business that don't have any management capability16:46
dtantsuretingof: the provision state API accepts network_data now, already. it's done, we're not discussing changing that.16:46
dtantsurJayF: that's why we cannot have nice things..16:46
* TheJulia looks at the 4 port router/switch on her desk in the 3d printed case where the switch looks and reports that it can be configured, but doesn't actually support any running configuation16:46
JayFdtantsur: if your world was true, I could say "that's why we cannot have cheap things" :D16:46
etingofdtantsur, then I do not understand how will `openstack baremetal node --set-ip 1.2.3.4 ` would work16:46
dtantsuretingof: I'm saying the opposite thing: let's not worry about final instances, only about the deployment process16:46
TheJuliadtantsur: agreed, this is purely a deployment problem to solve16:47
dtantsuretingof: more of $ openstack baremetal node set <node> --ramdisk-network-data /my/file.json16:47
etingofwell, that's not your ideal, is not it? ;)16:47
dtantsurclose to what you're proposing, just without instance_network_data and with some validation on the API side16:47
dtantsurmy ideal.. may never come to life.16:47
dtantsurmy ideal would be more of $ openstack baremetal port set <port> --network-data ipv4=1.2.3.4 --network-data ipv6=blah:blah::blah16:48
JayFThat's going to get really gross for super-complex setups.16:48
dtantsurbut I can see a point in accepting the whole network_data file: you can easily reuse it in the (existing) deployment API16:49
JayFUnless you're going to limit Ironic's capabilities to less than what glen/network_data supports16:49
JayFi.e. how would you model a bonded network interface ala what OnMetal used to use16:49
dtantsurJayF: we're not sure we want to support super-complex setups, but fair enough16:49
dtantsurdid you use bonding for deploy ramdisks?16:49
JayFWe did not, but totally would've if I could've16:49
*** alexmcleod has quit IRC16:49
dtantsurJayF: interesting, why?16:50
etingofdtantsur, let's consider the edge case where we have a single network with exactly the same IP config for deploy and instance16:50
etingofdtantsur, would not it be operator friendlier to pass network config to ramdisk and instance in exactly the same way?16:50
etingofe.g. network_data.json or config-drive16:50
JayFdtantsur: failure semantics around switch failures, mainly. We could've had a better attempt at making control plane stay up in switch failure cases, or at a minimum, short outages on the switch wouldn't have put nodes in weird-ish states (that might have been related to our hacky long running ramdisk stuff though)16:51
dtantsuretingof: it probably will, hence my doubts. If we accept network_data, then it can be the same file, but put into different places for deployment and for final instance.16:52
TheJuliaThat moemnt when you can't find the screen beeping16:52
etingofdtantsur, that's what in the latest spec...16:52
dtantsuretingof: yes, but we need to remove instance_network_data and add API validation16:52
dtantsur(and probably settle on using glean as a backend)16:53
dtantsurthis ^^ is the minimal proposal to finish the spec. there can be options.16:53
dtantsurJayF: ah, long running ramdisks (which we now have upstream btw)16:53
JayFdtantsur: Yeah, but I suspect what you are doing now has to be better than what we were doing :P16:54
etingofdtantsur, that all makes sense to me except my doubt wrt network_data validation vs generation debate16:54
dtantsurJayF: I'd not be that confident16:54
TheJulia\o/ success it was a hangouts chat16:54
dtantsuretingof: I don't find "try and get deploy timeouts if you fail" a good failure model16:54
dtantsurespecially when it comes to silly things like wrong JSON format16:55
etingofI see both approaches as formally equivalent for as long as the operator uses config generation tool16:55
dtantsuretingof: they don't. just live with it, they won't use such a tool.16:56
dtantsurtripleo people tried to create a networking template generator - people still write them by hand (and often enough wrongly)16:56
etingofthen they will fail at instance config16:56
etingofanyway16:56
etingofgiven that they might generate instance network_data by hand - we are not guarding that16:57
dtantsurwe don't verify instance configdrive because we're not on the other end. we don't know how it should look.16:57
dtantsurfor the ramdisk we are the other end16:58
* dtantsur wouldn't mind at least trying to verify the instance configdrive16:58
etingofI know, I am just noting that the end result is the same - either ramdisk or instance won't come up if the operator is messing with network_data.json16:58
dtantsurSure, the area of responsibility is different. We're responsible for the ramdisk, the operators are responsible for the final instance.16:59
*** lucasagomes has quit IRC17:00
*** hwoarang has quit IRC17:01
TheJuliarpittau: re 700912, the env-setup script gets it I think..17:02
rpittauTheJulia: yeah, I was going to reply to my own comment :/17:03
rpittauTheJulia: one thing, running the test it seems python-ironicclient doesn't get installed and the test fails at openstack baremetal node list17:04
etingofdtantsur, right, but still, to my opinion adding glean validator into ironic is barely justified in this case. given all pros and cons17:04
etingofbut anyway, let me update the spec accordingly17:04
dtantsuretingof: it's not "glean validator", it's our API validator17:04
*** hwoarang has joined #openstack-ironic17:04
dtantsurwhatever we accept must be validated as much as possible17:04
dtantsurglean or anything else is an implementation detail17:04
etingofthat perhaps makes it even more hairy then17:05
dtantsuryou're even free to invent your own format (and convert it to glean behind the scenes) as long as it's documented and validated17:05
etingofif we gonna support more than one syntax17:05
*** pcaruana has quit IRC17:05
dtantsurwe aren't. I'm merely pointing out that it doesn't matter for the API discussion.17:05
dtantsurunless we're building something that we truly never ever interpret, we have to validate it17:06
etingofyou see, I am reluctant to teach ironic parsing a specific dialect of network_data.json17:06
dtantsur(an examples of something that we do not interpret are whole disk images or instance configdrive)17:06
dtantsuretingof: s/dialect/subset/. this has been your proposal since the beginning, no? the question was only "which subset"17:07
etingofdtantsur, not really. the only thing that I had to account for is the need to merge multiple network_data.json files17:08
TheJuliain my 1-on-1 atm17:08
etingofbut that need is gone if we attach network_data to ironic node instead of ironic port17:08
*** hwoarang has quit IRC17:09
dtantsuretingof: if you'd like to support the complete network_data.json, we only need to add the missing parts to glean.17:09
etingof;-)17:09
dtantsurstill easier than implement everything from the ground up17:09
rpittaugood night! o/17:09
*** rpittau is now known as rpittau|afk17:09
etingofI am leaning towards making ironic (conductor) neutral to network config format17:10
*** hwoarang has joined #openstack-ironic17:10
etingofso it would be like end-to-end: the operator builds a well-formed network-data.json for ironic ramdisk via a tool, network_data.json passes through ironic conductor to ironic ramdisk for consumption17:11
dtantsurs/well-formed// we cannot assume that17:11
etingofthe same network_data.json could work for instance as well17:12
* dtantsur reminds that we're still arguing about 100-200 lines of basic validation code......17:12
JayFdtantsur: I'd be curious if that validation code would live in glean and be libraried-in, or if we'd write it ourselves.17:12
etingofwe can only assume network_data.json sanity based on operator's willingness to play by the rules i.e. to use config generating tool17:12
dtantsurJayF: interesting point. given that format is not glean-specific, it could be just a JSON schema..17:12
JayFdtantsur: seems like it could be painful to require an ironic API validation change to support new features in network_data.json if glean was already updated to support them17:13
JayFyeah, you're seeing what I'm getting at a little bit17:13
dtantsurJayF: it's not the case we're facing, really. I don't even suggest we validate every aspect of the file.17:13
JayFdtantsur: you just wanna make sure it's valid json and has a few keys you expect?17:13
dtantsurmore of "somebody typed ipaddress instead of ip_address"17:13
dtantsuryeah17:13
JayFdtantsur: I still think that's a little dangerous, because the spec for that json could change a name. Right now, AFAICT, it'd be possible to implement all this in Ironic without having Ironic (or IPA ramdisk) care about the specifics of what's inside that json, as long as glean knows how to deal with it17:14
dtantsurJayF: yep, but it's barely a good API (and a failure will cost a lot)17:15
* etingof humbly reminds that ip_address field can still have a non-functional value that we can't catch17:15
JayFYou both are kinda right, it's a crappy failure case that we can avoid some aspects of17:15
*** ociuhandu has joined #openstack-ironic17:15
dtantsuretingof: never in this discussion have I suggested to cover 100% of failures, hence I'm referring to sanity checking17:15
JayFbut also you can't prevent the most likely cause of those (actaully badly-configured networking)17:15
JayFI'll also ask: does this spec consider at all using this network data for rescue ramdisks?17:16
dtantsuroh, we haven't talked about rescue much17:16
dtantsur(see, that's why it's useful to have you in this discussion :)17:16
etingofdtantsur, I know, I am just pointing out that there can be lesser sense in catching typos if semantic errors are likely and fatal17:16
JayFIt should just be able to use what deployment does, just s/deploy/rescue/ on the relevant added options :P17:16
dtantsuretingof: inability to prevent all errors doesn't mean we shouldn't prevent any17:17
dtantsurespecially since the costs of validation are negligible17:17
JayFdtantsur: etingof: I wonder if we have a good way to give non-fatal API warnings ... i.e. API user uploads a network_data.json that fails validation ... can we attempt to still set it on the node, but let the user know "we found potential issues with this file: X, Y, Z"17:17
etingofso what's the difference between deploy/rescue and clean ramdisks?17:18
dtantsuretingof: deploy and clean ramdisks are largely the same (ditto inspection)17:18
dtantsurrescue is running on a user network, so it may potentially have different networking17:18
JayFat OnMetal, for example, our rescue ramdisk was extremely similar to our deploy ramdisk, but with some useful tools added for users trying to rescue, and with the private firmware tools ripped out17:18
dtantsurwhich is a huge can of worms if we talk about non-admin users, but let's ignore it17:18
JayFoh man, rescue would be painful to support with this17:19
dtantsurindeed17:19
JayFbecause you'd have to cleanup the "ramdisk" networking configuraiton before applying the "user" networking configuration17:19
JayFwhich may not be that awful, glean might just do it by default, but it's case that'd have to be explicitly supported and tested17:19
*** hwoarang has quit IRC17:20
dtantsuroh17:20
*** hwoarang has joined #openstack-ironic17:20
dtantsuretingof: what JayF is talking about is that rescue starts on the provisioning network, then switching to the internal one17:20
dtantsurwith the networking configuration changing mid-way17:20
dtantsuroh, s/internal/user/17:21
JayFs/provisioning/rescue/17:21
dtantsurwe cannot even express that in the currently proposed model17:21
etingofhow does it do the switch-over? restarts DHCP client?17:21
JayFmany deployers use the same network for provisioning and rescue, but it's not required17:21
dtantsuretingof: I think so, but I'm not fluent in that code17:21
JayFetingof: basically it does a callback to conductor, the conductor tells neutron to flip to user network, sets the state to "rescued"17:22
JayFetingof: after performing that callback, and a short sleep, the ramdisk then applys the network configuration already populated on the configdrive17:22
etingofJayF, thanks for the insight!17:23
JayFit's ugly and horrible17:23
etingofJayF, why does it start on provisioning network at all?17:23
JayFand honestly worked way better than I expected in the real world :P17:23
etingofi.e. rescue17:23
JayFwe'll, as noted above, it starts on the *rescue* network17:23
JayFdeployers can optionally make that the same network through config17:23
dtantsuretingof: it needs a network that can access ironic API/TFTP/etc17:23
JayFbut think about multitenant cases: we can't ever have the user node, running user code, booted on a network that can hit the control plane17:24
etingofright, but do we need to even start on rescue network with vmedia/dhcpless?17:24
dtantsuretingof: you need to talk to ironic API anyway17:25
JayFetingof: to get the rescue password from it :)17:25
dtantsurnow, multitenancy is probably not about the Edge case17:25
dtantsurso these use cases may not overlap17:25
etingofcan't we embed it into iso?17:25
dtantsurbut we need to decide something about rescue17:25
JayFno, and I'm not even saying you have to support that in the matrix for this feature or w/e17:25
*** ociuhandu has quit IRC17:25
dtantsuretingof: we can, but right now we don't.17:25
JayFjust noting that rescue should be considered and explicitly excluded rather than just forgotten :P17:25
dtantsurJayF++ great catch17:26
* etingof is trying to avoid switching the networks17:26
dtantsuretingof: we can explicitly say that rescue is only supported in the limited way17:27
dtantsurand then think what we can do about it outside of this spec17:27
JayFI will say though, it sounds like rescue working could be crucial to the use case as I understand it17:27
etingofyeah17:27
*** pcaruana has joined #openstack-ironic17:28
dtantsurJayF: yep, but the case TheJulia was talking about assumes same network for everything17:28
JayFdtantsur: ah17:28
dtantsurrescue will work like that17:28
dtantsurmulti-tenancy... we'll have to solve it somehow.17:28
JayFI'm still about 30 months behind :P17:28
dtantsurheh17:28
dtantsurJayF: the Edge world is weird and changes every few months17:28
JayFand really, I don't even work much upstream right now, but I love getting in these chats when they happen :)17:28
JayFdtantsur: s/Edge//17:28
JayFlol17:28
dtantsurhaha, true17:29
*** hamzy has quit IRC17:29
dtantsurJayF: we love seeing you here :)17:29
etingofthat's why ironic is taking over the world17:29
JayFdtantsur: you know I work with Ruby/Jim at Verizon Media now, right?17:29
JayFdtantsur: so I do work on Ironic, just haven't had the chance to work as much upstream (yet, I hope)17:29
dtantsurI do, yes (\o/)17:29
JayFYou still at Red Hat?17:30
dtantsuryeah, same stuff, a bit more k8s in my life17:30
dtantsuraka http://metal3.io/17:30
JayFNice.17:30
JayFHonestly, I'm kinda glad I stepped away for a couple of years from OpenStack. It's very satisfying to leave, come back, and see all the stuff we all built together that much more mature.17:31
dtantsurmay be a nice surprise indeed :)17:31
JayFSuch as, for instance, a chat completely centered around the existance of a network_data.json file that didn't even exist until we all made it exist way back when17:31
dtantsur:D17:31
*** ociuhandu has joined #openstack-ironic17:31
dtantsurand we're approaching supporting non-admin API users, who could imagine17:32
JayFI thought the policy work enabled that years ago?17:32
dtantsurJayF: I mean, ownership of nodes by non-admin users17:32
JayFOh, I guess you mean *multitenant aware* non-admin API users17:32
dtantsuryeah17:32
JayFyeah that makes sense, and is likely awesome for my current use cases17:33
dtantsur(and leasing being proposed to ironic-specs)17:33
* dtantsur tries to remember when we last talked f2f17:33
dtantsuretingof: I suggest we make another iteration of the spec, incorporating all the feedback, and walk from there. I've been in this discussion for 4 hours, I need a break :)17:34
JayFAustin, TX PTG?17:34
dtantsurAustin was a Summit, wasn't it? I guess you mean ATL?17:34
JayFdtantsur: if you're ever in the Seattle area, we can have a mini-reunion. At least Aeva is in this area too (although I haven't seen her in a long time either)17:34
JayFmaybe? I don't remember, it all runs together17:34
JayFand there were a few months when I tried to forget it all after getting laid off17:35
dtantsuryeah..17:35
dtantsurI'd love to come to that area, barring all the visa complexities17:35
dtantsur(and my general dislike of the airplanes)17:35
JayFI'm going to try to get to go to the Vancouver summit since it's driving distance for me17:36
tenbraeAlso planning to attend Vancouver summit, FWIW :)17:36
dtantsurI don't have an idea if I go or not. Again, budgets, visas, flights :)17:36
*** ociuhandu has quit IRC17:37
* dtantsur will go get some drin^Wrest now17:37
*** dtantsur is now known as dtantsur|afk17:37
dtantsur|afksee you tomorrow!17:37
JayFdtantsur|afk: o/17:38
* TheJulia exits 1-on-117:38
TheJuliabrrraaains17:38
*** ociuhandu has joined #openstack-ironic17:39
TheJuliaso rescue networking, yes agree it has to be excluded, but could eventually be supported as long as we implment the agent token work or something like that so we can begin to add some additional layers of security17:40
*** aedc has joined #openstack-ironic17:41
* TheJulia replaces dtantsur's rest with a drink17:43
*** priteau has quit IRC17:45
*** ociuhandu has quit IRC17:46
*** jawad_axd has quit IRC17:53
*** derekh has quit IRC18:00
TheJuliare vancouver summit, I'll very likey be there. We've been asked if we would bring the puppy too.... I'm not sure if that is possible.18:02
*** jawad_axd has joined #openstack-ironic18:04
JayFTheJulia: if you all drive up there, plan a stopover for smoked meats in Tacoma :D18:04
* TheJulia looks around for the spare braincells and thinks it is just time to go get some lunch and run errands18:04
JayFTheJulia: I even have a big enough parking spot for the dark bug18:04
JayF*bus18:04
TheJuliaohhh!18:04
TheJuliaAnd we have solar on it now18:04
TheJuliaso we don't need to run the generator as much18:04
JayFYeah, I'm 100% seriously, just LMK18:04
*** bdodd has quit IRC18:04
TheJuliaokay!18:04
etingofdtantsur|afk yes, I will push another spec update18:06
TheJuliadking_desktop: Sorry, I think I got your last message as I was falling asleep last night. You should be fine if use stable/train. Depending on how your triggering it, you may need to explicitly set the git_branch settings. A quick grep -r will find them since it is like package_<git_branch>18:07
TheJuliaokay, brain needs to disconnect for a little bit18:07
TheJuliaerrands!18:07
dking_desktopThank you very much!18:09
*** dougsz has quit IRC18:16
TheJuliaand no need to do errands, found the hidden back of cat foot18:20
*** afasano has quit IRC18:29
*** afasano has joined #openstack-ironic18:30
*** jawad_axd has quit IRC18:44
*** trident has quit IRC18:48
*** trident has joined #openstack-ironic18:49
*** hwoarang has quit IRC19:10
*** hwoarang has joined #openstack-ironic19:11
*** jtomasek has quit IRC19:13
*** bdodd has joined #openstack-ironic19:24
*** dougsz has joined #openstack-ironic19:24
*** hamzy has joined #openstack-ironic19:26
*** dougsz has quit IRC19:27
*** jawad_axd has joined #openstack-ironic20:03
*** p0tr3c has quit IRC20:04
*** jawad_axd has quit IRC20:09
*** bfournie has quit IRC20:12
*** Lucas_Gray has joined #openstack-ironic20:30
*** pcaruana has quit IRC20:41
*** Goneri has quit IRC21:00
TheJuliadtantsur|afk: https://storyboard.openstack.org/#!/story/2007076 filed :\21:03
*** afasano has quit IRC21:08
*** afasano has joined #openstack-ironic21:09
*** mmethot has joined #openstack-ironic21:14
*** rcernin has joined #openstack-ironic21:37
*** rcernin has quit IRC21:37
*** rcernin has joined #openstack-ironic21:38
*** early has quit IRC21:55
stevebakerTheJulia: hey, I'm thinking of tackling the ironic pecan->flask conversion, but I should talk to someone first21:58
TheJuliastevebaker: this is encouraged all around21:58
stevebakertalking? yeah21:58
TheJuliawell, converting the api as well21:59
stevebakerTheJulia: ok, so first questions. Is there upstream consensus to start this? Should it be a story or a blueprint?22:00
TheJuliastevebaker: 1) there has long been consensus. 2) I think there is a story already. 3) Ironic doesn't use blueprints22:01
stevebakerTheJulia: cool, I didn't find the story when I looks, but will try again22:02
TheJuliasearching myself22:03
TheJuliahttps://storyboard.openstack.org/#!/story/200580622:03
stevebakerTheJulia: \o/22:04
TheJuliahmm22:04
TheJuliahttps://storyboard.openstack.org/#!/story/165134622:04
TheJuliaThe latter one has rfe approved tagged22:05
stevebakerTheJulia: huh, or is that one more about replacing WSME for validation, which is related but orthogonal?22:06
TheJuliathe general idea is get rid of pecan and wsme22:08
stevebakerfair enough22:08
TheJuliaironic-python-agent already got swapped over to flask and aside from a single bug, worked like a charm22:08
stevebakerTheJulia: finally, I'll see how the implementation goes as far as the risk of the switchover breaking things. But my first attempt will be to have a few refactoring changes then a big-bang switchover change.22:10
TheJuliastevebaker: a big single bang patch is super unlikely to ever merge22:11
TheJuliafrom a review bandwidth standpoint, it is just going to crawl... where as like one endpoint at a time should fly right through and allow for less conflicts/headaches22:11
TheJuliathe idea was always to do what keystone did, change a single endpoint, get happy with that, and then sweep through the rest of the api in rapid succession22:12
stevebakerTheJulia: oh they did pecan->flask?22:12
*** early has joined #openstack-ironic22:12
TheJuliayup22:12
stevebakergood to know, I'll check it out22:12
TheJuliastevebaker: it was like two years ago now22:13
stevebakerok22:13
TheJuliaI'm sure morgan is someplace... if there are questions22:13
stevebakersweet22:14
TheJuliaokay maybe he isnot on irc22:15
stevebakerTheJulia: it looks like pre-flask they used paste.deploy plus there own internal wsgi fu https://docs.openstack.org/releasenotes/keystone/rocky.html#prelude22:18
TheJuliaactually, come to think of it. ipa was ?werkzrug? which is what flask is based on22:19
TheJuliasince we didn't need all of the extra stuff for that api22:19
stevebakerok22:19
*** bfournie has joined #openstack-ironic22:23
*** afasano has quit IRC22:47
*** afasano has joined #openstack-ironic22:47
*** afasano has quit IRC23:04
*** afasano has joined #openstack-ironic23:06
*** hamzy has quit IRC23:13
*** tesseract has quit IRC23:13
*** goldyfruit_ has joined #openstack-ironic23:17
*** rh-jelabarre has quit IRC23:26
*** ociuhandu has joined #openstack-ironic23:30
*** afasano has quit IRC23:31
*** afasano has joined #openstack-ironic23:32
*** ociuhandu has quit IRC23:35
*** aedc has quit IRC23:36
*** bdodd has quit IRC23:41

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!