| opendevreview | Verification of a change to openstack/ironic master failed: Enable OVN CI https://review.opendev.org/c/openstack/ironic/+/885087 | 00:41 |
|---|---|---|
| opendevreview | Verification of a change to openstack/ironic master failed: CI: Fix our internal MTU settings https://review.opendev.org/c/openstack/ironic/+/893112 | 00:41 |
| opendevreview | Verification of a change to openstack/ironic master failed: Enable OVN CI https://review.opendev.org/c/openstack/ironic/+/885087 | 01:29 |
| opendevreview | Verification of a change to openstack/ironic master failed: Enable OVN CI https://review.opendev.org/c/openstack/ironic/+/885087 | 04:12 |
| rpittau | good morning ironic! o/ | 06:54 |
| opendevreview | Adam Rozman proposed openstack/ironic-python-agent master: implement basic-auth support for user-image download process https://review.opendev.org/c/openstack/ironic-python-agent/+/890272 | 07:02 |
| opendevreview | Verification of a change to openstack/ironic-python-agent master failed: Conditional creation of RAIDed ESP for UEFI Software RAID https://review.opendev.org/c/openstack/ironic-python-agent/+/891609 | 07:41 |
| opendevreview | Adam Rozman proposed openstack/ironic-python-agent master: implement basic-auth support for user-image download process https://review.opendev.org/c/openstack/ironic-python-agent/+/890272 | 08:11 |
| opendevreview | Merged openstack/ironic master: Add inspection hooks https://review.opendev.org/c/openstack/ironic/+/893533 | 08:12 |
| opendevreview | Adam Rozman proposed openstack/ironic-python-agent master: implement basic-auth support for user-image download process https://review.opendev.org/c/openstack/ironic-python-agent/+/890272 | 08:21 |
| opendevreview | Merged openstack/ironic-python-agent master: Conditional creation of RAIDed ESP for UEFI Software RAID https://review.opendev.org/c/openstack/ironic-python-agent/+/891609 | 11:07 |
| iurygregory | good morning Ironic | 11:36 |
| TheJulia | good morning | 12:58 |
| iurygregory | good morning TheJulia =) | 12:59 |
| opendevreview | Julia Kreger proposed openstack/sushy master: Add a boot progress indicator https://review.opendev.org/c/openstack/sushy/+/896835 | 13:04 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Reset parent_node values to uuids instead of names https://review.opendev.org/c/openstack/ironic/+/889750 | 13:06 |
| TheJulia | rpittau: fixed the typo you spotted ^ | 13:06 |
| TheJulia | another checksum failure. :\ | 13:10 |
| TheJulia | whiskey tango | 13:10 |
| opendevreview | Adam Rozman proposed openstack/ironic-python-agent master: implement basic-auth support for user-image download process https://review.opendev.org/c/openstack/ironic-python-agent/+/890272 | 13:27 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Extract generic image publishing code from image_utils https://review.opendev.org/c/openstack/ironic/+/897681 | 14:00 |
| dtantsur | rpittau, refactoring part 2 ^^ | 14:01 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [WIP] Generic API for attaching/detaching virtual media https://review.opendev.org/c/openstack/ironic/+/894918 | 14:01 |
| dtantsur | I hope it makes ^^^ testable | 14:01 |
| rpittau | dtantsur: ack, I'll have a look as soon as possible | 14:02 |
| opendevreview | Verification of a change to openstack/ironic master failed: Enable OVN CI https://review.opendev.org/c/openstack/ironic/+/885087 | 14:31 |
| TheJulia | hmm, a bug we have, it seems in the agent | 14:58 |
| JayF | TheJulia: the hashing things? I have a suspicion | 14:59 |
| TheJulia | it seems to be the last iteration of the transfer.... it almost seems like the buffer hangs up | 15:00 |
| TheJulia | under pressure | 15:00 |
| TheJulia | but we don't have enough information to know for sure from logs, but there is definitely a issue with retry logic needed | 15:00 |
| JayF | https://opendev.org/openstack/ironic-python-agent/src/branch/master/ironic_python_agent/__init__.py#L27 | 15:01 |
| JayF | I basically have been waiting for the right eventlet version combo for this to bite us | 15:01 |
| JayF | and this behavior is kinda how I'd expect this to be shaped if it's causing us problems | 15:01 |
| JayF | I'm not saying it's the issue for sure by any means, I'm saying I've been suspect this partial-monkey-patch would cause this kinda issue for a while | 15:02 |
| * JayF looks to see if pypi version has been bumped in u-c recently | 15:03 | |
| JayF | s/pypi/eventlet | 15:03 |
| TheJulia | That could definitely be part of it, if it silently lets us think the last packet hasn't been delivered yet | 15:03 |
| TheJulia | which seems super weird | 15:03 |
| TheJulia | we need to fix the retry handling around the failure anyhow. | 15:03 |
| JayF | latest eventlet, from Jan 2023, is in u-c | 15:03 |
| TheJulia | which will also make it just go away | 15:03 |
| JayF | so unlikely it's that | 15:03 |
| TheJulia | in theory | 15:03 |
| TheJulia | well | 15:03 |
| JayF | well lets see when it bumped in u-c | 15:03 |
| TheJulia | we only relatively recently added enough logic to log what was happening there | 15:03 |
| JayF | bumped 9 months ago | 15:04 |
| JayF | so we've been running this code for 9 months at least | 15:04 |
| JayF | you're suggesting it may have been failing this way and we couldn't tell, perhaps? | 15:04 |
| TheJulia | yeah, we added bytes transferred logic say... 3-ish months ago after seeing a failure in CI like this where checksum didn't match | 15:04 |
| JayF | well, we didn't know it was failing *this way* | 15:04 |
| JayF | TheJulia: https://github.com/eventlet/eventlet/issues/798 does this match our usage pattern, perhaps? | 15:05 |
| JayF | the bug matches up, the circumstances match up (high contention in CI) | 15:06 |
| TheJulia | it doesn't hang though | 15:08 |
| TheJulia | The iterator returns liek there are no more chunks and thus transfer is done | 15:08 |
| JayF | ...is it possible for that to happen and be a code problem? | 15:08 |
| JayF | and not a networking problem? | 15:09 |
| JayF | or I guess I should say; a code problem at the level we operate | 15:09 |
| TheJulia | I don't know yet, your jumping to the conclusion you think it is while I'm still trying to wrap my head around it | 15:09 |
| JayF | yeah I'll leave you to your digging | 15:09 |
| TheJulia | What I know of requests, it should only be able to happen If it thinks the buffer is empty, http downloads are notoriously problematic in that regard because all it has to do is not completely verify the transfer to be shorted | 15:10 |
| TheJulia | so it could be any of the above and just a lost packet, really | 15:10 |
| TheJulia | if it parsed the fin before the last packet, I could see that being the cause | 15:10 |
| TheJulia | but that is still super weird | 15:10 |
| rpittau | good night! o/ | 15:34 |
| ThiagoCMC | Hey folks, I've set up Ironic using Bifrost in both a test environment (`testenv`) and a production lab with actual bare metal machines using IPMI & RedFish. I've set the Bifrost variables to `dhcp_provider: none` and `enable_inspector_discovery: true`. Everything works smoothly; the bare metal machines can PXE boot, enroll automatically, and have an OS deployed successfully. However, there's a challenge. If an `active` machine is | 15:39 |
| ThiagoCMC | manually set to PXE and rebooted, it mistakenly starts the IPA process again. In systems like Foreman or MaaS, DHCP management ensures that provisioned nodes don't repeat the "discovery/enlistment" process, even if PXE is the primary boot option. Instead, they boot from local storage. How can this be achieved with standalone Bifrost Ironic? I should note that I used `dhcp_provider: none` because, without it, the `dhcp-boot` entries | 15:39 |
| ThiagoCMC | are missing in the `dnsmasq` configuration, preventing correct PXE booting. However, this setting feels counter intuitive. | 15:39 |
| ThiagoCMC | *Sorry for the multi-line message!* | 15:39 |
| JayF | ThiagoCMC: so I think there's supposed to be something that blocks known-servers from being rediscovered based on their mac | 15:42 |
| TheJulia | ThiagoCMC: yeah, that is a downside of not having a fully managed dhcp service. Realistically we do manage the network boot state, and if your using UEFI (you really really should be), we insert a next boot record anyway which overrides it all. There *is* some hardware out there which disregards new boot overrides in some weird cases (mainly supermicro gear), in particular when IPMI and not Redfish is being used. | 15:42 |
| JayF | but bifrost and inspection is not my primary area of expertise | 15:42 |
| JayF | or I'm wrong :D | 15:42 |
| TheJulia | ThiagoCMC: I *think* stevebaker[m] has done some work about the direct management of dnsmasq, but I just don't know where that is at. He should be awake in 3-4 hours. | 15:43 |
| TheJulia | JayF: there is not a thing to block known systems with bifrost, it is generally static. The model is we setup the machines, and then they stick to what they've been told to do | 15:44 |
| TheJulia | If one is using a full integrated stack/flow, then it is much easier to do, but generally with regards to inspection it is your moving the machine off the original network where you do those activities | 15:45 |
| JayF | I think I'm remembering some old-old-old model stuff where we used iptables to block off known servers from the dhcp server? or vice versa (to only allow known servers in?) | 15:47 |
| JayF | but the more I think of it, the more I think my recollection is from an implementation that was removed years ago | 15:48 |
| TheJulia | So that does exist in inspector, but not configured in bifrost | 15:48 |
| JayF | aha | 15:48 |
| TheJulia | in *large* part because ironic and inspector *share* the dnsmasq | 15:48 |
| TheJulia | it is a single configuration which serves both purposes | 15:48 |
| TheJulia | basically ThiagoCMC is ending back up in discovery, unfortunately | 15:49 |
| JayF | that fits, ty for filling the gaps in | 15:49 |
| JayF | and it sounds like you nailed the problem | 15:49 |
| TheJulia | yeah, downside of trying to be lightweight and compact too | 15:49 |
| TheJulia | you can't have that and have lockout of known entities | 15:49 |
| TheJulia | ThiagoCMC: ... we *should* have persistent config for those nodes stored locally. Are all the nics enrolled in ironic? | 15:50 |
| TheJulia | wheeee meeting time is about to start | 15:50 |
| * TheJulia needs more coffee | 15:50 | |
| opendevreview | Merged openstack/ironic master: Enable OVN CI https://review.opendev.org/c/openstack/ironic/+/885087 | 16:01 |
| JayF | TheJulia: no, you need a party hat | 16:02 |
| * TheJulia turns le castle vania back on and begins dancing on the finance committee call | 16:02 | |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Retry on checksum failures https://review.opendev.org/c/openstack/ironic-python-agent/+/897853 | 16:15 |
| TheJulia | JayF: ^ since we've sort of seen the same thing with unreliable NICs, I'm inclined to think some bytes got lost in a buffer and it just doesn't get detected due to the fun interactions at play | 16:24 |
| * TheJulia resumes dancing | 16:24 | |
| JayF | it's a good fix for some problems even if it may or may not address a root cause for this one (I'm not saying it doesn't; I'm saying *shrug*) | 16:25 |
| TheJulia | if we're loosing a fragmented packet, we may not be able to see it with the way the layering works | 16:26 |
| TheJulia | since there is no size verification when streaming content | 16:26 |
| TheJulia | yeah, it is a complex area, we can only do so much, so better is all we can really do short of being able to go "ohh, I can reproduce it right meow()!" | 16:27 |
| JayF | heads up; I'll be AFK most of the afternoon today | 17:01 |
| TheJulia | Ack ack | 17:01 |
| JayF | TheJulia: this works out for how we have things scheduled; but I'm basically going to miss the last half of Ironic Thursday vPTG | 17:27 |
| JayF | TheJulia: since that is OVN+Redfish day; would you mind owning moderation for that day? | 17:28 |
| JayF | (I'll be moderating the TC PTG during that time) | 17:28 |
| TheJulia | JayF: not at all | 17:33 |
| TheJulia | will do | 17:33 |
| JayF | thanks! | 17:33 |
| TheJulia | just add a note to the schedule | 17:33 |
| * TheJulia cackles with gleee | 17:33 | |
| JayF | I added moderators to each day | 17:34 |
| TheJulia | muchas gracias | 17:37 |
| ThiagoCMC | Thanks for the insights! I appreciate your help. To confirm, I use EFI exclusively. I now have a clearer understanding of Bifrost/Ironic's current implementation and realize it's not something I missed. I'll explore workarounds based on your guidance. Cheers! | 19:03 |
| stevebaker[m] | good morning | 19:05 |
| ThiagoCMC | Morning! ^_^ | 19:05 |
| JayF | ThiagoCMC: No problem, we are here to help. Feel free to hang out and ask more questions :D | 19:08 |
| JayF | ThiagoCMC: start customizing your agent and ask questions about that and maybe I can help, too ;) | 19:08 |
| TheJulia | ThiagoCMC: since we do set hardware to boot from disk and boot from specific records,, we just don't expect them to go back to network unless there is a specific cause, so if the concern is "it is happening", then that is a different issue that we would likely like to better understand | 19:08 |
| stevebaker[m] | ThiagoCMC: Setting dhcp_provider: dnsmasq should add a dhcp filter exclusion for known nodes, but I don't know what other effects it will have on your use case | 19:20 |
| ThiagoCMC | stevebaker[m], adding ` dhcp_provider: dnsmasq` disables the `dhcp-boot` entries in the `dnsmasq` conf, so the machines don't even PXE boot. Source: https://github.com/openstack/bifrost/blob/stable/2023.1/playbooks/roles/bifrost-ironic-install/templates/dnsmasq.conf.j2#L96 | 19:24 |
| ThiagoCMC | I just realized that this line is differnt in Bifrost `2023.2`! | 19:25 |
| ThiagoCMC | There is now a new `or enable_inspector_discovery | bool` in there. Which was something I initially thought about doing. | 19:25 |
| ThiagoCMC | JayF, TheJulia, thanks again for being so welcoming and helpful! I'll stick around. My challenges are more about the unpredictability of the machine life-cycle - sometimes people change settings without realizing it. I'm also comparing Ironic with MaaS and Foreman in the same lab, trying to weigh the pros and cons of each. JayF, I'll surely take you up on that offer; customizing my agent is on the horizon, and I'll need all the | 19:27 |
| ThiagoCMC | help I can get! 😄 | 19:27 |
| TheJulia | ThiagoCMC: understand completely! Pesky humans changing hosts! | 19:28 |
| ThiagoCMC | Yeah lol | 19:28 |
| ThiagoCMC | Maybe the solution is with Bifrost 2023.2! | 19:28 |
| * TheJulia goes off to $nextmeeting realizing that the odds of technical work being done today is rapidly approaching no chance | 19:28 | |
| ThiagoCMC | I want to try... But there's a error, as follows: | 19:29 |
| ThiagoCMC | stevebaker[m], I just tried to install Bifrost 2023.2, but there's an error: `ERROR: 404 Client Error: Not Found for url: https://releases.openstack.org/constraints/upper/stable/2023.2` - Any idea? | 19:29 |
| * TheJulia raises an eyebrow | 19:29 | |
| ThiagoCMC | This happens when I try `bifrost-cli install ...` from `stable/2023.2` branch. | 19:30 |
| JayF | Uh | 19:45 |
| JayF | I really should close IRC if I'm going AFK in the afternooon so I don't see scary things like that ;) | 19:45 |
| JayF | that looks like maybe a url change | 19:46 |
| JayF | I'm going to actually close this window and not dig for now | 19:47 |
| JayF | ThiagoCMC: please file a bug at bugs.launchpad.net about this, and re-link it here, if stevebaker[m] or someone else here doesn't get you fixed up | 19:47 |
| JayF | ThiagoCMC: under bifrost project, if it doesn't exist, under ironic project is fine | 19:47 |
| * JayF actually fades into the background | 19:47 | |
| JayF | ThiagoCMC: in the meantime, just run bifrost from master | 19:48 |
| JayF | 2023.2 bifrost and master bifrost are probably almost identical right now :) | 19:48 |
| ThiagoCMC | Ok, I'll do it. | 19:52 |
| stevebaker[m] | ThiagoCMC: The URL has moved, I'll have a poke soon | 20:29 |
| stevebaker[m] | ThiagoCMC: In the meantime setting the upper constraints env might help, TOX_CONSTRAINTS_FILE=https://releases.openstack.org/constraints/upper/2023.2/upper-constraints.txt bifrost-cli install ... | 20:41 |
| ThiagoCMC | stevebaker[m], cool! It seems to be working. But I used it like this instead: `TOX_CONSTRAINTS_FILE=https://releases.openstack.org/constraints/upper/2023.2/ bifrost-cli install ...` - Because it already appends `upper-constraints.txt`. | 21:08 |
| stevebaker[m] | ah good | 21:11 |
| JayF | ThiagoCMC: so just read your full thing re: machine life cycle; Ironic has a lot of features, if your hardware cooperates, around being able to reset things back to a sensible state between deployments. It's one of the real feature differences between Ironic and some of the alternatives. We try to follow the machines' life as it goes from role to role cleaning it between | 21:15 |
| JayF | each deployment | 21:15 |
| ThiagoCMC | Yep, absolutely! Ironic's flexibility is impressive. | 21:22 |
| ThiagoCMC | One of my next tasks is to see how to integrate vendor-specific tools like Lonevo's `onecli` into a custom IPA image, so the firmware gets upgraded during "enrollment/discovery". Hardware RAID on those machines are also going to be a nice challenge. | 21:25 |
| ThiagoCMC | BTW, how are you folks dealing with things like these (`onecli`, `storcli`, etc)? | 21:26 |
| TheJulia | We've generally been trying to drive vendors to improve out of band interfaces as opposed to internal CLIs and data passing | 21:26 |
| ThiagoCMC | Hmm... Makes sense | 21:26 |
| ThiagoCMC | RedFish BTW? | 21:26 |
| ThiagoCMC | I mean, FTW lol | 21:26 |
| TheJulia | since the Redfish APIs have been a standardization point as opposed to writing code to support each tool and random integration vendor for raid chips | 21:26 |
| ThiagoCMC | Sounds awesome! | 21:28 |
| TheJulia | vendor mileage does, unfortunately, vary there as well | 21:29 |
| ThiagoCMC | Yeah, I know the drill. I've been dealing with Lenovo, Dell, HP, and Supermicro. Each has its unique surprises. | 21:30 |
| ThiagoCMC | So far we were using Foreman, but it kind sucks lol | 21:31 |
| ThiagoCMC | MaaS is promising but limited in different ways. | 21:31 |
| ThiagoCMC | Ironic seems the way to go! | 21:31 |
| TheJulia | <3 | 21:31 |
| ThiagoCMC | ^_^ | 21:32 |
| TheJulia | the dust in the air today must be bad, my allergies are bad enough I'm thinking of calling it a day | 21:32 |
| * TheJulia lives in a desert area | 21:32 | |
| ThiagoCMC | TheJulia, thank you for your help today! I really appreciate it... Take a rest! =P | 21:33 |
| JayF | ThiagoCMC: I'll say; such a hardware manager would be something we'd potentially take upstream, as long as the tooling it uses is publically available (not that we'd build it in by default; but we'd need to point to where it is) | 22:07 |
| JayF | ThiagoCMC: in which case you get the benefit of our review :D (I wrote a majority of the hardware managers we used to secure hardware between tenants for rackspace onmetal) | 22:07 |
| ThiagoCMC | Ok, got it! I'll act as the "QA guy" :-D | 22:08 |
| JayF | lol you'd have to be designer, dev, and qa guy | 22:08 |
| JayF | I am "wisened old architect" ;) | 22:08 |
| ThiagoCMC | Sounds like fun! LOL | 22:08 |
| JayF | (That's kinda joking; but realistically if it was done upstream; we'd be consultants and you'd have to do the work and testing) | 22:09 |
| JayF | but I have done it a lot, and am willing to help -- upstream or not :D | 22:09 |
| TheJulia | JayF: what does that make me?! | 22:09 |
| JayF | TheJulia: I was about to make some kinda chair pun | 22:09 |
| JayF | TheJulia: then I realized you could just reflect it back upon me | 22:09 |
| JayF | TheJulia: Ironic is basically the living room of openstack | 22:09 |
| TheJulia | lol | 22:09 |
| ThiagoCMC | I'll definitely need help ^_^ | 22:09 |
| JayF | Much of the help you need already exists; we have good documentation around the interface there for IPA and several examples. | 22:10 |
| JayF | Those docs and examples were written by some clever wise young architect I once knew | 22:10 |
| JayF | before he got old, grizzled and grey | 22:10 |
| JayF | :P | 22:10 |
| TheJulia | JayF: There are pluses to being the chair! | 22:11 |
| TheJulia | JayF: get some hair dye, it fixes everything! | 22:11 |
| ThiagoCMC | lolol | 22:11 |
| JayF | https://docs.openstack.org/ironic-python-agent/latest/contributor/hardware_managers.html + https://opendev.org/openstack/ironic-python-agent/src/branch/master/examples | 22:11 |
| JayF | those examples might need to be freshened up with service and deploy step decorators | 22:11 |
| ThiagoCMC | Nice, thanks! | 22:12 |
| TheJulia | yeah, they do need to have notes about that | 22:12 |
| JayF | but that's mostly the pattern, you might have to add a decorator in an extra place to expose any steps you make for usage in a deploy template or service steps | 22:12 |
| TheJulia | I'll try to do that tomorrow | 22:12 |
| JayF | I mean, that is prime low hanging fruit | 22:12 |
| JayF | if you wanna write it down and delegate | 22:12 |
| TheJulia | true true | 22:12 |
| JayF | I've been trying hard not to eat all the ripe low hanging apples to leave some quick wins for outreachy/mlh folks incoming | 22:13 |
| TheJulia | The chair is very bad at delegating, good at convincing others | 22:13 |
| TheJulia | Next my wife will tell me it is my lot in life | 22:13 |
| TheJulia | but... not to travel back in time with a space station | 22:13 |
| JayF | (ThiagoCMC context on these jokes are; I am chair of the TC this cycle; TheJulia has been chair of Open Infrastructure Foundation for ... what feels like a long, long time?) | 22:13 |
| TheJulia | JayF: this is my 2nd year as chair, I served as vice chair for a year as well | 22:14 |
| JayF | that might as well be forever | 22:14 |
| JayF | those are post-covid years | 22:14 |
| JayF | those are roughly equivalent to seven normal human years :P | 22:14 |
| TheJulia | wait... seven!??!?!?!?! | 22:15 |
| TheJulia | I can retires now?! | 22:15 |
| * TheJulia feels sadness that retirement is so far away | 22:15 | |
| ThiagoCMC | Wow! It's very nice to reach you folks here so easy... S2 | 22:15 |
| ThiagoCMC | I'm feeling important lol | 22:15 |
| JayF | We always try to be friendly; today I'm extra friendly because I took a mental health afternoon off. It's very successful, you can tell by the fact I'm here :D | 22:16 |
| JayF | (it actually was successful, I feel much better now and got many useful things done) | 22:16 |
| JayF | and honestly, Ironic is used a lot by large faceless companies who don't even want to talk publically about their infra; it's always nice to have a new person in here interested and willing to chat/ask questions/etc | 22:17 |
| JayF | CPU Cores is how you measure success in press releases; new happy users I've had a conversation with is how my brain measures success. | 22:17 |
| ThiagoCMC | Cool, mental health is priority | 22:17 |
| TheJulia | I'm also generally super friendly with JayF... since we've known each other for $OMGIFEELOLDYEARS | 22:18 |
| ThiagoCMC | BTW, I also have some tips/suggestions for the Bifrost documentation (it's a bit hard to follow for starters, but I wrote my own internally to teach co-workers about Ironic and how to set it up easily - step-by-step). | 22:18 |
| JayF | TheJulia: I think we've crossed, for me at least, the "I've known you half my life" threshold :D | 22:18 |
| TheJulia | JayF: Yes, I believe so! | 22:18 |
| JayF | ThiagoCMC: happy to hear them; also I'll note we will likely liberally merge change requests just to edit them | 22:19 |
| JayF | ThiagoCMC: if you've not contributed to an opendev hosted repo before, I have some of the onboarding stuff collated for an MLH fellow who just started on my team, if/when you get to that point lmk and I can share it | 22:19 |
| TheJulia | As much as I want to go buy some hair dye, I need to roll towards the wifey's office and pick her up | 22:19 |
| TheJulia | Tonight is "meet the team" night for the local AHL team | 22:20 |
| JayF | (it's all documented; all I did was assemble the various links in one place to make it easier) | 22:20 |
| JayF | oooh, that's fun | 22:20 |
| JayF | we had sad news; the best defenseman on the team "stuck" in the NHL | 22:20 |
| JayF | so we're probably going the whole season without him here :( | 22:20 |
| * JayF stepping away for a bit | 22:21 | |
| TheJulia | JayF: ouch :( | 22:21 |
| TheJulia | JayF: anyway, have a wonderful remainder of the afternoon! | 22:21 |
| ThiagoCMC | JayF, sure! I'm down for it, never contributed thought (shame on me). I'm also using OpenStack Ansible (and Ceph Ansible) for about 5~6 years. Big fan of Bash/Python/Ansible. I'll write a `.md` somewhere soon (about the docs)! | 22:22 |
| ThiagoCMC | Gotta go... Night, night! ^_^ | 22:32 |
| TheJulia | g;night | 22:33 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!