| *** gkadam has joined #openstack-kayobe | 05:12 | |
| *** egonzalez has joined #openstack-kayobe | 06:58 | |
| *** priteau has joined #openstack-kayobe | 08:14 | |
| *** ktibi has joined #openstack-kayobe | 08:22 | |
| *** mgoddard has joined #openstack-kayobe | 08:28 | |
| mgoddard | morning kayobians | 08:38 |
|---|---|---|
| verdurin | Morning. Did mangle EFI booting yesterday, though there was something missing from the IPA image, so the node was stuck in a loop. | 08:43 |
| mgoddard | well that's progress. Any idea what is missing from IPA? | 08:44 |
| mgoddard | which IPA images are you using? | 08:44 |
| verdurin | About to start a meeting, so I'll check later. Saw it go past in the logs, something hardware-related. | 08:46 |
| verdurin | Was just using the image built by kayobe. | 08:48 |
| mgoddard | I think the default images are coreos, although we typically build centos-based IPA images locally | 08:50 |
| mgoddard | requires some config in etc/kayobe/ipa.yml | 08:50 |
| mgoddard | http://kayobe.readthedocs.io/en/latest/deployment.html#building-deployment-images | 08:51 |
| ktibi | mgoddard, kayobe use centos by default, no coreOS | 09:00 |
| ktibi | verdurin, I tried to use uefi on a platform but I abandoned for legacy boot :/ I use ipxe. | 09:03 |
| *** dougsz_ has joined #openstack-kayobe | 09:04 | |
| mgoddard | ktibi: default is coreos: https://github.com/openstack/kayobe/blob/master/ansible/group_vars/all/ipa#L65 | 09:09 |
| mgoddard | if you don't build the image. The default image built is CentOS | 09:09 |
| ktibi | mgoddard, yes I talked about build | 09:09 |
| mgoddard | ok | 09:10 |
| ktibi | hum you make me think maybe I'll try coreOS with my node which not work during inspection | 09:11 |
| ktibi | because I use devuser with centos not work. I build myself the IPA and copy in bifrost container, no work too | 09:11 |
| ktibi | I can't get the LOG for see why inspection fail | 09:11 |
| *** ktibi has quit IRC | 09:23 | |
| *** ktibi has joined #openstack-kayobe | 09:33 | |
| verdurin | mgoddard: here is the introspection error http://paste.openstack.org/show/2B38YztxgJQTR6WDU6zI/ | 10:15 |
| verdurin | The node was listed as enrolled on the seed, but clearly the error meant it kept trying | 10:16 |
| mgoddard | verdurin: did you add extra-hardware to ipa_collectors_extra in ipa.yml? | 10:17 |
| verdurin | mgoddard: I think I copied over the default one to the config path, so I could get it to recognise the build images = True setting | 10:21 |
| verdurin | mgoddard: there's nothing in ipa_collectors_extra | 10:23 |
| mgoddard | verdurin: hmm, is extra-hardware present in the kernel command line arguments of the default PXE config? | 10:28 |
| mgoddard | argument name would be ipa-inspection-collectors | 10:31 |
| verdurin | mgoddard: ah it is, yes. I copied that from elsewhere. Trying with that removed now | 10:31 |
| mgoddard | verdurin: that can be useful, but requires a ramdisk built with the python-hardware package | 10:32 |
| verdurin | Yes, I'd been using that with TripleO | 10:32 |
| mgoddard | we use https://github.com/stackhpc/stackhpc-image-elements/tree/master/elements/ipa-extra-hardware to install it | 10:32 |
| verdurin | And unlike with the deployment image, it looks like there is a mechanism for including the extra elements | 10:44 |
| mgoddard | correct | 10:44 |
| mgoddard | it uses stackhpc.os-image-elements | 10:45 |
| verdurin | It's now trying to set the node name to a switch interface picked up via LLDP: http://paste.openstack.org/show/wcx8UMCl4gkR1BkpbhTL/ and failing... | 10:50 |
| mgoddard | hmm, yeah that's not going to work | 10:53 |
| mgoddard | http://paste.openstack.org/show/727809/ | 10:55 |
| mgoddard | kayobe's probably a bit too opinionated there, we should change the default | 10:55 |
| mgoddard | although if your switch supports sending the interface description via LLDP it is a neat discovery trick | 10:56 |
| verdurin | mgoddard: grand, I'll try that. | 10:56 |
| mgoddard | normally it requires setting an option on the switch globally or per interface | 10:56 |
| mgoddard | verdurin: the early stages of use are very useful, before you become accustomed to the idiosyncracies of the software. If you have any feedback it would be appreciated | 10:59 |
| mgoddard | not now necessarily | 10:59 |
| verdurin | mgoddard: I've been trying to keep notes of everything | 10:59 |
| mgoddard | verdurin: perfect | 11:00 |
| *** dougsz_ has quit IRC | 11:50 | |
| *** dougsz_ has joined #openstack-kayobe | 12:54 | |
| *** egonzalez has quit IRC | 13:03 | |
| verdurin | Is there a way of using generic ipmitool with the iLO devices on this hardware? | 13:44 |
| mgoddard | I imagine so, but you might need to configure the server to allow IPMI | 13:45 |
| verdurin | I've already done that. | 13:48 |
| verdurin | Just need to add the values in kayobe-config/etc/kayobe/inventory/group_vars/all/bmc ? | 13:48 |
| mgoddard | are you running 'overcloud inventory discover'? | 13:50 |
| verdurin | NOt yet. | 13:50 |
| verdurin | Looking at this because when introspection finishes I noticed that it's unable to power off the node. | 13:50 |
| verdurin | And that failure is reported in the introspected node's record. | 13:51 |
| mgoddard | which failre? | 13:51 |
| verdurin | The power management failure. | 13:51 |
| mgoddard | have you configured the ipmi username & password? | 13:51 |
| mgoddard | ipmi_username, ipmi_password in etc/kayobe/bmc.yml | 13:52 |
| verdurin | I had configured the introspection IPMI details, I'm now looking at doing so more generally, hence the question | 13:52 |
| mgoddard | these will be applied to nodes during inspection via rules | 13:52 |
| verdurin | Okay. Referring to the earlier question, it's a bit confusing having these settings in multiple places. | 13:53 |
| mgoddard | most settings for kayobe are in etc/kayobe/*.yml | 13:54 |
| mgoddard | I'd put the bmc config in etc/kayobe/bmc.yml rather than etc/kayobe/inventory/group_vars/all/bmc | 13:55 |
| verdurin | It does explain the precedence here, admittedly http://kayobe.readthedocs.io/en/latest/configuration/kayobe.html#configuration-patterns | 13:56 |
| mgoddard | it takes a bit of getting used to, but there is a method to the madness | 13:57 |
| mgoddard | open to suggestions of course | 13:58 |
| verdurin | I keep getting caught out because (e.g.) I run 'ag ipmi_username' then pick what seems the likeliest one. | 13:58 |
| mgoddard | I guess the thing to keep in mind is that kayobe tends to set global defaults in kayobe/ansible/group_vars/all/<file>, and these variables can be set in kayobe-config/etc/kayobe/<file>.yml | 14:00 |
| mgoddard | there are exceptions, and sometimes you need to use inventory host_vars/group_vars when variables need to vary between hosts/groups | 14:01 |
| verdurin | Hmm. It's still not picking up the IPMI credentials. | 14:18 |
| mgoddard | are the inspector rules correct | 14:18 |
| verdurin | They're basically the same ones you pasted earlier, with LLDP included because I fixed that on the switch. | 14:21 |
| mgoddard | can you check the rules in inspector? | 14:21 |
| mgoddard | you can use python-ironic-inspector-client with OS_TOKEN=fake OS_URL=http://localhost:5050 | 14:22 |
| verdurin | First rule is 'Set IPMI driver_info if no credentials' | 14:26 |
| mgoddard | does it have the correct credentials? | 14:28 |
| mgoddard | openstack baremetal introspection rule show <rule> | 14:28 |
| verdurin | No, neither username nor password are in there. | 14:31 |
| verdurin | Both are set in etc/kayobe/bmc.yml and I rebuilt the image after doing so. | 14:31 |
| mgoddard | the rules get registered during seed service deploy | 14:32 |
| mgoddard | you could short cut via | 14:33 |
| mgoddard | kayobe playbook run ansible/seed-introspection-rules.yml | 14:33 |
| mgoddard | it should re-register the rule | 14:33 |
| verdurin | Ah. | 14:35 |
| verdurin | Yes, looks like that's made some changes. | 14:38 |
| mgoddard | great | 14:39 |
| mgoddard | you should be able to reprocess the inspection data | 14:39 |
| mgoddard | openstack baremetal introspection reprocess <node> | 14:39 |
| mgoddard | verdurin: have you read http://www.stackhpc.com/ironic-idrac-ztp.html? It covers our typical cloud discovery setup | 14:42 |
| verdurin | mgoddard: I hadn't seen that. Will read it properly. Sadly this pilot hardware is HPE, so I can't use all your nice stuff. | 14:44 |
| verdurin | The production hardware will be Dell with Dell management switches, albeit running Cumulus. | 14:45 |
| mgoddard | cumulus on dell has come up for us before, although haven't tried it yet | 14:45 |
| verdurin | Marvellous - power management works now, so I can add the remaining nodes. | 14:47 |
| mgoddard | great! | 14:47 |
| verdurin | Thanks for your forebearance mgoddard | 14:47 |
| mgoddard | I'm hoping that from this I can generate some better config docs for the seed side of things | 14:49 |
| mgoddard | no problem verdurin - this is one of the hardest parts of getting a system up and running, and it's inevitably the first thing you do, when you're least familiar | 14:50 |
| verdurin | mgoddard: I think this is the fourth different OpenStack installation I've used, but who's counting... | 14:52 |
| verdurin | (installation method) | 14:52 |
| mgoddard | still looking for 'the one'? | 14:53 |
| verdurin | I quite liked TripleO but debugging was just too much of a mither. | 14:54 |
| mgoddard | that has been my (limited) experience with it | 14:54 |
| mgoddard | once you get to know the patterns used in kolla ansible, I think it works quite well | 14:55 |
| mgoddard | provides a lot of control | 14:55 |
| mgoddard | seen this: https://www.slideshare.net/MarkGoddard2/to-kayobe-or-not-to-kayobe? | 14:56 |
| verdurin | I very much like Ansible, so it's an excellent base on which to build. | 14:56 |
| verdurin | Did skim through that. Should take a proper look. | 14:57 |
| verdurin | mgoddard: incidentally, the reprocessing of introspection data appears to require some extra configuration that I'm currently missing. | 15:08 |
| verdurin | "Inspector is not configured to store data. Set the [processing] store_data configuration option to change this" | 15:11 |
| verdurin | Not that it really matters. | 15:11 |
| mgoddard | oh, my bad | 15:14 |
| mgoddard | it requires https://review.openstack.org/460005 | 15:14 |
| mgoddard | didn't make it into queens | 15:14 |
| mgoddard | you can cherry pick & rebuild the container image if it's useful to you | 15:14 |
| *** gkadam has quit IRC | 16:06 | |
| *** dougsz_ has quit IRC | 16:58 | |
| *** mgoddard has quit IRC | 17:01 | |
| *** ktibi has quit IRC | 17:29 | |
| *** mgoddard has joined #openstack-kayobe | 17:30 | |
| *** priteau has quit IRC | 17:52 | |
| *** mgoddard has quit IRC | 18:30 | |
| *** priteau has joined #openstack-kayobe | 19:47 | |
| *** gkadam has joined #openstack-kayobe | 19:47 | |
| *** priteau has quit IRC | 20:12 | |
| *** gkadam has quit IRC | 22:34 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!