| *** gyee has quit IRC | 00:06 | |
| *** milan has joined #openstack-ironic | 00:06 | |
| *** harlowja has quit IRC | 00:26 | |
| *** v12aml has quit IRC | 00:32 | |
| *** v12aml has joined #openstack-ironic | 00:47 | |
| *** yolanda has quit IRC | 02:03 | |
| *** lbragstad has quit IRC | 02:18 | |
| *** jhesketh_ is now known as jhesketh | 03:14 | |
| *** phuongnh has joined #openstack-ironic | 03:36 | |
| *** jaypipes has quit IRC | 04:02 | |
| *** jaypipes has joined #openstack-ironic | 04:02 | |
| openstackgerrit | paresh sao proposed openstack/ironic master: [WIP]OOB RAID implementation for ilo5 based HPE Proliant servers. https://review.openstack.org/555664 | 04:35 |
|---|---|---|
| *** e0ne has joined #openstack-ironic | 05:00 | |
| *** bfournie has quit IRC | 05:04 | |
| *** moshele has joined #openstack-ironic | 05:56 | |
| *** e0ne has quit IRC | 05:58 | |
| *** e0ne has joined #openstack-ironic | 06:02 | |
| openstackgerrit | Kaifeng Wang proposed openstack/ironic-inspector master: [WIP] Wrap Flask into oslo.service https://review.openstack.org/561823 | 06:32 |
| *** hoonetorg has quit IRC | 06:33 | |
| *** pcaruana has joined #openstack-ironic | 06:36 | |
| *** yolanda has joined #openstack-ironic | 06:41 | |
| *** hoonetorg has joined #openstack-ironic | 06:49 | |
| *** e0ne has quit IRC | 07:02 | |
| *** serlex has joined #openstack-ironic | 07:03 | |
| *** gkadam has joined #openstack-ironic | 07:04 | |
| *** e0ne has joined #openstack-ironic | 07:35 | |
| *** e0ne has quit IRC | 07:41 | |
| *** skazi has quit IRC | 07:45 | |
| openstackgerrit | Merged openstack/ironic master: Simplify subclasses for PXERamdiskDeploy https://review.openstack.org/589206 | 07:50 |
| *** w-miller has joined #openstack-ironic | 07:51 | |
| *** e0ne has joined #openstack-ironic | 07:52 | |
| *** e0ne has quit IRC | 07:53 | |
| *** ajya has joined #openstack-ironic | 08:09 | |
| *** rcernin has quit IRC | 08:11 | |
| openstackgerrit | Madhuri Kumari proposed openstack/ironic master: Implement basic interfaces for GraphicalConsole Interface https://review.openstack.org/547356 | 08:33 |
| openstackgerrit | Moshe Levi proposed openstack/ironic-specs master: Add Support for SmartNIC https://review.openstack.org/582767 | 08:42 |
| *** derekh has joined #openstack-ironic | 08:46 | |
| *** priteau has joined #openstack-ironic | 08:55 | |
| *** milan has quit IRC | 08:58 | |
| openstackgerrit | Ilya Etingof proposed openstack/ironic master: Update docs on ironic boot mode management https://review.openstack.org/588582 | 09:15 |
| *** skazi has joined #openstack-ironic | 09:25 | |
| openstackgerrit | Ilya Etingof proposed openstack/virtualbmc master: Enable debug logging in tests https://review.openstack.org/588345 | 09:27 |
| *** milan has joined #openstack-ironic | 09:28 | |
| *** S4ren has joined #openstack-ironic | 09:31 | |
| *** S4ren has quit IRC | 09:39 | |
| *** S4ren has joined #openstack-ironic | 10:06 | |
| openstackgerrit | Olivier Bourdon proposed openstack/bifrost master: Fix path of ironic-rootwrap when using venv https://review.openstack.org/589422 | 10:25 |
| *** olivierbourdon38 has joined #openstack-ironic | 10:26 | |
| *** phuongnh has quit IRC | 10:47 | |
| *** galaxyblr has joined #openstack-ironic | 11:04 | |
| olivierbourdon38 | Morning ironicers | 11:18 |
| jroll | morning \o | 11:20 |
| openstackgerrit | Olivier Bourdon proposed openstack/bifrost master: Add missing sudoers rule for ironic-inspector-rootwrap https://review.openstack.org/589429 | 11:21 |
| olivierbourdon38 | I have a question in regard to the last patches I submitted https://review.openstack.org/589422 and https://review.openstack.org/589429 | 11:22 |
| patchbot | patch 589422 - bifrost - Fix path of ironic-rootwrap when using venv | 11:22 |
| patchbot | patch 589429 - bifrost - Add missing sudoers rule for ironic-inspector-root... | 11:22 |
| olivierbourdon38 | seems than when using venv in bifrost ironic code is still failing when calling sudo on it most probably due to a command not found in path | 11:23 |
| olivierbourdon38 | Aug 07 13:12:51 master.vagrantup.com ironic-inspector[2029]: 2018-08-07 13:12:51.213 2029 ERROR ironic_inspector CalledProcessError: Command '('sudo', 'ironic-inspector-rootwrap', '/etc/ironic-inspector/rootwra | 11:23 |
| olivierbourdon38 | p.conf', 'iptables', '-N', 'ironic-inspector')' returned non-zero exit status 1 | 11:23 |
| olivierbourdon38 | in y case the venv is located in ~vagrant/.venvs/bifrost and therefore the path to the ironic-inspector-rootwrap executable is ~vagrant/.venvs/bifrost/bin/ironic-inspector-rootwrap | 11:25 |
| olivierbourdon38 | any clue on how this should be dealt with | 11:25 |
| olivierbourdon38 | any clue on how this should be dealt with ? | 11:25 |
| olivierbourdon38 | simple solution would be to symlink from venv to /usr/local/bin aka ironic_rootwrap_dir | 11:26 |
| olivierbourdon38 | (for Ubuntu) | 11:26 |
| olivierbourdon38 | but I am not sure this is the best solution | 11:26 |
| jroll | hrm, ~vagrant/.venvs/bifrost/bin/ should be in the path, I would think | 11:28 |
| jroll | but if not | 11:28 |
| jroll | we could add PATH=~vagrant/.venvs/bifrost/bin:$PATH before the command that executes ironic-inspector | 11:29 |
| * etingof is wondering whom to ping to check out why his ptl ballot never arrived... | 11:30 | |
| jroll | etingof: only two projects had elections | 11:30 |
| jroll | senlin and tacker | 11:30 |
| jroll | if you didn't have commits in those recently, you wouldn't get a ballot | 11:31 |
| jroll | and g'morning :) | 11:31 |
| etingof | aha, I should have read the announcement more carefully perhaps ;) | 11:31 |
| jroll | :P | 11:31 |
| openstackgerrit | Olivier Bourdon proposed openstack/bifrost master: Add missing sudoers rule for ironic-inspector-rootwrap https://review.openstack.org/589429 | 11:31 |
| etingof | good morning jroll, olivierbourdon38 and all the ironicers o/ | 11:31 |
| olivierbourdon38 | morning etingof | 11:32 |
| olivierbourdon38 | mornig jroll | 11:32 |
| jroll | \o | 11:32 |
| olivierbourdon38 | furthermore some CI jobs are failing "msg": "Inspection of node 4e41df61-84b1-5856-bfb6-6b5f2cd3dd11 failed, last error: Failed to start inspection: Unable to establish connection to http://127.0.0.1:5050: but I am not sure it is related to my fix | 11:34 |
| *** slagle has joined #openstack-ironic | 11:39 | |
| TheJulia | good morning | 11:51 |
| *** skazi has quit IRC | 11:57 | |
| etingof | good morning, TheJulia ;) | 12:00 |
| olivierbourdon38 | jroll, for the ironic user, default path is set to /home/ironic/bin:/home/ironic/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin so no venv in there I am affraid | 12:11 |
| *** MattMan has quit IRC | 12:11 | |
| *** MattMan has joined #openstack-ironic | 12:11 | |
| sambetts|afk | the path will also change because of using sudo I think | 12:14 |
| sambetts|afk | it'll switch to the root users PATH | 12:14 |
| sambetts|afk | unless its done with sudo -E | 12:14 |
| jroll | ah yeah | 12:14 |
| jroll | in any case, can modify the path when we run inspector | 12:14 |
| sambetts|afk | will that pass through to the sudo environment? | 12:15 |
| jroll | sudo PATH=... /path/to/inspector :) | 12:17 |
| jroll | there may be a flag needed | 12:17 |
| jroll | from the man page: Environment variables to be set for the command may also be passed on the command line in the form of VAR=value, e.g. LD_LIBRARY_PATH=/usr/local/pkg/lib. | 12:18 |
| sambetts|afk | inspector isn't started with sudo or as the root user though, which is why the code calls out to rootwrap to do root actions :/ | 12:18 |
| jroll | ah, right | 12:19 |
| jroll | this must be a solved problem though, right? | 12:19 |
| sambetts|afk | I guess in inspector we should use the full path to the rootwrap application | 12:19 |
| sambetts|afk | instead of relying on the path | 12:20 |
| sambetts|afk | because whats happening is essentially like a normal user installing something in their /home/<user>/bin and then expecting the root user to find it in their path | 12:20 |
| jroll | really don't want to have to have a config for that | 12:20 |
| jroll | mailing list archives suggest symlinks into /usr/local/bin | 12:21 |
| jroll | (as did olivierbourdon38 ) | 12:21 |
| olivierbourdon38 | currently testing this latest symlinks solution | 12:21 |
| olivierbourdon38 | will most probably update aforementioned patchsets if successful | 12:22 |
| jroll | yeah, don't see why it wouldn't work | 12:22 |
| sambetts|afk | if we added the "-E" flag to the sudo call in inspector it would ensure that inspectors environment variables are passed through to the sudo run, including all the ones set by virtualenv when its active like the PATH changes | 12:25 |
| TheJulia | jroll: there is an awesome question on https://review.openstack.org/#/c/589277 that would be good to get your eyes on | 12:26 |
| patchbot | patch 589277 - ironic - Add conductor_group docs | 12:26 |
| sambetts|afk | the only reason I could see a symlink into /usr/local/bin fail is due to the fact the sudo environment doesn't have the virtualenv "activated" in it which could result in a failured to find certain libraries | 12:26 |
| sambetts|afk | its looks like virtualenv might inject something into the generated bin scripts to activate the env dynamically when its called so it maybe ok | 12:28 |
| TheJulia | if memory serves, it does. There is some magic there | 12:30 |
| olivierbourdon38 | sambetts|afk yes indeed, they all start with #!/home/vagrant/.venv/bifrost/bin/python2 in my case so this should be safe | 12:30 |
| openstackgerrit | Olivier Bourdon proposed openstack/bifrost master: Create symlink for ironic-rootwrap when using venv https://review.openstack.org/589422 | 12:31 |
| sambetts|afk | that selects the python interpreter to use, unsure if it will set the PYTHON_PATH etc too #eeeetomuchmagic | 12:31 |
| olivierbourdon38 | from what I experienced it also does. Magic is soooo deeeeeeeep | 12:32 |
| sambetts|afk | quick scan through the virtualenv docs "If you directly run a script or the python interpreter from the virtualenv’s bin/ directory (e.g. path/to/ENV/bin/pip or /path/to/ENV/bin/python-script.py) there’s no need for activation." | 12:34 |
| jroll | ilo-people, question from a friend: are there iLO APIs to change the serial console port without an OS being present? | 12:35 |
| TheJulia | dtantsur|afk: do you concur with https://review.openstack.org/#/c/589277/1 ? | 12:35 |
| patchbot | patch 589277 - ironic - Add conductor_group docs | 12:35 |
| jroll | TheJulia: kaifeng is correct there | 12:35 |
| jroll | (because we create nodes in the conductor) | 12:35 |
| TheJulia | yeah, and I guess we end up building the request with the information we have, we ship it over rpc to the conductor... and the node either fires up as a task, or not.. and we get a db entry | 12:36 |
| jroll | yep | 12:36 |
| jroll | leaving a comment | 12:36 |
| * jroll didn't realize you got this up | 12:36 | |
| TheJulia | jroll: thanks, I was thinking orpahned but yeah, I guess conductor group would be submitted along with | 12:37 |
| TheJulia | jroll: I thought I messaged you last night, but irc bankruptcy at EOD is easy :) | 12:37 |
| jroll | I know that feeling :) | 12:38 |
| openstackgerrit | Olivier Bourdon proposed openstack/bifrost master: Add missing sudoers rule for ironic-inspector-rootwrap https://review.openstack.org/589429 | 12:39 |
| *** e0ne has joined #openstack-ironic | 12:40 | |
| jroll | TheJulia: you has comments | 12:40 |
| openstackgerrit | Olivier Bourdon proposed openstack/bifrost master: Add missing sudoers rule for ironic-inspector-rootwrap https://review.openstack.org/589429 | 12:40 |
| *** bfournie has joined #openstack-ironic | 12:41 | |
| TheJulia | easy quick review https://review.openstack.org/#/c/407327/6 | 12:41 |
| patchbot | patch 407327 - ironic - Improve the "Ironic behind mod wsgi" documentation | 12:41 |
| TheJulia | jroll: thanks | 12:41 |
| * TheJulia goes and revises doc | 12:48 | |
| TheJulia | hmm, no dmitry today | 12:49 |
| *** weshay has quit IRC | 12:50 | |
| openstackgerrit | Merged openstack/ironic master: Improve the "Ironic behind mod wsgi" documentation https://review.openstack.org/407327 | 13:04 |
| *** tssurya has joined #openstack-ironic | 13:07 | |
| *** trown|outtypewww is now known as trown | 13:12 | |
| olivierbourdon38 | folks, unless I am mistaken bifrost setup does not create /etc/ironic-inspector/rootwrap.conf which makes calls to Aug 07 15:12:30 master.vagrantup.com ironic-inspector[16875]: 2018-08-07 15:12:30.369 16875 ERROR ironic_inspector CalledProcessError: Command '('sudo', 'ironic-inspector-rootwrap', '/etc/ironic-inspector/rootwrap.conf', 'iptables', '-N', 'ironic-inspector')' returned non-zero exit status 1 fail and therefore ironic-inspe | 13:16 |
| olivierbourdon38 | ctor process does not start, is this a known/recent issue ? | 13:16 |
| sambetts|afk | olivierbourdon38: is this in the virtualenv still? | 13:18 |
| olivierbourdon38 | yes | 13:18 |
| sambetts|afk | olivierbourdon38: I think that the files might end up in <virtualenv>/etc | 13:19 |
| sambetts|afk | they should be installed by the pip install process I think | 13:19 |
| sambetts|afk | but under a virtualenv I think any data files get put into the relative root directory | 13:19 |
| olivierbourdon38 | only for ironic, not ironic-inspector | 13:20 |
| olivierbourdon38 | $ ls /home/vagrant/.venv/bifrost/etc/ironic/ | 13:20 |
| olivierbourdon38 | rootwrap.conf rootwrap.d | 13:20 |
| olivierbourdon38 | $ ls /home/vagrant/.venv/bifrost/etc/ | 13:20 |
| olivierbourdon38 | ironic keystone pycadf | 13:20 |
| sambetts|afk | huh thats I'm surprised about then :/ I've had inspector working with bifrost downstream | 13:21 |
| olivierbourdon38 | so have I until I updated my env with latest codes this very morning :-( | 13:22 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Add conductor_group docs https://review.openstack.org/589277 | 13:23 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Add "noop" management and use it in the "ipmi" hardware type https://review.openstack.org/589101 | 13:25 |
| jroll | ok, I am thoroughly confused and probably just missing something... | 13:27 |
| jroll | http://logs.openstack.org/77/589277/1/check/build-openstack-sphinx-docs/fe7e6b6/html/ | 13:27 |
| TheJulia | If anyone has some time, I'd appreciate a quick review on https://review.openstack.org/#/c/589115/ | 13:27 |
| patchbot | patch 589115 - ironic - Switch the "snmp" hardware type to "noop" management | 13:27 |
| jroll | has "condutor groups" (see spelling error) in the table of contents | 13:27 |
| jroll | but the patch has no such spelling error | 13:27 |
| *** namnh has joined #openstack-ironic | 13:27 | |
| TheJulia | jroll: lol | 13:27 |
| jroll | said patch: https://review.openstack.org/#/c/589277/1/doc/source/admin/conductor-groups.rst | 13:28 |
| patchbot | patch 589277 - ironic - Add conductor_group docs | 13:28 |
| jroll | oh wait | 13:28 |
| sambetts|afk | jroll: that build is for patchset 1 | 13:28 |
| jroll | PS1 does specify it | 13:28 |
| jroll | and so does 2, but not in the 1->2 diff | 13:28 |
| jroll | my bad | 13:28 |
| * jroll updates | 13:28 | |
| sambetts|afk | ooo weird haha | 13:28 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Add conductor_group docs https://review.openstack.org/589277 | 13:29 |
| jroll | eh, the 1->2 diff didn't change that file | 13:29 |
| jroll | oh thanks TheJulia :) | 13:29 |
| jroll | +2'd | 13:29 |
| TheJulia | Teach me for typing out patches without wearing glasses.... | 13:29 |
| TheJulia | err, teaches | 13:29 |
| jroll | reviewed the snmp thing | 13:31 |
| *** cdearborn has joined #openstack-ironic | 13:32 | |
| * etingof is on it as well | 13:32 | |
| TheJulia | thanks guys | 13:32 |
| bfournie | hi sambetts|afk, it looks like this fix - https://review.openstack.org/#/c/565776/, wasn't backported to queens, correct? | 13:35 |
| patchbot | patch 565776 - ironic - Fix error thrown by logging in common/neutron.py (MERGED) | 13:35 |
| sambetts|afk | bfournie: no I don't think it ever got backported | 13:36 |
| openstackgerrit | Sam Betts proposed openstack/ironic stable/queens: Fix error thrown by logging in common/neutron.py https://review.openstack.org/589471 | 13:36 |
| sambetts|afk | bfournie: ^ just hit the cherry-pick gui button :-P | 13:37 |
| bfournie | sambetts|afk: thanks! | 13:37 |
| *** serlex has quit IRC | 13:37 | |
| bfournie | sambetts|afk: yeah, just wanted to make sure I didn't miss if a backport was in flight | 13:37 |
| *** moshele has quit IRC | 13:39 | |
| TheJulia | That is a trivial and clear fix, approved | 13:39 |
| *** jcoufal has joined #openstack-ironic | 13:42 | |
| *** mjturek has joined #openstack-ironic | 13:43 | |
| *** galaxyblr has quit IRC | 13:44 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic-specs master: Moving UCS specs to backlog https://review.openstack.org/589474 | 13:45 |
| openstackgerrit | Julia Kreger proposed openstack/ironic-specs master: Move specs completed in 11.1 https://review.openstack.org/589475 | 13:48 |
| cdearborn | good morning ironic! | 13:49 |
| TheJulia | good morning cdearborn! | 13:50 |
| cdearborn | we are running into an issue with ironic in OSP13/Queens: http://paste.openstack.org/show/727522/ | 13:50 |
| cdearborn | the ironic api service sometimes fails when trying to set target raid config on a node | 13:50 |
| cdearborn | it appears that the issue has something to do with wsgi | 13:50 |
| cdearborn | i would like to try shutting off wsgi and seeing if the issue reproduces | 13:51 |
| cdearborn | any thoughts on how to do that? | 13:51 |
| TheJulia | Well that is not good | 13:51 |
| sambetts|afk | cdearborn: is there an error log in the ironic-api service logs?? | 13:51 |
| *** ibmko has joined #openstack-ironic | 13:51 | |
| TheJulia | ++ | 13:52 |
| cdearborn | pwd | 13:52 |
| TheJulia | cdearborn: you'd have to manually stop the httpd running the ironic service, and should then be able to directly invoke the ironic-api in debug mode... although I don't rememer how that is assebled so it may require something like proxying through httpd | 13:53 |
| cdearborn | sambetts|afk: where would i find that? there is no ironic api log in /var/log/ironic | 13:53 |
| TheJulia | but more than likely, there is some error in the logs. Also, would help to have enough data to reproduce, including the blob of data being sent | 13:53 |
| TheJulia | cdearborn: journalctl | 13:53 |
| TheJulia | or in container logs | 13:53 |
| sambetts|afk | "journalctl -u ironic-api.service" or "docker logs <ironic api container>" | 13:54 |
| sambetts|afk | not sure if OSP13 pipe them out anywhere | 13:54 |
| sambetts|afk | pipes* | 13:54 |
| ibmko | Hi TheJulia, just a short question about the yesterday's thing. I've build fedora deploy image according to Ironic documentation and it seems it is missing some packages (sgdisk), was the recommended coreOS deploy image built using the disk-image-builder utility? If so, is it possible to find out which "elements" were used to build it? I just want to make sure that my image is not missing anything | 13:54 |
| cdearborn | sambetts|afk, TheJulia: will have to reproduce the issue. that's easily done, but it will take about an hour to run | 13:55 |
| sambetts|afk | ibmko: coreos isn't built using DIB, its build using a custom script in the openstack/ironic-python-agent repo | 13:56 |
| ibmko | sambetts|afk, aha, ok | 13:57 |
| sambetts|afk | ibmko: but I've not heard of the DIB build not working because of missing packages before | 13:57 |
| cdearborn | sambetts|afk, TheJulia: this is happening when we do RAID configuration prior to overcloud deployment. we are able to set target raid config on 8 out of 9 nodes: 3 controllers, 3 computes, 3 storage. the json being sent is identical to the json for at least 2 other nodes that work | 13:57 |
| TheJulia | cdearborn: look for apache log data | 13:58 |
| ibmko | sambetts|afk, my image ended up with this: u'/openstack/venvs/ironic-17.0.5/bin/ironic-rootwrap: Executable not found: sgdisk (filter match = sgdisk)\n' | 13:58 |
| ibmko | I am wondering, perhaps I need to include this element: https://docs.openstack.org/diskimage-builder/latest/elements/iscsi-boot/README.html | 13:58 |
| TheJulia | ibmko: no, coreos is built with some in-repo scripts and a docker file. If sgdisk is missing you can always explicitly add the package. | 13:59 |
| TheJulia | ibmko: is sgdisk present on the conductor node? | 13:59 |
| TheJulia | ibmko: what deploy_interface are you using? | 14:00 |
| cdearborn | TheJulia: exactly where would I find the apache log data that you want? | 14:00 |
| sambetts|afk | TheJulia, cdearborn: I've seen weirdness with wsgi proxying before caused by attempts to stream data through the proxy instead of httpd caching it then passing it on to the wsgi server, so if the tcp connection to the client is lost before the wsgi application is loaded and ready to read the data the service gets an IOError | 14:00 |
| TheJulia | cdearborn: interesting... | 14:00 |
| TheJulia | cdearborn: if it is being fronted by uwsgi, I think it would be in the container logs under uwsgi's logging | 14:01 |
| ibmko | TheJulia, i am using iscsi method, uh, I thought that the error message comes from the image, but when you ask this, I now realize that it might actually come from conductor ? | 14:01 |
| TheJulia | sambetts|afk: that is kind of what I suspect is going on, socket closing before reply processed which is weird. :\ | 14:02 |
| cdearborn | sambetts|afk, TheJulia: the issue seems to be load based. if i try rerunning the script after deployment errors out, then it works perfectly | 14:02 |
| olivierbourdon38 | forgive the "newbie" question but is there a way to trigger a complete CI pass on current bifrost without submitting a "dummy" patch ? | 14:02 |
| TheJulia | ibmko: yeah, the conductor needs the same utilities with the iscsi deploy method | 14:02 |
| TheJulia | s/method/interface/ | 14:03 |
| jroll | cdearborn: before we go too far in debugging, what's the error being returned? | 14:03 |
| ibmko | TheJulia, great, thanks, I think this might be it | 14:03 |
| cdearborn | jroll: http://paste.openstack.org/show/727522/ | 14:03 |
| sambetts|afk | cdearborn, TheJulia: uwsgi has an option for dealing with taking to long to read the data on the service side, its "post-buffering = 1" in the uwsgi config, I ran into similar issues on a different project | 14:05 |
| jroll | cdearborn: ah, yeah that does look like some sort of load issue or something | 14:05 |
| jroll | sambetts|afk: ++ | 14:05 |
| *** baha has joined #openstack-ironic | 14:05 | |
| TheJulia | sambetts|afk: yeah, that sounds really familar | 14:06 |
| jroll | I've also seen similar long long ago | 14:06 |
| cdearborn | jroll, TheJulia, sambetts|afk: i am at the point where i can launch a deployment. shall i launch it with no changes to reproduce the issue, or should i try it with the above change? | 14:08 |
| jroll | cdearborn: well, you have a traceback, I'm not sure what else we'd want to learn from reproducing :) | 14:09 |
| sambetts|afk | looks like its apache itself thats running the wsgi daemon not uwsgi so I'm not sure what the equivolent config option is :/ | 14:10 |
| TheJulia | cdearborn: I think your best bet is to file a bugzilla with as much detail as possible on reproduction steps | 14:11 |
| openstackgerrit | Olivier Bourdon proposed openstack/bifrost master: Dummy commit to trigger complete CI test pass https://review.openstack.org/589493 | 14:13 |
| cdearborn | TheJulia, y, will file a BZ. we've only seen this issue in our scripting where we are doing RAID config in parallel using 1 thread per node across 9 nodes, so reproducing outside our scripting will probably be hard | 14:13 |
| TheJulia | olivierbourdon38: good plan :) | 14:13 |
| TheJulia | cdearborn: include that as a note on the bz. We might be able to make the client better behave/be more tollerant, but.... that is a weird one. | 14:14 |
| olivierbourdon38 | TheJulia yes indeed, seems like ironic-inspector does not launch in my env and reverted changes in bifrost without any success, I guess something has been broken in ironic-inspector code itself | 14:14 |
| olivierbourdon38 | which will also explain bifrost CI failiures where :5050 is not reachable | 14:15 |
| *** jrist has quit IRC | 14:15 | |
| TheJulia | olivierbourdon38: or we merged something yesterday possibly... | 14:15 |
| * TheJulia needs more coffee | 14:17 | |
| NobodyCam | mmm Coffee | 14:17 |
| olivierbourdon38 | as mentioned previously I reverted a 1month old bifrost env of my own which I know worked like a charm but as it is dragging dependencies which might have changed this could explain why it now fails | 14:17 |
| NobodyCam | And Good Morning Ironic'ers | 14:17 |
| olivierbourdon38 | currently testing some potential causes | 14:18 |
| jroll | morning NobodyCam :) | 14:18 |
| olivierbourdon38 | morning NobodyCam | 14:18 |
| NobodyCam | Good Morning jroll olivierbourdon38 :) | 14:18 |
| *** mgoddard has joined #openstack-ironic | 14:20 | |
| cdearborn | good morning NobodyCam | 14:21 |
| NobodyCam | Good Morning cdearborn :) | 14:21 |
| *** rpioso|afk is now known as rpioso | 14:26 | |
| rpioso | Good morning, ironicers | 14:26 |
| *** gkadam_ has joined #openstack-ironic | 14:27 | |
| *** jrist has joined #openstack-ironic | 14:27 | |
| NobodyCam | Good Morning rpioso | 14:28 |
| openstackgerrit | Sam Betts proposed openstack/ironic master: [DNM] Testing Cisco Third Party CI https://review.openstack.org/514312 | 14:28 |
| *** gkadam has quit IRC | 14:29 | |
| rpioso | NobodyCam: :-) | 14:30 |
| NobodyCam | :) | 14:30 |
| *** yolanda has quit IRC | 14:31 | |
| etingof | good morning rpioso and sorry for not responding wrt Redfish console! | 14:34 |
| olivierbourdon38 | I think I found the cause of the ironic-inspector issue | 14:34 |
| rpioso | etingof: Great to hear from you! Are you feeling better? And how's the weather? | 14:35 |
| olivierbourdon38 | at least following the test I now have it running without issues after deploying bifrost from scratch | 14:35 |
| olivierbourdon38 | https://review.openstack.org/#/c/589020/ | 14:35 |
| patchbot | patch 589020 - ironic-inspector - Clean up pxe_filter related deprecations (MERGED) | 14:35 |
| rpioso | etingof: No worries about the Redfish console questions. I added a topic to the Stein PTG etherpad, which I see you added to :-) Thank you! | 14:36 |
| etingof | rpioso, I am better, thank you! The temperature over here is almost hitting Fahrenheit's feverish wife's body temperature :-/ | 14:36 |
| *** yolanda has joined #openstack-ironic | 14:37 | |
| *** hongbin has joined #openstack-ironic | 14:38 | |
| rpioso | etingof: Yikes! Hopefully, both with break soon. If it gives you any comfort, it's 88 here and heading to 95, which is very toasty for New Hampshire. | 14:39 |
| etingof | rpioso, so, the PTG topic I've seen is more about the Ironic event service rather than the Redfish console...? or did I miss it? | 14:39 |
| *** e0ne has quit IRC | 14:39 | |
| etingof | rpioso, nooo, it does not excite me at all that you guys have to suffer from the heat over there as well... | 14:40 |
| rpioso | etingof: I determined that the iDRAC virtual console does not offer the events. Rather, they're available through the iDRAC web client. The virtual console is visible and launchable from that client. | 14:42 |
| rpioso | etingof: So, I disentangled them. | 14:42 |
| etingof | rpioso, right, but can iDRAC console be managed through ironic? | 14:44 |
| rpioso | etingof: No, not presently :-( | 14:45 |
| etingof | rpioso, do you guys have plans to add Redfish interface to iDRAC console management? ;) | 14:45 |
| TheJulia | etingof: eek :( | 14:47 |
| *** priteau has quit IRC | 14:50 | |
| olivierbourdon38 | TheJulia see root cause above ^^^ | 14:50 |
| TheJulia | olivierbourdon38: I had a feeling that was it | 14:52 |
| olivierbourdon38 | reverting this very single change and everything is back to normal behaviour | 14:52 |
| sambetts|afk | I assume whats happened is that before manage_firewall was set to False | 14:54 |
| sambetts|afk | so then it never touched the sudo code that inspector had | 14:54 |
| olivierbourdon38 | seems very like this yes sambetts|afk | 14:55 |
| sambetts|afk | now the firewall_driver option needs to be set to noop to disable it | 14:55 |
| sambetts|afk | so its enabled by mistake | 14:55 |
| *** hoonetorg has quit IRC | 14:57 | |
| olivierbourdon38 | in my conf manage_firewall is set to false | 14:58 |
| TheJulia | yeah, should be a quick fix to the to set the filter to noop | 14:59 |
| rpioso | etingof: Checking ... | 14:59 |
| TheJulia | So a few random virtualbmc patches are up. some of them align fo the python3-first goal, others just fix a couple minor things. https://review.openstack.org/#/q/project:openstack/virtualbmc Not a priority in any sense | 15:02 |
| * etingof is thinking to file a story wrt Redfish console support in ironic... | 15:05 | |
| etingof | ...but he is never sure in his SB searching powers | 15:05 |
| *** pcaruana has quit IRC | 15:11 | |
| rpioso | etingof: https://topics-cdn.dell.com/pdf/idrac9-lifecycle-controller-v3151515_api-guide_en-us.pdf starting on p. 54 | 15:12 |
| openstackgerrit | Julia Kreger proposed openstack/bifrost master: Fix ironic-inspector config https://review.openstack.org/589514 | 15:15 |
| TheJulia | olivierbourdon38: ^^^ i'm hoping that fixes it | 15:15 |
| TheJulia | etingof: just be sure to define what you mean by console :) | 15:16 |
| olivierbourdon38 | TheJulia currently testing it | 15:17 |
| etingof | TheJulia, what I had in mind is smth similar to what ipmitool does for SOL, but over Redfish | 15:17 |
| TheJulia | etingof: so serial over lan basically, if supported that makes sense to me | 15:18 |
| TheJulia | I'm hoping we make progress on graphical/vnc-ish consoles this cycle | 15:19 |
| TheJulia | well, next cycle | 15:19 |
| TheJulia | which basically kind of begins any day now | 15:19 |
| etingof | TheJulia, at the Redfish level the mechanics is the same for all console types I guess... it's just that we ask Redfish server to configure a console for us to connect somehow | 15:19 |
| etingof | in theory, it can be SOL, VNC, SPICE whatever | 15:20 |
| TheJulia | Well, then it sounds like we would want to be one level abstracted further | 15:22 |
| etingof | rpioso, I can't find much details on console support in that doc... I am trying to understand if Dell servers could be configured (over Redfish) to provide some sort of console over network... | 15:22 |
| TheJulia | at least interacting with a remote redfish api | 15:22 |
| etingof | TheJulia, btw, sushy seems to have console support implemented | 15:24 |
| TheJulia | Serial I think | 15:24 |
| TheJulia | but yeah | 15:24 |
| etingof | TheJulia, abstracted one level further? should it be higher than ipmitool offers? | 15:25 |
| *** priteau has joined #openstack-ironic | 15:26 | |
| TheJulia | etingof: I say one step further in the fact our exisiting console interface is geared towards serial usage (nevermind the ilo patch somewhere along the way to slip vnc support into their driver)... but with graphical consoles the methods will be similar, just the user will be able to ask for graphical instead, and both profiles will need to work because some people want serial console logging... | 15:26 |
| etingof | ah, I see what you mean | 15:27 |
| NobodyCam | +1 for serial console logging | 15:27 |
| mjturek | is it possible to disable output from IPA? | 15:28 |
| TheJulia | mjturek: please clarify | 15:28 |
| mjturek | TheJulia: sorry, I'm debugging a deploy ramdisk that isn't getting an IP through serial over lan | 15:29 |
| mjturek | I'm logging into the ramdisk to investigate | 15:29 |
| mjturek | but IPA's output is interrupting | 15:29 |
| TheJulia | kill ipa? | 15:29 |
| mjturek | that's a good idea!!! | 15:29 |
| mjturek | I'll try that now | 15:30 |
| TheJulia | I mean, it is the only way ;) | 15:30 |
| etingof | I think we discussed one issue wrt Redfish console support with mraineri - it seems that with current schema console stuff resides at a wrong place e.g. at Manager while it should rather be at ComputerSystem | 15:30 |
| mjturek | gotcha, thanks TheJulia :) | 15:31 |
| *** gyee has joined #openstack-ironic | 15:32 | |
| openstackgerrit | Merged openstack/ironic master: Node gets stuck in ING state when conductor goes down https://review.openstack.org/586993 | 15:36 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Switch the "snmp" hardware type to "noop" management https://review.openstack.org/589115 | 15:40 |
| openstackgerrit | Olivier Bourdon proposed openstack/bifrost master: Fix ironic-inspector config https://review.openstack.org/589514 | 15:40 |
| olivierbourdon38 | TheJulia many thanks works perfectly with small mods ;-) | 15:41 |
| TheJulia | olivierbourdon38: thanks for updating the patch | 15:42 |
| olivierbourdon38 | TheJulia yavw | 15:43 |
| *** olivierbourdon38 has quit IRC | 15:46 | |
| *** S4ren has quit IRC | 15:50 | |
| * ibmko succesfully provisioned his first Ironic instance :) | 15:57 | |
| *** skazi has joined #openstack-ironic | 15:58 | |
| TheJulia | ibmko: \o/ | 15:59 |
| ibmko | TheJulia, its nice that the ToR switch multi-tenancy integration actually works | 15:59 |
| ibmko | with our Juniper switches | 15:59 |
| sambetts|afk | \o/ | 16:00 |
| rpioso | Can nova scheduler's determination of the baremetal host be tied to the deploy steps to execute? | 16:00 |
| ibmko | are there any plans for having security groups provisioned on the ToR switches too ? :) | 16:01 |
| *** jcoufal has quit IRC | 16:02 | |
| sambetts|afk | ibmko: security groups don't match quite 1-1 with the way ToRs do ACLs, depending on which neutron driver your using a vendor might have added support for them | 16:05 |
| ibmko | sambetts|afk, but given that Ironic project seems to try to match Nova project semantics, there shloud be a way how to get security groups on Ironic node, or am I wrong ? | 16:08 |
| *** namnh has quit IRC | 16:10 | |
| sambetts|afk | security groups are completely implemented on the network side never on the VM/instance thats booted, with VMs they do it by putting iptables between the VM and the network to filter out traffic which works for ingress and egress traffic, with ironic there is no iptables, so you have to put it in the ToR to mirror neutrons behaviour, but ToRs typically only supports ingress or egress not | 16:13 |
| sambetts|afk | both, but they are becoming more feature rich so some may be able to do it | 16:13 |
| ibmko | yeah | 16:14 |
| *** jaypipes_ has joined #openstack-ironic | 16:16 | |
| *** jaypipes has quit IRC | 16:17 | |
| *** jaypipes_ has quit IRC | 16:17 | |
| *** e0ne has joined #openstack-ironic | 16:18 | |
| *** anupn has joined #openstack-ironic | 16:25 | |
| TheJulia | ibmko: That is good to hear :) | 16:31 |
| TheJulia | ibmko: I believe arista implemented support in their ml2 driver to handle security groups that are set, fwiw. | 16:32 |
| *** jaypipes has joined #openstack-ironic | 16:33 | |
| *** trown is now known as trown|lunch | 16:33 | |
| TheJulia | rpioso: no, nova is not deploy steps aware and I think we would only really be able to be deploy template aware once we implement support to handle templates for deployment definitions. Why do you ask? | 16:34 |
| jroll | that's my recollection of the plan, allow the user to request traits or something via nova, which when set in instance_info trigger a particular deploy template | 16:35 |
| *** gkadam_ has quit IRC | 16:38 | |
| *** moshele has joined #openstack-ironic | 16:44 | |
| TheJulia | Reviews on https://review.openstack.org/#/c/588582/ and https://review.openstack.org/589277 would be appreciated | 16:45 |
| patchbot | patch 588582 - ironic - Update docs on ironic boot mode management | 16:45 |
| patchbot | patch 589277 - ironic - Add conductor_group docs | 16:45 |
| *** tssurya has quit IRC | 16:45 | |
| TheJulia | NobodyCam: If you have a few minutes, an you take a quick look at https://review.openstack.org/#/c/589514/2 | 16:45 |
| patchbot | patch 589514 - bifrost - Fix ironic-inspector config | 16:45 |
| *** ajya has quit IRC | 16:51 | |
| *** derekh has quit IRC | 17:00 | |
| *** w-miller has quit IRC | 17:01 | |
| *** mgoddard has quit IRC | 17:05 | |
| rpioso | TheJulia: I was thinking along the lines of templates. Roughly, how would that work? | 17:06 |
| *** milan has quit IRC | 17:06 | |
| TheJulia | rpioso: an operator would define approved templates for their environment, that would essentially be a blob of json for deploy steps, and we would have logic in the conductor that would compare the posted traits from the flavor to drive which template is used | 17:12 |
| *** e0ne has quit IRC | 17:20 | |
| *** e0ne has joined #openstack-ironic | 17:21 | |
| *** e0ne has quit IRC | 17:22 | |
| eandersson | When using the flat network layout, is there a reason why a provisioning port is never created/deleted in the code? | 17:24 |
| anupn | eandersson: I believe that is because there is no separate provision network. Only one network is created initially for tenant and that is used for all actions. | 17:27 |
| eandersson | Would that be the same network it uses once it is provisioned as well? | 17:27 |
| eandersson | Trying to figure out a simple (and temporary) design for an untrusted network. | 17:29 |
| eandersson | Ideally the provisioning network would be trusted, and the network used by the actual baremetal after provisioning could be untrusted. | 17:30 |
| rpioso | TheJulia: Cool. Thank you! | 17:30 |
| *** harlowja has joined #openstack-ironic | 17:31 | |
| anupn | eandersson: Yes it will be the same network | 17:32 |
| TheJulia | eandersson: You would need the multitenancy features or some code to actually toggle the network attachments.. which is what a number of folks did before we had support through neutron | 17:35 |
| rpioso | TheJulia: To answer your earlier question, it just seems like that's how it should work. I see it's a proposed topic for the PTG :) | 17:38 |
| eandersson | TheJulia, yea we are looking at something like that | 17:39 |
| TheJulia | eandersson: call something() in the dhcp code perhaps? | 17:39 |
| eandersson | We were thinking that maybe we could just attach the tftpd server and a swift proxy to the same namespace as the prov network | 17:41 |
| eandersson | and just switch of the port when we are done provisoning | 17:41 |
| *** trown|lunch is now known as trown | 17:42 | |
| eandersson | *off | 17:42 |
| eandersson | You wouldn't happen to know who did this before neutron by any chance? | 17:43 |
| *** harlowja has quit IRC | 17:43 | |
| TheJulia | oath had a downstream implementation, rackspace did as well although apparently it was a bitmore substantial code wise. jroll would be the person whose brain to tap for that. I think NobodyCam mentioned something similar once | 17:44 |
| jroll | oath doesn't have a downstream network switching thing, we use the flat networking model | 17:45 |
| jroll | rackspace had patches that evolved into what ironic has upstream | 17:46 |
| TheJulia | hogepodge: do you know of any that did networking integration with ironic for multitenancy before we actually did it in ironic? | 17:47 |
| jroll | honestly, whatever you write is going to be less simple than figuring out how to do it with upstream code | 17:47 |
| jroll | you're going to have to write software to configure the switch dynamically, etc | 17:47 |
| *** moshele has quit IRC | 17:50 | |
| TheJulia | Some shops already have that though. I feel like one of the mass open cloud folks kind of did it as well where the called their switch management tooling | 17:50 |
| jroll | yeah, fair | 17:50 |
| eandersson | jroll, that is kinda what I figured as well | 17:50 |
| jroll | just saying it isn't super simple :) | 17:50 |
| eandersson | We don't care about multitenacy as we only have one use case we support at the moment | 17:51 |
| jroll | well, you're still essentially doing what we call "multitenant networking", just only with one tenant :) | 17:52 |
| eandersson | But... ideally our implementation would allow us to move into multitenacy later on | 17:52 |
| eandersson | I see - yea makes sense | 17:52 |
| *** e0ne has joined #openstack-ironic | 18:04 | |
| TheJulia | I guess you could create your own dhcp driver module and then use that to navigate an upgrade | 18:04 |
| openstackgerrit | Merged openstack/ironic master: Add "noop" management and use it in the "ipmi" hardware type https://review.openstack.org/589101 | 18:11 |
| *** baha has quit IRC | 18:24 | |
| *** skazi has quit IRC | 18:29 | |
| *** openstackgerrit has quit IRC | 18:49 | |
| *** moshele has joined #openstack-ironic | 18:50 | |
| *** baha has joined #openstack-ironic | 19:04 | |
| *** moshele has quit IRC | 19:18 | |
| *** e0ne has quit IRC | 19:32 | |
| mbeierl | shoot - lost my notes. TheJulia, I know you told me before, but how do I run a single unit test again in Ironic? | 19:43 |
| *** skazi has joined #openstack-ironic | 19:44 | |
| jroll | mbeierl: tox -e py27 ironic.tests.unit.some.module.SomeTestCase.test_something | 19:44 |
| mbeierl | jroll: thanks! I knew it was something simple! | 19:44 |
| jroll | that's a regex, so you can even just pass something like test_pxe and it'll match any full path with that in it | 19:45 |
| jroll | and you're welcome :) | 19:45 |
| *** openstackgerrit has joined #openstack-ironic | 19:48 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Add admin-only provisioner reference arch https://review.openstack.org/586618 | 19:48 |
| TheJulia | jroll: thanks! | 19:49 |
| *** ceryx has joined #openstack-ironic | 20:17 | |
| *** hoonetorg has joined #openstack-ironic | 20:29 | |
| *** markguz_ has joined #openstack-ironic | 20:41 | |
| -openstackstatus- NOTICE: Due to a bug, Zuul has been unable to report on cherry-picked changes over the last 24 hours. This has now been fixed; if you encounter a cherry-picked change missing its results (or was unable to merge), please recheck now. | 20:42 | |
| TheJulia | Goodnight folks, I'm going to go get some dinner | 21:06 |
| *** baha has quit IRC | 21:10 | |
| rpioso | TheJulia: Good night! | 21:20 |
| *** trown is now known as trown|outtypewww | 21:27 | |
| *** bfournie has quit IRC | 21:32 | |
| openstackgerrit | Merged openstack/networking-baremetal master: Remove testrepository and .testr.conf https://review.openstack.org/581679 | 21:53 |
| *** bdodd has quit IRC | 21:56 | |
| *** slagle has quit IRC | 21:56 | |
| *** priteau has quit IRC | 22:09 | |
| *** rcernin has joined #openstack-ironic | 22:20 | |
| *** strigazi has joined #openstack-ironic | 22:20 | |
| *** strigazi has quit IRC | 22:21 | |
| *** strigazi has joined #openstack-ironic | 22:21 | |
| *** cdearborn has quit IRC | 22:25 | |
| *** strigazi has quit IRC | 22:27 | |
| *** strigazi has joined #openstack-ironic | 22:28 | |
| *** strigazi has quit IRC | 22:28 | |
| *** strigazi has joined #openstack-ironic | 22:29 | |
| *** strigazi has quit IRC | 22:32 | |
| *** strigazi has joined #openstack-ironic | 22:33 | |
| *** hongbin has quit IRC | 22:39 | |
| openstackgerrit | Merged openstack/ironic master: Switch the "snmp" hardware type to "noop" management https://review.openstack.org/589115 | 22:41 |
| *** rpioso is now known as rpioso|afk | 22:54 | |
| *** priteau has joined #openstack-ironic | 23:15 | |
| *** pmannidi has joined #openstack-ironic | 23:19 | |
| *** marlinc has quit IRC | 23:23 | |
| *** markguz_ has quit IRC | 23:41 | |
| *** marlinc has joined #openstack-ironic | 23:41 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!