Wednesday, 2020-08-19

stevebakerjanders: just for fun/fitness/dog-exercise00:01
*** markmcclain has quit IRC00:13
jandersnice! :) and I'm sure the dog would +100:14
*** ijw_ has quit IRC00:14
*** ijw has joined #openstack-ironic00:14
*** markmcclain has joined #openstack-ironic00:15
*** markmcclain has quit IRC00:24
*** Goneri has quit IRC00:41
*** rloo has quit IRC00:52
*** xinliang has joined #openstack-ironic00:54
jandersstevebaker have you installed bifrost and enrolled VMs into ironic lately?01:13
jandersI'm hitting this:01:14
jandershttp://paste.openstack.org/show/796931/01:14
jandersI wonder if others have seen this issue01:14
stevebakerjanders: I haven't used bifrost, I've got my own container based startup script derived from a metal3 script01:15
jandersit happens with both master codebase and a patch I'm testing which is based on master from end of July01:15
jandersright!01:15
jandersweird thing is if I run baremetal commands it's all good, it's just the playbook that's having hassles01:16
stevebakerjanders: I wonder if you're hit by the addition of basic-auth as an alternative to noauth01:16
janderswhat was the timeframe for adding this in?01:16
janderssounds very likely01:16
stevebakerjanders: added in this change https://review.opendev.org/#/c/741964/01:16
patchbotpatch 741964 - bifrost - Support HTTP basic auth and switch bifrost-cli to it (MERGED) - 26 patch sets01:16
jandersyep the timeframe matches01:17
jandersperhaps there is some work that needs to be done in playbooks so that they support the new ways01:18
jandersthanks for pointing me to this stevebaker01:18
stevebakerjanders: maybe, or that change broke noauth01:18
jandersyour first comment is pointing exactly to what Ansible is getting01:19
jandershttp://paste.openstack.org/show/796932/01:24
jandersreproduced!01:24
jandersnow I gotta figure out how to make ansible read the creds properly01:25
jandersone would think it would *just work* but apparently not01:25
stevebakerjanders: hmm, is something doing export OS_CLOUD=bifrost?01:29
jandersgood question01:29
janderschecking01:29
jandersI hardwired cloud and auth_type into the role to match clouds.yaml and it worked01:33
janderssomething is not getting passed correctly it seems - or it's an RTFM issue :)01:33
jandersmaybe I need to re-read the install docs01:33
jandersit doesn't seem to be the RTFM case so far01:36
stevebakerjanders: it could be a gap in supporting .config/openstack/clouds.yaml, because most folk just export OS_* variables01:43
stevebakerjanders: like, this is my stackrc ;) http://paste.openstack.org/show/796933/01:44
jandersI think I got it01:52
jandersif we deploy with http_basic auth by default, we need to change:01:52
jandersplaybooks/roles/bifrost-cloud-config/defaults/main.yml01:52
jandersfrom noauth_mode: true to noauth_mode: false by default too01:52
jandersseems our defaults are out of whack between how we configure ironic and how we configure the playbooks01:53
jandersdo you think this is making sense? shall I propose a change to cover this?01:54
stevebakerjanders: that sounds plausable01:59
jandersok! I will track it down further, cause on one hand we have this:02:00
jandershttps://github.com/openstack/bifrost/blob/master/playbooks/roles/bifrost-ironic-install/templates/ironic.conf.j2#L3202:01
jandersand this:02:01
jandershttps://github.com/openstack/bifrost/blob/master/playbooks/roles/bifrost-ironic-install/defaults/main.yml#L26302:01
*** ociuhandu has joined #openstack-ironic02:01
jandersand on the other hand we have this:02:02
jandershttps://github.com/openstack/bifrost/blob/master/playbooks/roles/bifrost-cloud-config/defaults/main.yml02:02
jandersyet for some reason bifrost seems to deploy ironic configured with http_basic while the enroll playbook tries to use noauth (and hence the error I was getting)02:03
janderslooks like something somewhere is overriding ironic-side auth settings even though it looks like it should be deployed with noauth02:04
*** gudrutis0 has joined #openstack-ironic02:04
stevebakerjanders: yeah it sounds like the desired noauth_mode is not propagating to all roles for some reason02:04
*** another_larsks has joined #openstack-ironic02:05
jandersI have a sneaking suspicion it has to do with the http_basic is the "catch-all" in the ironic.conf template02:05
*** zzzeek has quit IRC02:05
*** gudrutis has quit IRC02:05
*** mgoddard has quit IRC02:05
*** weshay|ruck has quit IRC02:05
*** larsks has quit IRC02:05
*** gryf has quit IRC02:05
*** gudrutis0 is now known as gudrutis02:05
jandersstevebaker I fully agree02:05
*** weshay has joined #openstack-ironic02:06
*** benj_ has quit IRC02:06
janderswell now that I have a workaround I fill finish my testing and doco and then I might look into this further02:06
jandersit's been giving me grief since yesterday02:06
*** tzumainn has quit IRC02:06
stevebakerI wonder why CI passes02:06
*** ociuhandu has quit IRC02:06
jandersthanks for oyur help, without it I would still be stuck and frantically grepping through the playbooks :)02:07
*** benj_ has joined #openstack-ironic02:07
*** ricolin has quit IRC02:09
stevebakerno problem02:09
*** zzzeek has joined #openstack-ironic02:11
jandersgood question re CI - maybe enroll playbook doesn't have enough coverage02:12
jandersthe code looks allright to me but clearly something somewhere is causing a disconnect between values passed to different playbooks02:12
*** Qianbiao has joined #openstack-ironic02:16
*** ricolin has joined #openstack-ironic02:19
*** rcernin has quit IRC02:21
*** gryf has joined #openstack-ironic02:22
openstackgerritHuang Cheng proposed openstack/ironic master: Fix: port attribute name propagate_uplink_status  https://review.opendev.org/74682802:45
*** rcernin has joined #openstack-ironic02:59
*** stevebaker has quit IRC03:14
openstackgerritVishal Manchanda proposed openstack/ironic-ui master: [goal] Migrate testing to ubuntu focal  https://review.opendev.org/74470803:34
*** mkrai has joined #openstack-ironic03:41
*** stevebaker has joined #openstack-ironic03:43
*** rcernin has quit IRC04:14
*** rcernin has joined #openstack-ironic04:21
*** xinliang has quit IRC04:28
*** rh-jelabarre has quit IRC04:29
*** rcernin has quit IRC04:37
*** yolanda has quit IRC04:41
*** rcernin has joined #openstack-ironic04:46
*** stevebaker has quit IRC04:51
*** abdysn has joined #openstack-ironic05:00
*** gudrutis9 has joined #openstack-ironic05:10
*** gudrutis92 has joined #openstack-ironic05:12
*** gudrutis has quit IRC05:13
*** gudrutis92 is now known as gudrutis05:13
*** gudrutis9 has quit IRC05:15
*** gyee has quit IRC05:20
*** ianychoi_ has joined #openstack-ironic05:25
*** ianychoi has quit IRC05:28
*** janders has quit IRC05:51
*** janders has joined #openstack-ironic05:52
*** dmellado has quit IRC06:11
*** janders has quit IRC06:27
*** janders has joined #openstack-ironic06:31
*** dmellado has joined #openstack-ironic06:41
*** patchbot has quit IRC06:41
*** yolanda has joined #openstack-ironic06:50
*** patchbot has joined #openstack-ironic06:52
*** xinliang has joined #openstack-ironic06:54
*** dmellado has quit IRC07:07
iurygregorygood morning Ironic07:12
jandershey iurygregory o/07:12
jandershow are you today?07:12
iurygregoryhey janders o/07:13
*** mkrai has quit IRC07:13
iurygregorystill about 85% =)07:13
jandersdefinitely "glass half full" then, but could be fuller :)07:13
*** mkrai has joined #openstack-ironic07:13
jandersI figured out the permission issue I was hitting yesterday with help from stevebaker07:14
jandershitting something new now07:14
jandersthe permission issue is somehow linked to the introduction of http_basic auth07:14
iurygregorywow07:15
janderseven though bifrost should be deploying ironic in noauth by default, it isn't07:15
jandersI haven't tracked down exactly where things get out of whack but I will07:16
janderslong story short ironic gets deployed with http_basic yet clouds.yaml is configured with noauth07:16
jandershence the mismatch and permission issues07:16
iurygregorylet me grab some more coffee and I can look with you =)07:16
jandersok! :)07:16
iurygregorythere is a way to make sure it gets deploy to noauth07:16
iurygregorybrb07:17
*** belmoreira has joined #openstack-ironic07:19
*** hjensas has joined #openstack-ironic07:27
*** mkrai has quit IRC07:31
iurygregoryjanders, I knew this would happen =X07:35
iurygregory`Support HTTP basic auth and switch bifrost-cli to it `07:35
iurygregory=D07:35
iurygregoryso well, we switched the cli to it...07:36
jandersI think this one will be an easy fix, we just gotta figure out where ironic-install roles and node-enroll nodes get out of whack07:37
janderscause they all seem configured to default to noauth07:37
jandershowever I hit something else nasty which I wanted to ask you about07:37
jandersI reinstalled my bifrost using my patch07:37
jandersI wasn't able to deploy the VMs07:37
iurygregoryhuh? =O07:38
jandersdnsmasq was acting silly, it seemed that I can't PXE anything07:38
jandersi went looking and ended up killing dnsmasq processes manually and copying ironic-generated config to libvirt dnsmasq config file07:38
iurygregorydid you get Dmitry's setup? lol07:38
iurygregoryI remember he had some issues with PXE lol07:38
jandersnow VMs booted (we'll see if they deploy) but this was _really_ hacky07:39
jandersI wonder what went wrong07:39
jandersit looks to me like bifrost didn't touch libvirt's dnsmasq07:39
iurygregoryyeah seems strange .-.07:39
jandersand libvirt's dnsmasq isn't configured for PXE over httpd by default I think07:39
jandershave you seen this before?07:40
janderslooks like Dmitry might have07:40
iurygregoryI haven't seen .-.07:40
jandersit was uber weird... reminded me the very early days of using ironic07:41
jandershunting for tftp config issues07:41
jandersit's nasty cause there is so little troubleshooting tools at that stage07:41
iurygregoryabout the http_basic problem07:42
iurygregoryI think you may be right about the enroll using noauth07:43
jandersI've tracked it down to about 80%07:43
janderslet me dig out code references07:43
iurygregoryhttps://github.com/openstack/bifrost/blob/master/playbooks/roles/ironic-enroll-dynamic/tasks/main.yml07:43
iurygregoryuses bifrost-cloud-config07:43
jandersthe only issue is even though it my seem that both ironic-install and cloud-config are using noauth by default07:44
jandersone is and one is not07:44
iurygregorythat is default to noauth, so maybe when using the bifrost-cli we need to make sure to set a few things before running the command so it will override things in the bifrost-cloud-config07:44
jandersand I haven't figured out where exactly things get mixed up07:44
* iurygregory cheks ironic-install07:44
jandersgood news is my deploy worked (on hacked dnsmasq config)07:44
jandersso let's look at auth issues togetehr07:45
*** ijw has quit IRC07:45
janderscause its a matter of time and others will hit it07:45
jandersit needs fixing (or at least better understanding)07:45
jandershttps://github.com/openstack/bifrost/blob/master/playbooks/roles/bifrost-cloud-config/defaults/main.yml07:45
jandersso this one defaults to noauth07:45
jandershttps://github.com/openstack/bifrost/blob/master/playbooks/roles/bifrost-ironic-install/defaults/main.yml#L26307:46
jandersthis one defaults to noauth, too07:46
iurygregorymaybe we need to make it read a variable that is set...07:46
jandershowever in the jinja2 template, http_basic is the "catch-all"07:46
iurygregoryand re use the value07:46
jandershttps://github.com/openstack/bifrost/blob/master/playbooks/roles/bifrost-ironic-install/templates/ironic.conf.j2#L3207:46
jandersin my env ironic-install var was set to noauth: true yet it still installed with http_basic07:47
jandersI think it's not passed correctly somewhere and then the catch-all kicks in07:47
iurygregoryyeah07:47
jandersone good news is that if http_basic is configured correctly across the board it *just works*07:47
janders(unless it's http_basic auth that broke my httpboot :) )07:48
iurygregoryI have the feeling we should change to07:49
iurygregory"{{ lookup('env','NOAUTH_MODE') }}"07:49
iurygregorysomething like this...07:49
iurygregorybecause since it's hardcoded to true it will probably try to execute things with noauth and BOOM07:50
jandersright!07:51
jandersso this way the user sets an env variable and then both bifrost-cli and ansible base their settings on it?07:51
iurygregoryI know for CI it works like that07:52
iurygregoryand there are ofc some env vars you can set that will identify your setup07:52
iurygregoryfor example if you want an specific location for your venv you can set VENV07:52
jandersyep, I used that a lot07:53
*** dmellado has joined #openstack-ironic07:53
janderson the good news front, all the breakages aside I managed to get bifrost to use the ironic codebase from my patch07:53
jandersand re-tested successfully07:54
iurygregory\o/07:54
jandershttp://paste.openstack.org/show/796935/07:54
jandersthis is the steps07:54
janders(this includes a workaround to the issue we are discussing; I haven't included the dnsmasq fix as I don't yet fully understand what happened there)07:55
iurygregorygotcha07:55
iurygregorythere is also one thing since you run the playbooks by hand you may need to pass -e noauth_mode=false07:56
jandersthat's a good idea in general however I don't think it will help for this particular issue07:56
iurygregoryyeah07:56
iurygregoryonly something that came up on my mind hehehe07:57
janderscause (to my best understanding) the issue is that ironic gets installed with noauth_mode=false and clouds.yaml gets populated with noauth_mode=true07:57
jandersand that happens from bifrost-cli level07:57
iurygregoryyup07:57
jandersand by the time we run enroll, damage is already done07:57
jandersas much as I agree it would be safer to control behaviour of both playbooks from one var07:58
jandersim really curious where the disconnect happens cause the way Dmitry wrote the code seems slick and logical, to me it looks like it should *just work*07:58
iurygregoryyep07:58
jandersbut for some reason the value of noauth_mode gets mangled between here: https://github.com/openstack/bifrost/blob/master/playbooks/roles/bifrost-ironic-install/defaults/main.yml#L26307:59
iurygregoryCI is also happy hehe07:59
jandersand here: https://github.com/openstack/bifrost/blob/master/playbooks/roles/bifrost-ironic-install/templates/ironic.conf.j2#L3207:59
iurygregoryhttps://media.giphy.com/media/Ykttn9oms8Iii1V7QD/giphy.gif our daily life08:01
jandersthis reminds me coming back from holiday early this year when we had the big fires going08:01
janderswe were supposed to overnight in a town which almost burned down the day before08:01
iurygregoryyeah =(08:02
iurygregorysorry bad joke >.<08:02
iurygregoryI forgot about the fire =(08:02
janderscalled the hotel and the local police and they were both like "yesterday it was terrible, but today the wind changed, so it's all good, the fire is 10km away, you can come"08:02
jandersno worries :) I see the analogy too08:02
iurygregorywow08:02
jandersinsane what climate change can do08:03
iurygregorytruth08:03
rpittaugood morning ironic! o/08:09
jandersgood morning rpittau o/08:10
rpittauhey janders :)08:11
jandersrpittau we tracked down that 401 error I was hitting yesterday to some disconnect in auth config between ironic server and client side08:13
jandersin my lab it seems that if default settings are used, ironic is deployed with http_basic while the client side is deployed with noauth and BOOM08:13
jandersstill looking at it08:13
*** ijw has joined #openstack-ironic08:13
iurygregorygood morning rpittau o/08:14
rpittauhey iurygregory :)08:14
jandersI will have a quick break and will be back to troubleshooting this08:15
iurygregoryrpittau, just fyi the removal of the filter in ironic didn't help =(08:15
*** lucasagomes has joined #openstack-ironic08:16
rpittauiurygregory: yeah, I was looking at that08:16
rpittaumaybe adding some logging there would help08:16
*** dtantsur|afk is now known as dtantsur08:16
dtantsurmorning ironic08:16
rpittauhey dtantsur :)08:17
iurygregorymorning dtantsur =)08:17
iurygregoryrpittau, yeah I will look in a bit, going to try something about the tempest tests now =)08:17
*** ijw has quit IRC08:18
*** belmoreira has quit IRC08:21
openstackgerritDmitry Tantsur proposed openstack/ironic-inspector bugfix/10.3: Correct the constraints link in tox.ini  https://review.opendev.org/74685308:23
dtantsurrpittau: please ^^^08:23
rpittauyeah08:23
*** mkrai has joined #openstack-ironic08:23
openstackgerritDmitry Tantsur proposed openstack/bifrost master: Validate that the services are running after installation  https://review.opendev.org/74356908:26
jandersback08:27
jandersgood morning dtantsur08:27
jandersI have some questions if you have time?08:27
dtantsurtotally08:28
openstackgerritDmitry Tantsur proposed openstack/bifrost bugfix/8.3: Update TOX_CONSTRAINTS_FILE for bugfix/8.3  https://review.opendev.org/74617908:29
jandersmaybe let's start with the clean-steps patch08:29
jandershttps://review.opendev.org/#/c/744117/08:29
patchbotpatch 744117 - ironic - [WIP]Generic way to configure clean step priorites - 10 patch sets08:29
jandersI did my best to address your comments - does this revision look better?08:30
openstackgerritDmitry Tantsur proposed openstack/ironic bugfix/15.2: Update TOX_CONSTRAINTS_FILE for bugfix/15.2  https://review.opendev.org/74679708:30
dtantsurjanders: I haven't had a chance to look yet - a large backlog this morning08:31
jandersokay - please take your time08:31
jandershopefully the other questions will be quicker :)08:31
*** mkrai has quit IRC08:31
*** mkrai_ has joined #openstack-ironic08:32
jandersI hit auth issues while doing a fresh install of bifrost this morning08:32
jandersit seems that ironic-install role ends up using http_basic auth by default08:32
openstackgerritDmitry Tantsur proposed openstack/ironic-python-agent bugfix/6.3: Update TOX_CONSTRAINTS_FILE for bugfix/6.3  https://review.opendev.org/74679508:32
janderswhile client config is still on noauth08:32
dtantsurjanders: you mean, the client config is not updated correctly?08:32
jandersmy understanding is that on default settings both ironic-server and client config should be noauth08:33
jandersor am I wrong on this?08:33
dtantsurbifrost-cli defaults to HTTP basic, the playbooks default to no-auth for now (will change in W)08:33
dtantsurit's a bit unfortunate, but the dynamic playbooks are supposed to take the authentication information from clouds.yaml08:34
jandersok so that's expected08:34
jandersI was thinking I'm hitting a bug08:34
*** ociuhandu has joined #openstack-ironic08:34
janderswould it make sense to switch over the playbooks to match?08:34
jandersit should be a one liner change08:35
openstackgerritDmitry Tantsur proposed openstack/bifrost master: dynamic: always use clouds.yaml when it is available  https://review.opendev.org/74685908:35
dtantsurin the W cycle because of backward compatibility. you may be hitting ^^^ though08:36
dtantsurthis patch should be a bit more sophisticated, but you get an idea08:36
jandersok! I will look into this a bit more08:37
jandersmy fix for the day was "sed -i s/^noauth_mode: true/noauth_mode: false’ ~/bifrost/playbooks/roles/bifrost-cloud-config/defaults/main.yml" :)08:37
openstackgerritDmitry Tantsur proposed openstack/bifrost master: dynamic: always use clouds.yaml when it is available  https://review.opendev.org/74685908:37
dtantsurjanders: yeah, rather ^^^08:37
janderslast question: when doing a fresh install of bifrost, I noticed dnsmasq setup was totally busted08:38
dtantsurOo08:38
jandersit wasn't passing next-server info08:38
jandersI think libvirt's default config was left behind08:38
dtantsurwe haven't changed it for a while...08:38
jandersI ended up manually copying the bifrost-generated dnsmasq.conf over the top of libvirt and that fixed it08:38
rpittaujanders: you mean re-installing on top of an old install ?08:39
* dtantsur blinks08:39
jandersyes, but I ran my uninstall script first08:39
dtantsurjanders: the libvirt dnsmasq.conf should not come into play at all08:39
dtantsurmaybe the uninstall script was too eager at some point?08:39
jandersprobably not in regards to the libvirt stuff, it only destroys and undefines VMs08:40
jandershow does bifrost inject the appropriate dnsmasq config during install?08:41
jandersI suppose it needs to change what libvirt set up by default to get VMs to get ipxe from /httpboot right?08:41
jandersI think in my case this didn't happen for some reason08:41
jandersit's possible dnf updates contributed to this08:42
dtantsurI suspect we configure libvirt to not provide its own DHCP08:42
dtantsuri.e. using a bridged network or whatever08:42
dtantsurI need to look into bifrost-create-vms to be sure08:42
jandersI will check if I had any libvirt updates coming in08:43
jandersthis would explain what happened08:43
*** ijw has joined #openstack-ironic08:44
openstackgerritKaifeng Wang proposed openstack/ironic-inspector master: Identify accelerator devices during introspection  https://review.opendev.org/74528908:45
jandersI did update libvirt since the last bifrost install, so this might have messed up dnsmasq configs :(08:48
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic-tempest-plugin master: Update basic_ops  https://review.opendev.org/74144508:49
*** ijw has quit IRC08:49
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic master: Add tempest default_boot_option and altflavor  https://review.opendev.org/74188608:52
openstackgerritHarald JensÃ¥s proposed openstack/ironic master: Switch Ironic to openstacksdk for Neutron  https://review.opendev.org/73487308:54
*** mkrai_ has quit IRC08:55
*** mkrai has joined #openstack-ironic08:55
iurygregoryok, fingers crossed now to see how nova will deal with it08:55
openstackgerritMerged openstack/ironic bugfix/15.2: Update .gitreview for bugfix/15.2  https://review.opendev.org/74679608:55
openstackgerritMerged openstack/ironic-python-agent bugfix/6.3: Update .gitreview for bugfix/6.3  https://review.opendev.org/74679408:55
openstackgerritMerged openstack/ironic-inspector bugfix/10.3: Correct the constraints link in tox.ini  https://review.opendev.org/74685308:55
*** k_mouza has joined #openstack-ironic09:14
*** ijw has joined #openstack-ironic09:15
*** ijw has quit IRC09:20
dtantsurI'd like to pull the trigger on removing non-venv support in bifrost. any objections? or anyone who wants to do it themselves?09:20
jandersno objections from me09:27
janderssounds reasonable09:27
rpittaudtantsur: no objections at all09:27
jandersI will have an hour's break before the team meeting -  braindead after a day of wrestling auth and pxeboot issues, at least I am winning :)09:36
*** mkrai has quit IRC09:39
*** mkrai_ has joined #openstack-ironic09:39
*** Lucas_Gray has joined #openstack-ironic09:42
iurygregorydtantsur, no objections also09:42
*** rcernin has quit IRC09:43
*** Qianbiao has quit IRC09:44
*** Qianbiao has joined #openstack-ironic09:45
*** ijw has joined #openstack-ironic09:46
*** rcernin has joined #openstack-ironic09:49
*** ijw has quit IRC09:51
*** mkrai_ has quit IRC09:54
*** mkrai__ has joined #openstack-ironic09:54
*** Lucas_Gray has quit IRC09:58
*** Lucas_Gray has joined #openstack-ironic10:01
openstackgerritVerification of a change to openstack/bifrost failed: Update .gitreview for bugfix/8.3  https://review.opendev.org/74617810:15
*** ijw has joined #openstack-ironic10:18
*** ijw has quit IRC10:23
*** xinliang has quit IRC10:29
*** rcernin has quit IRC10:30
openstackgerritDmitry Tantsur proposed openstack/bifrost master: dynamic: always use clouds.yaml when it is available  https://review.opendev.org/74685910:30
*** rcernin has joined #openstack-ironic10:34
*** mkrai__ has quit IRC10:36
*** rcernin has quit IRC10:48
*** ijw has joined #openstack-ironic10:50
*** rcernin has joined #openstack-ironic10:52
*** Qianbiao has quit IRC10:53
*** ijw has quit IRC10:54
*** uzumaki has joined #openstack-ironic11:01
*** rcernin has quit IRC11:07
dtantsurjanders: https://review.opendev.org/#/c/746859/ passes CI, a test is appreciated11:18
patchbotpatch 746859 - bifrost - dynamic: always use clouds.yaml when it is available - 3 patch sets11:18
jandersdtantsur ACK; happy to help :)11:18
dtantsurjanders: it probably boils down to testing if it works for you without hacking the defaults11:18
*** rcernin has joined #openstack-ironic11:21
*** ijw has joined #openstack-ironic11:21
*** ijw has quit IRC11:26
*** belmoreira has joined #openstack-ironic11:27
janderssounds good - I can run the patch through my bifrost install process11:29
*** k_mouza has quit IRC11:30
openstackgerritRiccardo Pittau proposed openstack/ironic-python-agent-builder master: Build centos8 on centos8  https://review.opendev.org/74690111:50
*** ijw has joined #openstack-ironic11:52
openstackgerritRiccardo Pittau proposed openstack/ironic-python-agent master: Prepare to move ipa-builder jobs  https://review.opendev.org/74676611:54
openstackgerritRiccardo Pittau proposed openstack/ironic-python-agent-builder master: [DNM] Test CI  https://review.opendev.org/74676711:54
*** Qianbiao has joined #openstack-ironic11:55
*** rh-jelabarre has joined #openstack-ironic11:58
*** ijw has quit IRC11:59
*** Qianbiao has quit IRC12:00
*** ijw has joined #openstack-ironic12:23
*** sshnaidm is now known as sshnaidm|afk12:26
*** ijw has quit IRC12:28
*** ijw has joined #openstack-ironic12:30
openstackgerritKaifeng Wang proposed openstack/ironic-specs master: Support node history  https://review.opendev.org/65281112:33
*** k_mouza has joined #openstack-ironic12:34
*** rh-jelabarre has quit IRC12:41
janderssee you tomorrow Ironic o/12:44
rpittaubye janders12:45
iurygregorybye janders12:46
*** rcernin has quit IRC12:46
*** weshay is now known as weshay|interview12:47
*** Goneri has joined #openstack-ironic12:47
*** rcernin has joined #openstack-ironic12:48
*** mkrai has joined #openstack-ironic12:52
*** guilhermesp has joined #openstack-ironic12:55
guilhermesphey there! are we bumping ironic-inspector victoria in stable/ussuri? iurygregory has been helping me and we ended up finding out this weird thing below12:56
guilhermesphttps://www.irccloud.com/pastebin/BT7U7n5I/12:56
guilhermespironic-inspector==10.3.0 which is victoria, right?12:57
guilhermespso our inspector is not happy12:57
guilhermesphttps://www.irccloud.com/pastebin/xVAoOxar/12:57
iurygregorycorrect 10.3.0 is Victoria12:57
iurygregorysome how your upgrade is just looking at 10.x tags and think it's ussuri I would say12:58
iurygregorybut it's funny that it doesn't push ironic-lib to victoria (and boom problems)12:59
guilhermespyep, im trying to get the bottom of it12:59
*** k_mouza has quit IRC12:59
dtantsurdefine "we" in "we bumping" please13:00
iurygregoryosa13:00
dtantsur10.3.0 is a recent master (victoria) release13:00
guilhermespbtw, we use openstack-ansible stable/ussuri, so we are getting this sha13:01
guilhermesphttps://www.irccloud.com/pastebin/uyUEsgdT/13:01
rpittaummm ironic 21 looks coming from the future :)13:01
dtantsurstable/ussuri is b26838b2913bbc739e88555c2441d1bc7fcd8429 now13:02
*** rh-jelabarre has joined #openstack-ironic13:02
dtantsurbut yeah, the sha was probably valid at that date13:02
iurygregoryrpittau, yep13:03
*** k_mouza has joined #openstack-ironic13:05
*** ijw has quit IRC13:06
*** weshay|interview is now known as weshay13:10
*** rloo has joined #openstack-ironic13:13
iurygregorymaybe somehow osa thinks that 10.x is valid for ussuri13:13
iurygregorythats the only thing that comes to my mind13:13
guilhermespyeah im trying to find out how inspector is being bumped. We only clone the repo and then create a venv based on the sha we were pointing https://github.com/openstack/openstack-ansible/blob/a21b05dd378c267ee3235e2532d01acb599d3fb6/playbooks/defaults/repo_packages/openstack_services.yml#L21713:16
*** weshay is now known as weshay|ruck13:21
*** ijw has joined #openstack-ironic13:25
*** mkrai has quit IRC13:27
*** trandles_ has joined #openstack-ironic13:29
dtantsurfolks, it looks like we could use stable releases of ironic on ussuri and train13:31
*** priteau has joined #openstack-ironic13:31
rpittauguilhermesp: you probably need to change the sha to b26838b2913bbc739e88555c2441d1bc7fcd8429 first to have the latest HEAD13:31
guilhermespyep i just changed rpittau13:32
rpittaudtantsur: I can propose those after this meeting (~30 mins) if noone else do it first13:32
guilhermesphttps://www.irccloud.com/pastebin/o9pVN3J1/13:32
iurygregoryI can do it now13:32
guilhermesp"ironically" it is still getting 10.3.0 for inspector13:32
guilhermesphttps://www.irccloud.com/pastebin/T6Zc8Kml/13:32
iurygregorycan I go ahead rpittau ? =)13:32
rpittauiurygregory: sure! :)13:32
* dtantsur likes that we have a competition for this role :)13:32
iurygregoryit's not a competition lol13:33
iurygregorymy mind is going crazy with nova filters already13:33
iurygregoryso it's good to switch13:33
rpittauguilhermesp: I don't see any upper constraints being set for ironic-inspector, that's why it gets 10.3.013:36
rpittauguilhermesp: I'm talking about this file https://opendev.org/openstack/requirements/raw/commit/e770c44a9f83ff4a1d06366cb563c5907a8b21a0/upper-constraints.txt13:37
rpittauwe probably need to add it :)13:38
rpittauactually it shouldn't be there, but osa reads it from there, I guess the logic behind ironic installation in osa is not perfect on that side13:39
*** tzumainn has joined #openstack-ironic13:42
iurygregorydtantsur, train 13.0.6 and ussuri 15.0.1 wdyt?13:45
dtantsuriurygregory: whatever the next versions are, I guess :)13:46
iurygregorydtantsur, done13:47
TheJuliaCan I go back to bed?13:49
*** trandles_ is now known as trandles13:50
dtantsurTheJulia: yes. good night.13:50
dtantsur:)13:50
iurygregoryTheJulia, you can13:51
*** seba has quit IRC13:52
*** ijw has quit IRC13:56
*** ijw_ has joined #openstack-ironic13:56
TheJuliaI've got an open case in front of me with more lacp + agent issues13:56
* TheJulia sighs13:57
*** mkrai has joined #openstack-ironic13:58
rpittauheh :/13:58
guilhermespyeah rpittau that is probably a OSA thing, im taking a look here and will let you guys know when we fix in our side :P13:58
guilhermespvalheeeeeu iurygregory :P13:58
rpittauguilhermesp: ok! :)13:59
iurygregoryguilhermesp, lol =P13:59
iurygregoryyou owe me a beer in CG =)13:59
guilhermespwonder when you will be able to do it lol13:59
iurygregoryyeah, fingers crossed for Dec/Jan14:00
guilhermespfor now, lets do the imaginary beer14:00
iurygregoryor only Sept next year hehe14:00
* rpittau looks like a good time for tea and cookies14:01
*** ijw_ has quit IRC14:01
iurygregoryrpittau, don't forget to give cookie to zuul14:02
*** abdysn has quit IRC14:15
openstackgerritKaifeng Wang proposed openstack/ironic-specs master: Snapshot support  https://review.opendev.org/74693514:15
iurygregoryhumm only the uefi jobs failed and the multinode...14:24
*** Lucas_Gray has quit IRC14:27
*** ijw has joined #openstack-ironic14:27
guilhermespiurygregory: rpittau solution https://review.opendev.org/#/c/746931/ https://review.opendev.org/#/c/746933/14:28
patchbotpatch 746931 - openstack-ansible - Add ironic-inspector to the openstack_services - 2 patch sets14:28
patchbotpatch 746933 - openstack-ansible-os_ironic - Add ironic-inspector to constraints - 1 patch set14:28
iurygregoryguilhermesp, ++ =)14:29
rpittauguilhermesp: that will help :)14:30
guilhermespyep ive got inspector running now :)14:30
trandlesusing bifrost, inspector is giving me grief14:31
trandleshttp://paste.openstack.org/show/796953/14:31
*** ijw has quit IRC14:32
*** tkajinam has quit IRC14:33
*** Lucas_Gray has joined #openstack-ironic14:34
*** rcernin has quit IRC14:37
*** Lucas_Gray has quit IRC14:39
iurygregorytrandles, inspector is running fine?14:39
iurygregoryhave you checked the logs14:39
*** Lucas_Gray has joined #openstack-ironic14:40
*** ijw has joined #openstack-ironic14:41
*** sshnaidm|afk is now known as sshnaidm14:42
trandlesrunning fine, but the logs (only in journalctl) are nothing but debug messages about syncing with ironic and periodic clean up14:42
trandlesno logs in /var/log/ironic-inspector14:43
iurygregoryO.o14:43
trandlesindeed14:43
dtantsurtrandles: this is a client misconfiguration14:44
dtantsurthe inspector client tries to talk to ironic14:44
dtantsurI wonder what's in your clouds.yaml14:44
dtantsurand yes, /var/log is no longer used since recently14:45
dtantsuror rather: is only used for ramdisk logs14:47
trandleswhere can I find clouds.yaml?  Apologies, this is the first I've tried bifrost in probably 2ish years.14:48
dtantsurtrandles: ~/.config/openstack/14:49
trandlesI see references to it in the docs howto page14:49
trandlesupdated with clouds.yaml: http://paste.openstack.org/show/796955/14:50
*** belmoreira has quit IRC14:55
dtantsurhmm, looks fine to me14:56
dtantsurtrandles: you don't have any OS_ variables defined, do you? check with `env | grep OS_`14:57
trandlesI do, sourced from env-vars14:57
trandlesOS_AUTH_TYPE and OS_ENDPOINT14:57
trandlesthat was it14:58
trandlesunset those two and now I don't get the 404 (not trying to hit ironic-api with introspection requests)14:59
trandlesI do get "endpoint of type baremetal-introspection was not found"14:59
trandlesit works if I add "--os-endpoint=http://localhost:5050" though15:01
dtantsurtrandles: just export OS_CLOUD=bifrost and provide nothing else15:05
trandles:+1:15:06
rpittaunot sure I should laugh, but the centos8 patch for ipa-builder is failing ALL tinyipa jobs15:07
rpittau"Could not find a version that satisfies the requirement bcrypt>=3.1.3" ok....15:07
iurygregoryops =(15:07
rpittauyeah ipa-builder ci is broken because of that, probably cache as bcrypt is at 3.2.0 already15:08
openstackgerritDmitry Tantsur proposed openstack/python-ironic-inspector-client master: Provide a clear error message when trying to access ironic (not inspector)  https://review.opendev.org/74695615:09
dtantsurthis should help debugging in the future ^^^15:09
*** rloo has quit IRC15:24
*** rloo has joined #openstack-ironic15:25
openstackgerritDmitry Tantsur proposed openstack/ironic-python-agent master: Enable the logs collection by default  https://review.opendev.org/74695915:25
dtantsurbfournie: ^^^15:25
openstackgerritDmitry Tantsur proposed openstack/sushy master: Add a CI job with UEFI+vmedia and clean up the job definitions  https://review.opendev.org/74696215:35
openstackgerritDmitry Tantsur proposed openstack/sushy master: DNM try breaking vmedia  https://review.opendev.org/74696315:36
*** gyee has joined #openstack-ironic15:44
*** seba has joined #openstack-ironic15:45
dtantsurTheJulia: https://storyboard.openstack.org/#!/story/2008038 lemme know if it makes sense or I'm still delirious from an alpine air poisoning15:45
*** patchbot has quit IRC15:47
*** patchbot has joined #openstack-ironic15:47
*** dougsz has joined #openstack-ironic15:49
TheJuliadtantsur: it seems reasonable to me15:54
TheJulianot all bmc's expose it though, but it is better than nothing and improves the odds overall15:54
dtantsurTheJulia: we'll add "exposes EthernetInterfaces" to our Redfish profile :)15:59
TheJulia++15:59
TheJuliaof course, that only works with cards that can interact with the bmc firmware16:00
* TheJulia ducsk16:00
TheJuliaducks16:00
*** lucasagomes has quit IRC16:01
dtantsursigh16:01
dtantsurI hope most of cards can16:01
* dtantsur is curious if janders wants to give https://storyboard.openstack.org/#!/story/2008038 a try16:02
TheJuliaThe HPE's and Dells of the world yes. Supermicro's I think are limited to one type of card if my memory's last reading of their docs are correct16:03
* TheJulia needs to replace her lab router so she can have an actual dedicated BMC network16:04
*** mkrai has quit IRC16:04
openstackgerritMerged openstack/bifrost master: dynamic: always use clouds.yaml when it is available  https://review.opendev.org/74685916:17
openstackgerritDmitry Tantsur proposed openstack/bifrost bugfix/8.3: dynamic: always use clouds.yaml when it is available  https://review.opendev.org/74697816:23
openstackgerritMerged openstack/ironic-inspector stable/ussuri: Fix URL parsing to enable redfish_address matching  https://review.opendev.org/74671816:27
*** ociuhandu_ has joined #openstack-ironic16:30
*** Lucas_Gray has quit IRC16:32
*** ociuhandu has quit IRC16:34
*** ociuhandu_ has quit IRC16:34
*** dougsz has quit IRC16:36
openstackgerritMerged openstack/ironic-inspector master: Replace retrying with tenacity  https://review.opendev.org/74566716:44
*** ociuhandu has joined #openstack-ironic16:48
dtantsura simple patch adding better CI coverage for sushy: https://review.opendev.org/#/c/746962/16:52
patchbotpatch 746962 - sushy - Add a CI job with UEFI+vmedia and clean up the job... - 1 patch set16:52
*** ociuhandu has quit IRC16:52
*** dtantsur is now known as dtantsur|afk17:01
dtantsur|afko/17:01
*** k_mouza has quit IRC17:05
*** k_mouza has joined #openstack-ironic17:15
*** k_mouza has quit IRC17:20
*** k_mouza has joined #openstack-ironic17:54
*** k_mouza has quit IRC17:59
*** k_mouza has joined #openstack-ironic18:08
*** k_mouza has quit IRC18:14
*** ijw has quit IRC18:18
*** k_mouza has joined #openstack-ironic18:23
*** k_mouza has quit IRC18:26
*** ijw has joined #openstack-ironic18:33
*** ijw has quit IRC18:37
*** ijw has joined #openstack-ironic18:38
*** k_mouza has joined #openstack-ironic18:40
*** k_mouza has quit IRC18:40
*** ociuhandu has joined #openstack-ironic19:04
*** ociuhandu has quit IRC19:34
*** priteau has quit IRC19:44
*** Lucas_Gray has joined #openstack-ironic20:22
*** ociuhandu has joined #openstack-ironic20:22
openstackgerritJay Faulkner proposed openstack/ironic-python-agent master: Eventlet should be monkey patched as early as possible  https://review.opendev.org/74677420:25
*** ijw has quit IRC20:27
*** ijw has joined #openstack-ironic20:28
*** ijw has quit IRC20:31
*** ijw has joined #openstack-ironic20:31
*** dustinc has quit IRC20:49
*** stevebaker has joined #openstack-ironic20:53
*** uzumaki has quit IRC21:01
*** ociuhandu has quit IRC21:06
JayFIs there any special setup required for `tox -efunctional` to work?21:11
JayFand/or should they be expected to pass outside of devstack generally?21:12
JayF(Talking about in Ironic-Python-Agent)21:14
*** hjensas has quit IRC21:18
*** ijw_ has joined #openstack-ironic21:29
*** ijw has quit IRC21:33
TheJuliafor ipa, I don't know...22:29
TheJuliaI think it should work but I just don't remember22:29
jandersgood morning Ironic o/22:36
*** rloo has quit IRC22:40
TheJuliagood morning22:51
TheJuliain case anyone ever wants to modify https://github.com/OpenStackweb/ironic-website feel free to ping me, I should now be able to approve pull requests22:52
JayFWell FWIW, functional tests fail pretty rapidly, without much useful output, at least when run on CentOS 7.22:54
TheJuliawoot22:54
JayF>     requests.exceptions.HTTPError: 404 Client Error: NOT FOUND for url: http://localhost:9999/v1/commands/22:54
JayFso little output it's extremely difficult to tell how/where it's actually breaking.22:54
TheJuliahmm22:54
JayFIt looks like it's missing cached_node['instance_info'], but I'm unsure where it'd be getting that from22:54
JayFunless it's actually attempting to do a lookup into ironic's API, with a prepopulated node22:54
JayFwhich is what I was presuming, since it only runs in the gate under devstack22:55
TheJuliaoh... I bet it is22:55
TheJuliaor wants to :(22:55
*** Lucas_Gray has quit IRC22:55
JayFif that's what it needs; that's fine. I am just trying to answer the questions "should these be run downstream?" and "how would we run them downstream?"22:55
JayFthe answer seems to be shrug and shrug22:56
TheJuliaI'm sitting here feeling like I looked at them about 9 months ago and sobbed some22:56
TheJuliaand don't remember the rest22:56
JayFit is worth noting that AFAICT there are zero docs about how these run or what they require, outside of bindep.txt22:56
JayFYeah; I sorta took the sign they were nonvoting as an implied lack of confidence in them22:57
TheJuliait is more a I don't think they've really been modified much at all over its lifetime22:57
TheJuliaThe entire project is more integration scenario test driven anyway22:57
JayFsounds like a pretty firm answer to me of "not worth it", which I'll take22:58
*** tkajinam has joined #openstack-ironic22:58
*** rcernin has joined #openstack-ironic22:59
*** Lucas_Gray has joined #openstack-ironic23:02
*** ociuhandu has joined #openstack-ironic23:06
*** ociuhandu has quit IRC23:11
*** Lucas_Gray has quit IRC23:24
*** Goneri has quit IRC23:27
*** trandles has quit IRC23:28
openstackgerritBill Dodd proposed openstack/ironic master: WIP: Add Redfish RAID management to Ironic  https://review.opendev.org/73000023:31

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!