Friday, 2019-01-04

« Thursday, 2019-01-03 Index Saturday, 2019-01-05 »
* TheJulia head desks due to the fun of neutron devstack00:20
NobodyCamTheJulia: Coffee break time?00:27
TheJulianah, I was thinking cooking dinner time00:28
TheJuliaI started at like 5:30 AM this morning00:28
NobodyCamieek00:28
TheJuliayeah00:28
TheJuliaI kind of want to find an excuse to go downtown or something for dinner, but I suspect I'll just end up cooking00:28
*** gyee has quit IRC00:45
NobodyCamhehhee00:48
NobodyCamI'm in Irvine won't be back home until about 7:3000:49
*** mjturek has quit IRC01:19
*** rloo has quit IRC01:50
*** dnuka has joined #openstack-ironic02:54
dnukagood morning ironic02:55
*** dnuka has quit IRC03:08
*** dnuka has joined #openstack-ironic03:09
*** hwoarang has quit IRC03:09
*** lifeless has joined #openstack-ironic03:10
*** hwoarang has joined #openstack-ironic03:10
*** hwoarang has quit IRC03:18
*** hwoarang_ has joined #openstack-ironic03:18
jiapeiGood morning dnuka03:19
dnukagood morning jiapei :)03:19
*** mkrai has joined #openstack-ironic04:20
mkraiHi, what is the timeline for ironic-spec freeze?04:21
*** c0dyhi11 has joined #openstack-ironic04:30
*** codyhill has quit IRC04:31
*** c0dyhi11 is now known as codyhill04:31
openstackgerritDebayan Ray proposed openstack/ironic master: Allow missing ``local_gb`` property  https://review.openstack.org/62809606:01
*** hwoarang_ has quit IRC06:25
*** hwoarang has joined #openstack-ironic06:29
openstackgerritDhanuka Warusadura proposed openstack/sushy master: [WIP] Introduce ``convert_to_date``  https://review.openstack.org/62588906:31
openstackgerritDhanuka Warusadura proposed openstack/sushy master: [WIP] Introduce ``convert_to_date``  https://review.openstack.org/62588906:34
*** mkrai has quit IRC06:37
*** rcernin has quit IRC06:47
*** rpittau has joined #openstack-ironic07:53
rpittaugood morning ironic! o/07:53
dnukamorning rpittau o/07:54
rpittauhi dnuka :)07:54
*** moshele has joined #openstack-ironic07:54
openstackgerritMerged openstack/ironic-python-agent master: Attempt to read the partition table after writing an image  https://review.openstack.org/55768708:01
*** moshele has quit IRC08:17
iurygregorymorning everyone o/08:59
rpittauhey iurygregory :)08:59
iurygregoryyo rpittau o/08:59
dnukamorning iurygregory o/08:59
iurygregorydnuka, o/09:00
iurygregoryrpittau, i made networking-baremetal work with python2 \o/09:00
rpittauI saw it! that's great!09:02
iurygregoryi think the failure for python3 was because it took like +20min and the timeout condition is 1200 (20min)09:06
rpittauit could be as it doesn't seem like there's any actual error09:08
openstackgerritIury Gregory Melo Ferreira proposed openstack/networking-baremetal master: [DNM] Tests  https://review.openstack.org/62467109:08
iurygregoryyeah09:08
iurygregorytrying only python3 now in my patch to see =)09:08
rpittaufingers crossed :)09:09
iurygregorybut i think give +20min for python3 is a strange way to fix things09:09
iurygregoryalso seems that BUILD_TIMEOUT is the one that it uses instead of IRONIC_TEMPEST_BUILD_TIMEOUT09:10
rpittauit's very long, maybe it's stuck somewhere, I found the timeout exception now09:10
iurygregoryim looking at the logs now too09:10
rpittauhttp://logs.openstack.org/26/614026/16/check/networking-baremetal-multitenant-vlans/6b14293/job-output.txt.gz#_2019-01-03_17_03_01_04208609:11
openstackgerritMerged openstack/ironic master: Fix SushyError namespacing in Redfish inspection  https://review.openstack.org/62809509:12
*** S4ren has joined #openstack-ironic09:17
dnukaI've been trying to fix this for the whole day and I've tried a lot of stuff, thank you very much rpittau for the review :)09:31
rpittaudnuka, ur welcome :)09:31
dnuka:)09:31
*** derekh has joined #openstack-ironic09:35
*** S4ren has quit IRC09:36
* rpittau crawl to the kitchen to make some coffee09:44
*** e0ne has joined #openstack-ironic09:52
iurygregoryrpittau, check this out https://zuul.openstack.org/stream/0b54fd004c65488a867edaaed0aedc08?logfile=console.log10:20
* iurygregory LOL10:22
rpittauthat succeeded10:22
iurygregoryyep same conf =P10:22
rpittauoO10:23
iurygregoryhome the test_baremetal_multitenancy doesnt fail10:23
iurygregoryHAHAHA10:23
iurygregoryi changed to python3 to run in the DNM lets see XD10:24
iurygregoryaprox 20 min to finish so it wont timeout10:24
rpittaulet's see :/10:25
rpittauI have the feeling this is an issue with nova conf10:33
rpittauit's missing the connection parameter under the database section10:33
rpittauand causing this: http://logs.openstack.org/26/614026/16/check/networking-baremetal-multitenant-vlans/6b14293/controller/logs/screen-n-cond-cell1.txt.gz#_Jan_03_16_31_07_29788110:34
iurygregorylooking10:36
iurygregorybut the conf have the db information10:41
rpittauyes, I just noticed that, it doesn't like it for some reason :/10:42
* iurygregory flips nova (╯°□°）╯︵ ┻━┻)10:44
iurygregorynow im mad '-' the job gave networking-baremetal-multitenant-vlansTIMED_OUT in 1h 33m 04s10:45
iurygregorybut both tests are ok now http://logs.openstack.org/71/624671/11/check/networking-baremetal-multitenant-vlans/0b54fd0/job-output.txt.gz10:46
iurygregory'-'10:46
openstackgerritIury Gregory Melo Ferreira proposed openstack/networking-baremetal master: [DNM] Tests  https://review.openstack.org/62467110:55
dnukahey iurygregory, can you please explain this a bit :) https://review.openstack.org/#/c/625889/5/sushy/tests/unit/resources/test_settings.py@7911:05
patchbotpatch 625889 - sushy - [WIP] Introduce ``convert_to_date`` - 7 patch sets11:05
openstackgerritDerek Higgins proposed openstack/ironic master: Testing UEFI job on bionic  https://review.openstack.org/62675511:08
openstackgerritDhanuka Warusadura proposed openstack/sushy master: Introduce ``convert_to_date``  https://review.openstack.org/62588911:22
*** S4ren has joined #openstack-ironic11:28
*** rpittau is now known as rpittau|lunch11:28
*** dtantsur|afk is now known as dtantsur11:32
dtantsurmorning/afternoon ironic11:33
dnukamorning dtantsur11:34
*** dnuka has quit IRC11:39
*** rpittau|lunch has quit IRC11:43
*** rpittau|lunch has joined #openstack-ironic11:44
iurygregorymorning dtantsur o/11:44
iurygregorydtantsur, would be a problem extend the max time for the job in 10 minutes?11:45
dtantsuriurygregory: not in theory, but we should at least try to understand why it becomes longer11:54
iurygregorydtantsur, ack i made python3 for network-baremetal work but it took 1h 33min and gave TIME_OUT since was 1h 30 the max time11:55
iurygregoryand seems that some times oslo doesnt find nova configuration http://logs.openstack.org/26/614026/16/check/networking-baremetal-multitenant-vlans/6b14293/controller/logs/screen-n-cond-cell1.txt.gz#_Jan_03_16_31_07_29788111:56
dtantsurthe latter should definitely be reported to nova11:57
dtantsurotherwise, I can imagine python 3 being slightly slower, so maybe it's fine11:57
*** rpittau|lunch has quit IRC11:58
dtantsuriurygregory: actually, the ironic's multinode job seems to have 3 hours for timeout: https://github.com/openstack/ironic/blob/master/zuul.d/ironic-jobs.yaml#L33011:58
iurygregorythe time for python2 was fine so maybe have a different timeout for each job11:58
dtantsurso it should be safe to bump11:58
iurygregorycool11:59
iurygregorytesting to see if everything goes well so i can see a +1 before give another try in Julia`s patch11:59
*** rpittau|lunch has joined #openstack-ironic12:01
dtantsurTheJulia: spotted at least three different random failures :( see the whiteboard12:08
*** rpittau|lunch has quit IRC12:13
*** rpittau|lunch has joined #openstack-ironic12:13
*** moshele has joined #openstack-ironic12:14
*** moshele has quit IRC12:17
*** rpittau|lunch has quit IRC12:28
*** bfournie has quit IRC12:29
*** rpittau has joined #openstack-ironic12:33
rpittauapparently my internet connection also went for lunch :/12:34
dtantsureveryone needs lunch!12:37
rpittaulol yeah12:38
rpittauhi dtantsur :)12:38
dtantsurhi12:38
openstackgerritDmitry Tantsur proposed openstack/ironic master: Allocation API: allow skipping retries in TaskManager  https://review.openstack.org/62842812:39
*** lifeless has quit IRC12:42
*** moshele has joined #openstack-ironic12:46
openstackgerritzhouxinyong proposed openstack/bifrost master: Delete the duplicate words in  CONTRIBUTING.rst  https://review.openstack.org/62843312:47
*** moshele has quit IRC12:47
*** moshele has joined #openstack-ironic13:00
*** jcoufal has joined #openstack-ironic13:02
openstackgerritDerek Higgins proposed openstack/ironic master: Testing UEFI job on bionic  https://review.openstack.org/62675513:02
TheJuliadtantsur: ugh :(13:03
*** moshele has quit IRC13:04
* TheJulia goes back to sleep13:04
openstackgerritDmitry Tantsur proposed openstack/ironic master: [WIP] Allocation API: implementation without HA and take over  https://review.openstack.org/62822313:14
*** bfournie has joined #openstack-ironic13:14
iurygregoryTheJulia, i took a look at the state of the nodes they were in BUILD13:16
iurygregoryand some strange errors in nova13:17
*** trown|outtypewww is now known as trown13:26
dtantsurthis week everything fails at random. my bet is that the CI has serious hangover :)13:48
iurygregorydtantsur, for sure13:48
iurygregorytimeout everywhere13:49
* etingof lost hope to get his patches verified13:51
*** mmethot has joined #openstack-ironic14:00
*** rloo has joined #openstack-ironic14:06
openstackgerritIury Gregory Melo Ferreira proposed openstack/networking-baremetal master: [DNM] Tests  https://review.openstack.org/62467114:10
*** iurygregory has quit IRC14:15
*** iurygregory has joined #openstack-ironic14:15
*** bfournie has quit IRC14:41
TheJuliagood morning everyone14:41
TheJuliaiurygregory: What was the status in ironic though? Nova can generate and log some funky errors before ironic is online and hosts are available, but afterwords it should be error free unless there was an underlying failure. That could be a quick failure for the baremetal node to deploy since nova checks every ?1 second?14:43
TheJuliamaybe 2, I don't remember what the loop defaults to14:43
TheJuliaderekh: hey, I remember you fighting with neutron plugins and some of our CI jobs, am I remembering correctly?14:43
iurygregorymorning TheJulia o/14:43
iurygregorylet me get the logs ive found14:43
iurygregoryhttp://logs.openstack.org/26/614026/16/check/networking-baremetal-multitenant-vlans/6b14293/controller/logs/screen-n-cond-cell1.txt.gz?level=ERROR14:44
iurygregoryhttp://logs.openstack.org/26/614026/16/check/networking-baremetal-multitenant-vlans/6b14293/controller/logs/screen-n-cpu.txt.gz?level=ERROR14:44
iurygregoryhttp://logs.openstack.org/26/614026/16/check/networking-baremetal-multitenant-vlans/6b14293/controller/logs/screen-n-api.txt.gz?level=ERROR14:44
dtantsurmorning TheJulia14:44
iurygregoryFailed to provision instance doesnt seems good =(14:45
*** mjturek has joined #openstack-ironic14:47
rpittauhi TheJulia :)14:48
TheJuliahttp://logs.openstack.org/26/614026/16/check/networking-baremetal-multitenant-vlans/6b14293/controller/logs/screen-ir-cond.txt.gz#_Jan_03_16_26_58_037622 <-- smells like delayed IO14:49
TheJuliabecause forcing a refresh can become blocked on a direct io transaction still in flight14:49
TheJuliaor just high latency on the disk because the refresh will only work if the disk is not locked for writes and direct io is blocking to the seek operation for the partition table refresh14:50
dtantsurTheJulia: oh, the same thing as the one I put on the whiteboard14:57
dtantsurplease add your findings there as well14:57
TheJuliastill looking at it trying to cut the data out of the dstat log15:00
TheJuliaoh yeah, it does look like we have multi-second io lag15:01
TheJuliaand we're likely compensating with weirdness due to buffer cache15:02
*** Goneri has quit IRC15:05
TheJulialooks like it is across providers, but that could just be noisy neighbors. The first looks like a resurgance of the neutron/ironic port attachment stuff but the actual timings need to be checked. The feel like delayed io :\15:10
*** bfournie has joined #openstack-ironic15:13
iurygregorytimeouts =(15:17
TheJuliaiurygregory: did you look into the neutron logs?15:34
iurygregorynow python3 works and python2 fails ¯\_(ツ)_/¯ tks OpenStack <315:34
openstackgerritDerek Higgins proposed openstack/ironic master: Testing UEFI job on bionic  https://review.openstack.org/62675515:34
TheJulialets start a list of jobs that could use a recheck on a sunday or something, and I'll go kick them off15:34
TheJuliajust so we can see if a quieter time is better behaved15:35
iurygregoryTheJulia, looking now give me a minute, was trying 4 devstacks since morning to figure out some confs15:35
TheJuliaiurygregory: no rush, I'm in the middle of like 4, soon to be 3 other things at the moment15:35
iurygregoryTheJulia, ack15:35
iurygregoryweekend will be fun \o/15:36
derekhTheJulia: I was looking into removing our use of Q_USE_PROVIDERNET_FOR_PUBLIC if thats what your remembering? Have to get back to it, I dropped it when CI moved to bionic to fix those jobs (still one left to get moved to bionic)15:37
TheJuliaderekh: yeah, I think that might be it15:39
TheJuliaderekh: which would allow us to get rid of the q-* service names right?15:40
derekhTheJulia: possibly (not sure), it wasn't what I was aiming to do but can try it once I'm back looking into it15:42
derekhTheJulia: btw about the uefi on bionic, the main problem is that in bionic net.netfilter.nf_conntrack_helper defaults to 0 (so tftp doesn't work) and I'm having trouble figuring out how to change it in the router namespace...15:43
iurygregoryTheJulia, jsut some warnings in q-* logs, and in ir-neutronagt only errors when trying to get the verions15:45
*** naved001 has quit IRC15:51
TheJuliaderekh: I'm hitting it trying to get a v6 job going15:55
TheJuliabtw, my v6 patch returned the right value to the client locally for ipxe to know where to connect to exclusively over v615:55
TheJuliaderekh: it being the service names and various options no longer being available that multitenant stuff is built upon for our CI runs15:56
derekhTheJulia: your hitting the nf_conntrack_helper defaulting to 0?16:01
TheJuliaderekh: sorry, no. things like methods not existing invoked by other methods that get pulled in16:02
TheJuliaits all a mess :(16:02
derekhahh ok, hopefully I'll get back to it soon16:03
*** mkrai has joined #openstack-ironic16:04
TheJuliaI'm going to try and get the v6 job going that way since I'm doing it as a multitenant job to force what we want for networking16:05
*** jrist has joined #openstack-ironic16:09
openstackgerritRuby Loo proposed openstack/ironic stable/queens: Allow disabling instance image cache  https://review.openstack.org/62852716:13
openstackgerritRuby Loo proposed openstack/ironic stable/pike: Allow disabling instance image cache  https://review.openstack.org/62852816:15
NobodyCamGood Morning Ironic'ers16:17
TheJuliagood morning NobodyCam16:18
NobodyCam:)16:18
NobodyCamand ofc16:18
NobodyCamTGIF!16:19
*** munimeha1 has joined #openstack-ironic16:19
dtantsurhey hey NobodyCam16:19
dtantsurthe first TGIF of the year, nice :)16:20
TheJulia\o/16:20
iurygregorymorning NobodyCam =)16:23
rpittauhi NobodyCam :)16:25
*** mkrai has quit IRC16:27
etingofTheJulia, re the irmc job you've mentioned - is it run in CI? -- https://review.openstack.org/#/c/624084/16:28
patchbotpatch 624084 - ironic - Build UEFI-only ISO for UEFI boot - 1 patch set16:28
TheJuliayes16:28
etingofwhat's its name there?16:28
TheJuliahmmm16:29
TheJulialooks like it didn't trigger either time16:29
TheJuliawerid16:29
TheJuliaoh16:29
etingofdoes it need to be backed by a real hw?16:29
TheJuliahmm, it should trigger on common code changes16:29
TheJuliairmc, yes16:30
etingofis it run it 3rd party CI?16:30
etingofah, I found it in some other patch16:32
*** mkrai has joined #openstack-ironic16:32
TheJuliayeah, they file filter so they don't over tax the hardware pool, but common should be triggered upon :\16:33
*** Goneri has joined #openstack-ironic16:34
etingofat any rate, the logs I can see for irmc are not exceptionally useful16:34
TheJuliathen again, common code changes for us are not super... common16:34
TheJuliayeah, I noticed that too :(16:34
etingofno openstack ironic logs, for instance16:34
TheJuliaetingof: would you be up for sending an email on the subject?16:35
etingofsure, who is the right person?16:36
*** e0ne has quit IRC16:36
etingofother than that, would it be completely crazy to make sushy-tools http-pulling an iso image (triggered by vmedia insert call) and booting libvirt domain from it?16:36
TheJuliaI can think of their face right now, but not their name. hshinna is a good person to start out with, he can point you in the right direction. I would just mention that you noticed the irmc job not triggering on ironic/common code changes and a lack of ironic logs being stored from the CI job run16:37
etingofbecause even if irmc CI would work perfectly, it's still not 100% redfish vmedia16:37
TheJuliaetingof: not toooooo crazy... we need the support in ironic..16:38
TheJuliayeah16:38
etingofright, I'm coding vmedia/redfish in ironic16:38
TheJulia<316:38
etingofbut I can't test it, it seems16:38
*** sthussey has quit IRC16:39
TheJuliayeah, makes sense then to add it in sushy to get it to the boot from file behavior16:39
etingofironically, my patches keep failing for all reasons lately, but (!) the actual changes are not being tested at all %-)16:39
* TheJulia wonders if libvirt can essentially bfv16:39
TheJuliaerr16:39
TheJuliaboot from url16:39
*** hjensas|brb is now known as hjensas16:39
etingofmore likely if it can boot from uefi iso over http I think16:40
rpiosoGood morning, ironicers16:41
TheJuliathat would almost functionally be boot from url, but boot from url is supposed to be dhcp driven16:41
mkraiHi, what is the timeline for ironic-spec freeze?16:42
TheJuliakind of sounds like vmedia booting since AIUI ovmf2 doesn't have http support enabled in the pre-built images16:42
TheJuliamkrai: there is no freeze for specs16:42
etingofthat's why I am thinking of booting libvirt domain from locally stored uefi iso16:42
mkraiTheJulia: Thank you16:42
TheJuliamkrai: we do stop landing user visible/impacting changes to the API upon the common freeze and gradually become more and more cautious as time goes on before release16:43
TheJuliaso massive changes at the last minute will very generally not be permitted to merge16:43
mkraiTheJulia: Got it, makes sense.16:44
TheJuliathey get held over until we start merging for the next cycle16:44
openstackgerritDmitry Tantsur proposed openstack/ironic master: [WIP] Allocation API: implementation without HA and take over  https://review.openstack.org/62822316:45
etingofperhaps it's not bfv, but uefi iso boot - the image layouts are completely different16:46
etingofvmedia requires iso16:47
TheJuliayeah, iso is different16:47
TheJuliathe boot from url stuff allows... at least by spec, vmedia ramdisk style16:47
iurygregorybye everyone \o/ happy weekend16:48
TheJuliao/16:48
*** iurygregory has quit IRC16:48
*** sthussey has joined #openstack-ironic16:50
etingofbut wait, with vmedia boot BMC is pulling iso, not the machine16:50
etingofthe machine is booted from emulated CDROM16:51
NobodyCamhey hey dtantsur rpittau16:51
NobodyCamhappy new years16:51
NobodyCamgood morning TheJulia16:51
*** derekh has quit IRC17:04
*** derekh has joined #openstack-ironic17:04
rpittaubye all! have a great weekend o/17:12
*** rpittau has quit IRC17:12
NobodyCamo/17:14
*** mkrai has quit IRC17:37
*** e0ne has joined #openstack-ironic17:43
*** e0ne has quit IRC17:44
*** S4ren has quit IRC17:58
*** derekh has quit IRC18:03
*** gyee has joined #openstack-ironic18:04
*** MattMan has quit IRC18:09
*** MattMan has joined #openstack-ironic18:09
*** dtantsur is now known as dtantsur|afk18:11
dtantsur|afkhave a nice weekend folks18:11
TheJuliayou too18:12
*** baha has joined #openstack-ironic18:13
*** jcoufal has quit IRC18:18
*** jcoufal has joined #openstack-ironic19:00
*** jaypipes has quit IRC20:04
*** lifeless has joined #openstack-ironic20:12
*** mlavalle has joined #openstack-ironic20:22
*** mlavalle has left #openstack-ironic20:22
*** bdodd has quit IRC20:40
*** mjturek has quit IRC20:45
*** jtomasek has quit IRC20:51
openstackgerritJulia Kreger proposed openstack/ironic master: WIP ipv6 job  https://review.openstack.org/61316921:11
*** baha has quit IRC21:12
*** S4ren has joined #openstack-ironic21:33
*** baha has joined #openstack-ironic21:34
*** S4ren has quit IRC21:38
*** trown is now known as trown|outtypewww22:04
*** jcoufal has quit IRC22:10
*** baha has quit IRC22:16
*** mjturek has joined #openstack-ironic22:19
*** mjturek has quit IRC22:20
*** hwoarang has quit IRC23:03
*** hwoarang has joined #openstack-ironic23:04
*** munimeha1 has quit IRC23:12
*** bdodd has joined #openstack-ironic23:19
openstackgerritJulia Kreger proposed openstack/ironic master: WIP ipv6 job  https://review.openstack.org/61316923:34
*** rloo has quit IRC23:40
*** openstack has joined #openstack-ironic23:45
*** ChanServ sets mode: +o openstack23:45
openstackgerritJulia Kreger proposed openstack/ironic master: Add a locking interface  https://review.openstack.org/59919123:51
« Thursday, 2019-01-03 Index Saturday, 2019-01-05 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!