Monday, 2019-09-23

*** Lucas_Gray has quit IRC00:11
*** yamamoto has joined #openstack-infra00:40
*** zhurong has quit IRC00:52
*** jamesmcarthur has joined #openstack-infra01:13
*** zhurong has joined #openstack-infra01:26
*** yamamoto has quit IRC01:33
*** sgw1 has quit IRC01:35
*** rfolco has joined #openstack-infra01:40
*** armax has joined #openstack-infra01:45
*** ykarel|away has joined #openstack-infra01:57
openstackgerritMerged openstack/diskimage-builder master: Install Python 3 libselinux packages for Fedora  https://review.opendev.org/68329502:05
*** rfolco has quit IRC02:08
ianwi'll do a point release, see if we can get f30 working ...02:15
*** dklyle has quit IRC02:33
*** dklyle has joined #openstack-infra02:48
*** jamesmcarthur has quit IRC02:58
*** larainema has joined #openstack-infra03:00
*** jamesmcarthur has joined #openstack-infra03:01
*** jamesmcarthur has quit IRC03:01
*** jamesmcarthur has joined #openstack-infra03:01
*** ricolin has joined #openstack-infra03:20
*** dklyle has quit IRC03:20
*** psachin has joined #openstack-infra03:27
*** ykarel|away has quit IRC03:37
*** markvoelker has joined #openstack-infra03:54
*** ykarel|away has joined #openstack-infra03:57
*** markvoelker has quit IRC03:58
*** ykarel|away is now known as ykarel04:11
*** setuid is now known as zZZzzZZetuid04:16
*** ramishra has joined #openstack-infra04:24
*** dave-mccowan has quit IRC04:48
*** HenryG has quit IRC04:51
*** jamesmcarthur has quit IRC04:52
*** jamesmcarthur has joined #openstack-infra04:53
*** HenryG has joined #openstack-infra04:54
*** slaweq has joined #openstack-infra04:57
*** odicha has joined #openstack-infra04:57
*** slaweq has quit IRC05:05
*** soniya29 has joined #openstack-infra05:07
*** jamesmcarthur has quit IRC05:09
*** jamesmcarthur has joined #openstack-infra05:09
*** slaweq has joined #openstack-infra05:10
*** whoami-rajat has joined #openstack-infra05:11
*** iurygregory has joined #openstack-infra05:12
*** jamesmcarthur has quit IRC05:13
*** odicha has quit IRC05:16
*** odicha has joined #openstack-infra05:18
*** slaweq has quit IRC05:28
*** slaweq has joined #openstack-infra05:30
*** yamamoto has joined #openstack-infra05:31
*** yamamoto has quit IRC05:35
*** yolanda has quit IRC05:42
*** jaosorior has joined #openstack-infra05:43
*** jamesmcarthur has joined #openstack-infra05:44
*** yolanda has joined #openstack-infra05:49
AJaegerianw: looking at grafana, f29 is not working either - want to fix that as well or focus on f30?05:50
*** jamesmcarthur has quit IRC05:51
ianwAJaeger: um, f29 should be ~ ok, although probably unstable.  but yeah, i kicked off a build so the next upload of f30 should *hopefully* allow ansible to run against it05:51
AJaegergrafana showed it in red when I checked the nodepool-dib grafana board yesterday AFAIR05:51
AJaegerso, might be unstable - didn't dig deeper05:52
*** AJaeger has quit IRC05:52
*** AJaeger has joined #openstack-infra05:56
AJaegerianw: http://grafana.openstack.org/d/NcFtwKriz/nodepool-dib-status?orgId=1 is the graph I looked at06:00
*** dpawlik has joined #openstack-infra06:00
*** raukadah is now known as chandankumar06:01
*** rpioso is now known as rpioso|afk06:05
*** jamesmcarthur has joined #openstack-infra06:20
*** yolanda has quit IRC06:22
*** jamesmcarthur has quit IRC06:25
*** florianf has joined #openstack-infra06:25
*** florianf has left #openstack-infra06:25
*** slaweq has quit IRC06:27
*** slaweq has joined #openstack-infra06:34
*** rpittau|afk is now known as rpittau06:40
*** pgaxatte has joined #openstack-infra06:52
*** tesseract has joined #openstack-infra07:00
*** rcernin has quit IRC07:04
*** slaweq has quit IRC07:04
*** slaweq has joined #openstack-infra07:06
*** jamesmcarthur has joined #openstack-infra07:13
*** ccamacho has joined #openstack-infra07:16
*** xek has joined #openstack-infra07:17
*** jamesmcarthur has quit IRC07:19
*** tosky has joined #openstack-infra07:20
*** ramishra has quit IRC07:21
openstackgerritIan Wienand proposed opendev/infra-specs master: [wip] Spec to retire static.openstack.org  https://review.opendev.org/68385207:34
*** e0ne has joined #openstack-infra07:34
*** e0ne has quit IRC07:35
*** ramishra has joined #openstack-infra07:36
*** apetrich has joined #openstack-infra07:39
*** rosmaita has joined #openstack-infra07:41
*** pcaruana has joined #openstack-infra07:41
rosmaitahttps://docs.google.com/presentation/d/1tJmFrL_c5_Rp_u7hZGjv8UQgWsYilFMjUUiHyMnf9WY/edit#slide=id.g4786e183e0_0_1007:41
*** jpena|off is now known as jpena07:42
rosmaitasorry about that07:42
*** rosmaita has left #openstack-infra07:42
*** eharney has joined #openstack-infra07:44
*** slaweq has quit IRC07:47
*** jtomasek has quit IRC07:47
*** jtomasek has joined #openstack-infra07:48
*** ralonsoh has joined #openstack-infra07:59
*** ykarel is now known as ykarel|lunch08:00
*** toabctl has joined #openstack-infra08:02
*** dchen has quit IRC08:03
*** yolanda has joined #openstack-infra08:09
*** xenos76 has joined #openstack-infra08:11
*** jamesmcarthur has joined #openstack-infra08:15
yoctozeptohey infra, something went wrong: https://zuul.opendev.org/t/openstack/build/23299c38822041b99b36abe9444148a0/logs08:16
yoctozepto(I should've probably quoted that)08:16
*** e0ne has joined #openstack-infra08:18
*** iurygregory has quit IRC08:19
*** jamesmcarthur has quit IRC08:20
*** gregoryo has joined #openstack-infra08:21
*** rcernin has joined #openstack-infra08:45
*** iurygregory has joined #openstack-infra08:45
*** derekh has joined #openstack-infra08:49
*** yolanda has quit IRC08:50
*** gregoryo has quit IRC08:55
*** markvoelker has joined #openstack-infra08:57
*** gfidente has joined #openstack-infra08:58
*** ykarel|lunch is now known as ykarel08:58
*** markvoelker has quit IRC09:02
*** dpawlik has quit IRC09:04
*** dpawlik has joined #openstack-infra09:04
*** ociuhandu has joined #openstack-infra09:05
*** dtantsur|afk is now known as dtantsur09:07
*** jamesmcarthur has joined #openstack-infra09:16
openstackgerritJan Kubovy proposed zuul/zuul master: Unify Github/Github Enterpise read timeout  https://review.opendev.org/68336609:18
*** jamesmcarthur has quit IRC09:21
AJaegeryoctozepto: So, https://zuul.opendev.org/t/openstack/build/23299c38822041b99b36abe9444148a0 works but then switching to logs fails. Interesting, hope somebody will be around later to check...09:26
yoctozeptoAJaeger: indeed :-)09:28
*** Lucas_Gray has joined #openstack-infra09:30
*** yamamoto has joined #openstack-infra09:31
frickleryoctozepto: AJaeger: I can see this error in my browser's webconsole: URIError: "Pathname "/t/openstack/build/23299c38822041b99b36abe9444148a0/log/primary/logs/kolla/haproxy/haproxy_latest.%Y%m%d.log.gz" could not be decoded. This is likely caused by an invalid percent-encoding."09:35
fricklerlikely the zuul js doesn't handle that filename well09:35
fricklerplease create a story for zuul if there isn't one already09:35
*** dpawlik has quit IRC09:37
*** dpawlik has joined #openstack-infra09:38
*** Lucas_Gray has quit IRC09:46
*** Lucas_Gray has joined #openstack-infra09:48
*** ricolin has quit IRC09:48
*** ociuhandu has quit IRC09:59
*** ociuhandu has joined #openstack-infra10:00
*** yamamoto has quit IRC10:02
*** ociuhandu has quit IRC10:06
yoctozeptofrickler: thx, done: https://storyboard.openstack.org/#!/story/200659910:07
*** jamesmcarthur has joined #openstack-infra10:17
*** jamesmcarthur has quit IRC10:23
AJaegeryoctozepto: but you should fix the filename as well, it looks really broken10:24
yoctozeptoAJaeger: indeed it does, added checking to my TODO already10:25
*** zZZzzZZetuid is now known as setuid10:25
*** yamamoto has joined #openstack-infra10:32
fricklerianw: AJaeger: F29 and possibly other builds seems to be failing with a missing "openstack-repos" element: http://paste.openstack.org/show/778858/ . This doesn't even seem to retain a log file for those builds.10:34
*** zigo_ has joined #openstack-infra10:35
AJaegermmh, openstack-repos is include in nodepool/nodepool.yaml everywhere as far as I can see.10:39
AJaegerhope ianw has an idea ^10:41
*** ociuhandu has joined #openstack-infra10:41
*** prometheanfire has quit IRC10:41
*** prometheanfire has joined #openstack-infra10:42
*** snapiri has quit IRC10:43
frickleryeah, maybe one needs to execute dib from a different directory, so the latter test may be misleading. still the builds fail with errors and the build log that is mentioned in the nodepool log doesn't seem to exist10:43
*** ociuhandu has quit IRC10:46
*** Lucas_Gray has quit IRC10:46
*** Lucas_Gray has joined #openstack-infra10:48
*** ociuhandu has joined #openstack-infra10:50
*** soniya29 has quit IRC10:53
*** ociuhandu has quit IRC10:54
*** dpawlik has quit IRC10:57
*** jaosorior has quit IRC10:58
*** dchen has joined #openstack-infra11:00
*** pgaxatte has quit IRC11:03
*** dpawlik has joined #openstack-infra11:04
*** dave-mccowan has joined #openstack-infra11:08
*** dpawlik has quit IRC11:09
*** snapiri has joined #openstack-infra11:10
*** yamamoto has quit IRC11:11
*** dpawlik has joined #openstack-infra11:14
*** shachar has joined #openstack-infra11:15
*** snapiri has quit IRC11:16
*** soniya29 has joined #openstack-infra11:16
*** ociuhandu has joined #openstack-infra11:19
*** jamesmcarthur has joined #openstack-infra11:19
*** dave-mccowan has quit IRC11:21
*** dave-mccowan has joined #openstack-infra11:23
*** ociuhandu has quit IRC11:23
*** jamesmcarthur has quit IRC11:24
*** ccamacho has quit IRC11:24
*** yamamoto has joined #openstack-infra11:27
*** jpena is now known as jpena|lunch11:28
*** yamamoto has quit IRC11:29
*** yamamoto has joined #openstack-infra11:29
*** jbadiapa has quit IRC11:30
*** lucasagomes has joined #openstack-infra11:31
*** rh-jelabarre has joined #openstack-infra11:31
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Add more languages for build-releasenotes  https://review.opendev.org/68392311:35
AJaegerfrickler: I noticed this reviewing the designate-dashboard releasenotes ^11:35
*** rh-jelabarre has quit IRC11:45
*** rh-jelabarre has joined #openstack-infra11:45
*** zul has joined #openstack-infra11:46
*** rcernin has quit IRC11:49
fricklerAJaeger: in the reno build I see "cs" instead of "cz", not sure how that matches up in the end or whether that is the correct code for czech?11:50
*** pgaxatte has joined #openstack-infra11:52
*** ociuhandu has joined #openstack-infra11:57
*** goldyfruit_ has quit IRC11:57
*** ociuhandu has quit IRC11:58
*** ociuhandu has joined #openstack-infra11:59
*** rledisez has joined #openstack-infra11:59
fricklerseems wikipedia also says "cs" https://en.wikipedia.org/wiki/Czech_language11:59
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Add more languages for build-releasenotes  https://review.opendev.org/68392312:01
AJaegerfrickler: indeed, cs is right - for language, not country12:01
ianwfricker: that element comes from project-config ... where did you see that failure?12:01
*** iurygregory has quit IRC12:01
fricklerianw: the top failure is in the log on nb02. the second is when I try to reproduce manually, which I possibly did wrong12:02
fricklerianw: both referring to my paste http://paste.openstack.org/show/778858/12:02
AJaegerfrickler: see also https://translate.openstack.org/iteration/view/designate-dashboard/master?dswid=5162 (sort by percent translated)12:02
*** markvoelker has joined #openstack-infra12:03
ianwfrickler: interesting, the top looks like a nodepool problem?  the lower, yeah i'd say you need to add project-config/nodepool/elements to the elements path12:03
ianwgotta head off sorry, but will look in morning if no progress, thanks12:04
AJaegerfrickler: https://review.opendev.org/683938 for designate-dashboard. I was irritated by the bad formatting, this should fix it.12:06
AJaegerOh, mugsie is ultra-fast ;)12:07
mugsiei happened to click into the dashboard review folder in my mail client :)12:08
fricklerAJaeger: I was still trying to parse what you did there, but yeah ;) this will also conflict with what I did in https://review.opendev.org/683925 , but I can fix that later12:08
AJaegerfrickler: want to take my change in and I'll abandon?12:09
AJaegerfrickler: or shall I update yours?12:10
fricklerAJaeger: no, just leave things as they are, I'll update mine later12:10
AJaegerok12:10
fricklerAJaeger: luckily those jobs are pretty fast, no tempest 2h beasts :D12:11
AJaeger\o/12:11
*** rfolco has joined #openstack-infra12:12
*** weshay has joined #openstack-infra12:15
*** weshay is now known as weshay|rover12:16
*** jamesmcarthur has joined #openstack-infra12:20
*** jamesmcarthur has quit IRC12:25
*** rpioso|afk is now known as rpioso12:28
*** rlandy has joined #openstack-infra12:29
*** rascasoft has joined #openstack-infra12:31
*** rcernin has joined #openstack-infra12:33
lennybHi, following #link https://docs.openstack.org/infra/system-config/third_party.html   gerrit plugin configuration I am see that my CI comments under "Code-Review" section and not under "Zuul check" .  Is there an option for CI using rest api or gerrit plugin to comment under "Zuul check" section like all the others. ( I am trying to use it as a temporal phase during transition to zuul3 )12:35
*** lbragstad_ is now known as lbragstad12:39
*** psachin has quit IRC12:39
openstackgerritJan Kubovy proposed zuul/zuul master: Unify Github/Github Enterpise read timeout  https://review.opendev.org/68336612:45
*** jamesmcarthur has joined #openstack-infra12:45
*** jpena|lunch is now known as jpena12:47
clarkblennyb: the voting is part of your pipeline config in zuul12:48
clarkblennyb: you want to be voting on the "verified" category instead of the code-review category12:48
*** jcoufal has joined #openstack-infra12:50
clarkblennyb: https://opendev.org/openstack/project-config/src/branch/master/zuul.d/pipelines.yaml#L47 is how we do it for opendev's zuul12:50
*** iurygregory has joined #openstack-infra12:51
*** aaronsheffield has joined #openstack-infra12:53
fricklerlennyb: can you show a sample patch where you see this? the issue might just be that you need to name your CI account properly12:54
*** haleyb has joined #openstack-infra12:55
*** ricolin has joined #openstack-infra12:59
openstackgerritJan Kubovy proposed zuul/zuul master: Unify Github/Github Enterpise read timeout  https://review.opendev.org/68336612:59
*** e0ne_ has joined #openstack-infra12:59
*** goldyfruit_ has joined #openstack-infra13:01
*** e0ne has quit IRC13:01
*** larainema has quit IRC13:01
lennybclarkb: I want to vote on "Zuul check" . !!! I am NOT using zuul at the moment,  ex #link http://paste.openstack.org/show/778943/13:02
lennybfrickler; ^13:03
*** rlandy is now known as rlandy|mtg13:03
corvusi'm going to restart the zuul scheduler13:03
fricklerlennyb: see item 2. in https://docs.openstack.org/infra/system-config/third_party.html#creating-a-service-account , your CI name has to have three pieces, like: "Mellanox Whatever CI"13:06
*** yamamoto has quit IRC13:06
fungilennyb: where it says "zuul check" there, that's the ci which uses the account name "zuul" reporting on the status from its "check" pipeline. if you look at a project with lots of third-party ci systems reporting on changes you'll see the distinction: https://review.opendev.org/68041313:08
fungifor example, that change has other ci systems like "Intel NFV CI" reporting a "check" result13:10
*** ociuhandu has quit IRC13:10
fricklerfungi: oh, I assumed the issue was whether the CI results appear in that table at all, but the mellanox CI does show up there13:10
fungiso your results won't say "zuul check" but maybe "whatever your ci check" instead13:10
*** ociuhandu has joined #openstack-infra13:10
*** jaosorior has joined #openstack-infra13:11
*** e0ne has joined #openstack-infra13:12
*** e0ne_ has quit IRC13:12
corvusShrews: i restarted the zuul scheduler with current master13:14
fricklerlennyb: so in the example in your paste, you would need to add formatted results like in the last box in this section https://docs.openstack.org/infra/system-config/third_party.html#posting-result-to-gerrit13:15
corvus#status log restarted zuul scheduler on commit 80ef01534d90751aa2d9cd3bf4a356fca292bed813:15
openstackstatuscorvus: finished logging13:15
*** mriedem has joined #openstack-infra13:16
*** ociuhandu has quit IRC13:17
lennybfungi,frickler: thanks I see your point. So from the community perspective commenting is enough, it does not have to be under "Check" section like "Arista CI" in fungi's example , but if I want it to be there I need to use a formated message like frickler proposed13:21
fungi"check" is just a pipeline name13:24
*** beekneemech is now known as bnemec13:24
fungiin those examples13:24
Shrewscorvus: cool. basic autohold commands seem to work now: http://paste.openstack.org/show/778944/13:24
corvusShrews: \o/13:25
*** sthussey has joined #openstack-infra13:26
corvusinfra-root: ^ fyi snazzy new autohold tools13:26
fricklerShrews: so we should now be able to see held nodes from zuul directly? that'd be really cool13:26
Shrewsfrickler: yes. and tristanC is working on further improvements to organizing those in the output13:27
corvusfrickler: yes, and delete -- so only need to log into one host13:27
Shrewsfyi, https://review.opendev.org/68246613:27
frickleroh, with delete that's even better, great13:27
*** ociuhandu has joined #openstack-infra13:27
fungiShrews: awesome!13:28
*** goldyfruit___ has joined #openstack-infra13:30
*** dchen has quit IRC13:30
*** ociuhandu has quit IRC13:33
*** goldyfruit_ has quit IRC13:33
openstackgerritMerged zuul/zuul-registry master: Add README, license, and Zuul config  https://review.opendev.org/68342613:33
*** whoami-rajat has quit IRC13:33
openstackgerritJan Kubovy proposed zuul/zuul master: Report retried builds in a build set via mqtt.  https://review.opendev.org/63272713:34
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Initial implementation  https://review.opendev.org/68395813:36
Shrewsfrickler: oh, also, autohold requests persist across scheduler restarts now, too13:37
*** dpawlik has quit IRC13:37
*** eernst has quit IRC13:40
openstackgerritJan Kubovy proposed zuul/zuul master: Add support for smart reconfigurations  https://review.opendev.org/65211413:43
*** dklyle has joined #openstack-infra13:44
*** yolanda has joined #openstack-infra13:44
*** jamesmcarthur has quit IRC13:44
*** rcernin has quit IRC13:45
*** diga has joined #openstack-infra13:45
*** yamamoto has joined #openstack-infra13:47
*** zhurong has quit IRC13:47
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: Add unit tests for tox_instalL_sibling_packages  https://review.opendev.org/68396313:48
*** dklyle has quit IRC13:50
*** bdodd has joined #openstack-infra13:50
*** signed8bit has joined #openstack-infra13:50
*** yolanda has quit IRC13:52
*** redrobot has quit IRC13:53
*** yolanda has joined #openstack-infra13:56
weshay|roverpabelanger, re: https://review.opendev.org/#/c/683240/13:57
*** yamamoto has quit IRC13:57
*** rlandy|mtg is now known as rlandy14:00
*** jcoufal_ has joined #openstack-infra14:00
*** ociuhandu has joined #openstack-infra14:00
*** slaweq has joined #openstack-infra14:01
*** yolanda has quit IRC14:03
*** jcoufal has quit IRC14:04
*** ociuhandu has quit IRC14:04
*** Guest30550 has joined #openstack-infra14:05
*** Guest30550 is now known as redrobot14:08
AJaegerShrews, corvus , we have 4 autohold nodes - are those still needed? http://grafana.openstack.org/d/rZtIH5Imz/nodepool?orgId=114:10
*** jpena is now known as jpena|brb14:11
lennybfungi,frickler,clarkb: thanks a lot. works like a magic.14:11
fricklerAJaeger: two are mine, I'll delete those, other two are ianw's14:16
openstackgerritMerged zuul/zuul-jobs master: Fix tox_install_sibling_packages.py for py3  https://review.opendev.org/68380514:18
*** yolanda has joined #openstack-infra14:19
ShrewsAJaeger: i have no idea14:20
fricklerianw: this fresh f29 build on nb01 seems to have worked fine, maybe there's something wrong about nb02 specifically https://nb01.openstack.org/fedora-29-0000054155.log14:21
*** iurygregory has quit IRC14:23
*** yamamoto has joined #openstack-infra14:25
openstackgerritItxaka Serrano Garcia proposed opendev/system-config master: Revert to rit.edu mirror for Suse  https://review.opendev.org/68397814:26
*** yamamoto has quit IRC14:28
*** dklyle has joined #openstack-infra14:31
*** soniya29 has quit IRC14:36
*** pgaxatte has quit IRC14:38
openstackgerritJan Kubovy proposed zuul/zuul master: Add --check-config option to zuul scheduler  https://review.opendev.org/54216014:40
*** tosky has quit IRC14:40
*** kopecmartin is now known as kopecmartin|off14:44
*** ociuhandu has joined #openstack-infra14:51
*** jamesmcarthur has joined #openstack-infra14:54
*** yolanda has quit IRC14:54
*** ociuhandu has quit IRC14:57
*** gfidente has quit IRC15:02
*** ociuhandu has joined #openstack-infra15:04
*** whoami-rajat has joined #openstack-infra15:04
*** ociuhandu has quit IRC15:04
*** diablo_rojo has joined #openstack-infra15:04
*** jpena|brb is now known as jpena15:05
*** ociuhandu has joined #openstack-infra15:05
weshay|roverpabelanger, howdy.. just checking in re: https://review.opendev.org/#/c/683240/15:05
weshay|roveradd config for quay15:05
*** pcaruana has quit IRC15:05
weshay|roverany reviews here would be helpful :) thanks in advance15:06
*** eharney has quit IRC15:06
*** ociuhandu has quit IRC15:08
*** ykarel is now known as ykarel|afk15:08
*** ociuhandu has joined #openstack-infra15:08
*** yamamoto has joined #openstack-infra15:09
*** udesale has joined #openstack-infra15:09
*** roman_g has joined #openstack-infra15:11
*** ociuhandu has quit IRC15:13
*** timburke_ is now known as timburke15:15
donnydDo we have a good method to benchmark providers?15:16
*** ociuhandu has joined #openstack-infra15:19
clarkbdonnyd: not really15:20
openstackgerritTobias Henkel proposed zuul/zuul master: Support emitting warnings via zuul_return  https://review.opendev.org/65152615:20
clarkbproviders have heterogenous servers and we have no good way to scheduling to the different varieties and testing them all15:20
clarkbdonnyd: fwiw I hvae no interest in doing what you've proposed on the mailing list15:20
clarkbwe intentionally avoid scheduling based on parameters like that to avoid 1) everyone deciding their jobs must run on the fast cloud and 2) to be resilient to outages and othe rprovider needs15:20
donnydlol well ok then15:21
clarkbour software can run on these providers it has for years. Unfortunately more recently its super bloated with memory which leads to swapping which lead sto massive performance issues15:21
clarkbI've asked everyone in popsntack to fix this and haven't seen anyone move a finger15:21
*** ociuhandu has quit IRC15:21
*** ociuhandu has joined #openstack-infra15:22
clarkb(I was actually thinking about this recently and wondered if we could stop deploying etcd by default in devstack to save some memory becuase we don't actually use etcd anywhere as far as I know)15:22
*** KeithMnemonic has joined #openstack-infra15:22
donnydI think it's a bit unfortunate to not take advantage of scheduling jobs on the providers that are best at doing X task.15:22
*** mattw4 has joined #openstack-infra15:22
donnydAnd the idea would only be to prefer not require X job for Y provider15:24
donnydYea if there are things being turned on that are not strictly necessary for the job than that is a waste15:24
clarkbdonnyd: I just think we should fix the broken software first15:25
clarkbparticularly because we are consuming that broken software and its brokeness leads to fewer resources15:25
clarkbwe use half our vexxhost quota because cinder thinks we are using 2x the volumes we actually use15:25
clarkband run into quota limits15:25
*** dtantsur is now known as dtantsur|afk15:26
donnydI do have no disagreements with what you are saying here15:26
*** mattw4 has quit IRC15:26
clarkbif we can get to a happy place where the software is running well and doesn't swap and we still have problems then we start considering other changes15:26
clarkbbut if we know the software is broken as is we start with fixing htat15:26
donnydbut if vexxhost, limestone and FN are 3x faster at doing say a docs job.. doesn't it make sense to *try* and schedule them there15:26
clarkbno beause when limstone and vexxhost have an outage (as we currently do) those jobs will then start running elsewhere and if we havne't testing it works there in a month there is a good chance they fail (possibly due to timeouts)15:27
*** ykarel|afk is now known as ykarel15:27
donnydclarkb: you are total right - I am just trying to come up with ideas that can help15:27
donnydWell that is what it would need to prefer - not require15:28
clarkbI guess I don't really know how you'd balance that.15:29
*** jamesmcarthur has quit IRC15:30
clarkbI guess to tl'dr this I've been working really hard the last few weeks to proactively find and report bugs in openstack and suggest fixes as I am able. Doing this because I know feature freeze is often painful as people try to get code in last minute and any gate flakyness leads to these problems15:32
clarkbinstead of being taken up on these offers to help and addressing real problems we keep seeing people suggest we rewrite nodepool and zuul15:32
donnydI am not sure how it would work either - we would have to work it out . Maybe like a tag in the jobs or something that says what its bound by15:32
clarkbI personally don't find that productive because the broken sofwtare is the softwrae being tested15:32
clarkbwe should fix that and it shouldn't be controversial that I say it15:33
donnydclarkb: you are correct - and I agree with you15:33
donnydThese are just thoughts to work on down the road15:33
*** cmorpheus is now known as cmurphy15:34
Shrewsclarkb: do you have an example of the suggestion to rewrite np/zuul? Is that in a ML post?15:34
clarkbShrews: TheJulia has suggseted we give projects a "kill the jobs and report their logs" button15:34
clarkbShrews: donnyd has just suggsetd we allow job hints to prefer one provider of nodes over another15:35
clarkbthese ideas may not be bad, but we can't properly evaluate them (in my opinion) if we are doing so as an optimiation for the software is broken15:36
*** dklyle has quit IRC15:36
clarkbwe should optimize for the software is good because that should be the gola of our developers15:36
*** dklyle has joined #openstack-infra15:36
*** rkukura has joined #openstack-infra15:36
clarkbShrews: donnyd also suggested that we do per project prioritization of node assignments (we do it per pipeline queue today)15:37
*** mattw4 has joined #openstack-infra15:37
donnydclarkb: I didn't know we already did that part15:37
*** yamamoto has quit IRC15:37
donnydShrews: http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009595.html15:39
*** yolanda has joined #openstack-infra15:40
clarkbreally I odn't mind the suggestions I just think that we can't properly implement and evaluate them while hair is on fire and we are trying to merge a bunch of last minute features. We should be able to consider these things under normal circumstances and to get there we need to improve our test reliability15:40
clarkbwhich is why I keep pushing that direction as the first step15:40
donnydclarkb: yea this is a stressful time I am sure each cycle. I don't think anyone is pointing any fingers at infra (unless I missed someone pointing fingers). I am also sure we get the same requests each and every cycle for the same thing15:42
donnydJust food for thought is all15:42
clarkbdonnyd: I don't think its explicit finger pointing, but when instead of the reaction being everyone goes and fixes the flaky software we get a bunch of suggestions for work for the infra team it feels that way15:42
pabelangeralso, different clouds, having different speeds also do a good job at exposing problems too in the software.15:43
donnydYea I can definitely understand that15:43
pabelangerbut agree, don't want to home jobs to a specific cloud... it is painful when that cloud is down15:43
donnydWell that is why the job would have to prefer not require15:43
donnydif the cloud isn't available or is already loaded then it would run as normal15:44
donnydso like a best effort sort of thing - but it would likely get jobs through the pipelines faster15:44
donnydand the jobs would still need to be suitable for all providers because you wouldn't know where it will run15:45
clarkbpabelanger: the timing issue is a really good point particularly for openstack15:45
openstackgerritJames E. Blair proposed zuul/zuul master: Wait for poll twice in tests  https://review.opendev.org/68398915:45
clarkbbecause everything is async having different speeds is really good for shaking out problems15:45
donnydThe idea is just to best effort try to schedule on a particular provider for a particular workload15:46
*** ykarel is now known as ykarel|afk15:47
*** efried has joined #openstack-infra15:48
*** dave-mccowan has quit IRC15:49
AJaegerconfig-core, trivial change for translated releasenotes - please review https://review.opendev.org/68392315:49
*** rpittau is now known as rpittau|afk15:52
*** ociuhandu has quit IRC15:56
*** ociuhandu has joined #openstack-infra15:57
*** eernst has joined #openstack-infra15:58
donnydShrews: I am thinking all it would really take in NP is to have a method that allows you to associate a trait with a provider15:59
donnydso maybe something like cloud X is really good at Y, try your best to schedule there16:00
*** ociuhandu has quit IRC16:01
donnydBut also we would have to come up with a way to profile a cloud's performance - which is something else entirely16:01
*** ociuhandu has joined #openstack-infra16:02
*** eernst has quit IRC16:02
*** mattw4 has quit IRC16:03
*** Lucas_Gray has quit IRC16:04
*** mattw4 has joined #openstack-infra16:05
*** pcaruana has joined #openstack-infra16:06
*** Lucas_Gray has joined #openstack-infra16:06
donnydclarkb: I also think that maybe everyone comes up with ideas on how to make things better at time of the cycle because it gets pushed to the front of their minds - the rest of the cycle they are probably heads down in their own code16:06
*** ociuhandu has quit IRC16:06
donnydI think everyone sees that infra does a really fantastic job of keeping this *whole thing* moving forward16:08
*** yamamoto has joined #openstack-infra16:09
*** ociuhandu has joined #openstack-infra16:10
*** slaweq has quit IRC16:10
clarkbya that us why I try to proactively call it out periodically. But you are right,unless you are under a deadline the merge problems are far less annoying16:10
*** eernst has joined #openstack-infra16:11
*** ykarel|afk is now known as ykarel|away16:11
Shrewsdonnyd: i think saying "all it would really take" is trivializing the changes that would be required to nodepool. but i keep a growing list of requested nodepool features for a future rewrite. i'll add that there16:11
*** e0ne has quit IRC16:16
*** yamamoto has quit IRC16:20
donnydShrews: yea I probably could have worded that a little better16:21
openstackgerritWitold Bedyk proposed openstack/project-config master: Set Review-Priority label sticky  https://review.opendev.org/68399416:23
donnydShrews: if you think its an idea worth looking into I can write it up in an RFE for zuul/NP16:23
*** Lucas_Gray has quit IRC16:25
*** eernst has quit IRC16:26
Shrewsdonnyd: nah16:27
*** mattw4 has quit IRC16:28
donnydokie dokey16:29
*** eernst has joined #openstack-infra16:29
*** eernst has quit IRC16:29
*** Lucas_Gray has joined #openstack-infra16:29
*** ykarel|away has quit IRC16:36
openstackgerritColleen Murphy proposed openstack/project-config master: Update Grafana dashboard for openSUSE  https://review.opendev.org/68373816:46
*** pcaruana has quit IRC16:46
*** mattw4 has joined #openstack-infra16:48
donnydOn a completely unrelated topic it would seem that moving to Openstack routers for FN has made some positive impacts for performance16:49
donnydLooking at the dashboard we now regularly hit the 500Mb/s range in downloads from the mirror node16:49
donnydAnd traffic regularly hits 3GB/s in the network node whereas before it was less than half that16:51
weshay|roverany review love for https://review.opendev.org/#/c/683240/ ?16:53
*** Goneri has joined #openstack-infra16:54
AJaegerweshay|rover: it helps a lot if you say which repo and what kind of change, so that the right people can look at it ...16:57
*** ykarel|away has joined #openstack-infra16:57
weshay|roveropendev/system-config adding proxy config for the quay container registry16:58
*** udesale has quit IRC17:00
*** b3nt_pin is now known as beagles17:00
pabelangerweshay|rover: sorry, at ansiblefest this week. Can look, but won't have time to help deploy, would suggest working with other infra-root to bring it online. That said, I know other roots are at ansiblefest too17:05
*** mattw4 has quit IRC17:11
*** ykarel|away has quit IRC17:12
weshay|roverk17:12
*** whoami-rajat has quit IRC17:14
*** jpena is now known as jpena|off17:16
openstackgerritGary W. Smith proposed openstack/project-config master: Add airshipui project  https://review.opendev.org/67980617:20
*** derekh has quit IRC17:24
*** jamesmcarthur has joined #openstack-infra17:25
*** ociuhandu has quit IRC17:27
*** ociuhandu has joined #openstack-infra17:28
*** ociuhandu has quit IRC17:28
*** jamesmcarthur_ has joined #openstack-infra17:28
*** ociuhandu has joined #openstack-infra17:28
*** jamesmcarthur has quit IRC17:32
*** jamesmcarthur_ has quit IRC17:34
*** jamesmcarthur has joined #openstack-infra17:34
*** eernst has joined #openstack-infra17:34
*** ralonsoh has quit IRC17:35
*** eernst has quit IRC17:35
*** jamesmcarthur has quit IRC17:39
*** jamesmcarthur has joined #openstack-infra17:39
*** e0ne has joined #openstack-infra17:42
*** jtomasek has quit IRC17:42
*** ykarel has joined #openstack-infra17:43
openstackgerritMerged zuul/zuul-website master: Add faq entry about kubernetes and docker images  https://review.opendev.org/68074217:45
*** mattw4 has joined #openstack-infra17:46
*** jamesmcarthur has quit IRC17:59
openstackgerritMatt Riedemann proposed openstack/devstack-gate master: Fix MySQL log collection  https://review.opendev.org/68404217:59
*** e0ne has quit IRC18:01
openstackgerritMatt Riedemann proposed openstack/devstack-gate master: Fix MySQL log collection  https://review.opendev.org/68404218:01
*** ricolin has quit IRC18:02
clarkbweshay|rover: cloudnull for quay.io, it has inherited the cache override from dockerhub. That is safe for dockerhub because all objects are sha256 identified18:02
clarkbknowing what little I know of the registry protocol I imagine that quay must do the same thing so this override is safe ther etoo18:02
clarkbdo you know?18:03
weshay|rovercloudnull, may know ^18:03
cloudnullfrom what I can tell quay does the same thing. so we might even be able to use the same cache?18:04
cloudnullEmilienM ^18:04
*** igordc has joined #openstack-infra18:06
clarkbcloudnull: it is the same cache, but objects are identified by the whole url aiui18:06
clarkbso we won't actually overlap cache entries18:06
cloudnull++ I believe that is true, there should be no collisions.18:08
* cloudnull realizes I'm hanging myself with "Should"18:09
openstackgerritMerged zuul/zuul master: Fix weak dependencies to work with child_jobs  https://review.opendev.org/67793618:09
clarkbmostly we need to make sure its safe to override apache's normal cache rules because thing sare uniquely identified18:09
clarkband if they change then they get a new hash18:09
cloudnullclarkb - https://quay.io/repository/tripleomaster/centos-binary-aodh-api/manifest/sha256:6b426625a4b764b6e1ac5bcd8fdbcb1d5fb1c1397e61017d52aec5fe3d17ce8218:11
*** goldyfruit___ has quit IRC18:11
cloudnullyes, everything is unique18:11
clarkbperfect18:11
cloudnullso we should be perfectly safe18:11
clarkbyup agreed18:12
* cloudnull had to double check myself 18:12
corvusclarkb: cloudnull: the manifest endpoint also accepts tag names (eg https://quay.io/repository/tripleomaster/centos-binary-aodh-api/manifest/latest)18:13
clarkbcorvus: neat. In that case we'll cache thatdata for 24 hours18:14
clarkbwhich may be ok?18:14
clarkbnote we likely have this issue with the dockerhub proxy too18:14
corvusyeah, my comment is re protocol in general; sholud apply to every registry18:14
corvusi think that means that there could be up to 24 hour delay before we notice an updated tag18:15
clarkbya18:15
corvus(but everything re the old values sholud still work)18:15
*** Lucas_Gray has quit IRC18:16
openstackgerritDavid Shrewsbury proposed zuul/nodepool master: Do not overwrite image upload ZK data on delete  https://review.opendev.org/68185718:16
openstackgerritDavid Shrewsbury proposed zuul/nodepool master: Do not overwrite image upload ZK data on delete  https://review.opendev.org/68185718:18
*** Goneri has quit IRC18:18
*** ociuhandu has quit IRC18:19
*** diga has quit IRC18:28
*** jtomasek has joined #openstack-infra18:35
*** goldyfruit___ has joined #openstack-infra18:41
*** e0ne has joined #openstack-infra18:43
*** slaweq has joined #openstack-infra18:44
*** e0ne has quit IRC18:45
*** igordc has quit IRC18:51
*** e0ne has joined #openstack-infra18:51
*** ykarel has quit IRC19:01
*** goldyfruit___ has quit IRC19:01
*** lucasagomes has quit IRC19:06
*** lucasagomes has joined #openstack-infra19:06
*** slaweq has quit IRC19:11
*** lucasagomes has quit IRC19:14
*** mgoddard has quit IRC19:17
*** mgoddard has joined #openstack-infra19:20
*** michael-beaver has joined #openstack-infra19:20
openstackgerritBrian Haley proposed openstack/project-config master: Update Networking OVN failure rate dashboard  https://review.opendev.org/68412419:21
*** goldyfruit has joined #openstack-infra19:23
*** jcoufal_ has quit IRC19:30
*** jcoufal has joined #openstack-infra19:30
*** pcaruana has joined #openstack-infra19:43
*** diablo_rojo has quit IRC19:45
*** e0ne has quit IRC19:46
*** dave-mccowan has joined #openstack-infra19:50
*** diablo_rojo has joined #openstack-infra19:57
*** derekh has joined #openstack-infra20:01
*** markvoelker has quit IRC20:07
*** pcaruana has quit IRC20:12
*** rosmaita has joined #openstack-infra20:22
EmilienMclarkb: ack, thx20:30
EmilienMcloudnull: thx for the ping. I didn't know that20:30
*** Lucas_Gray has joined #openstack-infra20:31
*** diablo_rojo has quit IRC20:31
*** jtomasek has quit IRC20:43
*** diablo_rojo has joined #openstack-infra20:44
openstackgerritMerged zuul/zuul master: Wait for poll twice in tests  https://review.opendev.org/68398921:01
*** xenos76 has quit IRC21:03
*** jcoufal has quit IRC21:04
*** mattw4 has quit IRC21:05
*** mattw4 has joined #openstack-infra21:05
*** xek has quit IRC21:05
*** yamamoto has joined #openstack-infra21:15
paladoxmordred we've upgraded our bazel version! (we couldn't build 2.15.17 without it)21:16
*** tesseract has quit IRC21:23
*** Wryhder has joined #openstack-infra21:27
*** igordc has joined #openstack-infra21:27
*** Lucas_Gray has quit IRC21:27
*** Wryhder is now known as Lucas_Gray21:28
*** bdodd has quit IRC21:28
*** mattw4 has quit IRC21:29
*** mattw4 has joined #openstack-infra21:30
*** goldyfruit has quit IRC21:32
*** rcernin has joined #openstack-infra21:38
*** rcernin has quit IRC21:40
*** rcernin has joined #openstack-infra21:40
*** goldyfruit has joined #openstack-infra21:43
*** eernst has joined #openstack-infra21:46
*** eernst has quit IRC21:51
*** markvoelker has joined #openstack-infra21:53
*** mriedem has quit IRC21:55
openstackgerritMerged opendev/system-config master: Add proxy for quay  https://review.opendev.org/68324021:55
*** rosmaita has quit IRC21:58
*** markvoelker has quit IRC21:58
*** yamamoto has quit IRC22:19
*** mattw4 has quit IRC22:22
openstackgerritKerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters  https://review.opendev.org/68320522:26
*** rfolco has quit IRC22:29
*** rfolco has joined #openstack-infra22:37
*** adriant has quit IRC22:39
*** iokiwi has joined #openstack-infra22:42
*** dchen has joined #openstack-infra23:00
*** tkajinam has joined #openstack-infra23:01
rm_workin the vars section of a zuul job config, does anything I put there basically get made into ALL_CAPS_FORM and set during the devstack run?23:05
*** rh-jelabarre has quit IRC23:06
rm_worktrying to convert an old playbook that used ./safe-devstack-vm-gate-wrap.sh23:08
rm_worktips appreciated! I saw there was a migrating doc but it seemed to cover mostly creating the jobs.yaml, though I might have missed it scanning... will re-read that now23:11
*** adriant has joined #openstack-infra23:12
rm_workI've got a job that's inheriting from legacy-dsvm-base but would like it to use whatever the newest-fangled devstack-base is, and stop running that oldschool devstack-vm-gate-wrap thing in a playbook, lol23:14
rm_workchecking out https://review.opendev.org/#/c/500365/ per the docs :)23:15
*** Lucas_Gray has quit IRC23:26
*** mattw4 has joined #openstack-infra23:29
rm_workif someone has a sec, could you look at https://review.opendev.org/#/c/683285/6/.zuul.yaml and the run/post jobs I removed, and tell me if you think the stuff I added covers all of that? (I think the log-copy in the post.yaml is just... standard?)23:29
*** aaronsheffield has quit IRC23:33
*** mattw4 has quit IRC23:43
*** sthussey has quit IRC23:45
*** michael-beaver has quit IRC23:50

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!