Tuesday, 2020-09-22

*** eharney has quit IRC00:04
*** artom has quit IRC00:07
*** eharney has joined #openstack-infra00:07
*** hongbin has quit IRC00:10
*** zxiiro has quit IRC00:17
*** armax has quit IRC00:50
*** tbachman has joined #openstack-infra00:53
*** Goneri has quit IRC00:59
*** hamalq has quit IRC01:01
gmannfungi: ack,01:12
*** gyee has quit IRC01:22
*** eharney has quit IRC01:41
*** eharney has joined #openstack-infra01:54
*** hamalq has joined #openstack-infra02:25
*** ramishra has quit IRC02:29
*** zzzeek has quit IRC02:51
*** zzzeek has joined #openstack-infra02:54
*** hongbin has joined #openstack-infra03:00
*** ramishra has joined #openstack-infra03:15
*** ramishra has quit IRC03:48
*** ramishra has joined #openstack-infra03:55
*** ykarel|away has joined #openstack-infra04:05
*** ykarel|away is now known as ykarel04:10
*** eharney has quit IRC04:10
*** eharney has joined #openstack-infra04:23
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-infra04:33
*** ykarel has quit IRC04:44
*** ykarel has joined #openstack-infra04:46
*** eharney has quit IRC05:03
*** rcernin has quit IRC05:14
openstackgerritMerged openstack/project-config master: Linaro US: Add a 16GB RAM label for CentOS 8  https://review.opendev.org/75303605:22
*** jtomasek has joined #openstack-infra05:23
*** hongbin has quit IRC05:26
*** Ajohn has quit IRC05:26
openstackgerritMerged openstack/project-config master: Switch grafana nodepool to Fedora 32  https://review.opendev.org/75274505:28
*** ykarel has quit IRC05:33
*** rcernin has joined #openstack-infra05:34
*** xek has joined #openstack-infra05:44
*** Ajohn has joined #openstack-infra05:44
*** lmiccini has joined #openstack-infra05:45
openstackgerritCarlos Goncalves proposed openstack/project-config master: CentOS 8 Stream initial deployment  https://review.opendev.org/73479105:50
*** ysandeep|away is now known as ysandeep05:50
*** zzzeek has quit IRC05:54
*** matt_kosut has joined #openstack-infra05:54
*** zzzeek has joined #openstack-infra05:54
*** Ajohn has quit IRC05:56
*** vishalmanchanda has joined #openstack-infra05:58
*** ykarel has joined #openstack-infra06:08
*** eolivare has joined #openstack-infra06:10
*** hamalq has quit IRC06:11
*** jtomasek has quit IRC06:21
*** zzzeek has quit IRC06:26
*** ralonsoh has joined #openstack-infra06:28
*** zzzeek has joined #openstack-infra06:28
*** ykarel has quit IRC06:37
*** xek has quit IRC06:38
*** hashar has joined #openstack-infra06:49
*** andrewbonney has joined #openstack-infra06:53
*** dklyle has quit IRC06:56
*** jtomasek has joined #openstack-infra07:00
*** jcapitao has joined #openstack-infra07:07
*** slaweq has joined #openstack-infra07:09
*** nightmare_unreal has joined #openstack-infra07:10
*** rcernin has quit IRC07:13
*** jpena|off is now known as jpena07:15
*** rcernin has joined #openstack-infra07:40
*** tosky has joined #openstack-infra07:40
*** gfidente has joined #openstack-infra08:03
*** lucasagomes has joined #openstack-infra08:13
*** priteau has joined #openstack-infra08:23
*** jtomasek has quit IRC08:27
*** derekh has joined #openstack-infra08:35
*** priteau has quit IRC08:35
*** priteau has joined #openstack-infra08:50
*** rcernin has quit IRC08:59
*** dtantsur|afk is now known as dtantsur09:00
*** rcernin has joined #openstack-infra09:00
priteauHello. Is there known issue with some zuul jobs? We're hitting retry_limit in the blazar gate09:01
priteauERROR 1064 (42000) at line 4: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'identified by 'openstack_citest' WITH GRANT OPTION' at line 209:01
priteauThis is the failing script, but nothing has changed: https://opendev.org/openstack/blazar/src/branch/master/tools/test-setup.sh09:02
priteauI think I found the fix in Cinder: https://review.opendev.org/#/c/737364/09:05
*** ysandeep is now known as ysandeep|lunch09:08
*** Lucas_Gray has joined #openstack-infra09:14
*** ociuhandu has joined #openstack-infra09:16
*** rcernin has quit IRC09:19
*** ysandeep|lunch is now known as ysandeep09:56
*** hashar is now known as hasharAway10:08
*** jcapitao has quit IRC10:14
*** owalsh has quit IRC10:15
*** rcernin has joined #openstack-infra10:37
*** rcernin has quit IRC10:37
*** rcernin has joined #openstack-infra10:37
*** ldenny has quit IRC10:48
*** ldenny has joined #openstack-infra10:49
*** hasharAway is now known as hashar10:55
*** priteau has quit IRC11:08
*** rcernin has quit IRC11:11
*** eolivare has quit IRC11:13
*** jcapitao has joined #openstack-infra11:26
*** jcapitao is now known as jcapitao_lunch11:26
*** jpena is now known as jpena|lunch11:33
*** xek has joined #openstack-infra11:40
*** owalsh has joined #openstack-infra11:42
*** hashar has quit IRC11:43
*** owalsh has quit IRC11:49
*** ykarel has joined #openstack-infra11:50
*** jtomasek has joined #openstack-infra12:00
*** jcapitao_lunch is now known as jcapitao12:04
*** rfolco has joined #openstack-infra12:05
*** ysandeep is now known as ysandeep|brb12:05
*** owalsh has joined #openstack-infra12:07
*** rfolco is now known as rfolco|ruck12:12
*** rlandy has joined #openstack-infra12:14
*** hashar has joined #openstack-infra12:16
*** ysandeep|brb is now known as ysandeep12:16
*** mmethot has quit IRC12:27
*** jhesketh has quit IRC12:32
*** jpena|lunch is now known as jpena12:34
*** zxiiro has joined #openstack-infra12:43
*** rcernin has joined #openstack-infra12:55
*** mmethot has joined #openstack-infra12:58
*** mgoddard has quit IRC13:04
*** mgoddard has joined #openstack-infra13:09
*** rcernin has quit IRC13:10
iurygregoryinfra folks is it possible to extend the timeout for openstack-tox-docs? =)13:12
iurygregoryhttps://zuul.openstack.org/builds?job_name=openstack-tox-docs&result=TIMED_OUT13:18
fungiiurygregory: we need to roll back the texlive-full addition, and that should solve the recent issue. when trying to builds pdfs on ubuntu focal it complains that something is missing, so the quick solution was apparently to add the texlive-full metapackage which drags in hundreds of additional packages when probably only one or two actual dependencies needed to be added13:18
fungii have a held node i need to find time today to experiment with reproducing the build and figure out what additional package(s) it really needs13:19
*** jhesketh has joined #openstack-infra13:19
fungiiurygregory: https://review.opendev.org/752584 is what i'm planning to update and then i'll exercise it with rechecks of https://review.opendev.org/75258513:20
iurygregoryfungi, I see! we are trying to fix ironic CI and we noticed the docs failing13:20
iurygregorytks for the heads up!13:20
fungino sweat, i'm just sorry i haven't found time to work it out yet13:21
iurygregoryno worries13:23
iurygregorythe worst thing we can do would be put non-voting docs to fix our main jobs lol13:23
*** jtomasek has quit IRC13:38
bcafarelhi infra folks, our fedora job neutron-ovn-tempest-ovs-master-fedora is failing for ~2 weeks and it seems generic issue to all fedora jobs)13:39
bcafarelrecent devstack-platform-fedora-latest failure: https://zuul.opendev.org/t/openstack/build/ca1deb29e6e443779f26dd0ab1a2222c13:39
bcafarelI checked the configure-unbound role but last change seems unrelated (forcing file permissions) and seems to work fine locally13:40
bcafarel(failure is usually in Start unbound: "Service is in unknown state")13:40
iurygregoryfungi, if we override tox_skip_pdf (to true) we would probably get things green right?13:41
*** Lucas_Gray has quit IRC13:43
*** priteau has joined #openstack-infra13:46
fungiiurygregory: if that causes it to not run the prepare-build-pdf-docs role, then yes i expect so13:54
fungii'm narrowing it down now though. i suspect it was a font which moved from texlive-fonts-recommended to texlive-fonts-extra between bionic and focal13:55
*** Goneri has joined #openstack-infra13:56
fungibcafarel: what fedora version? we're in the process of switching the fedora-latest label to fedora-32 today i think13:56
iurygregoryfungi, we were discussing possible solutions on ironic side just in case =)13:56
fungiseems something broke with service management on f30 and maybe f3113:56
*** lpetrut has joined #openstack-infra14:02
bcafarelfungi: devstack-single-node-fedora-latest node indeed14:03
fricklerbcafarel: fungi: yes, f31 seems broken, lyarwood started to work on f32 but some issues still remaining https://review.opendev.org/75029214:04
bcafarelfrickler: nice, thanks! I will follow that review14:06
*** artom has joined #openstack-infra14:15
*** eolivare has joined #openstack-infra14:24
*** armax has joined #openstack-infra14:26
*** ysandeep is now known as ysandeep|away14:27
*** hashar has quit IRC14:29
*** Lucas_Gray has joined #openstack-infra14:31
*** dklyle has joined #openstack-infra14:36
*** dklyle has quit IRC14:37
*** iurygregory has quit IRC14:46
*** iurygregory has joined #openstack-infra14:51
*** dklyle has joined #openstack-infra14:55
*** abhishekk is now known as abhishekk-afk14:55
clarkbbcafarel: fungi thats the known f31 issue15:03
clarkbbasically they have to update certain packages when systemd updates and when they don't it breaks ansible or something15:04
clarkbI started a fix but then ianw said we should focus on f32 instead15:04
clarkbbcafarel: https://review.opendev.org/#/c/751872/ was the beginnings of a workaround but testing it is tricky to ensure we don't break all the jobs. https://bugzilla.redhat.com/show_bug.cgi?id=1853736 is the fedora bug aiui15:06
openstackbugzilla.redhat.com bug 1853736 in systemd "systemctl show service fails with "Failed to parse bus message: Invalid argument"" [Unspecified,Closed: errata] - Assigned to systemd-maint15:06
*** rcernin has joined #openstack-infra15:06
bcafarelclarkb++ thanks so overall the fix is f32 switch in https://review.opendev.org/#/c/750292/ (plus maybe follow ups?)15:08
clarkbbcafarel: ya we announced we're bumping the base fedora-latest label to f32 today as well15:09
*** rcernin has quit IRC15:10
clarkbpriteau: re nothing changed, I think the thing that cahnged in this case is the switch to focal15:12
priteauyes, I assumed so15:12
priteauit's fixed now15:12
*** Lucas_Gray has quit IRC15:23
*** Lucas_Gray has joined #openstack-infra15:26
*** ykarel is now known as ykarel|away15:27
*** abhishekk-afk is now known as abhishekk15:35
*** xek has quit IRC15:36
toskysome time ago we discussed about os-loganalyze (which contains a legacy devstack-gate job) and, if I understand it correctly, it was mentioned here that it should be retired15:45
toskyso what are the next steps to ensure it's really removed? :)15:45
*** harlowja has joined #openstack-infra15:45
clarkbtosky: I think you can push a change to remove the job15:46
clarkbthen we can separately work to sunset the project itself. Will probably take a little longer as third party ci may be using it15:46
toskyclarkb: oh, oki, thanks!15:46
fungithe odds that they're continually deploying it are fairly low though, i would think?15:47
clarkbfungi: yes :) more just that we should send a note and give them a few days to offer to maintain it. The d-g job doesn't add a ton of value though I don't think15:47
clarkbthe unittests have lots of example log formats and should be sufficient if people do want to maintain it15:48
fungiif we don't plan to accept new fixes/improvements for that project then i expect the only way people who are trying to deploy it from git are likely to notice is when that stops working15:48
clarkbthats a good point15:48
fungisure we can announce the removal (likely to the third-party ci ml?)15:48
*** lpetrut has quit IRC15:49
fungibut folks whose deployments are that old are for the most part not paying attention anywhere we'd be announcing such things15:49
openstackgerritLuigi Toscano proposed openstack/os-loganalyze master: Remove the experimental legacy-dsvm-os-loganalyze job  https://review.opendev.org/75339015:49
*** artom has quit IRC15:52
*** ociuhandu_ has joined #openstack-infra15:55
*** bdodd has quit IRC15:55
*** bdodd has joined #openstack-infra15:56
*** ociuhandu has quit IRC15:58
fungias an update on the texlive-full rollback, researching the opaque error message went nowhere, so now i'm in the process of bisecting the packages pulled in by texlive-full to work out which one is making things succeed15:58
*** lucasagomes has quit IRC15:58
*** ykarel|away has quit IRC15:59
*** jcapitao has quit IRC16:00
*** ociuhandu_ has quit IRC16:00
iurygregoryfungi, http://paste.openstack.org/show/798225/ this would be the way to set tox_skip_pdf?16:01
iurygregoryI'm a bit puzzled since the job comes from the template (not sure if this would work)16:02
toskyclarkb: I suspect we may need to restore AJaeger's https://review.opendev.org/#/c/734600/16:02
fungiiurygregory: i think you need to add openstack-tox-docs to jobs lists for the same pipelines that template adds it, and then stick the vars under them16:04
clarkbtosky: hrm if pep8 isn't working maybe we should just go ahead and properly retire it16:04
clarkbAJaeger: ^ do you have an opinion on the preferred path there?16:05
iurygregoryfungi, gotcha16:05
fungizuul will deduplicate the extra job additions, and still keep the variant with the vars override16:05
fungi(i think!)16:05
iurygregoryI will see in a few =)16:07
*** irclogbot_2 has quit IRC16:08
*** bhagyashri|rover has quit IRC16:08
*** irclogbot_2 has joined #openstack-infra16:09
*** dosaboy has quit IRC16:09
*** bhagyashris has joined #openstack-infra16:10
fungiand it turns out my bisection was not idempotent, even after removing all the packages brought in by texlive-full i continued being able to successfully build, so i expect i need to clean additional artifacts between tests16:12
AJaegerclarkb: not really an opinion - I can restore if you want and let others take over16:12
*** harlowja has quit IRC16:12
*** dosaboy has joined #openstack-infra16:13
openstackgerritClark Boylan proposed openstack/project-config master: Add noop jobs to os-loganalyze  https://review.opendev.org/75339816:14
clarkbwhy don't we just do the retirement ^ I think that is the first step16:14
clarkbI'm going to pop out for a bike ride now but can push up the other changes when I get back16:14
*** hamalq has joined #openstack-infra16:16
*** jpena is now known as jpena|off16:21
*** redrobot has quit IRC16:25
*** johnsom has quit IRC16:26
*** johnsom has joined #openstack-infra16:27
*** dtantsur is now known as dtantsur|afk16:34
*** gfidente is now known as gfidente|afk16:43
*** lmiccini has quit IRC16:55
fungigetting close! i have a candidate package my cursory testing identified makes the pdf build work; now working on focused testing to confirm16:58
*** derekh has quit IRC16:58
*** Lucas_Gray has quit IRC17:00
fungithe interdependencies between hundreds of texlive ecosystem packages and indirect deps on metapackages and reliance on recommends (which we don't install by default) is making this really, really hard to track down. but i've narrowed it to two leaf packages now17:05
*** andrewbonney has quit IRC17:13
*** xek has joined #openstack-infra17:16
*** eolivare has quit IRC17:26
openstackgerritJeremy Stanley proposed openstack/openstack-zuul-jobs master: Replace texlive-full with cm-super  https://review.opendev.org/75258417:26
*** zxiiro has quit IRC17:29
fungii really don't understand ^17:35
fungiinstalling cm-super seems to make nova's docs pdf buildable without texlive-full on focal17:36
fungibut it's only a "suggests" of a few packages (not even a recommends), and we don't seem to be installing the packages suggesting it even on bionic17:38
*** ociuhandu has joined #openstack-infra17:38
*** artom has joined #openstack-infra17:39
fungitexlive-full depends on it, which is why that got the builds to succeed17:39
*** ociuhandu has quit IRC17:39
*** ociuhandu has joined #openstack-infra17:40
fungibut the only other rdepends for it on bionic are texlive-fonts-extra and fig2ps neither of which we seem to install, and even if we did i don't think the ansible package module installs suggests? at most recommends17:41
*** hamalq has quit IRC17:41
*** ramishra has quit IRC17:42
fungianyway, baffled17:44
fungii should have ci results on 752585 shortly to confirm my findings, at least17:44
*** ociuhandu has quit IRC17:45
fungioh, i also ruled out that it was any of cm-super's dependencies (cm-super-minimal, pfb2t1c2pfb) since leaving those installed and removing cm-super itself was enough to reproduce the error17:45
*** ralonsoh has quit IRC17:46
*** eharney has joined #openstack-infra17:46
fungistarting to think this isn't a missing/added package issue, but rather a behavior change between python 3.6 and 3.817:47
*** tbachman has quit IRC18:07
*** tbachman has joined #openstack-infra18:08
*** gyee has joined #openstack-infra18:34
openstackgerritClark Boylan proposed openstack/project-config master: Add noop jobs to os-loganalyze  https://review.opendev.org/75339818:35
openstackgerritClark Boylan proposed openstack/os-loganalyze master: Retire os-loganalyze  https://review.opendev.org/75342618:40
openstackgerritClark Boylan proposed openstack/project-config master: Retire os-loganalyze  https://review.opendev.org/75342818:43
clarkbAJaeger: ^ thanks for the review. I think that is all the required changes18:43
*** xek has quit IRC18:50
*** vishalmanchanda has quit IRC18:58
*** zzzeek has quit IRC19:08
*** hamalq has joined #openstack-infra19:09
*** zzzeek has joined #openstack-infra19:09
*** priteau has quit IRC19:23
*** Tengu has quit IRC19:32
*** Tengu has joined #openstack-infra19:33
*** matt_kosut has quit IRC19:47
*** zxiiro has joined #openstack-infra19:53
*** bdodd has quit IRC19:54
*** tbachman_ has joined #openstack-infra20:00
*** tbachman has quit IRC20:01
*** tbachman_ is now known as tbachman20:01
*** matt_kosut has joined #openstack-infra20:03
*** gfidente|afk has quit IRC20:04
*** matt_kosut has quit IRC20:08
*** tbachman_ has joined #openstack-infra20:19
*** tbachman has quit IRC20:20
*** tbachman_ is now known as tbachman20:20
*** ianychoi_ has joined #openstack-infra20:23
*** ianychoi has quit IRC20:27
*** ociuhandu has joined #openstack-infra20:34
*** rfolco|ruck has quit IRC20:59
*** tosky_ has joined #openstack-infra21:09
*** tosky is now known as Guest3320021:10
*** tosky_ is now known as tosky21:10
*** eharney has quit IRC21:11
*** pcrews has joined #openstack-infra21:13
*** slaweq has quit IRC21:18
*** mwhahaha has quit IRC22:04
*** gmann has quit IRC22:04
*** arne_wiebalck has quit IRC22:04
*** mwhahaha has joined #openstack-infra22:04
*** seongsoocho has quit IRC22:04
*** zxiiro has quit IRC22:04
*** nicolasbock has quit IRC22:04
*** arne_wiebalck has joined #openstack-infra22:05
*** seongsoocho has joined #openstack-infra22:05
*** nicolasbock has joined #openstack-infra22:05
*** gmann has joined #openstack-infra22:06
*** zxiiro has joined #openstack-infra22:06
*** gouthamr has quit IRC22:06
*** gouthamr has joined #openstack-infra22:11
fungistill waiting for my nova canary recheck to sing on the pdf build change22:21
fungiand as i say that, it got a node assignment!22:22
fungijust shy of 5 hours in check ;)22:22
clarkbyou know the suns out and its not too cold. I should go outside and listen to the birds while I do code review22:22
fungiyou should!22:22
clarkbfungi: its release candidate time and we had to disable half of inap22:22
fungiyeah, i mean, i'm not surprised it took five hours22:23
fungiat least i was conscientious and rejiggered the dnm change so it only runs that one docs job22:23
funginope, my mistake. looked like it got a node but now it's back to queued22:25
fungii wonder if we have nodes falling over somewhere22:25
*** calbers has quit IRC22:26
fungiguess i'll double check how many restarts this one logs after it's reported22:26
fungilooking at the node requests graph, apparently an hour ago it suddenly doubled22:28
fungiahh, looks like there may have been a massive surge in the check pipeline at that point22:28
fungiokay, it's showing a node assignment for 752585 again22:29
fungiseems like it might actually be running22:30
fungizuul's estimating ~20 minutes for that job, probably because so many of them are timing out at that point currently22:30
*** calbers has joined #openstack-infra22:33
fungiit's now reached the task of interest22:34
fungi(install pdf prereqs)22:34
fungiand already done22:35
clarkbabout 10 minutes quicker :)22:35
fungi~4 minutes into the job it was past the pdf prereqs installation22:35
funginow to hope it successfully builds like it did for me manually22:35
fungilooks like it completed the pdf build and is wrapping up, so i think we have a winner22:39
fungiand done22:41
fungihttps://zuul.opendev.org/t/openstack/build/d973bcabf3b74d5e83ce56b6b7b2d221 does not show multiple tries, so i probably imagined it going back to queued22:42
fungigmann: heads up, https://review.opendev.org/752584 seems to work for running the docs job in ~half the time22:44
fungiconfig-core: ^ a quick second review on that would be most appreciated so we can unbreak lots of projects22:44
fungizuul clocked that build in at 10m 45s22:45
*** ociuhandu has quit IRC22:46
*** tosky has quit IRC22:48
openstackgerritMerged openstack/openstack-zuul-jobs master: Replace texlive-full with cm-super  https://review.opendev.org/75258422:52
fungiiurygregory: ^ you can probably remove whatever workarounds you put in place now22:52
fungii did not test ironic's docs jobs with it, but it worked with nova's22:53
*** rcernin has joined #openstack-infra22:55
*** rlandy has quit IRC22:56
*** tkajinam has joined #openstack-infra22:57
*** artom has quit IRC23:01
*** zzzeek has quit IRC23:30
*** zzzeek has joined #openstack-infra23:31
*** tetsuro has joined #openstack-infra23:57

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!