Monday, 2020-01-06

« Sunday, 2020-01-05 Index Tuesday, 2020-01-07 »
*** tosky has quit IRC00:05
*** bhavikdbavishi has joined #zuul02:54
*** bhavikdbavishi1 has joined #zuul02:59
*** bhavikdbavishi has quit IRC03:01
*** bhavikdbavishi1 is now known as bhavikdbavishi03:01
*** sanjayu_ has joined #zuul03:55
*** chandankumar has joined #zuul04:50
*** sanjayu__ has joined #zuul04:52
*** sanjayu_ has quit IRC04:55
*** chandankumar has quit IRC05:32
*** evrardjp has quit IRC05:33
*** evrardjp has joined #zuul05:33
*** bhavikdbavishi has quit IRC05:37
*** bhavikdbavishi has joined #zuul05:38
*** chandankumar has joined #zuul05:53
*** threestrands has quit IRC05:58
*** chandankumar has quit IRC06:14
*** chandankumar has joined #zuul06:17
*** bhavikdbavishi has quit IRC06:21
*** bhavikdbavishi has joined #zuul06:23
*** chandankumar has quit IRC06:42
*** chandankumar has joined #zuul07:05
*** themroc has joined #zuul08:26
*** jangutter has joined #zuul08:54
*** bhavikdbavishi has quit IRC08:56
*** bhavikdbavishi has joined #zuul08:57
*** themroc has quit IRC08:58
*** themroc has joined #zuul09:00
*** electrofelix has joined #zuul09:03
*** bhavikdbavishi has quit IRC09:09
*** bhavikdbavishi has joined #zuul09:45
*** pcaruana has joined #zuul09:54
*** bhavikdbavishi has quit IRC10:04
*** pcaruana has quit IRC10:11
*** pcaruana has joined #zuul10:12
*** bhavikdbavishi has joined #zuul10:20
*** reiterative has joined #zuul10:20
*** sanjayu__ has quit IRC10:33
*** saneax has joined #zuul10:33
openstackgerritAkihiro Motoki proposed zuul/zuul-jobs master: Allow to use ensure-babel without ensure-sphinx  https://review.opendev.org/70120911:35
*** rfolco has joined #zuul12:10
*** armstrongs has joined #zuul12:13
*** themroc has quit IRC12:18
*** themroc has joined #zuul12:18
*** rlandy has joined #zuul12:58
Shrewsohai #zuul13:51
*** pabelanger has joined #zuul13:55
pabelangero/13:55
pabelanger#zuul: Cannot join channel (+r) - you need to be identified with services - see https://freenode.net/kb/answer/registration13:55
pabelangerare we at a point now, where we can try dropping +r on zuul?13:55
pabelangerI seem to remember some discussions around that13:56
tristanChello folks o/   could someone help me confirm that pipeline-config.queue attribute modification are not evaluated speculatively and they are only effective after merge?13:59
pabelangertristanC: yes, need to be merged to be applied14:00
*** tosky has joined #zuul14:01
tristanCpabelanger: alright, thank you14:01
*** Goneri has quit IRC14:03
*** nhicher has joined #zuul14:05
openstackgerritTristan Cacqueray proposed zuul/zuul master: doc: add note for speculative queue attribute  https://review.opendev.org/70123014:06
openstackgerritMerged zuul/nodepool master: Fix resource warnings when running tests  https://review.opendev.org/70027514:41
openstackgerritMerged zuul/nodepool master: Parallelize test_ec2_machine  https://review.opendev.org/70027614:47
openstackgerritMerged zuul/nodepool master: Support userdata for instances in aws  https://review.opendev.org/63271214:59
AJaegerhttps://review.opendev.org/#/c/700899/ for zuul-jobs ensures that we have the right python3 interpreter installed, I have two +2s on it but no +A - I'll self-approve later today unless somebody -1s or +As earlier15:06
*** Goneri has joined #zuul15:10
corvusAJaeger: lgtm, but i'll leave it out there a little more in case someone else thinks of something we should check15:15
openstackgerritMerged zuul/nodepool master: Make public ip configurable in aws  https://review.opendev.org/63271515:16
AJaegerthanks, corvus15:16
ShrewsAJaeger: i though tox would throw an error if you requested a python env (e.g., -e py35) where the right version of python was not installed?15:22
AJaegerShrews: It apparently does not, see recent runs of zuul-jobs, let me grab one for you15:23
AJaegerhttps://zuul.opendev.org/t/zuul/build/dfd018207e9d449cae0fef12f95b86a9/log/job-output.txt#41215:24
AJaegerShrews, " UserWarning: conflicting basepython version (set 3.6, should be 3.5) f"15:24
AJaegerso, not fatal15:24
Shrews$ tox -epy3715:25
Shrewspy37 create: /media/psf/Home/Devel/git/zuul/nodepool/.tox/py3715:25
ShrewsERROR: InterpreterNotFound: python3.715:25
Shrewsweird15:26
fungiAJaeger: is that with https://tox.readthedocs.io/en/latest/config.html#conf-skip_missing_interpreters set?15:27
Shrewsmaybe it's not fatal if it can use a different available minor version?15:27
*** chandankumar has quit IRC15:27
Shrewsi'm not opposed to the change. seems good to me. just confused why it isn't fatal in this scenario15:28
Shrewsmaybe because of the config setting fungi pointed out15:28
fungialternatively, maybe an unexpected side-effect of https://tox.readthedocs.io/en/latest/config.html#conf-ignore_basepython_conflict15:28
AJaegerfungi: zuul-jobs does not set skip_missing_interpreters15:29
AJaegerfungi: and neither is ignore_basepython_conflict set AFAIK15:29
fungi"In a future version of tox, this warning will become an error."15:30
fungimaybe the difference is the version of tox each of you is running?15:30
Shrews3.14.215:31
openstackgerritDavid Shrewsbury proposed zuul/zuul-jobs master: ensure-tox: Output tox version  https://review.opendev.org/70123615:36
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: DNM: testing  https://review.opendev.org/70123715:37
corvushappy new year everyone! :)15:50
AJaegerhappy new year, corvus - and everyone else ;)15:52
AJaegerhttps://review.opendev.org/696337 has two +2s but no +A - "Add pypi_fqdn to differentiate it package mirrors". tristanC any reason not to merge? Anybody wants to +A?15:53
corvusi think it's fine to merge changes like that until the new mirror_info stuff is viable (it isn't yet)15:55
*** electrofelix has quit IRC15:56
clarkbpabelanger: we still get spam in the unregistered channel so probably not (however that spam originates from one of about 120 channels so maybe it would be ok?)15:58
*** chandankumar has joined #zuul16:02
pabelangerclarkb: ack, we may want to update it from openstack-unregistered to opendev-unregistered for long term too16:09
*** arxcruz is now known as arxcruz|off16:11
*** themroc has quit IRC16:29
openstackgerritDavid Shrewsbury proposed zuul/zuul-jobs master: ensure-tox: Output tox version  https://review.opendev.org/70123616:29
pabelangerHmm, just upgrade zuul to 3.14.0 and think paramiko changed, I now see http://paste.openstack.org/show/788087/ for SSH keys for gerrit16:34
*** mattw4 has joined #zuul16:34
clarkbpabelanger: I think you may need to use rsa keys?16:35
clarkbseems like we ran into similar due to a bug in paramiko? and forcing rsa bypasses16:36
pabelangerclarkb: well, it was working before upgrades, let me see why it doesn't not16:36
openstackgerritMerged zuul/zuul-jobs master: Add pypi_fqdn to differentiate it package mirrors  https://review.opendev.org/69633716:36
clarkbpabelanger: look in the git history of the zuul quickstart testing16:39
clarkbI think that is where we hit it16:39
pabelangerk16:40
pabelangerparamiko 2.7.1 looks to have added new private key support: https://github.com/net-ssh/net-ssh/issues/63316:41
fungiparamiko had a couple releases last month, so maybe this is the first time you're running 2.7.116:41
pabelangeroops16:41
pabelangerthat isn't paramiko issue16:41
pabelangerfungi: yah, is16:41
fungihttps://www.paramiko.org/changelog.html16:43
pabelangeryah, I likely have to convert my PEM keys now16:44
fungi[Feature] #602: (via #1343, #1313, #618) Implement support for OpenSSH 6.5-style private key files (typically denoted as having BEGIN OPENSSH PRIVATE KEY headers instead of PEM format’s BEGIN RSA PRIVATE KEY or similar). If you were getting any sort of weird auth error from “modern” keys generated on newer operating system releases (such as macOS Mojave), this is the first update to try.16:44
fungipretty sure that's what we hit16:44
fungi(ni 2.7.0)16:44
clarkbpabelanger: b2e5500cc266052cb9c801a42ff6b74eb81fc46c zuul commit16:45
clarkbfungi: ya ^ worked around it via explicit format setting16:45
*** themroc has joined #zuul16:45
fungiahh, yeah, https://review.opendev.org/69070116:45
openstackgerritJames E. Blair proposed opendev/zone-zuul-ci.org master: Update gitreview  https://review.opendev.org/70124916:46
*** themroc has quit IRC16:46
openstackgerritJames E. Blair proposed opendev/zone-zuul-ci.org master: Delegate acme challenges to opendev  https://review.opendev.org/70125016:46
*** tosky has quit IRC16:46
fungiresulting in this note: https://zuul-ci.org/docs/zuul/admin/components.html#attr-executor.private_key_file16:47
fungiand a similar one in the gerrit driver doc16:47
pabelangerclarkb: fungi: thanks! I'll downgrade for now, and reformat keys16:47
pabelangerZuul version: 3.14.0 !!16:51
pabelangernow to migrate jobs to 2.916:51
openstackgerritMerged zuul/zuul master: doc: add note for speculative queue attribute  https://review.opendev.org/70123016:56
openstackgerritPaul Belanger proposed zuul/zuul master: Use -N '' for ssh-keygen  https://review.opendev.org/70125216:56
pabelangerclarkb: fungi: nit^ in docs16:57
openstackgerritJames E. Blair proposed opendev/zone-zuul-ci.org master: Use tabs instead of spaces  https://review.opendev.org/70125416:59
fungipabelanger: oh, yep, that matches what we fixed in the playbook. thanks!16:59
openstackgerritMerged opendev/zone-zuul-ci.org master: Update gitreview  https://review.opendev.org/70124917:01
openstackgerritMerged opendev/zone-zuul-ci.org master: Delegate acme challenges to opendev  https://review.opendev.org/70125017:01
*** chkumar246 has joined #zuul17:01
*** chandankumar has quit IRC17:04
*** chkumar246 has quit IRC17:07
*** saneax has quit IRC17:10
openstackgerritMerged opendev/zone-zuul-ci.org master: Use tabs instead of spaces  https://review.opendev.org/70125417:14
openstackgerritJames E. Blair proposed opendev/zone-zuul-ci.org master: Also delegate www. acme record to opendev  https://review.opendev.org/70125717:20
openstackgerritMerged zuul/zuul-jobs master: Ensure python interpreter is installed for tox-py3X  https://review.opendev.org/70089917:22
*** rishabhhpe has joined #zuul17:28
corvusfyi over in #openstack-infra we're working on getting a new ssl cert for zuul-ci.org17:29
*** sshnaidm is now known as sshnaidm|afk17:31
*** mattw4 has quit IRC17:31
*** mattw4 has joined #zuul17:32
rishabhhpeHello All,17:32
rishabhhpeHappy New Year, Can someone please help after looking at this issue from zuul.log. I had pasted the complete error along with layout.yaml file and output of17:32
rishabhhpecommand for status of gearman in this link -: http://paste.openstack.org/show/788088/17:32
rishabhhpeIt will be really helpful if anyone from the team can help in debugging this issue .17:32
rishabhhpeOur build is broken from past two weeks and showing status as N on community page.17:32
*** evrardjp has quit IRC17:33
*** evrardjp has joined #zuul17:33
openstackgerritMerged opendev/zone-zuul-ci.org master: Also delegate www. acme record to opendev  https://review.opendev.org/70125717:36
fungirishabhhpe: it's been a while since i've worked on old zuul versions. you're still using 2.x with jenkins i guess?17:39
rishabhhpe@fungi: Yes we are still using old zuul 2.x17:40
rishabhhpehere it is saying job is not registered with gearman17:40
fungiare the jobs actually defined in your jenkins master? is the gearman plug-in installed and enabled? i don't see any of the job names from your layout getting registered by your jenkins master in gearman17:40
fungii also have completely paged out any memory of how to troubleshoot the jenkins side of that. i vaguely recall that disabling and reenabling the plug-in should trigger the jenkins master to reregister all its jobs17:41
rishabhhpeYes job defination is defined in master ..properly but this issue is coming only after i restarted my master ci server17:41
clarkbit will only register if you have slaves connected that can run the job iirc17:42
fungioh, right, that too17:42
fungiare you using nodepool or registering your slaves in some other way?17:42
fungiremoving and re-adding the slaves should get those jobs to show up in gearman17:42
rishabhhpewe have nodepool which had already spawned multiple instances17:42
rishabhhpeexplicitly no slave is there17:43
fungido you set a min-ready for your nodes in nodepool?17:44
fungiso that there is at least one slave registered with jenkins?17:44
rishabhhpeYes it is defined in nodepool.yaml if you want i can share the content of it with you17:44
rishabhhpefungi: i had pasted my nodepool.yaml file here http://paste.openstack.org/show/788089/17:46
pabelangerrishabhhpe: you may want to rotate password, ^ has exposed them17:51
pabelangerpasswords*17:52
fungiokay, so you should have at least 4 ready d-p-c slaves and 4 ready d-p-c-fc slaves in jenkins. are they showing up there?17:52
rishabhhpepabelanger: thanks .. will take care of it from next time17:53
rishabhhpefungi: yes there 8 ready instances which i can see17:53
fungiso in that case, disabling and reenabling the gearman plug-in in your jenkins master should cause it to register its availability of nodes for those jobs in zuul's gearman (i think restarting the jenkins master will do it too, but that's more disruptive)17:55
rishabhhpeWell i did jenkins restart also but it wont workout ... will enable and disable of gearman plugin will not impact the current setup .. as we might be using very old version tools .. which are not updated from long time .17:56
pabelangerrishabhhpe: fungi: https://www.mediawiki.org/w/index.php?title=Continuous_integration/Zuul#Debugging maybe helpful, that is still 2.x based. For getting more info from gearman17:59
pabelangerhttps://docs.openstack.org/infra/openstackci/third_party_ci.html#setup-jenkins too17:59
fungiyes, it's unfortunately hard to support very old versions of software. we specifically designed jenkins out of zuul and released v3 almost two years ago now, so our focus is going to be on making it easy to run without jenkins and lots of us are quickly forgetting how all that stuff used to work18:00
pabelanger++18:01
fungiseeking help from people who are still running zuul 2.x and jenkins may be a more effective route, but i don't know where to find them (and hopefully their numbers are dwindling as they replace their old ci systems)18:02
rishabhhpepabelanger: fungi: i am not able to find this file only zuul-gearman.py18:07
*** pcaruana has quit IRC18:07
pabelangerhttps://github.com/wikimedia/puppet/blob/production/modules/zuul/files/zuul-gearman.py18:08
*** pcaruana has joined #zuul18:16
rishabhhpepabelanger: copying this file from the repo u have given and running the command zuul-gearman.py status it is showing command not found .. does it require any extra setup ?18:19
pabelangerrishabhhpe: not sure, I haven't used wikimedia tools before. I mostly was pointing to another data point to aid in debugging18:20
*** pcaruana has quit IRC18:21
rishabhhpeOk18:23
*** armstrongs has quit IRC18:24
rishabhhpefungi: enabling/disable also does not workout still getting the same error18:26
openstackgerritMerged zuul/zuul master: Use -N '' for ssh-keygen  https://review.opendev.org/70125218:32
corvusrishabhhpe: you may be able to turn on debug logging in the jenkins master to find out why the jobs aren't being registered.  i'm sorry i don't remember how to do that.18:39
rishabhhpecorvus: thanks i will check that part18:40
openstackgerritJames E. Blair proposed zuul/zuul master: Re-elect James Blair as project lead  https://review.opendev.org/70126719:41
*** rlandy is now known as rlandy|afk19:43
fungiit's a brand new year!19:50
corvusdeserving of a one-bit change19:52
*** armstrongs has joined #zuul19:57
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: DNM: testing  https://review.opendev.org/70123720:00
corvusmnaser: do you need anything to move forward on the helm stuff?  (i see you did work over the holidays while i was not around)20:01
*** rishabhhpe has quit IRC20:01
fungiindeed, even if that's ascii "0" to "1" the binary encoded value is changing from 00110000 to 0011000120:02
fungitruly a one-bit change20:02
AJaegercorvus: there are two changes up for zuul-jobs by mnaser, I think he needs both20:02
mnasercorvus: i'd appreciate it, actually ran into two small tidbits -- for some reason the helm jobs are not being seen (depends-on a untrusted projects seems to have shown as if the job doesn't exist?) and operational side of things, im seeing a lot of defunct processes in the executor pods so i think there's something to really to figure out20:02
*** ianychoi has quit IRC20:04
Shrewsi hope jeblair nominates himself to oppose corvus, just for the entertainment value!20:05
*** armstrongs has quit IRC20:05
corvusmnaser: ack, i'll look into the job thing after lunch; re executor -- like zombie procesess?  which processes are those?20:12
fungiShrews: i'm looking forward to the debate20:13
corvusfungi, Shrews: oh man you really don't want to hear the voices in my head arguing with each other20:13
fungiit'll be like watching a single-player chess match?20:13
corvuswith corvus playing checkers20:13
*** igordc has joined #zuul20:13
fungii'm imagining that pixar short "geri's game"20:14
*** igordc has quit IRC20:15
Shrewshttps://gph.is/2jTq91T20:15
fungiyep, that's the one20:16
*** ianychoi has joined #zuul20:19
*** rfolco has quit IRC20:34
*** tosky has joined #zuul20:55
*** zbr has quit IRC21:05
*** zbr has joined #zuul21:07
openstackgerritJames E. Blair proposed zuul/zuul-helm master: Added nodepool to charts  https://review.opendev.org/70046221:09
corvusmnaser: 2 things: 1) the depends-on url format was not one that zuul recognized (/c/<number>;  it does recognize /#/c/<number).  maybe we should add that?  where did you copy it from?)21:11
corvusmnaser: 2) i updated the zuul.d/project.yaml file to match the zuul "house style" for zuul config; not something i'm normally picky about in other projects, but it's nice for zuul's own config to match the other zuul projects and the docs (no leading '---', and indented list items)21:13
*** zbr has quit IRC21:14
corvusit looks like the job is hitting pre-run failures21:14
*** michael-beaver has joined #zuul21:15
*** zbr has joined #zuul21:22
*** zbr has quit IRC21:26
*** zbr has joined #zuul21:30
*** zbr_ has joined #zuul21:37
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Stop compressing files during intermediate steps  https://review.opendev.org/70128221:38
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Test swift upload logs without encoding type for gzip files  https://review.opendev.org/70128321:38
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Test swift upload logs without encoding type for gzip files  https://review.opendev.org/70128421:38
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Swift upload logs without encoding type for gzip files  https://review.opendev.org/70128421:38
*** zbr has quit IRC21:39
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Stop compressing files during intermediate steps  https://review.opendev.org/70128221:48
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Test swift upload logs without encoding type for gzip files  https://review.opendev.org/70128321:48
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Swift upload logs without encoding type for gzip files  https://review.opendev.org/70128421:48
corvusclarkb: should we do an announcement before 701282?  and for opendev, maybe go ahead and set that as a site var or something?21:48
corvus(i'm not too worried about fetch-subunit-output, but rather stage-output)21:49
clarkbthe flip of the default value you mean? ya I agree21:49
corvusya21:49
clarkbI guess I should split this into two stacks then as opdnev can set the var early21:49
corvusclarkb: ++; we can even make sure we get it right before sending the deprecation announcement21:49
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Test swift upload logs without encoding type for gzip files  https://review.opendev.org/70128321:50
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Swift upload logs without encoding type for gzip files  https://review.opendev.org/70128421:50
clarkbOk thats the stacks split21:50
clarkbI'll work on the opendev change21:50
*** zbr_ has quit IRC22:02
*** zbr has joined #zuul22:02
*** zbr has quit IRC22:25
*** zbr has joined #zuul22:25
*** zbr has quit IRC22:30
*** jamesmcarthur has joined #zuul22:44
*** Goneri has quit IRC22:46
*** jamesmcarthur has quit IRC22:57
*** jamesmcarthur has joined #zuul23:03
*** zbr has joined #zuul23:04
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: Add basic Helm jobs  https://review.opendev.org/70022223:07
*** armstrongs has joined #zuul23:07
*** zbr_ has joined #zuul23:08
*** zbr has quit IRC23:09
*** mattw4 has quit IRC23:09
*** mattw4 has joined #zuul23:10
*** Goneri has joined #zuul23:10
*** rlandy|afk is now known as rlandy23:11
*** zbr_ has quit IRC23:14
*** armstrongs has quit IRC23:17
*** jamesmcarthur has quit IRC23:17
*** zbr has joined #zuul23:21
*** tosky has quit IRC23:23
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: DNM: testing  https://review.opendev.org/70123723:52
*** mattw4 has quit IRC23:56
« Sunday, 2020-01-05 Index Tuesday, 2020-01-07 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!