Thursday, 2020-02-06

*** sshnaidm is now known as sshnaidm|afk00:08
*** igordc has quit IRC00:08
*** armstrongs has joined #zuul00:12
*** decimuscorvinus_ has joined #zuul00:16
*** dmellado_ has joined #zuul00:16
*** paulalbertella has joined #zuul00:17
*** wxy-xiyuan has joined #zuul00:20
*** decimuscorvinus has quit IRC00:21
*** dmellado has quit IRC00:21
*** reiterative has quit IRC00:22
*** dmellado_ is now known as dmellado00:22
*** Miouge has quit IRC00:22
*** armstrongs has quit IRC00:22
*** Miouge has joined #zuul00:22
*** johanssone has quit IRC00:23
*** dustinc has quit IRC00:25
*** mnasiadka has quit IRC00:25
*** dustinc has joined #zuul00:25
*** zxiiro has quit IRC00:25
*** andreaf has quit IRC00:25
*** evgenyl has quit IRC00:25
*** Shrews has quit IRC00:25
*** stevthedev has quit IRC00:25
*** donnyd has quit IRC00:25
*** logan- has quit IRC00:25
*** andreaf has joined #zuul00:25
*** evgenyl has joined #zuul00:25
*** mnasiadka has joined #zuul00:25
*** donnyd has joined #zuul00:25
*** Shrews has joined #zuul00:25
*** stevthedev has joined #zuul00:26
*** johanssone has joined #zuul00:26
*** zxiiro has joined #zuul00:26
*** zbr has quit IRC00:27
*** logan- has joined #zuul00:27
*** mattw4 has quit IRC00:28
*** openstackstatus has quit IRC00:28
*** zbr has joined #zuul00:31
*** jamesmcarthur has joined #zuul01:12
*** jhesketh has quit IRC01:13
*** jhesketh has joined #zuul01:14
*** jamesmcarthur has quit IRC01:17
*** jamesmcarthur has joined #zuul01:29
*** jamesmcarthur has joined #zuul01:53
*** jamesmcarthur has quit IRC02:01
*** jamesmcarthur has joined #zuul02:02
*** jamesmcarthur has quit IRC02:08
*** jamesmcarthur has joined #zuul02:29
*** bhavikdbavishi has joined #zuul02:48
*** bhavikdbavishi1 has joined #zuul02:52
*** bhavikdbavishi has quit IRC02:54
*** bhavikdbavishi1 is now known as bhavikdbavishi02:54
*** jamesmcarthur has quit IRC03:01
*** jamesmcarthur has joined #zuul03:03
*** jamesmcarthur has quit IRC03:08
*** jamesmcarthur has joined #zuul03:18
*** jamesmcarthur has quit IRC03:44
*** jamesmcarthur has joined #zuul03:44
*** jamesmcarthur has quit IRC03:50
*** jamesmcarthur has joined #zuul04:00
*** jamesmcarthur has quit IRC04:03
*** jamesmcarthur has joined #zuul04:05
*** jamesmcarthur has quit IRC04:10
*** jamesmcarthur has joined #zuul04:35
*** zxiiro has quit IRC05:11
*** jamesmcarthur has quit IRC05:14
*** raukadah is now known as chkumar|rover05:18
*** rlandy|bbl has quit IRC05:25
*** evrardjp has quit IRC05:33
*** evrardjp has joined #zuul05:34
*** bolg has joined #zuul05:36
openstackgerritJan Kubovy proposed zuul/zuul master: WIP: Store unparsed branch config in Zookeeper  https://review.opendev.org/70571605:37
*** paulalbertella has quit IRC05:39
*** reiterative has joined #zuul05:39
*** swest has joined #zuul05:41
*** yolanda has quit IRC06:01
*** bhavikdbavishi has quit IRC06:02
*** bhavikdbavishi has joined #zuul06:02
*** felixedel has joined #zuul06:14
*** felixedel has quit IRC06:18
*** saneax has joined #zuul06:20
*** fdegir has quit IRC06:20
*** felixedel has joined #zuul06:20
*** fdegir has joined #zuul06:20
*** felixedel has quit IRC07:01
*** felixedel has joined #zuul07:07
*** felixedel has quit IRC07:12
openstackgerritFelix Schmidt proposed zuul/zuul master: Implement github checks API  https://review.opendev.org/70516807:16
*** jpena|off is now known as jpena07:54
*** yolanda has joined #zuul08:19
*** saneax has quit IRC08:22
*** saneax has joined #zuul08:23
openstackgerritFelix Schmidt proposed zuul/zuul master: Implement github checks API  https://review.opendev.org/70516808:30
*** sugaar has joined #zuul08:31
openstackgerritJan Kubovy proposed zuul/zuul master: WIP: Store unparsed branch config in Zookeeper  https://review.opendev.org/70571608:35
*** tosky has joined #zuul08:40
*** sshnaidm|afk is now known as sshnaidm08:54
*** bhavikdbavishi has quit IRC09:14
ofososHey, can someone help me with this threaddump? Paste.debian.net/1129400/09:50
ofososThis is the Bitbucket driver after running for five days. Is it possible that all my threads disappeared, because of an uncaught exception?09:51
ofososWell, the first part is from shortly after the start, the second part is after the breakage10:01
mnaserofosos: i dont know enough about these deep internals but could it be possible that something happened to your zk connection?10:07
mnaserofosos: anything else in the rest of the logs that can be helpful?10:07
openstackgerritDaniel Pawlik proposed zuul/zuul-jobs master: Fix ensure-tox issue when tox was installed on the host  https://review.opendev.org/70621610:22
*** apevec has joined #zuul10:33
openstackgerritSimon Westphahl proposed zuul/zuul master: Add Zuul's event id to Ansible inventory  https://review.opendev.org/70622210:33
openstackgerritDaniel Pawlik proposed zuul/zuul-jobs master: Fix ensure-tox issue when tox was installed on the host  https://review.opendev.org/70621610:35
openstackgerritSimon Westphahl proposed zuul/zuul-jobs master: Add event id to emit-job-header  https://review.opendev.org/70622510:37
openstackgerritDaniel Pawlik proposed zuul/zuul-jobs master: Fix ensure-tox issue when tox was installed on the host  https://review.opendev.org/70621610:43
*** bolg_ has joined #zuul10:52
*** bolg has quit IRC10:52
*** bolg_ has quit IRC10:55
openstackgerritDaniel Pawlik proposed zuul/zuul-jobs master: Fix ensure-tox issue when tox was installed on the host  https://review.opendev.org/70621611:07
*** dpawlik has joined #zuul11:07
dpawlikAJaeger, hi. Changed commit msg: https://review.opendev.org/#/c/706216/11:08
dpawlikAJaeger, example of failing job: https://review.rdoproject.org/zuul/build/148f86eb24a6474aa729000a55a7c5ac11:09
*** bhavikdbavishi has joined #zuul11:32
*** bhavikdbavishi1 has joined #zuul11:35
*** bhavikdbavishi has quit IRC11:37
*** bhavikdbavishi1 is now known as bhavikdbavishi11:37
*** avass has joined #zuul11:46
*** apevec has quit IRC11:48
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Adds variable to toggle whether to revoke sudo  https://review.opendev.org/70624811:58
*** jpena is now known as jpena|lunch12:00
*** wxy-xiyuan has quit IRC12:00
*** rfolco has joined #zuul12:05
mnaserdpawlik: ah crap good catch12:06
mnaserdpawlik: damnit this was my fault12:07
mnaseri am pretty sure i forgot to git add defaults/main.yml12:07
dpawlikmnaser, oh, cool solution. I didn't think about such way12:08
mnaserwait, there is `tox_executable: tox`12:08
mnaserin defaults12:08
mnaseri didnt forget..12:08
dpawlikmnaser, I was performing a test of the playbook on local VM12:08
mnaserdpawlik: i guess the functional change is that `tox_executable` is actually being used now in that role12:09
mnaserso maybe you have that overridden before?12:09
dpawlikmnaser, let me check on Fedora 28 (its mostly failing on that iamge)12:09
mnaserbefore it would do tox and now it uses `tox_executable` so *maybe* someone override that in your environment (because otherwise i would imagine opendev would have blew up)12:09
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Adds variable to toggle whether to revoke sudo  https://review.opendev.org/70624812:10
dpawlikmnaser, let me check one more time12:11
dpawlikhttps://logserver.rdoproject.org/80/24780/1/gate/tox-docs/148f86e/job-output.txt 2020-02-05 18:18:36.93772612:12
dpawlikso its different problem. Checking12:12
pabelanger/usr/bin/python3 -m tox --version that seems wrong is that because tox_executable is "/usr/bin/python3 -m tox" ?12:13
pabelangeryes12:13
pabelangerhttps://logserver.rdoproject.org/80/24780/1/gate/tox-docs/148f86e/zuul-info/inventory.yaml12:13
pabelangerso, there is some logic errors12:13
*** pcaruana has quit IRC12:13
pabelangerthe type tox || pip install foo12:13
pabelangershould be12:13
pabelangertype: {{ tox_executable || || pip install tox --user12:13
pabelangertype: {{ tox_executable }} || pip install tox --user12:13
mnaseri wonder if that will always pass though12:14
mnaserbecause python3 would be there but tox would not be12:14
mnaserin dpawlik case at least12:14
mnaseralmost like we should do tox version check instead of type12:14
pabelangerwe need to look at old rdo job12:15
pabelangerbefore this change12:15
mnaseri think that whole bit can probaly be better ansible-ified12:15
pabelangeryah, we should revert until we know12:16
pabelanger    tox_executable: '{{ ansible_python.executable }} -m tox'12:16
pabelangerhas been used for a while in rdo12:16
mnaserid rather fix because a revert will break me :<12:16
mnaseri can work on the fix12:16
pabelangersure, but rdo is completely broken, so we broken them first?12:17
dpawliklet me compare both version and fix mine12:18
mnaseris all of rdo broken, or just f28 (anything that uses python3 -m tox)?12:20
pabelangerthen job that uses tox is broke12:22
pabelangerany*12:22
pabelangerlooks like tox-doc jobs for example12:22
pabelangerhttps://review.rdoproject.org/r/#/c/24803/12:22
mnasergive me 2 minutes12:24
pabelangerI might have mispoken, I'm not sure if all of rdo is broken. That looks related to fedora-28 removal12:24
pabelanger/usr/bin/python3 -m tox --version does work for me locally12:26
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Adds variable to toggle whether to revoke sudo  https://review.opendev.org/70624812:26
pabelangermaybe something limited to fedora-2812:26
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: ensure-tox: refactor to work with tox_executable  https://review.opendev.org/70625412:29
mnaserpabelanger, dpawlik: ^ have a look at this12:29
mnaserlet me know what you think and lets wait for tests12:29
pabelangerdpawlik: can you add depends-on from RDO to see if that fixes your issue?12:32
dpawlikpabelanger, k12:33
mnaseri think that is doing the right thing(tm)12:33
pabelangerbtw: https://zuul.opendev.org/t/zuul/status12:33
pabelangerthere is a paused job for 64hrs12:34
mnaserout for vacation :)12:34
mnasercorvus: would it be ok if we (vexxhost) did 3rd party ci similar to SF for zuul/zuul-jobs ?12:35
mnaserin this case, a change like this i have to manually do a depends-on to see if it works or not12:35
dpawlikhttps://review.rdoproject.org/r/#/c/24814/12:36
dpawlikmnaser, doesn't work :(12:37
mnaserpoop12:37
mnasergot a link to console/logs handy?12:37
dpawlikmnaser, https://review.rdoproject.org/zuul/stream/ff553764215a4a049a2ca02339bd26c7?logfile=console.log12:39
pabelangerI wonder if even a revert at this point would fix, it looks like tox is not installed for python3 now12:39
openstackgerritPaul Belanger proposed zuul/zuul-jobs master: Revert "ensure-tox: save tox_executable fact"  https://review.opendev.org/70625512:41
dpawlikpabelanger, mnaser: output from my PS https://review.rdoproject.org/zuul/stream/75e8f33706304d4694ccf161ec6c95fa?logfile=console.log12:42
mnaserok so i think i see whats happening here12:42
mnaserpabelanger, dpawlik: python3 does *not* have tox, but by default we try to install in py2 and not py312:42
dpawlikcan be12:43
dpawlikits f2812:43
mnaserso actually this might regress in the fact that you'll be using tox from py2 instead of py312:43
openstackgerritSimon Westphahl proposed zuul/zuul-jobs master: Add event id to emit-job-header  https://review.opendev.org/70622512:43
mnaserand i messed up12:43
mnaserand removed register: result12:43
mnaserbut this seems to be the fix12:43
mnaserone sec12:43
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: ensure-tox: refactor to work with tox_executable  https://review.opendev.org/70625412:43
mnaserdpawlik: ^ try that12:44
dpawlikrechecking...12:44
mnaserit will work but it will use py2 tox .. im not sure how we can work around dat12:44
ttxcorvus, fungi, mordred: Looks like my new pipeline is an efficient infinite loop: https://review.opendev.org/#/c/705991/112:45
ttxThis should fix it -- https://review.opendev.org/70625712:45
dpawliklets say, that if {{ ansible_python.executable }} is py3 it should use pip312:45
pabelangermnaser: dpawlik: I see the issue, tox_executable is python3 -m tox, but image doesn't have it today. But old type tox check, found it in python2, because of hardcoded 'tox'12:45
pabelangerand now, because pip2 install is first, we cannot install tox12:46
mnaserpabelanger: exactly.  and now the fix i put above will actually install tox inside py212:46
dpawlikyup12:46
mnaserso realistically it will regress to the same old behaviour12:46
mnaseryou'll be using pip212:46
pabelangerhttps://review.rdoproject.org/zuul/build/e1682e34f679428e80e2a5bc20022604/log/job-output.txt#23212:46
pabelangerso, this causes another issue12:47
pabelangeras current tox_executable is pip312:47
pabelangerbut ensure-tox now, will pip2 install12:47
pabelangerand change tox_executable under cover12:48
pabelangerso, we likely now need to expose ensure-tox python version via defaults12:48
pabelangerso jobs can force pip3 or pip212:49
mnaserpabelanger: yeah the autodetection seemed a little meh12:50
zbrhappy to see current discussions, i had the impression nobody liked my patches around these.12:50
mnaserzbr: you'll always attract attention when you break things T_T12:50
zbrmnaser: true, maybe better to be hidden in the shadows12:51
zbrlets undig https://review.opendev.org/#/c/690057/ for start12:52
mnaserzbr: interesting, i think this might solve help resolve this case12:57
zbrmnaser: you seen the date on it? ;)12:57
mnaserpfft, only 3 weeks? :P12:58
mnaserif it hasnt merge conficted yet i'd say you're doing well =P12:58
mnaserzbr: i dont know if we can assume ansible_python.executable will be the python that the user intends to run tox under though12:58
mnaserthat makes that assumption12:58
mnaserfor a while we ran ansible with py2 in opendev i think ?12:58
zbrand is a valid assumption, if anyone wants something else, they can specify tox_executable12:59
mnaserthats true, in this case thats exactly what rdo did12:59
zbrwe already to the same assumption with pip, ansible,...12:59
zbrunless overriden, any python tool is expected to use the default interpreter.12:59
* mnaser hmms12:59
mnaseri like the approach zbr has but i also feel like its step 2, step 1 is to unbreak rdo imho13:00
*** rlandy has joined #zuul13:01
mnaseri wonder if13:02
mnaservariables > facts13:02
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Adds variable to toggle whether to revoke sudo  https://review.opendev.org/70624813:03
mnaserhmm...13:03
mnaserpabelanger, dpawlik: how did things work before? what actaully installing tox?13:03
AJaegerpabelanger: did you see https://review.opendev.org/#/c/706216/ ?13:04
*** yolanda has quit IRC13:04
AJaegeroh, you did...13:04
mnaserbecause previously, it must have been using python2 tox..13:04
mnaserbecause type tox would evaluate fine and it would skip the install13:05
AJaegerpabelanger: please review again, I think you misread the change13:05
mnaserAJaeger: i also have another alternative here https://review.opendev.org/#/c/706254/13:05
AJaegermnaser: I don't see how that can work, tox_executable is unset initially, isn't it?13:06
mnaserAJaeger: defaults/main.yml is something i added to the first change13:07
mnaserthat has tox_executable predefined as "tox"13:07
mnaserso it tries tox --version (or whatever is defined) -- if that works, yay.  if it doesn't, then it installs it locally and stores that fact13:07
AJaegermnaser: ah...13:07
pabelangermnaser: this worked because tox --version returned true13:07
pabelangerbut now python3 -m tox --version doesn't13:07
pabelangeradn they have a task after ensure-tox to pip3 install -U tox13:08
mnaserpabelanger: oh that doesnt work?  how can we then pass "args" to python3 -m tox then?13:08
AJaegermnaser: Ok, removed -1 - but have no time to dig deeper into this now.13:08
mnaserwould it be like python3 -m tox -- --version or something?13:08
* AJaeger steps out again13:08
pabelangerpython2 -m tox --version would work13:08
pabelangeras the image has py2 tox13:09
pabelangerit is missing py3 tox right now13:09
mnaseroh i see your point13:09
pabelangerbecause we added tox_executable into ensure-tox, now we get overlap13:09
mnaserpabelanger: so before my change, how did py3 tox every get installed then?13:09
mnaserbecause it would detect the image had py2 tox13:09
mnaserand not install anything13:09
pabelangermnaser: https://review.rdoproject.org/zuul/build/e1682e34f679428e80e2a5bc20022604/log/job-output.txt#24713:10
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general.  https://review.opendev.org/70626213:10
pabelangerold way, py2 tox --version worked, and ignored tox_exectuable13:10
pabelangerthen, next playbook did upgrade, to then make tox_executable work13:10
pabelangerIMO, we need more refactor to allow for pip2 / pip3 now in ensure-tox, but that is going to grow into PRs that zbr proposed a while back13:11
mnaseraah ok13:11
mnaserso rdo had a thing to upgrade13:11
mnaserok i think im starting to follow a bit13:11
pabelangerIMO, we should look at what zbr did, as it is more indepth13:12
pabelangerbut, that is going to take some time to review / land13:12
*** ofosos has quit IRC13:12
pabelangerwhy I think, we should revert to working jobs, then try to add new support into ensure-tox13:12
pabelangerwhich, will break you again13:12
*** Goneri has quit IRC13:12
mnaseryeah i'm trying to find the fastest least intrusive solution for now13:13
mnaserand then we refactor into zbr work imho13:13
mnaseri wonder if it makes sense to introduce something like tox_python13:13
pabelangeras work around for you, you could set tox_executable: ~/local/bin/tox in your base job, and old way works13:13
mnaserwhich defaults to "2" (the old behaviour)13:13
mnaseror it can default to whatever it detects the system python to be13:13
mnaserand installs accordingly13:13
zbrmnaser: we have tons of python based tools, starting to create vars for each does not scale very well.13:13
zbrIn "python -m" I trust. Almost never failed me ;)13:14
openstackgerritDaniel Pawlik proposed zuul/zuul-jobs master: Fix ensure-tox issue when tox was installed on the host  https://review.opendev.org/70621613:15
*** guilhermesp has quit IRC13:16
pabelangermnaser: I am a fan of the pattern I did with https://opendev.org/windmill/ansible-role-openstacksdk/src/branch/master/defaults/main.yaml which exposed 3 install methods (git / pip / pkg) then for pip / git allowed to also install via virtualenv and select python version13:16
*** clayg has quit IRC13:16
pabelangerresulting in something like: https://opendev.org/windmill/ansible-role-openstacksdk/src/branch/master/tasks/install/pip.yaml13:16
mnaserbecause ensure-tox right now doesnt know if you want py2 or py313:17
pabelangeryah13:17
*** kklimonda has joined #zuul13:18
*** jpena|lunch is now known as jpena13:19
mnaserpabelanger, dpawlik i'm more than ready to come up with a fix to unblock y'all, i feel badz, so if we can find something, i can implement it right now13:20
pabelangermnaser: I think we need to revist https://review.opendev.org/676464/13:22
pabelangerthis is what zbr was been working on for a bit13:22
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general.  https://review.opendev.org/70626213:23
*** gmann has quit IRC13:23
mnasercould we maybe quickly gather consensus on a direction? :13:24
mnaseri like the python -m idea13:24
*** jamesmcarthur has joined #zuul13:24
*** hashar has joined #zuul13:24
*** bhavikdbavishi has quit IRC13:26
*** irclogbot_1 has quit IRC13:26
*** bhavikdbavishi has joined #zuul13:27
pabelangerI'm low on time today, head deep into ansible collection testing13:28
pabelangerbut agree, I think we need to discuss it13:29
*** irclogbot_3 has joined #zuul13:30
*** bhavikdbavishi has quit IRC13:31
*** jamesmcarthur has quit IRC13:35
*** jamesmcarthur has joined #zuul13:49
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general.  https://review.opendev.org/70626213:49
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general.  https://review.opendev.org/70626213:52
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general.  https://review.opendev.org/70626213:55
openstackgerritDaniel Pawlik proposed zuul/zuul-jobs master: [DNM] Fix ensure-tox issue when tox was installed on the host  https://review.opendev.org/70621613:58
*** Goneri has joined #zuul13:58
dpawlikpabelanger, so do we have a "smart" solution to enable gates?13:59
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general.  https://review.opendev.org/70626214:00
*** guilhermesp has joined #zuul14:01
tristanCmnaser: well it seems like moving rdo tox job to centos8 fixes the situation, but other zuul-jobs user might be affected by the change. shouldn't we revert it in that case?14:04
*** clayg has joined #zuul14:05
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general.  https://review.opendev.org/70626214:07
mnasertristanC: if we revert it, i break again, so i dont know what to do14:11
mnaseri guess it would suck to be me14:11
mnaserlols14:11
mordredmnaser: so - I just read the scollback, but I'm still pre-coffee - could you summarize for me?14:11
pabelangermnaser: you should be able to update your base job, for tox_executable for now14:11
openstackgerritDaniel Pawlik proposed zuul/zuul-jobs master: [DNM] Fix ensure-tox issue when tox was installed on the host  https://review.opendev.org/70621614:11
mnasermordred: sure!14:11
mnaserthe change we did the other day had a small behaviour change that it started using tox_executable in the command the output the tox version14:12
mnaserrdo overrides tox_executable to: `python3 -m tox` in their jobs14:12
mnasertheir image also had tox built in14:12
mnaserso when the job ran: it would do type tox >> oh ok i got tox, i'm cool, move on, dont try to install anything14:12
mnaserthen it would run tox --version (that works, even tho its py2 tox)14:13
mnaserand then the later have a little task that installed tox inside py314:13
mordredok. those steps all seem reasonable :)14:13
mnaserNOW since my change happened -- it tried to check if pip was installed (it was in the image, so no need to install)14:14
mnaserand then instead of doing tox --version, it did {{ tox_executable }} --version -- aka python3 -m venv --version14:14
mnaserorry, python -m tox --version14:14
mnaserwhich would bomb out, because it wasn't installed. and their pre fails14:14
mnasernow i came up with this -- https://review.opendev.org/#/c/706254/14:14
mordredok. so - wait a sec14:15
mordredlet me understand that bit14:15
mnaserok14:15
mordredtox_executable was set to /home/zuul/.local/bin/tox  ... OH14:15
mordredok. I got it14:15
mnaserright, we set it to that *if* we install it14:16
mordredit's not that the thing you set didn't work - it's that the thing they overrode it with now didn't work14:16
mordredyeah14:16
mnaserexactly14:16
mnaserbecause now the variable is being used14:16
mordredoh - but wait - in the original behavior before your change ...14:16
mnaseryes, it was also kinda broken. because ensure-tox wasnt actually doing anything in their case, they had another pre.yaml which installed tox in py314:16
mordredit worked because tox was pre-installed so type tox works14:17
mnasermordred: see https://review.rdoproject.org/zuul/build/e1682e34f679428e80e2a5bc20022604/log/job-output.txt#24714:17
mnasercorrect14:17
mordredif tox is pre-installed, shouldn't your new change not install anything or set tox_executable?14:17
mordredwhy didn't type tox in the role not cause the role to skip doing anything?14:18
mnasermordred: that part is fine, but because i changed tox --version to {{ tox_executable }} --version14:18
mnaserand they have that overridden14:18
mnaserit was running python3 -m tox --version (even tho tox was not installed until the task after that one)14:18
mordredOH14:19
mnaserand it fails14:19
mordredwow14:19
mnaseryes there was a behaviour change with tox_executable being a thing in there now in the command after14:19
mordredok ... SO ... can I make a stupid suggestion for a short fix while we ponder potentially a larger overall design?14:20
mnasermordred: https://review.opendev.org/#/c/706254/2/roles/ensure-tox/tasks/main.yaml this was my short fix idea14:20
mnaserbut the problem is it would still install tox inside pip2 and override the fact14:20
mnaserso i welcome other ideas :)14:20
mnaser(fyi there was also talk of a more concrete fix by using python$version -m tox everywhere instead of tox command, but thats a whole another discussion)14:21
mordredyeah - that's bigger14:21
mordredyour suggestion works and is better than my suggestion - I think let's go with yours, it'll unbreak, then we can think about this larger issue deeply14:21
mnasermordred: it doesnt fully work, because i think the set_fact might override their var?14:22
tristanCmnaser: it seems like if tox_executable is a documented rolevar, then a role shouldn't mutate rolevar, as it will likely cause unexpected behavior14:22
mnaseri dont know the order of 27 layers of  ansible vars14:22
mnaserthis whole issue is a bit of a deadlock really14:23
mordredmnaser: maybe we should just not run your chunk if tox_executable is set at all?14:23
mnaserif we dont touch role vars, then we mess with path, but if we mess with path, that's really clunky and its messing with system level stuff that can touch/break other things14:23
mnaseroooooooh that's interesting, so an assumption if tox_executable is set that means it exists14:24
mordredif the user has set tox_executable, they're saying they want to use tox that's somewhere else14:24
mordredyeah14:24
mordredand that ensure_tox doesn't need to do anything because someone wants something different14:24
mnasermordred: but that means printing the version out will break in rdo's case at least14:24
mnaserunless we omit that too14:24
mordredthat also addresses tristanC's concern about not mutating a set role var14:24
mordredomit that too14:24
mnaserso whole role is skipped if tox_executable is set14:24
mordredbasically, have ensure-tox do _nothing_ if tox_executable is set14:25
mordredyeah14:25
*** chkumar|rover is now known as raukadah14:25
mordredI think it should unbreak the largest number of people14:25
mordredwhat a FASCINATING problem first thing in the morning :)14:25
* mnaser feels badz14:25
mordreddon't feel bad - I think your original thing was solid - just none of us thought about this other thing14:26
mordred(also - I mean - we should probably take this usecase in to account so that we can make ensure-tox do what rdo wants too so they can get rid of their pre task)14:26
mnasergiven this is trivial ill push up a patch and we can discuss in reviews14:27
mordredmnaser: cool, thanks!14:27
mnasermordred: hmm, but we have tox_executable inside defaults/main.yml14:36
mnaserso we should drop that then?14:36
tristanCmnaser: can't we check if tox_executable is different from the default?14:36
mnasertristanC: so maybe define _tox_executable and tox_executable: "{{ _tox_executable }}"14:37
mnaserand condition if they're not equal14:37
mnaserya?14:37
pabelangerwe should create ensure_tox_executable, then default to tox. Then modify tox_executable if unset and tox is missing14:37
openstackgerritDaniel Pawlik proposed zuul/zuul-jobs master: [DNM] Fix ensure-tox issue when tox was installed on the host  https://review.opendev.org/70621614:38
Shrewsi thought we kinda-sorta had a policy of prefixing role vars with the role name to prevent accidental overrides?14:39
tristanCShrews: yep, but in that case, the variable is used by ensure-tox and tox roles14:39
mnaserShrews: it would make sense from an ansible pov but my idea was to help have that var used later in the actualt tox role14:39
openstackgerritSimon Westphahl proposed zuul/zuul master: wip: Add support for tracing using OpenTelemetry  https://review.opendev.org/70517014:39
openstackgerritSimon Westphahl proposed zuul/zuul master: wip: Enable tracing of trigger events  https://review.opendev.org/70628914:39
openstackgerritSimon Westphahl proposed zuul/zuul master: wip: Trace Github webhook trigger events  https://review.opendev.org/70629014:39
openstackgerritSimon Westphahl proposed zuul/zuul master: wip: Trace Gerrit trigger events  https://review.opendev.org/70629114:39
Shrewsmnaser: ah14:40
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general.  https://review.opendev.org/70626214:40
tristanCwould be nice if we could declare and document common vars such as tox_executable or zuul_work_dir, it's kind of tedius to copy paste the same rolevar documentation14:40
mnaser++14:41
dpawlikmordred, https://review.rdoproject.org/r/#/c/24814/ => this change depends on mnasar change https://review.opendev.org/#/c/706254/14:41
dpawlikmordred, so its still doesn't unlock us14:41
mordreddpawlik: yeah - I think we just figured out the flaw in the mnaser change14:43
corvusalso, i owe a test job for this -- we can probably cover all these cases14:48
mordredmnaser: I like your _tox_executable idea above14:49
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general.  https://review.opendev.org/70626214:51
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general.  https://review.opendev.org/70626214:52
mordredcorvus: https://review.opendev.org/#/c/706222 has 2x+2 but I think your opinion on it would be good14:52
corvusmordred: can you summarize the problem with 706254/2 in one line?14:53
mordredcorvus: 706254 still has tox_executable set in defaults14:53
corvusbut it defaults to 'tox' which seems to approximate the old behavior so why is that bad?14:53
corvusoh got it14:54
mordredcorvus: this is a nice first-thing-in-the-morning one isn't it?14:54
corvusnope don't got ti14:54
corvusmordred: keep going pls :)14:54
mordredcorvus: having it set means we can't do a test of "if tox_executable is set then don't run these tasks"14:55
mordredoh - wait14:55
corvusmordred: but that's not the test that it runs, it runs "does the tox_executable as set work?"14:55
mordredright - sorry - I just realized that the latest version of the patch that we discussed hasn't been written yet14:56
mordredcorvus: ok - so the summary of the problem with the version of the patch that is there ...14:57
mordredoh - nope. I don't understand why that version doens't work. let me go read more things14:59
mnasermordred, corvus: rdo has tox_executable overridden to: python3 -m tox -- so when it tries to run "python3 -m tox --version" -- it fails, because tox is not installed in python 315:00
mordredright - bit you have a rc != 0 trap in there15:00
mordredoh. duh. != 015:00
mordredk. I'm back on the same page again15:00
mnasermy "fix" in its current state, will make you end up using py2 tox15:01
mordredthat's why we were going to have the check change to if tox_executable is set, don't try to install tox - but that has the flaw that we have a default value set for tox_executable15:01
pabelangerrdo also has a later playbook in the job, that will pip3 install tox, that is why they have tox_executable: python3 -m pip15:01
mordreddo we need the tox_executable value in defaults? what does that get us?15:02
mnasermordred: not much, i just thought it was a cleaner more streamlined way15:02
corvusmnaser: because of lines 12-16 defaulting to pip (py2) if it is installed as well as py3?15:02
mordredyeah15:03
corvusmnaser: ^ in re "< mnaser> my "fix" in its current state, will make you end up using py2 tox"15:03
corvusok, so that's the problem i didn't see.  thanks :)15:03
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general.  https://review.opendev.org/70626215:03
pabelangerfor rdo, which tox originally worked, because py27 tox is installed. Then, they upgraded to py3 tox, and pre set tox_executable to that15:03
mnaserso _tox_executable: tox and tox_executable: "{{ _tox_executable }}" and be noop if _tox_executable != tox_executable ?15:04
mordredmnaser: yah - totally. I think in this case switching to if tox_executable is not defined and removing tox_executable from defaults seems cleanest15:04
mordredoh - yeah - that would work too15:04
mnaserat least that way i can detect changes i guess15:05
mordredand would prevent us from breaking people who set tox_executable to tox for some reason15:05
pabelangeryup15:05
mnaseri'd like to have a clean solution eventually once we unbreak rdo15:05
mnaserbut thats a longer discussion15:05
mordredyes. I agree - it turns out there are several variables at play here we should discuss... because "what python should tox be installed with" is a valid concept15:06
tristanCmnaser: ftr, rdo is no longer broken, the affected job got migrated to centos815:06
mordredoh - so we can take a minute and just think about the end-state we'd liek to be in then15:07
mnaserok thats reasonable15:07
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: ensure-tox: refactor to work with tox_executable  https://review.opendev.org/70625415:09
mnasermordred: for whatever its worth ^15:09
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general.  https://review.opendev.org/70626215:14
*** jamesmcarthur has quit IRC15:18
*** jamesmcarthur has joined #zuul15:19
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general.  https://review.opendev.org/70626215:22
*** hashar has quit IRC15:22
*** jamesmcarthur has quit IRC15:25
*** zxiiro has joined #zuul15:45
*** jamesmcarthur has joined #zuul15:46
openstackgerritMerged zuul/zuul master: Add Zuul's event id to Ansible inventory  https://review.opendev.org/70622215:47
*** jamesmcarthur has quit IRC15:51
*** jamesmcarthur has joined #zuul15:55
openstackgerritJeremy Stanley proposed zuul/zuul master: Flesh out the glossary significantly  https://review.opendev.org/70439116:05
*** sshnaidm is now known as sshnaidm|afk16:05
*** jpena is now known as jpena|off16:06
fungithat's ^ no longer wip as far as i'm concerned. please feel free to tear it apart16:11
Shrewsfungi: i will do my best Pelosi impersonation on it16:12
fungidomo16:13
*** mattw4 has joined #zuul16:13
*** bhavikdbavishi has joined #zuul16:20
sugaarhi, the ssh keys need by gerrit, does it need them in order to communicate with the scheduler? or is it to communicate with the workers?16:26
*** bhavikdbavishi1 has joined #zuul16:29
*** bhavikdbavishi has quit IRC16:29
*** bhavikdbavishi1 is now known as bhavikdbavishi16:29
fungithe scheduler and the mergers need an ssh key to be able to communicate with gerrit. the scheduler uses it to watch gerrit stream-events for triggering events, while the mergers use it to fetch git refs for proposed changes16:29
sugaarall right16:30
sugaarthanks16:30
fungiand just to be clear, executors generally run a convenience merger, so need it too16:30
fungii believe it's possible to configure the mergers to use https instead of ssh, though it generally makes sense to have them connect the same way the scheduler does16:33
fungihttps://zuul-ci.org/docs/zuul/reference/drivers/gerrit.html#attr-%3Cgerrit%20connection%3E.password16:34
fungithough that only talks about using https for reporting, not for fetching change refs16:36
fungiokay, this is a weird situation, i think... normally zuul should trigger a build for any job which gets modified by a proposed change even if file filters would normally discount it. in this case two jobs were altered, but a build was only triggered for one of them: https://review.opendev.org/70605916:39
fungiin the .zuul.d/centos.yaml file, kolla-ansible-centos-source-zun is being altered and a build of that happened16:40
fungibut in the .zuul.d/ubuntu.yaml file kolla-ansible-ubuntu-source-zun is being altered in the same pipeline and no build was triggered16:41
pabelangerfungi: kolla-ansible-ubuntu-source-zun is duplicated16:45
fungioh interesting16:46
pabelangerleft comment16:46
fungigood eye!16:46
pabelangerI wonder what scheduler debug says16:46
fungiso i guess that's a possible hole in how zuul auto-triggers builds for jobs being changed. if the job is listed twice then altering the second instance of it and not the first might not trigger a build automatically16:47
clarkbya I think they should stop splitting that config16:47
pabelangerfungi: yah, maybe16:47
fungiwell, in this case the job entry is duplicated within the same file anyway16:47
clarkbah16:47
clarkbwell stop duplicating unnecessarily across the board then :)16:48
fungiso i don't think the split project configuration led to that, though i agree it's harder to wrangle in general16:48
openstackgerritJames E. Blair proposed zuul/zuul master: Extract allow/disallow filter into util function  https://review.opendev.org/70614416:55
openstackgerritClark Boylan proposed zuul/nodepool master: Don't build universal wheels  https://review.opendev.org/70632517:03
clarkbI noticed ^ debugging a similar issue in openstack17:04
openstackgerritJames E. Blair proposed zuul/zuul master: Gerrit: add polling support for refs  https://review.opendev.org/70613817:08
corvusclarkb, fungi: do i understand correctly that opendev images "pip install tox" ?  or maybe pip3 install tox?  if so -- how do i know which, pip or pip3?17:18
clarkbcorvus: it runs `$DIB_PYTHON_PIP install $package` where package is tox and DIB_PYTHON_PIP is going to be pip for platform default python17:19
clarkbcentos7 == pip2, bionic and centos8 == pip317:20
corvusis there like a table somewhere with what that value is?17:20
corvusok i think that'll get me started, thx17:20
clarkbcorvus: its based on whatever the distro declares to be their default python, but I am not aware of a table for that17:22
*** yolanda has joined #zuul17:22
fungithough also tox is *supposed* to be python interpreter agnostic... but what its default interpreter selection is might still depend on which one you installed it with17:25
corvusmostly i want to know how to uninstall it17:25
yoctozeptopabelanger: thanks, I removed the duplicate - though it still does not run ubuntu https://review.opendev.org/70605917:26
yoctozepto:/17:26
clarkbcorvus: removing the bin/tox entry is probably sufficient for that?17:26
yoctozeptofungi, clarkb: ^^17:26
corvusclarkb: i'm writing a test, so i want to run ensure-tox as if it ran on a node without tox having ever been installed17:26
clarkbyoctozepto: did you collapse the pipeline configs that were split up?17:27
*** jpena|off is now known as jpena17:27
yoctozeptoclarkb: pipeline configs..?17:28
corvusyeah ps3 of https://review.opendev.org/706059 removes the dup17:28
yoctozeptoah, you mean that project is split in two files17:28
clarkbyoctozepto: everything under project in https://review.opendev.org/#/c/706059/3/.zuul.d/centos.yaml and https://review.opendev.org/#/c/706059/3/.zuul.d/ubuntu.yaml17:28
yoctozeptoit is still split in there17:28
corvusoh there were 3 instances before17:28
clarkbtry collapsing them and see if the behavior changes17:29
yoctozeptocorvus: 3?17:29
corvusoh, no those are 2 different job17:29
yoctozepto:D17:30
corvusclarkb: so your theory is that only one of the 2 project stanzas is triggering the check?17:30
yoctozeptocorvus: I proposed the same idea in #infra17:30
yoctozeptobut like 6 hours ago17:30
corvus(and the one that worked happened to be in the stanza that changed?)17:30
clarkbcorvus: ya thoguh its mostly just a "I've not see nthat before, not sure zuul will handle it the way yoctozepto expects it " hunch17:30
corvusyoctozepto: i was asleep 6 hours ago17:30
yoctozeptocorvus: yeah, no problem17:31
yoctozeptojust saying I had a similar "hunch" ;-)17:31
corvusclarkb, yoctozepto: all of the patches i have seen from yoctozepto should result in the expected behavior of both jobs running.  so hopefully this will help us confirm the bug and find a temporary workaround.17:31
yoctozeptocorvus, clarkb: https://review.opendev.org/70605917:33
yoctozeptostill the same17:33
yoctozeptoI moved two to see if it matters17:33
clarkbyoctozepto: rm the extra file entirely :)17:33
yoctozeptoclarkb: yeah, though this makes my head spin already :-)17:33
clarkband any others if there are more than 217:33
yoctozeptoclarkb: also there is 3rd one for debian17:33
yoctozeptoyeah17:33
yoctozepto:D17:33
*** evrardjp has quit IRC17:33
yoctozeptook, let's check it out in a local editor17:34
*** evrardjp has joined #zuul17:34
*** tosky has quit IRC17:34
yoctozeptoclarkb, corvus: still nah17:40
yoctozeptohttps://review.opendev.org/70605917:40
clarkbok we can rule that hunch out at least17:42
yoctozeptoclarkb: now got rid of centos one17:42
yoctozeptoand it does not run either17:42
* yoctozepto out of hunches for now17:43
yoctozepto;D17:43
corvusit's kolla-ansible-ubuntu-source-zun that's not running?17:43
yoctozeptocorvus: yeah17:43
yoctozeptomultiple files ruled out17:44
yoctozeptocommon parent ruled out17:44
corvuschecking logs17:46
corvushttps://etherpad.openstack.org/p/U82ltjzkVY17:48
yoctozeptowhat does it mean?17:49
yoctozeptoit did regular check for it17:49
corvusdon't know yet, just sharing data17:49
mordredso - just as an aside - google is totally indexing zuul job pages. I just google searched (without thinking about it) for kolla-ansible-ubuntu-source-zun ... and the first google hit was http://zuul.openstack.org/job/kolla-ansible-ubuntu-source-zun17:51
yoctozeptoyeah, the job runs normally, also for kolla17:52
yoctozeptoI was just expecting to see it here due to the "check changed" rule17:52
corvusthere's actually 2303 debug log lines related to that change (you have a lot of jobs)17:53
yoctozeptocorvus: thanks, doing our best ;-)17:53
*** saneax has quit IRC17:53
yoctozepto"where quality meets quantity" or so it goes...17:53
corvushaha17:53
mordredthat should be a t-shirt17:54
*** saneax has joined #zuul17:54
*** saneax has quit IRC17:55
*** saneax has joined #zuul17:58
openstackgerritMerged zuul/zuul master: add_host: enable using ansible_fqdn and ansible_private_key_file  https://review.opendev.org/70612418:00
openstackgerritMerged zuul/zuul master: Increase timeout of zuul-build-image  https://review.opendev.org/70568918:11
tobiashmordred: that's awesome18:21
*** jpena is now known as jpena|off18:22
corvusyoctozepto, clarkb, fungi, pabelanger, mordred: got it.  i hope you're sitting down for this.18:23
yoctozeptocorvus: just sat down and was going to ask :-)18:23
corvusyoctozepto, clarkb, fungi, pabelanger, mordred: there is a clue in the log output.  notice that the centos job ran because it was a "newly created job"18:23
yoctozeptocorvus: oh yeah, it truly was18:24
* fungi straps in18:24
corvusthat's the main difference between the two, and ultimately why it ran but not ubuntu18:24
yoctozeptothat's part of the solution, brilliant indeed18:24
yoctozeptothanks corvus18:24
corvusthe ubuntu job was not newly created, so zuul is falling back on the configuration change detection18:24
corvusyou might *think* that it should run since its configuration has changed18:24
yoctozeptonow onto why the "touched" rule does not apply here18:24
yoctozeptoyeah18:25
* fungi would think that18:25
* yoctozepto too ;-)18:25
* mordred can't wait18:25
yoctozeptothe ground is shaking as spectators await the final word from corvus18:25
AJaegerdrum roll...18:26
* yoctozepto assumes corvus just can't handle the laughter atm :-)18:27
corvus(just getting details right, sorry)18:27
fungisure, sure, i bet you're actually getting a sandwich ;)18:28
yoctozeptothen it's better to really sit down as corvus is going to drop something heavy18:29
Shrewssuch a tech tease18:29
yoctozepto(pun heavily intended)18:29
* yoctozepto starts playing heavy metal18:33
corvushrm, this isn't quite holding up as i type it up.  let me dig a bit more.18:36
corvussorry about that18:36
yoctozeptothat's exactly how it's to write an academic paper, I know the pain :-)18:38
*** dpawlik has quit IRC18:50
fungimaybe you should have gotten a sandwich after all18:51
corvusi'm going to fire up the repl; this will take a few minutes.  probably long enough for a sandwich.18:52
mordredmmm. sandwich18:55
mnaserbtw, pabelanger pointed this out a while back but earlier in the day.. http://zuul.opendev.org/t/zuul/status -- there's a job that has been paused for 70 hours18:57
*** tosky has joined #zuul18:59
fungiunrelated, we've also discovered in #openstack-infra that it's possible when setting debug on a project-pipeline in a proposed change for the resulting report to be too large to fit into the message column of the buildset table, and this seems to also prevent the gerrit reporter from reporting19:01
mordred\o/19:07
*** jamesmcarthur has quit IRC19:10
Shrewsfungi: zuul source seems to indicate that message is a TEXT column. that doesn't quite make sense19:18
Shrewsunless a packet size is being exceeded maybe?19:19
fungimaybe?19:19
fungihow wide is a text column allowed to be?19:20
Shrewsfungi: as large as you need19:20
fungineat19:20
mordrednope19:20
mordredTEXT – 64KB (65,535 characters)19:20
Shrewslimited by memory and max_allowed_packet19:20
Shrewsorly?19:20
fungiyeah, so the message is >2^16 which makes sense if mordred is to be believed19:21
fungi'(pymysql.err.DataError) (1406, "Data too long for column \'message\' at row 1")19:21
*** bhavikdbavishi has quit IRC19:21
fungi... (65967 characters truncated) ...19:21
Shrewsok, that makes more sense then and blames my own failing RAM19:22
mordredhttps://dev.mysql.com/doc/refman/8.0/en/storage-requirements.html#data-types-storage-reqs-strings19:22
fungiproblem is, this is with all project-templates and all jobs but the one in question commented out of the project-pipeline19:22
fungiso not sure how to actually get the debug report for the change19:22
Shrewsmordred: but no one could ever possibly need more than 64K19:23
mordredShrews: that19:23
fungithanks, bill19:23
Shrewsthat's basically infinity19:23
johnsommysql's column storage is... interesting19:24
fungii wonder if we could just truncate the message ourselves when storing? or still report to gerrit even if the mysql reporter fails?19:24
Shrewsfungi: might be better/easier/lazier to change to MEDIUMTEXT/LONGTEXT19:24
fungithat sounds reasonable, but presumably needs a migration19:25
Shrewsyah19:25
fungii mean, 64k seems like not that much when you can have lengthy debug reports19:25
johnsomThe interesting part is this job overflows that field even without debug on19:25
fungioh, you're not getting any results reported for the change even before adding debug:true?19:27
fungii'll look back at an older patchset's logging in that case19:27
johnsomRight, the last two runs had the debug line commented out. The failures prior to Andreas asking for debug didn't have the line at all19:27
corvusfungi: also, btw, all the same info should be in the scheduler debug log, so you can dig it out for johnsom even with the reporter failure19:28
corvusfungi, clarkb, mordred, yoctozepto, pabelanger: okay, re the kolla job: the reason zuul doesn't think that there is a job configuration change for the ubuntu zun job is a bug in zuul.  it maintains both a text and regex form of the filematcher in the job data structure, and it uses the text version to compare for differences.  however, the text version (which is only used for this and rest/json api19:28
corvusserialization) is not updated when variants are applied.  so basically it didn't see the difference in this check (but the actual file matchers which are stored in another attribute work as expected).  this is a relatively straightforward fix, and we should do it so that the json serialization of jobs is correct.  however, we may want to consider breaking it again, but intentionally this time -- we might19:28
corvusdecide that changing a file matcher shouldn't automatically run a job since it doesn't actually change how the job is run.19:28
fungioh, good point, i probably have to find the buildset id and then grep for that instead of the change id19:28
corvusfungi: grep for the event id19:29
fungicorvus: agreed on only changing a file matcher not triggering a build, rerunning that job doesn't actually test anything19:30
corvusfungi: the "e:" at the start of the log line -- get that by grepping for the buildset19:30
johnsomThe last change was 706329,1419:30
corvusi'll propose the fix (and intentional re-break) after lunch19:33
fungijohnsom: yep, trimming off the leader from all the matching loglines may still be too large for paste.o.o but i'll try to link something in #openstack-infra in a moment19:33
fungicorvus: thanks!!!19:33
johnsom+119:34
yoctozeptocorvus: thanks, let's first see if the fix helps and then break again, totally fine by me19:38
corvusyoctozepto: oh, we'll know.  there will be a test :)19:38
yoctozeptowell "break" as it makes sense not to run it in this particular case19:39
corvusianw, fungi, AJaeger: there's a notable absence in this change: https://review.opendev.org/695828 -- the change did not add the header to the update script.19:39
yoctozeptocorvus: I'll restore ps3 to get it back in order19:39
yoctozeptoif you don't mind19:39
corvusyoctozepto: yep, all done, thanks!19:39
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: WIP: add an ensure-tox test job  https://review.opendev.org/70637119:40
corvusianw, fungi, AJaeger: see ^19:40
corvusmnaser: ^ that change is a start on the ensure-tox test19:40
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: WIP: add an ensure-tox test job  https://review.opendev.org/70637119:42
yoctozeptobtw, is there an easier way to restore ps than review -d, amend and review?19:43
*** armstrongs has joined #zuul19:43
AJaegeryoctozepto: git review -d X,3 ; git review19:44
AJaeger(assuming you want to go back to 3)19:44
mnasercorvus: i know a lot of text got lost in the big scrollback, but i was wondering if it might be ok for us (vexxhost) to do third party ci on zuul/zuul-jobs19:44
yoctozeptoAJaeger: yeah, I do it with amend inbetween cause this rejects it with "already present"19:44
fungiyoctozepto: AJaeger: you may need to rebase or make a minor alteration however, as gerrit won't allow a new patchset with the same exact git sha as an earlier patchset19:44
yoctozeptoexactly19:44
fungior even amend, yeah19:44
yoctozeptoso my procedure really is minimal19:45
fungishould be sufficient19:45
yoctozeptothanks, folks19:45
AJaegerfungi, I think it worked for me without changes - but have nothing to test right now...19:45
AJaegerwhen I pushed back the old change, it even restored the votes ;)19:45
AJaegercorvus: are you going to add the autogenerated lines or shall I?19:46
*** michael-beaver has joined #zuul19:49
mordredmnaser: that seems like a good idea to me. software factory also does a 3pci on zuul-jobs19:52
AJaegerargh, adding those lines in is too tricky for me for tonight, hope ianw can take it on...19:52
*** armstrongs has quit IRC19:53
*** Goneri has quit IRC19:54
*** openstackstatus has joined #zuul19:57
*** ChanServ sets mode: +v openstackstatus19:57
mnasermordred: yep i saw that, so i just figured i wanted to ask if it was ok before doing it :)19:58
mordredmnaser: ++19:59
openstackgerritDavid Shrewsbury proposed zuul/zuul master: Alter message column type in buildset table  https://review.opendev.org/70637720:14
Shrewsfungi: mordred: it may be as simple as that? ^^20:15
Shrewslooks like the sql reporter automatically runs the migrations on load20:15
clarkbShrews: we should check if that is valid in postgres20:17
Shrewsclarkb: oh good point20:17
clarkba quick google shows postgres may just have "text" but unsure if they'll automagic the mediumtext to text20:17
clarkb(I think tristanC is using postgres if we need someone to confirm in the wild)20:19
Shrewsclarkb: yeah, not seeing support for that, just TEXT20:19
Shrewshttps://docs.sqlalchemy.org/en/13/dialects/postgresql.html#postgresql-data-types20:19
Shrewswelp, so much for that20:19
Shrewsmaybe fungi's suggestion to trim the message is the route to take20:20
clarkbwe can probably check the connection type and do the migration if mysql else noop20:20
clarkb(we'll still want to record that we migrated just do nothing in the postgres case)20:20
*** hashar has joined #zuul20:21
Shrewsclarkb: that only half solves the problem though20:21
clarkbShrews: postgres' text type isn't size limited20:21
clarkb(at least that is what googling says)20:22
Shrewsclarkb: oh, well maybe that's what i was recalling earlier  :)20:22
ianwcorvus: hrm ... i feel sure i did.  i also remember rebasing things several times when working on that and i guess it must have got lost in a merge at some point, let me see20:23
Shrewsi think only doing the migration for one db would unnecessarily complicate things due to having to keep the ORM model in sync with the db20:23
clarkbShrews: yes i think you still mark postgres as having migrated, you just don't do anything20:24
clarkb(I'm guessing that a function that passes is sufficient but have never tried that with alembic)20:25
Shrewsclarkb: i mean the change needed in https://review.opendev.org/#/c/706377/1/zuul/driver/sql/sqlconnection.py20:25
clarkbya that would have to be conditional too20:26
Shrewsif using mysql, that needs to be sa.MEDIUMTEXT. if using pg, it should remain sa.TEXT20:26
clarkbyup20:26
clarkbI'm sure we can introspect that from sql_alchemy but not sure how20:26
clarkbself.engine20:27
clarkbif mysql in self.engine.driver: text_type=sa.MEDIUMTEXT else text_type=sa.TEXT ?20:29
clarkbthere is more than one mysql driver so that test needs to be flexible enough to catch them20:29
*** jamesmcarthur has joined #zuul20:38
*** gothicmindfood has quit IRC20:51
*** jlk has quit IRC20:51
*** irclogbot_3 has quit IRC20:53
*** irclogbot_1 has joined #zuul20:55
*** rlandy is now known as rlandy|afk21:00
*** Goneri has joined #zuul21:11
*** Goneri has quit IRC21:18
corvusmnaser: ++3pci on zuul-jobs21:31
corvusmnaser: (though, for things like ensure-tox, we ought to be able to cover most of the cases in-repo)21:31
corvusShrews: are we sure we want to store >64k?21:32
corvusit, erm, you know, sounds like a lot :)21:32
Shrewscorvus: nope21:33
Shrewsi think trimming is just as viable, imo21:34
corvusShrews: k.  i'll just throw it out there that i think it's a valid question and i kindasorta lean toward the answer may be to trim but am totally open to whatever other ppl think21:34
Shrewsagreed 64k seems like way more than you'd need. i have no idea the content they're trying to store there21:35
corvusShrews: it's usually <100 bytes, like this: https://zuul.opendev.org/t/zuul/buildset/5505f6b6f9804f289c7faadf289bb0e321:35
corvuswith debug on, we might expect like 50-100 lines.  i have no idea how we got to 64k.21:36
corvus(i'd bet a nickel on some bug that would be obvious if we had even a truncated sample)21:36
*** jamesmcarthur has quit IRC21:36
Shrewsyeah. maybe we should look at the actual content to see what's happening21:36
Shrewsfungi: do you have that sample you referred to earlier? (if you aren't afloat somewhere)21:37
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: WIP: add an ensure-tox test job  https://review.opendev.org/70637121:41
*** hashar has quit IRC21:42
clarkbI think truncating is probably fine too21:43
clarkbShrews: is that 65k characters or 65k bytes?21:43
clarkbbecause utf8 is 4 bytes per character would get us closer to 12k characters21:43
clarkbeasier to hit that limit I expect21:43
fungiShrews: i can get that sample again, yes21:50
fungiShrews: the "content" in this case was a debug report21:51
fungiwhich is why it was so large21:51
mnasercorvus: ok cool, i may have something in the way tomorrow / this weekend.21:51
fungiShrews: though as the report was truncated by the db driver, i don't exactly know why it was so very, very large21:52
fungiafter the db rejected it, the buildset went on to not report to gerrit either21:52
*** jlk has joined #zuul21:54
*** Goneri has joined #zuul21:56
corvusclarkb, Shrews: oh, is that the way it's stored?  /me mumbles something about drizzle fixing this21:57
clarkbcorvus: ya confirmed its 64KB not characters21:58
clarkbso utf8 will be 1/4 that21:58
corvusclarkb: utf32 is 4 bytes per char, but yeah21:59
fungibest to truncate just shy of 16kb just to make sure21:59
clarkbcorvus: mysql doesn't do variable length utf8, mysql utf8 is 4 bytes per char (or 3 if you do the 3 byte encoding instead)21:59
corvusclarkb: that is not exactly how i read https://dev.mysql.com/doc/refman/8.0/en/charset-unicode.html  but this may vary by version?22:02
corvusnot sure what opendev is using right now22:02
clarkbcorvus: ah it is variable length but not when it comes to calculating index sizes22:04
clarkb(index sizes is the place I've run into this)22:04
clarkbwith index sizes they assume the max char byte size22:05
corvusah, makes sense22:06
fungioh, right, since that can't be predicted22:06
corvusso if this holds for our version, in practice it shouldn't be an issue and we can assume char=byte and go with 64k22:06
fungiso they go with the upper-bound22:06
clarkbcorvus: yup22:06
fungionly an issue if zuul decides to report a bunch of snowman22:06
clarkbfungi: Gerrit Frozen mode: Code review with your 4 year olds22:07
clarkband now I've got that song stuck in my head22:07
fungialso if truncating, we can probably use one of the stdlib unicodedata modules to calculate the byte length22:07
*** tjgresha has joined #zuul22:08
clarkbfungi: we can also truncate after converting to a byte type? though I'm not sure if sqlalchemy will accept that for input on a text type22:09
fungiyeah, i meant if sqla wants it in unicode string form22:10
funginevermind, i was thinking of where i've used the unicodedata module to determine aggregate glyph widths for line wrapping22:14
fungiwe care about encoded width not displayed width22:14
fungiwe could len(message.encode('utf-8')) but the trick is finding a spot to truncate it without splitting within a character22:15
clarkbya22:15
fungialso we'd have to guess at what sort of normalizing any of the layers might decide to perform22:15
clarkbwe could also just assume the max length like indexes do and that would be safe and still give us ~11k characters22:16
fungicompromise: catch the exception and retry truncated to what we think is safe22:16
openstackgerritJames E. Blair proposed zuul/zuul master: Don't run jobs if only their file matchers are updated  https://review.opendev.org/70639922:17
corvusyoctozepto: the bug you found ^ thanks :)22:17
fungialready reading it22:17
corvusclarkb: i use the word 'frozen' a lot in that commit message22:18
clarkbcorvus: you would be a hero in my household22:18
corvusbut it's always next to the word 'job' which is boring22:19
clarkbI once said "else" and that was enough to have the full attention of my children22:19
corvushaha22:19
clarkbbecause it is close enough to "Elsa" and they heard "Elsa"22:19
fungijust wait until they find out you know an olaph22:20
*** tjgresha has quit IRC22:20
*** yolanda has quit IRC22:24
clarkbthey would immediately demand we go visit22:25
corvusand would be rewarded with farm animals22:26
*** mattw4 has quit IRC22:29
*** Goneri has quit IRC22:31
*** rlandy|afk is now known as rlandy22:32
*** mattw4 has joined #zuul22:38
ianwcorvus: i can't replicate that comment issue that your change has exhibited -- ruamel should be preserving comments, and when i add the job and run ./tools/update-test-platforms.py it doesn't wipe out all the other files comments?22:38
corvusianw: maybe i have an old/defective version of ruamel?22:39
ianwhrm, it's vendored in ./tools22:40
corvusi don't think it is, only the helper methods in ruamellib22:41
corvusi'm running 0.15.34-1 from ubuntu22:42
ianwhttp://paste.openstack.org/show/789249/22:42
*** avass has quit IRC22:44
ianwoohhh, right, ruamellib is a wrapper22:45
ianw0.16.5 from f3122:46
corvusi'll try in a venv22:47
corvusianw: yeah, that works better22:48
corvus0.16.722:48
corvusso the answer is: don't run with old ruamel22:48
ianwok, maybe we should put a tox env in for it22:48
corvusi wonder if we can check the version in the script22:48
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: WIP: add an ensure-tox test job  https://review.opendev.org/70637122:49
clarkbany other zuulians want to review (and hopefully approve) an easy nodepool change https://review.opendev.org/#/c/706325/ that stops us building universal wheels for a python3 only project22:56
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Add tox env for update-test-platforms  https://review.opendev.org/70640422:59
clarkbhttps://review.opendev.org/#/c/705755/3 has a bit more to it but I'm sure will make its users happy too. This adds ebs-optimized flag support for ec2 instances23:04
*** Goneri has joined #zuul23:12
corvusSpamapS: if you or someone from your org has a minute to review an aws change, that'd be great: https://review.opendev.org/70575523:13
clarkbcorvus: can you check my comments on https://review.opendev.org/#/c/706138/2 it isn't clear to me how the testing exercises the code that was changed23:28
clarkb(because there was no connection config updates to poll those oprojects)23:28
corvusclarkb: responded23:33
corvusclarkb: and added a comment23:35
clarkbcorvus: gotcha so self.connection.projects comes from the zuul tenant config?23:35
clarkbI think that was the bit of info I was missing23:36
corvusyep.23:36
openstackgerritMerged zuul/nodepool master: Don't build universal wheels  https://review.opendev.org/70632523:36
clarkbfungi: left some thoughts on https://review.opendev.org/#/c/704391/3 I'd be happy with followups for Shrews' and my comments too if we want to just land what we've got23:38
fungii saw, thanks! i'll spin another, but a proposed definition for "pipeline manager" would be appreciated23:41
fungithis still needs cross-linking of terms, though i'm not sure yet if i should do that in the same change or make a separate one for simplicity23:42
fungii'm guessing by pipeline manager, you mean the archetypal algorithms we use to determine queuing order, windowing, coalescence and implicit dependency23:43
fungi(dependent, independent, supersedent...)23:43
fungiif so, i can take a stab at it23:44
clarkbcorvus: left thoughts on https://review.opendev.org/#/c/705856/3 now as well. I +2'd though if you want to approve. I think the proposed short circuit and test simplification can happen in followups23:45
clarkbfungi: yup that23:48
corvusi'm having a ball reviewing fungi's change!23:49
clarkbfungi: I believe they are documented already. Something like Pipeline Manager: pipeline configuration item that determines the queuing behavior of events. They determine if events are queued together or independent or if they should be superceded entirely.23:50
fungiwfm23:52
corvusfungi, clarkb: i got in on reviewing 704391 too.  it looks great, and i have enjoyed "helping".23:52
fungiand yeah, i really don't mind folks proposing entirely different definitions for the terms i'm adding there, i mostly just wanted good placeholders as a starting point to build on23:53
corvusoh i have nothing other than editorial nits to offer :)23:53
fungialso fine ;)23:53
*** tosky has quit IRC23:55
corvusi'm very grateful to be provided the opportunity to further the cause of establishing the "Maine comma".  (when you feel you have to add an oxford comma just to eliminate purposeful misreadings)23:56
fungiheh23:56
fungiyeah, i never got in the oxford habit. have to remind myself when style guides call for it23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!