*** sshnaidm is now known as sshnaidm|afk | 00:08 | |
*** igordc has quit IRC | 00:08 | |
*** armstrongs has joined #zuul | 00:12 | |
*** decimuscorvinus_ has joined #zuul | 00:16 | |
*** dmellado_ has joined #zuul | 00:16 | |
*** paulalbertella has joined #zuul | 00:17 | |
*** wxy-xiyuan has joined #zuul | 00:20 | |
*** decimuscorvinus has quit IRC | 00:21 | |
*** dmellado has quit IRC | 00:21 | |
*** reiterative has quit IRC | 00:22 | |
*** dmellado_ is now known as dmellado | 00:22 | |
*** Miouge has quit IRC | 00:22 | |
*** armstrongs has quit IRC | 00:22 | |
*** Miouge has joined #zuul | 00:22 | |
*** johanssone has quit IRC | 00:23 | |
*** dustinc has quit IRC | 00:25 | |
*** mnasiadka has quit IRC | 00:25 | |
*** dustinc has joined #zuul | 00:25 | |
*** zxiiro has quit IRC | 00:25 | |
*** andreaf has quit IRC | 00:25 | |
*** evgenyl has quit IRC | 00:25 | |
*** Shrews has quit IRC | 00:25 | |
*** stevthedev has quit IRC | 00:25 | |
*** donnyd has quit IRC | 00:25 | |
*** logan- has quit IRC | 00:25 | |
*** andreaf has joined #zuul | 00:25 | |
*** evgenyl has joined #zuul | 00:25 | |
*** mnasiadka has joined #zuul | 00:25 | |
*** donnyd has joined #zuul | 00:25 | |
*** Shrews has joined #zuul | 00:25 | |
*** stevthedev has joined #zuul | 00:26 | |
*** johanssone has joined #zuul | 00:26 | |
*** zxiiro has joined #zuul | 00:26 | |
*** zbr has quit IRC | 00:27 | |
*** logan- has joined #zuul | 00:27 | |
*** mattw4 has quit IRC | 00:28 | |
*** openstackstatus has quit IRC | 00:28 | |
*** zbr has joined #zuul | 00:31 | |
*** jamesmcarthur has joined #zuul | 01:12 | |
*** jhesketh has quit IRC | 01:13 | |
*** jhesketh has joined #zuul | 01:14 | |
*** jamesmcarthur has quit IRC | 01:17 | |
*** jamesmcarthur has joined #zuul | 01:29 | |
*** jamesmcarthur has joined #zuul | 01:53 | |
*** jamesmcarthur has quit IRC | 02:01 | |
*** jamesmcarthur has joined #zuul | 02:02 | |
*** jamesmcarthur has quit IRC | 02:08 | |
*** jamesmcarthur has joined #zuul | 02:29 | |
*** bhavikdbavishi has joined #zuul | 02:48 | |
*** bhavikdbavishi1 has joined #zuul | 02:52 | |
*** bhavikdbavishi has quit IRC | 02:54 | |
*** bhavikdbavishi1 is now known as bhavikdbavishi | 02:54 | |
*** jamesmcarthur has quit IRC | 03:01 | |
*** jamesmcarthur has joined #zuul | 03:03 | |
*** jamesmcarthur has quit IRC | 03:08 | |
*** jamesmcarthur has joined #zuul | 03:18 | |
*** jamesmcarthur has quit IRC | 03:44 | |
*** jamesmcarthur has joined #zuul | 03:44 | |
*** jamesmcarthur has quit IRC | 03:50 | |
*** jamesmcarthur has joined #zuul | 04:00 | |
*** jamesmcarthur has quit IRC | 04:03 | |
*** jamesmcarthur has joined #zuul | 04:05 | |
*** jamesmcarthur has quit IRC | 04:10 | |
*** jamesmcarthur has joined #zuul | 04:35 | |
*** zxiiro has quit IRC | 05:11 | |
*** jamesmcarthur has quit IRC | 05:14 | |
*** raukadah is now known as chkumar|rover | 05:18 | |
*** rlandy|bbl has quit IRC | 05:25 | |
*** evrardjp has quit IRC | 05:33 | |
*** evrardjp has joined #zuul | 05:34 | |
*** bolg has joined #zuul | 05:36 | |
openstackgerrit | Jan Kubovy proposed zuul/zuul master: WIP: Store unparsed branch config in Zookeeper https://review.opendev.org/705716 | 05:37 |
---|---|---|
*** paulalbertella has quit IRC | 05:39 | |
*** reiterative has joined #zuul | 05:39 | |
*** swest has joined #zuul | 05:41 | |
*** yolanda has quit IRC | 06:01 | |
*** bhavikdbavishi has quit IRC | 06:02 | |
*** bhavikdbavishi has joined #zuul | 06:02 | |
*** felixedel has joined #zuul | 06:14 | |
*** felixedel has quit IRC | 06:18 | |
*** saneax has joined #zuul | 06:20 | |
*** fdegir has quit IRC | 06:20 | |
*** felixedel has joined #zuul | 06:20 | |
*** fdegir has joined #zuul | 06:20 | |
*** felixedel has quit IRC | 07:01 | |
*** felixedel has joined #zuul | 07:07 | |
*** felixedel has quit IRC | 07:12 | |
openstackgerrit | Felix Schmidt proposed zuul/zuul master: Implement github checks API https://review.opendev.org/705168 | 07:16 |
*** jpena|off is now known as jpena | 07:54 | |
*** yolanda has joined #zuul | 08:19 | |
*** saneax has quit IRC | 08:22 | |
*** saneax has joined #zuul | 08:23 | |
openstackgerrit | Felix Schmidt proposed zuul/zuul master: Implement github checks API https://review.opendev.org/705168 | 08:30 |
*** sugaar has joined #zuul | 08:31 | |
openstackgerrit | Jan Kubovy proposed zuul/zuul master: WIP: Store unparsed branch config in Zookeeper https://review.opendev.org/705716 | 08:35 |
*** tosky has joined #zuul | 08:40 | |
*** sshnaidm|afk is now known as sshnaidm | 08:54 | |
*** bhavikdbavishi has quit IRC | 09:14 | |
ofosos | Hey, can someone help me with this threaddump? Paste.debian.net/1129400/ | 09:50 |
ofosos | This is the Bitbucket driver after running for five days. Is it possible that all my threads disappeared, because of an uncaught exception? | 09:51 |
ofosos | Well, the first part is from shortly after the start, the second part is after the breakage | 10:01 |
mnaser | ofosos: i dont know enough about these deep internals but could it be possible that something happened to your zk connection? | 10:07 |
mnaser | ofosos: anything else in the rest of the logs that can be helpful? | 10:07 |
openstackgerrit | Daniel Pawlik proposed zuul/zuul-jobs master: Fix ensure-tox issue when tox was installed on the host https://review.opendev.org/706216 | 10:22 |
*** apevec has joined #zuul | 10:33 | |
openstackgerrit | Simon Westphahl proposed zuul/zuul master: Add Zuul's event id to Ansible inventory https://review.opendev.org/706222 | 10:33 |
openstackgerrit | Daniel Pawlik proposed zuul/zuul-jobs master: Fix ensure-tox issue when tox was installed on the host https://review.opendev.org/706216 | 10:35 |
openstackgerrit | Simon Westphahl proposed zuul/zuul-jobs master: Add event id to emit-job-header https://review.opendev.org/706225 | 10:37 |
openstackgerrit | Daniel Pawlik proposed zuul/zuul-jobs master: Fix ensure-tox issue when tox was installed on the host https://review.opendev.org/706216 | 10:43 |
*** bolg_ has joined #zuul | 10:52 | |
*** bolg has quit IRC | 10:52 | |
*** bolg_ has quit IRC | 10:55 | |
openstackgerrit | Daniel Pawlik proposed zuul/zuul-jobs master: Fix ensure-tox issue when tox was installed on the host https://review.opendev.org/706216 | 11:07 |
*** dpawlik has joined #zuul | 11:07 | |
dpawlik | AJaeger, hi. Changed commit msg: https://review.opendev.org/#/c/706216/ | 11:08 |
dpawlik | AJaeger, example of failing job: https://review.rdoproject.org/zuul/build/148f86eb24a6474aa729000a55a7c5ac | 11:09 |
*** bhavikdbavishi has joined #zuul | 11:32 | |
*** bhavikdbavishi1 has joined #zuul | 11:35 | |
*** bhavikdbavishi has quit IRC | 11:37 | |
*** bhavikdbavishi1 is now known as bhavikdbavishi | 11:37 | |
*** avass has joined #zuul | 11:46 | |
*** apevec has quit IRC | 11:48 | |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds variable to toggle whether to revoke sudo https://review.opendev.org/706248 | 11:58 |
*** jpena is now known as jpena|lunch | 12:00 | |
*** wxy-xiyuan has quit IRC | 12:00 | |
*** rfolco has joined #zuul | 12:05 | |
mnaser | dpawlik: ah crap good catch | 12:06 |
mnaser | dpawlik: damnit this was my fault | 12:07 |
mnaser | i am pretty sure i forgot to git add defaults/main.yml | 12:07 |
dpawlik | mnaser, oh, cool solution. I didn't think about such way | 12:08 |
mnaser | wait, there is `tox_executable: tox` | 12:08 |
mnaser | in defaults | 12:08 |
mnaser | i didnt forget.. | 12:08 |
dpawlik | mnaser, I was performing a test of the playbook on local VM | 12:08 |
mnaser | dpawlik: i guess the functional change is that `tox_executable` is actually being used now in that role | 12:09 |
mnaser | so maybe you have that overridden before? | 12:09 |
dpawlik | mnaser, let me check on Fedora 28 (its mostly failing on that iamge) | 12:09 |
mnaser | before it would do tox and now it uses `tox_executable` so *maybe* someone override that in your environment (because otherwise i would imagine opendev would have blew up) | 12:09 |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds variable to toggle whether to revoke sudo https://review.opendev.org/706248 | 12:10 |
dpawlik | mnaser, let me check one more time | 12:11 |
dpawlik | https://logserver.rdoproject.org/80/24780/1/gate/tox-docs/148f86e/job-output.txt 2020-02-05 18:18:36.937726 | 12:12 |
dpawlik | so its different problem. Checking | 12:12 |
pabelanger | /usr/bin/python3 -m tox --version that seems wrong is that because tox_executable is "/usr/bin/python3 -m tox" ? | 12:13 |
pabelanger | yes | 12:13 |
pabelanger | https://logserver.rdoproject.org/80/24780/1/gate/tox-docs/148f86e/zuul-info/inventory.yaml | 12:13 |
pabelanger | so, there is some logic errors | 12:13 |
*** pcaruana has quit IRC | 12:13 | |
pabelanger | the type tox || pip install foo | 12:13 |
pabelanger | should be | 12:13 |
pabelanger | type: {{ tox_executable || || pip install tox --user | 12:13 |
pabelanger | type: {{ tox_executable }} || pip install tox --user | 12:13 |
mnaser | i wonder if that will always pass though | 12:14 |
mnaser | because python3 would be there but tox would not be | 12:14 |
mnaser | in dpawlik case at least | 12:14 |
mnaser | almost like we should do tox version check instead of type | 12:14 |
pabelanger | we need to look at old rdo job | 12:15 |
pabelanger | before this change | 12:15 |
mnaser | i think that whole bit can probaly be better ansible-ified | 12:15 |
pabelanger | yah, we should revert until we know | 12:16 |
pabelanger | tox_executable: '{{ ansible_python.executable }} -m tox' | 12:16 |
pabelanger | has been used for a while in rdo | 12:16 |
mnaser | id rather fix because a revert will break me :< | 12:16 |
mnaser | i can work on the fix | 12:16 |
pabelanger | sure, but rdo is completely broken, so we broken them first? | 12:17 |
dpawlik | let me compare both version and fix mine | 12:18 |
mnaser | is all of rdo broken, or just f28 (anything that uses python3 -m tox)? | 12:20 |
pabelanger | then job that uses tox is broke | 12:22 |
pabelanger | any* | 12:22 |
pabelanger | looks like tox-doc jobs for example | 12:22 |
pabelanger | https://review.rdoproject.org/r/#/c/24803/ | 12:22 |
mnaser | give me 2 minutes | 12:24 |
pabelanger | I might have mispoken, I'm not sure if all of rdo is broken. That looks related to fedora-28 removal | 12:24 |
pabelanger | /usr/bin/python3 -m tox --version does work for me locally | 12:26 |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds variable to toggle whether to revoke sudo https://review.opendev.org/706248 | 12:26 |
pabelanger | maybe something limited to fedora-28 | 12:26 |
openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-tox: refactor to work with tox_executable https://review.opendev.org/706254 | 12:29 |
mnaser | pabelanger, dpawlik: ^ have a look at this | 12:29 |
mnaser | let me know what you think and lets wait for tests | 12:29 |
pabelanger | dpawlik: can you add depends-on from RDO to see if that fixes your issue? | 12:32 |
dpawlik | pabelanger, k | 12:33 |
mnaser | i think that is doing the right thing(tm) | 12:33 |
pabelanger | btw: https://zuul.opendev.org/t/zuul/status | 12:33 |
pabelanger | there is a paused job for 64hrs | 12:34 |
mnaser | out for vacation :) | 12:34 |
mnaser | corvus: would it be ok if we (vexxhost) did 3rd party ci similar to SF for zuul/zuul-jobs ? | 12:35 |
mnaser | in this case, a change like this i have to manually do a depends-on to see if it works or not | 12:35 |
dpawlik | https://review.rdoproject.org/r/#/c/24814/ | 12:36 |
dpawlik | mnaser, doesn't work :( | 12:37 |
mnaser | poop | 12:37 |
mnaser | got a link to console/logs handy? | 12:37 |
dpawlik | mnaser, https://review.rdoproject.org/zuul/stream/ff553764215a4a049a2ca02339bd26c7?logfile=console.log | 12:39 |
pabelanger | I wonder if even a revert at this point would fix, it looks like tox is not installed for python3 now | 12:39 |
openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: Revert "ensure-tox: save tox_executable fact" https://review.opendev.org/706255 | 12:41 |
dpawlik | pabelanger, mnaser: output from my PS https://review.rdoproject.org/zuul/stream/75e8f33706304d4694ccf161ec6c95fa?logfile=console.log | 12:42 |
mnaser | ok so i think i see whats happening here | 12:42 |
mnaser | pabelanger, dpawlik: python3 does *not* have tox, but by default we try to install in py2 and not py3 | 12:42 |
dpawlik | can be | 12:43 |
dpawlik | its f28 | 12:43 |
mnaser | so actually this might regress in the fact that you'll be using tox from py2 instead of py3 | 12:43 |
openstackgerrit | Simon Westphahl proposed zuul/zuul-jobs master: Add event id to emit-job-header https://review.opendev.org/706225 | 12:43 |
mnaser | and i messed up | 12:43 |
mnaser | and removed register: result | 12:43 |
mnaser | but this seems to be the fix | 12:43 |
mnaser | one sec | 12:43 |
openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-tox: refactor to work with tox_executable https://review.opendev.org/706254 | 12:43 |
mnaser | dpawlik: ^ try that | 12:44 |
dpawlik | rechecking... | 12:44 |
mnaser | it will work but it will use py2 tox .. im not sure how we can work around dat | 12:44 |
ttx | corvus, fungi, mordred: Looks like my new pipeline is an efficient infinite loop: https://review.opendev.org/#/c/705991/1 | 12:45 |
ttx | This should fix it -- https://review.opendev.org/706257 | 12:45 |
dpawlik | lets say, that if {{ ansible_python.executable }} is py3 it should use pip3 | 12:45 |
pabelanger | mnaser: dpawlik: I see the issue, tox_executable is python3 -m tox, but image doesn't have it today. But old type tox check, found it in python2, because of hardcoded 'tox' | 12:45 |
pabelanger | and now, because pip2 install is first, we cannot install tox | 12:46 |
mnaser | pabelanger: exactly. and now the fix i put above will actually install tox inside py2 | 12:46 |
dpawlik | yup | 12:46 |
mnaser | so realistically it will regress to the same old behaviour | 12:46 |
mnaser | you'll be using pip2 | 12:46 |
pabelanger | https://review.rdoproject.org/zuul/build/e1682e34f679428e80e2a5bc20022604/log/job-output.txt#232 | 12:46 |
pabelanger | so, this causes another issue | 12:47 |
pabelanger | as current tox_executable is pip3 | 12:47 |
pabelanger | but ensure-tox now, will pip2 install | 12:47 |
pabelanger | and change tox_executable under cover | 12:48 |
pabelanger | so, we likely now need to expose ensure-tox python version via defaults | 12:48 |
pabelanger | so jobs can force pip3 or pip2 | 12:49 |
mnaser | pabelanger: yeah the autodetection seemed a little meh | 12:50 |
zbr | happy to see current discussions, i had the impression nobody liked my patches around these. | 12:50 |
mnaser | zbr: you'll always attract attention when you break things T_T | 12:50 |
zbr | mnaser: true, maybe better to be hidden in the shadows | 12:51 |
zbr | lets undig https://review.opendev.org/#/c/690057/ for start | 12:52 |
mnaser | zbr: interesting, i think this might solve help resolve this case | 12:57 |
zbr | mnaser: you seen the date on it? ;) | 12:57 |
mnaser | pfft, only 3 weeks? :P | 12:58 |
mnaser | if it hasnt merge conficted yet i'd say you're doing well =P | 12:58 |
mnaser | zbr: i dont know if we can assume ansible_python.executable will be the python that the user intends to run tox under though | 12:58 |
mnaser | that makes that assumption | 12:58 |
mnaser | for a while we ran ansible with py2 in opendev i think ? | 12:58 |
zbr | and is a valid assumption, if anyone wants something else, they can specify tox_executable | 12:59 |
mnaser | thats true, in this case thats exactly what rdo did | 12:59 |
zbr | we already to the same assumption with pip, ansible,... | 12:59 |
zbr | unless overriden, any python tool is expected to use the default interpreter. | 12:59 |
* mnaser hmms | 12:59 | |
mnaser | i like the approach zbr has but i also feel like its step 2, step 1 is to unbreak rdo imho | 13:00 |
*** rlandy has joined #zuul | 13:01 | |
mnaser | i wonder if | 13:02 |
mnaser | variables > facts | 13:02 |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds variable to toggle whether to revoke sudo https://review.opendev.org/706248 | 13:03 |
mnaser | hmm... | 13:03 |
mnaser | pabelanger, dpawlik: how did things work before? what actaully installing tox? | 13:03 |
AJaeger | pabelanger: did you see https://review.opendev.org/#/c/706216/ ? | 13:04 |
*** yolanda has quit IRC | 13:04 | |
AJaeger | oh, you did... | 13:04 |
mnaser | because previously, it must have been using python2 tox.. | 13:04 |
mnaser | because type tox would evaluate fine and it would skip the install | 13:05 |
AJaeger | pabelanger: please review again, I think you misread the change | 13:05 |
mnaser | AJaeger: i also have another alternative here https://review.opendev.org/#/c/706254/ | 13:05 |
AJaeger | mnaser: I don't see how that can work, tox_executable is unset initially, isn't it? | 13:06 |
mnaser | AJaeger: defaults/main.yml is something i added to the first change | 13:07 |
mnaser | that has tox_executable predefined as "tox" | 13:07 |
mnaser | so it tries tox --version (or whatever is defined) -- if that works, yay. if it doesn't, then it installs it locally and stores that fact | 13:07 |
AJaeger | mnaser: ah... | 13:07 |
pabelanger | mnaser: this worked because tox --version returned true | 13:07 |
pabelanger | but now python3 -m tox --version doesn't | 13:07 |
pabelanger | adn they have a task after ensure-tox to pip3 install -U tox | 13:08 |
mnaser | pabelanger: oh that doesnt work? how can we then pass "args" to python3 -m tox then? | 13:08 |
AJaeger | mnaser: Ok, removed -1 - but have no time to dig deeper into this now. | 13:08 |
mnaser | would it be like python3 -m tox -- --version or something? | 13:08 |
* AJaeger steps out again | 13:08 | |
pabelanger | python2 -m tox --version would work | 13:08 |
pabelanger | as the image has py2 tox | 13:09 |
pabelanger | it is missing py3 tox right now | 13:09 |
mnaser | oh i see your point | 13:09 |
pabelanger | because we added tox_executable into ensure-tox, now we get overlap | 13:09 |
mnaser | pabelanger: so before my change, how did py3 tox every get installed then? | 13:09 |
mnaser | because it would detect the image had py2 tox | 13:09 |
mnaser | and not install anything | 13:09 |
pabelanger | mnaser: https://review.rdoproject.org/zuul/build/e1682e34f679428e80e2a5bc20022604/log/job-output.txt#247 | 13:10 |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general. https://review.opendev.org/706262 | 13:10 |
pabelanger | old way, py2 tox --version worked, and ignored tox_exectuable | 13:10 |
pabelanger | then, next playbook did upgrade, to then make tox_executable work | 13:10 |
pabelanger | IMO, we need more refactor to allow for pip2 / pip3 now in ensure-tox, but that is going to grow into PRs that zbr proposed a while back | 13:11 |
mnaser | aah ok | 13:11 |
mnaser | so rdo had a thing to upgrade | 13:11 |
mnaser | ok i think im starting to follow a bit | 13:11 |
pabelanger | IMO, we should look at what zbr did, as it is more indepth | 13:12 |
pabelanger | but, that is going to take some time to review / land | 13:12 |
*** ofosos has quit IRC | 13:12 | |
pabelanger | why I think, we should revert to working jobs, then try to add new support into ensure-tox | 13:12 |
pabelanger | which, will break you again | 13:12 |
*** Goneri has quit IRC | 13:12 | |
mnaser | yeah i'm trying to find the fastest least intrusive solution for now | 13:13 |
mnaser | and then we refactor into zbr work imho | 13:13 |
mnaser | i wonder if it makes sense to introduce something like tox_python | 13:13 |
pabelanger | as work around for you, you could set tox_executable: ~/local/bin/tox in your base job, and old way works | 13:13 |
mnaser | which defaults to "2" (the old behaviour) | 13:13 |
mnaser | or it can default to whatever it detects the system python to be | 13:13 |
mnaser | and installs accordingly | 13:13 |
zbr | mnaser: we have tons of python based tools, starting to create vars for each does not scale very well. | 13:13 |
zbr | In "python -m" I trust. Almost never failed me ;) | 13:14 |
openstackgerrit | Daniel Pawlik proposed zuul/zuul-jobs master: Fix ensure-tox issue when tox was installed on the host https://review.opendev.org/706216 | 13:15 |
*** guilhermesp has quit IRC | 13:16 | |
pabelanger | mnaser: I am a fan of the pattern I did with https://opendev.org/windmill/ansible-role-openstacksdk/src/branch/master/defaults/main.yaml which exposed 3 install methods (git / pip / pkg) then for pip / git allowed to also install via virtualenv and select python version | 13:16 |
*** clayg has quit IRC | 13:16 | |
pabelanger | resulting in something like: https://opendev.org/windmill/ansible-role-openstacksdk/src/branch/master/tasks/install/pip.yaml | 13:16 |
mnaser | because ensure-tox right now doesnt know if you want py2 or py3 | 13:17 |
pabelanger | yah | 13:17 |
*** kklimonda has joined #zuul | 13:18 | |
*** jpena|lunch is now known as jpena | 13:19 | |
mnaser | pabelanger, dpawlik i'm more than ready to come up with a fix to unblock y'all, i feel badz, so if we can find something, i can implement it right now | 13:20 |
pabelanger | mnaser: I think we need to revist https://review.opendev.org/676464/ | 13:22 |
pabelanger | this is what zbr was been working on for a bit | 13:22 |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general. https://review.opendev.org/706262 | 13:23 |
*** gmann has quit IRC | 13:23 | |
mnaser | could we maybe quickly gather consensus on a direction? : | 13:24 |
mnaser | i like the python -m idea | 13:24 |
*** jamesmcarthur has joined #zuul | 13:24 | |
*** hashar has joined #zuul | 13:24 | |
*** bhavikdbavishi has quit IRC | 13:26 | |
*** irclogbot_1 has quit IRC | 13:26 | |
*** bhavikdbavishi has joined #zuul | 13:27 | |
pabelanger | I'm low on time today, head deep into ansible collection testing | 13:28 |
pabelanger | but agree, I think we need to discuss it | 13:29 |
*** irclogbot_3 has joined #zuul | 13:30 | |
*** bhavikdbavishi has quit IRC | 13:31 | |
*** jamesmcarthur has quit IRC | 13:35 | |
*** jamesmcarthur has joined #zuul | 13:49 | |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general. https://review.opendev.org/706262 | 13:49 |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general. https://review.opendev.org/706262 | 13:52 |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general. https://review.opendev.org/706262 | 13:55 |
openstackgerrit | Daniel Pawlik proposed zuul/zuul-jobs master: [DNM] Fix ensure-tox issue when tox was installed on the host https://review.opendev.org/706216 | 13:58 |
*** Goneri has joined #zuul | 13:58 | |
dpawlik | pabelanger, so do we have a "smart" solution to enable gates? | 13:59 |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general. https://review.opendev.org/706262 | 14:00 |
*** guilhermesp has joined #zuul | 14:01 | |
tristanC | mnaser: well it seems like moving rdo tox job to centos8 fixes the situation, but other zuul-jobs user might be affected by the change. shouldn't we revert it in that case? | 14:04 |
*** clayg has joined #zuul | 14:05 | |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general. https://review.opendev.org/706262 | 14:07 |
mnaser | tristanC: if we revert it, i break again, so i dont know what to do | 14:11 |
mnaser | i guess it would suck to be me | 14:11 |
mnaser | lols | 14:11 |
mordred | mnaser: so - I just read the scollback, but I'm still pre-coffee - could you summarize for me? | 14:11 |
pabelanger | mnaser: you should be able to update your base job, for tox_executable for now | 14:11 |
openstackgerrit | Daniel Pawlik proposed zuul/zuul-jobs master: [DNM] Fix ensure-tox issue when tox was installed on the host https://review.opendev.org/706216 | 14:11 |
mnaser | mordred: sure! | 14:11 |
mnaser | the change we did the other day had a small behaviour change that it started using tox_executable in the command the output the tox version | 14:12 |
mnaser | rdo overrides tox_executable to: `python3 -m tox` in their jobs | 14:12 |
mnaser | their image also had tox built in | 14:12 |
mnaser | so when the job ran: it would do type tox >> oh ok i got tox, i'm cool, move on, dont try to install anything | 14:12 |
mnaser | then it would run tox --version (that works, even tho its py2 tox) | 14:13 |
mnaser | and then the later have a little task that installed tox inside py3 | 14:13 |
mordred | ok. those steps all seem reasonable :) | 14:13 |
mnaser | NOW since my change happened -- it tried to check if pip was installed (it was in the image, so no need to install) | 14:14 |
mnaser | and then instead of doing tox --version, it did {{ tox_executable }} --version -- aka python3 -m venv --version | 14:14 |
mnaser | orry, python -m tox --version | 14:14 |
mnaser | which would bomb out, because it wasn't installed. and their pre fails | 14:14 |
mnaser | now i came up with this -- https://review.opendev.org/#/c/706254/ | 14:14 |
mordred | ok. so - wait a sec | 14:15 |
mordred | let me understand that bit | 14:15 |
mnaser | ok | 14:15 |
mordred | tox_executable was set to /home/zuul/.local/bin/tox ... OH | 14:15 |
mordred | ok. I got it | 14:15 |
mnaser | right, we set it to that *if* we install it | 14:16 |
mordred | it's not that the thing you set didn't work - it's that the thing they overrode it with now didn't work | 14:16 |
mordred | yeah | 14:16 |
mnaser | exactly | 14:16 |
mnaser | because now the variable is being used | 14:16 |
mordred | oh - but wait - in the original behavior before your change ... | 14:16 |
mnaser | yes, it was also kinda broken. because ensure-tox wasnt actually doing anything in their case, they had another pre.yaml which installed tox in py3 | 14:16 |
mordred | it worked because tox was pre-installed so type tox works | 14:17 |
mnaser | mordred: see https://review.rdoproject.org/zuul/build/e1682e34f679428e80e2a5bc20022604/log/job-output.txt#247 | 14:17 |
mnaser | correct | 14:17 |
mordred | if tox is pre-installed, shouldn't your new change not install anything or set tox_executable? | 14:17 |
mordred | why didn't type tox in the role not cause the role to skip doing anything? | 14:18 |
mnaser | mordred: that part is fine, but because i changed tox --version to {{ tox_executable }} --version | 14:18 |
mnaser | and they have that overridden | 14:18 |
mnaser | it was running python3 -m tox --version (even tho tox was not installed until the task after that one) | 14:18 |
mordred | OH | 14:19 |
mnaser | and it fails | 14:19 |
mordred | wow | 14:19 |
mnaser | yes there was a behaviour change with tox_executable being a thing in there now in the command after | 14:19 |
mordred | ok ... SO ... can I make a stupid suggestion for a short fix while we ponder potentially a larger overall design? | 14:20 |
mnaser | mordred: https://review.opendev.org/#/c/706254/2/roles/ensure-tox/tasks/main.yaml this was my short fix idea | 14:20 |
mnaser | but the problem is it would still install tox inside pip2 and override the fact | 14:20 |
mnaser | so i welcome other ideas :) | 14:20 |
mnaser | (fyi there was also talk of a more concrete fix by using python$version -m tox everywhere instead of tox command, but thats a whole another discussion) | 14:21 |
mordred | yeah - that's bigger | 14:21 |
mordred | your suggestion works and is better than my suggestion - I think let's go with yours, it'll unbreak, then we can think about this larger issue deeply | 14:21 |
mnaser | mordred: it doesnt fully work, because i think the set_fact might override their var? | 14:22 |
tristanC | mnaser: it seems like if tox_executable is a documented rolevar, then a role shouldn't mutate rolevar, as it will likely cause unexpected behavior | 14:22 |
mnaser | i dont know the order of 27 layers of ansible vars | 14:22 |
mnaser | this whole issue is a bit of a deadlock really | 14:23 |
mordred | mnaser: maybe we should just not run your chunk if tox_executable is set at all? | 14:23 |
mnaser | if we dont touch role vars, then we mess with path, but if we mess with path, that's really clunky and its messing with system level stuff that can touch/break other things | 14:23 |
mnaser | oooooooh that's interesting, so an assumption if tox_executable is set that means it exists | 14:24 |
mordred | if the user has set tox_executable, they're saying they want to use tox that's somewhere else | 14:24 |
mordred | yeah | 14:24 |
mordred | and that ensure_tox doesn't need to do anything because someone wants something different | 14:24 |
mnaser | mordred: but that means printing the version out will break in rdo's case at least | 14:24 |
mnaser | unless we omit that too | 14:24 |
mordred | that also addresses tristanC's concern about not mutating a set role var | 14:24 |
mordred | omit that too | 14:24 |
mnaser | so whole role is skipped if tox_executable is set | 14:24 |
mordred | basically, have ensure-tox do _nothing_ if tox_executable is set | 14:25 |
mordred | yeah | 14:25 |
*** chkumar|rover is now known as raukadah | 14:25 | |
mordred | I think it should unbreak the largest number of people | 14:25 |
mordred | what a FASCINATING problem first thing in the morning :) | 14:25 |
* mnaser feels badz | 14:25 | |
mordred | don't feel bad - I think your original thing was solid - just none of us thought about this other thing | 14:26 |
mordred | (also - I mean - we should probably take this usecase in to account so that we can make ensure-tox do what rdo wants too so they can get rid of their pre task) | 14:26 |
mnaser | given this is trivial ill push up a patch and we can discuss in reviews | 14:27 |
mordred | mnaser: cool, thanks! | 14:27 |
mnaser | mordred: hmm, but we have tox_executable inside defaults/main.yml | 14:36 |
mnaser | so we should drop that then? | 14:36 |
tristanC | mnaser: can't we check if tox_executable is different from the default? | 14:36 |
mnaser | tristanC: so maybe define _tox_executable and tox_executable: "{{ _tox_executable }}" | 14:37 |
mnaser | and condition if they're not equal | 14:37 |
mnaser | ya? | 14:37 |
pabelanger | we should create ensure_tox_executable, then default to tox. Then modify tox_executable if unset and tox is missing | 14:37 |
openstackgerrit | Daniel Pawlik proposed zuul/zuul-jobs master: [DNM] Fix ensure-tox issue when tox was installed on the host https://review.opendev.org/706216 | 14:38 |
Shrews | i thought we kinda-sorta had a policy of prefixing role vars with the role name to prevent accidental overrides? | 14:39 |
tristanC | Shrews: yep, but in that case, the variable is used by ensure-tox and tox roles | 14:39 |
mnaser | Shrews: it would make sense from an ansible pov but my idea was to help have that var used later in the actualt tox role | 14:39 |
openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Add support for tracing using OpenTelemetry https://review.opendev.org/705170 | 14:39 |
openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Enable tracing of trigger events https://review.opendev.org/706289 | 14:39 |
openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace Github webhook trigger events https://review.opendev.org/706290 | 14:39 |
openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace Gerrit trigger events https://review.opendev.org/706291 | 14:39 |
Shrews | mnaser: ah | 14:40 |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general. https://review.opendev.org/706262 | 14:40 |
tristanC | would be nice if we could declare and document common vars such as tox_executable or zuul_work_dir, it's kind of tedius to copy paste the same rolevar documentation | 14:40 |
mnaser | ++ | 14:41 |
dpawlik | mordred, https://review.rdoproject.org/r/#/c/24814/ => this change depends on mnasar change https://review.opendev.org/#/c/706254/ | 14:41 |
dpawlik | mordred, so its still doesn't unlock us | 14:41 |
mordred | dpawlik: yeah - I think we just figured out the flaw in the mnaser change | 14:43 |
corvus | also, i owe a test job for this -- we can probably cover all these cases | 14:48 |
mordred | mnaser: I like your _tox_executable idea above | 14:49 |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general. https://review.opendev.org/706262 | 14:51 |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general. https://review.opendev.org/706262 | 14:52 |
mordred | corvus: https://review.opendev.org/#/c/706222 has 2x+2 but I think your opinion on it would be good | 14:52 |
corvus | mordred: can you summarize the problem with 706254/2 in one line? | 14:53 |
mordred | corvus: 706254 still has tox_executable set in defaults | 14:53 |
corvus | but it defaults to 'tox' which seems to approximate the old behavior so why is that bad? | 14:53 |
corvus | oh got it | 14:54 |
mordred | corvus: this is a nice first-thing-in-the-morning one isn't it? | 14:54 |
corvus | nope don't got ti | 14:54 |
corvus | mordred: keep going pls :) | 14:54 |
mordred | corvus: having it set means we can't do a test of "if tox_executable is set then don't run these tasks" | 14:55 |
mordred | oh - wait | 14:55 |
corvus | mordred: but that's not the test that it runs, it runs "does the tox_executable as set work?" | 14:55 |
mordred | right - sorry - I just realized that the latest version of the patch that we discussed hasn't been written yet | 14:56 |
mordred | corvus: ok - so the summary of the problem with the version of the patch that is there ... | 14:57 |
mordred | oh - nope. I don't understand why that version doens't work. let me go read more things | 14:59 |
mnaser | mordred, corvus: rdo has tox_executable overridden to: python3 -m tox -- so when it tries to run "python3 -m tox --version" -- it fails, because tox is not installed in python 3 | 15:00 |
mordred | right - bit you have a rc != 0 trap in there | 15:00 |
mordred | oh. duh. != 0 | 15:00 |
mordred | k. I'm back on the same page again | 15:00 |
mnaser | my "fix" in its current state, will make you end up using py2 tox | 15:01 |
mordred | that's why we were going to have the check change to if tox_executable is set, don't try to install tox - but that has the flaw that we have a default value set for tox_executable | 15:01 |
pabelanger | rdo also has a later playbook in the job, that will pip3 install tox, that is why they have tox_executable: python3 -m pip | 15:01 |
mordred | do we need the tox_executable value in defaults? what does that get us? | 15:02 |
mnaser | mordred: not much, i just thought it was a cleaner more streamlined way | 15:02 |
corvus | mnaser: because of lines 12-16 defaulting to pip (py2) if it is installed as well as py3? | 15:02 |
mordred | yeah | 15:03 |
corvus | mnaser: ^ in re "< mnaser> my "fix" in its current state, will make you end up using py2 tox" | 15:03 |
corvus | ok, so that's the problem i didn't see. thanks :) | 15:03 |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general. https://review.opendev.org/706262 | 15:03 |
pabelanger | for rdo, which tox originally worked, because py27 tox is installed. Then, they upgraded to py3 tox, and pre set tox_executable to that | 15:03 |
mnaser | so _tox_executable: tox and tox_executable: "{{ _tox_executable }}" and be noop if _tox_executable != tox_executable ? | 15:04 |
mordred | mnaser: yah - totally. I think in this case switching to if tox_executable is not defined and removing tox_executable from defaults seems cleanest | 15:04 |
mordred | oh - yeah - that would work too | 15:04 |
mnaser | at least that way i can detect changes i guess | 15:05 |
mordred | and would prevent us from breaking people who set tox_executable to tox for some reason | 15:05 |
pabelanger | yup | 15:05 |
mnaser | i'd like to have a clean solution eventually once we unbreak rdo | 15:05 |
mnaser | but thats a longer discussion | 15:05 |
mordred | yes. I agree - it turns out there are several variables at play here we should discuss... because "what python should tox be installed with" is a valid concept | 15:06 |
tristanC | mnaser: ftr, rdo is no longer broken, the affected job got migrated to centos8 | 15:06 |
mordred | oh - so we can take a minute and just think about the end-state we'd liek to be in then | 15:07 |
mnaser | ok thats reasonable | 15:07 |
openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-tox: refactor to work with tox_executable https://review.opendev.org/706254 | 15:09 |
mnaser | mordred: for whatever its worth ^ | 15:09 |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general. https://review.opendev.org/706262 | 15:14 |
*** jamesmcarthur has quit IRC | 15:18 | |
*** jamesmcarthur has joined #zuul | 15:19 | |
openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Make revoke-sudo more general. https://review.opendev.org/706262 | 15:22 |
*** hashar has quit IRC | 15:22 | |
*** jamesmcarthur has quit IRC | 15:25 | |
*** zxiiro has joined #zuul | 15:45 | |
*** jamesmcarthur has joined #zuul | 15:46 | |
openstackgerrit | Merged zuul/zuul master: Add Zuul's event id to Ansible inventory https://review.opendev.org/706222 | 15:47 |
*** jamesmcarthur has quit IRC | 15:51 | |
*** jamesmcarthur has joined #zuul | 15:55 | |
openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Flesh out the glossary significantly https://review.opendev.org/704391 | 16:05 |
*** sshnaidm is now known as sshnaidm|afk | 16:05 | |
*** jpena is now known as jpena|off | 16:06 | |
fungi | that's ^ no longer wip as far as i'm concerned. please feel free to tear it apart | 16:11 |
Shrews | fungi: i will do my best Pelosi impersonation on it | 16:12 |
fungi | domo | 16:13 |
*** mattw4 has joined #zuul | 16:13 | |
*** bhavikdbavishi has joined #zuul | 16:20 | |
sugaar | hi, the ssh keys need by gerrit, does it need them in order to communicate with the scheduler? or is it to communicate with the workers? | 16:26 |
*** bhavikdbavishi1 has joined #zuul | 16:29 | |
*** bhavikdbavishi has quit IRC | 16:29 | |
*** bhavikdbavishi1 is now known as bhavikdbavishi | 16:29 | |
fungi | the scheduler and the mergers need an ssh key to be able to communicate with gerrit. the scheduler uses it to watch gerrit stream-events for triggering events, while the mergers use it to fetch git refs for proposed changes | 16:29 |
sugaar | all right | 16:30 |
sugaar | thanks | 16:30 |
fungi | and just to be clear, executors generally run a convenience merger, so need it too | 16:30 |
fungi | i believe it's possible to configure the mergers to use https instead of ssh, though it generally makes sense to have them connect the same way the scheduler does | 16:33 |
fungi | https://zuul-ci.org/docs/zuul/reference/drivers/gerrit.html#attr-%3Cgerrit%20connection%3E.password | 16:34 |
fungi | though that only talks about using https for reporting, not for fetching change refs | 16:36 |
fungi | okay, this is a weird situation, i think... normally zuul should trigger a build for any job which gets modified by a proposed change even if file filters would normally discount it. in this case two jobs were altered, but a build was only triggered for one of them: https://review.opendev.org/706059 | 16:39 |
fungi | in the .zuul.d/centos.yaml file, kolla-ansible-centos-source-zun is being altered and a build of that happened | 16:40 |
fungi | but in the .zuul.d/ubuntu.yaml file kolla-ansible-ubuntu-source-zun is being altered in the same pipeline and no build was triggered | 16:41 |
pabelanger | fungi: kolla-ansible-ubuntu-source-zun is duplicated | 16:45 |
fungi | oh interesting | 16:46 |
pabelanger | left comment | 16:46 |
fungi | good eye! | 16:46 |
pabelanger | I wonder what scheduler debug says | 16:46 |
fungi | so i guess that's a possible hole in how zuul auto-triggers builds for jobs being changed. if the job is listed twice then altering the second instance of it and not the first might not trigger a build automatically | 16:47 |
clarkb | ya I think they should stop splitting that config | 16:47 |
pabelanger | fungi: yah, maybe | 16:47 |
fungi | well, in this case the job entry is duplicated within the same file anyway | 16:47 |
clarkb | ah | 16:47 |
clarkb | well stop duplicating unnecessarily across the board then :) | 16:48 |
fungi | so i don't think the split project configuration led to that, though i agree it's harder to wrangle in general | 16:48 |
openstackgerrit | James E. Blair proposed zuul/zuul master: Extract allow/disallow filter into util function https://review.opendev.org/706144 | 16:55 |
openstackgerrit | Clark Boylan proposed zuul/nodepool master: Don't build universal wheels https://review.opendev.org/706325 | 17:03 |
clarkb | I noticed ^ debugging a similar issue in openstack | 17:04 |
openstackgerrit | James E. Blair proposed zuul/zuul master: Gerrit: add polling support for refs https://review.opendev.org/706138 | 17:08 |
corvus | clarkb, fungi: do i understand correctly that opendev images "pip install tox" ? or maybe pip3 install tox? if so -- how do i know which, pip or pip3? | 17:18 |
clarkb | corvus: it runs `$DIB_PYTHON_PIP install $package` where package is tox and DIB_PYTHON_PIP is going to be pip for platform default python | 17:19 |
clarkb | centos7 == pip2, bionic and centos8 == pip3 | 17:20 |
corvus | is there like a table somewhere with what that value is? | 17:20 |
corvus | ok i think that'll get me started, thx | 17:20 |
clarkb | corvus: its based on whatever the distro declares to be their default python, but I am not aware of a table for that | 17:22 |
*** yolanda has joined #zuul | 17:22 | |
fungi | though also tox is *supposed* to be python interpreter agnostic... but what its default interpreter selection is might still depend on which one you installed it with | 17:25 |
corvus | mostly i want to know how to uninstall it | 17:25 |
yoctozepto | pabelanger: thanks, I removed the duplicate - though it still does not run ubuntu https://review.opendev.org/706059 | 17:26 |
yoctozepto | :/ | 17:26 |
clarkb | corvus: removing the bin/tox entry is probably sufficient for that? | 17:26 |
yoctozepto | fungi, clarkb: ^^ | 17:26 |
corvus | clarkb: i'm writing a test, so i want to run ensure-tox as if it ran on a node without tox having ever been installed | 17:26 |
clarkb | yoctozepto: did you collapse the pipeline configs that were split up? | 17:27 |
*** jpena|off is now known as jpena | 17:27 | |
yoctozepto | clarkb: pipeline configs..? | 17:28 |
corvus | yeah ps3 of https://review.opendev.org/706059 removes the dup | 17:28 |
yoctozepto | ah, you mean that project is split in two files | 17:28 |
clarkb | yoctozepto: everything under project in https://review.opendev.org/#/c/706059/3/.zuul.d/centos.yaml and https://review.opendev.org/#/c/706059/3/.zuul.d/ubuntu.yaml | 17:28 |
yoctozepto | it is still split in there | 17:28 |
corvus | oh there were 3 instances before | 17:28 |
clarkb | try collapsing them and see if the behavior changes | 17:29 |
yoctozepto | corvus: 3? | 17:29 |
corvus | oh, no those are 2 different job | 17:29 |
yoctozepto | :D | 17:30 |
corvus | clarkb: so your theory is that only one of the 2 project stanzas is triggering the check? | 17:30 |
yoctozepto | corvus: I proposed the same idea in #infra | 17:30 |
yoctozepto | but like 6 hours ago | 17:30 |
corvus | (and the one that worked happened to be in the stanza that changed?) | 17:30 |
clarkb | corvus: ya thoguh its mostly just a "I've not see nthat before, not sure zuul will handle it the way yoctozepto expects it " hunch | 17:30 |
corvus | yoctozepto: i was asleep 6 hours ago | 17:30 |
yoctozepto | corvus: yeah, no problem | 17:31 |
yoctozepto | just saying I had a similar "hunch" ;-) | 17:31 |
corvus | clarkb, yoctozepto: all of the patches i have seen from yoctozepto should result in the expected behavior of both jobs running. so hopefully this will help us confirm the bug and find a temporary workaround. | 17:31 |
yoctozepto | corvus, clarkb: https://review.opendev.org/706059 | 17:33 |
yoctozepto | still the same | 17:33 |
yoctozepto | I moved two to see if it matters | 17:33 |
clarkb | yoctozepto: rm the extra file entirely :) | 17:33 |
yoctozepto | clarkb: yeah, though this makes my head spin already :-) | 17:33 |
clarkb | and any others if there are more than 2 | 17:33 |
yoctozepto | clarkb: also there is 3rd one for debian | 17:33 |
yoctozepto | yeah | 17:33 |
yoctozepto | :D | 17:33 |
*** evrardjp has quit IRC | 17:33 | |
yoctozepto | ok, let's check it out in a local editor | 17:34 |
*** evrardjp has joined #zuul | 17:34 | |
*** tosky has quit IRC | 17:34 | |
yoctozepto | clarkb, corvus: still nah | 17:40 |
yoctozepto | https://review.opendev.org/706059 | 17:40 |
clarkb | ok we can rule that hunch out at least | 17:42 |
yoctozepto | clarkb: now got rid of centos one | 17:42 |
yoctozepto | and it does not run either | 17:42 |
* yoctozepto out of hunches for now | 17:43 | |
yoctozepto | ;D | 17:43 |
corvus | it's kolla-ansible-ubuntu-source-zun that's not running? | 17:43 |
yoctozepto | corvus: yeah | 17:43 |
yoctozepto | multiple files ruled out | 17:44 |
yoctozepto | common parent ruled out | 17:44 |
corvus | checking logs | 17:46 |
corvus | https://etherpad.openstack.org/p/U82ltjzkVY | 17:48 |
yoctozepto | what does it mean? | 17:49 |
yoctozepto | it did regular check for it | 17:49 |
corvus | don't know yet, just sharing data | 17:49 |
mordred | so - just as an aside - google is totally indexing zuul job pages. I just google searched (without thinking about it) for kolla-ansible-ubuntu-source-zun ... and the first google hit was http://zuul.openstack.org/job/kolla-ansible-ubuntu-source-zun | 17:51 |
yoctozepto | yeah, the job runs normally, also for kolla | 17:52 |
yoctozepto | I was just expecting to see it here due to the "check changed" rule | 17:52 |
corvus | there's actually 2303 debug log lines related to that change (you have a lot of jobs) | 17:53 |
yoctozepto | corvus: thanks, doing our best ;-) | 17:53 |
*** saneax has quit IRC | 17:53 | |
yoctozepto | "where quality meets quantity" or so it goes... | 17:53 |
corvus | haha | 17:53 |
mordred | that should be a t-shirt | 17:54 |
*** saneax has joined #zuul | 17:54 | |
*** saneax has quit IRC | 17:55 | |
*** saneax has joined #zuul | 17:58 | |
openstackgerrit | Merged zuul/zuul master: add_host: enable using ansible_fqdn and ansible_private_key_file https://review.opendev.org/706124 | 18:00 |
openstackgerrit | Merged zuul/zuul master: Increase timeout of zuul-build-image https://review.opendev.org/705689 | 18:11 |
tobiash | mordred: that's awesome | 18:21 |
*** jpena is now known as jpena|off | 18:22 | |
corvus | yoctozepto, clarkb, fungi, pabelanger, mordred: got it. i hope you're sitting down for this. | 18:23 |
yoctozepto | corvus: just sat down and was going to ask :-) | 18:23 |
corvus | yoctozepto, clarkb, fungi, pabelanger, mordred: there is a clue in the log output. notice that the centos job ran because it was a "newly created job" | 18:23 |
yoctozepto | corvus: oh yeah, it truly was | 18:24 |
* fungi straps in | 18:24 | |
corvus | that's the main difference between the two, and ultimately why it ran but not ubuntu | 18:24 |
yoctozepto | that's part of the solution, brilliant indeed | 18:24 |
yoctozepto | thanks corvus | 18:24 |
corvus | the ubuntu job was not newly created, so zuul is falling back on the configuration change detection | 18:24 |
corvus | you might *think* that it should run since its configuration has changed | 18:24 |
yoctozepto | now onto why the "touched" rule does not apply here | 18:24 |
yoctozepto | yeah | 18:25 |
* fungi would think that | 18:25 | |
* yoctozepto too ;-) | 18:25 | |
* mordred can't wait | 18:25 | |
yoctozepto | the ground is shaking as spectators await the final word from corvus | 18:25 |
AJaeger | drum roll... | 18:26 |
* yoctozepto assumes corvus just can't handle the laughter atm :-) | 18:27 | |
corvus | (just getting details right, sorry) | 18:27 |
fungi | sure, sure, i bet you're actually getting a sandwich ;) | 18:28 |
yoctozepto | then it's better to really sit down as corvus is going to drop something heavy | 18:29 |
Shrews | such a tech tease | 18:29 |
yoctozepto | (pun heavily intended) | 18:29 |
* yoctozepto starts playing heavy metal | 18:33 | |
corvus | hrm, this isn't quite holding up as i type it up. let me dig a bit more. | 18:36 |
corvus | sorry about that | 18:36 |
yoctozepto | that's exactly how it's to write an academic paper, I know the pain :-) | 18:38 |
*** dpawlik has quit IRC | 18:50 | |
fungi | maybe you should have gotten a sandwich after all | 18:51 |
corvus | i'm going to fire up the repl; this will take a few minutes. probably long enough for a sandwich. | 18:52 |
mordred | mmm. sandwich | 18:55 |
mnaser | btw, pabelanger pointed this out a while back but earlier in the day.. http://zuul.opendev.org/t/zuul/status -- there's a job that has been paused for 70 hours | 18:57 |
*** tosky has joined #zuul | 18:59 | |
fungi | unrelated, we've also discovered in #openstack-infra that it's possible when setting debug on a project-pipeline in a proposed change for the resulting report to be too large to fit into the message column of the buildset table, and this seems to also prevent the gerrit reporter from reporting | 19:01 |
mordred | \o/ | 19:07 |
*** jamesmcarthur has quit IRC | 19:10 | |
Shrews | fungi: zuul source seems to indicate that message is a TEXT column. that doesn't quite make sense | 19:18 |
Shrews | unless a packet size is being exceeded maybe? | 19:19 |
fungi | maybe? | 19:19 |
fungi | how wide is a text column allowed to be? | 19:20 |
Shrews | fungi: as large as you need | 19:20 |
fungi | neat | 19:20 |
mordred | nope | 19:20 |
mordred | TEXT – 64KB (65,535 characters) | 19:20 |
Shrews | limited by memory and max_allowed_packet | 19:20 |
Shrews | orly? | 19:20 |
fungi | yeah, so the message is >2^16 which makes sense if mordred is to be believed | 19:21 |
fungi | '(pymysql.err.DataError) (1406, "Data too long for column \'message\' at row 1") | 19:21 |
*** bhavikdbavishi has quit IRC | 19:21 | |
fungi | ... (65967 characters truncated) ... | 19:21 |
Shrews | ok, that makes more sense then and blames my own failing RAM | 19:22 |
mordred | https://dev.mysql.com/doc/refman/8.0/en/storage-requirements.html#data-types-storage-reqs-strings | 19:22 |
fungi | problem is, this is with all project-templates and all jobs but the one in question commented out of the project-pipeline | 19:22 |
fungi | so not sure how to actually get the debug report for the change | 19:22 |
Shrews | mordred: but no one could ever possibly need more than 64K | 19:23 |
mordred | Shrews: that | 19:23 |
fungi | thanks, bill | 19:23 |
Shrews | that's basically infinity | 19:23 |
johnsom | mysql's column storage is... interesting | 19:24 |
fungi | i wonder if we could just truncate the message ourselves when storing? or still report to gerrit even if the mysql reporter fails? | 19:24 |
Shrews | fungi: might be better/easier/lazier to change to MEDIUMTEXT/LONGTEXT | 19:24 |
fungi | that sounds reasonable, but presumably needs a migration | 19:25 |
Shrews | yah | 19:25 |
fungi | i mean, 64k seems like not that much when you can have lengthy debug reports | 19:25 |
johnsom | The interesting part is this job overflows that field even without debug on | 19:25 |
fungi | oh, you're not getting any results reported for the change even before adding debug:true? | 19:27 |
fungi | i'll look back at an older patchset's logging in that case | 19:27 |
johnsom | Right, the last two runs had the debug line commented out. The failures prior to Andreas asking for debug didn't have the line at all | 19:27 |
corvus | fungi: also, btw, all the same info should be in the scheduler debug log, so you can dig it out for johnsom even with the reporter failure | 19:28 |
corvus | fungi, clarkb, mordred, yoctozepto, pabelanger: okay, re the kolla job: the reason zuul doesn't think that there is a job configuration change for the ubuntu zun job is a bug in zuul. it maintains both a text and regex form of the filematcher in the job data structure, and it uses the text version to compare for differences. however, the text version (which is only used for this and rest/json api | 19:28 |
corvus | serialization) is not updated when variants are applied. so basically it didn't see the difference in this check (but the actual file matchers which are stored in another attribute work as expected). this is a relatively straightforward fix, and we should do it so that the json serialization of jobs is correct. however, we may want to consider breaking it again, but intentionally this time -- we might | 19:28 |
corvus | decide that changing a file matcher shouldn't automatically run a job since it doesn't actually change how the job is run. | 19:28 |
fungi | oh, good point, i probably have to find the buildset id and then grep for that instead of the change id | 19:28 |
corvus | fungi: grep for the event id | 19:29 |
fungi | corvus: agreed on only changing a file matcher not triggering a build, rerunning that job doesn't actually test anything | 19:30 |
corvus | fungi: the "e:" at the start of the log line -- get that by grepping for the buildset | 19:30 |
johnsom | The last change was 706329,14 | 19:30 |
corvus | i'll propose the fix (and intentional re-break) after lunch | 19:33 |
fungi | johnsom: yep, trimming off the leader from all the matching loglines may still be too large for paste.o.o but i'll try to link something in #openstack-infra in a moment | 19:33 |
fungi | corvus: thanks!!! | 19:33 |
johnsom | +1 | 19:34 |
yoctozepto | corvus: thanks, let's first see if the fix helps and then break again, totally fine by me | 19:38 |
corvus | yoctozepto: oh, we'll know. there will be a test :) | 19:38 |
yoctozepto | well "break" as it makes sense not to run it in this particular case | 19:39 |
corvus | ianw, fungi, AJaeger: there's a notable absence in this change: https://review.opendev.org/695828 -- the change did not add the header to the update script. | 19:39 |
yoctozepto | corvus: I'll restore ps3 to get it back in order | 19:39 |
yoctozepto | if you don't mind | 19:39 |
corvus | yoctozepto: yep, all done, thanks! | 19:39 |
openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: add an ensure-tox test job https://review.opendev.org/706371 | 19:40 |
corvus | ianw, fungi, AJaeger: see ^ | 19:40 |
corvus | mnaser: ^ that change is a start on the ensure-tox test | 19:40 |
openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: add an ensure-tox test job https://review.opendev.org/706371 | 19:42 |
yoctozepto | btw, is there an easier way to restore ps than review -d, amend and review? | 19:43 |
*** armstrongs has joined #zuul | 19:43 | |
AJaeger | yoctozepto: git review -d X,3 ; git review | 19:44 |
AJaeger | (assuming you want to go back to 3) | 19:44 |
mnaser | corvus: i know a lot of text got lost in the big scrollback, but i was wondering if it might be ok for us (vexxhost) to do third party ci on zuul/zuul-jobs | 19:44 |
yoctozepto | AJaeger: yeah, I do it with amend inbetween cause this rejects it with "already present" | 19:44 |
fungi | yoctozepto: AJaeger: you may need to rebase or make a minor alteration however, as gerrit won't allow a new patchset with the same exact git sha as an earlier patchset | 19:44 |
yoctozepto | exactly | 19:44 |
fungi | or even amend, yeah | 19:44 |
yoctozepto | so my procedure really is minimal | 19:45 |
fungi | should be sufficient | 19:45 |
yoctozepto | thanks, folks | 19:45 |
AJaeger | fungi, I think it worked for me without changes - but have nothing to test right now... | 19:45 |
AJaeger | when I pushed back the old change, it even restored the votes ;) | 19:45 |
AJaeger | corvus: are you going to add the autogenerated lines or shall I? | 19:46 |
*** michael-beaver has joined #zuul | 19:49 | |
mordred | mnaser: that seems like a good idea to me. software factory also does a 3pci on zuul-jobs | 19:52 |
AJaeger | argh, adding those lines in is too tricky for me for tonight, hope ianw can take it on... | 19:52 |
*** armstrongs has quit IRC | 19:53 | |
*** Goneri has quit IRC | 19:54 | |
*** openstackstatus has joined #zuul | 19:57 | |
*** ChanServ sets mode: +v openstackstatus | 19:57 | |
mnaser | mordred: yep i saw that, so i just figured i wanted to ask if it was ok before doing it :) | 19:58 |
mordred | mnaser: ++ | 19:59 |
openstackgerrit | David Shrewsbury proposed zuul/zuul master: Alter message column type in buildset table https://review.opendev.org/706377 | 20:14 |
Shrews | fungi: mordred: it may be as simple as that? ^^ | 20:15 |
Shrews | looks like the sql reporter automatically runs the migrations on load | 20:15 |
clarkb | Shrews: we should check if that is valid in postgres | 20:17 |
Shrews | clarkb: oh good point | 20:17 |
clarkb | a quick google shows postgres may just have "text" but unsure if they'll automagic the mediumtext to text | 20:17 |
clarkb | (I think tristanC is using postgres if we need someone to confirm in the wild) | 20:19 |
Shrews | clarkb: yeah, not seeing support for that, just TEXT | 20:19 |
Shrews | https://docs.sqlalchemy.org/en/13/dialects/postgresql.html#postgresql-data-types | 20:19 |
Shrews | welp, so much for that | 20:19 |
Shrews | maybe fungi's suggestion to trim the message is the route to take | 20:20 |
clarkb | we can probably check the connection type and do the migration if mysql else noop | 20:20 |
clarkb | (we'll still want to record that we migrated just do nothing in the postgres case) | 20:20 |
*** hashar has joined #zuul | 20:21 | |
Shrews | clarkb: that only half solves the problem though | 20:21 |
clarkb | Shrews: postgres' text type isn't size limited | 20:21 |
clarkb | (at least that is what googling says) | 20:22 |
Shrews | clarkb: oh, well maybe that's what i was recalling earlier :) | 20:22 |
ianw | corvus: hrm ... i feel sure i did. i also remember rebasing things several times when working on that and i guess it must have got lost in a merge at some point, let me see | 20:23 |
Shrews | i think only doing the migration for one db would unnecessarily complicate things due to having to keep the ORM model in sync with the db | 20:23 |
clarkb | Shrews: yes i think you still mark postgres as having migrated, you just don't do anything | 20:24 |
clarkb | (I'm guessing that a function that passes is sufficient but have never tried that with alembic) | 20:25 |
Shrews | clarkb: i mean the change needed in https://review.opendev.org/#/c/706377/1/zuul/driver/sql/sqlconnection.py | 20:25 |
clarkb | ya that would have to be conditional too | 20:26 |
Shrews | if using mysql, that needs to be sa.MEDIUMTEXT. if using pg, it should remain sa.TEXT | 20:26 |
clarkb | yup | 20:26 |
clarkb | I'm sure we can introspect that from sql_alchemy but not sure how | 20:26 |
clarkb | self.engine | 20:27 |
clarkb | if mysql in self.engine.driver: text_type=sa.MEDIUMTEXT else text_type=sa.TEXT ? | 20:29 |
clarkb | there is more than one mysql driver so that test needs to be flexible enough to catch them | 20:29 |
*** jamesmcarthur has joined #zuul | 20:38 | |
*** gothicmindfood has quit IRC | 20:51 | |
*** jlk has quit IRC | 20:51 | |
*** irclogbot_3 has quit IRC | 20:53 | |
*** irclogbot_1 has joined #zuul | 20:55 | |
*** rlandy is now known as rlandy|afk | 21:00 | |
*** Goneri has joined #zuul | 21:11 | |
*** Goneri has quit IRC | 21:18 | |
corvus | mnaser: ++3pci on zuul-jobs | 21:31 |
corvus | mnaser: (though, for things like ensure-tox, we ought to be able to cover most of the cases in-repo) | 21:31 |
corvus | Shrews: are we sure we want to store >64k? | 21:32 |
corvus | it, erm, you know, sounds like a lot :) | 21:32 |
Shrews | corvus: nope | 21:33 |
Shrews | i think trimming is just as viable, imo | 21:34 |
corvus | Shrews: k. i'll just throw it out there that i think it's a valid question and i kindasorta lean toward the answer may be to trim but am totally open to whatever other ppl think | 21:34 |
Shrews | agreed 64k seems like way more than you'd need. i have no idea the content they're trying to store there | 21:35 |
corvus | Shrews: it's usually <100 bytes, like this: https://zuul.opendev.org/t/zuul/buildset/5505f6b6f9804f289c7faadf289bb0e3 | 21:35 |
corvus | with debug on, we might expect like 50-100 lines. i have no idea how we got to 64k. | 21:36 |
corvus | (i'd bet a nickel on some bug that would be obvious if we had even a truncated sample) | 21:36 |
*** jamesmcarthur has quit IRC | 21:36 | |
Shrews | yeah. maybe we should look at the actual content to see what's happening | 21:36 |
Shrews | fungi: do you have that sample you referred to earlier? (if you aren't afloat somewhere) | 21:37 |
openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: add an ensure-tox test job https://review.opendev.org/706371 | 21:41 |
*** hashar has quit IRC | 21:42 | |
clarkb | I think truncating is probably fine too | 21:43 |
clarkb | Shrews: is that 65k characters or 65k bytes? | 21:43 |
clarkb | because utf8 is 4 bytes per character would get us closer to 12k characters | 21:43 |
clarkb | easier to hit that limit I expect | 21:43 |
fungi | Shrews: i can get that sample again, yes | 21:50 |
fungi | Shrews: the "content" in this case was a debug report | 21:51 |
fungi | which is why it was so large | 21:51 |
mnaser | corvus: ok cool, i may have something in the way tomorrow / this weekend. | 21:51 |
fungi | Shrews: though as the report was truncated by the db driver, i don't exactly know why it was so very, very large | 21:52 |
fungi | after the db rejected it, the buildset went on to not report to gerrit either | 21:52 |
*** jlk has joined #zuul | 21:54 | |
*** Goneri has joined #zuul | 21:56 | |
corvus | clarkb, Shrews: oh, is that the way it's stored? /me mumbles something about drizzle fixing this | 21:57 |
clarkb | corvus: ya confirmed its 64KB not characters | 21:58 |
clarkb | so utf8 will be 1/4 that | 21:58 |
corvus | clarkb: utf32 is 4 bytes per char, but yeah | 21:59 |
fungi | best to truncate just shy of 16kb just to make sure | 21:59 |
clarkb | corvus: mysql doesn't do variable length utf8, mysql utf8 is 4 bytes per char (or 3 if you do the 3 byte encoding instead) | 21:59 |
corvus | clarkb: that is not exactly how i read https://dev.mysql.com/doc/refman/8.0/en/charset-unicode.html but this may vary by version? | 22:02 |
corvus | not sure what opendev is using right now | 22:02 |
clarkb | corvus: ah it is variable length but not when it comes to calculating index sizes | 22:04 |
clarkb | (index sizes is the place I've run into this) | 22:04 |
clarkb | with index sizes they assume the max char byte size | 22:05 |
corvus | ah, makes sense | 22:06 |
fungi | oh, right, since that can't be predicted | 22:06 |
corvus | so if this holds for our version, in practice it shouldn't be an issue and we can assume char=byte and go with 64k | 22:06 |
fungi | so they go with the upper-bound | 22:06 |
clarkb | corvus: yup | 22:06 |
fungi | only an issue if zuul decides to report a bunch of snowman | 22:06 |
clarkb | fungi: Gerrit Frozen mode: Code review with your 4 year olds | 22:07 |
clarkb | and now I've got that song stuck in my head | 22:07 |
fungi | also if truncating, we can probably use one of the stdlib unicodedata modules to calculate the byte length | 22:07 |
*** tjgresha has joined #zuul | 22:08 | |
clarkb | fungi: we can also truncate after converting to a byte type? though I'm not sure if sqlalchemy will accept that for input on a text type | 22:09 |
fungi | yeah, i meant if sqla wants it in unicode string form | 22:10 |
fungi | nevermind, i was thinking of where i've used the unicodedata module to determine aggregate glyph widths for line wrapping | 22:14 |
fungi | we care about encoded width not displayed width | 22:14 |
fungi | we could len(message.encode('utf-8')) but the trick is finding a spot to truncate it without splitting within a character | 22:15 |
clarkb | ya | 22:15 |
fungi | also we'd have to guess at what sort of normalizing any of the layers might decide to perform | 22:15 |
clarkb | we could also just assume the max length like indexes do and that would be safe and still give us ~11k characters | 22:16 |
fungi | compromise: catch the exception and retry truncated to what we think is safe | 22:16 |
openstackgerrit | James E. Blair proposed zuul/zuul master: Don't run jobs if only their file matchers are updated https://review.opendev.org/706399 | 22:17 |
corvus | yoctozepto: the bug you found ^ thanks :) | 22:17 |
fungi | already reading it | 22:17 |
corvus | clarkb: i use the word 'frozen' a lot in that commit message | 22:18 |
clarkb | corvus: you would be a hero in my household | 22:18 |
corvus | but it's always next to the word 'job' which is boring | 22:19 |
clarkb | I once said "else" and that was enough to have the full attention of my children | 22:19 |
corvus | haha | 22:19 |
clarkb | because it is close enough to "Elsa" and they heard "Elsa" | 22:19 |
fungi | just wait until they find out you know an olaph | 22:20 |
*** tjgresha has quit IRC | 22:20 | |
*** yolanda has quit IRC | 22:24 | |
clarkb | they would immediately demand we go visit | 22:25 |
corvus | and would be rewarded with farm animals | 22:26 |
*** mattw4 has quit IRC | 22:29 | |
*** Goneri has quit IRC | 22:31 | |
*** rlandy|afk is now known as rlandy | 22:32 | |
*** mattw4 has joined #zuul | 22:38 | |
ianw | corvus: i can't replicate that comment issue that your change has exhibited -- ruamel should be preserving comments, and when i add the job and run ./tools/update-test-platforms.py it doesn't wipe out all the other files comments? | 22:38 |
corvus | ianw: maybe i have an old/defective version of ruamel? | 22:39 |
ianw | hrm, it's vendored in ./tools | 22:40 |
corvus | i don't think it is, only the helper methods in ruamellib | 22:41 |
corvus | i'm running 0.15.34-1 from ubuntu | 22:42 |
ianw | http://paste.openstack.org/show/789249/ | 22:42 |
*** avass has quit IRC | 22:44 | |
ianw | oohhh, right, ruamellib is a wrapper | 22:45 |
ianw | 0.16.5 from f31 | 22:46 |
corvus | i'll try in a venv | 22:47 |
corvus | ianw: yeah, that works better | 22:48 |
corvus | 0.16.7 | 22:48 |
corvus | so the answer is: don't run with old ruamel | 22:48 |
ianw | ok, maybe we should put a tox env in for it | 22:48 |
corvus | i wonder if we can check the version in the script | 22:48 |
openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: add an ensure-tox test job https://review.opendev.org/706371 | 22:49 |
clarkb | any other zuulians want to review (and hopefully approve) an easy nodepool change https://review.opendev.org/#/c/706325/ that stops us building universal wheels for a python3 only project | 22:56 |
openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Add tox env for update-test-platforms https://review.opendev.org/706404 | 22:59 |
clarkb | https://review.opendev.org/#/c/705755/3 has a bit more to it but I'm sure will make its users happy too. This adds ebs-optimized flag support for ec2 instances | 23:04 |
*** Goneri has joined #zuul | 23:12 | |
corvus | SpamapS: if you or someone from your org has a minute to review an aws change, that'd be great: https://review.opendev.org/705755 | 23:13 |
clarkb | corvus: can you check my comments on https://review.opendev.org/#/c/706138/2 it isn't clear to me how the testing exercises the code that was changed | 23:28 |
clarkb | (because there was no connection config updates to poll those oprojects) | 23:28 |
corvus | clarkb: responded | 23:33 |
corvus | clarkb: and added a comment | 23:35 |
clarkb | corvus: gotcha so self.connection.projects comes from the zuul tenant config? | 23:35 |
clarkb | I think that was the bit of info I was missing | 23:36 |
corvus | yep. | 23:36 |
openstackgerrit | Merged zuul/nodepool master: Don't build universal wheels https://review.opendev.org/706325 | 23:36 |
clarkb | fungi: left some thoughts on https://review.opendev.org/#/c/704391/3 I'd be happy with followups for Shrews' and my comments too if we want to just land what we've got | 23:38 |
fungi | i saw, thanks! i'll spin another, but a proposed definition for "pipeline manager" would be appreciated | 23:41 |
fungi | this still needs cross-linking of terms, though i'm not sure yet if i should do that in the same change or make a separate one for simplicity | 23:42 |
fungi | i'm guessing by pipeline manager, you mean the archetypal algorithms we use to determine queuing order, windowing, coalescence and implicit dependency | 23:43 |
fungi | (dependent, independent, supersedent...) | 23:43 |
fungi | if so, i can take a stab at it | 23:44 |
clarkb | corvus: left thoughts on https://review.opendev.org/#/c/705856/3 now as well. I +2'd though if you want to approve. I think the proposed short circuit and test simplification can happen in followups | 23:45 |
clarkb | fungi: yup that | 23:48 |
corvus | i'm having a ball reviewing fungi's change! | 23:49 |
clarkb | fungi: I believe they are documented already. Something like Pipeline Manager: pipeline configuration item that determines the queuing behavior of events. They determine if events are queued together or independent or if they should be superceded entirely. | 23:50 |
fungi | wfm | 23:52 |
corvus | fungi, clarkb: i got in on reviewing 704391 too. it looks great, and i have enjoyed "helping". | 23:52 |
fungi | and yeah, i really don't mind folks proposing entirely different definitions for the terms i'm adding there, i mostly just wanted good placeholders as a starting point to build on | 23:53 |
corvus | oh i have nothing other than editorial nits to offer :) | 23:53 |
fungi | also fine ;) | 23:53 |
*** tosky has quit IRC | 23:55 | |
corvus | i'm very grateful to be provided the opportunity to further the cause of establishing the "Maine comma". (when you feel you have to add an oxford comma just to eliminate purposeful misreadings) | 23:56 |
fungi | heh | 23:56 |
fungi | yeah, i never got in the oxford habit. have to remind myself when style guides call for it | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!