*** igordc has joined #zuul | 00:09 | |
*** mattw4 has quit IRC | 00:16 | |
*** fdegir has quit IRC | 00:23 | |
*** fdegir has joined #zuul | 00:24 | |
*** spsurya has joined #zuul | 01:05 | |
*** saneax has quit IRC | 01:19 | |
*** bhavikdbavishi has joined #zuul | 01:33 | |
*** bhavikdbavishi has quit IRC | 01:38 | |
*** bhavikdbavishi has joined #zuul | 01:50 | |
*** rfolco has quit IRC | 01:52 | |
*** bhavikdbavishi has quit IRC | 02:05 | |
*** saneax has joined #zuul | 02:05 | |
*** sshnaidm|afk has quit IRC | 03:10 | |
*** sshnaidm|afk has joined #zuul | 03:23 | |
*** bhavikdbavishi has joined #zuul | 03:27 | |
*** raukadah is now known as chkumar|ruck | 04:33 | |
*** igordc has quit IRC | 05:02 | |
*** bjackman has joined #zuul | 05:56 | |
*** themroc has joined #zuul | 07:26 | |
*** jpena|off is now known as jpena | 07:39 | |
*** jangutter has joined #zuul | 08:24 | |
*** hwangbo has quit IRC | 08:53 | |
mnasiadka | is there a way to use github project in job.required-projects, when the main project is on opendev gerrit? | 09:17 |
---|---|---|
*** sshnaidm|afk is now known as sshnaidm | 09:23 | |
*** bjackman has quit IRC | 11:04 | |
*** bjackman has joined #zuul | 11:19 | |
*** jpena is now known as jpena|lunch | 11:35 | |
openstackgerrit | Jean-Philippe Evrard proposed zuul/zuul master: Revert "Expose date time as facts" https://review.opendev.org/676393 | 11:37 |
*** bjackman has quit IRC | 11:49 | |
*** bjackman has joined #zuul | 11:52 | |
mordred | mnasiadka: https://opendev.org/openstack/openstacksdk/src/branch/master/.zuul.yaml#L323-L324 is an example of doing that | 12:13 |
mordred | mnasiadka: it requires adding the github repo to the zuul tenant config file | 12:13 |
mordred | mnasiadka: and obviously the opendev zuul isn't going to be gating the github project, so care should be taken - but depending on what you're doing, it can be very helpful | 12:14 |
mnasiadka | mordred: I just need ceph-ansible code being cloned in one of kolla-ansible jobs, what's the downside of just doing git clone in bash scripts run by Zuul? | 12:15 |
*** jangutter_ has joined #zuul | 12:19 | |
*** jangutter has quit IRC | 12:20 | |
*** rlandy has joined #zuul | 12:22 | |
*** rlandy is now known as rlandy|rover | 12:22 | |
*** rfolco has joined #zuul | 12:26 | |
mordred | mnasiadka: cloning things from the internet in jobs is inherently unstable - because of the internet :) | 12:30 |
mnasiadka | mordred: ok, got the point :) | 12:30 |
mordred | mnasiadka: when the repo comes in via zuul, the zuul mergers will attempt to clone it, and will retry things if there are temporary issues, etc | 12:30 |
mordred | also - if it's in required-projects, you can do depends-on with PRs to the project, if needed | 12:31 |
*** jpena|lunch is now known as jpena | 12:31 | |
mnasiadka | mordred: ok, sounds fancy - let me find the zuul tenant config file and create a PR... | 12:32 |
mnasiadka | mordred: no configuration needed on github project side? | 12:32 |
mordred | nope - it's a public git repo :) | 12:33 |
mordred | mnasiadka: https://opendev.org/openstack/project-config/src/branch/master/zuul/main.yaml#L1541-L1559 | 12:34 |
*** jangutter_ has quit IRC | 12:39 | |
*** jangutter has joined #zuul | 12:39 | |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: executor: resolve build root real path https://review.opendev.org/676404 | 12:50 |
tristanC | ftr, zuul-3.10.0 doesn't work when /var/lib/zuul is a symlink and executor.job_dir is not set without https://review.opendev.org/676404 | 12:53 |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: config: add tenant.toDict() method and REST endpoint https://review.opendev.org/621344 | 12:55 |
tristanC | to be more precise, executor can't find roles from zuul-jobs and jobs fail with retry_limit | 12:55 |
*** AJaeger_ is now known as AJaeger | 12:57 | |
*** jeliu_ has joined #zuul | 13:11 | |
*** bjackman_ has joined #zuul | 13:15 | |
*** bjackman has quit IRC | 13:17 | |
panda | I'm having problem testing a change from a bare role. the role is added to a job as required_project: and as role: and it' included in the run: playbook. The inclusion of the role with a tasks_from: seems to not find the task file. | 13:19 |
panda | is it even possible to check and gate the role with zuul itself ? | 13:29 |
panda | maybe more clear: if I want a job to check a change in a role that is used in the job itself | 13:34 |
panda | is it possible ? | 13:34 |
*** bjackman_ has quit IRC | 13:35 | |
panda | is the role cloned with or without the change I want to test ? | 13:37 |
panda | I see in inventory theyr're using the same src_dir, so if the role is prepared after, may master checkout overrides the patch | 13:38 |
pabelanger | panda: we have something in zuul-jobs now, but unsure where it is documented. It is something corvus has been driving | 13:39 |
panda | ooh, so it's not currently possible ? | 13:40 |
pabelanger | https://zuul-ci.org/docs/zuul-jobs/policy.html#testing | 13:40 |
panda | pabelanger: OpenDev only ... I would need it for rdo ... | 13:42 |
panda | pabelanger: and the role is not even a zuul-jobs, is from tripleo | 13:43 |
pabelanger | panda: you'd need to setup the same test framework | 13:44 |
pabelanger | setup minimal base job, then parent testing to it | 13:45 |
panda | pabelanger: so make rdo read zuul-tests.d, then setting up a minimal base job and ... I'm not sure I understood the last one. | 13:48 |
pabelanger | panda: for the test to work, you shouldn't run the role, when you are testing the role. So for that, the base minimal job, will not include it, then your new test job needs to be setup to parent to minimal base | 13:50 |
pabelanger | if that makes sense | 13:50 |
pabelanger | https://opendev.org/zuul/zuul-jobs/src/branch/master/zuul-tests.d/general-roles-jobs.yaml#L98 | 13:52 |
pabelanger | you can see, that parents to base-minimal which has a lot of the roles from base missing | 13:52 |
tristanC | panda: we can help you with that, we actually have a story for this sprint to document zuul-tests.d use-case: https://tree.taiga.io/project/morucci-software-factory/us/2775 | 13:53 |
pabelanger | what is zob? | 13:58 |
pabelanger | 'zob testing' | 13:59 |
panda | pabelanger: tristanC I see ... so the child job is not using the roles: in your case because you have the role directly in the same repository. | 14:01 |
tristanC | pabelanger: i think it's a shortcut for zuul jobs | 14:03 |
panda | to test a role in tripleo that is used in base jobs in rdo ... we would need to enable the zuul-test.d lookup ..and | 14:03 |
tristanC | panda: and define testing job that have a file matcher for the role | 14:04 |
*** noorul has joined #zuul | 14:04 | |
panda | tristanC: in the zuul-tests.d in the role repo ? | 14:05 |
noorul | hi all | 14:05 |
tristanC | panda: you can also make that job inherit from tripleo-ci-base and use the match-on-config-updates attribute so that the test job also run when tripleo-ci-base definition is changed | 14:05 |
noorul | I see that in Zuul documentation there is not mention of GCP driver. Is there any work in progress for this? | 14:05 |
tristanC | panda: yes, add role testing job in the same repos as the role is located | 14:05 |
pabelanger | noorul: I don't think once was started | 14:06 |
pabelanger | one* | 14:06 |
pabelanger | I think azure is | 14:06 |
panda | tristanC: that is probably the main problem. The role must be tested in rdo | 14:06 |
noorul | How nodepool behaves in the case of static driver? | 14:07 |
tristanC | panda: that's ok, we can load the zuul-tests.d only from rdo's zuul, or let's name it zuul-tests-rdo.d | 14:07 |
noorul | Will it ensure that one node is used by only one task at a time? | 14:07 |
pabelanger | noorul: yes, you can set a limit | 14:07 |
pabelanger | https://zuul-ci.org/docs/nodepool/configuration.html#static-driver | 14:07 |
pabelanger | https://zuul-ci.org/docs/nodepool/configuration.html#attr-providers.max-concurrency is the setting | 14:08 |
noorul | Thank you! | 14:08 |
tristanC | panda: if you want to do that now, please let us start a review to add such documentation to zuul/doc/source/user/howtos so that we can take notes | 14:08 |
panda | tristanC: so, add a dir zuul-tests-rdo.d in the repo, add the registry-login-integration job in that dir, that will inherit from a base rdo job and test the role ? | 14:09 |
panda | tristanC: I should have done it three weeks ago ... | 14:09 |
tristanC | panda: alright, let me write down now a draft for the process in a review | 14:10 |
panda | tristanC: my saviour ! | 14:10 |
tristanC | panda: actually, that taiga story was initially planned for last sprint too :) | 14:10 |
panda | tristanC: just to be more specific, this is the change I have to test on the repo https://review.opendev.org/673481. and this is hte job I've tried to run in both config and rdo-jobs https://review.rdoproject.org/r/21593 | 14:11 |
panda | the goal is to reuse a role that deals with docker registry to make all job pre-login to rdo registry | 14:11 |
panda | but using the role from tripleo, I want to be sure that any change to the role are tested with the jobs | 14:12 |
panda | pabelanger: thanks for your hints. | 14:23 |
clarkb | note that if you are running those jobs jn opendev talking through the mirror proxies that is currently done jn the clear. I made bot of that with cloudnull when he added the new rdo registry to those mirrors | 14:24 |
clarkb | there is work to deploy them with https but not all regions have hadtheir mirror replaced yet (so is in progress) | 14:26 |
panda | clarkb: oh, because even if thery are intheritd from jobs in rdo , they are run upstream ... | 14:28 |
panda | tristanC: clarkb: I mean in opendev ... mmhh, that may be a complication for the nodes | 14:28 |
clarkb | Yes. I specifically called this out as that registry didnt appear fully public | 14:29 |
clarkb | but was told it was fine at the time | 14:29 |
pabelanger | dmellado: o/ | 14:29 |
dmellado | hi pabelanger o/ | 14:29 |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: WIP: docs: add test jobs howto https://review.opendev.org/676424 | 14:30 |
panda | clarkb: the job will not access the registry anyway, it will not have the correct credentials, we're probably going to mock the registry and it will be good enough. | 14:30 |
panda | tristanC: clarkb but I'm worried by the fact that the job will not be tested in rdo environemnt | 14:30 |
panda | the integration will differ from the final job we're going to run | 14:31 |
tristanC | panda: hum, that sounds more tricky than just testing a job and or role, perhaps the draft ( https://review.opendev.org/676424 ) could use a dedicated section for testing base component used by child jobs accross multiple tenant... | 14:32 |
noorul | In openstack project there are two pipelines, one is check and another one is gate. Is it possible to use different set of tasks for check and gates? | 14:35 |
pabelanger | noorul: yup, we usually try to keep the same jobs in both, but sometime non-voting jobs in check are unstable. So they are left out of gate | 14:36 |
panda | tristanC: hold on, so is there a specific reason why the job defined here https://review.rdoproject.org/r/21593 is not running the container role with the change in https://review.opendev.org/673481 ? | 14:38 |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: WIP: docs: add test jobs howto https://review.opendev.org/676424 | 14:39 |
tristanC | panda: you mean this error: https://logs.rdoproject.org/93/21593/21/check/registry-login-integration/c89fd2e/job-output.txt.gz#_2019-08-14_12_03_46_544226 ? | 14:42 |
panda | tristanC: yes | 14:42 |
pabelanger | could we promote https://opendev.org/zuul/zuul/src/branch/master/tools/zuul-changes.py to be installable in zuul wheel? Or some other method to save queues to disk? This will help avoid the step of manually copying it to zuul server | 14:43 |
*** noorul has quit IRC | 14:46 | |
panda | tristanC: crap, I think I understand why, there's a job with the same name defined in config, that's the job that is running | 14:47 |
tristanC | panda: also, where is playbooks/integration/registry-login.yaml ? | 14:48 |
openstackgerrit | Paul Belanger proposed zuul/zuul master: Update zuul-changes.py to python3 only https://review.opendev.org/676429 | 14:49 |
panda | tristanC: forgot to git add ... and didn't realize it was missing because the job was running. ... well at elast I know that config jobs take precedence over other configuration s... | 14:49 |
openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Switch to fetch-sphinx-tarball for tox-docs https://review.opendev.org/676430 | 14:49 |
pabelanger | zuul-maint: should be easier review^ | 14:49 |
panda | tristanC: added it now, changed the name, and retrying. | 14:49 |
* panda facepalm | 14:49 | |
panda | tristanC: this is all because we can't test in config :) | 14:50 |
tristanC | panda: shouldn't this job be hosted in opendev, using a zuul-rdo-tests.d that will be loaded in rdo? | 14:50 |
tristanC | panda: or do you prefer if rdo report result as a regular third party job | 14:51 |
tristanC | panda: right now, https://review.rdoproject.org/r/#/c/21593/ says that it will gate, but it doesn't seems like this can gate, only opendev can gate openstack/ansible-role-container-registry | 14:51 |
tristanC | or i'm missing something :) | 14:52 |
panda | tristanC: yes, well, with the former, you're not running in the rdo environemnt, so you get only partial results, with the latter, you can't really stop breaking changes to be merged | 14:52 |
panda | tristanC: I think I'd prefer to have real results over a real stopper taht decides on partial results. | 14:54 |
panda | with the first choice, we can blame the approver :) | 14:54 |
panda | tristanC: the new patch worked ... renaming the job and adding the playbook. | 14:55 |
openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Return preview artifact in fetch-sphinx-output https://review.opendev.org/676437 | 14:59 |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: WIP: docs: add test jobs howto https://review.opendev.org/676424 | 15:00 |
tristanC | panda: could you please review https://review.opendev.org/676424 , it should contains the instructions for what you did | 15:00 |
pabelanger | tobiash: left comment on https://review.opendev.org/674034/ | 15:02 |
*** chkumar|ruck is now known as raukadah | 15:05 | |
*** jangutter has quit IRC | 15:07 | |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Add appending yaml log plugin https://review.opendev.org/623256 | 15:08 |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Process yaml log files if they exist https://review.opendev.org/676246 | 15:08 |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Write errors from ansible execution into yaml log https://review.opendev.org/676250 | 15:08 |
tobiash | pabelanger: I'm open to drop py2 there | 15:08 |
mordred | corvus: it helps when the new callback plugin actually has symlinks in the various ansible dir. :) | 15:08 |
pabelanger | tobiash: k, I only noticed it because of https://review.opendev.org/676429/ | 15:09 |
panda | tristanC: is match-on-config-update a shortcut for files: - zuul.d/myjobconfiguration.yaml ? | 15:09 |
mordred | pabelanger: how about let's rebase 676429 on 674034? | 15:10 |
mordred | pabelanger: (I was thinking of hitting +A on 674034 just now) | 15:10 |
pabelanger | mordred: wfm | 15:10 |
tristanC | panda: it's not a file matcher, it will match when the parent job definition change, e.g. timeout, run, roles... any of the parent attributes | 15:10 |
openstackgerrit | Merged zuul/zuul master: executor: resolve build root real path https://review.opendev.org/676404 | 15:11 |
panda | tristanC: uh, proactive testing, you launch the jobs that may be affected by your definition change. that's very cool | 15:13 |
*** ianychoi has joined #zuul | 15:20 | |
corvus | mordred: regarding your change at https://review.opendev.org/676430 (and https://review.opendev.org/676432 ) it does look like it will work (which is great, that means we're halfway to making promote work for everyone) -- but there's one concern to work through: it doesn't support nearly as many role variables as fetch-sphinx-output, so any jobs which are setting those will bork. otoh, the *job* doesn't | 15:21 |
corvus | actually *document* any of the role variables as job variables. | 15:21 |
tristanC | panda: it is indeed, that's the reason why we wanted to documented that previous sprint | 15:21 |
corvus | mordred: so i think the question there is -- is it okay to change the "tox-docs" job in zuul-jobs in such a way that will cause it to ignore all the implied role variables? | 15:22 |
corvus | AJaeger: ^ fyi | 15:22 |
mordred | corvus: I think those are *highly* unlikely to be used by anyone outside of openstack publishing jobs | 15:26 |
corvus | mordred: you're not worried about non-openstack users of tox-docs? | 15:26 |
corvus | mordred: oh sorry | 15:26 |
corvus | i misread | 15:26 |
corvus | mordred: you mean that the only folks likely to tweak those vars are openstack publishing | 15:27 |
mordred | yeah. the uses of those vars I can see are all direct uses of the role itself | 15:27 |
corvus | mordred: yeah, i agree with that | 15:27 |
mordred | rather than via tox-docs | 15:27 |
mordred | so I thni kthe tox-docs change is safe - and we can deal with the other variables when we make new publish jobs for openstack | 15:27 |
corvus | mordred: should we send something out to zuul-discuss or zuul-annonce first though? because if there's someone out there that's using tox-docs and has found those variables, we'll break them and we have no way of knowing | 15:28 |
corvus | (fwiw, i have looked at the inheritance hierarchy of tox-docs in the openstack tenant and confirmed that there are no jobs which inherit from it and set those vars; that doesn't cover project-pipeline variants, but that seems unlikely) | 15:29 |
mordred | corvus: yes, probably so. in the meantime - we COULD just *add* fetch-sphinx-tarball | 15:29 |
mordred | corvus: I just searched for those variables | 15:30 |
mordred | and found them only set in publication playbooks | 15:30 |
mordred | http://codesearch.openstack.org/?q=sphinx_output_suffix&i=nope&files=&repos= and http://codesearch.openstack.org/?q=sphinx_output_src&i=nope&files=&repos= | 15:30 |
corvus | mordred: how about you email zuul-annonce telling folks that we're going to switch the implementation under tox-docs in 2 weeks (to better support promote pipelines) and that if they're using any role variables that it'll be a problem. then we merge that change in 2 weeks. meanwhile, i think the change to the fetch-sphinx-output role can cover the immediate preview url need. how's that sound for a plan? | 15:32 |
corvus | (i'm not super keen on adding the second fetch role to the job because we'll end up pulling the data back to the executor twice) | 15:34 |
mordred | corvus: ++ | 15:34 |
fbo | Hi, is it normal that on a post job the repo on the test node get an origin set on "origin file:///dev/null" ? | 15:40 |
clarkb | fbo: yes | 15:41 |
clarkb | fbo: Zuul curated repos don't have true origins since they tend to be forward looking (post may be an exception to this but zuul doesn't differentiate) | 15:41 |
clarkb | and to avoid pollution of the repo during a job via a remote update zuul sets an origin (because some tools want one) but set it to dev null to avoid it being a source for unwanted commits | 15:42 |
fbo | alright, ok I understand, the tool I run in post expects a valid remote. But I can set it prior to run the tool | 15:43 |
clarkb | fbo: you should consider if that invalidates the job results | 15:43 |
mordred | fbo: yeah - hopefully the tool isn't pulling from remotes or such :) | 15:44 |
clarkb | fbo: zuul ensures that all branch heads are set to the correct ref for the job that is running | 15:44 |
clarkb | and if something demands a valid remote it may change the heads or other repo state | 15:44 |
fbo | but could it make sense to provide an option to have zuul set the correct remote ? | 15:45 |
fbo | the tool is fedpkg, and it looks at the remote to tell/send to koji (fedora build system) where is the commit published. | 15:46 |
mordred | ah - that's an interesting use case | 15:46 |
fbo | nothing is done with the local repo except looking at the origin. | 15:46 |
*** themroc has quit IRC | 15:49 | |
*** mattw4 has joined #zuul | 15:52 | |
openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Correct version of ansible from test-requirements.txt https://review.opendev.org/676448 | 16:06 |
pabelanger | would be nice to have option to toggle it for post job | 16:08 |
clarkb | pabelanger: it isn't necessarily safe to do that in a post job either though | 16:08 |
clarkb | since a post job is tied to a specific commit | 16:08 |
pabelanger | clarkb: isn't that promote? | 16:09 |
clarkb | pabelanger: it is both. Post jobs run against a merge commit | 16:09 |
mordred | well - *may* be tied to a specific commit | 16:09 |
mordred | depending on whether supercedent is being used or not | 16:09 |
* pabelanger nod | 16:10 | |
*** jpena is now known as jpena|off | 16:10 | |
pabelanger | when this last came up, I was running code from git location on disk manually or zuul could. Solution was, don't do that :) | 16:10 |
pabelanger | only do it it rare conditions, when zuul is down for some reason | 16:10 |
*** bhavikdbavishi has quit IRC | 16:13 | |
fungi | fbo: there isn't necessarily a "correct" remote, since the repository is pushed onto the node | 16:14 |
fungi | at least from a zuul job node's perspective | 16:15 |
clarkb | zuul does know the canonical repo location from its perspective though | 16:16 |
fbo | fungi: yeah but zuul knows the remote loc of the repo, I was surprised to find it set to /dev/null. I undertand the why, but it bring a limitation. At least for my usecase :) | 16:17 |
pabelanger | fbo: you can fix it via prepare-workspace too | 16:18 |
fbo | imo that should be configurable and keep the default like it is today | 16:18 |
fbo | pabelanger: just added a task in my role https://pagure.io/zuul-distro-jobs/c/3cd4aedd0e6087775e36679dbb55c75f30646c74?branch=master | 16:19 |
fungi | yeah, zuul knows *a* remote which it used to get the code it pushed, but whether that's the "correct" remote is a more existential discussion | 16:21 |
openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Allow ensure-tox to upgrade tox version https://review.opendev.org/672760 | 16:28 |
openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Create zookeeper operator and zuul CR to k8s test https://review.opendev.org/676458 | 16:35 |
pabelanger | so, we just upgraded to zuul 3.10.0, but having issue getting web working | 16:36 |
pabelanger | The path '/t/ansible/status' was not found. is from cherrypy | 16:36 |
openstackgerrit | Merged zuul/zuul master: Make tenant and pipeline optional in zuul-changes https://review.opendev.org/674034 | 16:36 |
pabelanger | https://dashboard.zuul.ansible.com/t/ansible/status | 16:36 |
pabelanger | zuul-maint: any suggestions where to look? | 16:38 |
pabelanger | ^ | 16:38 |
clarkb | pabelanger: did you update your js too? (I think those tend to need to update together on big updates?) | 16:38 |
pabelanger | clarkb: what ever was installed via whell | 16:39 |
pabelanger | wheel* | 16:39 |
pabelanger | https://dashboard.zuul.ansible.com/api/info works and I can see github using it | 16:40 |
pabelanger | so, just UI seems to be issue | 16:40 |
*** fungi has quit IRC | 16:42 | |
*** fungi has joined #zuul | 16:43 | |
corvus | pabelanger, mordred: i think we found yesterday that 3.10.0 wasn't correctly built with the js | 16:46 |
corvus | mordred was working on fixing the build jobs | 16:47 |
corvus | when that happens we can either rebuild 3.10.0 or issue 3.10.1 | 16:47 |
pabelanger | Oh | 16:47 |
corvus | mordred: where'd you get on that? | 16:48 |
pabelanger | okay, I'll revert | 16:48 |
pabelanger | corvus: is it safe just to revert zuul-web or do I need everything? | 16:48 |
corvus | pabelanger: i bet zuul-web will be ok | 16:48 |
pabelanger | k, let me try that | 16:49 |
pabelanger | okay, back | 16:49 |
pabelanger | so, that is likely it | 16:50 |
pabelanger | I'd vote for 3.10.1 | 16:50 |
pabelanger | let me reply to ML too, so others are aware | 16:50 |
openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Allow ensure-tox to upgrade tox version https://review.opendev.org/676464 | 16:52 |
pabelanger | corvus: mordred: yes, that looks like the issue static/static is missing | 16:54 |
corvus | i'm trying to figure out what we're missing from the release job | 16:55 |
openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: POC: Enable tox-molecule on ensure-tox https://review.opendev.org/672760 | 16:55 |
mordred | corvus: sorry - got distracted - looking in to it now | 16:56 |
pabelanger | yah, same | 16:58 |
openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Allow ensure-tox to upgrade tox version https://review.opendev.org/676464 | 16:58 |
pabelanger | maybe yarn is missing? | 16:58 |
pabelanger | https://logs.opendev.org/62/6254985dd726b8df8f4b73ef148dbec9ea7c48d9/release/opendev-release-python/4acda91/job-output.txt.gz#_2019-08-13_18_15_57_741961 | 16:59 |
pabelanger | https://logs.opendev.org/62/6254985dd726b8df8f4b73ef148dbec9ea7c48d9/release/opendev-release-python/4acda91/job-output.txt.gz#_2019-08-13_18_16_01_291661 confirms it is empty | 16:59 |
corvus | where was release-zuul-python defined? | 17:00 |
corvus | that's the job we used to run | 17:00 |
corvus | ah it's in openstack/project-config | 17:00 |
pabelanger | yah | 17:00 |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Install yarn before building python artifacts https://review.opendev.org/676466 | 17:01 |
pabelanger | so, wrong job? | 17:01 |
corvus | mordred: comment on that | 17:01 |
pabelanger | mordred: should we just move https://opendev.org/openstack/project-config/src/branch/master/playbooks/zuul-tarball/pre.yaml intree? | 17:02 |
mordred | yah - was just fixing that :) | 17:02 |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Install yarn before building python artifacts https://review.opendev.org/676466 | 17:02 |
pabelanger | and revoke sudo | 17:02 |
corvus | i'm not sure about revoke sudo | 17:03 |
corvus | yeah, it's in run | 17:03 |
corvus | we don't need it in pre | 17:03 |
pabelanger | wfm | 17:03 |
mordred | ++ | 17:03 |
mordred | corvus: this should let us make new artifacts ... I think we want a followup to do what we were discussing yesterday and build tarballs without built javascript but wheels with | 17:04 |
corvus | mordred: i think this is still incomplete | 17:04 |
pabelanger | however | 17:04 |
corvus | mordred: agree re followup | 17:04 |
pabelanger | opendev-release-python is what we use for release pipeline and promote | 17:04 |
pabelanger | so we also need them there too (yarn) | 17:04 |
corvus | yeah that ^ | 17:04 |
corvus | so this will fix the branch-tip artifacts | 17:04 |
corvus | pabelanger's thing will fix the released ones | 17:04 |
pabelanger | yah | 17:05 |
corvus | pabelanger: given that you aren't using the pypi branch-tip releases, can we stop releasing there on every commit? | 17:05 |
pabelanger | corvus: we still use them, for nodepool we were on specific commit | 17:05 |
corvus | hrm. | 17:06 |
pabelanger | I mean, we can drop the pre-release but it was to avoid asking for a new release (tag) | 17:06 |
corvus | yeah, it's just it makes the pypi page look terrible | 17:07 |
corvus | have you tried to use it? | 17:07 |
pabelanger | they were hidden before, did that change? | 17:07 |
pabelanger | or thought they were hidden | 17:08 |
corvus | pabelanger: https://pypi.org/project/zuul/#history | 17:08 |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Install yarn before building python artifacts https://review.opendev.org/676466 | 17:08 |
mordred | corvus, pabelanger: how's that look now? | 17:08 |
corvus | mordred: makes sense | 17:09 |
pabelanger | we missed promote, but that is what we are discussing now | 17:09 |
corvus | pabelanger: no, nothing in promote needs to change to fix the problem | 17:10 |
pabelanger | corvus: it is helpful to have a wheel per commit, means we don't have to change a lot of tooling to install a commit we want. But also understand it is ungly | 17:10 |
pabelanger | ugly* | 17:10 |
pabelanger | so, we can drop for now if want | 17:10 |
pabelanger | corvus: mordred: +3 | 17:11 |
pabelanger | also, if we do 3.10.1, we need renonote right? | 17:14 |
corvus | yep, can't have a release without a reason :) | 17:14 |
*** spsurya has quit IRC | 17:14 | |
pabelanger | mordred: want me to add that, in follow up? | 17:14 |
corvus | pabelanger: yes please :) | 17:18 |
pabelanger | k | 17:19 |
openstackgerrit | Paul Belanger proposed zuul/zuul master: Add release note for yarn dependencies missing https://review.opendev.org/676468 | 17:25 |
pabelanger | corvus: mordred: ^ | 17:26 |
corvus | we should be able to verify all of that is working before we tag the release | 17:26 |
pabelanger | agree | 17:26 |
pabelanger | maybe add check to error if they are missing in future too | 17:27 |
mordred | pabelanger, corvus: crappit. there's still one more thing we'r emissing | 17:29 |
mordred | we need to set release_python - one sec, patch coming | 17:29 |
corvus | pabelanger: how about you *don't* make your change depend on mordred's | 17:30 |
mordred | :) | 17:30 |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Install yarn before building python artifacts https://review.opendev.org/676466 | 17:32 |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Build wheels with javascript and tarballs without https://review.opendev.org/676470 | 17:32 |
corvus | mordred: oh, for the py3 thing | 17:32 |
mordred | yeah | 17:32 |
mordred | otherwise the wheels are wrong | 17:32 |
mordred | also - there's a stab at the more complex thing of not putting javascript into source tarballs | 17:33 |
corvus | mordred: cool, do we want to hold that for after this fix release? | 17:33 |
mordred | probably so, yes | 17:33 |
mordred | I think we should warn people about that, just in case | 17:33 |
corvus | yeah. maybe a -discuss post is warranted | 17:34 |
corvus | mordred: the good news is that the "py2" wheel looks like it is the right size: https://71dff8999a9cb00d39ca-d5fbed16a2dc3218104e32d2a6839727.ssl.cf2.rackcdn.com/676466/3/gate/zuul-build-python-release/d239f76/artifacts/ :) | 17:34 |
mordred | yay! | 17:35 |
mordred | sending the -announce post now for the other thing | 17:35 |
mordred | corvus: the original job in project-config also set node_version: 10 ... do we care? | 17:38 |
mordred | I can update this one more time real quick | 17:38 |
corvus | mordred: we seem to still use that everywhere, so probably so | 17:38 |
corvus | for the build-javascript-content-tarball job | 17:39 |
mordred | kk. one sec - I can also suck in pabelanger's release note patch too | 17:39 |
corvus | and maybe updating that is another followup change | 17:39 |
mordred | corvus: wait - updating what is another followup change? | 17:40 |
corvus | mordred: changing the node version? | 17:41 |
corvus | or is 10 the better one to use? | 17:41 |
openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Sync fetch-sphinx-tarball with fetch-sphinx-output https://review.opendev.org/676472 | 17:41 |
mordred | 10 is the better one - it's an override of the older default version in install-npm | 17:41 |
mordred | which defaults to node version 6 | 17:43 |
pabelanger | sorry, stepped away | 17:43 |
pabelanger | mordred: go for it | 17:43 |
openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Partial sync fetch-sphinx-tarball with fetch-sphinx-output https://review.opendev.org/676472 | 17:44 |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Install yarn before building python artifacts https://review.opendev.org/676466 | 17:44 |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Build wheels with javascript and tarballs without https://review.opendev.org/676470 | 17:44 |
clarkb | ya I updated the node version across the board for zuul not too long ago to 10 because one job was using 6 another 8 and another 10 | 17:44 |
mordred | pabelanger, corvus: THAT should finally be accurate | 17:44 |
pabelanger | lulz | 17:44 |
clarkb | and I wanted to eliminate any potential behavior differences while I was debugging a problem | 17:44 |
mordred | yeah | 17:44 |
pabelanger | +2 | 17:44 |
mordred | clarkb: https://review.opendev.org/#/c/676466 look ok to you? | 17:45 |
clarkb | ya not approving because distracted this week | 17:46 |
mordred | totes | 17:46 |
corvus | okay, that's approved. when it lands, we'll inspect the wheel/tarball and if it looks good, cut a .1 | 17:47 |
corvus | pabelanger: also, it looks like we already stopped doing the pre-release stuff on pypi | 17:47 |
corvus | after this release, i kinda want to go in and 'hide' all those | 17:48 |
pabelanger | corvus: okay, thats fine. usually is really an issue when we want to get something from master, but unable to tag release for some reason. Hopefully things are getting more stable the longer we do this :) | 17:48 |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Build wheels with javascript and tarballs without https://review.opendev.org/676470 | 17:49 |
mordred | pabelanger: ++ | 17:56 |
openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Log errors better in case of unknown REST errors https://review.opendev.org/676476 | 17:57 |
openstackgerrit | Merged zuul/zuul-jobs master: Return preview artifact in fetch-sphinx-output https://review.opendev.org/676437 | 18:00 |
openstackgerrit | Merged zuul/zuul-jobs master: Partial sync fetch-sphinx-tarball with fetch-sphinx-output https://review.opendev.org/676472 | 18:00 |
*** rlandy|rover is now known as rlandy|rover|brb | 18:10 | |
fungi | mordred: corvus: it's failing zuul-build-python-release in check | 18:22 |
fungi | oh i guess 676466 is the one we need for the patch release, not 676470 | 18:23 |
*** rlandy|rover|brb is now known as rlandy|rover | 18:26 | |
openstackgerrit | Merged zuul/zuul master: Install yarn before building python artifacts https://review.opendev.org/676466 | 18:31 |
pabelanger | https://logs.opendev.org/66/676466/5/promote/opendev-promote-python/24e32e2/job-output.txt.gz | 18:37 |
pabelanger | promote failed | 18:37 |
pabelanger | https://zuul.opendev.org/api/tenant/zuul/builds?change=676466&patchset=5&pipeline=gate&job_name=build-python-release | 18:40 |
pabelanger | is empty | 18:40 |
pabelanger | but download-artifact requires some value? | 18:40 |
*** mattw4 has quit IRC | 18:41 | |
pabelanger | corvus: mordred: do you know why that would be empty? | 18:41 |
pabelanger | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/download-artifact/tasks/main.yaml#L5 seems to expect a result | 18:41 |
*** mattw4 has joined #zuul | 18:44 | |
mordred | yeah - it's looking for artifacts from build-python-release -- but we replaced that with zuul-build-python-release -- one sec | 19:05 |
pabelanger | tobiash: github driver seems faster, but also could just be excited about improvements I know about | 19:08 |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Override the job name from which we promote https://review.opendev.org/676494 | 19:09 |
mordred | pabelanger, fungi: ^^ | 19:09 |
tobiash | pabelanger: it is definitely faster in our deployment | 19:10 |
pabelanger | tobiash: \o/ | 19:10 |
pabelanger | mordred: +2 | 19:11 |
pabelanger | tobiash: only think we are waiting for now, is new github3.py release. Happy to move to next thing and not worry about github now | 19:12 |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Override the job name from which we promote https://review.opendev.org/676494 | 19:14 |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Build wheels with javascript and tarballs without https://review.opendev.org/676470 | 19:14 |
mordred | fungi: ^^ that should fix the other thing | 19:14 |
mordred | corvus: whence you en-backen ... https://zuul.opendev.org/t/zuul/build/cfbe3f43bd604e509cc1b196cce66f15/log/job-output.txt#340 has an error message that shows as SUCCESS in the console https://zuul.opendev.org/t/zuul/build/cfbe3f43bd604e509cc1b196cce66f15/console | 19:19 |
openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Create zookeeper operator and zuul CR to k8s test https://review.opendev.org/676458 | 19:36 |
corvus | back; catching up | 19:44 |
mordred | corvus: we did some things, and also some other things | 19:47 |
mordred | corvus: mostly just ran around throwing our hands in the air | 19:48 |
corvus | mordred: ah yep -- https://review.opendev.org/676494 is almost the solution but i think you missed a thing | 19:49 |
mordred | corvus: HAHAHAHAHAHA | 19:50 |
mordred | corvus: that wasn't so much like overriding as just redefining to the same value | 19:50 |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Override the job name from which we promote https://review.opendev.org/676494 | 19:50 |
corvus | mordred: yeah, as overrides go, it wasn't the most effective | 19:51 |
mordred | corvus: I thnik for the console thing above it's because the fake thing we inject into the log for errors like that doesn't have a status with it? | 19:51 |
corvus | pabelanger: want to +3 https://review.opendev.org/676494 | 19:51 |
pabelanger | done! | 19:51 |
mordred | corvus: but maybe we shoudlnt' spend too much energy on fixing it until we're all the way to the yaml - bcause I thnik defining a data structure for "this is what an injected fake message should look like" is probably important there so that we can also inject the messages from the executor | 19:52 |
mordred | corvus: speaking of - https://logs.opendev.org/56/623256/7/check/tox-py36/eaa1c6d/testr_results.html.gz - | 19:54 |
mordred | yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/new:ansible.parsing.yaml.objects.AnsibleUnicode' | 19:54 |
mordred | corvus: we've got a trick somewhere already that lets us dump yaml in python3 without putting in unicode objects right? | 19:54 |
fungi | i have some, just a sec | 19:58 |
mordred | fungi: I thnik the trick in this case might be that I nede to import from zuul.lib.yamlutil instead of yaml? | 19:59 |
fungi | hrm, yeah i thought i had an example handy but didn't find it | 20:00 |
* pabelanger starts reading AWS driver docs for nodepool | 20:00 | |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Add appending yaml log plugin https://review.opendev.org/623256 | 20:02 |
fungi | mordred: not sure, looking at yamlutil.py it seems to mostly just be dealing with optional yamllib availability | 20:03 |
fungi | the dumper doesn't look like it does anything fancy | 20:04 |
mordred | hrm. yeah | 20:04 |
mordred | maybe just the difference between safe_dump and dump? | 20:04 |
mordred | yeah. I think that's it | 20:05 |
mordred | safe_dump(data, stream=None, **kwds) | 20:05 |
mordred | Serialize a Python object into a YAML stream. | 20:05 |
mordred | Produce only basic YAML tags. | 20:05 |
fungi | oh, got it | 20:05 |
fungi | yeah i guess that will do it | 20:06 |
mordred | we'll see :) | 20:06 |
fungi | anyway, using the class from zuul.lib.yamlutil seems like a good idea regardless for consistency | 20:06 |
*** saneax has quit IRC | 20:07 | |
openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Revert "Partial sync fetch-sphinx-tarball with fetch-sphinx-output" https://review.opendev.org/676511 | 20:13 |
corvus | mordred: yeah, i think this is fundamentally a different problem than the other one, but the solution will textually conflict, so we may as well solve them together? | 20:19 |
corvus | mordred: basically, i think you identified a place where we create a fake task record, but we forgot to set "failed: true" on it. | 20:20 |
corvus | that's a simple enough fix | 20:20 |
mordred | yeah | 20:20 |
corvus | mordred: hrm, now that i think about it, maybe we should just go ahead and fix that quickly then base the yaml appending stuff on that? | 20:21 |
mordred | corvus: or - actually ... | 20:21 |
mordred | corvus: I don't think that's actually the issue here | 20:21 |
corvus | mordred: just because i think this is a one-liner, and we're not sure how quickly we can roll out the yaml thing | 20:21 |
mordred | I think this is passthrough of incomplete data | 20:22 |
corvus | mordred: hrm. aiui, https://zuul.opendev.org/t/zuul/build/cfbe3f43bd604e509cc1b196cce66f15/log/job-output.json#6332 should have "failed": true under tasks.hosts.ubuntu-bionic | 20:22 |
mordred | corvus: yup | 20:23 |
mordred | but I don't think we're creating that | 20:23 |
corvus | oh neat | 20:23 |
mordred | yeah. "neat" | 20:25 |
mordred | I'm staring at it to see if there's any "good" way to do a fix in a couple of lines | 20:25 |
corvus | mordred: do you know which method gets called for that? | 20:25 |
mordred | the json plugin is very simple - it calls v2_runner_on_ok for both | 20:26 |
mordred | since all we're really doing is passing through the structure | 20:26 |
corvus | mordred: ah, so maybe we need to fan that out and add in a failed field if it isn't there? | 20:26 |
mordred | but what I think we might want to do instead is make actual methods | 20:26 |
mordred | yeah | 20:26 |
corvus | mordred: maybe it makes sense to have all 4 methods and store all 4 things, then we can make sure our statuses match ansible's | 20:27 |
corvus | right now, i think we're actually missing unreachable :) | 20:27 |
mordred | yeah | 20:27 |
mordred | and something like for host in self.results[-1]['tasks'][-1]['hosts'].values(): host.setfault('failed', True) | 20:28 |
mordred | in v2_runner_on_ok | 20:28 |
mordred | gah | 20:28 |
mordred | in v2_runner_on_failed | 20:29 |
corvus | mordred: how about just host.setfault('failed', True) ? | 20:29 |
mordred | should there be skipped=true and unreachable=true too then? | 20:29 |
corvus | mordred: we don't need to set the failed attribute on successful hosts | 20:29 |
mordred | ah - good point | 20:29 |
openstackgerrit | Merged zuul/zuul-jobs master: Revert "Partial sync fetch-sphinx-tarball with fetch-sphinx-output" https://review.opendev.org/676511 | 20:30 |
*** saneax has joined #zuul | 20:31 | |
corvus | mordred: and yeah, i think either that, or we should use one var (eg 'result') | 20:31 |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Set failed, unreachable, skipped statuses in json plugin https://review.opendev.org/676516 | 20:33 |
mordred | corvus: ^^ somethign liek that? | 20:33 |
corvus | mordred: if i'm reading the really roundabout code in the upstream json plugin correctly, it should set "failed: true" and "skipped: true" on the host result object | 20:35 |
mordred | oh yeah? | 20:36 |
openstackgerrit | Merged zuul/zuul master: Override the job name from which we promote https://review.opendev.org/676494 | 20:36 |
corvus | mordred: so i thkn your change mirrors the upstream behavior, except that i have no idea what happens on unreachable | 20:36 |
mordred | oh - gotcha. so set each of those | 20:36 |
corvus | mordred: so maybe we don't need to handle unreachable because something already does? i just don't know what | 20:37 |
corvus | mordred: https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/json.py#L139 | 20:39 |
corvus | i interpret that as "if the name of the method that is being called is 'v2_runner_on_failed' then set 'failed: True' on the host result | 20:40 |
corvus | ditto for skipped | 20:40 |
corvus | so maybe the host result already has a field for unreachable? | 20:40 |
mordred | maybe so? | 20:40 |
mordred | or maybe you only get a host if the host got reached? | 20:40 |
corvus | mordred: oh interesting | 20:41 |
corvus | mordred: so i think we're good on failed/skipped -- can you set up a local test to figure out unreachable? (i'm guessing if you just put a bogus thing in inventory...) | 20:41 |
mordred | corvus: yeah | 20:42 |
corvus | cool, i'm going to go do the swift thing, biab. | 20:42 |
mordred | corvus: cool. I'm about to EOD so I'll do the local test in the morning | 20:42 |
mordred | (well, I'm about to do a phone call - but when it's done I'll be EOD - so same result) | 20:43 |
corvus | ack | 20:43 |
pabelanger | so, just happen to look at tower release notes, new release today. And https://nvd.nist.gov/vuln/detail/CVE-2019-12439 was linked as security issue for bubblewrap | 20:46 |
pabelanger | looking at ubuntu bionic, I don't see any fix published yet | 20:47 |
*** rfolco has quit IRC | 20:51 | |
pabelanger | seems low chance | 20:53 |
pabelanger | likely why the delay on release | 20:53 |
pabelanger | however, first needs local SSH access to executor | 20:53 |
mordred | corvus: yes - unreachable=True is on the host record http://paste.openstack.org/show/757014/ | 20:55 |
openstackgerrit | Monty Taylor proposed zuul/zuul master: Set failed, unreachable, skipped statuses in json plugin https://review.opendev.org/676516 | 20:57 |
pabelanger | mordred: corvus: release job looks correct now: https://logs.opendev.org/94/676494/3/gate/zuul-build-python-release/eaefe32/artifacts/ | 20:57 |
pabelanger | and promote worked | 20:57 |
mordred | corvus: but also there was an error in the first patch | 20:57 |
mordred | pabelanger: woot | 20:57 |
pabelanger | think we are ready to tag | 20:57 |
pabelanger | confirmed, wheel is fine now | 20:59 |
pabelanger | Oh, looks like we'll pick up 2 commits too | 21:08 |
pabelanger | https://review.opendev.org/676466/ and https://review.opendev.org/676404/ | 21:09 |
*** saneax has quit IRC | 21:10 | |
*** jeliu_ has quit IRC | 21:16 | |
pabelanger | corvus: if you don't mind sending ping if you tag today, I'll then upgrade to 3.10.1 for zuul.a.c. For now, I have to #dadops for a bit | 21:18 |
pabelanger | thanks! | 21:18 |
openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Log errors better in case of unknown REST errors https://review.opendev.org/676476 | 21:22 |
openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Retry more operations https://review.opendev.org/676518 | 21:22 |
corvus | mordred, pabelanger: ^ i've tested that in rax and can confirm that it doesn't break the case where everything works :) | 21:22 |
corvus | mordred: (i also fixed a couple of things with your earlier patch) | 21:23 |
corvus | mordred: that's great news on the unreachable thing; i think 676516 should take care of it! it'll be easier to update the javascript once that's in production, so i say we wait for that for the next step | 21:25 |
corvus | pabelanger: i will start tagging .1 now | 21:25 |
corvus | 3aa36dd2d141a018ad5ba26146d979d4ac0d3eea will be the commit | 21:25 |
pabelanger | corvus: ack thanks, will look at swift stuff once back | 21:26 |
mordred | corvus: fwiw - since you're already importing openstacksdk - the iterate_timeout function is available if you want to use it | 21:36 |
mordred | corvus: openstack.utils.iterate_timeout | 21:36 |
mordred | corvus: (I like what you have - just mentioning it's availability) | 21:37 |
corvus | mordred: oh neat. well, i thought about using iterate_timeout but i thought this approach might be more suitable (with the built-in exception handler, it makes the calling site much simpler (at the expense of throwing in a lambda)) | 21:40 |
openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Retry more operations https://review.opendev.org/676518 | 21:42 |
openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Retry more operations https://review.opendev.org/676518 | 21:42 |
corvus | mordred: okay, i think i have added the required whitespace :) | 21:42 |
mordred | +@ | 21:46 |
*** rfolco has joined #zuul | 21:56 | |
*** mattw4 has quit IRC | 21:57 | |
*** mattw4 has joined #zuul | 21:58 | |
corvus | i sent out an announcement email, so i think that concludes the "forgot the javascript in the release" fire drill :) | 22:06 |
*** rlandy|rover has quit IRC | 23:12 | |
*** sshnaidm is now known as sshnaidm|afk | 23:38 | |
pabelanger | \o/ | 23:54 |
pabelanger | starting upgrades now | 23:54 |
pabelanger | oh, I'll start in a few hours, periodic jobs about to fire :) | 23:55 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!