Monday, 2019-05-13

*** jamesmcarthur has joined #zuul00:24
openstackgerritTristan Cacqueray proposed zuul/zuul master: web: honor job dependencies in trigger modal  https://review.opendev.org/65756700:30
*** armstrongs_ has quit IRC00:42
*** armstrongs has quit IRC00:43
*** altlogbot_3 has quit IRC00:50
*** altlogbot_1 has joined #zuul00:56
*** jamesmcarthur has quit IRC01:08
*** jamesmcarthur has joined #zuul01:08
openstackgerritTristan Cacqueray proposed zuul/zuul master: [WIP] zuul_stream: add debug to investigate tox-remote failures  https://review.opendev.org/65791401:14
*** jamesmcarthur has quit IRC01:24
*** jamesmcarthur has joined #zuul01:54
*** jamesmcarthur has quit IRC01:59
*** jamesmcarthur has joined #zuul01:59
openstackgerritTristan Cacqueray proposed zuul/zuul master: [WIP] zuul_stream: add debug to investigate tox-remote failures  https://review.opendev.org/65791402:25
*** jamesmcarthur has quit IRC02:31
*** jamesmcarthur has joined #zuul02:34
*** jamesmcarthur has quit IRC02:41
*** jamesmcarthur has joined #zuul02:43
*** jamesmcarthur has quit IRC02:57
*** bhavikdbavishi has joined #zuul03:15
*** lennyb has joined #zuul04:06
openstackgerritTristan Cacqueray proposed zuul/zuul master: [WIP] zuul_stream: add debug to investigate tox-remote failures  https://review.opendev.org/65791404:14
*** bhavikdbavishi1 has joined #zuul04:17
*** bhavikdbavishi has quit IRC04:17
*** bhavikdbavishi1 is now known as bhavikdbavishi04:17
*** bhavikdbavishi has quit IRC04:32
*** tjgresha has quit IRC04:57
*** tjgresha has joined #zuul04:57
*** jamesmcarthur has joined #zuul04:58
*** jamesmcarthur has quit IRC05:03
*** bhavikdbavishi has joined #zuul05:03
*** quiquell|off is now known as quiquell05:17
*** AJaeger has quit IRC05:59
*** AJaeger has joined #zuul06:03
*** pcaruana has joined #zuul06:16
*** saneax has joined #zuul06:24
*** bjackman has joined #zuul06:37
*** abraverm has quit IRC06:53
openstackgerritTristan Cacqueray proposed zuul/zuul master: [WIP] zuul_stream: add debug to investigate tox-remote failures  https://review.opendev.org/65791406:53
*** bhavikdbavishi has quit IRC07:01
*** gtema has joined #zuul07:04
openstackgerritTristan Cacqueray proposed zuul/zuul master: quick-start: add docker debug  https://review.opendev.org/65873807:17
openstackgerritTristan Cacqueray proposed zuul/zuul master: [WIP] zuul_stream: add debug to investigate tox-remote failures  https://review.opendev.org/65791407:42
*** abraverm has joined #zuul07:43
openstackgerritTristan Cacqueray proposed zuul/zuul master: quick-start: add docker debug  https://review.opendev.org/65873807:44
*** jpena|off is now known as jpena07:44
*** jangutter has joined #zuul08:05
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: wip: use-buildset-registry: install root config  https://review.opendev.org/65874608:15
openstackgerritTristan Cacqueray proposed zuul/zuul master: quick-start: add docker debug  https://review.opendev.org/65873808:16
*** abraverm has quit IRC08:30
openstackgerritTristan Cacqueray proposed zuul/zuul master: [WIP] zuul_stream: add debug to investigate tox-remote failures  https://review.opendev.org/65791408:32
*** bjackman has quit IRC08:44
*** bjackman has joined #zuul08:44
*** abraverm has joined #zuul08:44
*** hashar has joined #zuul08:45
*** bjackman_ has joined #zuul08:51
openstackgerritTristan Cacqueray proposed zuul/zuul master: zuul-quick-start: set buildset_registry_docker_user to root  https://review.opendev.org/65873808:53
*** bjackman has quit IRC08:54
*** gchenuet has joined #zuul09:00
openstackgerritTristan Cacqueray proposed zuul/zuul master: web: honor job dependencies in trigger modal  https://review.opendev.org/65756709:04
*** electrofelix has joined #zuul09:17
openstackgerritTristan Cacqueray proposed zuul/zuul master: zuul-quick-start: set buildset_registry_docker_user to root  https://review.opendev.org/65873809:26
tristanCzuul-maint: i think the zuul-quick-start job doesn't test speculative image without https://review.opendev.org/65873809:27
*** bhavikdbavishi has joined #zuul09:31
openstackgerritMark Meyer proposed zuul/zuul master: Add Bitbucket Server source functionality  https://review.opendev.org/65783709:34
openstackgerritMark Meyer proposed zuul/zuul master: Create a basic Bitbucket build status reporter  https://review.opendev.org/65833509:34
aspiersit's zuul socks day today09:36
*** wxy-xiyuan has joined #zuul09:39
openstackgerritTristan Cacqueray proposed zuul/nodepool master: static: fix missing python-path update  https://review.opendev.org/65851509:54
openstackgerritTristan Cacqueray proposed zuul/zuul master: [WIP] zuul_stream: add debug to investigate tox-remote failures  https://review.opendev.org/65791410:12
*** gtema has quit IRC10:43
*** themroc has joined #zuul10:43
*** bhavikdbavishi has quit IRC10:59
*** themroc has quit IRC11:14
*** bhavikdbavishi has joined #zuul11:26
*** jpena is now known as jpena|lunch11:33
*** snapiri has quit IRC11:59
*** gtema has joined #zuul12:05
*** rfolco|ruck has joined #zuul12:11
*** lennyb has quit IRC12:12
*** jamesmcarthur has joined #zuul12:12
*** rlandy has joined #zuul12:17
*** bhavikdbavishi has quit IRC12:21
*** quiquell is now known as quiquell|lunch12:25
*** jamesmcarthur has quit IRC12:31
*** saneax has quit IRC12:32
*** jamesmcarthur has joined #zuul12:49
*** pcaruana|afk| has joined #zuul12:55
*** pcaruana has quit IRC12:57
*** rlandy_ has joined #zuul12:58
*** wxy-xiyuan has quit IRC12:59
*** iamweswilson has quit IRC12:59
*** aprice has quit IRC12:59
*** iamweswilson has joined #zuul12:59
*** dmellado_ has joined #zuul12:59
*** aprice has joined #zuul12:59
*** maxamillion has quit IRC13:00
*** rlandy has quit IRC13:00
*** persia has quit IRC13:00
*** dmellado has quit IRC13:00
*** dmellado_ is now known as dmellado13:00
*** persia has joined #zuul13:00
*** rlandy_ is now known as rlandy13:02
openstackgerritMark Meyer proposed zuul/zuul master: Add Bitbucket Server source functionality  https://review.opendev.org/65783713:02
openstackgerritMark Meyer proposed zuul/zuul master: Create a basic Bitbucket build status reporter  https://review.opendev.org/65833513:02
*** maxamillion has joined #zuul13:03
*** wxy-xiyuan has joined #zuul13:14
*** markwork has joined #zuul13:15
*** jamesmcarthur has quit IRC13:26
*** gtema has quit IRC13:34
*** abraverm has quit IRC13:39
*** jpena|lunch is now known as jpena13:43
*** jamesmcarthur has joined #zuul13:46
*** snapiri has joined #zuul13:47
*** bjackman_ has quit IRC13:48
*** abraverm has joined #zuul13:53
*** bjackman_ has joined #zuul13:53
markworkDoes the recheck keyword in gerrit trigger a re-check in zuul?13:54
pabelangermarkwork: yes13:56
dmsimardmarkwork: the keyword is a configuration on the pipeline13:56
pabelangeryah13:56
pabelangerhttps://opendev.org/openstack/project-config/src/branch/master/zuul.d/pipelines.yaml#L2313:56
markworkdmsimard, pabelanger, Cool, thanks13:57
pabelangertristanC: tobiash: left a soft -1 on 63733914:15
tobiashpabelanger: thx14:17
pabelangerI can likely help mid-week if needed too14:18
pabelangerzuul-maint: I'd like to ask for reviews on https://review.opendev.org/631933/, this adds ansible 2.8 support to zuul, which is expected to land this week. It would be super awesome if we could get a release of zuul soon after! Big thanks to tobiash14:20
pabelangerit is -2 only because 2.8 isn't released, just yet14:21
openstackgerritMerged zuul/nodepool master: static: fix missing python-path update  https://review.opendev.org/65851514:21
Shrewspabelanger: you add new tasks in https://review.opendev.org/#/c/631933/18/tests/fixtures/config/remote-zuul-stream/git/org_project/playbooks/command.yaml but i don't see where you attempt to make sure those are not output14:29
Shrewswithin test_remote_zuul_stream.py, that is14:31
Shrewsmaybe i'm missing the point14:32
pabelangerShrews: I think I can remove them, IIRC I added them when trying to debug another issue14:34
openstackgerritMark Meyer proposed zuul/zuul master: Add Bitbucket Server source functionality  https://review.opendev.org/65783714:36
openstackgerritMark Meyer proposed zuul/zuul master: Create a basic Bitbucket build status reporter  https://review.opendev.org/65833514:36
openstackgerritMark Meyer proposed zuul/zuul master: Create a basic Bitbucket event source  https://review.opendev.org/65883514:36
pabelangerShrews: yah, that was to try and figure out the test_command failure in tox-remote. I thought it was related to 2.8, but that isn't the case. I'll remove that shortly14:38
Shrewsk14:38
*** jamesmcarthur_ has joined #zuul14:39
openstackgerritMerged zuul/nodepool master: Cleanup kube_config temp files between test runs  https://review.opendev.org/65710014:40
Shrewspabelanger: also  "We've had to update zuul_console to deal with new tasks stats, along with gather_facts also now being exposed directly to the job."     Where is that done?  zuul_console didn't change afaict14:41
*** jamesmcarthur has quit IRC14:43
pabelangerShrews: that is a typo, it should be zuul_stream14:43
pabelangerI'll fix that too14:43
pabelangerhttps://review.opendev.org/#/c/631933/18/zuul/ansible/2.8/callback/zuul_stream.py14:43
pabelangerline 524-52514:43
Shrewspabelanger: can you point out those points in that file to help us review since the entire file is "new"?14:44
pabelangerthe more I think of it, maybe we should add a version check to current zuul_stream rather then copypata 640 lines14:44
Shrewsi think that would be better, imo14:44
*** snapiri has quit IRC14:44
pabelangerk, let me figure that out14:45
*** bjackman_ has quit IRC14:49
*** abraverm has quit IRC14:49
*** quiquell|lunch is now known as quiquell|off14:49
*** abraverm has joined #zuul14:50
*** hashar has quit IRC14:51
*** pcaruana|afk| has quit IRC14:55
openstackgerritMark Meyer proposed zuul/zuul master: Create a basic Bitbucket build status reporter  https://review.opendev.org/65833514:57
openstackgerritMark Meyer proposed zuul/zuul master: Create a basic Bitbucket event source  https://review.opendev.org/65883514:57
corvuszuul-maint: i have another weird schedule this week -- today is a full day for me, but starting tomorrow i'm going to be mostly unavailable until next week.  things should return to normal next wednesday.15:00
tobiashcorvus: thanks for the note15:00
corvusi should be able to work on the review backlog on the plane with gertty15:00
corvusbut while i'm here today, is there anything i should focus on?15:01
Shrewscorvus: https://review.opendev.org/590335 is ready for your eyes15:01
corvus(i just sent out the delayed release announcements for the releases we made last week)15:01
tobiashcorvus: I'd appreciate if you could have a quick look (no thorrow review yet) at https://review.opendev.org/#/q/status:open+project:zuul/zuul+branch:master+topic:annotate-logs to see if I'm heading towards a good direction before I continue on that15:02
tobiashbut if it doesn't fi into your scheduler it's fine to wait for a week too15:03
tobiashs/fi/fit15:03
* Shrews AFK for PT15:03
corvusShrews: +3 thx15:05
corvustobiash: i like the direction there.  we have some existing annotations, like the build id on the executor.  i think it would also be useful to have the build id on all nodepool requests too.  so are you thinking that once the scheduler creates the builds for a change, we'll switch to using build-id-annotated loggers?  or should we include both the event id and the build id?15:09
tobiashcorvus: my intention was that when we have the build to have the event as well as the build15:10
tobiashand also supply the event to nodepool via the node request and annotate there as well if the information is there15:10
*** nickx-intel has joined #zuul15:11
corvustobiash: ok, sounds good.  if the log lines get too long, maybe we should think about using shorter uids to make things friendlier for the standard text logging config (of course, advanced logging systems like journald or elastic can just take the info as extra metadata).  but we probably don't have to solve that in advance.15:12
tobiashyes, that could be done easyly by some sort of transformation when generating the event id15:13
tobiashthe build id also probably could be shortened easily15:13
nickx-intelmorning zuulies. I'm having issues getting zuul check & gate checking & gating in a new install .... my initial git review(s) aren't showing in gerrit15:22
nickx-intelI did `git push` as admin but my push doesn't show in gerrit changes (host:8080) nor in tenent projects (host:9000)15:27
*** abraverm has quit IRC15:29
openstackgerritMerged zuul/nodepool master: Implement an OpenShift Pod provider  https://review.opendev.org/59033515:30
pabelangernickx-intel: have you installed and tried using git-review? https://pypi.org/project/git-review/15:32
pabelangerthat should help with pushing changes to gerrit15:33
*** yolanda_ has quit IRC15:34
nickx-intelI initially attempted via git review pabelanger15:35
nickx-intelmy initial commit shows in git log but not in gerrit15:35
pabelangernickx-intel: possible your remotes are not setup correctly, if you run git-review -s, what is the output?15:37
*** yolanda_ has joined #zuul15:37
pabelangerthen you can run git-review -n for dry-run15:37
nickx-intelgit-review -s outputs nothing pabelanger15:38
nickx-intel(I did just restart my containers)15:38
nickx-intelgit log still shows my past commits15:39
pabelangernickx-intel: do you have a .gitreview file in your local repo? for example: https://opendev.org/zuul/zuul/src/branch/master/.gitreview15:39
pabelangerthat should be setup with your gerrit / project info15:39
nickx-intelI have ../zuul-config/.git-review [gerrit]15:40
pabelangerlooks like some info is missing for some reason15:41
nickx-intelsorry I said .git-review; it's .gitreview15:41
nickx-intel`git remote` shows remote gerrit15:43
*** pcaruana|afk| has joined #zuul15:44
markworktobiash: I integrated most of the comments you made, can you also kick of the review process on https://review.opendev.org/#/c/658335/?15:45
markworkOk, I'm off, I'll have a log at the logs, laters15:47
nickx-intelI added --- to the .yaml's; git add *yaml; git review - this reports SUCCESS via commandline - pabelanger15:47
nickx-intelthe changes still don't appear on the gerrit dashboard; however I can follow the link of remote changes to the changes and review (with link)15:48
nickx-inteloh, if I log out, the changes appear on the dashboard15:50
nickx-intelweird15:50
pabelangercool15:51
pabelangerunfortunately, I have to AFK for a few hours, but can help more when I get back15:51
*** markwork has quit IRC15:51
nickx-intelsure pabelanger - just sheep shearing over here haha15:51
*** jangutter has quit IRC15:52
*** bjackman_ has joined #zuul15:52
nickx-intelI'm now trying to get zuul to pick up its check and gate pipelines .. it looks like zuul still isn't interacting with the git review process in my environment15:54
openstackgerritMerged zuul/zuul master: Improve proxy settings support for compose env  https://review.opendev.org/65514015:55
corvusnickx-intel: when git-review pushes a change up to gerrit, the zuul-scheduler process should see that event and start processing.  you may find clues in the scheduler logs.15:56
*** hashar has joined #zuul15:59
nickx-intelcorvus - logs within scheduler container? shortest path to access?16:03
corvusnickx-intel: yes, if you're using the quick-start, they go to stdout or stderr, so you can run "docker logs examples_scheduler_1" (i think that's the container name)16:04
openstackgerritMerged zuul/zuul master: Add some packages for basic python jobs  https://review.opendev.org/65514116:06
nickx-inteldo I have to code-review as admin to get zuul to notice corvus? or can I configure what zuul wants to review?16:10
corvusnickx-intel: if you are following the instructions in the quick-start docs ( https://zuul-ci.org/docs/zuul/admin/quick-start.html ) then by the end, zuul should run check jobs on any change that is uploaded.16:11
nickx-intelI'm suspecting "Timeout waiting for gerrit" is causal16:11
corvusnickx-intel: yes, that means that zuul hasn't started -- something has gone very wrong with the docker-compose setup.16:12
nickx-intelhmm16:12
corvusnickx-intel: do you have any http_proxy env vars set?16:13
nickx-intelYes, good call - I unset and re-`docker-compose up` - executor container looks happier16:14
corvusnickx-intel: we also *just* merged a change (20 minutes ago) to improve that as well: https://review.opendev.org/655140  perhaps that would address the issue you saw (or suggest another improvement)16:15
nickx-intelhmm ... I've unset all environment variables pertaining to proxy & no_proxy16:18
Shrewscorvus: i didn't land the zk auth nodepool change before the ptg b/c of the mem leak work (didn't want any big changes to land). I guess we're ok to land that now unless you can think of a reason not to do so.16:28
corvusShrews: are you happy with the python-malloc explanation of the memory change?  so we're shelving further investigation into potentials leaks?16:29
clarkbI think it is possible the leaking loggers were more costly than initially assumed too16:29
corvusoh i see16:29
nickx-intelI'm getting via executor_1: executor server / executor merger : exception connecting to <gear.Connection [container] Scheduler port 4730>16:29
Shrewscorvus: the graphs still look good16:29
corvusShrews: but yeah, can't think of another reason to hold it off.16:29
clarkbwe launch tens of thousands of nodes a day on opendev nodepool and many more if things fail a lot16:29
corvusnickx-intel: that's expected if the scheduler isn't starting16:30
corvusclarkb: i thought the thought on that was that it probably wasn't a change (ie, that's been in there since at least nodepool v3, if not earlier)16:30
corvusnickx-intel: so if your scheduler container is still waiting for gerrit, then the rest of the system would emit errors like that16:31
Shrewscorvus: i still cannot explain the sudden jump around nov-dev timeframe. but the jemalloc stuff seems to have helped16:31
clarkbcorvus: ah16:32
clarkbjemalloc is better at packing and reusing blocks right?16:33
clarkb(possible some change was good at fragmenting memory but isn't a direct leak?)16:33
Shrewsyeah, the logger leak has been around since the beginning. i don't think that was the major cause16:33
Shrewspossible a python or other library update in that time period exposed the problem for us more?16:35
Shrewsbut memory is sooo much more leveled off now   http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=64816&rra_id=0&view_type=tree&graph_start=1556555738&graph_end=155776533816:36
nickx-intelit looks like I have a few containers failing corvus16:36
nickx-intelI'm manually starting zuul-scheduler, zuul-executor, zuul-web16:36
Shrewscorvus: actually, maybe we should wait until we can land the zuul zk-auth change at the same time16:38
Shrewsstill a -1 on that one16:38
corvusShrews: ack16:38
*** mgoddard has quit IRC16:50
*** mgoddard has joined #zuul16:52
*** markwork has joined #zuul16:53
nickx-intelit looks like these containers are failing to run because they're failing to authenticate, corvus; "authentication from 172.x.x.x failed for admin: password does not match the one stored in gerrit" then the containers exit with code 116:55
nickx-intelthis one's google'able so I'm looking ..16:57
*** mgoddard has quit IRC16:57
*** mgoddard has joined #zuul16:58
*** jpena is now known as jpena|off17:08
clarkbpabelanger: when you say facts are displayed by default in 2.8 do you mean they will show up in our console logs?17:09
ShrewstristanC: I left some comments on the runc driver. I'm pretty uncomfortable with using Ansible for provisioning without some pretty thorough discussion around that.  My fear is we are going to be doing the ansible dependency dance we are currently doing with zuul. Perhaps mordred, corvus and tobiash can add some thoughts to that.17:11
tobiashI also think using ansible inside nodepool must be thought through carefully. I also think it can become a scalability problem on the nodepool side because of the amount of resources ansible needs.17:15
corvusi'm excited about adding new provider drivers to nodepool, but i think we need to be cautious about making nodepool itself a cloud provider.17:15
nickx-intelcan you expound on that corvus?17:16
*** mgoddard has quit IRC17:16
corvusnickx-intel: on what?17:16
nickx-intel> "i think we need to be cautious about making nodepool itself a cloud provider."17:17
corvusnickx-intel: well, nodepool's job is to talk to cloud provider apis.  but the runc driver doesn't do that.17:17
*** gtema has joined #zuul17:18
corvusif we don't draw the line somewhere, we might end up creating our own container orchestration engine or virtualization manager.  there are already projects for that.17:20
nickx-intelcan you link me to runc documentation please corvus?17:21
*** mgoddard has joined #zuul17:21
corvusnickx-intel: sorry i don't understand the question17:22
nickx-intelnvm I can just google runc driver - thanks corvus +117:22
Shrewsnickx-intel: for reference, https://review.opendev.org/535556 is the change we are discussing.17:23
corvusnickx-intel: the runc driver i was talking to Shrews and tobiash about is not merged into nodepool yet, and there are serious concerns about it.  i would not recommend using it for new deployments.17:23
nickx-intelthanks Shrews17:24
*** gchenuet has quit IRC17:25
openstackgerritFabien Boucher proposed zuul/zuul master: A reporter for Elasticsearch with the capability to index build and buildset results in an index.  https://review.opendev.org/64492717:27
*** hashar has quit IRC17:29
*** mattw4 has joined #zuul17:33
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate merger logs with event id  https://review.opendev.org/65865217:34
openstackgerritFabien Boucher proposed zuul/zuul master: A reporter for Elasticsearch with the capability to index build and buildset results in an index.  https://review.opendev.org/64492717:36
*** markwork has quit IRC17:47
*** jamesmcarthur_ has quit IRC17:58
*** jamesmcarthur has joined #zuul18:03
*** gtema has quit IRC18:20
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate job freezing logs with event id  https://review.opendev.org/65888818:30
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate node request processing with event id  https://review.opendev.org/65888918:30
*** jamesmcarthur has joined #zuul18:31
*** markwork has joined #zuul18:33
*** hashar has joined #zuul18:38
*** hashar has quit IRC18:38
*** abraverm has joined #zuul18:46
*** jamesmcarthur has quit IRC18:49
*** bjackman_ has quit IRC18:52
openstackgerritTobias Henkel proposed zuul/zuul master: WIP: Annotate builds with event id  https://review.opendev.org/65889518:57
openstackgerritTobias Henkel proposed zuul/zuul master: WIP: Annotate builds with event id  https://review.opendev.org/65889519:01
*** jamesmcarthur has joined #zuul19:07
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate merger logs with event id  https://review.opendev.org/65865219:21
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate job freezing logs with event id  https://review.opendev.org/65888819:21
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate node request processing with event id  https://review.opendev.org/65888919:21
openstackgerritTobias Henkel proposed zuul/zuul master: WIP: Annotate builds with event id  https://review.opendev.org/65889519:21
*** nhicher has joined #zuul19:24
pabelangerclarkb: yes19:24
*** persia has quit IRC19:35
*** persia has joined #zuul19:36
*** markwork has quit IRC20:10
*** panda has quit IRC20:19
*** panda has joined #zuul20:21
*** pcaruana|afk| has quit IRC20:46
*** abraverm has quit IRC20:56
*** mattw4 has quit IRC21:05
*** openstackgerrit has quit IRC21:09
*** mattw4 has joined #zuul21:32
*** jamesmcarthur has quit IRC21:40
*** gothicmindfood has joined #zuul22:29
*** saneax has joined #zuul23:05
*** openstackgerrit has joined #zuul23:18
openstackgerritTristan Cacqueray proposed zuul/zuul master: DNM: show zuul-quick-start doesn't test speculative images  https://review.opendev.org/65893523:18
*** rlandy is now known as rlandy|bbl23:25
*** mattw4 has quit IRC23:35
*** mattw4 has joined #zuul23:35
*** mattw4 has quit IRC23:43
clarkbtristanC: am I reading the output of the quick-start job for ^ properly? looks like it waited for gearman to start and gave up?23:57
clarkb(I think that would imply your chagnes were used in that test?)23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!