| *** threestrands has joined #openstack-infra | 00:04 | |
| clarkb | fungi: does gertty use gerritlib? | 00:06 |
|---|---|---|
| clarkb | (I think its all http so no) | 00:06 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add static01.opendev.org records https://review.opendev.org/704469 | 00:07 |
| fungi | it does not | 00:07 |
| fungi | corvus observed that the gerrit rest api is plenty well-defined that wrapping it in anything fancier than the requests library is somewhat pointless | 00:08 |
| fungi | though i think several of us wound up at the same conclusion | 00:09 |
| fungi | when doing the rewrite of openstack's election tooling a while back it was faster and easier to read the rest api docs than to figure out some additional python library | 00:10 |
| clarkb | ya gerritlib is ssh only too | 00:10 |
| paladox | pygerrit2 is rest api :) | 00:11 |
| *** smarcet has quit IRC | 00:12 | |
| clarkb | right I want ssh :) | 00:12 |
| openstackgerrit | Clark Boylan proposed opendev/gerritlib master: Use newer paramiko https://review.opendev.org/704471 | 00:13 |
| openstackgerrit | Clark Boylan proposed opendev/gerritlib master: Convert gerrit output to utf8 unicode https://review.opendev.org/704472 | 00:13 |
| clarkb | fungi: ^ the first one bumping paramiko should fix status upgrade | 00:14 |
| fungi | cool, will take a look | 00:14 |
| clarkb | the second was something I noticed when testing the change. I ran listProjects against review-dev and old code only worked with python2, new code seems to wrok with 2 and 3 | 00:14 |
| clarkb | ianw: ^ you too | 00:14 |
| paladox | clarkb https://pypi.org/project/pygerrit/ supports ssh :) | 00:14 |
| clarkb | paladox: ya iirc it came years after gerritlib | 00:14 |
| paladox | ah | 00:14 |
| clarkb | so we were already using this lib for a long time in various places | 00:15 |
| clarkb | it would probably make sense to convert, but updating paramiko is much easier right now | 00:15 |
| fungi | there have been a bunch of active gerritlib reviews recently, so we may want to take a stroll through those before blindly approving new patches for it | 00:16 |
| clarkb | fungi: ya I skimmed it and couldn't find any that bumped paramiko | 00:16 |
| clarkb | and https://review.opendev.org/#/c/412536/6/gerritlib/gerrit.py for example has the python3 problem (out.split's arg is a unicode string but out is a bytestring and it goes boom) | 00:17 |
| clarkb | I don't think any of the changes there address these problems | 00:17 |
| fungi | aah, the only recently-active ones are pelix's 412535 and 412536 | 00:18 |
| fungi | tristanC's 412757 is also borderline recently-active | 00:19 |
| clarkb | paladox: http://codesearch.openstack.org/?q=gerritlib&i=nope&files=&repos= gives a rough idea of what would need converting. It isn't all that much, but does tend to be things that are neglected :/ | 00:19 |
| paladox | heh | 00:19 |
| clarkb | germqtt, elastic-recheck, gerritbot and jeepyb | 00:19 |
| fungi | i mostly mean look through them in terms of approving them if we can and think they're reasonable so we're not just approving each others' changes and ignoring other contributors ;) | 00:20 |
| clarkb | thats fair, but those repos are pretty dead due to lack of contributors helping :/ | 00:20 |
| clarkb | it would be really really nice if the QA team could help with e-r more | 00:21 |
| clarkb | gmann: ^ | 00:21 |
| clarkb | I think the mqtt stuff was on my list of services to consider cleaning up due to lack of use | 00:21 |
| clarkb | the other three should probably use pygerrit assuming it is still maintained | 00:21 |
| *** rlandy is now known as rlandy|afk | 00:24 | |
| fungi | it looks like pelix's two changes and tristanC's change to gerritlib just need one more +2 anyway | 00:25 |
| fungi | i had apparently already reviewed some of them | 00:26 |
| fungi | i approved 412757 and i already had a +2 on 412535 and 412536 (the parent change needs a second +2 but the child change already has 2x +2 | 00:28 |
| clarkb | 757 is a bit odd, looking at fd internals | 00:31 |
| clarkb | it would probably be better to call read() until you get back '', then split what you got | 00:32 |
| clarkb | fungi: 757 conflicts with 535 fwiw | 00:32 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add static01.opendev.org https://review.opendev.org/704476 | 00:34 |
| fungi | yeah, i hadn't seen that trick with readline()'s _pos and _realpos attributes before | 00:35 |
| ianw | ^ with that in i think we'll be able to test governance/security with host overrides | 00:35 |
| ianw | before we flip the dns switch | 00:35 |
| clarkb | fungi: I've +2'd 535 but not approved as 757 will conflict with it | 00:39 |
| fungi | i removed my votes from 757 following your comments in here | 00:40 |
| clarkb | ianw: +2 | 00:40 |
| clarkb | fungi: ah in that case feel free to approve the other two | 00:40 |
| fungi | ianw: in playbooks/roles/letsencrypt-create-certs/handlers/main.yaml how do the repeated roles/letsencrypt-create-certs/handlers/restart_apache.yaml entries get scoped to different vhosts? | 00:40 |
| *** jhesketh has joined #openstack-infra | 00:41 | |
| clarkb | fungi: they should get run multiple times | 00:41 |
| clarkb | fungi: but since its a graceful restart I'm not too worried about it. Also only happens every 2 months | 00:41 |
| fungi | more just wondering why it's the same role included multiple times under different names with no conditional | 00:41 |
| clarkb | fungi: re gerritlib if we land those changes we should plan to do a release tomorrow? | 00:41 |
| fungi | clarkb: i think so, yes | 00:42 |
| clarkb | fungi: the handler name is the conditional, since we trigger the specific handler by name only when that cert is updated | 00:42 |
| clarkb | fungi: its like a function look up table | 00:42 |
| fungi | oh, so that "letsencrypt updated static01-opendev-org-main" name gets referenced explicitly elsewhere? | 00:42 |
| clarkb | fungi: yes, by the code that updates the contents on disk for static01-opendev-org-main (we basically call the update code for each entry at https://review.opendev.org/#/c/704476/1/playbooks/host_vars/static01.opendev.org.yaml it decides if it needs to do anything and if so triggers the handler via that lookup table) | 00:43 |
| *** mattw4 has quit IRC | 00:43 | |
| fungi | i guess that happens in a future patch not yet written | 00:43 |
| *** hwoarang has quit IRC | 00:43 | |
| clarkb | no thats already written | 00:43 |
| clarkb | its just variable based on what is in the host/group file | 00:44 |
| clarkb | so not explicit in the code | 00:44 |
| clarkb | ianw: can you review https://review.opendev.org/#/c/704471/ as that is another fix necessary for status.o.o | 00:44 |
| clarkb | that and a gerritlib release including it | 00:44 |
| openstackgerrit | Merged opendev/system-config master: Flot sources moved, update our minimization to accomodate https://review.opendev.org/704467 | 00:46 |
| *** hwoarang has joined #openstack-infra | 00:46 | |
| clarkb | re yui minimization is the concern that it might produce code that isn't valid in modern browsers? | 00:47 |
| clarkb | it doesn't do the fancy mapping files to make sense of what is what which I guess is another downside | 00:47 |
| openstackgerrit | Julia Kreger proposed openstack/diskimage-builder master: Add handling for no activate_this.py in venv https://review.opendev.org/704478 | 00:48 |
| clarkb | TheJulia: ianw ^ does that inappropriately detect the no venv case as a venv case? | 00:49 |
| *** smarcet has joined #openstack-infra | 00:50 | |
| clarkb | I guess it probably doesn't matter if that happens | 00:50 |
| TheJulia | clarkb: no, venv lacks actiate_this.py which virtualenv for python2 provided | 00:51 |
| TheJulia | python3.3 introduced the other variables in sys as handled by default as well | 00:51 |
| clarkb | TheJulia: one more comment left inline | 00:55 |
| openstackgerrit | Julia Kreger proposed openstack/diskimage-builder master: Add handling for no activate_this.py in venv https://review.opendev.org/704478 | 00:58 |
| clarkb | upon further investigation I think my initial concern is still valid | 00:58 |
| clarkb | because we'll end up with /usr/local/bin in sys.path which isn't quite right | 00:58 |
| clarkb | (probably won't break anything in most cases though) | 00:58 |
| clarkb | oh except we are guarded at the top | 00:59 |
| clarkb | I am up to speed now, thanks | 00:59 |
| ianw | TheJulia: ahhh, yes i think i hit this and meant to do something about it :/ | 01:00 |
| ianw | i think that seems about right, thanks | 01:01 |
| TheJulia | ianw: it shows up nicely in bifrost's build jobs when we switched over to venv becuase virutalenv seemed to implode on us. | 01:01 |
| ianw | TheJulia: is there a depends-on i can monitor just to be sure? | 01:02 |
| TheJulia | nope, but should be easy to give a spin. will double check real quick | 01:03 |
| TheJulia | ianw: https://review.opendev.org/704480 | 01:05 |
| ianw | i wonder if, since we moved dib-run-parts into lib/, if we actually need all that any more. one to put in the todo list i guess | 01:06 |
| TheJulia | the dibipa job should explode nicely if it doesn't work | 01:06 |
| ianw | cool, thanks, will keep an eye | 01:06 |
| *** signed8b_ has joined #openstack-infra | 01:07 | |
| *** signed8bit has quit IRC | 01:11 | |
| TheJulia | ianw: we might not be installing from git.... I feel like that might have changed since we didn't feel the need to pull in latest items :\ | 01:14 |
| TheJulia | heh, wrong patch | 01:15 |
| openstackgerrit | sebastian marcet proposed opendev/puppet-openstackid master: Fixed broken deployment https://review.opendev.org/704482 | 01:16 |
| *** gyee has quit IRC | 01:17 | |
| *** jamesmcarthur has joined #openstack-infra | 01:18 | |
| clarkb | fungi: were you going to approve those gerritlib changes? | 01:21 |
| clarkb | 535 and 536 | 01:22 |
| TheJulia | ianw: yeah... some recent changes appear to have nuked out the git install | 01:22 |
| fungi | clarkb: yep, just about to circle back around to them | 01:23 |
| fungi | git sidetracked refreshing my memory of how the letsencrypt handlers are mapped | 01:24 |
| TheJulia | ianw: https://review.opendev.org/704480 updated to swap install orders so it doesn't get downgraded to the latest release | 01:25 |
| *** KeithMnemonic1 has joined #openstack-infra | 01:25 | |
| *** KeithMnemonic2 has joined #openstack-infra | 01:26 | |
| *** yamamoto has joined #openstack-infra | 01:26 | |
| *** KeithMnemonic has quit IRC | 01:29 | |
| *** KeithMnemonic1 has quit IRC | 01:30 | |
| openstackgerrit | Merged opendev/gerritlib master: Reuse same paramiko connection recreating on errors https://review.opendev.org/412535 | 01:32 |
| openstackgerrit | Merged opendev/gerritlib master: Support querying for single group info https://review.opendev.org/412536 | 01:32 |
| clarkb | ianw: I've got to pop out for dinner soon but review on https://review.opendev.org/#/c/704471/ would be excellent if possible before tomorrow | 01:33 |
| clarkb | I'll try to make a gerritlib release in the morning if that lands | 01:33 |
| openstackgerrit | Merged opendev/system-config master: Add static01.opendev.org https://review.opendev.org/704476 | 01:37 |
| *** jamesmcarthur has quit IRC | 01:43 | |
| *** factor has joined #openstack-infra | 01:46 | |
| *** diablo_rojo has joined #openstack-infra | 01:59 | |
| openstackgerrit | Merged opendev/gerritlib master: Use newer paramiko https://review.opendev.org/704471 | 02:00 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: WIP: add CONTRIBUTING guidelines https://review.opendev.org/704484 | 02:02 |
| *** jamesmcarthur has joined #openstack-infra | 02:07 | |
| openstackgerrit | sebastian marcet proposed opendev/puppet-openstackid master: Fixed broken deployment https://review.opendev.org/704482 | 02:15 |
| *** jamesmcarthur has quit IRC | 02:19 | |
| *** yamamoto has quit IRC | 02:19 | |
| *** yamamoto has joined #openstack-infra | 02:20 | |
| openstackgerrit | Merged opendev/gerritlib master: Convert gerrit output to utf8 unicode https://review.opendev.org/704472 | 02:28 |
| *** roman_g has quit IRC | 02:34 | |
| *** yamamoto has quit IRC | 02:39 | |
| *** gfidente has quit IRC | 02:40 | |
| ianw | hrm, static.opendev got certs but didn't seem to get a webserver | 02:42 |
| *** yamamoto has joined #openstack-infra | 02:42 | |
| *** yamamoto has quit IRC | 02:42 | |
| ianw | fatal: [mirror01.regionone.linaro-us.opendev.org]: UNREACHABLE! => {"changed": false, "msg": "Data could not be sent to remote host \"2604:1380:4111:3e54:f816:3eff:fea4:899d\" | 02:43 |
| ianw | that seems bad too, but one thing at a time | 02:43 |
| *** yamamoto has joined #openstack-infra | 02:43 | |
| *** jamesmcarthur has joined #openstack-infra | 02:44 | |
| *** jamesmcarthur has quit IRC | 02:45 | |
| ianw | oh i've forgotten to add it to the cron rub | 02:46 |
| ianw | run | 02:46 |
| *** jamesmcarthur has joined #openstack-infra | 02:50 | |
| *** yamamoto has quit IRC | 02:52 | |
| *** diablo_rojo has quit IRC | 02:53 | |
| *** jamesmcarthur_ has joined #openstack-infra | 02:54 | |
| *** jamesmcarthur has quit IRC | 02:54 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add service-static.yaml to cron run https://review.opendev.org/704491 | 02:54 |
| *** jamesmcarthur_ has quit IRC | 03:03 | |
| *** jamesmcarthur has joined #openstack-infra | 03:03 | |
| *** factor has quit IRC | 03:05 | |
| *** factor has joined #openstack-infra | 03:06 | |
| *** jamesmcarthur has quit IRC | 03:08 | |
| *** rkukura has quit IRC | 03:13 | |
| ianw | kevinz: so, related to ^^ it seems the linaro-us mirror has gone down and i can't see anything useful; filed https://storyboard.openstack.org/#!/story/2007195 task #38491 | 03:16 |
| ianw | i'm also pretty sure that along with other things going on in CN it's new year, which i imagine means holidays too so not sure on latency for a fix (no rush, as such, we still have the other comms issues to fix up too) | 03:18 |
| ianw | i'll leave it for now. my comms eqiupment has arrived so offline for a bit to plug it all in | 03:19 |
| *** yamamoto has joined #openstack-infra | 03:29 | |
| *** jamesmcarthur has joined #openstack-infra | 03:34 | |
| *** jamesmcarthur has quit IRC | 03:41 | |
| *** zxiiro has quit IRC | 03:41 | |
| *** diablo_rojo has joined #openstack-infra | 04:01 | |
| *** rcernin has quit IRC | 04:18 | |
| *** rcernin has joined #openstack-infra | 04:18 | |
| *** armax has quit IRC | 04:24 | |
| *** smarcet has quit IRC | 04:27 | |
| *** goldyfruit_ has quit IRC | 04:27 | |
| *** goldyfruit_ has joined #openstack-infra | 04:27 | |
| *** jamesmcarthur has joined #openstack-infra | 04:37 | |
| *** jamesmcarthur has quit IRC | 04:41 | |
| *** rlandy|afk has quit IRC | 04:48 | |
| *** ykarel|away is now known as ykarel | 04:49 | |
| *** smarcet has joined #openstack-infra | 04:55 | |
| *** smarcet has left #openstack-infra | 05:10 | |
| *** jamesmcarthur has joined #openstack-infra | 05:12 | |
| *** goldyfruit_ has quit IRC | 05:13 | |
| *** goldyfruit_ has joined #openstack-infra | 05:13 | |
| *** jamesmcarthur has quit IRC | 05:18 | |
| *** udesale has joined #openstack-infra | 05:25 | |
| *** raukadah is now known as chkumar|rover | 05:34 | |
| mordred | ianw: in the service-static playbook - those a2ensite commands have creates: on them - wouldn't that prevent the notify from firing if the file exists but the contents change? (I have no idea of the answer- wondering if you do) | 05:43 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add service-static.yaml to cron run https://review.opendev.org/704491 | 05:46 |
| mordred | ianw: also - ^^ fixed a copy-pasta | 05:46 |
| *** toabctl has quit IRC | 06:05 | |
| *** factor has quit IRC | 06:05 | |
| *** threestrands has quit IRC | 06:08 | |
| *** lmiccini has joined #openstack-infra | 06:09 | |
| *** toabctl has joined #openstack-infra | 06:12 | |
| *** jamesmcarthur has joined #openstack-infra | 06:13 | |
| *** rkukura has joined #openstack-infra | 06:16 | |
| *** lpetrut has joined #openstack-infra | 06:16 | |
| *** jamesmcarthur has quit IRC | 06:18 | |
| *** jtomasek has joined #openstack-infra | 06:35 | |
| *** psachin has joined #openstack-infra | 06:39 | |
| openstackgerrit | Merged opendev/system-config master: Add service-static.yaml to cron run https://review.opendev.org/704491 | 06:43 |
| *** sshnaidm|afk is now known as sshnaidm | 06:48 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add Apache to Ansible for Gerrit https://review.opendev.org/704498 | 06:48 |
| *** ahosam has quit IRC | 06:50 | |
| *** jamesmcarthur has joined #openstack-infra | 06:51 | |
| mordred | ianw: have we gotten anywhere yet with letsencrypt for openstack.org domains? | 06:54 |
| *** jamesmcarthur has quit IRC | 06:56 | |
| *** pgaxatte has joined #openstack-infra | 07:18 | |
| *** jtomasek has quit IRC | 07:24 | |
| *** ociuhandu has joined #openstack-infra | 07:30 | |
| *** dpawlik has joined #openstack-infra | 07:34 | |
| *** ykarel is now known as ykarel|lunch | 07:35 | |
| *** ociuhandu has quit IRC | 07:35 | |
| *** yamamoto has quit IRC | 07:37 | |
| *** slaweq has joined #openstack-infra | 07:39 | |
| *** xek has joined #openstack-infra | 07:45 | |
| *** jamesmcarthur has joined #openstack-infra | 07:52 | |
| *** jamesmcarthur has quit IRC | 07:57 | |
| *** aedc has quit IRC | 08:04 | |
| *** rpittau|afk is now known as rpittau | 08:07 | |
| *** jtomasek has joined #openstack-infra | 08:07 | |
| *** yamamoto has joined #openstack-infra | 08:12 | |
| *** yamamoto has quit IRC | 08:12 | |
| *** yamamoto has joined #openstack-infra | 08:12 | |
| *** tosky has joined #openstack-infra | 08:13 | |
| AJaeger | config-core, could you review https://review.opendev.org/704306 and https://review.opendev.org/704321, please? | 08:15 |
| *** tesseract has joined #openstack-infra | 08:16 | |
| *** dchen has quit IRC | 08:20 | |
| *** iurygregory has joined #openstack-infra | 08:24 | |
| *** jamesmcarthur has joined #openstack-infra | 08:27 | |
| *** ykarel|lunch is now known as ykarel | 08:31 | |
| *** jamesmcarthur has quit IRC | 08:32 | |
| *** tkajinam has quit IRC | 08:34 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Handle draft pull requests in canMerge https://review.opendev.org/704150 | 08:37 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Enforce sql connections for scheduler and web https://review.opendev.org/630472 | 08:37 |
| *** ralonsoh has joined #openstack-infra | 08:38 | |
| openstackgerrit | Merged openstack/project-config master: check-release-approval job improvements https://review.opendev.org/704306 | 08:39 |
| openstackgerrit | Merged openstack/project-config master: Updated x/networking-mlnx project for pypi and neutron https://review.opendev.org/704321 | 08:39 |
| *** kjackal has joined #openstack-infra | 08:40 | |
| *** jpena|off is now known as jpena | 08:52 | |
| *** ccamacho has joined #openstack-infra | 08:57 | |
| *** tetsuro has joined #openstack-infra | 08:58 | |
| *** priteau has joined #openstack-infra | 08:59 | |
| openstackgerrit | Merged zuul/nodepool master: Remove duplicate log info from OpenStack driver https://review.opendev.org/704379 | 09:00 |
| *** hashar has joined #openstack-infra | 09:03 | |
| *** lucasagomes has joined #openstack-infra | 09:11 | |
| *** lucasagomes has quit IRC | 09:11 | |
| *** lucasagomes has joined #openstack-infra | 09:13 | |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: Add client_id to RPC client https://review.opendev.org/704201 | 09:16 |
| openstackgerrit | Miklos Vajna proposed opendev/git-review master: Make it possible to specify who is notified https://review.opendev.org/702861 | 09:25 |
| openstackgerrit | Merged zuul/zuul master: Be more specific with remaining time https://review.opendev.org/704191 | 09:32 |
| ianw | mordred: thanks | 09:42 |
| ianw | mordred: yes, we redirect the _acme-challenge.name.openstack.org CNAME to acme.opendev.org and use the "usual" configuration | 09:43 |
| ianw | that's a point about changing the content of the file and creates: ... i will have to investigate | 09:44 |
| *** yamamoto has quit IRC | 09:44 | |
| ianw | it looks like we still have issues with the openafs roles :/ --> http://paste.openstack.org/show/788863/ | 09:46 |
| *** yamamoto has joined #openstack-infra | 09:46 | |
| ianw | no hang on ... Cache size (50000000) must be less than 95% of partition size (which is 38447165). Lower cache size | 09:47 |
| openstackgerrit | Merged zuul/zuul master: Update screenshot for gerrit profile in quick-start https://review.opendev.org/704222 | 09:48 |
| *** yamamoto has quit IRC | 09:50 | |
| *** gfidente has joined #openstack-infra | 09:53 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: openafs-client: add option for OpenAFS cache location https://review.opendev.org/704523 | 10:05 |
| mordred | ianw: cool - so we could go ahead and LE review-dev if we wanted to at this point | 10:06 |
| * mordred is mostly pondering the "best" way to deal with review-dev and review currently doing ssl differently for apache - don't really want to make the ansible overly clever | 10:08 | |
| *** ociuhandu has joined #openstack-infra | 10:08 | |
| ianw | mordred: yep, just gotta log in to rax and point the openstack.org challenge domains by hand once | 10:08 |
| mordred | but the current review.o.o cert doesn't expire til mar 26 - and I think we don't get a TON of win unless we LE both of them | 10:08 |
| mordred | nod | 10:08 |
| ianw | it should be pretty straight forward to get the LE certs on the host; can be done asynchronously with actually using them of course | 10:09 |
| mordred | ianw: oh - that's a good point | 10:10 |
| *** Lucas_Gray has joined #openstack-infra | 10:10 | |
| *** electrofelix has joined #openstack-infra | 10:10 | |
| ianw | the "dummy" handler is to just touch a file when the certs are updated, i think there's an example for graphite in there | 10:11 |
| mordred | maybe I'll work on doing that for both next - that way I can have the ansible role just point to the LE certs like you did with static | 10:11 |
| openstackgerrit | Merged zuul/zuul master: Fix path in quick-start localtest https://review.opendev.org/704385 | 10:11 |
| ianw | mordred: should i ask why you're awake? | 10:12 |
| mordred | ianw: I'm in Thailand :) | 10:12 |
| mordred | ianw: but - today's my last day on this side of the world, I start the long trip back stateside tonight | 10:12 |
| ianw | heh, where is the world is Mordred Sandiego | 10:13 |
| mordred | come Monday we'll be moved in to our new place in New Orleans and I'll be more predictably located and available | 10:13 |
| mordred | ianw: yah. everyone's favorite game | 10:13 |
| ianw | all those places sound fun :) | 10:14 |
| ianw | well, as long as you're above sea level in New Orleans | 10:14 |
| mordred | ianw: I mean ... we'll do our best ... | 10:15 |
| ianw | i guess you have a lot of scuba gear anyway :) | 10:16 |
| mordred | that's all we need, right? | 10:18 |
| *** ociuhandu has quit IRC | 10:21 | |
| *** Lucas_Gray has quit IRC | 10:32 | |
| *** Lucas_Gray has joined #openstack-infra | 10:40 | |
| *** aedc has joined #openstack-infra | 10:55 | |
| *** ociuhandu has joined #openstack-infra | 10:55 | |
| *** Lucas_Gray has quit IRC | 11:00 | |
| *** Lucas_Gray has joined #openstack-infra | 11:06 | |
| *** dtantsur|afk is now known as dtantsur | 11:07 | |
| *** kjackal has quit IRC | 11:07 | |
| *** aedc has quit IRC | 11:12 | |
| *** rpittau is now known as rpittau|bbl | 11:18 | |
| *** tetsuro has quit IRC | 11:19 | |
| *** rm_work has quit IRC | 11:21 | |
| *** rm_work has joined #openstack-infra | 11:21 | |
| *** hashar has quit IRC | 11:35 | |
| *** yamamoto has joined #openstack-infra | 11:42 | |
| *** yamamoto has quit IRC | 11:43 | |
| *** kjackal has joined #openstack-infra | 11:44 | |
| *** lpetrut has quit IRC | 11:50 | |
| *** Lucas_Gray has quit IRC | 11:50 | |
| *** hashar has joined #openstack-infra | 11:58 | |
| *** Lucas_Gray has joined #openstack-infra | 12:01 | |
| *** Lucas_Gray has quit IRC | 12:06 | |
| *** Lucas_Gray has joined #openstack-infra | 12:09 | |
| *** yamamoto has joined #openstack-infra | 12:18 | |
| *** yamamoto has quit IRC | 12:19 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: Link the policy to the CONTRIBUTING.rst file https://review.opendev.org/704484 | 12:22 |
| *** philroche has quit IRC | 12:22 | |
| *** philroche has joined #openstack-infra | 12:23 | |
| *** jpena is now known as jpena|lunch | 12:23 | |
| *** rcernin has quit IRC | 12:29 | |
| *** ociuhandu has quit IRC | 12:29 | |
| *** ociuhandu has joined #openstack-infra | 12:30 | |
| *** ociuhandu has quit IRC | 12:35 | |
| *** Lucas_Gray has quit IRC | 12:39 | |
| *** Lucas_Gray has joined #openstack-infra | 12:40 | |
| *** mwhahaha has quit IRC | 12:40 | |
| *** mwhahaha has joined #openstack-infra | 12:41 | |
| *** yamamoto has joined #openstack-infra | 12:51 | |
| *** aedc has joined #openstack-infra | 12:51 | |
| slaweq | hi infra team, I have a question about launchpad | 12:54 |
| slaweq | I would like to add new tag to registered tags list for neutron | 12:54 |
| slaweq | do You know how can I do this in LP maybe? Any guide? | 12:54 |
| *** yamamoto has quit IRC | 12:57 | |
| *** rlandy has joined #openstack-infra | 12:59 | |
| *** rh-jelabarre has joined #openstack-infra | 13:01 | |
| *** portdirect has quit IRC | 13:01 | |
| *** portdirect has joined #openstack-infra | 13:02 | |
| *** rpittau|bbl is now known as rpittau | 13:04 | |
| *** psachin has quit IRC | 13:06 | |
| *** ykarel is now known as ykarel|mtg | 13:06 | |
| *** ociuhandu has joined #openstack-infra | 13:10 | |
| *** jamesmcarthur has joined #openstack-infra | 13:11 | |
| *** jamesmcarthur has quit IRC | 13:13 | |
| *** jamesmcarthur has joined #openstack-infra | 13:13 | |
| *** eharney has joined #openstack-infra | 13:17 | |
| *** jpena|lunch is now known as jpena | 13:20 | |
| *** dtantsur is now known as dtantsur|lunch | 13:20 | |
| *** ociuhandu has quit IRC | 13:20 | |
| *** udesale_ has joined #openstack-infra | 13:22 | |
| *** udesale has quit IRC | 13:25 | |
| *** Darcidride has joined #openstack-infra | 13:27 | |
| *** jamesmcarthur has quit IRC | 13:35 | |
| *** psachin has joined #openstack-infra | 13:38 | |
| *** jamesmcarthur has joined #openstack-infra | 13:39 | |
| *** jamesmcarthur has quit IRC | 13:45 | |
| *** jamesmcarthur has joined #openstack-infra | 13:47 | |
| *** Goneri has joined #openstack-infra | 13:48 | |
| *** aaronsheffield has joined #openstack-infra | 13:50 | |
| *** ociuhandu has joined #openstack-infra | 13:53 | |
| *** jaicaa has quit IRC | 13:54 | |
| *** signed8b_ has quit IRC | 13:56 | |
| *** signed8bit has joined #openstack-infra | 13:57 | |
| *** ociuhandu has quit IRC | 14:00 | |
| *** tkajinam has joined #openstack-infra | 14:00 | |
| *** AJaeger has quit IRC | 14:02 | |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: Speed up test_inventory https://review.opendev.org/704570 | 14:03 |
| *** jaicaa has joined #openstack-infra | 14:06 | |
| *** ykarel|mtg is now known as ykarel | 14:09 | |
| *** sdoran has quit IRC | 14:09 | |
| *** sdoran has joined #openstack-infra | 14:10 | |
| *** redrobot_ has joined #openstack-infra | 14:10 | |
| *** redrobot_ is now known as redrobot | 14:11 | |
| *** dtantsur|lunch is now known as dtantsur | 14:12 | |
| openstackgerrit | Fabien Boucher proposed opendev/system-config master: DNM: Test ara-report artifact https://review.opendev.org/697683 | 14:13 |
| *** ykarel is now known as ykarel|away | 14:18 | |
| *** AJaeger has joined #openstack-infra | 14:19 | |
| *** yamamoto has joined #openstack-infra | 14:20 | |
| *** Lucas_Gray has quit IRC | 14:23 | |
| *** Wryhder has joined #openstack-infra | 14:23 | |
| *** Wryhder is now known as Lucas_Gray | 14:24 | |
| *** pcaruana has quit IRC | 14:24 | |
| *** jamesden_ has joined #openstack-infra | 14:28 | |
| *** hamzy__ is now known as hamzy | 14:28 | |
| *** jamesdenton has quit IRC | 14:29 | |
| *** pkopec has joined #openstack-infra | 14:31 | |
| *** tobias-urdin has joined #openstack-infra | 14:36 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Enforce sql connections for scheduler and web https://review.opendev.org/630472 | 14:40 |
| *** smarcet has joined #openstack-infra | 14:43 | |
| *** Lucas_Gray has quit IRC | 14:47 | |
| *** Lucas_Gray has joined #openstack-infra | 14:49 | |
| *** Lucas_Gray has quit IRC | 14:53 | |
| *** Lucas_Gray has joined #openstack-infra | 14:56 | |
| *** Wryhder has joined #openstack-infra | 14:58 | |
| *** jtomasek has quit IRC | 15:00 | |
| *** Lucas_Gray has quit IRC | 15:01 | |
| *** Wryhder has quit IRC | 15:01 | |
| *** tkajinam has quit IRC | 15:02 | |
| clarkb | slaweq: tags? | 15:02 |
| clarkb | infra-root I'd like to make a gerritlib release today. I think I'll dot 0.8.2 on c234e11b513e0a636d487140728e9830ca1753b9 and avoid tagging 945d2088a7dc5c433020542f53b4381765fc7830 as 0.9.0 just yet since I've realized that gerritlibs unittest framework is a nop | 15:03 |
| clarkb | my concern there is that the type change might affect jeepyb. Once I've got status.o.o up I'll look at maybe adopting zuul's gerrit docker-compose stuff to do an integration job of gerritlib, jeepyb and gerrit | 15:04 |
| clarkb | infra-root ^ if that plan sounds good to you I'll push the tag after the board meeting | 15:05 |
| *** Lucas_Gray has joined #openstack-infra | 15:06 | |
| *** jamesmcarthur has quit IRC | 15:07 | |
| openstackgerrit | Sorin Sbarnea proposed opendev/git-review master: Adopt flake8+black https://review.opendev.org/704051 | 15:07 |
| *** jamesden_ is now known as jamesdenton | 15:07 | |
| *** psachin has quit IRC | 15:07 | |
| *** michael-beaver has joined #openstack-infra | 15:08 | |
| slaweq | clarkb: ok, I found it https://bugs.launchpad.net/neutron/+manage-official-tags :) | 15:08 |
| *** pcaruana has joined #openstack-infra | 15:11 | |
| fungi | clarkb: sgtm | 15:12 |
| *** jamesden_ has joined #openstack-infra | 15:15 | |
| corvus | clarkb: what's gerritlib used for? | 15:15 |
| corvus | and did i miss the part where we changed our minds on flake8? | 15:16 |
| corvus | er black | 15:16 |
| *** jamesdenton has quit IRC | 15:18 | |
| *** ociuhandu has joined #openstack-infra | 15:18 | |
| corvus | clarkb: iiuc, gerritlib is used by some processes on status.o.o so is a blocker for updates there? | 15:20 |
| corvus | (there's a lot of scrollback, i'm try to tease out what's relevant) | 15:21 |
| clarkb | corvus: yes, elastic-recheck uses gerritlib and that pulls in pycrypto which fails to build. We should be using newer paramiko which uses cyptography and has wheels instead (and is more secure and up to date) | 15:21 |
| clarkb | corvus: we've updated gerritlib to bump paramiko and rudimentary testing doing listProjects against review-dev works | 15:21 |
| clarkb | corvus: there is a followon change that attempts to make gerritlib work with python3 that I think deserves more testing before adding it to a release | 15:22 |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: Log duration of AnsibleJob.execute() https://review.opendev.org/704589 | 15:22 |
| clarkb | corvus: jeepyb also uses gerritlib so trying to keep the delta small | 15:22 |
| *** hashar has quit IRC | 15:22 | |
| corvus | clarkb: thanks. sgtm. | 15:23 |
| clarkb | re black I don't think there is a plan. zbr seems to just be pusing the change. I've reviewed it as I'm curious to see what that looks like in practice | 15:23 |
| *** jamesden_ is now known as jamesdenton | 15:23 | |
| *** Lucas_Gray has quit IRC | 15:24 | |
| *** Wryhder has joined #openstack-infra | 15:24 | |
| *** ociuhandu has quit IRC | 15:24 | |
| corvus | as you know since we've had this conversation many times, i'm strongly opposed. | 15:24 |
| *** Wryhder is now known as Lucas_Gray | 15:25 | |
| *** ociuhandu has joined #openstack-infra | 15:25 | |
| *** Lucas_Gray has quit IRC | 15:27 | |
| *** Lucas_Gray has joined #openstack-infra | 15:29 | |
| corvus | my understanding is that we have a consensus on code style and linters which we have arrived at after much discussion. if we're going to re-open the discussion, let's do so deliberately. otherwise, we should let people know that the status quo is intentional so we all don't wast time going over ground we've gone over before. | 15:31 |
| clarkb | yes, I think any changes should be agreed upon globally and applied globally so that we don't have to perform gymnastics when writing code for various repos | 15:33 |
| zbr | corvus: clarkb: feel free to -2 the black experiment, i would have raised it as draft/private if possible. | 15:33 |
| corvus | zbr: ok, i've -2d that. i think it's fine to push up changes like that for experiment/show-and-tell. i just want to make sure we're on the same page about that and no one is wasting effort. :) | 15:35 |
| *** ociuhandu has quit IRC | 15:37 | |
| *** jamesmcarthur has joined #openstack-infra | 15:37 | |
| zbr | corvus: would be better if I put something like "POC:" prefix on it? | 15:46 |
| corvus | zbr: yeah | 15:47 |
| *** diablo_rojo has quit IRC | 15:47 | |
| *** Goneri has quit IRC | 15:47 | |
| corvus | helps everyone know what the intention is | 15:47 |
| openstackgerrit | Sorin Sbarnea proposed opendev/git-review master: POC: Adopt flake8+black https://review.opendev.org/704051 | 15:47 |
| *** Lucas_Gray has quit IRC | 15:48 | |
| zbr | at some point we will have to discuss about pre-commit tool, i know that infra was against it, but nova already adopted it, i doubt we could block its use. | 15:50 |
| zbr | better to think more about implications and if there is something to make it easier to use | 15:50 |
| fungi | i don't think we're particularly against projects using any tools they want to use | 15:51 |
| *** jamesmcarthur has quit IRC | 15:52 | |
| fungi | many of the core reviewers for our infra team projects were against using the pre-commit tool in the projects they're reviewing | 15:52 |
| fungi | just because the infra team doesn't prefer a tool, that doesn't mean other projects shouldn't use it if their reviewers/maintainers have an interest in doing so | 15:53 |
| zbr | if I rememeber, the controversial part was always clone implementation, which on zuul would have impose some load issues, right? | 15:53 |
| fungi | we pointed out that it would likely be unstable for that reason | 15:54 |
| fungi | it only clones itself and its plugins though, right? | 15:54 |
| zbr | the number of linter/hooks repos is quite limited, probably less than 8 for entire openstack so far. | 15:54 |
| zbr | which means, we can mirror them. | 15:54 |
| zbr | in fact this was my attempt when I proposed the new project yesterday. | 15:55 |
| fungi | if it can clone from file paths, projects could probably configure zuul to provide those directly on the filesystem when requested | 15:55 |
| zbr | fungi: that needs to work for devs too! no way to hardcode paths there. | 15:56 |
| fungi | yeah, i didn't mean hard-code them | 15:56 |
| fungi | dev's aren't going to use our ci mirrors either | 15:56 |
| *** jamesmcarthur has joined #openstack-infra | 15:57 | |
| fungi | but that's the closest we come to "mirroring" git repositories for projects we're not hosting | 15:57 |
| zbr | pre-commit uses its own cache, quite well managed, but we cannot used it until we unable caching in zuul. | 15:57 |
| fungi | so could the job prepopulate pre-commit's repository cache? | 15:57 |
| zbr | interesting question... | 15:58 |
| clarkb | I think I have a local docker-compose bootstrap of gerrit working based on zuul quickstart and from there have managed to manually install jeepyb and gerritlib from source then run manage projects against the gerrit | 15:59 |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: Log duration of AnsibleJob.execute() https://review.opendev.org/704589 | 16:00 |
| clarkb | now to encode that into a job | 16:00 |
| *** chkumar|rover is now known as raukadah | 16:02 | |
| zbr | fungi: take a look at how cache looks like http://paste.openstack.org/show/788885/ | 16:04 |
| *** armax has joined #openstack-infra | 16:05 | |
| zbr | it has a db, and these folders, they are reused between projects when (repo,version) touple matches | 16:05 |
| fungi | yeah, that's not too bad. looks like sqlite i guess? | 16:06 |
| zbr | yep | 16:06 |
| zbr | each folder contains a git sparse clone and one/more virtualenvs. | 16:07 |
| fungi | and the repo dirs in the cache seem to be randomly-generated suffixes referenced from the cache table | 16:07 |
| zbr | not sure if is random, or hash based, i would use a hash myself. | 16:07 |
| fungi | ahh, so the cache embeds virtualenvs inside it | 16:07 |
| zbr | yep, because once run, it will instantly run the tool | 16:08 |
| *** ociuhandu has joined #openstack-infra | 16:08 | |
| zbr | one of the reasons is so fast | 16:08 |
| zbr | but this does not make the cache portable between different machines | 16:08 |
| corvus | is nova using pre-commit in the gate? | 16:08 |
| *** jackedin has joined #openstack-infra | 16:09 | |
| fungi | i guess the suffix *could* be some sort of base64 encoding (i see upper and lower case, digits and _ which is 63 at least) | 16:09 |
| corvus | because if it isn't, then it isn't really an infra issue; but if it is (or wants to), then that's a big change that the larger community needs to agree to. updating this document is how that would happen: https://governance.openstack.org/tc/reference/pti/python.html | 16:12 |
| *** ociuhandu has quit IRC | 16:12 | |
| *** psachin has joined #openstack-infra | 16:12 | |
| fungi | as far as i can tell they merged a configuration for the tool and some brief documentation on how to run it locally | 16:15 |
| *** zxiiro has joined #openstack-infra | 16:15 | |
| *** aedc has quit IRC | 16:15 | |
| corvus | that's cool | 16:15 |
| corvus | and it's certainly the case that figuring out how to run it stably in the gate is a pre-requisite to poposing a change to the pti | 16:16 |
| fungi | yeah, it seems to have exposed challenges such as the one mordred fixed for them in https://review.opendev.org/674057 | 16:18 |
| *** adriancz has quit IRC | 16:18 | |
| *** adriancz has joined #openstack-infra | 16:18 | |
| fungi | apparently always downloading whatever you want can be at odds with more than just network stability | 16:19 |
| corvus | but i just wanted to clarify that there is a project policy around how to run tests in the gate, and it's not the case that infra plays "catch-up" to what projects are doing -- we work together deliberately on changes like this. | 16:20 |
| *** gyee has joined #openstack-infra | 16:20 | |
| fungi | yeah, in the case of openstack's project testing interface, that was carefully built in collaboration with infra so that we could arrive at something stable and sustainable | 16:21 |
| openstackgerrit | Clark Boylan proposed opendev/gerritlib master: Add integration testing with jeepyb and gerrit https://review.opendev.org/704623 | 16:21 |
| *** udesale_ has quit IRC | 16:22 | |
| clarkb | considering ^ is so close I think I may hold off on a release until I get that stable and am happy with it just as another layer of sanity checking | 16:22 |
| fungi | corvus: granted, nova's already in a risky state of pti compliance doing things like https://opendev.org/openstack/nova/src/branch/master/tox.ini#L46 | 16:24 |
| zbr | on nova alone i see it pointing to two external repos, one is the main hooks repo and another one is of more personal nature, and adds a very tiny check. | 16:24 |
| zbr | i was considering that if we create our own fork of hooks we could avoid that external dependency, even add our own extra checks inside. | 16:24 |
| zbr | i do see this approach much safer than just pointing to outside repos. | 16:25 |
| fungi | seems like they could avoid that flake8 wrapper script for the "pep8" tox testenv and just make a separate testenv for the current purpose it claims to have (checking only files touched in the current patch) | 16:26 |
| clarkb | is there a generic "I want to run ansible from docker container" image? | 16:29 |
| clarkb | I'm using the zuul-executor above since that is what quickstart does (and it needs taht image anyway so makes sense there) but am wondering if there is a better option in the context of my new job which isn't running a zuul | 16:30 |
| corvus | clarkb: i'm not sure; maybe ansible/ansible-runner ? | 16:31 |
| clarkb | thanks. I'll add that to the list of things to test once the base version is running. Also need to run a python3 version too | 16:34 |
| openstackgerrit | Clark Boylan proposed opendev/gerritlib master: Add integration testing with jeepyb and gerrit https://review.opendev.org/704623 | 16:38 |
| *** pgaxatte has quit IRC | 16:45 | |
| *** jpena is now known as jpena|brb | 16:45 | |
| *** rfolco is now known as rfolco|bbl | 16:46 | |
| smarcet | fungi: clarkb: afternoon guys, when u have some time please review https://review.opendev.org/#/c/704482/ and https://review.opendev.org/#/c/704329/ thx u ! :) | 16:51 |
| *** tesseract has quit IRC | 17:01 | |
| *** aedc has joined #openstack-infra | 17:02 | |
| *** iurygregory has quit IRC | 17:04 | |
| *** aedc has quit IRC | 17:08 | |
| *** aedc has joined #openstack-infra | 17:09 | |
| openstackgerrit | Clark Boylan proposed opendev/gerritlib master: Add integration testing with jeepyb and gerrit https://review.opendev.org/704623 | 17:10 |
| clarkb | manage-projects will just retry in a loop if auth fails bceause it can't read its ssh key | 17:11 |
| clarkb | that was an interesting one to debug | 17:11 |
| *** aedc has quit IRC | 17:11 | |
| clarkb | smarcet: in that change you remove the database config? does that mean the db is no longer used? | 17:12 |
| smarcet | clarkb: the DB config removed there is the CMS DB, IDP was refactored to be a AUTHZ/AUTH Microservice so will have its own store of users , all data from member table ( from CMS side) will be move to the IDP new store, and will be managed from there :) | 17:14 |
| *** lucasagomes has quit IRC | 17:15 | |
| *** psachin has quit IRC | 17:15 | |
| fungi | smarcet: on the permissions change, it looks from the commit message like you're saying you want the www-data user to be able to run the artisan command but you're only setting it executable by owner/group and i don't see where ownership of the file is set to www-data | 17:17 |
| smarcet | fungi: yes the ownership is set on line 51 of that file i think | 17:19 |
| fungi | chown -R $FILE_OWNER:$FILE_GROUP $target_dir | 17:19 |
| fungi | aha, so $FILE_OWNER and $FILE_GROUP are being set elsewhere outside the script, thankls | 17:19 |
| smarcet | well its on the same function i think | 17:20 |
| smarcet | site_init and site_update | 17:20 |
| fungi | nothing in that script sets values for FILE_OWNER and $FILE_GROUP so it must be in another script which is calling those functions | 17:21 |
| *** artom has quit IRC | 17:22 | |
| fungi | anyway, thanks for pointing out the chown -R, i missed it was also being done for all of $target_dir | 17:22 |
| *** artom has joined #openstack-infra | 17:22 | |
| *** jpena|brb is now known as jpena | 17:24 | |
| openstackgerrit | Clark Boylan proposed opendev/gerritlib master: Add integration testing with jeepyb and gerrit https://review.opendev.org/704623 | 17:24 |
| *** hashar has joined #openstack-infra | 17:25 | |
| *** artom has quit IRC | 17:27 | |
| openstackgerrit | Clark Boylan proposed opendev/gerritlib master: Add integration testing with jeepyb and gerrit https://review.opendev.org/704623 | 17:27 |
| openstackgerrit | sebastian marcet proposed opendev/puppet-openstackid master: Refactored Deployment Script Functions https://review.opendev.org/704329 | 17:28 |
| *** artom has joined #openstack-infra | 17:29 | |
| *** lmiccini has quit IRC | 17:30 | |
| *** otherwiseguy has joined #openstack-infra | 17:33 | |
| fungi | heads up! there's a beta out for virtualenv 20, so we should probably brace for possible behavior changes when it releases: https://discuss.python.org/t/virtualenv-20-0-0-beta1-is-available/3077/4 | 17:42 |
| *** ociuhandu has joined #openstack-infra | 17:42 | |
| fungi | under python3 it's basically just using the venv module now | 17:42 |
| *** jrosser has quit IRC | 17:42 | |
| fungi | or basically the same as just using the venv module now | 17:43 |
| *** jrosser has joined #openstack-infra | 17:43 | |
| *** igordc has joined #openstack-infra | 17:46 | |
| *** ociuhandu_ has joined #openstack-infra | 17:46 | |
| *** ociuhandu has quit IRC | 17:47 | |
| zbr | corvus: can you look at https://review.opendev.org/#/q/topic:mac-support+(status:open+OR+status:merged) ? | 17:48 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: DNM: experiment with pygit2 https://review.opendev.org/704644 | 17:50 |
| *** ociuhandu_ has quit IRC | 17:51 | |
| *** gfidente has quit IRC | 17:52 | |
| openstackgerrit | Clark Boylan proposed opendev/jeepyb master: Don't use the db in manage-projects https://review.opendev.org/704645 | 17:52 |
| openstackgerrit | Clark Boylan proposed opendev/gerritlib master: Add integration testing with jeepyb and gerrit https://review.opendev.org/704623 | 17:53 |
| clarkb | I've discovered that jeepyb's use of the db makes things difficult | 17:53 |
| *** KeithMnemonic2 has quit IRC | 17:53 | |
| clarkb | what I have realized though is that once we have a basic integration test in place it will make rewriting jeepyb safely much easier | 17:54 |
| clarkb | still work pushing on | 17:54 |
| clarkb | that said I'm fairly confident in the paramiko changes now since ssh isn't what is breaking the testing | 17:55 |
| clarkb | its the layer on top of ssh | 17:55 |
| clarkb | I want to see how this last run goes then I'll likely push a 0.8.2 tag as proposed earlier | 17:55 |
| *** nickv1985 has quit IRC | 17:57 | |
| *** nickv1985 has joined #openstack-infra | 17:58 | |
| *** dustinc has quit IRC | 17:58 | |
| *** dustinc has joined #openstack-infra | 17:58 | |
| *** electrofelix has quit IRC | 17:59 | |
| *** derekh has quit IRC | 18:00 | |
| fungi | #status log a reboot migration of static.openstack.org is scheduled for 2020-01-30 at 20:15 UTC per provider ticket #200126-ord-0000316 | 18:00 |
| openstackstatus | fungi: finished logging | 18:00 |
| *** hashar has quit IRC | 18:01 | |
| fungi | #status log a reboot migration of wiki.openstack.org is scheduled for 2020-01-31 at 02:16 UTC per provider ticket #200127-ord-0000056 | 18:01 |
| openstackstatus | fungi: finished logging | 18:01 |
| *** priteau has quit IRC | 18:01 | |
| *** diablo_rojo has joined #openstack-infra | 18:02 | |
| openstackgerrit | Clark Boylan proposed opendev/gerritlib master: Add integration testing with jeepyb and gerrit https://review.opendev.org/704623 | 18:07 |
| openstackgerrit | Sagi Shnaidman proposed openstack/project-config master: WIP: don't require CLA for contributors of Ansible Openstack modules https://review.opendev.org/704648 | 18:07 |
| clarkb | I think it might actually pass this time \o/ | 18:07 |
| sshnaidm | mordred, mnaser take a look ^ | 18:07 |
| *** dtantsur is now known as dtantsur|afk | 18:07 | |
| *** pkopec has quit IRC | 18:08 | |
| *** Shrews has quit IRC | 18:09 | |
| *** Shrews has joined #openstack-infra | 18:09 | |
| *** armax has quit IRC | 18:10 | |
| *** rkukura has quit IRC | 18:14 | |
| *** smarcet has left #openstack-infra | 18:20 | |
| openstackgerrit | Clark Boylan proposed opendev/gerritlib master: Add integration testing with jeepyb and gerrit https://review.opendev.org/704623 | 18:20 |
| clarkb | it passed but there was a small issue in the jeepyb git commits | 18:21 |
| *** pkopec has joined #openstack-infra | 18:25 | |
| *** jpena is now known as jpena|off | 18:32 | |
| openstackgerrit | Clark Boylan proposed opendev/gerritlib master: Add integration testing with jeepyb and gerrit https://review.opendev.org/704623 | 18:34 |
| clarkb | this will go green! | 18:34 |
| *** rpittau is now known as rpittau|afk | 18:40 | |
| *** jdelaros1 has quit IRC | 18:41 | |
| *** jdelaros1 has joined #openstack-infra | 18:41 | |
| *** armax has joined #openstack-infra | 18:42 | |
| openstackgerrit | Clark Boylan proposed opendev/jeepyb master: Log errors when committing and pushing acls https://review.opendev.org/704660 | 18:47 |
| openstackgerrit | Clark Boylan proposed opendev/gerritlib master: Add integration testing with jeepyb and gerrit https://review.opendev.org/704623 | 18:47 |
| *** yamamoto has quit IRC | 18:48 | |
| *** yamamoto has joined #openstack-infra | 18:51 | |
| *** pkopec has quit IRC | 18:57 | |
| openstackgerrit | Clark Boylan proposed opendev/gerritlib master: Add integration testing with jeepyb and gerrit https://review.opendev.org/704623 | 19:01 |
| *** yamamoto has quit IRC | 19:05 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: DNM: experiment with pygit2 https://review.opendev.org/704644 | 19:06 |
| *** yamamoto has joined #openstack-infra | 19:06 | |
| *** yamamoto has quit IRC | 19:06 | |
| *** yamamoto has joined #openstack-infra | 19:07 | |
| openstackgerrit | Mattias Jernberg proposed opendev/git-review master: Allow the default of notopic to be configurable https://review.opendev.org/697448 | 19:10 |
| *** yamamoto has quit IRC | 19:14 | |
| *** rfolco|bbl has quit IRC | 19:19 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Enforce sql connections for scheduler and web https://review.opendev.org/630472 | 19:19 |
| *** jamesmcarthur has quit IRC | 19:21 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Flesh out the glossary significantly https://review.opendev.org/704391 | 19:24 |
| ianw | fungi: virtualenv-20 looks ... fun | 19:25 |
| fungi | yup | 19:26 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Enforce sql connections for scheduler and web https://review.opendev.org/630472 | 19:30 |
| *** rkukura has joined #openstack-infra | 19:41 | |
| openstackgerrit | Clark Boylan proposed opendev/jeepyb master: Run gerritlib and Gerrit integration test https://review.opendev.org/704671 | 19:42 |
| *** hashar has joined #openstack-infra | 19:42 | |
| osmanlicilegi | greetings. can someone help me with logging into wiki? I'm getting error "an invalid token was found" for a long time. | 19:43 |
| clarkb | osmanlicilegi: at what point in the login process does that happen? | 19:44 |
| openstackgerrit | Daniel Lublin proposed opendev/git-review master: Allow choosing which field to use as author when naming branch https://review.opendev.org/444574 | 19:46 |
| osmanlicilegi | clarkb: after I login over ubuntu one, wiki asks me to choose a nickname although I have one. when I select a nickname and click login, it says: An error occurred: an invalid token was found. | 19:49 |
| *** rchanter has joined #openstack-infra | 19:50 | |
| *** ociuhandu has joined #openstack-infra | 19:50 | |
| clarkb | osmanlicilegi: ok, we are finishing up our weekly team meeting then I expect someone can go take a look at the logs | 19:50 |
| osmanlicilegi | clarkb: ah, ok. thx! | 19:52 |
| fungi | osmanlicilegi: i've seen a handful of new users get that error when trying to create accounts and can't figure out what causes it. it seems to be consistent for that fraction of users, so it could be something to do with their openids served by login.ubuntu.com | 19:54 |
| *** ociuhandu has quit IRC | 19:54 | |
| clarkb | I've pushed gerritlib 0.8.2 tag | 19:57 |
| clarkb | infra-root I also expect the stack ending at https://review.opendev.org/704671 is ready for review if interested in some of the testing related to getting that 0.8.2 tag out | 19:57 |
| clarkb | I'm hoping that makes it easier for us to modify our gerrit management tools as we look towards self service or project managed gerrit | 19:57 |
| paladox | \o/ | 19:58 |
| paladox | clarkb we have a task on self service :P | 19:59 |
| clarkb | paladox: related to that it would help us test a swap out of gerrit python library | 19:59 |
| clarkb | paladox: so that is somethign we can look at as a next step from that test job | 19:59 |
| paladox | oh | 19:59 |
| clarkb | though we likely need to cover more of the cases that use gerritlib first | 19:59 |
| hashar | hi there, is there a way to list all repositories that might trigger a specific job ( tox-linters ) | 20:01 |
| fungi | zbr: redirecting here from the meeting, if we track 404 responses and add redirects, we build a solution which solves popular broken links in a variety of different search engines rather than just fixing them for one search engine (or having to independently manage accounts and relationships with the countless web search engines users may rely on) | 20:02 |
| hashar | and if so a way to trigger the job against those repositories with a proposed config change applied ( https://review.opendev.org/#/c/704414/1/zuul.d/python-jobs.yaml ) | 20:02 |
| clarkb | hashar: you can probably crawl the zuul api to find all uses of that job, but I don't think such a tool exists already. And I don't think there is a way to adhoc trigger it against all projects (you might want #zuul though?) | 20:02 |
| corvus | clarkb, hashar: i think that's accurate. you could probably use codesearch to find a bunch manually -- enough to get a sense of whether it works. | 20:03 |
| zbr | fungi: sounds like a brave challenge, only if we would have endless time for a DIY that would avoid spending 5 minutes on something that already exists. by the time we have it up and running, it will already be too late to fix the original problem. | 20:04 |
| hashar | ah the codesearch | 20:04 |
| osmanlicilegi | fungi: so you mean there's no alternate solution for this issue? | 20:04 |
| corvus | zbr: your tone is not constructive | 20:05 |
| corvus | zbr: you have made the point that we should consider users and avoid breaking urls (with which, i believe, everyone agrees and many have been working diligently on) | 20:05 |
| fungi | osmanlicilegi: none that i've found yet, other than to try a different login.ubuntu.com openid. something causes the error to eventually cease for some of the users who have reported it, but i'm not sure if that's because of something which changed in the openid provider or with their specific openid account | 20:05 |
| corvus | zbr: you have also made the point that we should use google site tools to do that (with which many people disagree and have suggested alternatives). | 20:05 |
| zbr | corvus: i should clarify: i am trying to find a practical solution. | 20:05 |
| hashar | clarkb: corvus: thank you :) | 20:06 |
| zbr | i will not go deepend into that. | 20:06 |
| osmanlicilegi | fungi: got it, thanks. | 20:06 |
| fungi | zbr: your underlying assertion seems to be that open tools are not "practical" and that we should give up on them and use proprietary ones instead. i reject that premise | 20:06 |
| clarkb | osmanlicilegi: does your launchpad openid url have "weird" characters in it? I've seen other openid imlpementation not properly handle them in the past | 20:07 |
| fungi | and now i need to go run some errands | 20:07 |
| clarkb | things like spaces | 20:07 |
| clarkb | osmanlicilegi: if you right clean on your lp user page and view source then grep for openid you should see your openid values | 20:07 |
| fungi | and grab early dinner. i hope to not be more than an hour or so | 20:07 |
| clarkb | (Not sure if there is a better/easier way to get those urls) | 20:07 |
| clarkb | *if you right click on your | 20:08 |
| osmanlicilegi | clarkb: only latin chars | 20:08 |
| fungi | i was able to reproduce the problem myself at one point by creating a new launchpad account/openid and then trying to authenticate to wiki.o.o to autocreate an account. i picked apart all the available logging for it, but nothing jumped out at me indicating what was going wrong with the openid response | 20:10 |
| fungi | not every new launchpad openid seems to cause this either, i seem to have gotten "lucky" with that particular test account | 20:12 |
| osmanlicilegi | maybe someone from canonical should examine it | 20:15 |
| fungi | it's not clear to me whether the error is coming from the mediawiki openid consumer implementation or the ubuntu one openid provider | 20:16 |
| fungi | i just reproduced it with my test account again, so it does seem to still be affected at least | 20:16 |
| openstackgerrit | Andrii Ostapenko proposed zuul/zuul-jobs master: Fix periodic image build jobs https://review.opendev.org/704680 | 20:19 |
| fungi | anyway, i have to pop out, will be back as soon as i can | 20:20 |
| *** rfolco has joined #openstack-infra | 20:24 | |
| *** armax has quit IRC | 20:30 | |
| *** armax has joined #openstack-infra | 20:32 | |
| *** diablo_rojo has quit IRC | 20:34 | |
| openstackgerrit | Merged opendev/system-config master: openafs-client: add option for OpenAFS cache location https://review.opendev.org/704523 | 20:35 |
| *** Darcidride has quit IRC | 20:45 | |
| *** harlowja has joined #openstack-infra | 20:48 | |
| *** jamesmcarthur has joined #openstack-infra | 20:51 | |
| *** ralonsoh has quit IRC | 20:57 | |
| *** Lucas_Gray has joined #openstack-infra | 21:07 | |
| *** jamesmcarthur has quit IRC | 21:17 | |
| *** kjackal has quit IRC | 21:17 | |
| *** jamesmcarthur has joined #openstack-infra | 21:18 | |
| *** slaweq has quit IRC | 21:23 | |
| *** slaweq has joined #openstack-infra | 21:29 | |
| ianw | infra-root / fungi in particular: if you'd like to override governance.openstack.org & security.openstack.org to 23.253.245.150 and make sure it looks the same, and is letsencrypted, that would be great | 21:29 |
| *** slaweq has quit IRC | 21:33 | |
| clarkb | ianw: looking now. Can you do similar for 104.130.141.217 as status.openstack.org no https to check though and it appears not all e-r graphs are generating | 21:42 |
| *** slaweq has joined #openstack-infra | 21:43 | |
| clarkb | ianw: I can confirm both names have LE certs. I am able to browse most of the content but https://governance.openstack.org/election/ is forbidden | 21:43 |
| ianw | clarkb: yeah, i see nothing on the e-r page, but otherwise all content seems to be there for me on status.o.o @ 104... | 21:45 |
| clarkb | I'm manually running the e-r graph generation in the foreground to try and generate errors | 21:46 |
| ianw | Alias "/election/" "${AFS_ROOT}/election/" ... hrm | 21:46 |
| clarkb | ah that gets published from another repo to another location? might need to add that one in before switching governance over? | 21:47 |
| *** slaweq has quit IRC | 21:47 | |
| *** jackedin has quit IRC | 21:48 | |
| ianw | yep for sure, let me dig | 21:48 |
| clarkb | or change it to be a url redirect instead? | 21:48 |
| ianw | no, i think it's not publishing | 21:48 |
| clarkb | ah | 21:48 |
| *** yamamoto has joined #openstack-infra | 21:50 | |
| fungi | ianw: back and testing now | 21:51 |
| fungi | there are something like four repos which publish stuff under governance.o.o | 21:53 |
| ianw | i think the elections is the only we've missed ... sigs tc uc should be there | 21:54 |
| fungi | yep, tested those and they seem to be fine | 21:54 |
| fungi | i've got +2 on openstack/elections if we need to fix any publishing jobs in it | 21:55 |
| *** yamamoto has quit IRC | 21:55 | |
| *** dpawlik has quit IRC | 21:55 | |
| ianw | i always have to teach myself how it works from scratch, i don't have ajaeger's innate knowledge :) | 21:55 |
| clarkb | fungi: the changes to gerritlib to manage connections separately broke the watcher thread :/ | 21:57 |
| clarkb | fungi: http://paste.openstack.org/show/788900/ | 21:57 |
| fungi | that's from elastic-recheck? | 21:58 |
| fungi | i guess we can revert and tag a patch release | 21:58 |
| clarkb | fungi: yes, I don't think its urgent since most of what e-r does is the periodic bug updates. | 21:58 |
| clarkb | I think I see the issue too | 21:58 |
| *** rcernin has joined #openstack-infra | 22:00 | |
| openstackgerrit | Clark Boylan proposed opendev/gerritlib master: Pass Gerrit not GerritConnection to GerritWatcher https://review.opendev.org/704693 | 22:01 |
| *** jamesmcarthur has quit IRC | 22:01 | |
| clarkb | fungi: ^ I think that will fix it, but not in a great spot to debug as I am trying to figure out why e-r isn't working in generating its graphs | 22:01 |
| clarkb | if someone wants to help push that through the newly added test job could be used to set up a stream watcher | 22:01 |
| *** jamesmcarthur has joined #openstack-infra | 22:03 | |
| fungi | ianw: security.o.o from the new server lgtm | 22:03 |
| clarkb | its probably sufficient to set up a local watcher against review.o.o | 22:05 |
| clarkb | and see that it works rather than crashing | 22:05 |
| *** bdodd has quit IRC | 22:05 | |
| fungi | should we try editing the copy on the replacement status.o.o? | 22:06 |
| clarkb | I can try that once I've got e-r graphs sorted out. I'm going to rerun graph generation with the verbose flag | 22:06 |
| fungi | okay, not wanting to interrupt your troubleshooting there | 22:06 |
| ianw | openstack/transparency-policy is one i hadn't considered yet | 22:08 |
| *** bdodd has joined #openstack-infra | 22:09 | |
| clarkb | fungi: ok realized that my first patch is buggy as username and port etc all come from gerrit.connection now | 22:09 |
| fungi | ianw: that hasn't had any changes since long before the move to zuul v3, so getting it to republish might be a challenge. where was it publishing to? | 22:10 |
| fungi | we could just copy the content over into afs | 22:10 |
| ianw | fungi: not sure yet, but i don't think https://www.openstack.org/legal/transparency-policy/ is showing a "live" copy of it from us | 22:11 |
| fungi | ahh, yeah, the osf is probably maintaining that anyway | 22:11 |
| clarkb | authentication via openid is broken on review-dev beacuse we redirect back to review.opendev.org | 22:12 |
| fungi | ianw: so it wasn't published to governance.o.o? what brought it to your attention? | 22:12 |
| ianw | fungi: https://opendev.org/openstack/project-config/src/branch/master/zuul.d/jobs.yaml#L592 ... but it looks like the job is never used | 22:13 |
| fungi | ahh | 22:13 |
| fungi | so cruft | 22:14 |
| openstackgerrit | Clark Boylan proposed opendev/gerritlib master: Pass Gerrit not GerritConnection to GerritWatcher https://review.opendev.org/704693 | 22:14 |
| clarkb | fungi: ^ that seems to work using a simple startWatching then getEvent against review-dev | 22:14 |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: ansible manager: only failed if last ansible failed https://review.opendev.org/704699 | 22:15 |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: zuul-manage-ansible: add validate option https://review.opendev.org/704701 | 22:16 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add openstack/election AFS promotion job https://review.opendev.org/704702 | 22:18 |
| clarkb | '2020-01-28 22:18:29 DEBUG [ergraph ] Starting query for bug 1708704' <- seems like e-r is spending a lot of time on that query | 22:21 |
| openstack | bug 1708704 in OpenStack-Gate "yum client: [Errno 256] No more mirrors to try" [Undecided,New] https://launchpad.net/bugs/1708704 | 22:21 |
| clarkb | I suspect that query makes elasticsearch sad | 22:25 |
| ianw | fungi: that -> https://review.opendev.org/704703 Add promote-governance-election job <- should be the pair to 704702 | 22:25 |
| ianw | i'll make a (wip) change to cleanup while i'm here | 22:25 |
| fungi | thanks | 22:25 |
| clarkb | ok it is OOMing | 22:26 |
| clarkb | and it isn't processing any other bug queries | 22:26 |
| fungi | ouch | 22:26 |
| *** diablo_rojo has joined #openstack-infra | 22:27 | |
| clarkb | I'm trying to rerun it without that query file in place | 22:27 |
| clarkb | it seems to be moving down its query list now | 22:30 |
| clarkb | if it manages to complete without OOMing I think we can remove that query for now until we sort out why it is so memory hungry | 22:31 |
| clarkb | while that is running I'm going to try and get a bike ride in as the sun just came out | 22:32 |
| clarkb | back in a bit | 22:32 |
| *** jamesmcarthur has quit IRC | 22:34 | |
| fungi | it's the e-r python process which was oom'ing yah? | 22:36 |
| *** jamesmcarthur has joined #openstack-infra | 22:36 | |
| fungi | maybe that query just returns too many hits | 22:36 |
| clarkb | yes | 22:41 |
| fungi | applying the query from https://opendev.org/opendev/elastic-recheck/src/branch/master/queries/1708704.yaml in kibana over the last 7 days takes quite a while to return | 22:45 |
| *** jamesmcarthur has quit IRC | 22:45 | |
| fungi | the loglines it returns don't look like they contain that message substring either | 22:46 |
| *** rfolco has quit IRC | 22:46 | |
| fungi | though it's still churning so i don't entirely trust the matched lines being displayed | 22:47 |
| openstackgerrit | Merged openstack/project-config master: Add openstack/election AFS promotion job https://review.opendev.org/704702 | 22:48 |
| *** rh-jelabarre has quit IRC | 22:50 | |
| *** jamesmcarthur has joined #openstack-infra | 22:50 | |
| *** signed8bit has quit IRC | 22:52 | |
| fungi | clarkb: oh! | 22:53 |
| fungi | try it (on a small range) and expand one of the resulting hits | 22:53 |
| fungi | it seems to be all tripleo-ci jobs encoding thousands of lines as a single log entry somehow | 22:53 |
| fungi | looks like maybe embedded ansible output | 22:54 |
| fungi | so something like an entire ansible run to deploy openstack crammed into one logline in elasticsearch | 22:55 |
| *** hashar has quit IRC | 22:56 | |
| fungi | it wouldn't take too many of those to explode the process memory in elastic-recheck | 22:56 |
| *** jamesmcarthur has quit IRC | 23:03 | |
| *** tkajinam has joined #openstack-infra | 23:06 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** slaweq has quit IRC | 23:16 | |
| *** dchen has joined #openstack-infra | 23:24 | |
| *** aaronsheffield has quit IRC | 23:29 | |
| *** rlandy is now known as rlandy|bbl | 23:32 | |
| clarkb | fungi: that would do it, thanks for the investigating | 23:44 |
| fungi | well, i'm unfortunately not coming up with great suggestions for how to fix that console log though | 23:44 |
| clarkb | weshay|ruck: zbr ^ is that something you might know about? | 23:44 |
| clarkb | fungi: I think we either stop indexing those files or fix the indexing of them to not treat the whole file as a single event | 23:44 |
| fungi | well, that's the job's console log i think? | 23:45 |
| clarkb | fwiw my manual removal of that query allowed all those graphs to populate | 23:45 |
| clarkb | hrm console logs should parse properly /me looks more closely | 23:45 |
| fungi | or those entries were tagged "console" anyway | 23:45 |
| fungi | i didn't think to check the file path it returned for them | 23:45 |
| clarkb | fungi: ianw care to rereview 104.130.141.217 as http://status.openstack.org ? if it loosk good now I'll go ahead and update dns then can work to remove the old host tomorrow assuming nothing new pops up | 23:47 |
| fungi | sure thing | 23:47 |
| *** michael-beaver has quit IRC | 23:47 | |
| clarkb | also ianw corvus I'm reasonably happy with https://review.opendev.org/#/c/704693/ now after testing it locally against review-dev.o.o | 23:48 |
| clarkb | we'll need to make a new release of gerritlib (maybe tomorrow) and that should fix e-r's irc bot | 23:48 |
| clarkb | fungi: https://7ae6145b803acbd97c06-6efb80aba20f3eb99d58ca8635d16bb1.ssl.cf2.rackcdn.com/704345/1/check/tripleo-ci-centos-7-containers-multinode-train/07ab6f6/logs/undercloud/var/log/extra/logstash.txt it is that file | 23:50 |
| *** smarcet has joined #openstack-infra | 23:50 | |
| fungi | oh neat | 23:50 |
| clarkb | I think ti may be breaking parsing because timestamps and/or hostname data isn't there in the prefix of the lines | 23:50 |
| fungi | i love that it's specifically named "logstash.txt" | 23:50 |
| fungi | so anyway, yeah, likely needs ingestion rules tweaked | 23:50 |
| clarkb | 2020-01-28 22:16:25.546979 | Job console starting... <- good lines from job-output.txt | 23:50 |
| clarkb | 2020-01-27 16:58:32 INFO | "INFO:tripleo_common.image.image_uploader: xusr: true", <- line from logstash.txt | 23:51 |
| weshay|ruck | clarkb, ah crud.. let me check out a recent change.. sorry | 23:51 |
| clarkb | weshay|ruck: thanks. let me pull up the parsing rules too as that might be a useful reference | 23:51 |
| fungi | ianw: https://governance.openstack.org/election/ is rendering correctly from 23.253.245.150 since that job addition merged | 23:51 |
| clarkb | weshay|ruck: https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf | 23:51 |
| weshay|ruck | thanks | 23:52 |
| clarkb | fungi: ianw agreed Ithink governance.o.o via new webserver lgtm | 23:52 |
| fungi | clarkb: no https for status.o.o? | 23:53 |
| fungi | looks like it's listening on 443/tcp but not doing ssl there | 23:53 |
| clarkb | fungi: I bet that is the default vhost | 23:54 |
| clarkb | I don't think we ever actually configured ssl there | 23:54 |
| fungi | okay, just making sure i'm hitting the right vm | 23:55 |
| fungi | apache access log on that server shows my ip address making requests, so seems i am | 23:56 |
| clarkb | I've added the status01.openstack.org dns records. Will add status.o.o cname once you think you are happy with it | 23:56 |
| fungi | Delay in Elastic Search: Indexing behind by 29 hours | 23:57 |
| fungi | yeesh | 23:57 |
| clarkb | ya that was separate issue | 23:57 |
| clarkb | though I wonder if the giant tripleo docs played a part in that | 23:57 |
| fungi | also doesn't seem to be rendering graphs for me at http://status.openstack.org/elastic-recheck/ | 23:57 |
| clarkb | (we had a corrupt index shard, so I sacrificed it to the ether) | 23:57 |
| clarkb | we were about 72 hours behind yesterday | 23:57 |
| fungi | ouch | 23:58 |
| clarkb | fungi: hrm it renders for me. Anything in the js console? | 23:58 |
| clarkb | iirc it loads json file and renders it with js | 23:58 |
| *** rkukura has quit IRC | 23:58 | |
| fungi | was just looking at page source, and yeah, seems that's all being done with linked scripts | 23:59 |
| fungi | i miss when people used to make web sites with html as something more than a wrapper for javascript | 23:59 |
| clarkb | ok I am able to reproduce after a hard refresh | 23:59 |
| clarkb | I bet I had the old js stuff cached and now the newer js stuff isn't working against our html :/ | 23:59 |
| clarkb | so we fixed the puppetry and installed js that doesn't work with our html I bet | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!