| docaedo | Also the dead-link checker is holding for https://review.openstack.org/#/c/260230/ to merge. jenkins-slaves need requestsexceptions (I failed to check that manifest for the slave...) | 00:22 |
|---|---|---|
| docaedo | In all my tests so far though, it works great, finds the bad, and updates the yaml. | 00:22 |
| docaedo | of course for that test to be of any real use we need to settle the new shed color on https://review.openstack.org/#/c/257663/ ;) | 00:23 |
| docaedo | (and I'm still happy with using attributes field for this and a few other things under the assumption we'll NOT be using this method after glare, which might even be in before next summit!) | 00:24 |
| *** kzaitsev_mb has quit IRC | 01:34 | |
| *** kzaitsev_mb has joined #openstack-app-catalog | 04:55 | |
| *** kzaitsev_mb has quit IRC | 06:34 | |
| *** kzaitsev_mb has joined #openstack-app-catalog | 06:37 | |
| *** kzaitsev_mb has quit IRC | 06:43 | |
| *** kzaitsev_mb has joined #openstack-app-catalog | 06:53 | |
| *** kzaitsev_mb has quit IRC | 07:03 | |
| *** kzaitsev_mb has joined #openstack-app-catalog | 07:04 | |
| *** kzaitsev_mb has quit IRC | 07:09 | |
| *** kzaitsev_mb has joined #openstack-app-catalog | 09:10 | |
| *** kzaitsev_mb has quit IRC | 14:53 | |
| *** kzaitsev_mb has joined #openstack-app-catalog | 17:00 | |
| kfox1111 | morning. looking | 17:17 |
| kfox1111 | +2'ed the kloudbuster image. I'll let you workflow when the image is ready in storage. | 17:18 |
| kfox1111 | What do you think of the code change I suggested in the yaml2json.py? | 17:20 |
| kfox1111 | do you have a review up for the link checker itself yet? | 17:21 |
| *** kzaitsev_mb has quit IRC | 17:23 | |
| openstackgerrit | Merged openstack/app-catalog: Update KloudBuster Image to v5 https://review.openstack.org/259659 | 17:23 |
| docaedo | Dead link checker job is merged as an experimental job: https://review.openstack.org/260115 but it needs this one to merge: https://review.openstack.org/260230 | 17:24 |
| docaedo | I used requestsexceptions but that didn't happen to be on the slaves (and I assumed it was as infra said "oh you just need to use requestexceptions here!") | 17:25 |
| docaedo | but I think that'll merge today or so, then test it with adding a comment "check experimental" to any review | 17:25 |
| docaedo | in theory, should result in proposal-bot suggesting a new yaml, but I'll stay on it | 17:26 |
| kfox1111 | https://review.openstack.org/#/c/260115 is not it. | 17:27 |
| docaedo | re: the code change in yaml2json, I think it still confused me, maybe it was suggested in the wrong place? If the idea was just to shorten the script by two lines, I'm kind of .. eh :) But maybe there's a different/better reason | 17:27 |
| kfox1111 | just easier to read to me. just does the filtering in one spot. | 17:27 |
| docaedo | oops, this one https://review.openstack.org/259232 is the dead link checker but that particular patch was missing a critical line that got added in 260115 | 17:28 |
| kfox1111 | instead of adding 8 lines, it just adds one. | 17:29 |
| docaedo | you sure? Still need to expand the heat environment section I think | 17:30 |
| kfox1111 | wow.... there's a lot of code in check_app_catalog_yaml. | 17:30 |
| kfox1111 | I'm just counting green lines. | 17:31 |
| kfox1111 | if you look at the left hand side, | 17:31 |
| kfox1111 | all you would have to do is add the one line I suggested at the point I suggested to do the same thing I think. | 17:31 |
| kfox1111 | y = [a for in in y] makes a new list with every member of y in it. | 17:32 |
| kfox1111 | y = [a for a in y] I mean. | 17:32 |
| kfox1111 | y = [a for a in y if ...] adds a into the new list then, only if it matches the conditional. | 17:32 |
| kfox1111 | so it does all the same filtering using a new list the other code does. just a lot less code. | 17:33 |
| kfox1111 | wow... | 17:34 |
| kfox1111 | is all that yaml code just to get it not to munge the yaml file? | 17:34 |
| docaedo | yes, all that is needed to keep the yaml in the same order | 17:35 |
| kfox1111 | :/ | 17:35 |
| kfox1111 | woudl have been way easier with just dumping a second yaml and merging it. | 17:35 |
| kfox1111 | would have been 2 lines in the yaml2json script instead of one. | 17:35 |
| kfox1111 | deactivated = yaml.load(open('linkchecker.yaml')) | 17:36 |
| kfox1111 | y = [a for a if a not in deactivated] | 17:36 |
| kfox1111 | hmm... well... | 17:37 |
| kfox1111 | y = [a for a if a['name'] not in deactivated] | 17:37 |
| docaedo | maybe, but that won't help me when I start checking hashes | 17:37 |
| kfox1111 | why not? | 17:38 |
| docaedo | actually, maybe it wouldn't hurt .. I guess I'm still opposed to multiple yamls for this, rather than keeping everything in one | 17:39 |
| kfox1111 | just makes for a lot more code, trying to ensure the code doesn't munge the human written yaml. | 17:40 |
| kfox1111 | having 2 lets you have the machine generated one where no one cares about ordering stuff, and one for humans that do. | 17:40 |
| docaedo | easier for humans to look in one yaml and see their asset has been marked inactive though | 17:41 |
| kfox1111 | true. | 17:41 |
| * kfox1111 shurgs. | 17:41 | |
| kfox1111 | I figured thats what the email thing woudl be fore. | 17:41 |
| kfox1111 | for | 17:41 |
| kfox1111 | getting a notification by email's a lot easier for a human then watching a gerrit review board. | 17:42 |
| kfox1111 | meeting next. bbiab. | 17:42 |
| docaedo | we're nowhere near being able to mail people when their asset has a dead link, and all in that would be way more work than something that works right now | 17:43 |
| docaedo | like literally, all of what has been written, works, and achieves the goal of automatically updating the catalog to remove dead links | 17:44 |
| kfox1111 | true. | 17:56 |
| kfox1111 | its great step in the right direction to get us the ability to remove dead liks. | 17:57 |
| kfox1111 | links | 17:57 |
| kfox1111 | but thats about all we can do with it. it will be hard most of the time for us to update links. :/ | 17:57 |
| *** openstackgerrit has quit IRC | 18:02 | |
| *** openstackgerrit has joined #openstack-app-catalog | 18:03 | |
| docaedo | when you say "hard to update links", are you talking about keeping hashes up to date for things like fedora beta, debian, etc? | 18:40 |
| docaedo | also keep in mind, I expect all of this to need to be re-though/re-worked when glare is implemented, but I'm very much looking forward to ditching the yaml entirely :) | 18:45 |
| kfox1111 | keeping a hash up to date can be automated with the link checker. if it finds a hash that doesn't match, it can post the update as part of the gerrit review. odds are if the hash matches more then once a new file, thats the correct hash. I'm worried more about the bad link case, where they move the images around. :/ | 19:17 |
| kfox1111 | then it might not be clear the right thing for us to do. it would take the origional submitter or someone else knowlegable on that image to post a new update, which would mean we would have to contact them somehow to let them know it broke. | 19:18 |
| docaedo | I don't follow though, when you say "move the image around"? who's doing that and expecting the catalog to keep it up to date for them | 19:19 |
| kfox1111 | thats the problem. if the person who maintains the app catalog entries for an image is a different person then maintains the site hosting the image, things can get "cleaned up" / moved around on the site. left hand vs right hand kind of thing. | 19:20 |
| kfox1111 | leaves stale entries in the app catalog. | 19:21 |
| docaedo | but if the link dies because someone moved it, then the link is dead, and inactivated, right? | 19:21 |
| kfox1111 | right. | 19:22 |
| kfox1111 | but we don't nessisarilyh have the knowlege to fix it and reenable it. while as the maintainer does. hense the need to email them. :/ | 19:23 |
| docaedo | yes - I agree that's an issue, but I don't see that being an issue in the next 2-3 months do you? | 19:23 |
| kfox1111 | no idea. will see I guess once the link checker's in place. | 19:23 |
| docaedo | As great as it would be, I don't envision a sudden massive influx of content into the catalog. I like what we're doing, but it still sits in a very odd place, where we want to deliver what you would get from a PaaS application catalog, but without a PaaS | 19:25 |
| kfox1111 | right. :/ | 19:26 |
| kfox1111 | I think the biggest problem though is lack of pluing support in any of the distro's (except debian) yet. | 19:26 |
| kfox1111 | though I don't think anyone uses debian except for testing at home. :/ | 19:26 |
| kfox1111 | if a user cant easily find any of it, then its unlikely for a developer to bother to post anything to the catalog. :/ | 19:27 |
| kfox1111 | we've gotten the plugin into rdo-delorean, but I don't think its in the liberty repo like I was hoping. | 19:27 |
| kfox1111 | haven't had time to go figure out how to make that happen. :/ | 19:28 |
| kfox1111 | we also need to make sure it gets into packstack, rdomanager, into ubuntu, and to fuel. | 19:28 |
| openstackgerrit | Christopher Aedo proposed openstack/app-catalog: Skip assets with active==false attribute https://review.openstack.org/257663 | 19:28 |
| kfox1111 | then we'll have some visibility. :/ | 19:28 |
| docaedo | ^^ is the script with shorter lines, finally grok'd what you were getting at, and agree :) | 19:29 |
| docaedo | er, not shorter lines - fewer lines. you know what I mean | 19:29 |
| docaedo | agree that the horizon plugin would change things for sure, and have kept "improving adoption" in the back of my mind... | 19:30 |
| kfox1111 | right. | 19:30 |
| kfox1111 | hmm... | 19:30 |
| kfox1111 | you pulled too much code... got rid of the env stuff... | 19:31 |
| kfox1111 | let me take a quick stab at it... | 19:31 |
| docaedo | Should also see some good movement on general catalog adoption when IBM includes it in bluemix. Don't know the scale of the audience, but it could be significant, and would be the biggest pickup of the catalog yet. Hoping still for first quarter next year | 19:31 |
| docaedo | I think the env stuff still worked though, at least when I looked at the output, it was there and formatted as expected | 19:31 |
| kfox1111 | yeah, that would be nice. | 19:31 |
| docaedo | unless the heat environment formatting needs something even more special? | 19:31 |
| kfox1111 | no, I think it will break. it won't be formatted as a json string. | 19:32 |
| docaedo | service": {"environment": {"parameters": {"db_port": 1000}}, "type": "heat", "format": "HOT"} | 19:33 |
| kfox1111 | yeah. should be {"environment": "{\"parameters\": {\"db_port\": 1000}}", "type": "heat", "format": "HOT"} | 19:34 |
| docaedo | ah got it .. well that will help for all those people deploying the hello world template :) | 19:37 |
| kfox1111 | :) | 19:37 |
| kfox1111 | it was easier to do it that way. the javascript in the plugin just needs a string, and can pass it along without converting json to a string. | 19:37 |
| kfox1111 | and the user can just put yaml into the yaml file as expected and it will convert properly. | 19:38 |
| docaedo | sure but you know - that comes at the expense of like .. THREE EXTRA LINES in that yaml script! | 19:38 |
| docaedo | :P | 19:38 |
| kfox1111 | :) | 19:39 |
| kfox1111 | yeah, but way more then 3 lines to fix on the js side. ;) | 19:39 |
| kfox1111 | needed like a whole nother external js file. :/ | 19:39 |
| docaedo | I'm going to be AFK for a while, but will check back in on this later, I'm still holding out hope that dead link checker is implemented before 2016 :D | 19:39 |
| kfox1111 | k. | 19:40 |
| kfox1111 | still think skipping attributes is a good idea too. | 19:40 |
| kfox1111 | why bother to implement a schema if we ignore it? | 19:40 |
| openstackgerrit | Kevin Fox proposed openstack/app-catalog: Skip assets with active==false attribute https://review.openstack.org/257663 | 19:54 |
| kfox1111 | it checks for things like active: "false" and rejects them. | 19:55 |
| kfox1111 | hmm.... you know, the other interesting way to do this would be with a tag... | 19:57 |
| kfox1111 | we could then allow users to still search on and see inactive things, but default them to hidden. | 19:57 |
| kfox1111 | if we bump asset version to 2 when adding the tags: ['inactive'], existing horizon plugins will just not show the asset at all. giving the correct behavior. and then in an updated horizon plugin, we could provide a checkbox to show inactive assets that would know asset version v2. | 20:01 |
| kfox1111 | hmm... actually, the code right now looks like it still shows the asset, but marks it as disabled, so you can still find it. | 20:36 |
| kfox1111 | damn... found a bug in the horizon plugin.. | 20:54 |
| kfox1111 | I could have sworn I tested schema versions... | 20:55 |
| kfox1111 | I think dep handling broke it. :/ | 21:01 |
| docaedo | Score one for free form attributes field :D haha | 21:13 |
| kfox1111 | no, free form wouldn't have the desired affect either. | 21:53 |
| kfox1111 | basically I was thinking of this: http://pastebin.com/3JXRg4KM | 21:55 |
| kfox1111 | if it wasn't for the horizon plugin bug, that would have worked too. | 21:56 |
| docaedo | OR we just exclude inactive assets from the JSON and call it a year | 21:57 |
| kfox1111 | we only have one horizon plugin version that's affected by it. since we have the python api in place now, we could fix it like in the patch, and filter out all version>2 stuff in the api if horizon plugin version 1.0.0 is found. | 21:57 |
| kfox1111 | I'm partial to tagging it so that others that want to go look for broken things can. | 21:58 |
| kfox1111 | they can easily setup cron jobs for example to check if their assets are marked inactive in the server. | 21:58 |
| kfox1111 | we can then put reasons things are disabled too in the asset. like, checksum mismatch, or broken link. | 21:59 |
| docaedo | they can do the same with a cron job hitting the repo, or by subscribing to app-catalog updates in which case they'll see the email | 21:59 |
| docaedo | I'm not proposing the checksum updater download every asset and do the hash | 22:00 |
| kfox1111 | I really don't think a git repo should be an api endpoint. | 22:00 |
| kfox1111 | email might work, but an api endpoint would work better with nagios for example. | 22:00 |
| kfox1111 | then you can mail the oncall person for example. | 22:00 |
| kfox1111 | ah. so its just checking for 404's? any plan to do checksum checking too? | 22:01 |
| docaedo | how many tb of bandwidth per day should the catalog be allocated? :P | 22:01 |
| docaedo | for checking, it's checking basically for a 200 response, or a 301/302 (in which case it will follow that looking for 200( | 22:02 |
| kfox1111 | good question. | 22:02 |
| kfox1111 | depends I guess on the cloud providers... | 22:03 |
| docaedo | also downloading and comparing the hash against expected is a waste of time - it will only reflect that one download that the catalog did | 22:03 |
| kfox1111 | with amazon, bandwidth in is free, so download checking would be free. | 22:03 |
| docaedo | would have no reflection on what the end user gets - it's on the user to download and compare the hash | 22:03 |
| kfox1111 | yeah, but it woudl detect cases where they updated image a with image b at the same location, but didn't update the catalog with the new sum. | 22:04 |
| docaedo | (which is something glance should probably do - take a remote URL and a hash, and compare the hash before importing) | 22:04 |
| kfox1111 | the user thinkgs its a bad link, and its true in a wy. they don't know if its new or tampered with. | 22:04 |
| kfox1111 | agreed. glance artifacts shoudl have checksumming built in I think. | 22:04 |
| docaedo | thats walking too close to guaranteeing what's in the catalog (vs. making a best effort to keep an index of things up to date) | 22:04 |
| * kfox1111 shrugs. | 22:05 | |
| kfox1111 | daily checking links, weekly checking checksums... seems best effortish to me. | 22:05 |
| kfox1111 | its validating what we believe the metadata in our repo is reasonably current. | 22:06 |
| docaedo | it still solves little because the user HAS to check when they get the asset, in this case it just makes the catalog be a middleman that makes a claim about a hash - the user still has to compare the hash | 22:06 |
| docaedo | I can see that as a mechanism for confirming the metadata is accurate | 22:06 |
| docaedo | but then it becomes very costly, unless the app catalog is hosting the binary assets (in which case they would have been hash-verified upon entry anyway) | 22:07 |
| docaedo | would be awesome if there was an http header for hash :) | 22:07 |
| kfox1111 | right. if we're paying for bandwidth, its too costly probably. if its free, I'm less sure. | 22:08 |
| kfox1111 | yeah. that would be awesome. :) | 22:08 |
| docaedo | someone is paying for the bandwidth, and when it becomes excessive that someone stops giving hosting away for free | 22:09 |
| kfox1111 | yeah, but take amazon for example. I figured out the reason I think they give away free bandwidth. | 22:10 |
| kfox1111 | netflix. | 22:10 |
| docaedo | what free bandwidth does amazon give away? | 22:10 |
| docaedo | you mean like the first gb free stuff? | 22:10 |
| kfox1111 | netflix essentially pays for the majority of the bandwidth out of amazon. they have to run wires to cover the cloud->internet direction. | 22:10 |
| kfox1111 | that means they have a ton of bandwidth going from the internet->cloud that's unused. | 22:11 |
| kfox1111 | so they give it away. it doesn't cost them much. | 22:11 |
| kfox1111 | if you upload stuff into aws, they don't charge you per gig. | 22:11 |
| kfox1111 | downoad, they do. | 22:11 |
| kfox1111 | download | 22:11 |
| kfox1111 | it all depends on how much unused capacity they have, in which directions. | 22:12 |
| kfox1111 | netflix is almost 37% of all internet traffic. | 22:13 |
| docaedo | ah sure but they always make it free or cheap to bring in, because your stuff is captive then, the easier and cheaper it is to move in the better (as long as you have to pay through the nose to move out) | 22:13 |
| kfox1111 | that means the corisponding upload to the aws cloud pipe must be huge and relatively idle. :) | 22:13 |
| kfox1111 | right. but with talking to them, they don't seem to care, if you pull data in, process it down to something small, and output that. | 22:14 |
| kfox1111 | or just chew on it and never spit anything back out. | 22:14 |
| kfox1111 | they have so much, they really just don't seem to care when we've talked to them. | 22:14 |
| docaedo | true | 22:14 |
| kfox1111 | so putting a checkummer in aws for example, woudl be just the cost of the vm doing the summing, not the bandwidth. | 22:15 |
| kfox1111 | I kind of wonder which other cloud providers run into that situation. | 22:15 |
| kfox1111 | most cloud hosts are hosting websites, which is bandwidth cloud->internet, so a very similar issue. | 22:15 |
| kfox1111 | internet->cloud's much less common, so may be cheeper/free. | 22:16 |
| docaedo | I'm going to be mostly away for a while now :) Will check back in on occasion - hope you have great holidays! | 22:19 |
| kfox1111 | yup. have a great one! :) | 22:21 |
| openstackgerrit | Christopher Aedo proposed openstack/app-catalog: Add dead-link test script to tools https://review.openstack.org/260198 | 23:15 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!