| *** jesusaur has joined #zuul | 00:21 | |
| SpamapS | tristanC: dmsimard any advice on an IAM policy to attach to nodepool's creds? | 00:27 |
|---|---|---|
| SpamapS | botocore.exceptions.ClientError: An error occurred (UnauthorizedOperation) when calling the DescribeImages operation: You are not authorized to perform this operation. | 00:30 |
| SpamapS | despite having ec2:* allowed | 00:30 |
| tristanC | SpamapS: I don't know, maybe look for the aws glance service name and add it? | 01:28 |
| *** bhavikdbavishi has joined #zuul | 02:55 | |
| SpamapS | tristanC: aws's glance is ec2:* | 03:55 |
| SpamapS | tristanC: I wonder how you get your boto creds in. I'm mounting mine in as ~/.aws/credentials | 03:55 |
| SpamapS | tristanC: n/m.. somehow my pods are picking up the ARN for the worker node role, and not the one I made for nodepool | 04:10 |
| tristanC | SpamapS: iirc i run sudo -u nodepool aws configure | 04:11 |
| tristanC | what is ARN? | 04:12 |
| SpamapS | amazon resource n-something | 04:23 |
| SpamapS | ?? | 04:23 |
| SpamapS | dunno | 04:23 |
| * SpamapS id detaching for a while | 04:23 | |
| SpamapS | I think I got it figured out | 04:23 |
| *** bhavikdbavishi has quit IRC | 06:16 | |
| *** pcaruana has joined #zuul | 06:49 | |
| *** dmellado has quit IRC | 07:00 | |
| *** dmellado has joined #zuul | 07:02 | |
| *** eumel8 has joined #zuul | 07:16 | |
| *** hashar has joined #zuul | 07:20 | |
| *** jpena|off is now known as jpena | 07:45 | |
| *** electrofelix has joined #zuul | 08:22 | |
| *** lennyb has joined #zuul | 09:46 | |
| *** hashar has quit IRC | 10:15 | |
| *** hashar has joined #zuul | 10:22 | |
| *** pcaruana has quit IRC | 10:32 | |
| *** pcaruana has joined #zuul | 10:32 | |
| *** threestrands has quit IRC | 11:05 | |
| *** hashar has quit IRC | 11:11 | |
| *** swest has joined #zuul | 11:16 | |
| *** jpena is now known as jpena|lunch | 11:22 | |
| *** hashar has joined #zuul | 11:24 | |
| *** snapiri has joined #zuul | 11:39 | |
| *** jpena|lunch is now known as jpena | 12:24 | |
| *** rfolco has joined #zuul | 12:29 | |
| *** rlandy has joined #zuul | 12:30 | |
| *** samccann has joined #zuul | 12:36 | |
| *** _ari_|pto has quit IRC | 12:48 | |
| *** myoung|training is now known as myou7ng | 12:57 | |
| *** myou7ng is now known as myoung | 12:57 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: rewrite interface in react https://review.openstack.org/591604 | 13:22 |
| tristanC | mordred: corvus: 591604 should now be feature complete with the angular/jquery code, and i've added a couple of jest tests for good measure. | 13:25 |
| Shrews | SpamapS: speaking of containers, you may find https://review.openstack.org/593125 interesting | 13:29 |
| pabelanger | would love to see a +3 this week to start testing them ^ | 13:59 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: rewrite interface in react https://review.openstack.org/591604 | 13:59 |
| dmsimard | SpamapS: for the time being, I'm running with ~/.aws/credentials (from the perspective of nodepool) however the name of the credentials must be "default" right now | 14:16 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: rewrite interface in react https://review.openstack.org/591604 | 14:16 |
| dmsimard | hoping to be able to change that so it's possible to use multiple credentials (not unlike clouds.yaml) | 14:17 |
| Shrews | pabelanger: i cannot think of a better way to test the infinite launch, other than just waiting for manager.createServer_fails to reach < 1 | 14:17 |
| Shrews | pabelanger: i tried (unsuccessfully) a few things this morning, but that's the best i came up with | 14:19 |
| corvus | tristanC: cool! mordred is mostly afk until ~ the PTG; would you mind if we wait until he's back to merge it? | 14:20 |
| pabelanger | Shrews: okay, I'll refresh the patch this morning and see if others can review. Thanks for helping | 14:20 |
| Shrews | corvus: the iptables fail logs are really weird | 14:20 |
| corvus | Shrews: neat! let's go over to infra and look at those | 14:21 |
| tristanC | corvus: sure, well if you are ok with this change, then i'll rebase the other angular page on top of it | 14:27 |
| corvus | tristanC: yeah, i like the change, and i think mordred does too, and i haven't heard anyone argue to keep angular, so i think we're going to do it. it's just a matter of having all hands on deck when it goes in. :) | 14:29 |
| tristanC | corvus: that's great to hear :-) | 14:30 |
| *** ianychoi has quit IRC | 14:36 | |
| *** ianychoi has joined #zuul | 14:42 | |
| *** hashar is now known as hasharAway | 14:49 | |
| *** pcaruana has quit IRC | 15:29 | |
| SpamapS | dmsimard: yeah for some reason nodepool isn't finding them in ~/.aws so I'm setting them as env vars instead | 15:32 |
| SpamapS | which actually kinda makes more sense anyway | 15:32 |
| dmsimard | tristanC: I was looking at the build ( http://logs.openstack.org/04/591604/22/check/zuul-build-dashboard/6e880c6/npm/html/status ) and it looks really cool. Filtering is instantaneous and the page seems faster to load (in firefox). I can't comment on the code but the result is nice, good job :D | 15:33 |
| dmsimard | SpamapS: in my world that means /var/lib/nodepool/.aws (since that's the homedir of nodepool) | 15:33 |
| Shrews | corvus: can you reapply your +W on https://review.openstack.org/593125 ? looks like a random test failure | 15:38 |
| corvus | Shrews: i just left a recheck comment (it has to go through check again anyway) | 15:39 |
| Shrews | ah | 15:39 |
| dmsimard | SpamapS: how are you setting those env vars ? | 15:39 |
| tristanC | dmsimard: thanks :) React is designed to only render element that needs update, thus status loading and filtering is indeed much faster | 15:46 |
| tristanC | it could be even faster if the create_tree method was rewritten to be per pipelines, it's still a copy of the legacy code that walk through all change_queues to figure out box indentation. | 15:47 |
| SpamapS | dmsimard: in the deployment container spec | 15:53 |
| SpamapS | dmsimard: in mine I just run nodepool as "root" since it's not real root inside the container. :-P | 15:53 |
| dmsimard | SpamapS: sure, I guess that reading env variables is provided by boto and not nodepool itself though.. I think the credentials file approach mirrors well what we have for openstack's clouds.yaml though so that would probably be what is the cleanest approach (from a driver standpoint) | 15:57 |
| SpamapS | dmsimard: Right, I want the credentials approach to work.. just not sure what's making it fail | 15:58 |
| dmsimard | SpamapS: are they under a [default] block ? | 15:58 |
| dmsimard | SpamapS: like http://paste.openstack.org/raw/728840/ | 15:59 |
| SpamapS | dmsimard: it's failing the isfile check | 16:04 |
| SpamapS | because secrets mounted in are symlinks I guess | 16:04 |
| dmsimard | SpamapS: have a stack trace ? I can look into it | 16:04 |
| *** jpena is now known as jpena|off | 16:09 | |
| SpamapS | bah no, it's dumber than that | 16:09 |
| SpamapS | Helps if you name the fields right | 16:09 |
| SpamapS | dmsimard: this is my new stack trace | 16:13 |
| SpamapS | http://paste.openstack.org/show/728842/ | 16:13 |
| SpamapS | looks like maybe that's just my AMI | 16:15 |
| dmsimard | So that looks like an error in the code I wrote on that last patchset | 16:16 |
| SpamapS | yeah some ami's have 0 block device mappings | 16:17 |
| SpamapS | they're instance-store-only | 16:17 |
| dmsimard | SpamapS: I guess line 136 at https://review.openstack.org/#/c/535558/10/nodepool/driver/aws/provider.py@136 was a bit naive | 16:17 |
| SpamapS | I'm trying an EBS AMI now | 16:18 |
| dmsimard | SpamapS: yeah, I was basically trying to address that some AMIs ship with an EBS by default and we definitely need to make sure they are set to delete on termination | 16:18 |
| dmsimard | Let me try and check what an AMI without EBS looks like | 16:18 |
| SpamapS | lol, and the EBS ones fail | 16:19 |
| SpamapS | Exception: The AMI %s has more than one block device | 16:19 |
| SpamapS | so ubuntu's images either have 0 b_d_m, or > 1. | 16:20 |
| dmsimard | SpamapS: are those AMIs on the marketplace ? | 16:20 |
| SpamapS | dmsimard: I have a quick patch for this, which is just to wrap it in an if block_device_mapping | 16:20 |
| SpamapS | dmsimard: https://cloud-images.ubuntu.com/locator/ | 16:21 |
| SpamapS | that's how I find 'em | 16:21 |
| dmsimard | SpamapS: ok, let me just have a look at what they look like | 16:21 |
| SpamapS | dmsimard: I'm testing a patch that just wraps the mapping check in a boolean check of block_device_mapping | 16:22 |
| SpamapS | in case you want to run with that on your next patchset | 16:22 |
| * SpamapS has to task switch to real work now ;) | 16:25 | |
| SpamapS | I'm hoping that over the next couple weeks I will evolve my k8s deployment into helm charts | 16:25 |
| SpamapS | and maybe we'll at least have a "got a k8s cluster? Want some zuul in it? -> chart" | 16:25 |
| dmsimard | SpamapS: looks like this from boto's perspective http://paste.openstack.org/raw/728844/ | 16:26 |
| *** pcaruana has joined #zuul | 16:26 | |
| dmsimard | SpamapS: happy to try your patch out and submit a new patchset | 16:27 |
| SpamapS | dmsimard: right, try an hvm-instance | 16:27 |
| SpamapS | dmsimard: that will have 0 mappings | 16:27 |
| dmsimard | SpamapS: there's two AMIs in that paste, the first is hvm-instance and yeah it has nothing | 16:27 |
| SpamapS | oh right | 16:27 |
| SpamapS | yeah so that's what I'd want to use for test nodes | 16:28 |
| SpamapS | No reason to spend $ on gp2 EBS | 16:28 |
| dmsimard | sure | 16:28 |
| dmsimard | SpamapS: if your patch was just to check if block_device_mappings was empty, I'll take care of it | 16:28 |
| SpamapS | and for those with Ebs, probably quite common to have > 1 | 16:28 |
| SpamapS | dmsimard: yeah that's the patch I'm going to test now | 16:28 |
| SpamapS | though I'm switching to other stuff for a few hours | 16:28 |
| dmsimard | SpamapS: okay, the bdm > 1 part I'm not entirely sure how to handle it.. like that image has one ebs but three devices ? or is that actually three volumes ? the first has delete on termination but the other two do not | 16:29 |
| * dmsimard prefers OpenStack | 16:30 | |
| *** electrofelix has quit IRC | 16:38 | |
| *** jlviva-viva is now known as jlvillal | 16:40 | |
| *** openstackgerrit has quit IRC | 17:04 | |
| *** openstackgerrit has joined #zuul | 17:06 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: add-fileserver: fileserver_leading_path isn't from the secret https://review.openstack.org/587768 | 17:06 |
| openstackgerrit | Merged openstack-infra/zuul master: Add container push job https://review.openstack.org/593125 | 17:18 |
| Shrews | ooh, neat. somebody merge something into zuul | 17:44 |
| Shrews | :) | 17:44 |
| *** jimi|ansible has quit IRC | 18:28 | |
| SpamapS | dmsimard: the non EBS ones are ephemeral storage | 18:59 |
| dmsimard | SpamapS: yeah I understood that part | 19:06 |
| *** pcaruana has quit IRC | 19:13 | |
| SpamapS | Shrews: does that mean we'll get a dockerhub that we can FROM zuul:latest on? | 19:16 |
| Shrews | SpamapS: yes | 19:17 |
| pabelanger | I was thinking about nodepool, how did we solve the diskimage-builder dependencies for nodepool-builder? Will pbrx be smart enough to pull those in from bindep in diskimage-builder? | 19:18 |
| pabelanger | or do you first build diskimage-builder container, then FROM it when doing nodepool-builder images | 19:20 |
| Shrews | pabelanger: yes, it uses bindep | 19:22 |
| SpamapS | Shrews: neat | 19:23 |
| pabelanger | Shrews: cool, so we don't need to add DIB dependencies directly into nodepool bindep.txt? | 19:25 |
| pabelanger | eg: kpartx | 19:25 |
| SpamapS | something we might want to put in zuul's bindep: ssh client | 19:26 |
| SpamapS | FileNotFoundError: [Errno 2] No such file or directory: 'ssh-agent': 'ssh-agent' | 19:26 |
| SpamapS | that's my current speed bump to running jobs in my AWS+K8S+Zuul monster | 19:26 |
| Shrews | pabelanger: shouldn't. it builds a "base" container first that contains all dependencies from bindep installed. then creates the other containers based on that one | 19:27 |
| pabelanger | okay cool | 19:28 |
| Shrews | pabelanger: i'm making the assumption that these things actually work. i haven't tested them, personally :) | 19:29 |
| pabelanger | yah, we'll likely learn soon enough :) | 19:31 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul master: Need SSH client for Ansible https://review.openstack.org/596875 | 19:37 |
| SpamapS | ^^ | 19:38 |
| SpamapS | need that so the docker images get ssh client | 19:38 |
| dmsimard | corvus: I'm able to find where the counters for SUCCESS/FAILURE are incremented for the different phases by the executor with "self.executor_server.statsd" but I'm unable to find where we might be reporting durations or timers.. is it done elsewhere ? The only duration/timer calculations I've seen were inferred from the database (i.e, for display in the web ui) | 20:09 |
| corvus | dmsimard: https://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/scheduler.py#n452 | 20:10 |
| dmsimard | ah, ty | 20:11 |
| *** samccann has quit IRC | 20:18 | |
| *** samccann has joined #zuul | 20:19 | |
| *** rlandy is now known as rlandy|brb | 20:31 | |
| dmsimard | corvus: I'm not able to figure it out... :( I took for example the Zuul tox-py36 job and on graphite.o.o, neither "stats.zuul.tenant.openstack.pipeline.check.project.git_openstack_org.openstack-infra_zuul.master.job.tox-py36.SUCCESS" or "stats_counts.zuul.tenant.openstack.pipeline.check.project.git_openstack_org.openstack-infra_zuul.master.job.tox-py36.SUCCESS" seem to hold any kind of duration data | 20:32 |
| dmsimard | I did confirm that the timer is reporter properly and on the right key in my environment, the value was reported was "103275" which seems appropriate for a job that ran for 1m43s (103275/1000 = 103 seconds) | 20:35 |
| corvus | dmsimard: the equivalent timer key for that is stats.timers.zuul.tenant.openstack.pipeline.check.project.git_openstack_org.openstack-infra_zuul.master.job.tox-py35.SUCCESS.mean | 20:41 |
| corvus | dmsimard: well, the last part ("mean") is just one item of data recorded by timers, there's a slew of them; you'll see them if you expand under "SUCCESS" | 20:41 |
| dmsimard | OH | 20:42 |
| dmsimard | it's under timers | 20:42 |
| corvus | right, statsd puts different kinds of metrics under different roots | 20:42 |
| dmsimard | okay, I found it now -- awesome, thanks :D | 20:43 |
| dmsimard | corvus: I'm not sure I understand why we're multiplying by 1000 -- the raw value appears to be in seconds already ? | 20:49 |
| corvus | dmsimard: right, that puts it in milliseconds. statsd only deals with integers, so if we want subsecond, we multiply. time in milliseconds is also semi-standard in javascript, so it's convenient in some cases. | 20:51 |
| dmsimard | oh, for precision -- got it | 20:51 |
| *** rlandy|brb is now known as rlandy | 20:53 | |
| *** samccann has quit IRC | 21:08 | |
| *** hasharAway has quit IRC | 21:56 | |
| *** eventingmonkey has quit IRC | 23:26 | |
| *** eventingmonkey has joined #zuul | 23:27 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: write-inventory: add ansible_python_interpreter variable https://review.openstack.org/596961 | 23:30 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!