| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Add helper to find full slots and pep8 check https://review.openstack.org/317809 | 00:03 |
|---|---|---|
| *** jamielennox is now known as jamielennox|away | 00:04 | |
| *** jamielennox|away is now known as jamielennox | 00:04 | |
| *** r-mibu has quit IRC | 00:05 | |
| pabelanger | fungi: clarkb: is the rate setting for provider some sort of rate limit? | 00:05 |
| fungi | yeah, would be better termed a delay-based throttle | 00:06 |
| *** ddieterly has joined #openstack-infra | 00:06 | |
| pabelanger | fungi: cause I am noticing when osic is not under any load, we are not leaking any FIPs. So I wonder if we can import some sort of throttle between our API calls | 00:07 |
| pabelanger | see if that helps | 00:07 |
| pabelanger | seems after we hit x amount of nodes being launched, do we start leaking the IPs | 00:07 |
| *** weshay has joined #openstack-infra | 00:07 | |
| *** gyee has quit IRC | 00:08 | |
| mordred | pabelanger: that's an excellent observation | 00:08 |
| * mordred isn't really here | 00:08 | |
| fungi | we mostly implemented it to deal with providers rate-limiting our api calls | 00:08 |
| *** gyee has joined #openstack-infra | 00:08 | |
| mordred | yah. but the delay-based throttle characteristics could keep us below the rate where we see leaks | 00:08 |
| pabelanger | Ya, cause nodepool is pretty happy right now. no expections in logs | 00:09 |
| mordred | it seems there may be two related issues: a) at some rate, the cloud derps - which is a bug in openstack, but is workaroundable maybe by throttling | 00:09 |
| mordred | b) when the cloud derps in that way, we're not detecting and responding to it properly | 00:10 |
| mordred | now - it might be that we're failing to detect and cleanup because the cloud is already behind and unable to respond to what we're asking of it | 00:10 |
| mordred | so we might be detecting properly and then just unable to recover | 00:10 |
| mordred | but I dont' think we know that for sure | 00:10 |
| *** harlowja_ has joined #openstack-infra | 00:11 | |
| mordred | pabelanger: it would be interesting to set a conservative rate, and then ramp up and see where we start leaking | 00:11 |
| pabelanger | We are definitely asking a lot of a cloud once we are leaking FIPs, launching upwards of 80 nodes at a time | 00:12 |
| *** banix has quit IRC | 00:12 | |
| *** Qiming has quit IRC | 00:12 | |
| pabelanger | mordred: like 1 second or longer? | 00:12 |
| *** mixos has joined #openstack-infra | 00:12 | |
| *** sdake has quit IRC | 00:13 | |
| *** markvoelker has quit IRC | 00:13 | |
| mordred | pabelanger: yah. that's probably a good start | 00:14 |
| *** bpokorny_ has joined #openstack-infra | 00:14 | |
| *** harlowja has quit IRC | 00:15 | |
| *** bpokorny_ has quit IRC | 00:15 | |
| mordred | pabelanger: we have rate: 0.1 for ovh-gra1 | 00:15 |
| *** bpokorny_ has joined #openstack-infra | 00:15 | |
| mordred | pabelanger: so if we do 1 and it works, maybe we can then try 0.1 and see if it can handle that | 00:15 |
| pabelanger | mordred: sure, let me get a patch up | 00:16 |
| *** bpokorn__ has joined #openstack-infra | 00:17 | |
| *** bpokorny_ has quit IRC | 00:17 | |
| *** bpokorny has quit IRC | 00:18 | |
| *** bpokorn__ has quit IRC | 00:18 | |
| *** bpokorny has joined #openstack-infra | 00:19 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Bump rate for OVH to 1 https://review.openstack.org/318384 | 00:19 |
| pabelanger | mordred: fungi: clarkb: ^ if we are interested in testing that | 00:20 |
| *** kzaitsev_mb has joined #openstack-infra | 00:20 | |
| *** r-mibu has joined #openstack-infra | 00:20 | |
| *** jamesmcarthur has joined #openstack-infra | 00:21 | |
| *** sheeprine has quit IRC | 00:22 | |
| *** nadya has joined #openstack-infra | 00:22 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Add helper to find full slots and pep8 check https://review.openstack.org/317809 | 00:22 |
| *** shashank_hegde has quit IRC | 00:23 | |
| *** amitgandhinz has joined #openstack-infra | 00:25 | |
| mordred | pabelanger: I may have confused you ... | 00:25 |
| mordred | pabelanger: osic is the one we want to lower the rate for | 00:25 |
| *** ajmiller_ has joined #openstack-infra | 00:25 | |
| mordred | I was just mentioning ovh as a point of comparison | 00:25 |
| *** dimtruck is now known as zz_dimtruck | 00:26 | |
| *** sheeprine has joined #openstack-infra | 00:26 | |
| *** ffrank has quit IRC | 00:27 | |
| *** nadya has quit IRC | 00:27 | |
| pabelanger | mordred: No, you are right. I was uploading something to ovh, and had it on my mind | 00:27 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: Publish os-api-ref to d.o.o https://review.openstack.org/318388 | 00:27 |
| *** thorst_ has joined #openstack-infra | 00:27 | |
| *** asettle has joined #openstack-infra | 00:27 | |
| openstackgerrit | David Lyle proposed openstack-infra/project-config: Update horizon dsvm tempest job https://review.openstack.org/314795 | 00:27 |
| *** sridhar_ram1 has quit IRC | 00:28 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Bump rate for osic to 1 https://review.openstack.org/318384 | 00:28 |
| *** vhosakot has quit IRC | 00:29 | |
| *** amitgandhinz has quit IRC | 00:29 | |
| *** ajmiller has quit IRC | 00:29 | |
| *** jamesmcarthur has quit IRC | 00:31 | |
| *** sridhar_ram has joined #openstack-infra | 00:32 | |
| *** asettle has quit IRC | 00:33 | |
| *** gyee has quit IRC | 00:34 | |
| *** gyee has joined #openstack-infra | 00:34 | |
| *** julim has joined #openstack-infra | 00:35 | |
| *** mikal_ is now known as mikal | 00:35 | |
| *** rfolco has quit IRC | 00:37 | |
| *** jerryz has quit IRC | 00:37 | |
| *** hichihara has joined #openstack-infra | 00:40 | |
| *** jamesmcarthur has joined #openstack-infra | 00:40 | |
| *** kzaitsev_mb has quit IRC | 00:41 | |
| *** mtanino has quit IRC | 00:43 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 00:45 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Bump rate for osic to 1.0 https://review.openstack.org/318384 | 00:46 |
| *** julim has quit IRC | 00:49 | |
| jeblair | mordred: pabelanger do we want to change the rate for the whole cloud, or just stick a sleep in when we're adding fips? | 00:50 |
| *** kzaitsev_mb has joined #openstack-infra | 00:50 | |
| jeblair | because 1 call/second overall will be s l o w. | 00:50 |
| *** Kennan has quit IRC | 00:50 | |
| pabelanger | jeblair: I'm open to suggestions. Figured if we wanted to test something, rate was the setting to use | 00:53 |
| jeblair | pabelanger: yeah, it's the only setting -- otherwise it's a code change; and possibly a difficult one to implement in nodepool | 00:54 |
| *** Apoorva has quit IRC | 00:54 | |
| jeblair | (since the tasks aren't defined in nodepool anymore, we'd have to either modify the task manager to introspect them and sleep on only the one(s) of interest, or modify shade) | 00:54 |
| *** pvaneck has quit IRC | 00:55 | |
| pabelanger | Agreed. Wasn't sure hacking shade was on the table, why I started with rate | 00:57 |
| *** kzaitsev_mb has quit IRC | 00:57 | |
| *** bpokorny has quit IRC | 00:57 | |
| jeblair | setting rate will work; just be prepared for it to reduce the overall throughput of the system more than you might expect | 00:58 |
| pabelanger | been about a 90mins now, with no leaked FIPs | 00:58 |
| *** zhurong has joined #openstack-infra | 00:58 | |
| mordred | jeblair: yah - I tink 1 sec is likely too slow | 00:58 |
| pabelanger | jeblair: ya, 1.0 might be a little aggressive | 00:58 |
| jeblair | (expect 10 minute 'time-to-ready' times) | 00:59 |
| jeblair | (compared to 1 minute now) | 00:59 |
| *** Kennan has joined #openstack-infra | 01:01 | |
| jeblair | mordred: nc 166.78.99.141 8088 | 01:02 |
| jeblair | if you want to follow along with a zuulv2.5 job :) | 01:02 |
| *** jamesmcarthur has quit IRC | 01:03 | |
| *** duobei has joined #openstack-infra | 01:03 | |
| jeblair | (it's finished now, but if you nc that, you'll get the whole console log) | 01:05 |
| *** kzaitsev_mb has joined #openstack-infra | 01:11 | |
| *** jamesmcarthur has joined #openstack-infra | 01:11 | |
| *** bodepd has quit IRC | 01:11 | |
| jeblair | and logs! http://logs.openstack.org/87/316187/2/check/gate-infra-puppet-apply-debian-jessie/f2003e4/ | 01:11 |
| pabelanger | jeblair: excellent | 01:13 |
| *** kzaitsev_mb has quit IRC | 01:16 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Several fixes to zuul launcher https://review.openstack.org/318397 | 01:19 |
| *** mriedem has joined #openstack-infra | 01:19 | |
| jeblair | pabelanger, Shrews: ^ there are the changes i made in place to get that job working | 01:20 |
| *** kzaitsev_mb has joined #openstack-infra | 01:21 | |
| *** Sukhdev has quit IRC | 01:22 | |
| jeblair | i'm going have nodepool send some load its way | 01:22 |
| openstackgerrit | Hirofumi Ichihara proposed openstack-infra/jeepyb: Don't create doc bug when stable branch https://review.openstack.org/318399 | 01:23 |
| *** roxanaghe has joined #openstack-infra | 01:24 | |
| *** doug-fish has joined #openstack-infra | 01:24 | |
| jeblair | okay it's too quiet right now | 01:24 |
| *** amitgandhinz has joined #openstack-infra | 01:25 | |
| mordred | jeblair: that's amazing | 01:25 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Remove hardcoded ext4 values https://review.openstack.org/318400 | 01:27 |
| pabelanger | ianw: ^ looks like some hardcoded ext4 values | 01:27 |
| pabelanger | jeblair: ack, will review in the morning | 01:27 |
| ianw | pabelanger: heh, no surprised ... looking | 01:27 |
| *** Qiming has joined #openstack-infra | 01:28 | |
| ianw | pabelanger: did you get a good run with that? | 01:28 |
| pabelanger | ianw: I haven't tested. Will rebuild it in the morning | 01:28 |
| ianw | pabelanger: ok, np ... when i have a sec i'll give it a go, just head-deep in mocking atm | 01:29 |
| *** roxanaghe has quit IRC | 01:29 | |
| jeblair | oops http://paste.openstack.org/show/497632/ | 01:29 |
| *** doug-fish has quit IRC | 01:29 | |
| *** Jeffrey4l has joined #openstack-infra | 01:30 | |
| *** amitgandhinz has quit IRC | 01:31 | |
| Shrews | i feel sad i cannot see live things, but happy 2.5 is progressing | 01:32 |
| *** ddieterly is now known as ddieterly[away] | 01:32 | |
| *** ddieterly[away] has quit IRC | 01:32 | |
| *** jamesmcarthur has quit IRC | 01:33 | |
| jeblair | i wonder what our $PATH is in jenkins | 01:33 |
| jeblair | since apparently we need to pass that in.... | 01:33 |
| Shrews | jeblair: that traceback hurts my head | 01:33 |
| pabelanger | ianw: okay, switch to ext3 manually rebooted, growroot failed | 01:34 |
| pabelanger | ianw: installed cloud-initramfs-growroot manually, rebooted, HDD setup as expected | 01:34 |
| *** baoli has joined #openstack-infra | 01:35 | |
| pabelanger | so, think we need cloud-initramfs-growroot for ubuntu-precise, like zigo mentioned | 01:35 |
| ianw | pabelanger: ok, so you just changed the mount flags? | 01:35 |
| ianw | didn't we merge something in dib removing that? | 01:35 |
| pabelanger | ianw: LABEL=cloudimg-rootfs / ext3 errors=remount-ro 0 1 | 01:36 |
| pabelanger | I manually changed that | 01:36 |
| pabelanger | and installed cloud-initramfs-growroot | 01:36 |
| ianw | still might be worth testing with an only ext3 created fs? | 01:37 |
| ianw | you remounted ext4 as ext3, right? | 01:37 |
| pabelanger | ianw: no, I simply rebooted and let the OS handle it | 01:38 |
| jeblair | okay, trying again with $PATH defined | 01:38 |
| ianw | pabelanger: i guess i mean how did you do "switch to ext3 manually rebooted, growroot failed" | 01:39 |
| pabelanger | ianw: I only switch /etc/fstab from ext4 to ext3 | 01:39 |
| *** Sukhdev has joined #openstack-infra | 01:39 | |
| pabelanger | then rebooted | 01:39 |
| *** jamesmcarthur has joined #openstack-infra | 01:39 | |
| ianw | right, so maybe it being originally an ext4 confused things ... it's *mostly* compatible, right? | 01:40 |
| pabelanger | /dev/vda1 on / type ext3 (rw,errors=remount-ro) | 01:40 |
| pabelanger | ya, not sure | 01:40 |
| pabelanger | but, it looks like cloud-initramfs-growroot helped it along | 01:41 |
| *** sridhar_ram has quit IRC | 01:41 | |
| ianw | so i'd say maybe let's try with a "real" ext3, but otherwise need to maybe revert https://git.openstack.org/cgit/openstack/diskimage-builder/commit/?id=e23b087505b85ae1da11054aa980a956f2ec7195 for precise? | 01:42 |
| *** thorst_ has quit IRC | 01:42 | |
| *** thorst_ has joined #openstack-infra | 01:42 | |
| pabelanger | ianw: well, it was a real ext3 partition, see: http://nodepool.openstack.org/dib.ubuntu-precise.log | 01:42 |
| pabelanger | the fstab was just incorrect | 01:43 |
| jeblair | nc 172.99.106.29 8088 | 01:43 |
| pabelanger | ianw: Ya, we can add a pkg-map for precise for it | 01:43 |
| jeblair | is a real build:gate-puppet-nova-puppet-lint job | 01:43 |
| pabelanger | ianw: or add it back | 01:43 |
| ianw | pabelanger: but didn't the build with it fail because it's hard-coded to ext4? | 01:43 |
| pabelanger | ianw: no, it worked surprisingly | 01:43 |
| jeblair | nc 149.202.161.176 8088 <-- gate-nova-docs | 01:44 |
| pabelanger | jeblair: time for some javascript updated :D | 01:44 |
| *** ewilson has joined #openstack-infra | 01:44 | |
| jeblair | let's see if these don't bomb first :) | 01:44 |
| pabelanger | pretty awesome, watching puppet job | 01:44 |
| jeblair | Shrews: ^ live things | 01:45 |
| *** Sukhdev has quit IRC | 01:45 | |
| flwang | hi can anybody tell me the difference between python-xxx-bitrot-jobs and python-xxx-bitrot-db-jobs ? thanks in advance | 01:46 |
| ianw | pabelanger: ahh, ok so the fs creation works, it's just the mount values | 01:46 |
| jeblair | there is one more thing we may want to do before throwing this into prod -- and that's some kind of estimated time reporting | 01:46 |
| pabelanger | ianw: ya, sorry if I was not clear. | 01:46 |
| pabelanger | flwang: -db will setup and used a database for the job. Other does not | 01:47 |
| ianw | pabelanger: yeah, in that case, it seems like the growroot element show know if it's building precise to just use the pkg | 01:47 |
| pabelanger | ianw: Right, so we can add a pkg-map for precise for it | 01:47 |
| flwang | pabelanger: so if we(zaqar) need a mongo/redis as backend, we should use the db one, right? | 01:48 |
| *** hichihara has quit IRC | 01:48 | |
| pabelanger | flwang: no, our -db jobs only setup mysql and postgres. So, you'll need to add logic to set that up for your jobs | 01:49 |
| *** duobei has quit IRC | 01:49 | |
| ianw | pabelanger: hmm, maybe more than that, as it tries to fiddle init scripts. it actually says "trusty and beyond" in the readme, i guess because of this :) | 01:49 |
| ianw | pabelanger: oh, but in debootstrap, i guess yeah | 01:50 |
| pabelanger | ianw: Ya, if you check backscroll zigo mention the version of kernel in precise is too old or something like that | 01:50 |
| flwang | pabelanger: but by default, it will call devstack to setup the env, right? | 01:50 |
| *** baoli has quit IRC | 01:51 | |
| pabelanger | flwang: if you job is configured to do that, yes. | 01:51 |
| flwang | pabelanger: ok, cool, thanks | 01:51 |
| *** baoli has joined #openstack-infra | 01:51 | |
| *** thorst_ has quit IRC | 01:51 | |
| jeblair | okay, i think there were some failures and some successes; i'll dig through the log and see what i can find | 01:53 |
| pabelanger | Ya, I am calling it for tonight. | 01:53 |
| pabelanger | jeblair: good luck! | 01:53 |
| openstackgerrit | Fei Long Wang proposed openstack-infra/project-config: Enable bitrot jobs for Zaqar https://review.openstack.org/318404 | 01:54 |
| *** jamesmcarthur has quit IRC | 01:55 | |
| *** yamahata has joined #openstack-infra | 01:55 | |
| *** duobei has joined #openstack-infra | 01:56 | |
| *** homeski has joined #openstack-infra | 01:56 | |
| *** baoli has quit IRC | 01:57 | |
| *** ewilson has quit IRC | 01:57 | |
| *** jyuso1 has joined #openstack-infra | 01:59 | |
| *** pradk has joined #openstack-infra | 01:59 | |
| *** kzaitsev_mb has quit IRC | 02:00 | |
| *** jamesmcarthur has joined #openstack-infra | 02:04 | |
| *** jamesmcarthur has quit IRC | 02:05 | |
| *** yamamoto_ has joined #openstack-infra | 02:06 | |
| *** pradk_ has joined #openstack-infra | 02:06 | |
| *** tqtran has quit IRC | 02:09 | |
| *** pradk has quit IRC | 02:10 | |
| *** pradk_ is now known as pradk | 02:10 | |
| *** mrmartin has joined #openstack-infra | 02:12 | |
| Nakato | Hey folks, what needs to happen to get this zuul change in? https://review.openstack.org/#/c/293194/ | 02:13 |
| *** mrmartin has quit IRC | 02:14 | |
| mordred | Nakato: jeblair has been in the middle of getting zuul 2.5 out - so I think he likely should look at it | 02:17 |
| *** vhosakot has joined #openstack-infra | 02:17 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: handle Ant paths in publishers https://review.openstack.org/318411 | 02:17 |
| *** yonglihe has joined #openstack-infra | 02:17 | |
| jeblair | Nakato: ah, thanks for the reminder -- that is blocking work. i'll try to look at it tomorrow | 02:18 |
| *** tqtran has joined #openstack-infra | 02:18 | |
| *** sdake has joined #openstack-infra | 02:19 | |
| Nakato | jeblair: Thanks | 02:20 |
| Nakato | Thanks mordred | 02:20 |
| *** nadya has joined #openstack-infra | 02:23 | |
| *** sdake has quit IRC | 02:23 | |
| *** rossella_s has quit IRC | 02:24 | |
| *** rossella_s has joined #openstack-infra | 02:24 | |
| *** tqtran has quit IRC | 02:26 | |
| *** nadya has quit IRC | 02:27 | |
| *** amitgandhinz has joined #openstack-infra | 02:27 | |
| *** sree has joined #openstack-infra | 02:30 | |
| openstackgerrit | Dong Ma proposed openstack-infra/subunit2sql: Fix db API get_ids_for_all_tests https://review.openstack.org/310323 | 02:32 |
| *** amitgandhinz has quit IRC | 02:32 | |
| *** sree has quit IRC | 02:33 | |
| *** mriedem has quit IRC | 02:33 | |
| *** yamahata has quit IRC | 02:34 | |
| *** links has joined #openstack-infra | 02:35 | |
| *** baoli has joined #openstack-infra | 02:35 | |
| *** tlian has quit IRC | 02:36 | |
| *** baoli has quit IRC | 02:39 | |
| *** baoli has joined #openstack-infra | 02:39 | |
| *** sree has joined #openstack-infra | 02:40 | |
| *** ramishra has joined #openstack-infra | 02:42 | |
| *** camunoz_ has joined #openstack-infra | 02:42 | |
| *** baoli has quit IRC | 02:42 | |
| *** baoli has joined #openstack-infra | 02:43 | |
| *** xarses has quit IRC | 02:43 | |
| *** woodster_ has quit IRC | 02:48 | |
| *** thorst_ has joined #openstack-infra | 02:48 | |
| *** jesusaur has quit IRC | 02:49 | |
| *** Wei_Liu has quit IRC | 02:52 | |
| *** banix has joined #openstack-infra | 02:52 | |
| *** lakshmiS has quit IRC | 02:52 | |
| *** jesusaur has joined #openstack-infra | 02:54 | |
| *** yuanying has quit IRC | 02:54 | |
| *** xarses has joined #openstack-infra | 02:55 | |
| *** thorst_ has quit IRC | 02:57 | |
| *** ajmiller has joined #openstack-infra | 02:59 | |
| *** binbincong has quit IRC | 03:01 | |
| *** ajmiller_ has quit IRC | 03:02 | |
| *** nemish has quit IRC | 03:02 | |
| *** nemish has joined #openstack-infra | 03:04 | |
| *** bhunter71 has quit IRC | 03:04 | |
| *** baoli has quit IRC | 03:09 | |
| *** jesusaur has quit IRC | 03:10 | |
| *** vhosakot has quit IRC | 03:11 | |
| *** jesusaur has joined #openstack-infra | 03:13 | |
| *** jswarren has quit IRC | 03:19 | |
| *** jswarren has joined #openstack-infra | 03:19 | |
| *** tqtran has joined #openstack-infra | 03:23 | |
| *** vhosakot has joined #openstack-infra | 03:25 | |
| *** yamamoto_ has quit IRC | 03:26 | |
| *** tqtran has quit IRC | 03:27 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Fix db API get_ids_for_all_tests https://review.openstack.org/310323 | 03:29 |
| *** zz_dimtruck is now known as dimtruck | 03:31 | |
| *** ajmiller has quit IRC | 03:33 | |
| *** weshay has quit IRC | 03:35 | |
| *** psachin has joined #openstack-infra | 03:38 | |
| *** ajmiller has joined #openstack-infra | 03:38 | |
| *** pilgrimstack1 has quit IRC | 03:41 | |
| *** pilgrimstack has joined #openstack-infra | 03:42 | |
| *** banix has quit IRC | 03:43 | |
| *** homeski has quit IRC | 03:44 | |
| *** yuanying has joined #openstack-infra | 03:46 | |
| *** duobei has quit IRC | 03:46 | |
| openstackgerrit | Matthew Treinish proposed openstack-dev/pbr: Add reST label to the generated changelog https://review.openstack.org/318430 | 03:50 |
| *** coreyob has quit IRC | 03:51 | |
| *** thorst_ has joined #openstack-infra | 03:54 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add link to published release notes in readme https://review.openstack.org/318433 | 03:55 |
| *** winggundamth has quit IRC | 03:55 | |
| *** jesusaur has quit IRC | 03:55 | |
| *** jesusaur has joined #openstack-infra | 03:58 | |
| *** links has quit IRC | 04:00 | |
| *** thorst_ has quit IRC | 04:01 | |
| mtreinish | krotscheck: I've been hitting that npm error a bunch (at least 3 times this evening) | 04:01 |
| mtreinish | krotscheck: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22npm%20ERR!%20registry%20error%20parsing%20json%5C%22 | 04:01 |
| mtreinish | krotscheck: looks like 37 hits in the last 24 hrs (which feels like a lot for js jobs) | 04:02 |
| mtreinish | krotscheck: maybe we should temporarily disable the mirror if it's causing problems | 04:03 |
| *** chlong has joined #openstack-infra | 04:04 | |
| *** flwang has quit IRC | 04:04 | |
| *** psachin has quit IRC | 04:05 | |
| *** amotoki has joined #openstack-infra | 04:06 | |
| *** yamamoto_ has joined #openstack-infra | 04:09 | |
| *** flwang has joined #openstack-infra | 04:11 | |
| *** mixos has quit IRC | 04:12 | |
| *** mixos has joined #openstack-infra | 04:12 | |
| *** links has joined #openstack-infra | 04:16 | |
| *** amitgandhinz has joined #openstack-infra | 04:16 | |
| *** Jeffrey4l has quit IRC | 04:16 | |
| *** dimtruck is now known as zz_dimtruck | 04:18 | |
| *** mixos has quit IRC | 04:18 | |
| *** ajmiller has quit IRC | 04:20 | |
| *** psachin has joined #openstack-infra | 04:20 | |
| *** Sukhdev has joined #openstack-infra | 04:20 | |
| *** amitgandhinz has quit IRC | 04:21 | |
| *** nadya has joined #openstack-infra | 04:23 | |
| *** Sukhdev has quit IRC | 04:24 | |
| *** Sukhdev has joined #openstack-infra | 04:24 | |
| *** vhosakot has quit IRC | 04:25 | |
| *** nadya has quit IRC | 04:28 | |
| *** shashank_hegde has joined #openstack-infra | 04:30 | |
| openstackgerrit | Chris Hoge proposed openstack-infra/project-config: Add DefCore guideline gate checks for json lint and schema https://review.openstack.org/317800 | 04:33 |
| *** shashank_hegde has quit IRC | 04:37 | |
| *** shashank_hegde has joined #openstack-infra | 04:39 | |
| *** armax has quit IRC | 04:40 | |
| *** kengo_sakai has joined #openstack-infra | 04:41 | |
| *** winggundamth has joined #openstack-infra | 04:42 | |
| *** zz_dimtruck is now known as dimtruck | 04:43 | |
| *** kengo_sakai has quit IRC | 04:46 | |
| *** sflanigan has quit IRC | 04:46 | |
| *** sflanigan has joined #openstack-infra | 04:47 | |
| *** sflanigan has joined #openstack-infra | 04:47 | |
| hogepodge | jhesketh: thanks for your help. think it finally sank in. | 04:48 |
| jhesketh | no worries :-) | 04:49 |
| jhesketh | hogepodge: looks like you might need to rebase your change | 04:49 |
| hogepodge | jhesketh: I'm depending on another patch to land. Could it be that? | 04:50 |
| openstackgerrit | Madhuri Kumari proposed openstack-infra/project-config: Add new project Higginsclient to openstack repo https://review.openstack.org/317699 | 04:50 |
| openstackgerrit | Chris Hoge proposed openstack-infra/project-config: Add DefCore guideline gate checks for json lint and schema https://review.openstack.org/317800 | 04:51 |
| *** shashank_hegde has quit IRC | 04:51 | |
| jhesketh | hogepodge: yeah it might have been outdated or something | 04:52 |
| *** shashank_hegde has joined #openstack-infra | 04:52 | |
| bkero | 04:52 | |
| *** gongysh has joined #openstack-infra | 04:53 | |
| *** armax has joined #openstack-infra | 04:54 | |
| *** thorst_ has joined #openstack-infra | 04:59 | |
| *** armax has quit IRC | 05:01 | |
| *** kengo_sakai has joined #openstack-infra | 05:02 | |
| *** armax has joined #openstack-infra | 05:03 | |
| *** gongysh has quit IRC | 05:04 | |
| *** thorst_ has quit IRC | 05:06 | |
| *** oanson has joined #openstack-infra | 05:07 | |
| *** doug-fish has joined #openstack-infra | 05:11 | |
| *** doug-fish has quit IRC | 05:15 | |
| *** amitgandhinz has joined #openstack-infra | 05:17 | |
| *** chlong has quit IRC | 05:17 | |
| *** kushal has joined #openstack-infra | 05:17 | |
| *** roxanaghe has joined #openstack-infra | 05:18 | |
| *** nadya has joined #openstack-infra | 05:20 | |
| *** roxanaghe has quit IRC | 05:20 | |
| *** amitgandhinz has quit IRC | 05:21 | |
| *** tqtran has joined #openstack-infra | 05:24 | |
| openstackgerrit | Merged openstack/python-jenkins: Fetch all builds from a job https://review.openstack.org/309554 | 05:26 |
| *** kengo_sakai has quit IRC | 05:27 | |
| *** markvoelker has joined #openstack-infra | 05:27 | |
| *** tqtran has quit IRC | 05:28 | |
| *** chlong has joined #openstack-infra | 05:31 | |
| SpamapS | hm | 05:32 |
| SpamapS | so, we're using the afs mirrors from infra, and seeing something inconsistent... | 05:33 |
| SpamapS | http://paste.openstack.org/show/497643/ | 05:33 |
| SpamapS | setuptools has a wheel reference | 05:33 |
| SpamapS | but the wheel is not there | 05:33 |
| *** markvoelker has quit IRC | 05:34 | |
| *** jesusaur has quit IRC | 05:34 | |
| *** javeriak has joined #openstack-infra | 05:35 | |
| *** savihou has joined #openstack-infra | 05:35 | |
| *** dimtruck is now known as zz_dimtruck | 05:35 | |
| *** jesusaur has joined #openstack-infra | 05:44 | |
| *** yamahata has joined #openstack-infra | 05:44 | |
| *** duobei has joined #openstack-infra | 05:48 | |
| *** ramishra has quit IRC | 05:50 | |
| *** xarses_ has joined #openstack-infra | 05:50 | |
| *** xarses_ has quit IRC | 05:51 | |
| *** xarses_ has joined #openstack-infra | 05:51 | |
| *** ramishra has joined #openstack-infra | 05:51 | |
| *** salv-orlando has joined #openstack-infra | 05:51 | |
| *** xarses has quit IRC | 05:52 | |
| *** javeriak has quit IRC | 05:53 | |
| *** kushal has quit IRC | 05:55 | |
| *** gyee has quit IRC | 05:56 | |
| openstackgerrit | Yanyan Hu proposed openstack-infra/project-config: Add Senlin support to rally-gate https://review.openstack.org/318453 | 05:56 |
| *** fawadkhaliq has joined #openstack-infra | 05:57 | |
| *** Sukhdev has quit IRC | 05:57 | |
| *** doug-fish has joined #openstack-infra | 05:57 | |
| *** Jeffrey4l has joined #openstack-infra | 05:58 | |
| *** armax has quit IRC | 06:00 | |
| *** doug-fish has quit IRC | 06:01 | |
| *** Ravikiran_K has joined #openstack-infra | 06:02 | |
| *** thorst_ has joined #openstack-infra | 06:04 | |
| *** chlong has quit IRC | 06:05 | |
| zaro | mordred: around? | 06:08 |
| *** kushal has joined #openstack-infra | 06:08 | |
| *** reedip_ has joined #openstack-infra | 06:10 | |
| *** abregman has quit IRC | 06:10 | |
| *** nadya has quit IRC | 06:10 | |
| *** thorst_ has quit IRC | 06:10 | |
| *** nadya has joined #openstack-infra | 06:10 | |
| *** rcernin has joined #openstack-infra | 06:13 | |
| *** nadya has quit IRC | 06:15 | |
| SpamapS | zaro: you around? see my message above about the pypi mirror? | 06:16 |
| *** amitgandhinz has joined #openstack-infra | 06:18 | |
| *** Jeffrey4l has quit IRC | 06:18 | |
| *** jgriffith has quit IRC | 06:19 | |
| *** flepied1 has quit IRC | 06:21 | |
| *** itisha has joined #openstack-infra | 06:22 | |
| *** chlong has joined #openstack-infra | 06:22 | |
| *** amitgandhinz has quit IRC | 06:22 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add magnum baymodel calls to shade. https://review.openstack.org/308009 | 06:23 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add magnum services call to shade https://review.openstack.org/313583 | 06:26 |
| *** jgriffith has joined #openstack-infra | 06:27 | |
| *** jgriffith is now known as Guest60864 | 06:27 | |
| *** pt_15 has joined #openstack-infra | 06:29 | |
| *** salv-orlando has quit IRC | 06:32 | |
| *** shashank_hegde has quit IRC | 06:33 | |
| *** shashank_hegde has joined #openstack-infra | 06:34 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add cloud-initramfs-growroot for Precise https://review.openstack.org/318461 | 06:38 |
| *** nmagnezi has joined #openstack-infra | 06:38 | |
| ianw | pabelanger: ^ lightly tested ... | 06:38 |
| *** nadya has joined #openstack-infra | 06:39 | |
| *** duobei has quit IRC | 06:39 | |
| *** duobei has joined #openstack-infra | 06:45 | |
| *** nmagnezi has quit IRC | 06:46 | |
| *** aeng has quit IRC | 06:46 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Use mock patching for most functions https://review.openstack.org/318464 | 06:48 |
| *** arxcruz has joined #openstack-infra | 06:49 | |
| *** nmagnezi has joined #openstack-infra | 06:49 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Remove hardcoded ext4 values https://review.openstack.org/318400 | 06:53 |
| *** slaweq has joined #openstack-infra | 06:53 | |
| *** slaweq has quit IRC | 06:54 | |
| *** chlong has quit IRC | 06:54 | |
| *** exploreshaifali has joined #openstack-infra | 06:54 | |
| *** slaweq has joined #openstack-infra | 06:54 | |
| *** slaweq has quit IRC | 06:54 | |
| *** slaweq has joined #openstack-infra | 06:57 | |
| *** andreas_s has joined #openstack-infra | 06:58 | |
| *** jaosorior has quit IRC | 06:59 | |
| *** flepied has joined #openstack-infra | 06:59 | |
| *** Hal1 has joined #openstack-infra | 07:00 | |
| *** jaosorior has joined #openstack-infra | 07:00 | |
| *** nadya has quit IRC | 07:01 | |
| *** nadya has joined #openstack-infra | 07:01 | |
| *** nadya has quit IRC | 07:06 | |
| *** thorst_ has joined #openstack-infra | 07:08 | |
| *** kushal has quit IRC | 07:10 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add cloud-initramfs-growroot for Precise https://review.openstack.org/318461 | 07:12 |
| *** mrmartin has joined #openstack-infra | 07:13 | |
| *** vincentll has joined #openstack-infra | 07:13 | |
| *** akaszuba has joined #openstack-infra | 07:15 | |
| *** thorst_ has quit IRC | 07:15 | |
| *** mikelk has joined #openstack-infra | 07:16 | |
| *** abregman has joined #openstack-infra | 07:16 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add IP addresses compliant with rfc5737 https://review.openstack.org/289250 | 07:17 |
| *** amitgandhinz has joined #openstack-infra | 07:18 | |
| *** javeriak has joined #openstack-infra | 07:20 | |
| *** ccamacho has joined #openstack-infra | 07:20 | |
| *** amitgandhinz has quit IRC | 07:23 | |
| *** jlanoux has joined #openstack-infra | 07:24 | |
| *** kushal has joined #openstack-infra | 07:25 | |
| *** exploreshaifali has quit IRC | 07:27 | |
| *** esikachev has joined #openstack-infra | 07:28 | |
| *** duobei has quit IRC | 07:29 | |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Remove more old Ironic ramdisk gates https://review.openstack.org/318477 | 07:34 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Export FS_TYPE and remove hardcoded ext4 values https://review.openstack.org/318400 | 07:34 |
| *** camunoz_ has quit IRC | 07:34 | |
| *** mrmartin has quit IRC | 07:35 | |
| *** ihrachys has joined #openstack-infra | 07:37 | |
| *** fawadkhaliq has quit IRC | 07:39 | |
| *** fawadkhaliq has joined #openstack-infra | 07:39 | |
| *** megm_ has quit IRC | 07:40 | |
| *** matrohon has joined #openstack-infra | 07:40 | |
| *** ihrachys has quit IRC | 07:41 | |
| *** megm has joined #openstack-infra | 07:42 | |
| *** gildub has joined #openstack-infra | 07:44 | |
| *** ccamacho is now known as ccamacho|afk | 07:46 | |
| *** jordanP has joined #openstack-infra | 07:46 | |
| *** salv-orl_ has joined #openstack-infra | 07:47 | |
| *** salv-orl_ has quit IRC | 07:47 | |
| *** salv-orlando has joined #openstack-infra | 07:48 | |
| *** dmk0202 has joined #openstack-infra | 07:50 | |
| *** zeih has joined #openstack-infra | 07:53 | |
| *** e0ne has joined #openstack-infra | 07:53 | |
| *** ifarkas has joined #openstack-infra | 07:56 | |
| *** exploreshaifali has joined #openstack-infra | 07:56 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add Higgins team meeting https://review.openstack.org/318335 | 07:56 |
| *** duobei has joined #openstack-infra | 07:57 | |
| *** HeOS has joined #openstack-infra | 07:57 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** yaume has joined #openstack-infra | 08:00 | |
| *** hichihara has joined #openstack-infra | 08:00 | |
| openstackgerrit | chenying proposed openstack-infra/project-config: Add smaug fullstack job https://review.openstack.org/317566 | 08:00 |
| *** exploreshaifali has quit IRC | 08:02 | |
| *** zzzeek has joined #openstack-infra | 08:02 | |
| *** shardy has joined #openstack-infra | 08:05 | |
| *** jamielennox is now known as jamielennox|away | 08:05 | |
| yolanda | good morning | 08:07 |
| *** HeOS has quit IRC | 08:07 | |
| *** dtantsur|afk is now known as dtantsur | 08:08 | |
| *** sflanigan has quit IRC | 08:09 | |
| *** thorst_ has joined #openstack-infra | 08:14 | |
| *** hashar has joined #openstack-infra | 08:17 | |
| *** yamahata has quit IRC | 08:17 | |
| *** amitgandhinz has joined #openstack-infra | 08:19 | |
| *** ihrachys has joined #openstack-infra | 08:20 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/glean: Do not overwrite readonly files https://review.openstack.org/309880 | 08:20 |
| *** daemontool has joined #openstack-infra | 08:21 | |
| *** thorst_ has quit IRC | 08:21 | |
| openstackgerrit | Lukas Bednar proposed openstack-infra/git-review: Fix AttributeError when can not connect to Gerrit https://review.openstack.org/318181 | 08:22 |
| *** amitgandhinz has quit IRC | 08:24 | |
| *** dingyichen has quit IRC | 08:24 | |
| *** asettle has joined #openstack-infra | 08:25 | |
| *** electrofelix has joined #openstack-infra | 08:26 | |
| *** lucas-dinner is now known as lucasagomes | 08:26 | |
| *** jordanP has quit IRC | 08:27 | |
| *** nadya has joined #openstack-infra | 08:29 | |
| *** jistr has joined #openstack-infra | 08:30 | |
| *** nadya has quit IRC | 08:34 | |
| *** liusheng has quit IRC | 08:34 | |
| *** liusheng has joined #openstack-infra | 08:35 | |
| *** Julien-zte has joined #openstack-infra | 08:36 | |
| *** zhurong has quit IRC | 08:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Promote OSA infrastructure Xenial/ jobs to non-voting https://review.openstack.org/318168 | 08:36 |
| *** zhurong has joined #openstack-infra | 08:37 | |
| *** shashank_hegde has quit IRC | 08:38 | |
| SotK | fungi: seems to be working, thanks for all your work on this :) | 08:38 |
| *** jordanP has joined #openstack-infra | 08:38 | |
| *** nadya has joined #openstack-infra | 08:39 | |
| electrofelix | zxiiro: I'd like to get https://review.openstack.org/309734 to allow testing non-default configs and https://review.openstack.org/314789 for prevention of cache corruption (small typo to fix) for 1.6 | 08:41 |
| *** Julien-w has joined #openstack-infra | 08:42 | |
| electrofelix | I think being able to create an additional job without needing to worry about avoiding it on a stable branch would be convenient (but not critical). | 08:42 |
| *** exploreshaifali has joined #openstack-infra | 08:42 | |
| *** sridhar_ram_ has quit IRC | 08:45 | |
| *** chem`` has quit IRC | 08:45 | |
| *** ccamacho|afk is now known as ccamacho | 08:45 | |
| *** dtantsur is now known as dtantsur|bbl | 08:46 | |
| *** Na3iL has joined #openstack-infra | 08:54 | |
| *** esikachev has quit IRC | 08:59 | |
| *** zeih has quit IRC | 09:00 | |
| *** derekh has joined #openstack-infra | 09:00 | |
| *** jcoufal has joined #openstack-infra | 09:00 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add Magnum bays support https://review.openstack.org/318518 | 09:03 |
| *** chem has joined #openstack-infra | 09:06 | |
| *** markusry has joined #openstack-infra | 09:06 | |
| *** duobei has quit IRC | 09:06 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Don't use DEVSTACK_GATE_TEMPEST for Ironic grenade https://review.openstack.org/318521 | 09:06 |
| *** duobei has joined #openstack-infra | 09:08 | |
| *** Na3iL has quit IRC | 09:11 | |
| *** chem has quit IRC | 09:12 | |
| *** chem has joined #openstack-infra | 09:12 | |
| zigo | SotK: fungi: I'm still getting an "UNSTABLE" result with "No SCP site is configured. This is likely a configuration problem.", however, there's no {site} stuff anymore in the publish macro. What's wrong? | 09:16 |
| zigo | https://review.openstack.org/#/c/318500/1 | 09:16 |
| *** esikachev has joined #openstack-infra | 09:17 | |
| *** sambetts|afk is now known as sambetts | 09:17 | |
| *** thorst_ has joined #openstack-infra | 09:19 | |
| *** amitgandhinz has joined #openstack-infra | 09:20 | |
| *** eliqiao has quit IRC | 09:21 | |
| *** ihrachys has quit IRC | 09:23 | |
| *** fawadkhaliq has quit IRC | 09:24 | |
| *** fawadkhaliq has joined #openstack-infra | 09:24 | |
| *** amitgandhinz has quit IRC | 09:25 | |
| *** thorst_ has quit IRC | 09:26 | |
| *** DevBox has joined #openstack-infra | 09:28 | |
| *** Zhoragh has joined #openstack-infra | 09:30 | |
| *** zqfan has joined #openstack-infra | 09:30 | |
| *** Zhoragh has left #openstack-infra | 09:30 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:31 | |
| *** javeriak has quit IRC | 09:34 | |
| *** zeih has joined #openstack-infra | 09:36 | |
| *** zeih has quit IRC | 09:40 | |
| *** zhurong has quit IRC | 09:43 | |
| *** gildub has quit IRC | 09:44 | |
| *** zhurong has joined #openstack-infra | 09:44 | |
| *** kzaitsev_mb has quit IRC | 09:45 | |
| *** gabriel-bezerra has quit IRC | 09:45 | |
| *** ericksonsantos has quit IRC | 09:46 | |
| *** Ravikiran_K has quit IRC | 09:46 | |
| *** eslamelhusseiny has joined #openstack-infra | 09:49 | |
| *** ericksonsantos has joined #openstack-infra | 09:49 | |
| *** gabriel-bezerra has joined #openstack-infra | 09:50 | |
| *** hichihara has quit IRC | 09:52 | |
| *** sridhar_ram_ has joined #openstack-infra | 09:56 | |
| *** eslamelhusseiny has quit IRC | 09:59 | |
| *** zeih has joined #openstack-infra | 10:03 | |
| *** matrohon has quit IRC | 10:04 | |
| *** zeih has quit IRC | 10:08 | |
| *** dtantsur|bbl is now known as dtantsur | 10:11 | |
| *** javeriak has joined #openstack-infra | 10:12 | |
| *** salv-orl_ has joined #openstack-infra | 10:14 | |
| *** salv-orlando has quit IRC | 10:17 | |
| *** claudiub has joined #openstack-infra | 10:18 | |
| *** gsagie has left #openstack-infra | 10:20 | |
| *** amitgandhinz has joined #openstack-infra | 10:21 | |
| *** duobei has quit IRC | 10:22 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add Sahara role to OpenStack-Ansible https://review.openstack.org/317931 | 10:23 |
| *** d0ugal has quit IRC | 10:24 | |
| *** amitgandhinz has quit IRC | 10:25 | |
| *** d0ugal has joined #openstack-infra | 10:28 | |
| *** exploreshaifali has quit IRC | 10:29 | |
| *** Qiming has quit IRC | 10:29 | |
| *** duobei has joined #openstack-infra | 10:30 | |
| *** d0ugal has quit IRC | 10:31 | |
| *** d0ugal has joined #openstack-infra | 10:31 | |
| *** jordanP has quit IRC | 10:36 | |
| *** sdague has joined #openstack-infra | 10:37 | |
| *** esikachev has quit IRC | 10:37 | |
| *** duobei has quit IRC | 10:39 | |
| *** ansiwen_ is now known as ansiwen | 10:41 | |
| *** javeriak has quit IRC | 10:41 | |
| *** gildub has joined #openstack-infra | 10:45 | |
| *** duobei has joined #openstack-infra | 10:45 | |
| *** ccamacho has quit IRC | 10:48 | |
| *** yonglihe has quit IRC | 10:48 | |
| *** duobei has quit IRC | 10:50 | |
| *** mrmartin has joined #openstack-infra | 10:50 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:52 | |
| *** njohnston has quit IRC | 10:54 | |
| *** weshay has joined #openstack-infra | 10:55 | |
| *** njohnston has joined #openstack-infra | 10:55 | |
| *** yamamoto_ has quit IRC | 10:56 | |
| *** javeriak has joined #openstack-infra | 10:58 | |
| *** esikachev has joined #openstack-infra | 11:00 | |
| *** mrmartin has quit IRC | 11:00 | |
| *** jamielennox|away is now known as jamielennox | 11:03 | |
| *** exploreshaifali has joined #openstack-infra | 11:04 | |
| *** tlian has joined #openstack-infra | 11:04 | |
| *** zhurong has quit IRC | 11:04 | |
| *** YorikSar has quit IRC | 11:05 | |
| *** esikachev has quit IRC | 11:05 | |
| *** ccamacho has joined #openstack-infra | 11:06 | |
| *** YorikSar has joined #openstack-infra | 11:07 | |
| *** Julien-w has quit IRC | 11:11 | |
| *** Julien-zte has quit IRC | 11:12 | |
| *** ccamacho has quit IRC | 11:13 | |
| *** YorikSar has quit IRC | 11:16 | |
| *** YorikSar has joined #openstack-infra | 11:16 | |
| *** jordanP has joined #openstack-infra | 11:17 | |
| *** rodrigods has quit IRC | 11:20 | |
| *** rodrigods has joined #openstack-infra | 11:20 | |
| *** amitgandhinz has joined #openstack-infra | 11:21 | |
| *** Qiming has joined #openstack-infra | 11:23 | |
| *** sree has quit IRC | 11:23 | |
| *** esikachev has joined #openstack-infra | 11:24 | |
| *** amitgandhinz has quit IRC | 11:26 | |
| *** mrmartin has joined #openstack-infra | 11:26 | |
| *** thorst_ has joined #openstack-infra | 11:27 | |
| *** ldnunes has joined #openstack-infra | 11:27 | |
| *** doug-fish has joined #openstack-infra | 11:28 | |
| *** doug-fish has quit IRC | 11:29 | |
| *** |-paul-| has joined #openstack-infra | 11:30 | |
| *** lucasagomes is now known as lucas-hungry | 11:30 | |
| *** fawadkhaliq has quit IRC | 11:30 | |
| *** javeriak has quit IRC | 11:31 | |
| *** mtanino has joined #openstack-infra | 11:31 | |
| *** zeih has joined #openstack-infra | 11:31 | |
| *** mrmartin has quit IRC | 11:32 | |
| *** rhallisey has joined #openstack-infra | 11:33 | |
| *** psachin has quit IRC | 11:33 | |
| *** YorikSar_ has joined #openstack-infra | 11:33 | |
| *** YorikSar has quit IRC | 11:35 | |
| *** tpsilva has joined #openstack-infra | 11:35 | |
| *** zeih has quit IRC | 11:38 | |
| *** kushal has quit IRC | 11:38 | |
| *** exploreshaifali has quit IRC | 11:39 | |
| *** jang has joined #openstack-infra | 11:40 | |
| *** gordc has joined #openstack-infra | 11:41 | |
| *** Hal1 has quit IRC | 11:45 | |
| *** ccamacho has joined #openstack-infra | 11:45 | |
| *** Na3iL has joined #openstack-infra | 11:45 | |
| *** deadnull has joined #openstack-infra | 11:49 | |
| *** ccamacho has quit IRC | 11:52 | |
| rcarrillocruz | yolanda: are you around to help me out on spinning logstash workers with the ansible cloud launcher? | 11:53 |
| rcarrillocruz | i need core eyes, to tell me params used for creating instances | 11:53 |
| rcarrillocruz | net names | 11:53 |
| rcarrillocruz | keys | 11:53 |
| rcarrillocruz | flavors | 11:53 |
| rcarrillocruz | etc | 11:53 |
| yolanda | i need over 30 minutes and i'll be available | 11:53 |
| rcarrillocruz | ok, no rush | 11:54 |
| *** lucas-hungry is now known as lucasagomes | 11:57 | |
| *** ggnel_t has joined #openstack-infra | 11:58 | |
| *** zeih has joined #openstack-infra | 11:59 | |
| *** psachin has joined #openstack-infra | 12:00 | |
| *** zhoragh has joined #openstack-infra | 12:01 | |
| zhoragh | in case of gerrit deployment i have trouble with 'bin/gerrit.war init -d /home/gerrit2/review_site --batch --no-auto-start' | 12:01 |
| zhoragh | currently we setup our own gerrit -> zuul -> jenkins infrastructure with help of openstack-infra/system-config | 12:01 |
| zhoragh | Downloading http://www.bouncycastle.org/download/bcpkix-jdk15on-151.jar ... !! FAIL !! | 12:01 |
| zhoragh | how can i avoid this error? | 12:02 |
| *** trown|outtypewww is now known as trown | 12:02 | |
| *** ddieterly has joined #openstack-infra | 12:03 | |
| *** gildub has quit IRC | 12:03 | |
| rcarrillocruz | zhoragh: https://review.openstack.org/#/c/318280/ | 12:04 |
| rcarrillocruz | check it out with zaro | 12:04 |
| rcarrillocruz | (zaro = Khai Do) | 12:04 |
| rcarrillocruz | also, you should really use puppet-openstackci | 12:06 |
| rcarrillocruz | trying to reconsume system-config is going to be painful for you | 12:06 |
| rcarrillocruz | (lot of people tried) | 12:06 |
| rcarrillocruz | s/lot/lots | 12:06 |
| *** exploreshaifali has joined #openstack-infra | 12:06 | |
| rcarrillocruz | puppet-openstackci is meant to be the puppet modules downstream users can reuse | 12:06 |
| rcarrillocruz | system-config is an implementation of openstack infra specific logic, which leverages puppet-openstackci modules | 12:07 |
| *** jistr has quit IRC | 12:08 | |
| *** coreyob has joined #openstack-infra | 12:08 | |
| *** timl0101 has joined #openstack-infra | 12:10 | |
| zhoragh | rcarrillocruz: puppet-openstackci do not deploy gerrit | 12:11 |
| rcarrillocruz | yep, and we need contributors | 12:11 |
| zhoragh | for the other stuff it is nice | 12:11 |
| rcarrillocruz | creating a class that is a shim of gerrit that just adds project_config should be easy | 12:11 |
| zhoragh | ok | 12:11 |
| rcarrillocruz | check it out with pschwartz, he told me he was working on puppet gerrit for puppet-openstackci | 12:11 |
| rcarrillocruz | he may have something on his laptop, awaiting for push | 12:12 |
| *** ccamacho has joined #openstack-infra | 12:12 | |
| *** nmagnezi has quit IRC | 12:13 | |
| zhoragh | can you give me the source for pschwartz | 12:13 |
| *** amrith is now known as _amrith_ | 12:14 | |
| *** |-paul-| has quit IRC | 12:14 | |
| rcarrillocruz | pscharwtz is a person | 12:14 |
| rcarrillocruz | not around now it seems | 12:14 |
| rcarrillocruz | pschwartz too | 12:15 |
| *** raildo-afk is now known as raildo | 12:15 | |
| *** ccamacho has quit IRC | 12:16 | |
| *** markvoelker has joined #openstack-infra | 12:16 | |
| zhoragh | rcarrillocruz: sorry i know, what i mean is how can i get contact to pscharwtz or his code | 12:16 |
| rcarrillocruz | dunno, wait for him to appear , he's usually on this channel | 12:17 |
| zhoragh | ok | 12:17 |
| *** Na3iL has quit IRC | 12:19 | |
| *** kushal has joined #openstack-infra | 12:19 | |
| *** javeriak has joined #openstack-infra | 12:20 | |
| *** eantyshev has quit IRC | 12:20 | |
| *** dprince has joined #openstack-infra | 12:20 | |
| *** javeriak_ has joined #openstack-infra | 12:21 | |
| *** markvoelker has quit IRC | 12:21 | |
| *** piet has joined #openstack-infra | 12:22 | |
| *** amitgandhinz has joined #openstack-infra | 12:22 | |
| *** markvoelker has joined #openstack-infra | 12:24 | |
| *** ddieterly is now known as ddieterly[away] | 12:25 | |
| timl0101 | the current shade release is 1.8.0. is it safe to assume the next feature release will be tagged 1.9.0? | 12:25 |
| *** javeriak has quit IRC | 12:25 | |
| yolanda | rcarrillocruz, i'm around | 12:26 |
| rcarrillocruz | timl0101: it depends if it will be a minor vs major release | 12:26 |
| rcarrillocruz | so no safe | 12:26 |
| rcarrillocruz | but mordred could confirm | 12:26 |
| timl0101 | thank you | 12:27 |
| *** amitgandhinz has quit IRC | 12:27 | |
| * Zara pokes around on the shiny dev server, thanks, fungi! | 12:28 | |
| *** jistr has joined #openstack-infra | 12:28 | |
| *** edmondsw has joined #openstack-infra | 12:28 | |
| rcarrillocruz | timl0101: are you asking to test the module version on the ansible group module you are working on by any chance | 12:28 |
| rcarrillocruz | ? | 12:28 |
| *** jistr is now known as jistr|bbl | 12:28 | |
| Zara | hm, 'no stories found' | 12:29 |
| *** kgiusti has joined #openstack-infra | 12:29 | |
| Zara | so it's a totally blank slate? cool | 12:29 |
| SpamapS | anybody around who can look into why the AFS mirror seems to be missing bits and pieces? | 12:30 |
| SpamapS | http://paste.openstack.org/show/497643/ | 12:30 |
| timl0101 | rcarrillocruz: yes. it occured to me I should explicitly document which version of shade is required. | 12:31 |
| rcarrillocruz | you're in luck , i asked the same thing to Shrews | 12:32 |
| rcarrillocruz | they are cool with non-exact version | 12:32 |
| rcarrillocruz | you can do something like https://github.com/ansible/ansible-modules-extras/pull/2240/files#diff-a96149ca617be9f96c2a8e447013f779R172 | 12:32 |
| rcarrillocruz | basicly, bailing out if shade is <= 1.8.0 | 12:32 |
| *** sdague has quit IRC | 12:32 | |
| timl0101 | thats perfect. thank you! | 12:32 |
| rcarrillocruz | np | 12:32 |
| *** Ravikiran_K has joined #openstack-infra | 12:33 | |
| *** weshay is now known as weshay_mt | 12:33 | |
| *** weshay_mt is now known as weshay_mtg | 12:33 | |
| krotscheck | mtreinish: Let me take alook | 12:34 |
| *** julim has joined #openstack-infra | 12:34 | |
| krotscheck | mtreinish: Hrm, looks like https://review.openstack.org/#/c/318279/ didn't get to the attention of enough infra cores to make the dib build deadline | 12:35 |
| *** oanson has quit IRC | 12:35 | |
| *** links has quit IRC | 12:36 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/devstack-gate: DO NOT REVIEW: This is for verifying CI with local.conf https://review.openstack.org/243545 | 12:36 |
| *** piet has quit IRC | 12:36 | |
| *** piet has joined #openstack-infra | 12:37 | |
| *** ccamacho has joined #openstack-infra | 12:37 | |
| *** banix has joined #openstack-infra | 12:39 | |
| *** _prad_ has joined #openstack-infra | 12:40 | |
| *** rhallisey has quit IRC | 12:41 | |
| *** markvoelker has quit IRC | 12:41 | |
| *** markvoelker has joined #openstack-infra | 12:42 | |
| *** amitgandhinz has joined #openstack-infra | 12:42 | |
| *** sdague has joined #openstack-infra | 12:43 | |
| *** rhallisey has joined #openstack-infra | 12:44 | |
| *** eharney has joined #openstack-infra | 12:45 | |
| *** markvoelker_ has joined #openstack-infra | 12:46 | |
| *** rlandy has quit IRC | 12:48 | |
| *** baoli has joined #openstack-infra | 12:48 | |
| *** ddieterly[away] is now known as ddieterly | 12:48 | |
| *** markvoelker_ has quit IRC | 12:48 | |
| *** exploreshaifali has quit IRC | 12:49 | |
| *** baoli_ has joined #openstack-infra | 12:49 | |
| *** rlandy has joined #openstack-infra | 12:49 | |
| *** markvoelker_ has joined #openstack-infra | 12:49 | |
| *** duobei has joined #openstack-infra | 12:50 | |
| *** markvoelker has quit IRC | 12:50 | |
| *** thiagop has joined #openstack-infra | 12:50 | |
| *** salv-orl_ has quit IRC | 12:50 | |
| thiagop | Hi guys. My instance of zuul is getting stuck on "looking for lost builds" pretty frequently | 12:51 |
| thiagop | restarting zuul works, but I cannot do that each and every day | 12:52 |
| thiagop | anybody has a clue on how to solve this? | 12:52 |
| *** baoli has quit IRC | 12:52 | |
| krotscheck | mtreinish: Digging through those logs, a lot of them are multiple hits in the same run, which is expected with a cold cache, large tarballs, and lots of mirrors. | 12:54 |
| *** markvoelker_ has quit IRC | 12:55 | |
| *** sridhar_ram_ has quit IRC | 12:55 | |
| *** markvoelker has joined #openstack-infra | 12:55 | |
| *** akshai has joined #openstack-infra | 12:56 | |
| *** ddieterly has quit IRC | 12:57 | |
| krotscheck | I'll keep an eye on it and see how things are impacted. | 12:58 |
| *** akshai has quit IRC | 12:58 | |
| krotscheck | thanks, fungi. | 12:59 |
| clarkb | SpamapS: I think that implies a bad bandernsatch run. we should only publish the volume if bandersnatch succeeds and that is not very succeedy | 13:00 |
| clarkb | when it is actually time to be awake I can look at it | 13:00 |
| openstackgerrit | Lukas Bednar proposed openstack-infra/gerritlib: bulk_query: added extra parameters https://review.openstack.org/235891 | 13:00 |
| *** nwkarsten has joined #openstack-infra | 13:02 | |
| rcarrillocruz | oh clarkb | 13:03 |
| *** akshai has joined #openstack-infra | 13:03 | |
| rcarrillocruz | wanted to chat with you about the cloud launcher | 13:03 |
| *** nwkarsten has quit IRC | 13:03 | |
| *** prometheanfire has quit IRC | 13:04 | |
| rcarrillocruz | can you folks use user-data when doing nova boot on RAX, and if so, would you be ok using it to boostrap puppet on newly created servers ? | 13:04 |
| *** nwkarsten has joined #openstack-infra | 13:04 | |
| vkramskikh | krotscheck: hi, it seems the new shrinkwrap format solved "module not found" issue: https://review.openstack.org/#/c/315119/ but it seems we ran again into the issue with partial JSON response from the mirror - almost all review requests to fuel-ui now get -1 :( i've posted an email to your npm mirror announcement thread | 13:04 |
| krotscheck | vkramskikh: Thanks! | 13:04 |
| clarkb | rcarrillocruz: what sort of user data? | 13:05 |
| rcarrillocruz | nova boot foo --user-data bootstrap_puppet.sh | 13:05 |
| rcarrillocruz | and the script would do essentially wget system-config/install_puppet.sh && install_puppet.sh | 13:06 |
| clarkb | rcarrillocruz: I think preference would be to use similar to current process instead of that | 13:06 |
| clarkb | because who knows how random clouds metadata is going to be set up | 13:06 |
| rcarrillocruz | thing is ansible cloud launcher only does provisioning, not bootstrapping. So in the case of keys, it gets messy and nasty to add a key to nova , create the server, then remove the key afterwards | 13:06 |
| clarkb | rcarrillocruz: eg just copy the file and run it yourself | 13:06 |
| openstackgerrit | Merged openstack-infra/project-config: Loosened npm cache timeout values. https://review.openstack.org/318279 | 13:06 |
| *** jamesmcarthur has joined #openstack-infra | 13:07 | |
| rcarrillocruz | so then i'm thinking of adding a pre-post section for resources in the cloud launcher 'dsl' | 13:07 |
| rcarrillocruz | so we can do a paramiko create key step | 13:07 |
| rcarrillocruz | then create the server with ansible | 13:07 |
| *** sdague has quit IRC | 13:07 | |
| rcarrillocruz | and a post step that removes the keys | 13:07 |
| clarkb | the issue with user data is glean doesnt support it | 13:07 |
| clarkb | thatd a cloud init thing | 13:08 |
| rcarrillocruz | i thought it was added to config drive ? | 13:08 |
| clarkb | but also metadata services are always funky | 13:08 |
| clarkb | rcarrillocruz: ya it would be on disk or on http server but nothing would run it | 13:08 |
| rcarrillocruz | ok, let me add the pre-post thing | 13:09 |
| clarkb | at that point you already need ssh access to fire the script | 13:09 |
| *** nwkarsten has quit IRC | 13:09 | |
| clarkb | whoch means you should just copy things yourself | 13:09 |
| rcarrillocruz | cos i belive that would also help solve the one-off tasks on servers | 13:09 |
| *** denisra has quit IRC | 13:09 | |
| rcarrillocruz | believe too | 13:09 |
| rcarrillocruz | thx | 13:09 |
| *** sdague has joined #openstack-infra | 13:10 | |
| zigo | clarkb: Any idea what's going on with the publish job? https://review.openstack.org/#/c/318500/1 | 13:11 |
| zigo | I don't have {site} anymore ... | 13:11 |
| *** jamesmcarthur has quit IRC | 13:11 | |
| openstackgerrit | Alexey Weyl proposed openstack-infra/project-config: Insert vitrage tempest to gate https://review.openstack.org/317851 | 13:12 |
| *** markvoelker_ has joined #openstack-infra | 13:12 | |
| *** markvoelker_ has quit IRC | 13:12 | |
| *** markvoelker_ has joined #openstack-infra | 13:12 | |
| clarkb | zigo: possibly the jenkins config updatr had not happened by the time that job ran? | 13:12 |
| clarkb | it is still very early morning for me so not really sure | 13:13 |
| krotscheck | What's that tool that tells us the size of used disk on our servers? | 13:13 |
| * krotscheck is having a daddy brain moment | 13:13 | |
| krotscheck | (The web UI) | 13:13 |
| zigo | clarkb: Sounds like it yes, because it's the same check job that failed the gating. | 13:13 |
| clarkb | krotscheck: cacti | 13:14 |
| krotscheck | clarkb: That's it, thanks! | 13:14 |
| SpamapS | clarkb: thanks | 13:14 |
| *** markvoelker has quit IRC | 13:15 | |
| *** nwkarsten has joined #openstack-infra | 13:16 | |
| *** markusry has quit IRC | 13:16 | |
| *** nadya has quit IRC | 13:16 | |
| *** sshnaidm has quit IRC | 13:18 | |
| *** Hal1 has joined #openstack-infra | 13:18 | |
| *** nwkarste_ has joined #openstack-infra | 13:18 | |
| *** duobei has quit IRC | 13:18 | |
| *** kushal has quit IRC | 13:19 | |
| *** nadya has joined #openstack-infra | 13:20 | |
| *** nwkarsten has quit IRC | 13:20 | |
| *** sshnaidm has joined #openstack-infra | 13:21 | |
| openstackgerrit | Alexey Weyl proposed openstack-infra/project-config: Insert vitrage tempest to gate https://review.openstack.org/317851 | 13:21 |
| *** nwkarsten has joined #openstack-infra | 13:22 | |
| *** nwkarste_ has quit IRC | 13:22 | |
| sdague | rax mirrors borked? | 13:23 |
| sdague | http://logs.openstack.org/59/318159/1/gate/gate-tempest-dsvm-cells/013ece6/logs/devstacklog.txt.gz#_2016-05-19_12_32_08_499 | 13:23 |
| *** jswarren has quit IRC | 13:24 | |
| krotscheck | Hrm. | 13:25 |
| krotscheck | Might be related to that npm thing vkramskikh mentioned ^^ | 13:25 |
| *** e0ne_ has joined #openstack-infra | 13:26 | |
| *** e0ne has quit IRC | 13:26 | |
| SpamapS | sdague: I had something similar in our internal jobs that use afs directly.. setuptools wheel missing | 13:26 |
| *** trown is now known as trown|brb | 13:26 | |
| SpamapS | sdague: clarkb said he'd look into a possible bandersnatch snafu | 13:26 |
| sdague | ok | 13:26 |
| SpamapS | a bandersnafu? | 13:26 |
| sdague | I like that word :) | 13:27 |
| vkramskikh | seems to be a similar issue - some http responses are not complete | 13:27 |
| SpamapS | sdague: would make a good race horse name | 13:27 |
| sdague | we've got a couple of gate fails right now over it | 13:27 |
| SpamapS | And here comes johnny on the rocks followed by brewster's millions and bandersnafu in the third position... | 13:27 |
| SpamapS | sounds like maybe checking latest bandersnatch runs needs to be bumped in priority | 13:28 |
| SpamapS | (also maybe checking AFS itself) | 13:28 |
| *** Julien-zte has joined #openstack-infra | 13:30 | |
| *** mriedem has joined #openstack-infra | 13:32 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Check openstack/openstack in rename process https://review.openstack.org/318665 | 13:32 |
| *** ddieterly has joined #openstack-infra | 13:32 | |
| *** mtanino has quit IRC | 13:33 | |
| *** kushal has joined #openstack-infra | 13:33 | |
| *** javeriak_ has quit IRC | 13:34 | |
| *** Julien-z_ has joined #openstack-infra | 13:34 | |
| *** Julien-zte has quit IRC | 13:34 | |
| krotscheck | Hrm hrm hrm. | 13:35 |
| phschwartz | asselin_: yolanda: There are a few bugs in puppet-openstackci I am running down and when I find them I will put up reviews. The 2 big ones I have found so far are around Apache SSL proxying not being setup correctly for jenkins and when doing a separate ndoe for zuul-merger, zuul is not installed so the zuul-merger process can't be started. | 13:35 |
| krotscheck | How would one even begin to triage that? | 13:35 |
| *** jtomasek_ has joined #openstack-infra | 13:35 | |
| jroll | we still don't support constraints in post jobs yet, right? | 13:36 |
| *** BigWillie has joined #openstack-infra | 13:37 | |
| openstackgerrit | Alexey Weyl proposed openstack-infra/project-config: Insert vitrage tempest to gate https://review.openstack.org/317851 | 13:37 |
| *** javeriak has joined #openstack-infra | 13:38 | |
| openstackgerrit | Lukas Bednar proposed openstack-infra/gerritlib: bulk_query: added extra parameters https://review.openstack.org/235891 | 13:39 |
| *** trown|brb is now known as trown | 13:41 | |
| *** Na3iL has joined #openstack-infra | 13:42 | |
| SpamapS | zigo: btw, is there anything blocking os-client-config 1.17.0 from going into unstable? https://tracker.debian.org/pkg/python-os-client-config? shade 1.8.0 has it as a lower bounds. | 13:43 |
| *** javeriak has quit IRC | 13:44 | |
| *** markvoelker_ has quit IRC | 13:44 | |
| zigo | SpamapS: upper-constrants.txt for Mitaka has os-client-config===1.16.0, so why do you think it'd be wise to upload an update to Sid? | 13:45 |
| *** ddieterly is now known as ddieterly[away] | 13:45 | |
| zigo | I could upload it to Experimental though ... | 13:45 |
| zigo | But I haven't really started working on the Newton branches. | 13:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Bump rate for osic to 0.5 https://review.openstack.org/318384 | 13:46 |
| *** BobBall_AWOL is now known as BobBall | 13:47 | |
| openstackgerrit | Ben Swartzlander proposed openstack-infra/project-config: Add bitrot jobs for Manila https://review.openstack.org/318676 | 13:47 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Import Panko https://review.openstack.org/318677 | 13:47 |
| *** piet has quit IRC | 13:48 | |
| krotscheck | fungi: What's our usual policy for dib build reruns? | 13:48 |
| krotscheck | As in, under what circumstances is it ok to do it? | 13:49 |
| *** prometheanfire has joined #openstack-infra | 13:49 | |
| pabelanger | krotscheck: it a bit of work to do it manually, if possible, it is usually easier to revert an image | 13:50 |
| pabelanger | just catching up on backscroll, is there an issue? | 13:50 |
| krotscheck | pabelanger: The npm cache relaxation patch only landed this morning, and missed last night's dib run | 13:50 |
| krotscheck | pabelanger: Also, we're seeing incomplete documents coming from our mirrors, which doesn't actually seem to be isolated to npm. | 13:51 |
| pabelanger | krotscheck: DIBs are actually building now. We start them at 10:35 UTC | 13:51 |
| pabelanger | krotscheck: which patch landed? | 13:52 |
| *** zhurong has joined #openstack-infra | 13:52 | |
| krotscheck | https://review.openstack.org/#/c/318279 | 13:52 |
| *** ddieterly[away] is now known as ddieterly | 13:52 | |
| *** akshai has quit IRC | 13:52 | |
| pabelanger | ubuntu-trusty should pick that up | 13:52 |
| krotscheck | pabelanger: We're on xenial :( | 13:53 |
| pabelanger | we missed it on centos-7 | 13:53 |
| pabelanger | krotscheck: xenial will catch it | 13:53 |
| pabelanger | that is the last DIB to build | 13:53 |
| krotscheck | Oh good! | 13:53 |
| *** markusry has joined #openstack-infra | 13:53 | |
| pabelanger | likely finished building at 1300UTC | 13:53 |
| pabelanger | then another few hours for an upload | 13:53 |
| pabelanger | just a waiting game until them | 13:54 |
| pabelanger | then* | 13:54 |
| krotscheck | vkramskikh ^^ - looks like the cache fix made it in. | 13:54 |
| vkramskikh | krotscheck: awesome, thanks | 13:54 |
| krotscheck | vkramskikh: I was able to get a clean install on fuel-ui by manually applying the changes in https://review.openstack.org/#/c/318279 - could you give it a try? | 13:55 |
| vkramskikh | sure | 13:55 |
| *** piet has joined #openstack-infra | 13:55 | |
| krotscheck | vkramskikh: You're so far away from our mirrors that natural net lag should hit you :) | 13:55 |
| vkramskikh | but it's not just me, this error happend a lot on jenkins slaves today: see https://review.openstack.org/#/c/318204/ | 13:56 |
| krotscheck | vkramskikh: Ok, let's try this then: https://review.openstack.org/318687 | 13:59 |
| *** esikachev has quit IRC | 13:59 | |
| *** jamesmcarthur has joined #openstack-infra | 14:00 | |
| *** jistr|bbl is now known as jistr | 14:00 | |
| *** akshai has joined #openstack-infra | 14:00 | |
| *** woodster_ has joined #openstack-infra | 14:02 | |
| *** trown is now known as jsitr | 14:02 | |
| *** jsitr is now known as trown | 14:02 | |
| *** esikachev has joined #openstack-infra | 14:03 | |
| *** _ari_ has joined #openstack-infra | 14:04 | |
| *** kien-ha has joined #openstack-infra | 14:05 | |
| krotscheck | vkramskikh: Why does fuel-ui have 500MB of node_module dependencies? | 14:07 |
| *** salv-orlando has joined #openstack-infra | 14:07 | |
| *** piet has quit IRC | 14:07 | |
| krotscheck | Wait, sorry- 250MB (500 on disk) | 14:07 |
| krotscheck | Still. 250 seems high? | 14:07 |
| vkramskikh | well, it's quite a large app :) | 14:07 |
| vkramskikh | and it uses npm also for its dependencies, not just devDependencies | 14:08 |
| *** Julien-z_ has quit IRC | 14:08 | |
| vkramskikh | after merge of shrinkwrap deduplication https://review.openstack.org/#/c/315119/ size of node_modules will go down | 14:09 |
| *** jamesmcarthur has quit IRC | 14:09 | |
| krotscheck | vkramskikh: Ah, right- forgot that hadn't merged yet. | 14:10 |
| openstackgerrit | Merged openstack-infra/project-config: Add gate-tempest-dsvm-lvm job to nova's experimental queue https://review.openstack.org/316298 | 14:11 |
| *** salv-orlando has quit IRC | 14:11 | |
| *** yamamoto has joined #openstack-infra | 14:12 | |
| *** jamesmcarthur has joined #openstack-infra | 14:13 | |
| *** slaweq has quit IRC | 14:13 | |
| *** rbrndt has joined #openstack-infra | 14:14 | |
| *** nwkarsten has quit IRC | 14:14 | |
| vkramskikh | krotscheck: even with deduplication it takes 201M on disk. well, that's what modern JS looks like :) | 14:15 |
| krotscheck | I just hope your shipped package ain't that big ;) | 14:16 |
| *** deadnull is now known as _deadnull | 14:17 | |
| *** zhurong has quit IRC | 14:18 | |
| vkramskikh | nope, it's about 300kB bundled with styles | 14:19 |
| fungi | krotscheck: sdague: any regions specifically where we're seeing incomplete documents returned, or is this everywhere? | 14:19 |
| krotscheck | fungi: I saw iad and dfw, and a couple of others. | 14:19 |
| krotscheck | So, probably not? | 14:20 |
| fungi | sdague's example was in rax ord | 14:20 |
| *** _amrith_ is now known as amrith | 14:20 | |
| fungi | okay | 14:20 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Make the docs-draft build point to storyboard-dev https://review.openstack.org/318702 | 14:20 |
| eharney | i'm seeing an odd "Invalid authentication method." message in gerrit when clicking "Add Me" on a patchset: https://i.imgur.com/dOgcKPl.png | 14:20 |
| jeblair | SpamapS, clarkb: re http://paste.openstack.org/show/497643/ i see http://paste.openstack.org/show/497720/ | 14:20 |
| mordred | rcarrillocruz: I agree with clarkb - it is specifically desirable to not use user-data | 14:21 |
| mordred | we have the ability to get ssh keys onto the machines, we have the abilty to run ansible -nothing needs to be bootstrapped more complicatedly than that | 14:21 |
| rcarrillocruz | yep, working on extending the role to accept pre/post actions by resource | 14:21 |
| mordred | rcarrillocruz: woot! | 14:22 |
| *** nwkarsten has joined #openstack-infra | 14:22 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update CVS Plugin to use convert_mapping_to_xml https://review.openstack.org/318706 | 14:23 |
| jeblair | clarkb, krotscheck, sdague: http://paste.openstack.org/show/497721/ | 14:23 |
| *** rossella_s has quit IRC | 14:23 | |
| fungi | sdague: so, just confirmed, at the moment logged into mirror.ord.rax.o.o i'm able to unzip lxml-3.6.0-cp27-none-linux_x86_64.whl with no problem, so the backing file seems to be | 14:24 |
| *** rossella_s has joined #openstack-infra | 14:24 | |
| jeblair | sdague, fungi: i double confirm that from both a direct afs read on my workstation as well as a wget from mirror.ord; the md5sums are the same | 14:25 |
| *** nwkarsten has quit IRC | 14:25 | |
| fungi | sdague: similarly i retrieved lxml-3.6.0-cp27-none-linux_x86_64.whl through apache with wget and it seems fine | 14:25 |
| fungi | so this must be intermittent | 14:25 |
| *** nwkarsten has joined #openstack-infra | 14:25 | |
| jeblair | sdague, fungi: hypothesis -- rax ord network gremlin caused the file transfer to be aborted and that's how pip decided to tell us? | 14:25 |
| *** _deadnull is now known as deadnull | 14:26 | |
| krotscheck | Does apache unceremoniously terminate outgoing requests if they take too long? | 14:26 |
| *** deadnull is now known as _deadnull | 14:27 | |
| jeblair | krotscheck: i'm not sure, but that one took ~1second according to the console log | 14:27 |
| vkramskikh | fresh failure on deduped shrinkwrap: http://logs.openstack.org/19/315119/6/check/gate-fuel-ui-npm-run-lint/2700f62/console.html | 14:27 |
| vkramskikh | shasum check failed for /tmp/npm-7468-89af237a/mirror.ord.rax.openstack.org/npm/webpack/-/webpack-1.12.9.tgz | 14:27 |
| *** _deadnull is now known as deadnull | 14:27 | |
| jeblair | also ord ^ | 14:27 |
| jeblair | (and i get the 'actual' sha1sum when i wget it) | 14:28 |
| *** fawadkhaliq has joined #openstack-infra | 14:29 | |
| *** hongbin has joined #openstack-infra | 14:29 | |
| *** whoops has joined #openstack-infra | 14:29 | |
| krotscheck | That's a pretty large json file :/ | 14:30 |
| *** mtanino has joined #openstack-infra | 14:31 | |
| vkramskikh | https://review.openstack.org/#/c/318687/ cache timeout request also has -1 from Jenkins, but due to another issue - module not found failure. i'll try to rebase it on the shrinkwrap request and see if it helps | 14:31 |
| openstackgerrit | Paul Belanger proposed openstack-infra/devstack-gate: Use local AFS mirror for network_sanity_check https://review.openstack.org/318714 | 14:33 |
| *** fawadkhaliq has quit IRC | 14:33 | |
| *** zhoragh has quit IRC | 14:34 | |
| *** zeih has quit IRC | 14:36 | |
| fungi | i've wondered for a while whether rackspace specifically has some sort of http monkeying going on in their networks (so-called "web accelerators" which can lead to very confusing failure modes for http traffic when they misbehave or get overtaxed) | 14:36 |
| krotscheck | fungi: I'm guessing there's a pattern then? | 14:37 |
| jroll | fungi: we don't have anything like that, though I'm not sure about specific endpoints like the mirrors | 14:37 |
| *** banix has quit IRC | 14:37 | |
| *** dtardivel has joined #openstack-infra | 14:38 | |
| jroll | oh, these are openstack's mirrors, there shouldn't be anything in the middle then above L3 | 14:38 |
| *** kushal has quit IRC | 14:39 | |
| fungi | jroll: right, it's just apache for us | 14:41 |
| fungi | so more likely is that it's simply the random network issues we've been witnessing for months in ord | 14:41 |
| *** cloudtrainme has joined #openstack-infra | 14:42 | |
| *** piet has joined #openstack-infra | 14:42 | |
| fungi | which rackspace's noc apparently can't track down without me managing to not delete a server instance exhibiting the problem behavior | 14:42 |
| *** yamahata has joined #openstack-infra | 14:43 | |
| * fungi rechecks his fishing change again to see if he gets a bite this time | 14:43 | |
| jeblair | fungi: one thing we could do is to hack nodepool to not delete any nodes that fail certain jobs in ord | 14:43 |
| *** sdake_ has joined #openstack-infra | 14:44 | |
| jroll | fungi: yeah, I guess they probably assume it's hypervisor-layer | 14:44 |
| fungi | yeah, if 315762 doesn't pan out, that's what i was considering next | 14:44 |
| jeblair | fungi: probably lots of false positives (negatives? whatevers).... | 14:44 |
| zxiiro | electrofelix: those patches look fine to me. SHould we ping some folks to review them? I'm under some pressure to get the features from 1.6 available on my project. | 14:45 |
| *** yamamoto has quit IRC | 14:45 | |
| fungi | jroll: anecdotal evidence is making me think that _when_ there is particularly heavy instance boot/delete churn, then within the first few seconds an instance is online sometimes it's unable to reach/unreachable from parts of rackspace's networks | 14:45 |
| jroll | fungi: interesting | 14:46 |
| fungi | jroll: like maybe reassigned l2 address lingering on some other part of the switching infrastructure maybe | 14:46 |
| fungi | which corrects itself as bridge table updates propagate, but by then we've failed the job | 14:47 |
| krotscheck | I'm sure there's english there somewhere... | 14:47 |
| jroll | fungi: hm, shouldn't be, we give IPs 15 minutes before re-assigning them, though I could totally see something getting stuck in the L3 controllers | 14:47 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:48 | |
| zaro | morning | 14:48 |
| fungi | jroll: what about macs though? are they reused? | 14:48 |
| zaro | zxiiro: you need help with jjb? | 14:48 |
| *** jlanoux has quit IRC | 14:48 | |
| jroll | fungi: in theory, yes, though I think the pools are larger than we'll ever hit | 14:49 |
| fungi | jroll: or, if randomly generated, what the chances of collision are? | 14:49 |
| *** salv-orlando has joined #openstack-infra | 14:49 | |
| *** tonytan4ever has joined #openstack-infra | 14:49 | |
| jroll | fungi: they're sequential and the db enforces uniqueness | 14:49 |
| pabelanger | clarkb: fungi: jeblair: I'm going a test with OSIC right now, setting the rate to 0.5. If you'd like to watch http://grafana.openstack.org/dashboard/db/nodepool-osic for the results | 14:49 |
| zxiiro | zaro: electrofelix needs these patches reviewed https://review.openstack.org/309734 and https://review.openstack.org/314789 and I just need the latest JJB to be released as there's a feature we need. | 14:49 |
| *** jamesmcarthur has quit IRC | 14:49 | |
| pabelanger | s/going/doing | 14:49 |
| fungi | also, i have no idea what's special about ord in particular that we don't really see this happen in other regions | 14:49 |
| jroll | fungi: this is basically the limits of my knowledge, though, maybe I could bug one of the data plane people | 14:50 |
| jroll | sure | 14:50 |
| fungi | but it's unfortunate that ord is also where they've decided to stick the bulk of our quota | 14:50 |
| pabelanger | the time to ready has increased like jeblair says, but we haven't leaked a FIP yet | 14:50 |
| jroll | fungi: have you seen evidence that the network comes up eventually? or do we kill things before that? | 14:51 |
| krotscheck | Well, the cache timeout didn't work: http://logs.openstack.org/87/318687/2/check/gate-fuel-ui-npm-run-lint/98bef6d/console.html | 14:51 |
| krotscheck | Partially terminated json response. | 14:51 |
| fungi | jroll: so far we kill things too soon to know, which is one reason i'm trying to catch one in the act | 14:51 |
| krotscheck | And it looks fine in AFS :/ | 14:51 |
| jroll | fungi: okay, just was going to state that as data if you had evidence | 14:52 |
| *** nwkarste_ has joined #openstack-infra | 14:52 | |
| fungi | jroll: right, no actual evidence other than some of our jobs have built-in retries for certain network operations and so we see these sorts of failures happen on succeeding jobs too | 14:52 |
| fungi | just less often, for obvious reasons | 14:53 |
| jroll | fungi: ok, cool, I'll poke the sdn team | 14:53 |
| *** fesp has joined #openstack-infra | 14:53 | |
| *** mixos has joined #openstack-infra | 14:54 | |
| fungi | jroll: and since most of our evidence is discovered post-mortem through job log analysis, we don't usually know about a hit until hours later | 14:54 |
| jroll | fungi: right | 14:54 |
| *** weshay_mtg is now known as weshay | 14:54 | |
| fungi | making catching a live specimen a particularly fun exercise | 14:54 |
| *** nwkarsten has quit IRC | 14:55 | |
| *** dprince has quit IRC | 14:56 | |
| *** julim has quit IRC | 14:56 | |
| *** dprince has joined #openstack-infra | 14:56 | |
| *** julim has joined #openstack-infra | 14:56 | |
| fungi | the tack i've taken for now is a trivial patch to devstack-gate which tries to ping git.openstack.org and short-circuits to passing if the ping succeeds but busy-waits until the job timeout if it fails | 14:57 |
| fungi | and then i recheck that change and watch to see if it has a d-g-based job running more than a minute or so | 14:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Several fixes to zuul launcher https://review.openstack.org/318397 | 14:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: handle Ant paths in publishers https://review.openstack.org/318411 | 14:57 |
| zaro | zxiiro: will take a look | 14:58 |
| fungi | i may switch from ping to wget or something if this doesn't pan out, in case icmp isn't being impacted the way http is | 14:58 |
| *** mriedem is now known as mriedem_meeting | 14:59 | |
| krotscheck | Does AFS return "current" vs "total" document size when reading from local disK? | 14:59 |
| *** jpr has joined #openstack-infra | 14:59 | |
| *** jaosorior has quit IRC | 14:59 | |
| jeblair | krotscheck: generally afs flushes on close, so usually you only see complete documents written | 15:00 |
| *** jaosorior has joined #openstack-infra | 15:00 | |
| krotscheck | I wonder if the reason the JSON docs from the npm mirror are terminated is because the http content-length header is read while there's only a partial doc availabele... | 15:00 |
| krotscheck | Hrm, gotcha. | 15:00 |
| jeblair | krotscheck: (that is, if a remote system is writing and you are reading locally; if you are reading and writing locally, you would probably see the in-progress state) | 15:00 |
| *** ddieterly is now known as ddieterly[away] | 15:01 | |
| *** ddieterly[away] is now known as ddieterly | 15:01 | |
| jeblair | krotscheck: and even so, with the atomic volume release, everything should be stable when it's updated | 15:01 |
| *** bhunter71 has joined #openstack-infra | 15:02 | |
| *** jaosorior has quit IRC | 15:03 | |
| *** jaosorior has joined #openstack-infra | 15:03 | |
| krotscheck | So reading the file size on disk of something that's in the process of being cached is unlikely to be the cause, got it. | 15:04 |
| * krotscheck is trying every angle. | 15:04 | |
| *** e0ne_ has quit IRC | 15:04 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Fix date format for agg_count and get_test_counts_in_date_range() https://review.openstack.org/316948 | 15:06 |
| *** shardy has quit IRC | 15:07 | |
| electrofelix | zaro: I'll quickly fix the typo zxiiro spotted in https://review.openstack.org/314789 | 15:09 |
| *** jaosorior has quit IRC | 15:10 | |
| *** doug-fish has joined #openstack-infra | 15:10 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Prevent corruption of cache by using atomic update https://review.openstack.org/314789 | 15:10 |
| *** claudiub has quit IRC | 15:10 | |
| kashyap | A small request: Anyone with access to a Xenial node, do you mind running `virsh capabilities` command and post the output to a pastebin? | 15:10 |
| *** links has joined #openstack-infra | 15:11 | |
| *** zz_dimtruck is now known as dimtruck | 15:12 | |
| *** doug-fish has quit IRC | 15:12 | |
| *** mrmartin has joined #openstack-infra | 15:12 | |
| fungi | sudo: virsh: command not found | 15:13 |
| jeblair | fungi: do you have a script to watch your job for you? | 15:13 |
| fungi | i assume it's not a standard part of the operating system? | 15:13 |
| *** heyongli has quit IRC | 15:13 | |
| *** alex_xu has quit IRC | 15:13 | |
| fungi | jeblair: good idea. i have a script i use to parse the zuul status.json into a console-friendly rendering, and could adapt that pretty easily | 15:14 |
| jeblair | fungi: here's a present for you: http://paste.openstack.org/show/497742/ | 15:14 |
| fungi | ooh, thanks! | 15:14 |
| jeblair | (uses the direct change url, so should be super efficient) | 15:14 |
| *** kien-ha has quit IRC | 15:14 | |
| fungi | oh, right. i even forgot we had those since we haven't hooked that back into hideci.js | 15:15 |
| fungi | my memory's fuzzy now, but i guess we were putting too much load on zuul with all the added status requests the last time we tried it? | 15:15 |
| *** pt_15 has quit IRC | 15:16 | |
| kashyap | clarkb: You about by any chance? When you are, mind looking at my comment above? It is for fixing the live migration jobs. | 15:16 |
| jeblair | fungi: yeah, i don't recall what happened | 15:16 |
| clarkb | kashyap: fungi responded | 15:16 |
| kashyap | clarkb: Ah, sorry - missed it fungi | 15:16 |
| fungi | kashyap: in a sec i'll get it installed and find out | 15:17 |
| kashyap | fungi: Oh, is it possible to install libvirt and associated client packages on that machine? | 15:17 |
| kashyap | fungi: Many thanks! | 15:17 |
| jeblair | fungi: oh, just realized the node name would be handy (so you can go directly to nodepool and hold it; updated script: http://paste.openstack.org/show/497743/ | 15:18 |
| *** jamesmcarthur has joined #openstack-infra | 15:18 | |
| * kashyap back in a min | 15:18 | |
| *** mrmartin has quit IRC | 15:18 | |
| fungi | jeblair: even better, yep! double-thanks | 15:19 |
| *** jtomasek_ has quit IRC | 15:19 | |
| *** itisha has quit IRC | 15:19 | |
| *** doug-fish has joined #openstack-infra | 15:20 | |
| *** jlanoux has joined #openstack-infra | 15:20 | |
| *** yamahata has quit IRC | 15:20 | |
| *** jamesmcarthur has quit IRC | 15:23 | |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: net-generic-switch: Do not create Neutron debug probes https://review.openstack.org/318739 | 15:24 |
| *** fawadkhaliq has joined #openstack-infra | 15:24 | |
| jeblair | fungi: oh, oops, it looks like we only get the node name on job completion, sorry | 15:25 |
| *** esikachev has quit IRC | 15:25 | |
| fungi | jeblair: yeah, i have a zuul debug log grep already written up to get me that anyway | 15:26 |
| fungi | so not a big deal | 15:26 |
| jeblair | cool. so the earlier script is probably better then | 15:26 |
| *** links has quit IRC | 15:26 | |
| *** akshai has quit IRC | 15:27 | |
| fungi | kashyap: happen to know what the package names i need to install to get a working virsh are? | 15:27 |
| fungi | it's not a tool i've ever tried to use before | 15:27 |
| kashyap | fungi: Hmm, good question, I know it by heart on Fedora, looking up on Ubuntu. 1 sec | 15:28 |
| *** claudiub has joined #openstack-infra | 15:28 | |
| kashyap | fungi: packages.ubuntu.com/xenial/libvirt-bin | 15:28 |
| fungi | the package doesn't seem to be named "virsh" or "libvirt" | 15:28 |
| fungi | aha | 15:28 |
| *** andreas_s has quit IRC | 15:29 | |
| kashyap | fungi: On Fedora, the `virsh` program is provided by 'libvirt-client' RPM package | 15:30 |
| kashyap | So, trying to find similar for Ubuntu | 15:30 |
| kashyap | fungi: `sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder` | 15:31 |
| openstackgerrit | Paul Belanger proposed openstack-infra/tripleo-ci: Delete more things to make room for centos-7 https://review.openstack.org/318743 | 15:31 |
| *** dmk0202 has quit IRC | 15:33 | |
| fungi | kashyap: http://paste.openstack.org/show/497747 | 15:33 |
| *** claudiub has quit IRC | 15:33 | |
| *** hashar has quit IRC | 15:33 | |
| fungi | that's from an ubuntu-xenial instance booted in rackspace ord from our current diskimage-builder created image | 15:33 |
| kashyap | fungi: Thank you | 15:34 |
| *** akshai has joined #openstack-infra | 15:34 | |
| *** savihou has quit IRC | 15:34 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add optional start_date parameter to get_recent*() methods https://review.openstack.org/307416 | 15:35 |
| *** savihou has joined #openstack-infra | 15:35 | |
| fungi | kashyap: i guess it's <arch>x86_64</arch> <model>qemu32</model> you were curious about? | 15:35 |
| kashyap | fungi: The problem is: On Xenial nodes, 'gate64' CPU model (defined here: https://review.openstack.org/#/c/168407/9/tools/cpu_map_update.py) is not being detected. | 15:36 |
| kashyap | fungi: So, trying to see which CPU flags 'Xenial' nodes are not exposing -- thus causing the error: | 15:36 |
| fungi | that's on a linux 4.4.0 kernel with libvirt 1.3.1 | 15:36 |
| kashyap | "[...] 2016-05-13T21:33:07.143245Z qemu-system-x86_64: Unable to find CPU definition: gate64" | 15:37 |
| fungi | "gate64" sounds like something specially created for devstack? | 15:37 |
| *** asettle has quit IRC | 15:37 | |
| kashyap | fungi: Yes, that model was created year ago to make live migration work in the Gate | 15:37 |
| *** vincentll has quit IRC | 15:37 | |
| *** akaszuba has quit IRC | 15:38 | |
| clarkb | fungi: yes we dont have heterogenous cpus so devstack tries to define a minimal x86_64 cpu model that allows live migration to work | 15:38 |
| fungi | kashyap: i'm curious why you think virsh capabilities would show it then if it's set up by devstack | 15:38 |
| clarkb | er we have heterogenous cpus | 15:38 |
| *** deadnull is now known as _deadnull | 15:38 | |
| kashyap | fungi: "You'll need to look at the `virsh capabilities` output to see what Nova is seeing, and identify the missing CPU flag(s)." | 15:39 |
| kashyap | The CI needs to be configured to cope with whatever the nodes expose. | 15:39 |
| *** bdemers has quit IRC | 15:39 | |
| clarkb | kashyap: it is though | 15:39 |
| *** psachin has quit IRC | 15:39 | |
| fungi | kashyap: wouldn't you need to boot a xenial instance and set up devstack on it to investigate this? | 15:39 |
| kashyap | (From conversation with danpb, libvirt dev on #openstack-nova) | 15:39 |
| clarkb | its not like those flags change across libvirt versons or distro releases right? | 15:39 |
| kashyap | clarkb: Let me check with danpb on -nova who can tell for certain | 15:40 |
| *** Swami has joined #openstack-infra | 15:40 | |
| clarkb | I think it is more likely that either the file location has changed or we dont restart libvirt in a way to pick up the updates with new libvirt | 15:40 |
| fungi | kashyap: as far as i know this is a cpu definition devstack creates. it doesn't exist in the wild on our ci server instances | 15:40 |
| clarkb | fungi: correct. And on truaty it works fine | 15:40 |
| *** dprince has quit IRC | 15:40 | |
| clarkb | and since cpu flags as exposed by kernel should be stable I dont expect the change from truaty to xenial breaking is due to flags chamging | 15:41 |
| kashyap | fungi: Yes, it's (that model) what DevStack creates | 15:41 |
| *** dprince has joined #openstack-infra | 15:41 | |
| kashyap | clarkb: We can talk with danpb on #openstack-nova, if you have a moment. Can you join there? | 15:42 |
| clarkb | sure | 15:42 |
| *** Na3iL has quit IRC | 15:45 | |
| *** bdemers has joined #openstack-infra | 15:46 | |
| *** yamamoto has joined #openstack-infra | 15:46 | |
| *** vhosakot has joined #openstack-infra | 15:47 | |
| waynr | electrofelix: yo just wanted to update you on rebase progress | 15:47 |
| openstackgerrit | Paul Belanger proposed openstack-infra/tripleo-ci: Delete more things to make room for centos-7 https://review.openstack.org/318743 | 15:47 |
| waynr | i think i've got around 15 patches in the line of work I did back in January, the first 5 or so i am expecting to be the most difficult and last night I plowed my way through the first three (making sure that both the rebase is clean and tests pass for each commit) | 15:48 |
| waynr | i'm planning to finish the entire rebase before pushing the new patchsets up for review on gerrit | 15:49 |
| pabelanger | krotscheck: DIB uploads are happening now, process usually takes about 4-5 hours to complete | 15:51 |
| *** IlyaG has joined #openstack-infra | 15:51 | |
| krotscheck | pabelanger: Sweet | 15:51 |
| waynr | i'm also planning on sending out an email to openstack-infra around the time i push the new patchsets up explaining how the commits build on each other and that i'm hoping deficiencies in the earlier commits can be forgiven where those concerns have been addressed later in the commit series | 15:52 |
| *** mikelk has quit IRC | 15:54 | |
| *** Qiming has quit IRC | 15:54 | |
| *** bpokorny has joined #openstack-infra | 15:57 | |
| *** yamamoto has quit IRC | 15:57 | |
| *** arxcruz has quit IRC | 15:57 | |
| *** dprince has quit IRC | 15:58 | |
| krotscheck | Argh, I cannot figure out why apache serving files form AFS would terminate partially. | 15:59 |
| *** armax has joined #openstack-infra | 15:59 | |
| *** dprince has joined #openstack-infra | 15:59 | |
| *** nadya has quit IRC | 16:00 | |
| *** gomarivera has joined #openstack-infra | 16:00 | |
| *** mriedem_meeting is now known as mriedem | 16:00 | |
| *** sdake_ is now known as sdake | 16:01 | |
| * krotscheck feels like he's playing whackamole. | 16:01 | |
| *** lakshmiS has joined #openstack-infra | 16:01 | |
| *** xarses_ is now known as xarses | 16:01 | |
| electrofelix | waynr: sounds good | 16:02 |
| *** julim has quit IRC | 16:02 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Add timeout to get_host_info task https://review.openstack.org/317937 | 16:03 |
| *** marcusvrn_ has quit IRC | 16:03 | |
| *** nadya has joined #openstack-infra | 16:03 | |
| *** rcernin has quit IRC | 16:04 | |
| *** bpokorny has quit IRC | 16:05 | |
| *** mriedem is now known as mriedem_afk | 16:06 | |
| *** esikachev has joined #openstack-infra | 16:09 | |
| *** fawadkhaliq has quit IRC | 16:11 | |
| *** nwkarste_ has quit IRC | 16:11 | |
| *** nadya has quit IRC | 16:13 | |
| *** bpokorny has joined #openstack-infra | 16:14 | |
| cloudnull | pabelanger mordred jeblair -- do you know off hand where the code lives that calls shade to build VMs? | 16:14 |
| clarkb | cloudnull: openstack-infra/nodepool is the project and service using shade to boot VMs | 16:15 |
| cloudnull | ok. | 16:15 |
| cloudnull | will look there. | 16:15 |
| *** dprince has quit IRC | 16:16 | |
| *** claudiub has joined #openstack-infra | 16:16 | |
| *** jpr has quit IRC | 16:18 | |
| *** nwkarsten has joined #openstack-infra | 16:19 | |
| *** dtantsur is now known as dtantsur|afk | 16:21 | |
| *** jswarren has joined #openstack-infra | 16:21 | |
| *** yamahata has joined #openstack-infra | 16:23 | |
| *** ewilson has joined #openstack-infra | 16:24 | |
| *** jcoufal has quit IRC | 16:27 | |
| *** tonytan4ever has quit IRC | 16:28 | |
| *** fesp has quit IRC | 16:29 | |
| *** Apoorva has joined #openstack-infra | 16:29 | |
| *** piet has quit IRC | 16:30 | |
| *** piet has joined #openstack-infra | 16:31 | |
| krotscheck | ARGH. Build fails. I try running the same patch against that mirror. Mine passes. | 16:32 |
| fungi | krotscheck: got a link to the failing log? curious which provider/region it was in | 16:32 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Move oslo.concurrency to test-requirements https://review.openstack.org/318771 | 16:32 |
| *** tonytan4ever has joined #openstack-infra | 16:33 | |
| *** gomarivera has quit IRC | 16:33 | |
| krotscheck | fungi: This one's dfw-> http://logs.openstack.org/87/318687/1/check/gate-fuel-ui-npm-run-lint/dcc7953/console.html | 16:33 |
| krotscheck | fungi: This one's r1 bluebox -> http://logs.openstack.org/87/318687/2/check/gate-fuel-ui-npm-run-lint/98bef6d/console.html | 16:33 |
| krotscheck | fungi: This one's IAD http://logs.openstack.org/87/318687/2/check/gate-fuel-ui-npm-run-lint/4335192/console.html | 16:34 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Move oslo.concurrency to test-requirements https://review.openstack.org/318771 | 16:34 |
| *** regXboi has joined #openstack-infra | 16:35 | |
| *** dprince has joined #openstack-infra | 16:36 | |
| *** kzaitsev_mb has quit IRC | 16:36 | |
| krotscheck | fungi: Furthermore, it's always on different packages. I _know_ it's an AFS cache issue, I just don't have the expertise to fix it. | 16:36 |
| krotscheck | fungi: The annoying part is that the patch I'm running with already has the timeout relaxation bits applied | 16:37 |
| *** Na3iL has joined #openstack-infra | 16:37 | |
| *** markusry has quit IRC | 16:37 | |
| *** savihou has quit IRC | 16:38 | |
| *** jordanP has quit IRC | 16:38 | |
| *** kzaitsev_mb has joined #openstack-infra | 16:40 | |
| *** akshai has quit IRC | 16:40 | |
| SpamapS | jeblair: my afs still shows no files in that dir | 16:40 |
| *** johnny__ has joined #openstack-infra | 16:40 | |
| SpamapS | jeblair: is there some way to invalidate the cache maybe? | 16:41 |
| *** ifarkas has quit IRC | 16:41 | |
| fungi | we're definitely pushing a lot of kernel cache memory on the afs servers and clients, though local afs cache on, for example, bluebox is only 6gb so far | 16:42 |
| jeblair | SpamapS: you could try restarting the cache manager (service opensafs-client restart) | 16:42 |
| *** esikachev has quit IRC | 16:42 | |
| jeblair | krotscheck: oh, do you think we're timing out on the first hit to a file? | 16:43 |
| fungi | jeblair: oddly it looks like we get part of the file and it's truncated, so maybe timing out on specific blocks? is afs access block-oriented? | 16:43 |
| krotscheck | jeblair: I think so, at least that's as much as I can tell. | 16:44 |
| jeblair | might be worth checking the apache logs for those requests | 16:44 |
| fungi | giving that a shot now | 16:45 |
| SpamapS | jeblair: already tried that. | 16:45 |
| SpamapS | will try again | 16:45 |
| *** fawadkhaliq has joined #openstack-infra | 16:46 | |
| SpamapS | jeblair: we had a full disk on that box yesterday. Is it possible the cache got screwed up by that? | 16:46 |
| krotscheck | jeblair: Good call, diggin | 16:46 |
| jeblair | SpamapS: oh, maybe. you could stop, fully remove /var/cache/openafs, then restart. | 16:47 |
| SpamapS | jeblair: to be clear, I want to remove the contents, not the dir, right? | 16:47 |
| jeblair | SpamapS: probably baste to remove the contents; not sure if the dir will be recreated by the initscript/daemon or if it's from the package. | 16:48 |
| jeblair | er best | 16:48 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Prevent corruption of cache by using atomic update https://review.openstack.org/314789 | 16:48 |
| fungi | i think the stale cache theory _might_ be debunked | 16:48 |
| electrofelix | zaro: that was caused by a rebase mistake | 16:48 |
| fungi | i have this in the log | 16:48 |
| *** earlephilhower has joined #openstack-infra | 16:48 | |
| jeblair | SpamapS: could also baste if it's not moist | 16:48 |
| *** shashank_hegde has joined #openstack-infra | 16:48 | |
| fungi | 169.45.71.79 - - [18/May/2016:12:35:36 +0000] "GET /npm/postcss-loader HTTP/1.1" 301 584 "install" "npm/3.5.2 node/v4.2.6 linux x64" | 16:49 |
| SpamapS | jeblair: that worked | 16:49 |
| electrofelix | though I'm stumped as to why it didn't cause something to fail completely | 16:49 |
| fungi | that's a 26 hours before the failure on that get at http://logs.openstack.org/87/318687/2/check/gate-fuel-ui-npm-run-lint/98bef6d/console.html | 16:50 |
| SpamapS | clarkb: FYI, the afs problem we had may have been unrelated to any bandersnatch issues. AFS cache clear seems to have resolved it | 16:50 |
| zxiiro | me too. that looks like a failure to me | 16:50 |
| zxiiro | but the job passed it | 16:50 |
| clarkb | SpamapS: ya jeblair's above pastes seemed to indicate it was owrking as expected | 16:50 |
| *** tqtran has joined #openstack-infra | 16:50 | |
| *** doug-fish has quit IRC | 16:50 | |
| *** doug-fish has joined #openstack-infra | 16:51 | |
| jeblair | fungi: that looks like a redirect -- i'm guessing to /npm/postcss-loader/ ? | 16:51 |
| fungi | yeah, i'm getting some better analysis looking for tarballs | 16:52 |
| *** sambetts is now known as sambetts|afk | 16:52 | |
| * krotscheck is still trying to figure out kibana | 16:53 | |
| *** mriedem has joined #openstack-infra | 16:53 | |
| fungi | in the current access log in bluebox we have 6 hits (over about 48 hours) for /npm/postcss-loader/-/postcss-loader-0.5.1.tgz all of which are 200 ok with reported length 3878 | 16:54 |
| *** shashank_hegde has quit IRC | 16:54 | |
| *** mriedem_afk has quit IRC | 16:54 | |
| fungi | no entries for it in the error log either | 16:54 |
| openstackgerrit | Pedro Alvarez Piedehierro proposed openstack-infra/storyboard-webclient: Close card-details modal when clicking on the Story link https://review.openstack.org/318779 | 16:54 |
| *** doug-fis_ has joined #openstack-infra | 16:54 | |
| jeblair | fungi, krotscheck: but is that error for the json at /npm/postcss-loader/ ; not the actual tarball? | 16:55 |
| *** electrofelix has quit IRC | 16:55 | |
| *** bpokorny has quit IRC | 16:55 | |
| fungi | ahh, yeah krotscheck needs to school me on what file that really is which got truncated | 16:55 |
| *** doug-fish has quit IRC | 16:55 | |
| *** mriedem1 has joined #openstack-infra | 16:55 | |
| krotscheck | fungi: both types, just different error codes for each. | 16:55 |
| *** derekh has quit IRC | 16:55 | |
| krotscheck | fungi: For the one you're currently investigating, it should be either http://mirror.regionone.bluebox-sjc1.openstack.org/npm/postcss-loader/ or http://mirror.regionone.bluebox-sjc1.openstack.org/npm/postcss-loader/index.json | 16:56 |
| jeblair | i see http://paste.openstack.org/show/497760/ | 16:56 |
| jeblair | i think 169.45.71.75 - - [19/May/2016:14:42:48 +0000] "GET /npm/postcss-loader/ HTTP/1.1" 200 36171 "http://mirror.regionone.bluebox-sjc1.openstack.org/npm/postcss-loader/" "npm/3.5.2 node/v4.2.6 linux x64" | 16:57 |
| jeblair | is the one for the job | 16:57 |
| jeblair | and it is suspicously small. | 16:57 |
| fungi | when i look at hits to /npm/postcss-loader/ there are varying sizes returned | 16:57 |
| jeblair | the one after is also a different size, but i suspect that may be due to client differences (different headers?) since it's being requested from a different platform/npm version | 16:58 |
| *** yamahata has quit IRC | 16:58 | |
| *** r-daneel has joined #openstack-infra | 16:58 | |
| *** piet has quit IRC | 16:58 | |
| SpamapS | jeblair: also I just basted my afs mirror and, frankly, it's a bit soupy now | 16:58 |
| *** mtanino has quit IRC | 16:58 | |
| *** yamahata has joined #openstack-infra | 16:58 | |
| fungi | yeah, have a feeling this is compression | 16:59 |
| *** trown is now known as trown|lunch | 16:59 | |
| *** doug-fis_ has quit IRC | 16:59 | |
| fungi | note the browser string | 16:59 |
| fungi | 188.164.131.220 - - [19/May/2016:16:56:54 +0000] "HEAD /npm/postcss-loader/ HTTP/1.1" 200 234 "-" "IrssiUrlLog/0.2" | 16:59 |
| *** mriedem has quit IRC | 16:59 | |
| jeblair | SpamapS: it'll set up once the cache repopulates | 16:59 |
| fungi | oh, those are head not get | 16:59 |
| *** esikachev has joined #openstack-infra | 16:59 | |
| fungi | filter failure on my part | 16:59 |
| *** doug-fish has joined #openstack-infra | 16:59 | |
| krotscheck | fungi, jeblair: That npm 2.15.1 is my desktop | 16:59 |
| fungi | now we know where to target the icbms | 17:00 |
| krotscheck | :-P | 17:00 |
| krotscheck | Lemme upgrade my version and rerun my tests. | 17:00 |
| jeblair | yeah, so i'd disregard that line, which leaves only the GET associated with the job as being an outlier | 17:00 |
| jeblair | (with a much-too-small response size) | 17:01 |
| SpamapS | jeblair: but of course | 17:01 |
| krotscheck | OH WAIT | 17:02 |
| * krotscheck remembers now. | 17:02 | |
| * krotscheck thinks | 17:02 | |
| *** piet has joined #openstack-infra | 17:02 | |
| krotscheck | I think there was an npm issue that munged up content sizes if gzip compression was used. | 17:02 |
| * krotscheck digs | 17:02 | |
| jeblair | krotscheck, fungi: npm reports an error ~5 minutes after the apache time (which, iirc, is request start time) | 17:03 |
| jeblair | (i think apache writes the log entry on request completion, but the timestamp is request start) | 17:03 |
| fungi | but yeah, if i filter on get requests with a browser agent string of "npm/3.5.2 node/v4.2.6 linux x64" then they're all the same size _except_ the request from 169.45.71.75 at 14:42:48 (the job in question) | 17:03 |
| *** pvaneck has joined #openstack-infra | 17:04 | |
| *** nwkarsten has quit IRC | 17:04 | |
| *** doug-fish has quit IRC | 17:04 | |
| krotscheck | Since so many other attempts work, it's likely not this gzip thing I remember :( | 17:04 |
| krotscheck | So, why would apache return a smaller document? | 17:04 |
| *** jpr has joined #openstack-infra | 17:04 | |
| krotscheck | fungi, jeblair: Where are y'all finding all these, kibana? | 17:05 |
| fungi | krotscheck: no, local apache logs on the server | 17:05 |
| krotscheck | fungi: Oh, you're cheating :-P | 17:05 |
| krotscheck | ;) | 17:05 |
| fungi | yes | 17:05 |
| fungi | the apache logs are not published | 17:06 |
| *** bpokorny has joined #openstack-infra | 17:06 | |
| *** lucasagomes is now known as lucas-afk | 17:06 | |
| fungi | though it seems like maybe there wouldn't be any harm in publishing our mirror server apache logs, especially since we tell people not to use them outside our ci | 17:06 |
| krotscheck | jeblair: A 5 minute interval between request and error implies that the client is waiting for the rest of the doc? | 17:06 |
| krotscheck | Though in that case, the client should throw something vis-a-vis unexpected termination | 17:07 |
| krotscheck | Instead of trying to parse the results | 17:07 |
| jeblair | krotscheck: i agree on the first part. i agree in principle on the second part, but don't know enough about npm to have a feel for whether it might manifest another way | 17:08 |
| fungi | though the delay between apache logging the request for /npm/postcss-loader and /npm/postcss-loader/ was only 3 seconds | 17:09 |
| *** mriedem1 is now known as mriedem | 17:09 | |
| jeblair | and the log entry for postcss-loader/ is scattered among others in the same timeframe (ie, it's not 5 minutes later) | 17:10 |
| *** julim has joined #openstack-infra | 17:10 | |
| *** _deadnull is now known as deadnull | 17:11 | |
| krotscheck | jeblair: Right- so why did that one request go froofy | 17:11 |
| *** abregman has quit IRC | 17:11 | |
| *** doug-fish has joined #openstack-infra | 17:11 | |
| *** doug-fish has quit IRC | 17:11 | |
| *** doug-fish has joined #openstack-infra | 17:12 | |
| *** ddieterly is now known as ddieterly[away] | 17:12 | |
| jeblair | the npm mirror update script did not log making an update to postcss-loader. it did do a mirror release at 14:40, and another at 14:50. unfortunately, we don't log duration or completion times for mirror releases there, so i don't know if that was happening at or near request time. | 17:12 |
| krotscheck | Well, I just updated my npm client to 3.5.2 and am getting WAY more errors. | 17:13 |
| *** akshai has joined #openstack-infra | 17:13 | |
| jeblair | oh, sorry, next release was at 14:45 | 17:13 |
| krotscheck | Ya know, we may have to back off from using the xenial npm package. | 17:14 |
| *** jlanoux has quit IRC | 17:15 | |
| *** shashank_hegde has joined #openstack-infra | 17:16 | |
| krotscheck | Ok, so npm version 2.14, same project, same mirror, installs fine. 10 minutes later, same project, same mirror, npm version 3.5.2: errors like the dickens. | 17:16 |
| jeblair | (afs logs are not helpful here, they only log errors) | 17:16 |
| kashyap | clarkb: When you have a sec, mind pastebin'ing contents of /etc/default/grub from Xenial? | 17:17 |
| jeblair | krotscheck, fungi: if the client (or network) aborts a transfer, it will look like the log entry we have | 17:19 |
| clarkb | kashyap: http://paste.openstack.org/show/497761/ that is what my dib built image has | 17:19 |
| pabelanger | clarkb: fungi: Got the go a head from tripleo to launch a test instances of: mirror.tripleo-test-cloud-rh1.regionOne.openstack.org | 17:20 |
| jeblair | krotscheck, fungi: so "client/network error aborted request early" is indistinguishable from "actually read truncated file from disk" in apache | 17:20 |
| krotscheck | jeblair: I suspect this is a bug inside of the npm client at the LTS version that's in xenial. | 17:20 |
| kashyap | clarkb: Ah, thanks. I was missing "serial console" in GRUB_TERMINAL. (I had just "serial") | 17:20 |
| openstackgerrit | Pedro Alvarez Piedehierro proposed openstack-infra/storyboard-webclient: Close card-details modal when clicking on the Story link https://review.openstack.org/318779 | 17:20 |
| jeblair | krotscheck: are all these errors on xenial? | 17:21 |
| krotscheck | jeblair: yep | 17:21 |
| krotscheck | I'm spinning up a xenial node to test. | 17:21 |
| *** Na3iL has quit IRC | 17:21 | |
| fungi | kashyap: http://paste.openstack.org/show/497762 | 17:22 |
| jeblair | krotscheck: then i agree that seems to fit the evidence so far. | 17:22 |
| jeblair | also, wow, apparently we make enough npm requests to see statistically significant errors in released software at scale. | 17:22 |
| *** BobBall is now known as BobBall_AWOL | 17:22 | |
| kashyap | fungi: Thanks. I was checking what black magic I was missing to get a serial console on Xenial. | 17:22 |
| fungi | kashyap: oh, nevermind, i see clarkb already did | 17:22 |
| kashyap | Yeah | 17:22 |
| *** nadya has joined #openstack-infra | 17:23 | |
| krotscheck | jeblair: no kidding | 17:23 |
| openstackgerrit | Zara proposed openstack-infra/python-storyboardclient: WIP Proof of Concept Storyboard Updating from Gerrit https://review.openstack.org/302912 | 17:23 |
| *** Na3iL has joined #openstack-infra | 17:23 | |
| openstackgerrit | Pedro Alvarez Piedehierro proposed openstack-infra/storyboard-webclient: Close card-details modal when clicking on the Story link https://review.openstack.org/318779 | 17:23 |
| fungi | krotscheck: jeblair: however, this doesn't explain sdague's example where we got a truncated wheel file in a job | 17:24 |
| krotscheck | jeblair: My first hope is that it'll all work if I revert this bit: https://review.openstack.org/#/c/312742/1/jenkins/jobs/macros.yaml | 17:24 |
| krotscheck | fungi: True | 17:24 |
| fungi | i'll see if i can do a similar investigation on that one | 17:24 |
| *** zzelle_ has joined #openstack-infra | 17:24 | |
| *** banix has joined #openstack-infra | 17:25 | |
| clarkb | kashyap: ok, fwiw I use dib and the devuser element and just ssh into things | 17:26 |
| *** regXboi has quit IRC | 17:26 | |
| kashyap | clarkb: I was using - http://libguestfs.org/virt-builder.1.html | 17:27 |
| fungi | kashyap: note we have a script to help you build identical images to what we use in our ci... http://git.openstack.org/cgit/openstack-infra/project-config/tree/tools/build-image.sh | 17:28 |
| * kashyap does: dnf whatprovides */disk-image-create | 17:29 | |
| kashyap | Ah, it's diskimage-builder | 17:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add timestamps to all stages of mirror scripts https://review.openstack.org/318824 | 17:29 |
| *** sridhar_ram_ has joined #openstack-infra | 17:30 | |
| jeblair | fungi: ^ for easier correlation in the future | 17:30 |
| mordred | kashyap: yah. we use diskimage-builder for all of our images | 17:30 |
| fungi | jeblair: yep, this will certainly help | 17:30 |
| *** mrmartin has joined #openstack-infra | 17:32 | |
| pabelanger | jeblair: Thank you! I was having troubles the other day too | 17:32 |
| *** fawadkhaliq has quit IRC | 17:33 | |
| *** Na3iL has quit IRC | 17:34 | |
| kashyap | mordred: Yeah, aware of the project. (Not to go on a tangent, it still uses 'sudo kpartx' while `guestfish` can handle that without needing root.) | 17:35 |
| clarkb | kashyap: doesn't it require libvirt group membershup though which is very similar to root? | 17:36 |
| *** mrmartin has quit IRC | 17:36 | |
| kashyap | clarkb: `guestfish` can operate completely independent without libvirt: | 17:36 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Add helper to find full slots and pep8 check https://review.openstack.org/317809 | 17:37 |
| kashyap | clarkb: It has two 'backends': (a) 'direct' (ie. QEMU, you can set it by: LIBGUESTFS_BACKEND=direct); (b) 'libvirt' | 17:37 |
| *** EricGonczer_ has joined #openstack-infra | 17:37 | |
| kashyap | So, 'libvirt' group membership is only applicable if one is using guestfish with its libvirt backend. | 17:38 |
| clarkb | I se | 17:38 |
| *** ddieterly[away] is now known as ddieterly | 17:39 | |
| krotscheck | jeblair, fungi: I can confirm that it's not the npm version. | 17:39 |
| kashyap | clarkb: Couple of years ago there was a discussion about it on fedora-cloud mailing list, where lifeless (IIRC) also chimed in on this subject | 17:39 |
| *** elo has quit IRC | 17:39 | |
| *** dmsimard is now known as dmsimard|afk | 17:39 | |
| * kashyap was trying to find that thread, but can't seem to find it now | 17:39 | |
| *** savihou has joined #openstack-infra | 17:39 | |
| clarkb | kashyap: I just rmember someone arguing its fast because libvirt which seemed to negate the security argument for me | 17:39 |
| clarkb | but I guess you have the option of fast or secure | 17:39 |
| kashyap | clarkb: Sorry, what's fast? (What is the 'it') | 17:40 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Add helper to find full slots and pep8 check https://review.openstack.org/317809 | 17:40 |
| clarkb | kashyap: image builds using alternatives to dib | 17:40 |
| kashyap | Yeah, if using libvirt, there's the sVirt mechanism | 17:40 |
| clarkb | kashyap: in this case the libguestfs stuff | 17:40 |
| krotscheck | Or rather, I can confim that 2.14.1 and 3.5.2 display the same errors. | 17:40 |
| *** nwkarsten has joined #openstack-infra | 17:40 | |
| *** nwkarsten has quit IRC | 17:41 | |
| kashyap | clarkb: `virt-builder` can churn out images super fast, once the image is cached. | 17:41 |
| *** nwkarsten has joined #openstack-infra | 17:41 | |
| *** EricGonc_ has joined #openstack-infra | 17:41 | |
| kashyap | About 20 seconds or less. | 17:42 |
| *** javeriak has joined #openstack-infra | 17:42 | |
| krotscheck | jeblair: npm version 3.9.0 does not appear to have this issue....? | 17:42 |
| fungi | krotscheck: npm has an lts series now, right? | 17:43 |
| krotscheck | fungi: On xenial, yes, and I've filed a bug. | 17:43 |
| fungi | maybe you can bisect to find the fix and look into backporting | 17:43 |
| krotscheck | Let me see what their official lts version is. | 17:43 |
| fungi | if it really is working on 3.9.0 fine | 17:43 |
| mordred | kashyap: guesfish handles a completely different problem | 17:44 |
| kashyap | mordred: Of course, I realize. I was talking about `virt-builder` | 17:44 |
| kashyap | mordred: Sorry, I realize I first mentioned 'guestfish' | 17:44 |
| *** r-daneel has quit IRC | 17:44 | |
| *** EricGonczer_ has quit IRC | 17:45 | |
| *** tonytan4ever has quit IRC | 17:45 | |
| kashyap | But yeah, the mounting of images with `sudo kpartx` is still a bit debatable. But won't go into that rabbit hole now :-) | 17:45 |
| mordred | it's all good - it's honestly the same problem - dib is a content description system, so I just want to make sure you don't get yourself caught into the rathole of reimplementing all of the elements we use | 17:45 |
| mordred | because I think that's a ton of work that is likely not particularly entertaining | 17:45 |
| *** degorenko is now known as _degorenko|afk | 17:46 | |
| mtreinish | fungi: found another storyboard thing that's bugging me. No newlines in comments, everytime I hit enter (tried shift+enter too) it saves the comment | 17:46 |
| fungi | mtreinish: huh, interesting. i thought i had been able to enter newlines in sb comments | 17:46 |
| *** sarob has joined #openstack-infra | 17:46 | |
| mordred | kashyap: I _do_ think that it would be interesting to explore having dib be able to use guestfish/virt-builder/libguestfs on the backend instead of chroot/qemu-ndb/kpartx | 17:46 |
| mtreinish | fungi: https://storyboard.openstack.org/#!/story/2000574 I clearly failed at it :) | 17:47 |
| mordred | kashyap: which is likely to also be a lot of work, but is probably more interesting work and more likely to be broadly useful to people | 17:47 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Add helper to find full slots and pep8 check https://review.openstack.org/317809 | 17:47 |
| *** esikachev has quit IRC | 17:47 | |
| *** akshai has quit IRC | 17:47 | |
| *** daemontool has quit IRC | 17:47 | |
| mordred | kashyap: I keep meaning to play with that myself, but keep having other thigns on my plate :) | 17:47 |
| kashyap | clarkb: This was the discussion from JUN-2013: https://goo.gl/7wS9BO | 17:47 |
| krotscheck | fungi: I'm checking the current LTS version (2.15.6) to make sure it works. | 17:48 |
| krotscheck | So far so good (on trusty and osx) | 17:48 |
| kashyap | mordred: Can understand. Feel free to ping me anything about `virt-builder`, FWIW, I use it extensively in my workflows | 17:48 |
| fungi | mtreinish: if i click in the "Enter your comment here" text area, i can create newlines with my enter key just fine (firefox 44) | 17:49 |
| fungi | mtreinish: this might be some specific browser behavior/extension fighting some sort of javascript dynamism in the webclient there? | 17:50 |
| mtreinish | fungi: with text already in the box? it works fine for me (firefox 46.0.1) with an empty box, but I was hitting enter out of habit in that bug and it wasn't making newlines | 17:50 |
| fungi | would be good to identify so it can be fixed | 17:50 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Add helper to find full slots and pep8 check https://review.openstack.org/317809 | 17:51 |
| fungi | mtreinish: yeah, i added a comment to the story you linked, in which i included newlines | 17:51 |
| *** kzaitsev_mb has quit IRC | 17:51 | |
| mtreinish | ah, it's shift+enter (which I guess I was doing out of habit) | 17:52 |
| fungi | ohhh | 17:52 |
| fungi | i will admit i never thought to try a shift-enter while typing in there | 17:52 |
| *** akshai has joined #openstack-infra | 17:54 | |
| mtreinish | fungi: I guess I do it by habit as a way around text boxes that submit when you hit enter (wasn't expecting the opposite here) | 17:55 |
| mordred | kashyap: I have enough things on my plate that I'll likely not do any work on this. but for the folks who have a problem with kpartx use in the openstack image buiding tool, I would really hope that we'd see collaboration from them on improving openstack's tool | 17:56 |
| mordred | kashyap: and that that would result in patches to dib to support different image generation backends, such as virt-builder | 17:56 |
| krotscheck | fungi: So, it turns out that the npm package on Xenial LTS? Not acutally npm's LTS version. | 17:57 |
| openstackgerrit | Thiago da Silva proposed openstack-infra/project-config: add python jobs to pyeclib project https://review.openstack.org/317672 | 17:57 |
| fungi | krotscheck: not entirely surprising | 17:57 |
| *** tonytan4ever has joined #openstack-infra | 17:57 | |
| fungi | jeblair: okay, this is perplexing. in sdague's example at http://logs.openstack.org/59/318159/1/gate/gate-tempest-dsvm-cells/013ece6/logs/devstacklog.txt.gz#_2016-05-19_12_32_08_499 there is _no_ corresponding get in the apache access log. i see it hit /pypi/simple/lxml/ and /wheel/ubuntu-14.04-x86_64/lxml/ but according to apache it never goes on to request | 17:59 |
| fungi | /wheel/ubuntu-14.04-x86_64/lxml/lxml-3.5.0-cp27-none-linux_x86_64.whl | 17:59 |
| kashyap | mordred: Sure - about time. On a related note, I came across `virt-dib` -- "Virt-dib is a tool for using the elements of diskimage-builder to build a new disk image, generate new ramdisks, etc." - http://libguestfs.org/virt-dib.1.html | 17:59 |
| *** SumitNaiksatam has joined #openstack-infra | 18:00 | |
| *** dtardivel has quit IRC | 18:00 | |
| odyssey4me | project-config admins/cores - I'd appreciate a review of https://review.openstack.org/317931 when you all have a moment to do so. | 18:00 |
| *** Apoorva_ has joined #openstack-infra | 18:00 | |
| *** esikachev has joined #openstack-infra | 18:00 | |
| mordred | kashyap: yup. that was the main suggestion in the thread about using not-dib for images | 18:00 |
| mordred | kashyap: and I think is a good one | 18:01 |
| mordred | kashyap: the commuity value to dib for me is in the elements, which are content | 18:01 |
| fungi | jeblair: pared down log sample... http://paste.openstack.org/show/497777 | 18:01 |
| mordred | kashyap: having different mechanisms to apply that content to produce a final image seems awesome | 18:01 |
| kashyap | mordred: Yep, the elements are very useful | 18:02 |
| *** ddieterly is now known as ddieterly[away] | 18:02 | |
| krotscheck | fungi: False alarm, I switched mirrors to ord and all the errors started popping up again | 18:03 |
| krotscheck | And they seem to go away after 3 or so runs | 18:03 |
| *** Apoorva has quit IRC | 18:04 | |
| jeblair | krotscheck: with what version? | 18:07 |
| *** trown|lunch is now known as trown | 18:07 | |
| jeblair | fungi: how does it know it's 2.9MB then? | 18:08 |
| *** esikachev has quit IRC | 18:08 | |
| fungi | jeblair: that's a very good question... possibly from the index? | 18:09 |
| krotscheck | jeblair: 2.5.16, 3.9.0, 2.15.1 | 18:09 |
| dstufft | pip doesn't look at the index for file size | 18:09 |
| fungi | jeblair: i vaguely recall discussing with dstufft an error mode where the actual http failure exception is masked and pip goes on to try to unpack an empty object | 18:09 |
| dstufft | it uses Content-Length IIRC | 18:09 |
| krotscheck | As soon as I switched from region1 (which was failing, then stopped) to ord, all the errors came back. | 18:10 |
| *** nadya has quit IRC | 18:10 | |
| fungi | okay, then that's doubly weird that it got the righth content length but apache recorded no request for the file | 18:10 |
| greghaynes | kashyap: mordred I'd really like to see someone work on a libguestfs-style backend for dib, but I too have way too much other stuff on my plate right now | 18:10 |
| jeblair | dstufft, fungi: so we should have a corresponding apache request | 18:10 |
| *** Shrews has quit IRC | 18:10 | |
| jeblair | dstufft, fungi: cause the "(2.9MB)" indicates it at least received a content-length header | 18:11 |
| *** notmorgan has quit IRC | 18:11 | |
| jeblair | krotscheck: same files involved? | 18:11 |
| dstufft | jeblair: fungi is this on a persistent node? | 18:12 |
| fungi | dstufft: nope | 18:12 |
| dstufft | what's the pip output | 18:12 |
| fungi | dstufft: http://logs.openstack.org/59/318159/1/gate/gate-tempest-dsvm-cells/013ece6/logs/devstacklog.txt.gz#_2016-05-19_12_32_08_499 | 18:13 |
| *** abregman has joined #openstack-infra | 18:14 | |
| krotscheck | jeblair: urrrrrr. | 18:14 |
| fungi | dstufft: i filtered the apache log for mirror.ord.rax.openstack.org for requests from 104.130.205.205 (the node that job ran on) to also get a representative sample since we burn through so many instances that rackspace ends up reassigning the same ip addresses to a few over the course of a day http://paste.openstack.org/show/497777/ | 18:15 |
| fungi | dstufft: the requests you see there at 09:10:29, 12:32:08 and 14:07:28 all ran on different nova instances that just happened to get the same ipv4 address reassigned | 18:16 |
| *** doug-fis_ has joined #openstack-infra | 18:16 | |
| krotscheck | jeblair: 3.9 failed on karma, 2.15.6 failed on beeper | 18:16 |
| *** DevBox has quit IRC | 18:17 | |
| *** eharney has quit IRC | 18:18 | |
| *** nadya has joined #openstack-infra | 18:18 | |
| jeblair | fungi: that is weird. i agree, i can't find any indication apache received that request. yet somehow pip knew about the size. | 18:18 |
| *** dprince has quit IRC | 18:18 | |
| *** doug-fish has quit IRC | 18:19 | |
| jeblair | [Thu May 19 12:32:09.077736 2016] [core:notice] [pid 8827:tid 140505418942336] AH00052: child pid 5896 exit signal Segmentation fault (11) | 18:19 |
| jeblair | fungi: ^ i'm at a point where i might entertain that is related. | 18:19 |
| dstufft | it wasn't cached either | 18:20 |
| dstufft | the file size doesn't get shown on cached responses | 18:20 |
| *** esikachev has joined #openstack-infra | 18:20 | |
| dstufft | I don't think we accurately fail if the download gets interupted | 18:21 |
| cloudnull | jeblair clarkb fungi pabelanger: I've been debugging shade and I have a couple of questions. | 18:21 |
| dstufft | typicall it gets exposed as a hash failure | 18:21 |
| cloudnull | I've built a small test script which is trying to do this https://github.com/openstack-infra/nodepool/blob/master/nodepool/provider_manager.py#L180-L226 | 18:21 |
| dstufft | but if these don't have hashes, then it would be reasonable to suggest that this is just an interuppted download failing later in the process | 18:22 |
| fungi | jeblair: oh! i looked in the vhost-specific error log but forgot to check the general error log. that timestamp does correlate disturbingly well | 18:22 |
| jeblair | fungi, krotscheck: i do not see similar erorrs on bluebox near where we were looking earlier; however, the errors krotscheck just saw on ord might be segfaults. | 18:22 |
| *** eharney has joined #openstack-infra | 18:23 | |
| cloudnull | and tracing shade I found that in meta it writes this into the server dict https://github.com/openstack-infra/shade/blob/master/shade/meta.py#L300-L304 | 18:23 |
| *** hashar has joined #openstack-infra | 18:23 | |
| jeblair | fungi, krotscheck: (so, my current thinking -- segfault for the wheel errror; npm responsible for sjc1 npm errors; segfault possibly responsible for ord npm errors -- further log analysis needed to determine that) | 18:23 |
| fungi | jeblair: and there do seem to be many, many, many of those | 18:23 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add CloudKitty role to OpenStack-Ansible https://review.openstack.org/318836 | 18:25 |
| cloudnull | which seems to create a false positive here https://github.com/openstack-infra/shade/blob/master/shade/meta.py#L123-L124 which returns https://github.com/openstack-infra/shade/blob/master/shade/openstackcloud.py#L3196-L3197 and seems to be the reason that some vms dont get an IP because of this https://github.com/openstack-infra/shade/blob/master/shade/openstackcloud.py#L3967-L3970 | 18:25 |
| cloudnull | so my question, is it "needed" in some clouds to write the accessIPv4 entry ? | 18:26 |
| *** notmorgan has joined #openstack-infra | 18:26 | |
| *** lennyb has quit IRC | 18:26 | |
| *** russellb has quit IRC | 18:26 | |
| *** dosaboy has quit IRC | 18:26 | |
| *** anthonyper has quit IRC | 18:26 | |
| *** zz_zz_ja has quit IRC | 18:26 | |
| *** _degorenko|afk has quit IRC | 18:26 | |
| *** pabelanger has quit IRC | 18:26 | |
| *** dmanchad has quit IRC | 18:26 | |
| *** dmellado has quit IRC | 18:26 | |
| *** Douhet has quit IRC | 18:26 | |
| *** bradm has quit IRC | 18:26 | |
| *** mahatic has quit IRC | 18:26 | |
| *** j_king has quit IRC | 18:26 | |
| *** coreycb has quit IRC | 18:26 | |
| *** crinkle has quit IRC | 18:26 | |
| *** lifeless has quit IRC | 18:26 | |
| *** Daviey has quit IRC | 18:26 | |
| *** mfisch has quit IRC | 18:26 | |
| *** apuimedo has quit IRC | 18:26 | |
| *** aviau has quit IRC | 18:26 | |
| *** j_king has joined #openstack-infra | 18:26 | |
| *** degorenko has joined #openstack-infra | 18:26 | |
| cloudnull | oh. netsplit. | 18:26 |
| *** apuimedo has joined #openstack-infra | 18:26 | |
| *** bradm has joined #openstack-infra | 18:26 | |
| *** coreycb has joined #openstack-infra | 18:26 | |
| *** deadnull is now known as _deadnull | 18:26 | |
| *** dmanchad has joined #openstack-infra | 18:26 | |
| *** lifeless has joined #openstack-infra | 18:26 | |
| *** dosaboy has joined #openstack-infra | 18:26 | |
| *** Daviey has joined #openstack-infra | 18:26 | |
| *** crinkle has joined #openstack-infra | 18:26 | |
| clarkb | cloudnull: that is how shade users know what to connect to because openstack doesn't actually tell you | 18:26 |
| *** zz_ja has joined #openstack-infra | 18:26 | |
| jeblair | cloudnull: i got your question | 18:26 |
| *** Douhet has joined #openstack-infra | 18:26 | |
| *** russellb has joined #openstack-infra | 18:26 | |
| *** lennyb has joined #openstack-infra | 18:27 | |
| *** mahatic has joined #openstack-infra | 18:27 | |
| *** dmellado has joined #openstack-infra | 18:27 | |
| *** Guest76268 has joined #openstack-infra | 18:27 | |
| *** anthonyper has joined #openstack-infra | 18:27 | |
| cloudnull | clarkb: but that seems to instruct shade to not assign a float to some instances | 18:27 |
| *** aviau has joined #openstack-infra | 18:27 | |
| cloudnull | in terms of what nodepool is doing | 18:27 |
| mordred | cloudnull: most clouds do not require floating ips | 18:27 |
| fungi | that netsplit seems to have been very minor | 18:28 |
| *** pabelanger has joined #openstack-infra | 18:28 | |
| *** oanson has joined #openstack-infra | 18:28 | |
| cloudnull | nodepool only instructs shade to auto_ip and doesnt provide a float pool | 18:29 |
| cloudnull | https://github.com/openstack-infra/nodepool/blob/master/nodepool/provider_manager.py#L234-L236 | 18:29 |
| mordred | cloudnull: that's right. that should do the right thing just about everywhere | 18:29 |
| mordred | providing a float pool is not needed | 18:29 |
| mordred | it is possible to discover where floating ips should come from on all clouds we've experienced | 18:29 |
| *** dham1 has quit IRC | 18:29 | |
| *** nemish has quit IRC | 18:30 | |
| mordred | float pool is only needed if there were more than one available subnet that had a gateway ip and a router associated with it - which is pretty uncommon | 18:30 |
| *** zhenguo_ has quit IRC | 18:30 | |
| *** igorbelikov has quit IRC | 18:30 | |
| *** mwhahaha has quit IRC | 18:30 | |
| cloudnull | so without the float pool it looks like a race here https://github.com/openstack-infra/shade/blob/master/shade/openstackcloud.py#L3955-L3971 | 18:31 |
| *** tpsilva has quit IRC | 18:31 | |
| *** monester has quit IRC | 18:31 | |
| *** vaidy has quit IRC | 18:31 | |
| *** sweston has quit IRC | 18:31 | |
| *** ildikov has joined #openstack-infra | 18:31 | |
| fungi | jeblair: it's pretty hard to turn up apache segfault examples which don't involve mod_php (because it's apparently really good at creating them) | 18:31 |
| cloudnull | which only hits the auto_ip condition | 18:31 |
| mordred | cloudnull: what's the race? | 18:31 |
| mordred | cloudnull: yah. it should only hit auto_ip | 18:31 |
| *** teran has quit IRC | 18:31 | |
| *** kgalanov has quit IRC | 18:31 | |
| *** andrewbogott has quit IRC | 18:31 | |
| *** zhiyan has quit IRC | 18:31 | |
| *** vdrok has quit IRC | 18:31 | |
| *** dham1 has joined #openstack-infra | 18:31 | |
| *** nemish has joined #openstack-infra | 18:31 | |
| *** bhunter71 has quit IRC | 18:32 | |
| mordred | cloudnull: then if it doesn't have a public IP (this is a floating ip cloud) - it'll add one in self._add_auto_ip | 18:32 |
| cloudnull | so the meta rewrites the accessIPv4 and then returns my 10.x address as its public | 18:32 |
| mordred | it shouldn't | 18:32 |
| mordred | if it's doing that, that's a bug | 18:32 |
| mordred | it should only write the 10. into accessIPv4 if you've configured it that you want to use private ips to access the instance | 18:33 |
| *** vaidy has joined #openstack-infra | 18:33 | |
| jeblair | cloudnull: note that a call to get_server_external_ipv4 precedes writing the accessIPv4 entry here: https://github.com/openstack-infra/shade/blob/master/shade/meta.py#L291 | 18:33 |
| mordred | cloudnull: https://github.com/openstack-infra/shade/blob/master/shade/meta.py#L300 | 18:33 |
| *** Shrews has joined #openstack-infra | 18:33 | |
| *** kgalanov has joined #openstack-infra | 18:33 | |
| *** teran has joined #openstack-infra | 18:33 | |
| *** igorbelikov has joined #openstack-infra | 18:33 | |
| jeblair | cloudnull: so the first time through, it will not short-circuit inside get_server_external_ipv4 | 18:33 |
| *** mwhahaha has joined #openstack-infra | 18:33 | |
| *** tpsilva has joined #openstack-infra | 18:33 | |
| *** _sarob has joined #openstack-infra | 18:34 | |
| *** zhenguo_ has joined #openstack-infra | 18:34 | |
| *** andrewbogott has joined #openstack-infra | 18:34 | |
| ildikov | fungi: hi | 18:34 |
| *** vdrok has joined #openstack-infra | 18:34 | |
| *** sweston has joined #openstack-infra | 18:34 | |
| ildikov | fungi: I have a quick question I hope you can point me to the right direction | 18:34 |
| *** zhiyan has joined #openstack-infra | 18:34 | |
| cloudnull | jeblair mordred if i run https://gist.github.com/cloudnull/2f3931e51062dc7b325faf9a24974006#file-serial-shade-test-py the server output contains a rewritten accessIPv4 | 18:35 |
| *** monester has joined #openstack-infra | 18:35 | |
| cloudnull | and that is using the args I found in nodepool | 18:35 |
| *** esikachev has quit IRC | 18:35 | |
| ildikov | fungi: we would like to test watchdog device in tempest as part of a gate job and we need a special image for that with the proper config | 18:35 |
| cloudnull | https://github.com/openstack-infra/nodepool/blob/master/nodepool/provider_manager.py#L188-L226 -- those args | 18:35 |
| *** kamsz has joined #openstack-infra | 18:36 | |
| fungi | ildikov: by special image, you mean an image booted by the nova which runs as part of devstack? | 18:36 |
| mordred | cloudnull: it contains the 10. address? | 18:36 |
| ildikov | fungi: yeap an image Nova can boot the VM from | 18:36 |
| fungi | ildikov: if so, several projects already do this to test agent-type applications (ironic-python-agent, trove-agent, et cetera) | 18:37 |
| ildikov | fungi: here's the WIP test code: https://review.openstack.org/#/c/315786/ | 18:37 |
| cloudnull | mordred: | 18:37 |
| cloudnull | http://cdn.pasteraw.com/ramrt7ujtqfvajshymae7fzdaf3ps78 | 18:37 |
| cloudnull | ha! | 18:37 |
| *** sarob has quit IRC | 18:37 | |
| ildikov | fungi: and does the image go under those repos? | 18:37 |
| mordred | cloudnull: ok. that seems like a bug. Shrews ^^ | 18:37 |
| cloudnull | and a query to nova returns http://cdn.pasteraw.com/t4a7qviixjpnjvmq98g3g7lyfo3ljdo | 18:37 |
| kamsz | hi there, is shade library capable of performing live migration? if not, are there any plans to add it? | 18:38 |
| fungi | ildikov: they generally use diskimage-builder to create and publish the image periodically to tarballs.openstack.org | 18:38 |
| jeblair | mordred, cloudnull: could it matter that we're using keystonev3? | 18:38 |
| asselin_ | dstufft, do you konw if upgrading from bandersnatch 1.10 to 1.11 requires a full sync? Also do you know if that bug would cause bandernatch mirror to go to 100% CPU? | 18:38 |
| mordred | jeblair: it shouldn't, no | 18:38 |
| fungi | ildikov: like https://tarballs.openstack.org/ironic-python-agent/coreos/ | 18:38 |
| cloudnull | jeblair: maybe ? | 18:38 |
| dstufft | asselin_: don't think so | 18:39 |
| mordred | jeblair, cloudnull: the logic related to figuring out the ip has nothing to do with keystone | 18:39 |
| ildikov | fungi: ah, ok, I will check that | 18:39 |
| Shrews | mordred: there are different rules for accessIPv4, iirc | 18:39 |
| jeblair | cloudnull: for reference, here is our clouds.yaml: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/nodepool/clouds.yaml.erb#n51 | 18:39 |
| Shrews | i seem to have lost some scrollback due to a weechat freeze | 18:39 |
| ildikov | fungi: is there any description regarding best practices on how to do this or just follow what we see as examples? | 18:40 |
| Shrews | so i do not have full context | 18:40 |
| asselin_ | dstufft, is that answer for both questions? | 18:40 |
| mordred | Shrews: so - at the moment, cloudnull is getting accessIPv4 rewritten with his private 10.x address | 18:40 |
| dstufft | asselin_: er, sorry it's an answer to the first | 18:40 |
| jeblair | Shrews: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-05-19.log.html#t2016-05-19T18:21:08 | 18:40 |
| dstufft | the second one is that I have no idea | 18:40 |
| fungi | ildikov: i could dig up the details on how their periodic jobs work, but you'd probably be better off syncing up with someone on the ipa core team https://review.openstack.org/#/admin/groups/307,members (maybe JayF has some expertise with that) | 18:40 |
| asselin_ | dstufft, ok thanks | 18:41 |
| cloudnull | maybe im doing it wrong? but im trying to figure out why FIP are being bastards on OSIC for everyone . | 18:41 |
| Shrews | mordred: is cloud.private == True? | 18:41 |
| mordred | which should never happen unless a) private=True is set b) we're discovering the information about the networks there incorrectly | 18:41 |
| jeblair | Shrews: if that doesn't quite have enough context, http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-05-19.log should have the rest | 18:41 |
| ildikov | fungi: sure, I'll do that, I just didn't want to necessarily ping more people with this if it's already written down | 18:41 |
| ildikov | fungi: thanks much for the pointers | 18:41 |
| mordred | Shrews: not to my knowledge, no | 18:41 |
| cloudnull | mordred: that may be my issue, im not setting cloud.private | 18:42 |
| mordred | cloudnull: can you put prints around https://github.com/openstack-infra/shade/blob/master/shade/meta.py#L300-L304 | 18:42 |
| mordred | cloudnull: you should not need to | 18:42 |
| *** baoli_ has quit IRC | 18:42 | |
| cloudnull | sure | 18:42 |
| mordred | cloudnull: but could you verify the values of cloud.private, server['accessIPv4'] and server['public_v4'] there? | 18:42 |
| *** rprakash has joined #openstack-infra | 18:43 | |
| *** jistr has quit IRC | 18:43 | |
| mordred | and maybe server['private_v4'] | 18:43 |
| rprakash | #info rprakash | 18:43 |
| *** ihrachys has joined #openstack-infra | 18:43 | |
| mordred | cloudnull: btw - pabelanger has found that if we lower the rate at which we do create server / create fip - we stop leaking | 18:43 |
| mordred | cloudnull: I believe he's currently testing one request per 0.5 seconds - one per 1 second definitely stopped the leak. one per 0.001 seconds definitely leaks | 18:44 |
| rprakash | Anyone how do i do git push for a new project? what's the target for git push after local commit? | 18:46 |
| *** abregman is now known as abregman|nb | 18:46 | |
| fungi | rprakash: git review -t new-project | 18:46 |
| rprakash | Like I get on local commit : 6 files changed, 55 insertions(+) create mode 100644 gerrit/acls/openstack/meghdwar.config create mode 100644 jenkins/jobs/meghdwar.yaml | 18:46 |
| *** kzaitsev_mb has joined #openstack-infra | 18:46 | |
| *** rcernin has joined #openstack-infra | 18:47 | |
| *** bhunter71 has joined #openstack-infra | 18:47 | |
| fungi | rprakash: that will cause the git-review tool to push your change into gerrit for review, with a review topic of "new-project" | 18:47 |
| rprakash | but git review does not work on my macbook | 18:48 |
| fungi | rprakash: what error does it emit? | 18:48 |
| rprakash | see Kedars-Macbook:project-config pramchan$ git review -t new-project git: 'review' is not a git command. See 'git --help'. Kedars-Macbook:project-config pramchan$ git-review -t new-project -bash: git-review: command not found | 18:48 |
| fungi | rprakash: you haven't even installed git-review from the looks of it | 18:48 |
| rprakash | No I did and check ypour email I sent you additional info | 18:49 |
| fungi | rprakash: that's what i meant yesterday about the new project creation instructions assuming you'd contributed code for review to other openstack projects before now | 18:49 |
| fungi | rprakash: http://docs.openstack.org/infra/manual/developers.html talks about installing git review, after the section about setting up your accounts | 18:50 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Temporarily start using the public registry again https://review.openstack.org/318875 | 18:50 |
| *** andrewbogott has quit IRC | 18:51 | |
| *** andrewbogott has joined #openstack-infra | 18:51 | |
| krotscheck | fungi, jeblair ^^ should make our builds start using the public registry until we figure out the segfaults | 18:51 |
| *** dimtruck is now known as zz_dimtruck | 18:51 | |
| cloudnull | mordred: prints around what you asked | 18:51 |
| cloudnull | http://cdn.pasteraw.com/dj7wurnhjeq9bxrezzsm9lp272tstru | 18:51 |
| Shrews | cloudnull: if you just call cloud.get_external_networks(), are any of the returned networks actually private? | 18:51 |
| cloudnull | it looks like get_server_external_ipv4 gets hit a lot | 18:51 |
| cloudnull | Shrews: looking | 18:52 |
| jeblair | krotscheck: i do not believe we have correlated a segfault with an npm error | 18:52 |
| *** bhunter71 has quit IRC | 18:53 | |
| jeblair | (the npm errors and the pypi error have manifested very differently; i would not assume they have the same cause) | 18:53 |
| *** bhunter71 has joined #openstack-infra | 18:53 | |
| rprakash | Kedars-Macbook:project-config pramchan$ pip install git-review Requirement already satisfied (use --upgrade to upgrade): git-review in /Library/Python/2.7/site-packages Requirement already satisfied (use --upgrade to upgrade): argparse in /Library/Python/2.7/site-packages (from git-review) Requirement already satisfied (use --u | 18:54 |
| *** eventingmonkey has joined #openstack-infra | 18:54 | |
| jeblair | krotscheck: if you supply timestamps for your attempts against ord, we can correlate those with log entries (but not me right now, i'm about to head to lunch) | 18:54 |
| *** Sukhdev has joined #openstack-infra | 18:55 | |
| fungi | rprakash: in that case it sounds like maybe git-review is not in your execution path. i really don't know the first thing about macintosh os, but hopefully someone else here who uses it knows what we're missing in the installation instructions there for it | 18:55 |
| fungi | rprakash: it's ambitious to attempt to create a new openstack project without having helped write patches to existing openstack projects. usually people creating new projects in our ecosystem have previously worked on other existing projects in openstack before coming to the conclusion that they need to make another one | 18:55 |
| Shrews | cloudnull: what network is 1057bd4b-fa35-4848-ae8a-0a5dbe661303 ? | 18:55 |
| cloudnull | http://cdn.pasteraw.com/88bf7rq86ieg27mz6apo6z95cqptff9 | 18:56 |
| rprakash | pip install git-review --upgrade Requirement already up-to-date: git-review in /Library/Python/2.7/site-packages Requirement already up-to-date: argparse in /Library/Python/2.7/site-packages (from git-review) Requirement already up-to-date: requests>=1.1 in /Library/Python/2.7/site-packages (from git-review) | 18:56 |
| fungi | rprakash: it's also ambitious to attempt to develop software on something other than linux which is intended to only run on linux, so if you choose to use macintosh you may be fighting an uphill battle | 18:56 |
| krotscheck | jeblair: urgh, I don' thave those | 18:56 |
| cloudnull | Shrews: thats the test net I've been using | 18:57 |
| cloudnull | http://cdn.pasteraw.com/cqcyrv7q368cpz9hw7joj5efepcevsb -- that has all of the return from cloud.get_external_networks() | 18:57 |
| mordred | cloudnull: wow. yah. public_v4 should NEVER have your 10.x address | 18:57 |
| *** nadya has quit IRC | 18:57 | |
| rprakash | I tried it earlier in ubuntu and ran into issues any way at this time once I get the project created will try handle tasks differently as I can pss the handle to experts | 18:58 |
| mordred | cloudnull: oh - your ADMIN_TEST_NET is a provider net | 18:58 |
| *** doug-fis_ has quit IRC | 18:58 | |
| *** shamail has joined #openstack-infra | 18:58 | |
| Shrews | yup | 18:58 |
| cloudnull | yes. its a vxlan type network | 18:59 |
| mordred | k. so, that's going to screw with our detection of the external qualities of the network | 18:59 |
| mordred | up until now, provider networks have only ever been public | 18:59 |
| mordred | so this is marking your ADMIN_TEST_NET network as an external network | 18:59 |
| clarkb | note we do not use provider networks in our tenant/project/wordoftheday | 19:00 |
| fungi | does anybody running git-review natively on mac osx happen to know why rprakash might see git saying review is not a valid subcommand even after doing a pip install git-review? | 19:00 |
| clarkb | we have a single neutron private network thingy thta dhcps to VMs then floating IP those things | 19:00 |
| Shrews | fungi: because OS X | 19:00 |
| mordred | cloudnull: is u'provider:physical_network': None what marks this as a private non-externally routable provider network? | 19:01 |
| fungi | Shrews: someone added a subsection in our developer guide about installing git review on osx and other unix variants, but it just mentions the pip install step | 19:01 |
| *** _sarob has quit IRC | 19:01 | |
| *** salv-orlando has quit IRC | 19:01 | |
| clarkb | fungi: rprakash if I were guessing the git install and the python git-review install do not share lookup paths so gits command automagic isn't working | 19:02 |
| fungi | so curious if there's more to do specifically on osx | 19:02 |
| *** rkukura_ has joined #openstack-infra | 19:02 | |
| cloudnull | mordred: yes . | 19:02 |
| mgagne | could it be that he has 2 git installed or 2 python installations and git isn't looking in the right python module path? | 19:02 |
| clarkb | but thats a huge guess from someone that has never owned an apple product | 19:02 |
| *** rkukura has quit IRC | 19:02 | |
| *** rkukura_ is now known as rkukura | 19:02 | |
| cloudnull | in the ml2 config that that item has a value it maps back to a physical device | 19:02 |
| cloudnull | **if that | 19:02 |
| *** ddieterly[away] has quit IRC | 19:02 | |
| Shrews | fungi: i do not know the answer, but that is how i have it installed and it WFM | 19:02 |
| cloudnull | otherwise its a private network | 19:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Be more precise in our detection of provider networks https://review.openstack.org/318880 | 19:03 |
| cloudnull | all of this is using the linuxbridge agent. | 19:03 |
| mordred | cloudnull: ^^ that should fix that bit | 19:03 |
| zaro | rprakash: it works fine for me, but i installed a while ago. | 19:03 |
| *** sarob has joined #openstack-infra | 19:03 | |
| mordred | and cause your external network to be reported properly | 19:03 |
| mordred | gha | 19:03 |
| zaro | rprakash: just did pip free on my mac and don't even see git-review. maybe homebrew? | 19:04 |
| mordred | internal network to be reported properly | 19:04 |
| mordred | sorry | 19:04 |
| cloudnull | testing nwo | 19:04 |
| *** claudiub has quit IRC | 19:04 | |
| cloudnull | meow | 19:04 |
| mordred | mmm. meow | 19:04 |
| * cloudnull cant type | 19:04 | |
| mordred | meow should always be tested | 19:04 |
| clarkb | my typing driver is defective | 19:04 |
| cloudnull | http://www.movienewsguide.com/super-troopers-poster-released/210901 -- right meow | 19:04 |
| Shrews | cloudnull: mordred: i was not aware that None was a valid value there. i don't think that's documented, or at least, i don't recall that in the docs | 19:05 |
| *** shamail has quit IRC | 19:05 | |
| rprakash | ok let me checkup paths and any env I have not set - thanks clarkb & fungi | 19:06 |
| cloudnull | Shrews: i dont think we're doing anything to define it as none i think if its undefined it is just set as none | 19:07 |
| cloudnull | but may be mistaken | 19:07 |
| *** kushal has joined #openstack-infra | 19:08 | |
| *** sdague has quit IRC | 19:08 | |
| zaro | rprakash: also if you plan to develop git-review on osx you'll need this change: https://review.openstack.org/#/c/285620/ | 19:08 |
| cloudnull | mordred: that works | 19:08 |
| mordred | cloudnull: woot | 19:09 |
| *** doug-fish has joined #openstack-infra | 19:09 | |
| rprakash | zaro let me look into it - thanks | 19:09 |
| *** sdague has joined #openstack-infra | 19:09 | |
| cloudnull | not that you want to see all this again http://cdn.pasteraw.com/8fhv1jvvdw54ntcmfha0ep968h2ie00 | 19:10 |
| *** salv-orlando has joined #openstack-infra | 19:10 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Reverse logic on key_word selection in uuid_to_int migration https://review.openstack.org/318886 | 19:12 |
| Shrews | kamsz: sorry, back to your question re: live migration in shade... | 19:13 |
| fungi | zaro: i think he's just trying to use git-review to propose a ne project addition into our gerrit, but good to keep in mind | 19:13 |
| mordred | cloudnull: woot. that output looks much more normal | 19:13 |
| Shrews | kamsz: currently supported, no. plans to support, not currently on our agenda | 19:13 |
| cloudnull | so i think that'll help quite abit with VMs not getting a FIP | 19:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Be more precise in our detection of provider networks https://review.openstack.org/318880 | 19:13 |
| Shrews | kamsz: but we happily accept patches :) | 19:13 |
| * cloudnull is asuming. | 19:14 | |
| mordred | Shrews: ^^ there is that but now with tests | 19:14 |
| mordred | kamsz: ++ to patches | 19:14 |
| rprakash | Kedars-Macbook:project-config pramchan$ whereis python /usr/bin/python Kedars-Macbook:project-config pramchan$ whereis git /usr/bin/git Kedars-Macbook:project-config pramchan$ whereis git-review | 19:15 |
| zaro | fungi: ok, thought there was some mention of development. anyways that makes me wonder how i'm able to use git-review at all now. | 19:16 |
| *** SumitNaiksatam has left #openstack-infra | 19:16 | |
| *** javeriak has quit IRC | 19:16 | |
| rprakash | whereis python /usr/bin/python whereis git /usr/bin/git whereis git-review (no response for this) | 19:16 |
| *** rbrndt has quit IRC | 19:17 | |
| fungi | my guess is that pip install on mac osx doesn't install entrypoints into the usual execution paths... dstufft: do you happen to know? | 19:17 |
| Shrews | mordred: don't we want an opposing check in the code below to include that provider net in the private nets? | 19:17 |
| mordred | Shrews: probably good idea | 19:18 |
| *** rbrndt has joined #openstack-infra | 19:18 | |
| Shrews | i mean, the logic there is hurting my head a bit, but i *think* that's sensible | 19:19 |
| kamsz | Shrews: thanks for the response :) | 19:21 |
| krotscheck | jeblair: I have a failure against ORD, the npm install run started around Thu May 19 19:02:04 UTC 2016, and it failed on fsevents | 19:23 |
| dstufft | fungi: depends on what your python thinks it should be | 19:23 |
| cloudnull | mordred: in testing that simple provider_net patch a few times all VMs are getting a FIP as expected. | 19:23 |
| mordred | cloudnull: woot! that's progress | 19:24 |
| cloudnull | I've only been testing w/ 25 at a time | 19:24 |
| pabelanger | mordred: cloudnull: Just reading backscrool, yay for progress | 19:24 |
| dstufft | fungi: something like `python -m sysconfig | head -n 14` should tell you the scripts dir | 19:24 |
| cloudnull | but it seems to be working | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Be more precise in our detection of provider networks https://review.openstack.org/318880 | 19:24 |
| pabelanger | backscroll* | 19:24 |
| mordred | Shrews: there it is with the inverse logic added to the internal network portion | 19:24 |
| cloudnull | I've also been testing using gnu-parallel instead of py multiprocessing to ensure I have mutual exclusion between the processes in every run. and with that patch so far so good. | 19:24 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Display the meeting_id if present https://review.openstack.org/318894 | 19:25 |
| *** ddieterly has joined #openstack-infra | 19:26 | |
| *** SumitNaiksatam has joined #openstack-infra | 19:28 | |
| cloudnull | pabelanger: I've now done 10 runs using 25 VMs and no leaked FIPS, no VMs without a FIP. | 19:29 |
| cloudnull | i dont know if that solves it for me, or my cloud is just being happier but, progress. | 19:29 |
| clarkb | cloudnull: mordred: but provider networks are not involved with our issue right? | 19:29 |
| clarkb | (sounds like we may have cuaght and fixed a different thing so yay but do we expect this to be related to nodepool's use of shade and osic?) | 19:29 |
| pabelanger | cloudnull: very nice | 19:30 |
| cloudnull | clarkb: i think they are inadvertently. | 19:30 |
| *** Guest76268 is now known as mfisch | 19:30 | |
| *** baoli has joined #openstack-infra | 19:30 | |
| *** mfisch has quit IRC | 19:30 | |
| *** mfisch has joined #openstack-infra | 19:30 | |
| Shrews | mordred: that doesn't seem right | 19:30 |
| cloudnull | the server dict was getting rewritten with an accessIPv4 address that was a private net | 19:30 |
| cloudnull | so it never requested a FIP | 19:31 |
| * cloudnull atleast thats how its been in my tests. | 19:31 | |
| clarkb | cloudnull: even when we don't use provider networks? | 19:31 |
| clarkb | cloudnull: the way our tenant is setup is a normal neutron netowrk thing with dhcp then floating IPs to that | 19:31 |
| *** SumitNaiksatam has quit IRC | 19:32 | |
| *** nadya has joined #openstack-infra | 19:33 | |
| mordred | right. this patch does not fix our case | 19:33 |
| *** bhunter71 has quit IRC | 19:33 | |
| *** banix has quit IRC | 19:33 | |
| mordred | but it does fix an issue that cloudnull's testing uncovered | 19:33 |
| *** hashar has quit IRC | 19:34 | |
| *** rprakash has quit IRC | 19:34 | |
| mordred | which was an incorrect assumption about what we learn from provider network metadata | 19:34 |
| mordred | Shrews: no? did I get the boolean's wrong? | 19:34 |
| cloudnull | clarkb: even though the network is a tenant network the settings for provider networks is present | 19:34 |
| *** sridhar_ram_ has quit IRC | 19:35 | |
| cloudnull | you should see something like this http://cdn.pasteraw.com/qd9wvmxawqzwvjlzodrg0bzo8j9fkb3 | 19:35 |
| Shrews | mordred: shouldn't that be OR before the provider check? | 19:35 |
| Shrews | mordred: so router:external == False OR provider:physical_network == None | 19:36 |
| cloudnull | clarkb: nevermind as tenant you dont see that op | 19:36 |
| cloudnull | as an admin you do | 19:36 |
| mordred | Shrews: no - I think both of them will always be that for internal networks | 19:37 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Remove the compass meeting https://review.openstack.org/318899 | 19:37 |
| mordred | if it's router:external == True, it's external. if it's provider:physical_network == vlan - it's external | 19:37 |
| cloudnull | Shrews: which may explain the discrepency in the return output you were talking about. | 19:37 |
| Shrews | mordred: but what if router:external is not present? | 19:37 |
| mordred | cloudnull: yah | 19:37 |
| *** tpsilva has quit IRC | 19:37 | |
| mordred | Shrews: router:external should always be present, no? | 19:37 |
| Shrews | mordred: dunno, but we have a check for it being present there | 19:37 |
| mordred | Shrews: no - that's covered | 19:37 |
| mordred | Shrews: that's in the ()s | 19:38 |
| mordred | Shrews: although we could change that to be a get with a default value of False if that would be clearer | 19:38 |
| Shrews | mordred: right, but if it is NOT present, we do an AND, which will fail | 19:38 |
| mordred | Shrews: http://paste.openstack.org/show/497797 how about that? | 19:39 |
| Shrews | this sounds like an excellent opportunity for a test case | 19:39 |
| mordred | Shrews: ++ | 19:39 |
| mordred | Shrews: well, a test case, ut also if we're arguing about the logic in the statement, then the code is not clear enough :) | 19:39 |
| Shrews | mordred: totes | 19:40 |
| mordred | Shrews: do you like the pasted version better for reading clarity | 19:40 |
| Shrews | mordred: but i think your paste solves the problem | 19:40 |
| mordred | ? | 19:40 |
| mordred | cool | 19:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Be more precise in our detection of provider networks https://review.openstack.org/318880 | 19:40 |
| *** _deadnull is now known as deadnull | 19:41 | |
| Shrews | also, i wasn't arguing. i was 'splainin' | 19:41 |
| *** Douhet has quit IRC | 19:41 | |
| Shrews | :-P | 19:41 |
| *** ams__ has joined #openstack-infra | 19:42 | |
| *** Douhet has joined #openstack-infra | 19:42 | |
| zaro | fungi, dstufft : thanks, looks like everything is pointing to a homebrew install | 19:43 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Remove cue meeting https://review.openstack.org/318908 | 19:43 |
| *** Apoorva_ has quit IRC | 19:43 | |
| mordred | Shrews: potato. potato | 19:43 |
| *** bhunter71 has joined #openstack-infra | 19:45 | |
| Shrews | mmm, potatoes | 19:46 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Remove hierarchical-multitenancy-meeting https://review.openstack.org/318912 | 19:46 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: fix the way we gather metadata https://review.openstack.org/318913 | 19:47 |
| phschwartz | Am I correct to assume that the cloud var for a provider in the nodepool config is used to associate a provider with an entry in clouds.yaml? | 19:48 |
| Shrews | phschwartz: yes | 19:48 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Remove mercador meeting https://review.openstack.org/318915 | 19:48 |
| *** chem` has joined #openstack-infra | 19:48 | |
| *** chem has quit IRC | 19:50 | |
| jroll | we still don't support constraints in post jobs yet, right? | 19:51 |
| clarkb | jroll: correct, jeblair mentioned he would review the change necessary to do that and I should probably go and rereview it | 19:51 |
| *** ddieterly is now known as ddieterly[away] | 19:51 | |
| jroll | clarkb: oh neat, thanks | 19:52 |
| clarkb | jroll: https://review.openstack.org/#/c/293194/ fwiw | 19:52 |
| jroll | cool | 19:53 |
| *** gomarivera has joined #openstack-infra | 19:53 | |
| openstackgerrit | David Lyle proposed openstack-infra/project-config: Update horizon dsvm tempest job https://review.openstack.org/314795 | 19:54 |
| ildikov | fungi: I have one more question regarding the custom image | 19:54 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Add meeting_id to poppy and improve the description https://review.openstack.org/318918 | 19:55 |
| ildikov | fungi: by discussing our case with jroll I realised that our image is already built and ready and we don't need to rebuild it all the time | 19:55 |
| *** e0ne has joined #openstack-infra | 19:56 | |
| ildikov | fungi: so basically the question would be whether there is a guideline on how to upload an image to tarballs.openstack.org? | 19:56 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Remove the trainers-team-meeting https://review.openstack.org/318920 | 19:57 |
| *** mhickey has joined #openstack-infra | 19:57 | |
| *** flepied has quit IRC | 19:58 | |
| *** gomarivera has quit IRC | 19:58 | |
| fungi | ildikov: that server is more for publishing things built in our automation. if the image already exists somewhere, then you should just be able to add the url for that in devstack (or a devstack plug-in) | 19:58 |
| fungi | ildikov: though if your testing depends on something in that image, don't you also want to be able to confirm newer versions of that thing don't break your use case? | 19:59 |
| fungi | ildikov: in which case periodically rebuilding it in the ci to update it might still make sense | 20:00 |
| ildikov | fungi: well, the change in it is to enable the watchdog device | 20:00 |
| fungi | ildikov: e.g. modprobe (or custom compiled kernel)? | 20:00 |
| ildikov | fungi: I think it's modprobe | 20:01 |
| fungi | ildikov: i assume this is the builtin kernel watchdog driver you're talking about anyway | 20:01 |
| ildikov | fungi: yeap | 20:01 |
| *** gyee has joined #openstack-infra | 20:02 | |
| *** deadnull has quit IRC | 20:02 | |
| ildikov | fungi: we didn't do too much magic, at least not for now | 20:02 |
| fungi | curious why your tempest test couldn't just ssh in and modprobe it in that case... the normal images used for existing tempest jobs don't include the kernel module for it? | 20:02 |
| *** sarob has quit IRC | 20:03 | |
| *** rhallisey has quit IRC | 20:03 | |
| *** Apoorva has joined #openstack-infra | 20:05 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: poppy: Add meeting_id, chair name, & better description https://review.openstack.org/318918 | 20:05 |
| *** rhallisey has joined #openstack-infra | 20:05 | |
| *** kushal has quit IRC | 20:05 | |
| ildikov | it wasn't me who played with it, but the guys were complaining about some black list modules that they had to remove from there in order to be able to load the driver finally | 20:05 |
| *** Sukhdev has quit IRC | 20:06 | |
| *** ddieterly[away] is now known as ddieterly | 20:06 | |
| ildikov | I also wasn't sure how much they tolerate to play with the guest image after boot in the test | 20:07 |
| *** sarob has joined #openstack-infra | 20:07 | |
| ildikov | fungi: ^^ | 20:07 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Add chair IRC nicks to Ironic meetings https://review.openstack.org/318924 | 20:08 |
| *** amrith is now known as _amrith_ | 20:09 | |
| *** Apoorva has quit IRC | 20:09 | |
| clarkb | ya I think having the test do what it needs is preferable to relying on another image | 20:10 |
| clarkb | or maybe update the single image in use to work in this case as well | 20:11 |
| fungi | ildikov: right, if it was just a question of handling the modprobe blacklist entries, that should also be pretty doable over ssh. we're probably not getting a complete picture of the challenges they ran into but it would be good to discuss the results of that exercise first rather than just writing it off | 20:12 |
| *** HeOS has joined #openstack-infra | 20:13 | |
| *** banix has joined #openstack-infra | 20:13 | |
| ildikov | fungi: sure, thanks for the heads up | 20:13 |
| *** matrohon has joined #openstack-infra | 20:13 | |
| *** denisra has joined #openstack-infra | 20:14 | |
| *** kgiusti has quit IRC | 20:15 | |
| *** doug-fish has quit IRC | 20:15 | |
| *** oanson has quit IRC | 20:15 | |
| *** markvoelker has joined #openstack-infra | 20:15 | |
| ildikov | fungi: I agree it would be better to live with the already available image so I will double check | 20:16 |
| ildikov | fungi: thanks for the help! | 20:16 |
| fungi | ildikov: you're welcome! | 20:18 |
| *** markvoelker has quit IRC | 20:20 | |
| ildikov | clarkb: well, from the test case PoV the another image sounds better for me, but if we consider the dependencies and workflows to have one the ssh option (if works) is back in the game | 20:20 |
| clarkb | ildikov: the problem with that is it means that you double your test infrastructure requirements for a single test | 20:21 |
| *** savihou has quit IRC | 20:21 | |
| clarkb | which is probably fine if necessary but likely not necessary in this case | 20:22 |
| *** savihou has joined #openstack-infra | 20:22 | |
| *** kushal has joined #openstack-infra | 20:22 | |
| *** akshai has quit IRC | 20:22 | |
| *** Apoorva has joined #openstack-infra | 20:23 | |
| ildikov | clarkb: yeah, I meant that so I agree | 20:23 |
| *** ayoung has quit IRC | 20:24 | |
| ildikov | clarkb: so I might need to annoy tempest people as well at a certain point, but I hope it will not be necessary :) | 20:24 |
| ildikov | clarkb: thanks | 20:25 |
| *** gomarivera has joined #openstack-infra | 20:25 | |
| pabelanger | clarkb: have a moment to help debug a mirror server? (mirror.tripleo-test-cloud-rh1.regionone.openstack.org). Having problems getting AFS to work correctly. 66.187.229.129 is the IP | 20:26 |
| clarkb | pabelanger: in a few, trying to finish up zuul cloner review | 20:26 |
| pabelanger | clarkb: sure, | 20:26 |
| *** mhickey has quit IRC | 20:27 | |
| clarkb | error handling has changed which is always fun to double check :) | 20:27 |
| *** e0ne has quit IRC | 20:30 | |
| *** ddieterly is now known as ddieterly[away] | 20:30 | |
| *** gomarivera has quit IRC | 20:30 | |
| *** Ravikiran_K has quit IRC | 20:30 | |
| clarkb | jeblair: I reviewed the post pipeline zuul-cloner support change and it has two +2's but I didn't approve as you indicated you would review it too. | 20:31 |
| clarkb | pabelanger: ok what is the problem? | 20:31 |
| *** Sukhdev has joined #openstack-infra | 20:32 | |
| *** daemontool has joined #openstack-infra | 20:32 | |
| pabelanger | clarkb: it doesn't look like /afs is responding on the server | 20:33 |
| pabelanger | clarkb: so, I am not sure if it is setup properly | 20:33 |
| clarkb | pabelanger: is it running ubuntu trusty? | 20:33 |
| pabelanger | clarkb: yes | 20:33 |
| pabelanger | clarkb: I used the cloud-image | 20:33 |
| clarkb | if so check the kernel version, iirc the hardware support newer kernel didn't dkms with afs package properly | 20:33 |
| *** ewilson has quit IRC | 20:33 | |
| *** piet has quit IRC | 20:35 | |
| *** HeOS has quit IRC | 20:35 | |
| *** bhunter71 has quit IRC | 20:36 | |
| pabelanger | clarkb: openafs modules is compiled and installed. There is also afs processes running | 20:36 |
| clarkb | pabelanger: btw that name doesn't resolve for me, have an ip addr? | 20:37 |
| pabelanger | clarkb: however ls /afs/openstack.org currently hangs. Wonder if it is downloading bits | 20:37 |
| pabelanger | clarkb: 66.187.229.129 | 20:37 |
| clarkb | oh you printed that earlire too, derp | 20:37 |
| clarkb | kernel version is the one I expect (3.13.something) | 20:38 |
| pabelanger | I see traffic between afs01.dfw.o.o, but pretty slow | 20:38 |
| *** doug-fish has joined #openstack-infra | 20:38 | |
| openstackgerrit | Matt Mulsow proposed openstack-infra/glean: Add support for bonded interfaces with a static IP. https://review.openstack.org/318940 | 20:39 |
| *** dmk0202 has joined #openstack-infra | 20:39 | |
| *** hemanthm is now known as hemanthm|afk | 20:40 | |
| *** tqtran has quit IRC | 20:40 | |
| clarkb | pabelanger: I think this is beyond my afs debugging skills | 20:40 |
| pabelanger | clarkb: okay, thanks for looking | 20:41 |
| *** jamesmcarthur has joined #openstack-infra | 20:41 | |
| *** flepied has joined #openstack-infra | 20:42 | |
| *** julim has quit IRC | 20:42 | |
| *** julim has joined #openstack-infra | 20:42 | |
| *** doug-fish has quit IRC | 20:43 | |
| *** nwkarste_ has joined #openstack-infra | 20:43 | |
| *** doug-fish has joined #openstack-infra | 20:43 | |
| *** yamamoto has joined #openstack-infra | 20:44 | |
| *** nwkarste_ has quit IRC | 20:44 | |
| *** nwkarste_ has joined #openstack-infra | 20:45 | |
| *** mamulsow has joined #openstack-infra | 20:45 | |
| clarkb | pabelanger: mtr doesn't show anything imeedaitely funny between there and afs01.dfw.openstack.org | 20:45 |
| clarkb | pabelanger: there is NAT in the way though | 20:45 |
| zigo | pabelanger: Hi there! Could you help me to figure out why I get the gating result as UNSTABLE ? https://review.openstack.org/#/c/318500/ | 20:47 |
| *** nwkarsten has quit IRC | 20:47 | |
| zigo | I don't get what's going on ... | 20:47 |
| clarkb | pabelanger: actually if I mtr the other direction they look like different paths? | 20:47 |
| clarkb | pabelanger: which may possibly make NAT unhappy | 20:48 |
| *** salv-orl_ has joined #openstack-infra | 20:48 | |
| clarkb | and prevent a potential stateful treatment of the udp packets as broken | 20:48 |
| clarkb | fungi: ^ | 20:48 |
| *** kushal has quit IRC | 20:48 | |
| pabelanger | clarkb: interesting | 20:49 |
| *** zzelle_ has quit IRC | 20:49 | |
| *** yaume has quit IRC | 20:50 | |
| *** baoli has quit IRC | 20:50 | |
| *** jamesmcarthur has quit IRC | 20:50 | |
| *** tonytan4ever has quit IRC | 20:51 | |
| *** salv-orlando has quit IRC | 20:51 | |
| clarkb | pabelanger: http://paste.openstack.org/show/497803/ | 20:51 |
| pabelanger | zigo: I'll have a look in a bitt | 20:51 |
| zigo | Cheers. | 20:51 |
| *** jamesmcarthur has joined #openstack-infra | 20:52 | |
| *** zqfan has quit IRC | 20:53 | |
| clarkb | pabelanger: sometimes routers/firewalls pretend udp is stateful and get over zealous about where the packets come from, I do not know if that is the case here with afs | 20:53 |
| clarkb | pabelanger: but the paths do seem to differ depending on direction | 20:53 |
| *** cody-somerville has joined #openstack-infra | 20:55 | |
| *** |-paul-| has joined #openstack-infra | 20:55 | |
| *** zz_dimtruck is now known as dimtruck | 20:57 | |
| fungi | clarkb: pabelanger: yes, afaik afs gets unhappy with complex address translation setups | 20:57 |
| *** BigWillie has quit IRC | 20:58 | |
| *** tqtran has joined #openstack-infra | 21:00 | |
| *** ddieterly[away] is now known as ddieterly | 21:00 | |
| *** chem` has quit IRC | 21:02 | |
| *** jamesmcarthur has quit IRC | 21:02 | |
| clarkb | pabelanger: I am not sure how to debu that further without better access to the machine running NAT | 21:02 |
| clarkb | pabelanger: maybe we can recruit the tripleoers to do some packet captures there? | 21:02 |
| pabelanger | clarkb: sure, I'll reach out to them in the morning. But think that is a good place to start | 21:02 |
| *** rhallisey has quit IRC | 21:03 | |
| pabelanger | clarkb: fungi: Thanks for the help. Stepping away for some food | 21:04 |
| openstackgerrit | Kaiyan Sheng proposed openstack-infra/project-config: [Monasca]Add Tempest Jobs as non-voting for monasca repos https://review.openstack.org/316293 | 21:06 |
| *** _ari_ has quit IRC | 21:07 | |
| *** rebase has joined #openstack-infra | 21:08 | |
| *** jamesmcarthur has joined #openstack-infra | 21:08 | |
| phschwartz | Shrews: The net-label in the networks section of a provider for nodepool is just the name of a network to use correct? | 21:08 |
| *** trown is now known as trown|outtypewww | 21:09 | |
| fungi | phschwartz: yes | 21:09 |
| phschwartz | fungi: ty | 21:09 |
| clarkb | and you can specify more than one if you want hosts to have multiple nics | 21:09 |
| *** weshay has quit IRC | 21:10 | |
| *** raildo is now known as raildo-afk | 21:10 | |
| *** yamahata has quit IRC | 21:11 | |
| *** jamesmcarthur has quit IRC | 21:13 | |
| nwkarste_ | does nodepool support v3 authentication? | 21:14 |
| *** vhosakot has quit IRC | 21:14 | |
| clarkb | nwkarste_: yes. More specifically it pushes that down into the client libs so as long as your client libs are new enough v3 should just work | 21:15 |
| clarkb | we run at least one cloud with v3 | 21:15 |
| *** ihrachys has quit IRC | 21:16 | |
| *** ldnunes has quit IRC | 21:16 | |
| *** ihrachys has joined #openstack-infra | 21:17 | |
| clarkb | nwkarste_: you may need to use os-client-config with a cloud specified in nodepoo.yaml though in order to get the domain stuff passed in properly | 21:18 |
| *** vhosakot has joined #openstack-infra | 21:18 | |
| jeblair | clarkb, Nakato: i don't understand why we're adding the project_revision argument | 21:19 |
| *** cody-somerville has quit IRC | 21:19 | |
| *** gomarivera has joined #openstack-infra | 21:20 | |
| nwkarste_ | clarkb: yep i've set that up. this is the error i'm getting http://paste.openstack.org/show/497806/ | 21:20 |
| jeblair | jhesketh: ^ | 21:20 |
| clarkb | jeblair: that is what specifies that specific revision to checkout which is the data we get in post | 21:20 |
| *** vhosakot has quit IRC | 21:20 | |
| *** julim has quit IRC | 21:20 | |
| jeblair | clarkb: that's in ZUUL_NEWREV | 21:20 |
| jeblair | clarkb: er ZUUL_REF | 21:20 |
| jeblair | well, both in a way | 21:20 |
| clarkb | nwkarste_: is the domain name or domain id specified in your clouds.yaml? | 21:21 |
| jeblair | clarkb: ggp logic is "if zuul_newrev, checkout zuul_ref". i'm wondering why that is unsatisfactory | 21:21 |
| clarkb | jeblair: I am guessing to make it more genericer? I know tripleo was looking at something that was basically z-c at one point that didn't require a full zuul env | 21:21 |
| clarkb | we could likely support both things though | 21:22 |
| nwkarste_ | clarkb: i have the domain_name | 21:22 |
| *** dmk0202 has quit IRC | 21:22 | |
| clarkb | nwkarste_: can you sahre your clouds.yaml and remove the sensitive bits? | 21:22 |
| clarkb | nwkarste_: let me find an example of ours for you too | 21:23 |
| *** thorst_ has quit IRC | 21:23 | |
| *** piet has joined #openstack-infra | 21:23 | |
| jeblair | clarkb: i'm pretty sure only 5 people understand our 3-level override system today, so i'm hesitant to add another level if it's not needed | 21:24 |
| *** vhosakot has joined #openstack-infra | 21:24 | |
| nwkarste_ | clarkb: http://paste.openstack.org/show/497807/ | 21:24 |
| clarkb | nwkarste_: https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/puppetmaster/all-clouds.yaml.erb#n113 is what ours looks like using v3 | 21:24 |
| *** gomarivera has quit IRC | 21:24 | |
| clarkb | nwkarste_: looks like project_domain_name should be part of the auth dict | 21:24 |
| *** matrohon has quit IRC | 21:25 | |
| nwkarste_ | clarkb: i will give that a shot | 21:25 |
| *** thorst_ has joined #openstack-infra | 21:25 | |
| clarkb | jeblair: thats fair, to be more like the normal operations of z-c it would accept both the env var and a cli arg right? do you just want to rename the cli arg to match the env var? | 21:25 |
| clarkb | and I guess there would be a singleton not many values? | 21:26 |
| *** cody-somerville has joined #openstack-infra | 21:26 | |
| openstackgerrit | David Lyle proposed openstack-infra/project-config: Update horizon dsvm tempest job https://review.openstack.org/314795 | 21:27 |
| jeblair | clarkb: i think those env vars are already supported. i get the logic behind doing the "project_revisions[project]=newrev" bit internally -- that's explained in the review history as wanting to avoid lots of errors if zc is used for multiple projects; i think i'm just thinking that exposing it as an option is premature without a really good use case | 21:27 |
| clarkb | jeblair: z-c supports that with branches though right? | 21:27 |
| jeblair | clarkb: mostly because i *really* don't want to have to unwind it when someone comes through with "but when i set --project-revision foo=bar my job works!" :) | 21:27 |
| clarkb | I think its trying to match that existing functionality with the more direct ref, maybe they get collapsed into a single thing | 21:28 |
| *** eventingmonkey has quit IRC | 21:28 | |
| jeblair | clarkb: yeah, there's a project-branches argument to handle the cross-branch testing case | 21:28 |
| *** thorst_ has quit IRC | 21:29 | |
| *** mamulsow has quit IRC | 21:30 | |
| *** vhosakot has quit IRC | 21:30 | |
| jeblair | clarkb: left comment on change; lemme know if that makes sense | 21:31 |
| nwkarste_ | clarkb: same issue. here's the current clouds.yaml http://paste.openstack.org/show/497810/ | 21:32 |
| *** hdd has joined #openstack-infra | 21:32 | |
| *** eharney has quit IRC | 21:33 | |
| clarkb | nwkarste_: maybe you need to set the user domain name too? | 21:33 |
| clarkb | nwkarste_: you'll see that in our clouds.yaml I linked | 21:33 |
| mordred | mwhahaha: yes. you also need user_domain_name | 21:33 |
| mordred | which also needs to be in the auth dict in clouds.yaml | 21:33 |
| lifeless | kashyap: guestfish needs the ability to make a vm, which is a root-like ability, but still the argument for it doesn't make sense for dib applications | 21:34 |
| *** jang has quit IRC | 21:34 | |
| *** chem` has joined #openstack-infra | 21:34 | |
| *** hdd has left #openstack-infra | 21:34 | |
| clarkb | jeblair: makes sense to me, lets see what Nakato says | 21:34 |
| mordred | nwkarste_: ^^ I meant that for you, not mwhahaha | 21:34 |
| *** elo has joined #openstack-infra | 21:34 | |
| mordred | nwkarste_: you can use the same value as you have for project_domain_name most likely | 21:35 |
| *** baoli has joined #openstack-infra | 21:36 | |
| nwkarste_ | mordred: thanks | 21:36 |
| phschwartz | Has there been changes recently to the smtp notifier in zuul? My layout's that use it are failing after an update. | 21:37 |
| *** doug-fish has quit IRC | 21:38 | |
| clarkb | phschwartz: possibly the connections changes which merged about 6 months ago | 21:39 |
| clarkb | not sure if that is recent :) | 21:39 |
| *** sdake has quit IRC | 21:39 | |
| *** sridhar_ram has joined #openstack-infra | 21:39 | |
| phschwartz | clarkb: no, what ever this is has to be in the past 2 months | 21:40 |
| phschwartz | clarkb: I was hoping someone knew off hand, I will start code surfing till I find the issue | 21:41 |
| fungi | phschwartz: what is the nature of the failure you're observing? | 21:41 |
| fungi | is it a parsing error, or a runtime issue? | 21:41 |
| fungi | is there a particular error message associated with it, or is the issue simply a change in observed behavior? | 21:42 |
| *** vhosakot has joined #openstack-infra | 21:42 | |
| *** aimeeu has joined #openstack-infra | 21:43 | |
| *** doug-fish has joined #openstack-infra | 21:44 | |
| *** thorst_ has joined #openstack-infra | 21:45 | |
| *** lakshmiS has quit IRC | 21:45 | |
| *** kzaitsev_mb has quit IRC | 21:47 | |
| *** mriedem has quit IRC | 21:47 | |
| phschwartz | fungi: config parsing error | 21:48 |
| *** ewilson has joined #openstack-infra | 21:48 | |
| *** doug-fish has quit IRC | 21:49 | |
| phschwartz | fungi: http://paste.openstack.org/show/497812/ | 21:49 |
| phschwartz | That is the trace while testing the config. | 21:49 |
| *** thorst_ has quit IRC | 21:49 | |
| *** aeng has joined #openstack-infra | 21:50 | |
| *** doug-fish has joined #openstack-infra | 21:50 | |
| phschwartz | fungi: And even more interesting, a --version actually says "Zuul version: 2.1.1.dev169" on both the running system and the new one | 21:50 |
| fungi | phschwartz: what does that pipeline's config look like? | 21:51 |
| fungi | do you have keys under smtp: other than from:, to: and subject:? | 21:52 |
| *** Sukhdev has quit IRC | 21:53 | |
| phschwartz | fungi: here is the first entry from pipelines that it is complaining about. http://paste.openstack.org/show/497813/ | 21:53 |
| aimeeu | Looking for help - I need to delete a pad on etherpad.openstack.org - do I have to open a bug so an administrator can do that? | 21:53 |
| *** ewilson has quit IRC | 21:54 | |
| *** doug-fish has quit IRC | 21:54 | |
| fungi | phschwartz: what's the next line after that? is it maybe incorrectly indented too far? (perhaps after a comment line?) | 21:56 |
| *** Sukhdev has joined #openstack-infra | 21:56 | |
| *** ewilson has joined #openstack-infra | 21:56 | |
| fungi | aimeeu: does it contain sensitive or legally restricted data? | 21:57 |
| *** daemontool has quit IRC | 21:57 | |
| fungi | phschwartz: another possibility... different distro/release? we've noticed some validation differences between precise and trusty | 21:58 |
| aimeeu | fungi: my supervisor feels it contains sensitive data | 21:58 |
| *** ewilson has quit IRC | 21:59 | |
| phschwartz | fungi: looking at the next line, and both are 14.04 | 21:59 |
| aimeeu | fungi: so I need the history erased | 21:59 |
| fungi | aimeeu: i'm about to step out for a while to eat, would need to take some time revisiting how i've deleted pads there in the past since it's probably been a couple years. one of our other sysadmins may recall the magic involved in doing that though | 22:00 |
| phschwartz | fungi: next line is a blank line, line after that is a propperly indented "- name: test" for another pipeline | 22:00 |
| *** flepied1 has joined #openstack-infra | 22:00 | |
| *** Swami has quit IRC | 22:01 | |
| fungi | phschwartz: yep, i'm out of ideas then. i guess recheck current assumptions | 22:01 |
| phschwartz | fungi: I am going to start from a clean layout and build it up and see what happens | 22:01 |
| fungi | phschwartz: you indicate that the systems and configs should be identical, but obviously there's some difference which matters. when i get in that situation i usually discover i've made an incorrect assumption in that regard | 22:02 |
| openstackgerrit | Kaiyan Sheng proposed openstack-infra/project-config: [Monasca]Add Tempest Jobs as non-voting for monasca repos https://review.openstack.org/316293 | 22:02 |
| *** ewilson has joined #openstack-infra | 22:02 | |
| aimeeu | fungi: I found the instructions but an admin needs to run the command https://docs.openstack.org/infra/system-config/etherpad.html | 22:02 |
| *** camunoz_ has joined #openstack-infra | 22:02 | |
| *** flepied has quit IRC | 22:02 | |
| *** IlyaG has quit IRC | 22:03 | |
| fungi | aimeeu: oh! i guess i documented that after all | 22:03 |
| fungi | git blame says i added that 3 years ago | 22:03 |
| fungi | here's hoping it still works with current etherpad versions | 22:04 |
| fungi | aimeeu: feel free to privately /msg me the pad url | 22:04 |
| phschwartz | fungi: I copied the layout over from one to the other, only difference is when puppet apply was run (3 months apart) | 22:04 |
| openstackgerrit | Kaiyan Sheng proposed openstack-infra/project-config: [Monasca]Add Tempest Jobs as non-voting for monasca repos https://review.openstack.org/316293 | 22:04 |
| aimeeu | fungi: thanks - will do! | 22:04 |
| *** ramishra has quit IRC | 22:05 | |
| *** edmondsw has quit IRC | 22:06 | |
| nwkarste_ | clarkb: still getting the expecting domain in project error. i've formatted my clouds.yaml to look like your example http://paste.openstack.org/show/497814/ | 22:06 |
| *** ramishra has joined #openstack-infra | 22:06 | |
| clarkb | nwkarste_: and you are using the domain name and not id there? | 22:07 |
| *** nadya has quit IRC | 22:07 | |
| *** ddieterly is now known as ddieterly[away] | 22:07 | |
| nwkarste_ | yeah. i was able to glance image-list using those same credentials | 22:08 |
| clarkb | glance supports clouds.yaml? | 22:08 |
| fungi | aimeeu: it's deleted successfully | 22:08 |
| clarkb | nwkarste_: can you use openstackclient with it? | 22:08 |
| *** ayoung has joined #openstack-infra | 22:08 | |
| fungi | aimeeu: let me know if you need any further help with that situation | 22:08 |
| aimeeu | fungi: thank you! you've saved my job (well, not that dire but you understand) | 22:09 |
| nwkarste_ | clarkb: no but i used those credentials in the call | 22:09 |
| *** jamesmcarthur has joined #openstack-infra | 22:09 | |
| clarkb | nwkarste_: right but these are completely different code ptahs, try with openstackclient as they should be similar there | 22:09 |
| fungi | aimeeu: i've been in the same situation more than once, totally know how it feels! | 22:10 |
| clarkb | nwkarste_: `openstack --os-cloud nwkarste image list` | 22:10 |
| fungi | anyway, now i'm stepping out for an hour or so to find foodishness | 22:10 |
| fungi | bbl | 22:10 |
| *** amitgandhinz has quit IRC | 22:11 | |
| *** HeOS has joined #openstack-infra | 22:12 | |
| *** ewilson has quit IRC | 22:13 | |
| nwkarste_ | clarkb: that works | 22:13 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Add helper to find full slots and pep8 check https://review.openstack.org/317809 | 22:13 |
| *** ewilson has joined #openstack-infra | 22:13 | |
| *** jamesmcarthur has quit IRC | 22:14 | |
| *** ewilson has quit IRC | 22:15 | |
| *** ewilson has joined #openstack-infra | 22:16 | |
| clarkb | nwkarste_: I am not sure then, you can try turning on shade debug logging wich should trace all the requests, it gets noisy but will give you a lot more info | 22:16 |
| *** jamesmcarthur has joined #openstack-infra | 22:17 | |
| *** sridhar_ram has quit IRC | 22:17 | |
| clarkb | nwkarste_: also did you restart nodepool after making those edits? I do not know if the clouds.yaml changes are automaticall picked up by nodepool | 22:17 |
| nwkarste_ | clarkb: i didn't restart. i will try that first | 22:17 |
| *** jamesmcarthur has quit IRC | 22:18 | |
| *** sridhar_ram has joined #openstack-infra | 22:20 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:20 | |
| *** rcernin has quit IRC | 22:23 | |
| *** rossella_s has quit IRC | 22:23 | |
| clarkb | if it doesn't pick up clouds.yaml changes without a restart I think we would consider that a bug | 22:24 |
| *** rossella_s has joined #openstack-infra | 22:24 | |
| *** ewilson has quit IRC | 22:25 | |
| nwkarste_ | clarkb: restarting didn't work. i did get the error here though https://review.openstack.org/#/c/233475/2 | 22:26 |
| *** markvoelker has joined #openstack-infra | 22:27 | |
| clarkb | nwkarste_: ok, then I don't know what is going on. My best suggestion is try running with shade debug logging and review the logs of the requests to galnce | 22:27 |
| *** mixos has quit IRC | 22:28 | |
| *** jamesmcarthur has joined #openstack-infra | 22:28 | |
| *** markvoelker has quit IRC | 22:29 | |
| openstackgerrit | David Lyle proposed openstack-infra/project-config: Update horizon dsvm tempest job https://review.openstack.org/314795 | 22:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Estimate job runtimes internally https://review.openstack.org/318966 | 22:29 |
| *** ddieterly[away] is now known as ddieterly | 22:29 | |
| *** markvoelker has joined #openstack-infra | 22:29 | |
| jeblair | clarkb, Shrews, pabelanger, jhesketh: ^ we're going to want that if we don't want to lose a user-facing feature in zuulv2.5 | 22:29 |
| *** sdake has joined #openstack-infra | 22:30 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Estimate job runtimes internally https://review.openstack.org/318966 | 22:30 |
| jeblair | fungi: cendora! nice! | 22:31 |
| jeblair | fungi: (though, are you sure it isn't 'fentos' [the freshmaker!]?) | 22:31 |
| *** jpr has quit IRC | 22:32 | |
| *** savihou has quit IRC | 22:34 | |
| *** HeOS has quit IRC | 22:35 | |
| *** sdague has quit IRC | 22:39 | |
| *** |-paul-| has quit IRC | 22:40 | |
| jhesketh | Morning | 22:43 |
| jeblair | jhesketh: morning! i refreshed those 2 zuulv2.5 changes and added the above ^ | 22:44 |
| *** kzaitsev_mb has joined #openstack-infra | 22:44 | |
| *** doug-fish has joined #openstack-infra | 22:44 | |
| *** bhunter71 has joined #openstack-infra | 22:44 | |
| *** doug-fish has quit IRC | 22:44 | |
| jhesketh | jeblair: cool | 22:44 |
| jhesketh | Will take a look | 22:45 |
| *** sdake has quit IRC | 22:45 | |
| *** IlyaG has joined #openstack-infra | 22:47 | |
| *** IlyaG has quit IRC | 22:48 | |
| *** IlyaG_ has joined #openstack-infra | 22:48 | |
| *** jamesmcarthur has quit IRC | 22:49 | |
| *** ddieterly has quit IRC | 22:51 | |
| *** banix has quit IRC | 22:52 | |
| *** vhosakot has quit IRC | 22:55 | |
| *** ffrank has joined #openstack-infra | 22:55 | |
| *** sridhar_ram has quit IRC | 22:56 | |
| *** markvoelker has quit IRC | 22:58 | |
| *** hongbin has quit IRC | 23:01 | |
| *** dimtruck is now known as zz_dimtruck | 23:01 | |
| *** IlyaG_ has quit IRC | 23:03 | |
| *** rbrndt has quit IRC | 23:03 | |
| *** yolanda has quit IRC | 23:03 | |
| *** ihrachys has quit IRC | 23:04 | |
| *** sdake has joined #openstack-infra | 23:04 | |
| *** Sukhdev has quit IRC | 23:05 | |
| *** yolanda has joined #openstack-infra | 23:05 | |
| *** Sukhdev has joined #openstack-infra | 23:08 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Estimate job runtimes internally https://review.openstack.org/318966 | 23:09 |
| *** thiagop has quit IRC | 23:09 | |
| *** cloudtrainme has quit IRC | 23:11 | |
| *** Qiming has joined #openstack-infra | 23:11 | |
| *** jamesmcarthur has joined #openstack-infra | 23:12 | |
| *** salv-orl_ has quit IRC | 23:12 | |
| *** sdake has quit IRC | 23:14 | |
| *** sdake has joined #openstack-infra | 23:15 | |
| *** jamesmcarthur has quit IRC | 23:17 | |
| prometheanfire | ianw: please don't abandon things without contacting first | 23:20 |
| *** nwkarsten has joined #openstack-infra | 23:21 | |
| *** amotoki has quit IRC | 23:21 | |
| *** sdake has quit IRC | 23:21 | |
| prometheanfire | ianw: and it IS needed because of how the element dependencies are set up in nodepool | 23:21 |
| *** amotoki has joined #openstack-infra | 23:21 | |
| *** ayoung has quit IRC | 23:22 | |
| *** nwkarste_ has quit IRC | 23:23 | |
| *** Swami has joined #openstack-infra | 23:23 | |
| prometheanfire | ianw: openstack-repos/element-deps deps on node-devstack | 23:24 |
| *** nwkarsten has quit IRC | 23:25 | |
| ianw | prometheanfire: devstack isn't the right place to maintain package lists for non-devstack platforms. we should come up with some other ideas | 23:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: include PATH https://review.openstack.org/318975 | 23:26 |
| *** amotoki has quit IRC | 23:26 | |
| prometheanfire | ianw: the other option would be to split the prepare-node call out into it's own thing | 23:26 |
| *** gordc has quit IRC | 23:26 | |
| *** Sukhdev has quit IRC | 23:27 | |
| prometheanfire | ianw: I think that's what you were talking about when you added that dependency | 23:27 |
| prometheanfire | 6cab81fec4bc686b1f4ee58f56f890ef40d68517 for refrence | 23:27 |
| *** rebase has quit IRC | 23:27 | |
| prometheanfire | ianw: would that be acceptable? | 23:30 |
| *** shashank_hegde has quit IRC | 23:30 | |
| *** zz_dimtruck is now known as dimtruck | 23:31 | |
| jeblair | from ansbile: "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!" whoops. | 23:32 |
| jeblair | that, um, is probably going to happen a lot. | 23:33 |
| *** ams__ has quit IRC | 23:33 | |
| prometheanfire | you can tell it to ignore iirc | 23:34 |
| jeblair | yeah, looks like we want to set env var ANSIBLE_HOST_KEY_CHECKING=false | 23:34 |
| mgagne | [defaults]/host_key_checking = False too | 23:34 |
| *** kzaitsev_mb has quit IRC | 23:34 | |
| jeblair | ah, thx | 23:34 |
| mgagne | you can have a global ansible.cfg in your home | 23:35 |
| jeblair | oh, hrm, i have that set; i think this might be coming from something else... | 23:35 |
| jeblair | yeah, it's from synchronize, which is separate... | 23:35 |
| mgagne | anyhow: https://github.com/ansible/ansible/blob/devel/lib/ansible/constants.py#L251 | 23:36 |
| jeblair | oh, that's all configured correctly; it's just a warning (that i guess we're going to see a lot of) | 23:37 |
| *** dimtruck is now known as zz_dimtruck | 23:41 | |
| *** zqfan has joined #openstack-infra | 23:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: create parent directories for SCP https://review.openstack.org/318979 | 23:42 |
| mordred | jeblair: oh goodie | 23:42 |
| *** kzaitsev_mb has joined #openstack-infra | 23:43 | |
| prometheanfire | ianw: I'm thinking prepare-node does more then set up the jenkins user... | 23:43 |
| ianw | prometheanfire: i'm thinking about this at https://etherpad.openstack.org/p/gentoo-ci | 23:43 |
| jeblair | mordred: ya, we might need to do something about redirecting the known_hosts file to keep things from getting insane | 23:44 |
| mordred | jeblair: ++ | 23:44 |
| *** baoli has quit IRC | 23:44 | |
| mordred | jeblair: isn't there a combo of telling ssh to auto-accept but also pointing known_hosts at /dev/null so that it accepts but writes to bong and avoids warnings? | 23:45 |
| prometheanfire | ianw: right | 23:46 |
| mordred | jeblair: next time a job starts, could you pop the ip into channel? I wanna watch a job stream while it's active, not just after the fact | 23:46 |
| mordred | jeblair: unless you don't want to - then don't | 23:47 |
| notmorgan | mordred: StrictHostKeyChecking no | 23:47 |
| jeblair | mordred: you mean symlink the known_hosts file? | 23:47 |
| notmorgan | mordred: and -q | 23:47 |
| mordred | jeblair: yah | 23:47 |
| jeblair | notmorgan: stricthostkeychecking means 'write the key to the file but don't check it' | 23:47 |
| notmorgan | jeblair: right i don't think you can say "don't write it" | 23:47 |
| jeblair | notmorgan: so we'll end up with millions of keys in the file, and a warning each time we connect to any host | 23:47 |
| notmorgan | jeblair: or at least i didn't see an option | 23:47 |
| notmorgan | jeblair: symlink to /dev/null | 23:48 |
| notmorgan | jeblair: and -q makes it not warn | 23:48 |
| mordred | notmorgan: does -q have a .ssh/config value? | 23:48 |
| notmorgan | hm.. looking | 23:49 |
| *** ayoung has joined #openstack-infra | 23:49 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: add -q to rsync ssh command https://review.openstack.org/318982 | 23:49 |
| mordred | LogLevel QUIET | 23:49 |
| mordred | jeblair: ^^ | 23:49 |
| *** jamielennox is now known as jamielennox|away | 23:49 | |
| notmorgan | mordred: there | 23:49 |
| jeblair | that's actually where we're seeing the warnings (i think maybe ansible handles it the rest of the time -- but it's an issue there because we're using local_command) | 23:49 |
| notmorgan | mordred: alias with -q | 23:49 |
| mordred | ah | 23:49 |
| notmorgan | jeblair: ah. | 23:50 |
| mordred | jeblair: yes. that's it | 23:50 |
| jeblair | mordred, notmorgan: it would be nice to use known_hosts when zuul talks to gerrit (this won't happen in our setup because they are different zuul hosts, but it would in a zuul all-in-one) | 23:50 |
| notmorgan | jeblair: so. | 23:50 |
| jeblair | mordred, notmorgan: it would also be nice to use it when talking to static.o.o when we publish artifacts | 23:50 |
| notmorgan | jeblair: i have an option for that. give me a sec | 23:50 |
| notmorgan | jeblair: basically, set in sshconfig host patterns we want the knownhost file to be used | 23:51 |
| *** zz_dimtruck is now known as dimtruck | 23:51 | |
| notmorgan | jeblair: and then patterns we don't, set UserKnownHostFile to /dev/null where you don't want it used | 23:51 |
| notmorgan | jeblair: and stricthostkeychecking back on for those | 23:51 |
| jeblair | notmorgan: that sounds very workable, cool | 23:51 |
| *** johnny__ has quit IRC | 23:51 | |
| notmorgan | jeblair: do you have a pattern for hosts/hostnames that we use that we want this for? | 23:51 |
| notmorgan | i'll come up with an example then do some puppet/sysconfig things to make ti a deal | 23:52 |
| jeblair | notmorgan: not right now -- i think we can punt on this for the moment and do better in v3 | 23:52 |
| *** sdake has joined #openstack-infra | 23:52 | |
| notmorgan | the example is something i can have one of the lovely infra-root folks test in either case. | 23:52 |
| jeblair | (where it will be different anyway, and for the moment, i'm okay with jenkins equivalency (which is don't care)) | 23:52 |
| notmorgan | well, i mean.. do we always have .o.o as a hostname for things we careabout? | 23:52 |
| mordred | ++ | 23:52 |
| *** gildub has joined #openstack-infra | 23:52 | |
| *** mixos has joined #openstack-infra | 23:52 | |
| notmorgan | and then ips for everything else? | 23:52 |
| notmorgan | cause ... i can work with that easily | 23:52 |
| notmorgan | and we can carry it forward. | 23:53 |
| jeblair | notmorgan: possibly, for the moment. | 23:53 |
| jeblair | that could work yeah | 23:53 |
| notmorgan | ok cool i'll do something like that and test it here (quickly) then post a paste up and if it looks right, i'll figure out puppet-y things :) | 23:53 |
| notmorgan | should be a quick diversion to do. | 23:53 |
| jeblair | mordred: if you're around for 15 mins, i'll divert another few jobs to v25 and like you to a console | 23:54 |
| mordred | jeblair: woot! | 23:54 |
| openstackgerrit | David Lyle proposed openstack-infra/project-config: Update horizon dsvm tempest job https://review.openstack.org/314795 | 23:55 |
| *** cody-somerville has quit IRC | 23:56 | |
| *** dingyichen has joined #openstack-infra | 23:56 | |
| jeblair | mordred: nc 63.251.114.139 8088 | 23:59 |
| jeblair | mordred: nc 169.45.92.97 8088 | 23:59 |
| jeblair | that's a build:gate-keystone-python34-db and build:gate-tempest-dsvm-full respectively | 23:59 |
| mordred | woot! | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
