| mat128 | Using sip connection seems to work | 00:00 |
|---|---|---|
| fungi | jeblair: do you have a moment to check whether we're out of quota on the pstn again? | 00:00 |
| pabelanger | fungi: looks good. I see 17 people on the server | 00:00 |
| *** dfflanders has quit IRC | 00:00 | |
| pabelanger | fungi: I would say we are likely out of $$ in voip.ms | 00:00 |
| fungi | it's possible we just ate it up with a virtual sprint | 00:00 |
| mat128 | pabelanger: Google hangouts dialer telling me line is busy | 00:00 |
| mat128 | ahhh | 00:01 |
| *** ddieterly has joined #openstack-infra | 00:01 | |
| pabelanger | mat128: Right, the issue is not pbx.o.o, likely our PSTN provider. Using a SIP client would likely work | 00:01 |
| jpmaxman | busy | 00:01 |
| mat128 | pabelanger: I can confirm a SIP client works | 00:01 |
| nibalizer | hrm | 00:01 |
| nibalizer | so i think | 00:01 |
| *** camunoz has quit IRC | 00:01 | |
| nibalizer | its because $GEM_HOME/bin/bundle install --without system_tests | 00:01 |
| nibalizer | maybe | 00:02 |
| *** aeng has quit IRC | 00:02 | |
| nibalizer | if you look at the puppet-lint macro definition in project-config | 00:02 |
| *** sree has quit IRC | 00:02 | |
| mat128 | pabelanger: ok, I'll note it in our pad | 00:02 |
| *** sarob_ has quit IRC | 00:02 | |
| mat128 | pabelanger, fungi: thanks for your help :) | 00:03 |
| *** pahuang has quit IRC | 00:03 | |
| *** sree has joined #openstack-infra | 00:03 | |
| pabelanger | mat128: fungi: One day we'll have a webrtc client hosted by infra, making it easier for people not used to setting up a SIP client | 00:04 |
| pabelanger | one day | 00:04 |
| fungi | mat128: it'll likely be back in service once we top up the account. i know clarkb was working with scott raschke (finance for the foundation) to set up direct billing for that so our community members didn't have to keep topping it up and expensing it to their employers, but i'm not sure how far he got on that front | 00:04 |
| mat128 | pabelanger: yesterday I had an issue with the SIP client (had 500ms echo delay with my own voice) that I didnt have using PSTN | 00:04 |
| notmorgan | mordred: jeblair, mordred: going to start rebasing the webhook patches onto the py3 stuff | 00:05 |
| * notmorgan mordred's mordred in a mordred when mordreding mordred in irc. | 00:05 | |
| pabelanger | mat128: possible. you can use echo@pbx.openstack.org to do a quick test of the network | 00:05 |
| mat128 | fungi: yeah I understand, thanks for the help :) | 00:05 |
| mat128 | pabelanger: it works fine right now with 2-3 participants | 00:05 |
| mat128 | no echo | 00:05 |
| mat128 | pabelanger: must have been someone on the bridge that did that | 00:05 |
| notmorgan | jeblair: i is this mostly aimed at the v3 branch only or also for the 2.5? | 00:06 |
| fungi | nibalizer: not sure why bundle install --without system_tests would cause that | 00:06 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-openstack_infra_spec_helper: Depend on the puppet gem https://review.openstack.org/332589 | 00:06 |
| notmorgan | cc pabelanger ^ greghaynes | 00:06 |
| pabelanger | mat128: Ya, I haven't profiled our server in a while. So not sure what happened. A lot of things could cause it | 00:06 |
| nibalizer | fungi: https://review.openstack.org/#/c/332589/ | 00:06 |
| fungi | hah | 00:06 |
| nibalizer | fungi: yea i got distracted by something unrelated | 00:06 |
| openstackgerrit | Merged openstack-infra/puppet-openstack_infra_spec_helper: Depend on the puppet gem https://review.openstack.org/332589 | 00:07 |
| *** sree has quit IRC | 00:08 | |
| fungi | gate jobs are rerunning for the puppet-accessbot guinea pig now | 00:08 |
| nibalizer | cool | 00:08 |
| *** sree has joined #openstack-infra | 00:10 | |
| jeblair | notmorgan: webhook == github? | 00:10 |
| nibalizer | omg my kingdom for github triggers in zuul | 00:11 |
| notmorgan | jeblair: yeah sorry | 00:12 |
| *** markvoelker has joined #openstack-infra | 00:12 | |
| notmorgan | nibalizer: it's on the list, and i'm actively taking interest in making them work since i've finished most of the py3 stuff [i' | 00:12 |
| jeblair | notmorgan: should that wait until the py3 stuff is ready? | 00:12 |
| fungi | nibalizer: passed the lint check | 00:12 |
| notmorgan | m sure more py3 work is done) | 00:13 |
| notmorgan | jeblair: i was going to rebase the chain on top of the py3 stuff | 00:13 |
| notmorgan | jeblair: since it needs py3 love afaict. | 00:13 |
| jeblair | notmorgan: right, but i thought the py3 stuff is in flux? | 00:13 |
| *** markvoelker_ has joined #openstack-infra | 00:13 | |
| nibalizer | fungi: i think we're passing | 00:13 |
| nibalizer | exciting! | 00:13 |
| notmorgan | jeblair: oh you mean re: gear things | 00:13 |
| jeblair | ya | 00:13 |
| jeblair | i mean, once we get the gear stuff settled out, hopefully that py3 change is like 80% of its current size :) | 00:14 |
| notmorgan | jeblair: right | 00:14 |
| *** sree has quit IRC | 00:14 | |
| notmorgan | jeblair: maybe 50% ;) 80 is optimistic | 00:14 |
| notmorgan | yeah derp. brain... it's been a long week | 00:14 |
| notmorgan | and i need to do more travel booking things | 00:14 |
| notmorgan | jeblair: don't want to miss ansible things. | 00:15 |
| *** markvoel_ has joined #openstack-infra | 00:15 | |
| *** markvoelker has quit IRC | 00:15 | |
| notmorgan | jeblair: so yeah let me send a response to your gear reply and i think we're 99% on the same page. maybe we just make a .get() (dict like) method that allows for in-line decode for cases you know you want to decode. | 00:15 |
| *** julim has joined #openstack-infra | 00:17 | |
| *** camunoz has joined #openstack-infra | 00:17 | |
| *** kzaitsev_mb has quit IRC | 00:18 | |
| nibalizer | fungi: one to go! | 00:18 |
| *** aeng has joined #openstack-infra | 00:18 | |
| *** cody-somerville has quit IRC | 00:19 | |
| *** sree has joined #openstack-infra | 00:19 | |
| fungi | nibalizer: yep, i'm streaming the console log to my terminal ;) | 00:19 |
| *** markvoelker_ has quit IRC | 00:19 | |
| mordred | fungi: woot! | 00:19 |
| *** ffrank has quit IRC | 00:19 | |
| nibalizer | i have this bbs reader fx plugin im using | 00:20 |
| nibalizer | it works ... ok | 00:20 |
| notmorgan | hmm. i wonder if i can convince linux to do the proper telnet:// uri opening... | 00:20 |
| fungi | nibalizer: gah, failed | 00:20 |
| mordred | notmorgan, nibalizer: pabelanger has etherpads for firefox and chrome | 00:20 |
| nibalizer | mordred: oh for relay things i think greghaynes has some nginx hax | 00:20 |
| *** pahuang has joined #openstack-infra | 00:20 | |
| fungi | rspec ./spec/acceptance/basic_spec.rb:25 # basic openstackci installation of packages Package "apache2" should be installed | 00:20 |
| mordred | nibalizer: when you say "relay" things | 00:20 |
| fungi | et cetera | 00:20 |
| mordred | nibalizer: what mean you? | 00:20 |
| notmorgan | mordred: on the desktop thats easy... on the laptop, KDE makes it more "fun" :P | 00:20 |
| fungi | nibalizer: are these now legitimate failures for that particular module at least | 00:20 |
| fungi | ? | 00:20 |
| mordred | notmorgan: you can configure it purely in firefox | 00:21 |
| fungi | nibalizer: oh, no they're not. openstackci (sub)module issues | 00:21 |
| mordred | notmorgan: basically, you add a url protocol handler - and then just configure it to be like konsole %s or something - but it would be good to capture the knowledge | 00:21 |
| mordred | notmorgan: as I have already forgotten what it is that we did | 00:22 |
| *** sree has quit IRC | 00:23 | |
| nibalizer | fungi: looks like a harness failure to me | 00:24 |
| nibalizer | mordred: whateevr thing that reads the nc and gives it back over the web | 00:24 |
| *** banix has joined #openstack-infra | 00:24 | |
| *** sree has joined #openstack-infra | 00:24 | |
| fungi | nibalizer: websockets? | 00:24 |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Move gerrit parameters into it's own file https://review.openstack.org/332591 | 00:24 |
| nibalizer | i think you have a pr against zuul | 00:24 |
| *** banix has quit IRC | 00:24 | |
| pabelanger | nibalizer: fungi: correct me if I am wrong, but don't you need to land the infra_spec_helper to puppet-openstackci first? | 00:25 |
| *** xarses has quit IRC | 00:25 | |
| pabelanger | since it is using old gems | 00:25 |
| nibalizer | oh | 00:25 |
| nibalizer | is that whats going on | 00:25 |
| fungi | interesting suggestion | 00:26 |
| nibalizer | cool yea lets do that | 00:26 |
| fungi | that could be it | 00:26 |
| fungi | rechecking now | 00:26 |
| nibalizer | i dont think i submitted one for openstackci | 00:26 |
| nibalizer | its 'special' | 00:26 |
| fungi | aha | 00:26 |
| fungi | yeah, i'm not finding one | 00:26 |
| nibalizer | incomming | 00:26 |
| fungi | we also don't have one (yet) for system-config? | 00:27 |
| zaro | nibalizer: arg! can't get that change to disable puppet on review-dev due to the apache error. anyway we can push it thru? https://review.openstack.org/#/c/332585/ | 00:27 |
| fungi | zaro: it's what we're working on solving now | 00:27 |
| *** weshay has quit IRC | 00:27 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-openstackci: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332592 | 00:27 |
| *** sree has quit IRC | 00:28 | |
| fungi | zaro: if 332592 works then we should be all set | 00:28 |
| *** nwkarsten has quit IRC | 00:29 | |
| *** haaj has quit IRC | 00:29 | |
| nibalizer | fungi: thanks for staying late to work on this | 00:30 |
| *** nwkarsten has joined #openstack-infra | 00:30 | |
| *** haaj has joined #openstack-infra | 00:30 | |
| mordred | nibalizer: yah. the websockets thing | 00:31 |
| *** nwkarsten has quit IRC | 00:31 | |
| *** gouthamr_ has joined #openstack-infra | 00:31 | |
| fungi | nibalizer: no need to thank me. it's a good improvement and avoids future pain when some other dependency decides to make non-backward-compatible releases | 00:32 |
| *** nwkarsten has joined #openstack-infra | 00:32 | |
| fungi | and i have changes i too wish to have merge without needing to override jobs | 00:32 |
| fungi | nibalizer: however, bad news. 332592 is now failing several jobs | 00:33 |
| *** mriedem has joined #openstack-infra | 00:34 | |
| *** sree has joined #openstack-infra | 00:34 | |
| *** haaj has quit IRC | 00:34 | |
| nibalizer | baw | 00:34 |
| openstackgerrit | Merged openstack-infra/system-config: disable puppet to install its-storyboard plugin https://review.openstack.org/332585 | 00:35 |
| fungi | zaro: ^ | 00:35 |
| *** gouthamr has quit IRC | 00:35 | |
| *** nwkarsten has quit IRC | 00:36 | |
| crinkle | nibalizer: anteaya sorry i was afk, i'm sure it is great | 00:38 |
| nibalizer | fungi: ah lol | 00:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-openstackci: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332592 | 00:38 |
| nibalizer | okay lets try that | 00:38 |
| *** sree has quit IRC | 00:38 | |
| nibalizer | crinkle: no sweat :) | 00:38 |
| *** banix has joined #openstack-infra | 00:39 | |
| zaro | fungi: thanks! | 00:40 |
| *** weshay has joined #openstack-infra | 00:40 | |
| fungi | nibalizer: hah, old mistake rears its head again | 00:40 |
| nibalizer | yep | 00:41 |
| nibalizer | getting punchy at the end of the day here | 00:41 |
| ianw | hmm, why does this look so terrible -> http://docs.openstack.org/developer/pbr/api/pbr.version.html | 00:41 |
| nibalizer | fungi: system-config doesn't have a Gemfile, so we don't need to modify it | 00:41 |
| *** tphummel has joined #openstack-infra | 00:42 | |
| fungi | oh, good, we just need the modules its jobs are installing to work | 00:43 |
| fungi | ianw: oh, wow. i wonder if the documentation themes got changed | 00:43 |
| *** banix has quit IRC | 00:43 | |
| *** raunak has quit IRC | 00:44 | |
| fungi | ianw: oh, i think that's from the modindex extension | 00:45 |
| fungi | we may not be theming that at all? | 00:45 |
| fungi | dhellmann: ^ any recollection of whether the modindex stuff ever got themed with oslosphinx? | 00:46 |
| *** sree_ has joined #openstack-infra | 00:47 | |
| nibalizer | fungi: failed lint? | 00:47 |
| *** thorst_ has joined #openstack-infra | 00:47 | |
| *** sree_ is now known as Guest24881 | 00:47 | |
| *** amitgandhinz has joined #openstack-infra | 00:48 | |
| fungi | nibalizer: yeah, looks like actual lint complaints | 00:48 |
| fungi | may have been passing under an older puppetlint? | 00:48 |
| fungi | e.g., "manifests/elasticsearch_node.pp - WARNING: class included by relative name on line 32" | 00:48 |
| *** sree has joined #openstack-infra | 00:48 | |
| nibalizer | oh you know what | 00:49 |
| nibalizer | i bet | 00:49 |
| nibalizer | because we haven't actually had the lint plugins | 00:49 |
| nibalizer | if you look at the diff, openstackci had a way old set | 00:49 |
| nibalizer | okay fixing this now | 00:50 |
| * nibalizer pulls thread | 00:50 | |
| * fungi coats sweater in superglue | 00:50 | |
| *** Guest24881 has quit IRC | 00:51 | |
| *** thorst_ has quit IRC | 00:51 | |
| *** lvdongbing has joined #openstack-infra | 00:52 | |
| *** amitgandhinz has quit IRC | 00:52 | |
| *** sree has quit IRC | 00:53 | |
| ianw | fungi: ahh, its shipping its own theme -> http://git.openstack.org/cgit/openstack-dev/pbr/tree/doc/source/_theme | 00:54 |
| fungi | ianw: yeah, i haven't played with the module index stuff in sphinx much, so not sure what state it's in wrt theming | 00:55 |
| *** nwkarsten has joined #openstack-infra | 00:57 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-openstackci: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332592 | 00:57 |
| nibalizer | okay updated | 00:57 |
| *** weshay has quit IRC | 00:58 | |
| *** jamesmcarthur has joined #openstack-infra | 00:58 | |
| *** xarses has joined #openstack-infra | 01:02 | |
| *** gordc has quit IRC | 01:03 | |
| *** Sukhdev_ has quit IRC | 01:06 | |
| nibalizer | fungi: self approving https://review.openstack.org/#/c/332592/3 | 01:07 |
| *** shashank_hegde has quit IRC | 01:08 | |
| *** jpeeler has joined #openstack-infra | 01:09 | |
| *** tphummel has quit IRC | 01:09 | |
| *** raunak has joined #openstack-infra | 01:10 | |
| *** droyo has joined #openstack-infra | 01:10 | |
| *** raunak has quit IRC | 01:11 | |
| fungi | nibalizer: yep, lgtm. +2 | 01:13 |
| openstackgerrit | Merged openstack-infra/puppet-openstackci: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332592 | 01:15 |
| nibalizer | okay | 01:15 |
| nibalizer | so now lets recheck accessbot again i guess? | 01:15 |
| nibalizer | (could also just send the horde) | 01:16 |
| fungi | yep, we should recheck that one first | 01:17 |
| *** sree has joined #openstack-infra | 01:18 | |
| *** bhavik has quit IRC | 01:19 | |
| *** thorst_ has joined #openstack-infra | 01:20 | |
| *** raunak has joined #openstack-infra | 01:21 | |
| *** sree has quit IRC | 01:22 | |
| *** Apoorva has quit IRC | 01:22 | |
| *** Apoorva has joined #openstack-infra | 01:23 | |
| *** thorst_ has quit IRC | 01:27 | |
| *** jpeeler has quit IRC | 01:27 | |
| nibalizer | out of check, into gate | 01:28 |
| *** jyuso1 has quit IRC | 01:29 | |
| *** sree has joined #openstack-infra | 01:32 | |
| *** jpeeler has joined #openstack-infra | 01:34 | |
| *** jpeeler has quit IRC | 01:34 | |
| *** jpeeler has joined #openstack-infra | 01:34 | |
| *** llu_ has joined #openstack-infra | 01:34 | |
| openstackgerrit | Merged openstack-infra/puppet-accessbot: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332466 | 01:35 |
| nibalizer | woo | 01:35 |
| nibalizer | okay im onna send the horde then | 01:35 |
| fungi | alright! | 01:35 |
| * fungi braces for impact once more | 01:35 | |
| *** sree has quit IRC | 01:36 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-ansible: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332509 | 01:37 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-apparmor: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332510 | 01:37 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-apps_site: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332511 | 01:37 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-askbot: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332512 | 01:37 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-asterisk: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332513 | 01:37 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-bandersnatch: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332514 | 01:37 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-bugdaystats: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332515 | 01:37 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-bup: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332516 | 01:37 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-cgit: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332517 | 01:37 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-ciwatch: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332518 | 01:37 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-diskimage_builder: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332520 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-drupal: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332521 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-elastic_recheck: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332522 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-elasticsearch: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332523 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-etherpad_lite: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332524 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-exim: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332525 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-gerrit: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332526 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-gerritbot: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332527 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-github: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332528 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-grafyaml: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332529 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-graphite: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332530 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-haveged: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332531 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-hound: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332532 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-httpd: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332533 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-infracloud: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332534 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-iptables: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332535 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-jeepyb: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332536 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-jenkins: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332537 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-kerberos: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332538 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-kibana: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332539 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-lodgeit: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332540 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-log_processor: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332541 | 01:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-logrotate: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332542 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-logstash: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332543 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-mailman: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332544 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-mediawiki: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332545 | 01:39 |
| notmorgan | nibalizer: o.O | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-meetbot: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332546 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-mysql_backup: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332547 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-nodepool: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332548 | 01:39 |
| notmorgan | nibalizer: well then. ;) | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-odsreg: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332549 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-openafs: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332550 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-openstack_health: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332551 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-openstackid: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332552 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-os_client_config: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332553 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-packagekit: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332554 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-pgsql_backup: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332555 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-phabricator: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332556 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-pip: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332557 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-planet: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332558 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-project_config: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332559 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-puppet: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332560 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-redis: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332561 | 01:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-refstack: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332562 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-releasestatus: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332563 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-reviewday: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332564 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-simpleproxy: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332565 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-snmpd: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332566 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-ssh: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332567 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-ssl_cert_check: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332568 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-stackalytics: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332569 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-statusbot: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332570 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-storyboard: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332571 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-subunit2sql: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332572 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-sudoers: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332573 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-tmpreaper: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332574 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-ulimit: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332575 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-unattended_upgrades: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332576 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-unbound: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332577 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-user: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332578 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-vcsrepo: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332579 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-vinz: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332580 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-yum: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332581 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-zanata: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332582 | 01:40 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-zuul: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332583 | 01:40 |
| nibalizer | notmorgan: i wanna be really sure i get ATC | 01:46 |
| fungi | hah | 01:46 |
| *** Apoorva has quit IRC | 01:47 | |
| notmorgan | nibalizer: i have proof you are ATC at least for barcelona | 01:47 |
| notmorgan | nibalizer: ;) | 01:47 |
| nibalizer | heh | 01:47 |
| fungi | verify +1 results are starting to come back. once i see them all green i'll +3 | 01:47 |
| nibalizer | awesome | 01:47 |
| *** Apoorva has joined #openstack-infra | 01:48 | |
| *** amitgandhinz has joined #openstack-infra | 01:48 | |
| nibalizer | fungi: im going to slip out, if thats ok with you | 01:49 |
| *** ddieterly has quit IRC | 01:51 | |
| *** pvaneck has quit IRC | 01:51 | |
| *** sree has joined #openstack-infra | 01:51 | |
| *** Apoorva has quit IRC | 01:53 | |
| fungi | nibalizer: go for it, and thanks for solving this! | 01:53 |
| *** zhurong has joined #openstack-infra | 01:53 | |
| *** amitgandhinz has quit IRC | 01:53 | |
| *** sree_ has joined #openstack-infra | 01:55 | |
| *** sree_ is now known as Guest2291 | 01:55 | |
| *** sree has quit IRC | 01:56 | |
| *** jamesmcarthur has quit IRC | 01:56 | |
| *** bhavik has joined #openstack-infra | 01:56 | |
| *** stevemar has quit IRC | 01:56 | |
| fungi | looks like we have a handful of modules not passing jobs for individual reasons that should be solvable by tweaking stuff and squashing it into their current packages | 01:58 |
| *** thorst_ has joined #openstack-infra | 01:58 | |
| *** Guest2291 has quit IRC | 01:59 | |
| *** salv-orlando has joined #openstack-infra | 02:00 | |
| *** raunak has quit IRC | 02:01 | |
| openstackgerrit | Merged openstack-dev/pbr: Updated from global requirements https://review.openstack.org/332269 | 02:01 |
| *** ddieterly has joined #openstack-infra | 02:03 | |
| *** nwkarsten has quit IRC | 02:04 | |
| *** nwkarsten has joined #openstack-infra | 02:04 | |
| jpmaxman | alright fungi et al I'm going to call it a night soon - anyone who wants to keep an eye on this page for suspicious activity - would be great: https://wiki.openstack.org/wiki/Special:RecentChanges | 02:06 |
| jpmaxman | I'll do the same :D | 02:06 |
| *** esberglu has joined #openstack-infra | 02:07 | |
| *** zhurong has quit IRC | 02:07 | |
| *** salv-orlando has quit IRC | 02:07 | |
| *** yamahata has joined #openstack-infra | 02:08 | |
| fungi | thanks again, jpmaxman! | 02:08 |
| *** nwkarsten has quit IRC | 02:09 | |
| fungi | okay, check results are back on all the status:open topic:gemfile-sync changes. i'll approve the ones that passed (12 are still failing and need further investigation) | 02:10 |
| *** zhurong has joined #openstack-infra | 02:10 | |
| nibalizer | ok | 02:10 |
| *** PollyZ has joined #openstack-infra | 02:10 | |
| fungi | something like 61 passing | 02:11 |
| rajinir | Having a weird problem in our CI system. Zuul is not passing a range of built-in parameters (eg: ZUUL_PROJECT) . Any clues where to debug this problem? | 02:11 |
| *** sarob has joined #openstack-infra | 02:11 | |
| nibalizer | hopefully youll take some evenjng now too | 02:11 |
| openstackgerrit | Merged openstack-infra/puppet-vcsrepo: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332579 | 02:11 |
| *** zhurong has quit IRC | 02:11 | |
| *** zhurong has joined #openstack-infra | 02:12 | |
| fungi | rajinir: if you're running jenkins 1.651.2 or later you probably want to read the thread starting at http://lists.openstack.org/pipermail/openstack-infra/2016-May/004284.html | 02:12 |
| *** esberglu has quit IRC | 02:13 | |
| *** ddieterly has quit IRC | 02:14 | |
| *** sarob has quit IRC | 02:15 | |
| *** jamielennox is now known as jamielennox|away | 02:16 | |
| *** yamamot__ has joined #openstack-infra | 02:16 | |
| *** erlon has quit IRC | 02:16 | |
| rajinir | fungi:Jenkins 1.651.3 | 02:16 |
| *** jamesmcarthur has joined #openstack-infra | 02:16 | |
| rajinir | fungi: It was working until two weeks back, so may be jenkins got upgraded recently | 02:17 |
| *** jamielennox|away is now known as jamielennox | 02:17 | |
| *** yamahata has quit IRC | 02:18 | |
| rajinir | fungi:thank you , will give it a try | 02:19 |
| rajinir | That was it. Thank you | 02:22 |
| *** thorst_ has quit IRC | 02:22 | |
| openstackgerrit | Merged openstack-infra/puppet-kerberos: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332538 | 02:22 |
| openstackgerrit | Merged openstack-infra/puppet-kibana: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332539 | 02:22 |
| openstackgerrit | Merged openstack-infra/puppet-lodgeit: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332540 | 02:22 |
| openstackgerrit | Merged openstack-infra/puppet-log_processor: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332541 | 02:23 |
| openstackgerrit | Merged openstack-infra/puppet-logrotate: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332542 | 02:23 |
| *** thorst_ has joined #openstack-infra | 02:23 | |
| *** aeng has quit IRC | 02:23 | |
| openstackgerrit | Merged openstack-infra/puppet-logstash: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332543 | 02:23 |
| openstackgerrit | Merged openstack-infra/puppet-mailman: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332544 | 02:23 |
| openstackgerrit | Merged openstack-infra/puppet-mediawiki: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332545 | 02:24 |
| openstackgerrit | Merged openstack-infra/puppet-meetbot: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332546 | 02:24 |
| *** reed has quit IRC | 02:24 | |
| fungi | it's... the mergening | 02:24 |
| openstackgerrit | Merged openstack-infra/puppet-vinz: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332580 | 02:24 |
| openstackgerrit | Merged openstack-infra/puppet-nodepool: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332548 | 02:24 |
| *** mriedem has quit IRC | 02:24 | |
| openstackgerrit | Merged openstack-infra/puppet-odsreg: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332549 | 02:24 |
| openstackgerrit | Merged openstack-infra/puppet-openafs: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332550 | 02:25 |
| *** rhochmuth has joined #openstack-infra | 02:26 | |
| *** rhochmuth has quit IRC | 02:26 | |
| openstackgerrit | Merged openstack-infra/puppet-os_client_config: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332553 | 02:26 |
| openstackgerrit | Merged openstack-infra/puppet-packagekit: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332554 | 02:26 |
| openstackgerrit | Merged openstack-infra/puppet-pgsql_backup: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332555 | 02:26 |
| openstackgerrit | Merged openstack-infra/puppet-phabricator: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332556 | 02:27 |
| openstackgerrit | Merged openstack-infra/puppet-pip: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332557 | 02:27 |
| openstackgerrit | Merged openstack-infra/puppet-planet: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332558 | 02:27 |
| *** rhochmuth has joined #openstack-infra | 02:27 | |
| *** rhochmuth has quit IRC | 02:27 | |
| openstackgerrit | Merged openstack-infra/puppet-project_config: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332559 | 02:27 |
| openstackgerrit | Merged openstack-infra/puppet-puppet: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332560 | 02:27 |
| openstackgerrit | Merged openstack-infra/puppet-redis: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332561 | 02:27 |
| *** rhochmuth has joined #openstack-infra | 02:27 | |
| openstackgerrit | Merged openstack-infra/puppet-releasestatus: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332563 | 02:27 |
| openstackgerrit | Merged openstack-infra/puppet-reviewday: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332564 | 02:27 |
| *** reed has joined #openstack-infra | 02:27 | |
| openstackgerrit | Merged openstack-infra/puppet-simpleproxy: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332565 | 02:27 |
| *** markmcclain has quit IRC | 02:28 | |
| *** jyuso1 has joined #openstack-infra | 02:29 | |
| *** tonytan4ever has joined #openstack-infra | 02:30 | |
| *** PollyZ_ has joined #openstack-infra | 02:30 | |
| zaro | fungi: how long does the disable puppet on review-dev take to apply? it looks like puppet is still reverting files i've modified :( | 02:31 |
| *** thorst_ has quit IRC | 02:31 | |
| openstackgerrit | Merged openstack-infra/puppet-snmpd: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332566 | 02:32 |
| openstackgerrit | Merged openstack-infra/puppet-ssh: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332567 | 02:32 |
| *** markmcclain has joined #openstack-infra | 02:32 | |
| openstackgerrit | Merged openstack-infra/puppet-ssl_cert_check: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332568 | 02:32 |
| *** reed has quit IRC | 02:32 | |
| openstackgerrit | Merged openstack-infra/puppet-stackalytics: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332569 | 02:32 |
| fungi | zaro: should have taken effect within a few minutes of merging... | 02:32 |
| zaro | hmm, was my change correct? because files i just changed are still getting reverted. | 02:33 |
| *** PollyZ has quit IRC | 02:33 | |
| zaro | https://review.openstack.org/#/c/332585/ | 02:34 |
| fungi | zaro: no wonder. you didn't add it to the disabled list there | 02:35 |
| *** PollyZ_ has quit IRC | 02:35 | |
| * zaro preps a new change for dat | 02:35 | |
| fungi | you needed to append it to the colon-separated list in the last line of that file | 02:35 |
| *** aeng has joined #openstack-infra | 02:35 | |
| openstackgerrit | Merged openstack-infra/puppet-statusbot: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332570 | 02:36 |
| openstackgerrit | Merged openstack-infra/puppet-storyboard: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332571 | 02:36 |
| openstackgerrit | Merged openstack-infra/puppet-subunit2sql: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332572 | 02:36 |
| *** dfflanders has joined #openstack-infra | 02:36 | |
| openstackgerrit | Merged openstack-infra/puppet-sudoers: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332573 | 02:36 |
| openstackgerrit | Merged openstack-infra/puppet-tmpreaper: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332574 | 02:36 |
| openstackgerrit | Merged openstack-infra/puppet-ulimit: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332575 | 02:36 |
| openstackgerrit | Merged openstack-infra/puppet-unattended_upgrades: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332576 | 02:36 |
| fungi | zaro: sorry about that, i didn't actually review it when i merged it, since nibalizer had already approved it | 02:36 |
| *** PollyZ has joined #openstack-infra | 02:37 | |
| openstackgerrit | Merged openstack-infra/puppet-unbound: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332577 | 02:37 |
| openstackgerrit | Merged openstack-infra/puppet-user: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332578 | 02:37 |
| openstackgerrit | Merged openstack-infra/puppet-yum: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332581 | 02:37 |
| openstackgerrit | Merged openstack-infra/puppet-ansible: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332509 | 02:37 |
| openstackgerrit | Merged openstack-infra/puppet-apparmor: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332510 | 02:37 |
| openstackgerrit | Merged openstack-infra/puppet-apps_site: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332511 | 02:37 |
| openstackgerrit | Merged openstack-infra/puppet-askbot: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332512 | 02:38 |
| openstackgerrit | Merged openstack-infra/puppet-asterisk: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332513 | 02:38 |
| openstackgerrit | Merged openstack-infra/puppet-bugdaystats: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332515 | 02:38 |
| openstackgerrit | Merged openstack-infra/puppet-bup: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332516 | 02:38 |
| fungi | zaro: and remove the line you added in 332585 since that's not needed (at least we don't _currently_ need a separate ansible group defined for review.o.o) | 02:38 |
| *** reed has joined #openstack-infra | 02:39 | |
| openstackgerrit | Merged openstack-infra/puppet-cgit: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332517 | 02:39 |
| openstackgerrit | Merged openstack-infra/puppet-diskimage_builder: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332520 | 02:39 |
| openstackgerrit | Merged openstack-infra/puppet-drupal: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332521 | 02:39 |
| openstackgerrit | Merged openstack-infra/puppet-elastic_recheck: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332522 | 02:39 |
| openstackgerrit | Merged openstack-infra/puppet-elasticsearch: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332523 | 02:39 |
| openstackgerrit | Merged openstack-infra/puppet-exim: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332525 | 02:39 |
| openstackgerrit | Merged openstack-infra/puppet-gerritbot: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332527 | 02:39 |
| openstackgerrit | Merged openstack-infra/puppet-github: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332528 | 02:40 |
| openstackgerrit | Merged openstack-infra/puppet-graphite: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332530 | 02:40 |
| openstackgerrit | Merged openstack-infra/puppet-haveged: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332531 | 02:40 |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332534 | 02:40 |
| openstackgerrit | Merged openstack-infra/puppet-iptables: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332535 | 02:42 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: disable puppet to install its-storyboard plugin https://review.openstack.org/332606 | 02:43 |
| zaro | fungi: ^ | 02:43 |
| openstackgerrit | Merged openstack-infra/puppet-jeepyb: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332536 | 02:43 |
| fungi | nibalizer: looks like all the passing spec helper changes have merged now, thanks again! | 02:44 |
| *** yuanying has quit IRC | 02:46 | |
| *** kzaitsev_mb has joined #openstack-infra | 02:46 | |
| *** amitgandhinz has joined #openstack-infra | 02:49 | |
| *** PollyZ has quit IRC | 02:50 | |
| *** PollyZ has joined #openstack-infra | 02:50 | |
| *** banix has joined #openstack-infra | 02:53 | |
| nibalizer | thsnk you fungi | 02:53 |
| nibalizer | the hero we need | 02:54 |
| *** amitgandhinz has quit IRC | 02:54 | |
| *** stevemar has joined #openstack-infra | 02:56 | |
| openstackgerrit | Merged openstack-infra/system-config: disable puppet to install its-storyboard plugin https://review.openstack.org/332606 | 02:59 |
| fungi | zaro: ^ it should stop shortly now | 03:03 |
| *** PollyZ has quit IRC | 03:04 | |
| *** rlandy has quit IRC | 03:07 | |
| zaro | fungi: thanks! | 03:09 |
| *** gouthamr_ has quit IRC | 03:11 | |
| *** PollyZ has joined #openstack-infra | 03:11 | |
| Srinu | fungi: hi | 03:12 |
| *** yamahata has joined #openstack-infra | 03:12 | |
| *** TaylorH has joined #openstack-infra | 03:13 | |
| *** salv-orlando has joined #openstack-infra | 03:14 | |
| *** zhurong has quit IRC | 03:18 | |
| openstackgerrit | Ian Wienand proposed openstack-dev/pbr: Move to oslosphinx https://review.openstack.org/332612 | 03:18 |
| openstackgerrit | Ian Wienand proposed openstack-dev/pbr: Move to oslosphinx https://review.openstack.org/332612 | 03:19 |
| *** ramishra has joined #openstack-infra | 03:20 | |
| *** raunak has joined #openstack-infra | 03:20 | |
| *** salv-orlando has quit IRC | 03:21 | |
| *** PollyZ has quit IRC | 03:21 | |
| *** zhurong has joined #openstack-infra | 03:23 | |
| *** banix has quit IRC | 03:23 | |
| *** gyee has quit IRC | 03:24 | |
| *** PollyZ has joined #openstack-infra | 03:26 | |
| openstackgerrit | Merged openstack-infra/system-config: Give storyboard-dev its own superusers list https://review.openstack.org/332505 | 03:27 |
| openstackgerrit | Merged openstack-infra/system-config: Add Zara to StoryBoard Dev Admin https://review.openstack.org/329092 | 03:28 |
| *** zhurong has quit IRC | 03:28 | |
| *** sree has joined #openstack-infra | 03:28 | |
| *** zhurong has joined #openstack-infra | 03:29 | |
| *** links has joined #openstack-infra | 03:29 | |
| *** thorst_ has joined #openstack-infra | 03:29 | |
| openstackgerrit | Merged openstack-infra/system-config: Add Adam Coldrick to StoryBoard Dev Admin https://review.openstack.org/332072 | 03:31 |
| *** aeng has quit IRC | 03:32 | |
| *** dimtruck is now known as zz_dimtruck | 03:33 | |
| *** zhurong has quit IRC | 03:33 | |
| *** kzaitsev_mb has quit IRC | 03:36 | |
| *** nwkarsten has joined #openstack-infra | 03:36 | |
| *** thorst_ has quit IRC | 03:37 | |
| *** cody-somerville has joined #openstack-infra | 03:39 | |
| *** jamesmcarthur has quit IRC | 03:39 | |
| *** PollyZ has quit IRC | 03:40 | |
| *** nwkarsten has quit IRC | 03:41 | |
| *** ramishra has quit IRC | 03:41 | |
| *** Kennan has quit IRC | 03:42 | |
| *** raunak has quit IRC | 03:44 | |
| *** raunak has joined #openstack-infra | 03:46 | |
| *** roxanaghe has joined #openstack-infra | 03:46 | |
| *** yuanying has joined #openstack-infra | 03:47 | |
| *** zhurong has joined #openstack-infra | 03:49 | |
| *** aeng has joined #openstack-infra | 03:49 | |
| *** amitgandhinz has joined #openstack-infra | 03:50 | |
| *** markmcclain has quit IRC | 03:50 | |
| *** dchen has joined #openstack-infra | 03:51 | |
| *** droyo has quit IRC | 03:52 | |
| *** tonytan4ever has quit IRC | 03:53 | |
| *** amitgandhinz has quit IRC | 03:55 | |
| *** zhurong has quit IRC | 03:58 | |
| *** zhurong has joined #openstack-infra | 03:58 | |
| *** njohnston|afk is now known as njohnston | 04:01 | |
| *** stevemar has quit IRC | 04:02 | |
| *** reed has quit IRC | 04:02 | |
| *** reed has joined #openstack-infra | 04:04 | |
| *** ramishra has joined #openstack-infra | 04:06 | |
| *** zz_dimtruck is now known as dimtruck | 04:07 | |
| *** ramishra has quit IRC | 04:09 | |
| *** rossella__ has joined #openstack-infra | 04:10 | |
| *** yamamot__ has quit IRC | 04:10 | |
| *** rossella_ has quit IRC | 04:13 | |
| *** zhurong has quit IRC | 04:13 | |
| *** reed has quit IRC | 04:13 | |
| *** zhurong has joined #openstack-infra | 04:13 | |
| *** ramishra has joined #openstack-infra | 04:14 | |
| *** cody-somerville has quit IRC | 04:14 | |
| *** shashank_hegde has joined #openstack-infra | 04:15 | |
| *** tobiash has quit IRC | 04:17 | |
| *** garyk1 has joined #openstack-infra | 04:18 | |
| *** zhurong has quit IRC | 04:19 | |
| *** raunak has quit IRC | 04:20 | |
| *** tobiash has joined #openstack-infra | 04:20 | |
| *** Kennan has joined #openstack-infra | 04:22 | |
| *** jaosorior has joined #openstack-infra | 04:23 | |
| *** salv-orlando has joined #openstack-infra | 04:28 | |
| *** raunak has joined #openstack-infra | 04:29 | |
| *** psachin has joined #openstack-infra | 04:30 | |
| *** Kennan2 has joined #openstack-infra | 04:30 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:32 | |
| *** Kennan has quit IRC | 04:33 | |
| *** thorst_ has joined #openstack-infra | 04:34 | |
| *** salv-orlando has quit IRC | 04:35 | |
| *** kzaitsev_mb has quit IRC | 04:38 | |
| *** thorst_ has quit IRC | 04:41 | |
| *** hichihara has joined #openstack-infra | 04:42 | |
| *** zhurong has joined #openstack-infra | 04:42 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Install deltarpm as it's needed for the undercloud https://review.openstack.org/332625 | 04:43 |
| *** Sukhdev has joined #openstack-infra | 04:49 | |
| *** Sukhdev has quit IRC | 04:49 | |
| *** Sukhdev has joined #openstack-infra | 04:50 | |
| *** amitgandhinz has joined #openstack-infra | 04:51 | |
| *** dfflanders has quit IRC | 04:52 | |
| *** yamamot__ has joined #openstack-infra | 04:53 | |
| *** Julien-zte has joined #openstack-infra | 04:55 | |
| *** amitgandhinz has quit IRC | 04:56 | |
| *** ihrachys has joined #openstack-infra | 04:56 | |
| *** rcernin has joined #openstack-infra | 04:56 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add networking-calico meeting https://review.openstack.org/331689 | 04:58 |
| *** pgadiya has joined #openstack-infra | 04:58 | |
| *** abregman_ has joined #openstack-infra | 04:59 | |
| *** njohnston is now known as njohnston|afk | 05:01 | |
| *** Jeffrey4l_ has quit IRC | 05:01 | |
| *** raunak has quit IRC | 05:02 | |
| *** reed has joined #openstack-infra | 05:02 | |
| *** stevemar has joined #openstack-infra | 05:02 | |
| *** dchen has quit IRC | 05:02 | |
| *** ihrachys has quit IRC | 05:03 | |
| *** ihrachys has joined #openstack-infra | 05:04 | |
| *** rajinir has quit IRC | 05:04 | |
| *** camunoz has quit IRC | 05:06 | |
| *** aeng has quit IRC | 05:06 | |
| *** stevemar has quit IRC | 05:07 | |
| *** zhurong has quit IRC | 05:07 | |
| AJaeger | nibalizer, fungi: Wow, that's a puppet merge party you had. Thanks! | 05:08 |
| *** shashank_hegde has quit IRC | 05:09 | |
| AJaeger | ianw, jhesketh, project-config cores: Could you review https://review.openstack.org/332460, please? That adds missing files to the files conditions for project-config | 05:09 |
| *** markmcclain has joined #openstack-infra | 05:11 | |
| *** roxanaghe has quit IRC | 05:11 | |
| *** roxanaghe has joined #openstack-infra | 05:12 | |
| *** pahuang has quit IRC | 05:12 | |
| *** zhurong has joined #openstack-infra | 05:13 | |
| *** dchen has joined #openstack-infra | 05:15 | |
| *** rcernin has quit IRC | 05:16 | |
| *** roxanaghe has quit IRC | 05:17 | |
| *** ihrachys_ has joined #openstack-infra | 05:18 | |
| *** ihrachys has quit IRC | 05:18 | |
| *** aeng has joined #openstack-infra | 05:18 | |
| *** camunoz has joined #openstack-infra | 05:19 | |
| *** shardy has joined #openstack-infra | 05:20 | |
| *** ihrachys_ has quit IRC | 05:20 | |
| openstackgerrit | Ofer Ben-Yacov proposed openstack-infra/project-config: Add new project: networking-icc https://review.openstack.org/332091 | 05:20 |
| *** pahuang has joined #openstack-infra | 05:26 | |
| *** pahuang has quit IRC | 05:28 | |
| *** prabu has joined #openstack-infra | 05:31 | |
| *** azvyagintsev has quit IRC | 05:31 | |
| *** kzaitsev_mb has joined #openstack-infra | 05:33 | |
| *** chlong has quit IRC | 05:34 | |
| *** kaisers has left #openstack-infra | 05:35 | |
| *** salv-orlando has joined #openstack-infra | 05:37 | |
| *** jkraj has joined #openstack-infra | 05:38 | |
| *** shardy has quit IRC | 05:38 | |
| *** kzaitsev_mb has quit IRC | 05:38 | |
| *** thorst_ has joined #openstack-infra | 05:39 | |
| *** Sukhdev_ has joined #openstack-infra | 05:40 | |
| *** Sukhdev has quit IRC | 05:41 | |
| *** winggundamth has quit IRC | 05:42 | |
| *** zhurong has quit IRC | 05:42 | |
| *** javeriak has joined #openstack-infra | 05:45 | |
| *** zhurong has joined #openstack-infra | 05:45 | |
| *** thorst_ has quit IRC | 05:46 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr: Updated from global requirements https://review.openstack.org/332335 | 05:46 |
| *** chlong has joined #openstack-infra | 05:47 | |
| *** rbrndt has joined #openstack-infra | 05:47 | |
| AJaeger | good morning, yolanda! Thanks for reviewing | 05:48 |
| yolanda | hi AJaeger , good morning | 05:48 |
| *** winggundamth has joined #openstack-infra | 05:48 | |
| yolanda | trying to find some time before summit starts :) | 05:48 |
| *** javeriak_ has joined #openstack-infra | 05:49 | |
| AJaeger | https://review.openstack.org/332287 seems to have some urgency - the rest can wait. Better enjoy breakfast ;) | 05:49 |
| *** javeriak has quit IRC | 05:50 | |
| *** zhurong_ has joined #openstack-infra | 05:51 | |
| *** zhurong has quit IRC | 05:51 | |
| *** amitgandhinz has joined #openstack-infra | 05:52 | |
| *** javeriak_ has quit IRC | 05:52 | |
| openstackgerrit | Ofer Ben-Yacov proposed openstack-infra/project-config: Add new project: networking-icc https://review.openstack.org/332638 | 05:52 |
| *** rcernin has joined #openstack-infra | 05:55 | |
| *** pahuang has joined #openstack-infra | 05:55 | |
| *** amitgandhinz has quit IRC | 05:56 | |
| *** rbrndt has quit IRC | 05:58 | |
| *** pfallenop has quit IRC | 06:00 | |
| *** rhochmuth has quit IRC | 06:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Remove skip test logic for project-config" https://review.openstack.org/330821 | 06:01 |
| *** amotoki has joined #openstack-infra | 06:01 | |
| *** dchen has quit IRC | 06:02 | |
| openstackgerrit | Merged openstack-infra/project-config: Add other-requirements to files list for project-config https://review.openstack.org/332460 | 06:02 |
| openstackgerrit | Merged openstack-infra/project-config: Start using pre-prov creds in dsvm-layer4 https://review.openstack.org/329723 | 06:03 |
| yolanda | reviewed | 06:04 |
| openstackgerrit | Ofer Ben-Yacov proposed openstack-infra/project-config: Add new project: networking-icc https://review.openstack.org/332638 | 06:05 |
| openstackgerrit | Merged openstack-infra/project-config: Quark Project Non-Voting Py3 https://review.openstack.org/332287 | 06:06 |
| *** ccamacho has joined #openstack-infra | 06:06 | |
| openstackgerrit | Merged openstack-infra/project-config: Moving live-migration job to check queue for nova https://review.openstack.org/332175 | 06:07 |
| openstackgerrit | Merged openstack-infra/project-config: Make murano-agent py34 job voting https://review.openstack.org/332093 | 06:07 |
| openstackgerrit | Merged openstack-infra/project-config: Enable storlets to show test_result https://review.openstack.org/332065 | 06:09 |
| *** zhurong_ has quit IRC | 06:09 | |
| *** Jeffrey4l_ has joined #openstack-infra | 06:09 | |
| *** bhavik has quit IRC | 06:10 | |
| *** bhavik has joined #openstack-infra | 06:10 | |
| *** _nadya_ has joined #openstack-infra | 06:10 | |
| *** zhurong has joined #openstack-infra | 06:11 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/332645 | 06:11 |
| *** aeng has quit IRC | 06:11 | |
| openstackgerrit | Merged openstack-infra/project-config: Migrated sahara-client-functional-job https://review.openstack.org/329497 | 06:11 |
| *** Julien-zte has quit IRC | 06:11 | |
| *** zhurong has quit IRC | 06:13 | |
| openstackgerrit | Merged openstack-infra/project-config: Use noop-jobs template https://review.openstack.org/331805 | 06:13 |
| openstackgerrit | Ofer Ben-Yacov proposed openstack-infra/project-config: Add new project: networking-icc https://review.openstack.org/332091 | 06:13 |
| openstackgerrit | Merged openstack-infra/project-config: Change puppet-4.0 unit test to puppet-4.5 https://review.openstack.org/332049 | 06:14 |
| *** linuxmaniac has joined #openstack-infra | 06:16 | |
| *** linuxmaniac_ has quit IRC | 06:17 | |
| openstackgerrit | Merged openstack-infra/project-config: Add KMIP Experiment Check to Barbican https://review.openstack.org/307820 | 06:18 |
| *** ihrachys has joined #openstack-infra | 06:18 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Update files list for project-config https://review.openstack.org/332647 | 06:19 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Merge gerrit jobs for project-config https://review.openstack.org/332648 | 06:19 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove projects tox environment https://review.openstack.org/332649 | 06:19 |
| *** zhurong has joined #openstack-infra | 06:19 | |
| *** ihrachys has quit IRC | 06:21 | |
| AJaeger | project-config cores, the above three changes update the file list for jobs and merge two jobs. I reviewed everything for this and noticed the duplicate conditions... | 06:22 |
| nibalizer | hi yolanda | 06:22 |
| *** javeriak has joined #openstack-infra | 06:23 | |
| *** armax has joined #openstack-infra | 06:23 | |
| zaro | nibalizer: have all the failing jobs been fixed? | 06:25 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-zuul: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332583 | 06:27 |
| nibalizer | some of them | 06:28 |
| nibalizer | https://review.openstack.org/#/q/topic:gemfile-sync+status:open is the list of things to still land | 06:28 |
| nibalizer | im going through the remainders now and trying to trace down the issues | 06:28 |
| AJaeger | So, 13 still open - but 66 or so merged. | 06:28 |
| nibalizer | hopefully yolanda wanders back and can +3 them :) | 06:28 |
| *** yolanda has quit IRC | 06:29 | |
| *** nmagnezi has joined #openstack-infra | 06:30 | |
| AJaeger | nibalizer: she's at a conference today. Yesterday she found some time in between for reviews but today she did her big round already. Let's see whether she has time later. | 06:31 |
| AJaeger | nibalizer: We can also ask jhesketh to review your changes. | 06:32 |
| openstackgerrit | Merged openstack-infra/puppet-ciwatch: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332518 | 06:32 |
| nibalizer | is he still on? | 06:33 |
| *** Nikolay_St has joined #openstack-infra | 06:33 | |
| jhesketh | yo | 06:33 |
| * jhesketh looks | 06:33 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-zanata: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332582 | 06:33 |
| *** dimtruck is now known as zz_dimtruck | 06:33 | |
| jhesketh | nibalizer: I thought these were all +w'd earlier.. how come some of these were missed? | 06:34 |
| *** mohankumar has joined #openstack-infra | 06:35 | |
| AJaeger | nibalizer, jhesketh: https://review.openstack.org/332647 plus the dependent change would be nice to review as well - but not urgent... | 06:35 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-openstackid: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332552 | 06:37 |
| *** salv-orl_ has joined #openstack-infra | 06:38 | |
| *** dfflanders has joined #openstack-infra | 06:38 | |
| AJaeger | thanks, jhesketh ! | 06:39 |
| jhesketh | anytime | 06:39 |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/332645 | 06:39 |
| openstackgerrit | Merged openstack-infra/puppet-openstack_health: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332551 | 06:40 |
| *** salv-orlando has quit IRC | 06:41 | |
| *** thorst_ has joined #openstack-infra | 06:44 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Puppet-refstack doesn't have any beaker jobs https://review.openstack.org/332654 | 06:44 |
| *** x00350071_ has joined #openstack-infra | 06:45 | |
| *** andreas_s has joined #openstack-infra | 06:45 | |
| nibalizer | jhesketh: fungi did not +3 ones that failed in the check | 06:45 |
| nibalizer | AJaeger: sure | 06:45 |
| nibalizer | gerrit is slowwww | 06:46 |
| *** david-lyle_ has joined #openstack-infra | 06:46 | |
| *** dangers_` has joined #openstack-infra | 06:46 | |
| *** llu__ has joined #openstack-infra | 06:47 | |
| *** tesseract- has joined #openstack-infra | 06:47 | |
| *** trash_ has joined #openstack-infra | 06:47 | |
| *** isaacb has joined #openstack-infra | 06:47 | |
| ianw | nibalizer: yeah, i got a few timeout errors before too. might be time for one of those restarts | 06:47 |
| nibalizer | jhesketh: +3 if you please https://review.openstack.org/#/c/332583/ | 06:47 |
| jhesketh | nibalizer: yeah just looking... I can +3 the ones passing | 06:48 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-openstack_health: Add build_newrev to the list of ignored keys https://review.openstack.org/328467 | 06:48 |
| *** bhavik1 has joined #openstack-infra | 06:48 | |
| jhesketh | nibalizer: done | 06:49 |
| nibalizer | some are like ..... | 06:49 |
| nibalizer | no | 06:49 |
| *** Guest23899 has quit IRC | 06:49 | |
| *** Guest23899 has joined #openstack-infra | 06:49 | |
| *** Guest23899 is now known as ioni | 06:49 | |
| jhesketh | nibalizer: ? | 06:50 |
| jhesketh | as in you don't know why they are failing? | 06:50 |
| *** x00350071_ has left #openstack-infra | 06:50 | |
| *** fabo_ has joined #openstack-infra | 06:50 | |
| *** Nikolay_1t has joined #openstack-infra | 06:50 | |
| *** slagle_ has joined #openstack-infra | 06:50 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-gerrit: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332526 | 06:51 |
| *** thorst_ has quit IRC | 06:51 | |
| nibalizer | or they are failing for dumb reasons | 06:51 |
| *** aviau_ has joined #openstack-infra | 06:51 | |
| nibalizer | still, i think we're really close | 06:52 |
| jhesketh | nibalizer: yep | 06:52 |
| nibalizer | and in theory this is the last time we have to do this | 06:52 |
| jhesketh | nice work baby sitting all of this btw :-) | 06:52 |
| *** dmellado_ has joined #openstack-infra | 06:52 | |
| *** snarwade has quit IRC | 06:52 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-jenkins: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332537 | 06:52 |
| *** amitgandhinz has joined #openstack-infra | 06:52 | |
| *** snarwade has joined #openstack-infra | 06:53 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-etherpad_lite: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332524 | 06:53 |
| *** dasm_ has joined #openstack-infra | 06:53 | |
| nibalizer | like this who knows | 06:53 |
| nibalizer | http://logs.openstack.org/14/332514/2/check/gate-puppet-bandersnatch-puppet-beaker-rspec-centos-7/a12e49b/console.html | 06:53 |
| *** jaosorior is now known as jaosorior_brb | 06:54 | |
| nibalizer | jeblair: an odd zuul fail http://logs.openstack.org/32/332532/2/check/gate-infra-puppet-apply-centos-7/a4df04c/console.html | 06:54 |
| *** charz has joined #openstack-infra | 06:54 | |
| *** dgonzalez_ has joined #openstack-infra | 06:55 | |
| *** shardy has joined #openstack-infra | 06:56 | |
| hosanai | AJaeger: could you please review patch 331535? We don't have other good way for python27 now so I would like to proceed development with this patch in order to not stopping the development. The patch has +1 from Monasca PTL. | 06:56 |
| *** oferby has joined #openstack-infra | 06:57 | |
| *** dmsimard1 has joined #openstack-infra | 06:57 | |
| *** amitgandhinz has quit IRC | 06:57 | |
| openstackgerrit | Merged openstack-infra/project-config: Update files list for project-config https://review.openstack.org/332647 | 06:58 |
| *** eantyshev1 has joined #openstack-infra | 06:58 | |
| openstackgerrit | Merged openstack-infra/puppet-zuul: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332583 | 06:59 |
| *** dfflanders has quit IRC | 06:59 | |
| *** megm_ has joined #openstack-infra | 07:00 | |
| openstackgerrit | Merged openstack-infra/puppet-mysql_backup: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332547 | 07:00 |
| openstackgerrit | Merged openstack-infra/puppet-zanata: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332582 | 07:00 |
| *** tlian2 has joined #openstack-infra | 07:01 | |
| *** vtapia_ has joined #openstack-infra | 07:01 | |
| *** hamzy_ has joined #openstack-infra | 07:01 | |
| *** gmann_ has joined #openstack-infra | 07:01 | |
| openstackgerrit | Merged openstack-infra/project-config: Merge gerrit jobs for project-config https://review.openstack.org/332648 | 07:01 |
| *** pcrews__ has joined #openstack-infra | 07:02 | |
| *** kdas__ has joined #openstack-infra | 07:02 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Fix tempest configuration https://review.openstack.org/331997 | 07:04 |
| *** ccamacho1 has joined #openstack-infra | 07:04 | |
| nibalizer | jhesketh: a couple more in https://review.openstack.org/#/q/topic:gemfile-sync+status:open can be pushed on through | 07:04 |
| *** yolanda has joined #openstack-infra | 07:04 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT MERGE TESTING https://review.openstack.org/316436 | 07:04 |
| *** Nikolay_St has quit IRC | 07:04 | |
| *** bhavik has quit IRC | 07:04 | |
| *** ccamacho has quit IRC | 07:04 | |
| *** pahuang has quit IRC | 07:04 | |
| *** hichihara has quit IRC | 07:04 | |
| *** david-lyle has quit IRC | 07:04 | |
| *** dangers_away has quit IRC | 07:04 | |
| *** aviau has quit IRC | 07:04 | |
| *** tlian has quit IRC | 07:04 | |
| *** kushal has quit IRC | 07:04 | |
| *** fungi_lounge has quit IRC | 07:04 | |
| *** pots has quit IRC | 07:04 | |
| *** gmann has quit IRC | 07:04 | |
| *** dgonzalez has quit IRC | 07:04 | |
| *** xiangxinyong has quit IRC | 07:04 | |
| *** megm has quit IRC | 07:04 | |
| *** vtapia has quit IRC | 07:04 | |
| *** eantyshev has quit IRC | 07:04 | |
| *** moravec1 has quit IRC | 07:04 | |
| *** hamzy has quit IRC | 07:04 | |
| *** pcrews_ has quit IRC | 07:04 | |
| *** trash has quit IRC | 07:04 | |
| *** dmsimard has quit IRC | 07:04 | |
| *** slagle has quit IRC | 07:04 | |
| *** jd__ has quit IRC | 07:04 | |
| *** Hunner has quit IRC | 07:04 | |
| *** llu_ has quit IRC | 07:04 | |
| *** rm_work has quit IRC | 07:04 | |
| *** dmellado has quit IRC | 07:04 | |
| *** fabo has quit IRC | 07:04 | |
| *** charz_ has quit IRC | 07:04 | |
| *** dasm has quit IRC | 07:04 | |
| *** dgonzalez_ is now known as dgonzalez | 07:04 | |
| *** dasm_ is now known as dasm | 07:04 | |
| *** bhavik1 is now known as bhavik | 07:04 | |
| *** sarob has joined #openstack-infra | 07:05 | |
| * jhesketh does the dance of button pushing | 07:05 | |
| *** llu__ is now known as llu_ | 07:05 | |
| *** llu_ has left #openstack-infra | 07:05 | |
| *** Ravikiran_K has joined #openstack-infra | 07:06 | |
| *** permalac_ has quit IRC | 07:06 | |
| *** chem has joined #openstack-infra | 07:06 | |
| *** kdas__ has quit IRC | 07:06 | |
| *** kushal has joined #openstack-infra | 07:07 | |
| *** hdaniel has joined #openstack-infra | 07:07 | |
| *** flepied1 has quit IRC | 07:07 | |
| *** shashank_hegde has joined #openstack-infra | 07:08 | |
| *** Kennan2 has quit IRC | 07:09 | |
| *** _nadya_ has quit IRC | 07:09 | |
| *** sarob has quit IRC | 07:09 | |
| *** ccamacho1 is now known as ccamacho|afk | 07:10 | |
| *** yolanda has quit IRC | 07:10 | |
| *** lindsayk_ has joined #openstack-infra | 07:10 | |
| openstackgerrit | Merged openstack-infra/puppet-gerrit: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332526 | 07:13 |
| Srinu | fungi:hi | 07:13 |
| *** armax has quit IRC | 07:14 | |
| *** hichihara has joined #openstack-infra | 07:14 | |
| *** pahuang has joined #openstack-infra | 07:14 | |
| *** fungi_lounge has joined #openstack-infra | 07:14 | |
| *** moravec1 has joined #openstack-infra | 07:14 | |
| *** jd__ has joined #openstack-infra | 07:14 | |
| *** Hunner has joined #openstack-infra | 07:14 | |
| *** rm_work has joined #openstack-infra | 07:14 | |
| Srinu | fungi: when jenkins job is running slave went offline while build can you check this logs http://paste.openstack.org/show/521057/ | 07:15 |
| openstackgerrit | Merged openstack-infra/puppet-hound: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332532 | 07:16 |
| openstackgerrit | Merged openstack-infra/puppet-etherpad_lite: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332524 | 07:17 |
| openstackgerrit | Merged openstack-infra/puppet-openstackid: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332552 | 07:17 |
| *** tlian2 has quit IRC | 07:17 | |
| *** shashank_hegde has quit IRC | 07:18 | |
| nibalizer | jhesketh: kk im out | 07:18 |
| nibalizer | later days | 07:18 |
| jhesketh | nibalizer: no worries... it must be silly o'clock for you | 07:18 |
| *** salv-orl_ has quit IRC | 07:20 | |
| openstackgerrit | Merged openstack-infra/puppet-grafyaml: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332529 | 07:20 |
| Srinu | nibalizer:when jenkins job is running slave went offline while build can you check this logs http://paste.openstack.org/show/521057/ | 07:21 |
| *** pcaruana has joined #openstack-infra | 07:22 | |
| *** fabo_ is now known as fabo | 07:23 | |
| *** chlong has quit IRC | 07:25 | |
| *** stevemar has joined #openstack-infra | 07:26 | |
| *** stevemar_ has joined #openstack-infra | 07:27 | |
| *** amoralej|off is now known as amoralej | 07:28 | |
| *** ccamacho|afk is now known as ccamacho | 07:28 | |
| *** stevemar has quit IRC | 07:30 | |
| *** mhickey has joined #openstack-infra | 07:31 | |
| *** abregman_ has quit IRC | 07:31 | |
| *** jianghuaw has joined #openstack-infra | 07:32 | |
| *** stevemar_ has quit IRC | 07:32 | |
| *** abregman has joined #openstack-infra | 07:32 | |
| *** jlanoux has joined #openstack-infra | 07:33 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:35 | |
| jianghuaw | hi, in my CI I noticed that the keypair dedicated for nodepool sometime will be deleted from the cloud provider. Then I see nodepool will try to delete it once failed to boot server. | 07:35 |
| jianghuaw | https://github.com/openstack-infra/nodepool/blob/master/nodepool/nodepool.py#L970 | 07:35 |
| jianghuaw | But is it really correct? | 07:35 |
| *** Na3iL has joined #openstack-infra | 07:35 | |
| *** mhickey has quit IRC | 07:35 | |
| jianghuaw | https://github.com/openstack-infra/nodepool/blob/master/nodepool/nodepool.py#L940 | 07:36 |
| jianghuaw | if I create a keypair in the cloud, and specify it in self.provider.keypair, why delete it with createServer exception. | 07:37 |
| jianghuaw | it's possible to fail the createServer - there is no enough resource on the cloud. If just delete this keypair, it can't update image any more even in the next turn there are more resource available again. | 07:39 |
| jianghuaw | So it seems to me a bug. Maybe just delete the keypair if it's added automatically at https://github.com/openstack-infra/nodepool/blob/master/nodepool/nodepool.py#L947 | 07:40 |
| jianghuaw | Any thoughts? thanks. | 07:41 |
| *** flepied has joined #openstack-infra | 07:41 | |
| *** vincentll has joined #openstack-infra | 07:43 | |
| *** aviau_ has quit IRC | 07:43 | |
| *** aviau has joined #openstack-infra | 07:44 | |
| *** dfflanders has joined #openstack-infra | 07:44 | |
| *** sheeprine has quit IRC | 07:45 | |
| *** sheeprine_ is now known as sheeprine | 07:45 | |
| *** jtomasek has quit IRC | 07:45 | |
| *** yaume has joined #openstack-infra | 07:47 | |
| *** jtomasek has joined #openstack-infra | 07:47 | |
| *** ramishra has quit IRC | 07:47 | |
| *** ramishra has joined #openstack-infra | 07:48 | |
| *** mhickey has joined #openstack-infra | 07:48 | |
| *** jpich has joined #openstack-infra | 07:49 | |
| *** thorst_ has joined #openstack-infra | 07:49 | |
| *** rossella__ has quit IRC | 07:49 | |
| *** ramishra has quit IRC | 07:51 | |
| *** ramishra has joined #openstack-infra | 07:53 | |
| *** amitgandhinz has joined #openstack-infra | 07:53 | |
| *** flepied has quit IRC | 07:53 | |
| openstackgerrit | Merged openstack-infra/puppet-jenkins: Use new infra_spec_helper for gem dependencies https://review.openstack.org/332537 | 07:53 |
| *** thorst_ has quit IRC | 07:56 | |
| *** shashank_hegde has joined #openstack-infra | 07:56 | |
| *** electrofelix has joined #openstack-infra | 07:57 | |
| *** amitgandhinz has quit IRC | 07:58 | |
| *** Sukhdev_ has quit IRC | 07:58 | |
| *** Sukhdev has joined #openstack-infra | 07:58 | |
| *** ramishra has quit IRC | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:02 | |
| *** mhickey has quit IRC | 08:04 | |
| *** jaosorior_brb is now known as jaosorior | 08:05 | |
| *** ramishra has joined #openstack-infra | 08:05 | |
| *** Sukhdev has quit IRC | 08:05 | |
| *** e0ne has joined #openstack-infra | 08:06 | |
| *** yamahata has quit IRC | 08:09 | |
| *** lucasagomes is now known as lucas-brb | 08:11 | |
| *** flepied has joined #openstack-infra | 08:11 | |
| *** eezhova has joined #openstack-infra | 08:12 | |
| *** salv-orlando has joined #openstack-infra | 08:14 | |
| *** acoles_ is now known as acoles | 08:16 | |
| *** mhickey has joined #openstack-infra | 08:18 | |
| *** yolanda has joined #openstack-infra | 08:18 | |
| *** zeih has joined #openstack-infra | 08:18 | |
| *** dizquierdo has joined #openstack-infra | 08:20 | |
| *** BobBall_AWOL is now known as BobBall | 08:21 | |
| *** kzaitsev_mb has quit IRC | 08:21 | |
| *** dmk0202 has joined #openstack-infra | 08:23 | |
| *** ams__ has joined #openstack-infra | 08:25 | |
| *** jkraj has quit IRC | 08:25 | |
| AJaeger | hosanai: I'd like to discuss that approach with the other reviewers first a bit more, will try to do later today. | 08:27 |
| *** abregman has quit IRC | 08:27 | |
| *** stevemar has joined #openstack-infra | 08:27 | |
| *** amotoki has quit IRC | 08:28 | |
| hosanai | AJaeger: thanks! it's a good approach. if you find out good solutions, please let me know. I will follow it. ;-) | 08:28 |
| *** mohankumar has quit IRC | 08:29 | |
| *** zeih has quit IRC | 08:29 | |
| *** stevemar_ has joined #openstack-infra | 08:29 | |
| *** hasharAway has quit IRC | 08:30 | |
| *** Hal has joined #openstack-infra | 08:30 | |
| *** Hal is now known as Guest7579 | 08:30 | |
| *** hashar has joined #openstack-infra | 08:32 | |
| *** stevemar has quit IRC | 08:32 | |
| *** stevemar_ has quit IRC | 08:34 | |
| *** _nadya_ has joined #openstack-infra | 08:35 | |
| *** rhochmuth has joined #openstack-infra | 08:36 | |
| *** rhochmuth has quit IRC | 08:36 | |
| *** sambetts|afk is now known as sambetts | 08:37 | |
| *** bapalm has quit IRC | 08:37 | |
| *** derekh has joined #openstack-infra | 08:38 | |
| *** _nadya_ has quit IRC | 08:39 | |
| *** haaj has joined #openstack-infra | 08:40 | |
| *** _nadya_ has joined #openstack-infra | 08:40 | |
| *** bapalm has joined #openstack-infra | 08:42 | |
| *** hashar has quit IRC | 08:44 | |
| *** hashar_ has joined #openstack-infra | 08:44 | |
| *** haaj has quit IRC | 08:45 | |
| *** rdx_matt has joined #openstack-infra | 08:45 | |
| *** ramishra has quit IRC | 08:48 | |
| *** ramishra has joined #openstack-infra | 08:48 | |
| *** shashank_hegde has quit IRC | 08:49 | |
| *** rdx_matt_ has quit IRC | 08:49 | |
| *** HeOS has quit IRC | 08:52 | |
| *** amotoki has joined #openstack-infra | 08:53 | |
| *** amitgandhinz has joined #openstack-infra | 08:54 | |
| *** thorst_ has joined #openstack-infra | 08:54 | |
| *** dizquierdo has quit IRC | 08:54 | |
| BobBall | How do we set up a project so we can vote on it with our CI? Do we have to have a <project>-ci group? What settings do we need in gerrit to permit +1/-1 verified votes from members of this group? Project is fuel-plugins-xenserver FYI | 08:56 |
| *** zeih has joined #openstack-infra | 08:57 | |
| *** amitgandhinz has quit IRC | 08:58 | |
| *** rdx_matt has quit IRC | 08:59 | |
| *** dtardivel has joined #openstack-infra | 08:59 | |
| *** ramishra has quit IRC | 09:00 | |
| *** ramishra has joined #openstack-infra | 09:01 | |
| *** rdx_matt has joined #openstack-infra | 09:01 | |
| *** rossella__ has joined #openstack-infra | 09:01 | |
| *** thorst_ has quit IRC | 09:02 | |
| *** flepied has quit IRC | 09:02 | |
| *** esikachev has joined #openstack-infra | 09:04 | |
| *** sarob has joined #openstack-infra | 09:05 | |
| *** mhickey has quit IRC | 09:08 | |
| *** zeih has quit IRC | 09:09 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add playbook to launch a new node https://review.openstack.org/320466 | 09:09 |
| *** zeih has joined #openstack-infra | 09:09 | |
| *** jaosorior has quit IRC | 09:09 | |
| *** ccamacho has quit IRC | 09:10 | |
| *** jaosorior has joined #openstack-infra | 09:10 | |
| *** sarob has quit IRC | 09:10 | |
| *** rossella__ has quit IRC | 09:11 | |
| *** rhochmuth has joined #openstack-infra | 09:13 | |
| *** Guest7579 has quit IRC | 09:13 | |
| *** rhochmuth has quit IRC | 09:17 | |
| *** jlanoux has quit IRC | 09:18 | |
| *** jlanoux has joined #openstack-infra | 09:18 | |
| *** mhickey has joined #openstack-infra | 09:20 | |
| *** flepied has joined #openstack-infra | 09:20 | |
| * bkero waves good morning | 09:21 | |
| *** Jeffrey4l__ has joined #openstack-infra | 09:25 | |
| *** Jeffrey4l_ has quit IRC | 09:26 | |
| *** Na3iL has quit IRC | 09:26 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Add Ironic multinode job https://review.openstack.org/332490 | 09:26 |
| *** ccamacho has joined #openstack-infra | 09:27 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add playbook/role to delete an SSH keypair https://review.openstack.org/320468 | 09:27 |
| *** Na3iL has joined #openstack-infra | 09:28 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add playbook to launch a new node https://review.openstack.org/320466 | 09:29 |
| *** ociuhandu has joined #openstack-infra | 09:33 | |
| *** Jeffrey4l__ has quit IRC | 09:35 | |
| yolanda | BobBall, look at http://docs.openstack.org/infra/system-config/third_party.html | 09:35 |
| *** flepied has quit IRC | 09:36 | |
| *** jkraj has joined #openstack-infra | 09:37 | |
| *** _degorenko|afk is now known as degorenko | 09:38 | |
| BobBall | We have the CI already running - it just can't vote +1/-1 on xenserver-fuel-plugins project (which I own); I didn't see anything in there for how to add it | 09:38 |
| *** tosky has joined #openstack-infra | 09:44 | |
| *** kushal has quit IRC | 09:44 | |
| *** ifarkas has joined #openstack-infra | 09:44 | |
| vgridnev | how logs of periodic jobs can be viewed (like translations imports)? is there a place where results are posted? | 09:45 |
| *** rdx_matt has quit IRC | 09:46 | |
| *** rdx_matt has joined #openstack-infra | 09:47 | |
| *** zeih has quit IRC | 09:47 | |
| *** bapalm has quit IRC | 09:48 | |
| *** HeOS has joined #openstack-infra | 09:49 | |
| *** flepied has joined #openstack-infra | 09:49 | |
| *** claudiub has joined #openstack-infra | 09:51 | |
| yolanda | BobBall, so you have the gerrit service account created? | 09:51 |
| BobBall | Yes - and are voting on Nova. "XenServer Citrix CI" | 09:51 |
| *** lvdongbing has quit IRC | 09:52 | |
| BobBall | We want to use the same account to vote on this other project, but it seems that the vote is ignored for fuel-plugins-xenserver project | 09:52 |
| yolanda | so you are publishing the results to gerrit using jenkins gerrit trigger plugin? | 09:53 |
| *** bapalm has joined #openstack-infra | 09:54 | |
| BobBall | Yes (see https://review.openstack.org/#/c/331977/ as an example for where we commented but the vote was not added) | 09:54 |
| rcarrillocruz | that sounds like an acl perm issue | 09:54 |
| BobBall | Patch set 3, not 4 | 09:54 |
| *** kzaitsev_mb has joined #openstack-infra | 09:54 | |
| BobBall | Where would I look for the ACL permissions? | 09:54 |
| *** amitgandhinz has joined #openstack-infra | 09:55 | |
| *** Hal has joined #openstack-infra | 09:55 | |
| BobBall | Then I can maybe figure out what needs to change :) | 09:55 |
| *** Hal is now known as Guest91686 | 09:55 | |
| yolanda | look at nova, there is that nova-ci in acl | 09:55 |
| vgridnev | any answers on my question about periodic jobs? sorry for repeating | 09:56 |
| yolanda | i see you have a fuel-plugins-ci in your acl, are you using same account to post results? | 09:56 |
| BobBall | Oh, I do? Hmmm... That's interesting. Then I can ask for permissions from the fuel-plugins-ci owner to add my CI to that group. | 09:57 |
| BobBall | I imagine that will fix it, thanks! | 09:57 |
| yolanda | ++ | 09:57 |
| *** sree has quit IRC | 09:58 | |
| *** zhurong has quit IRC | 09:58 | |
| *** amitgandhinz has quit IRC | 09:59 | |
| *** thorst_ has joined #openstack-infra | 09:59 | |
| *** sree has joined #openstack-infra | 10:00 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/os-testr: Fix whitelist_file's regex construction with regex https://review.openstack.org/332721 | 10:02 |
| *** sree has quit IRC | 10:04 | |
| *** kushal has joined #openstack-infra | 10:06 | |
| *** esikachev has quit IRC | 10:06 | |
| *** thorst_ has quit IRC | 10:07 | |
| *** daemontool has joined #openstack-infra | 10:07 | |
| *** matrohon has quit IRC | 10:09 | |
| *** yolanda has quit IRC | 10:10 | |
| *** eezhova has quit IRC | 10:11 | |
| *** rhochmuth has joined #openstack-infra | 10:13 | |
| *** haaj has joined #openstack-infra | 10:14 | |
| openstackgerrit | Alexander Evseev proposed openstack-infra/puppet-pip: Fix openstack_pip provider to support old pip https://review.openstack.org/332725 | 10:14 |
| *** rhochmuth has quit IRC | 10:14 | |
| *** hosanai has quit IRC | 10:14 | |
| *** zeih has joined #openstack-infra | 10:17 | |
| *** nwkarsten has joined #openstack-infra | 10:18 | |
| ajafo | hi, I'm trying to achive cross repo dependencies with zuul/gerrit/jenkins, I've repo1 and repo2, so I added shared job(return just success) in gate pipeline of both repos, then I create patch with Depends-on: but it doesn't work, should I do something else? could you please point me to the right direction? | 10:19 |
| *** daemontool has quit IRC | 10:19 | |
| *** ccamacho has quit IRC | 10:20 | |
| ajafo | is it must in this case to use zuul_cloner or I can just checkout $ZUUL_URL/$ZUUL_PROJECT $ZUUL_REF ? | 10:22 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: RFC: Make server-side resolving of worklist fields togglable https://review.openstack.org/332728 | 10:22 |
| *** nwkarsten has quit IRC | 10:23 | |
| *** Jeffrey4l has joined #openstack-infra | 10:27 | |
| *** zeih has quit IRC | 10:28 | |
| *** mescanef has joined #openstack-infra | 10:29 | |
| *** hashar_ has quit IRC | 10:29 | |
| AJaeger | vgridnev: http://status.openstack.org/openstack-health/#/g/build_queue/periodic?groupKey=build_queue&searchJob=translation | 10:29 |
| AJaeger | vgridnev: all logs are on logs.openstack.org. | 10:29 |
| AJaeger | vgridnev: what log file do you want to look at? | 10:29 |
| *** sdague has joined #openstack-infra | 10:30 | |
| *** stevemar has joined #openstack-infra | 10:31 | |
| *** lucas-brb is now known as lucasagomes | 10:32 | |
| AJaeger | vgridnev: example, http://status.openstack.org/openstack-health/#/job/sahara-dashboard-propose-translation-update | 10:32 |
| vgridnev | AJaeger, right now none, I just wondering how logs for bug https://bugs.launchpad.net/neutron/+bug/1568708 were found | 10:32 |
| openstack | Launchpad bug 1568708 in networking-midonet "translation sync broken due to wrong usage of venv" [High,In progress] - Assigned to YAMAMOTO Takashi (yamamoto) | 10:32 |
| AJaeger | vgridnev: using the first URL and looking at failures ;) | 10:32 |
| AJaeger | vgridnev: I changed the period to 1 day to only look at current day... | 10:33 |
| AJaeger | = last run | 10:33 |
| vgridnev | thanks, AJaeger seems like openstack-health is really good way for tracking jobs | 10:33 |
| AJaeger | vgridnev: it is indeed! And sahara-dashboard is fixed now... | 10:33 |
| *** chem is now known as chem|lunch | 10:35 | |
| *** stevemar has quit IRC | 10:36 | |
| *** ccamacho has joined #openstack-infra | 10:36 | |
| AJaeger | vgridnev: see also http://lists.openstack.org/pipermail/openstack-i18n/2016-June/002228.html | 10:37 |
| *** Jeffrey4l has quit IRC | 10:40 | |
| *** tlian has joined #openstack-infra | 10:41 | |
| *** derekh has quit IRC | 10:42 | |
| *** rdx_matt_ has joined #openstack-infra | 10:43 | |
| *** ramishra has quit IRC | 10:45 | |
| *** rdx_matt has quit IRC | 10:46 | |
| *** dtantsur|afk is now known as dtantsur | 10:48 | |
| *** ramishra has joined #openstack-infra | 10:50 | |
| *** amitgandhinz has joined #openstack-infra | 10:55 | |
| *** jlanoux has quit IRC | 10:58 | |
| *** javeriak has quit IRC | 11:00 | |
| *** amitgandhinz has quit IRC | 11:00 | |
| *** thorst_ has joined #openstack-infra | 11:04 | |
| *** sarob has joined #openstack-infra | 11:06 | |
| *** ramishra has quit IRC | 11:07 | |
| *** Ravikiran_K has quit IRC | 11:09 | |
| *** ramishra has joined #openstack-infra | 11:09 | |
| *** kushal has quit IRC | 11:09 | |
| *** jtyr has joined #openstack-infra | 11:10 | |
| *** sarob has quit IRC | 11:11 | |
| *** thorst_ has quit IRC | 11:12 | |
| jtyr | Does anybody know how to test a test for jenkins-jobs? Running "jenkins-jobs test tests/publishers/fixtures/pipeline001.yaml" is throwing an error: The topmost collection in file 'tests/publishers/fixtures/pipeline001.yaml' must be a list, not a <class 'collections.OrderedDict'> | 11:12 |
| *** zeih has joined #openstack-infra | 11:13 | |
| *** ramishra has quit IRC | 11:14 | |
| *** rhochmuth has joined #openstack-infra | 11:15 | |
| *** rhochmuth has quit IRC | 11:15 | |
| Zara | storyboard bugsquash has now started in #openstack-sprint! come say hi, in there or in #storyboard or even both! | 11:18 |
| *** ramishra has joined #openstack-infra | 11:18 | |
| *** chlong has joined #openstack-infra | 11:21 | |
| *** kushal has joined #openstack-infra | 11:22 | |
| *** ddieterly has joined #openstack-infra | 11:23 | |
| openstackgerrit | Jiri Tyr proposed openstack-infra/jenkins-job-builder: Adding property-file into the pipeline publisher https://review.openstack.org/311150 | 11:25 |
| *** jlanoux has joined #openstack-infra | 11:31 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Make `/v1/worklists/:id/filters` resolve criteria https://review.openstack.org/332764 | 11:31 |
| *** ddieterly has quit IRC | 11:33 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Add gate-ironic-python-agent-tox-func to IPA gate https://review.openstack.org/332766 | 11:38 |
| *** rodrigods has quit IRC | 11:39 | |
| *** weshay has joined #openstack-infra | 11:39 | |
| *** rodrigods has joined #openstack-infra | 11:39 | |
| *** ddieterly has joined #openstack-infra | 11:41 | |
| *** ldnunes has joined #openstack-infra | 11:42 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 11:43 | |
| *** rdx_matt_ has quit IRC | 11:44 | |
| *** amoralej is now known as amoralej|lunch | 11:44 | |
| *** rdx_matt has joined #openstack-infra | 11:44 | |
| *** prabu has quit IRC | 11:45 | |
| *** zhurong has joined #openstack-infra | 11:47 | |
| *** sdake has quit IRC | 11:48 | |
| *** chlong has quit IRC | 11:51 | |
| *** ccamacho has quit IRC | 11:55 | |
| *** ccamacho has joined #openstack-infra | 11:56 | |
| *** sree has joined #openstack-infra | 11:56 | |
| *** amitgandhinz has joined #openstack-infra | 11:56 | |
| *** yamamot__ has quit IRC | 11:57 | |
| *** mescanef has quit IRC | 11:58 | |
| *** ociuhandu has quit IRC | 11:59 | |
| *** ddieterly has quit IRC | 11:59 | |
| *** thorst_ has joined #openstack-infra | 11:59 | |
| *** mescanef has joined #openstack-infra | 12:00 | |
| *** dmsimard1 has quit IRC | 12:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Add glance api-ref jobs https://review.openstack.org/315191 | 12:00 |
| *** dmsimard has joined #openstack-infra | 12:00 | |
| *** hashar has joined #openstack-infra | 12:00 | |
| *** amitgandhinz has quit IRC | 12:01 | |
| *** Guest91686 has quit IRC | 12:01 | |
| *** jtomasek_ has joined #openstack-infra | 12:02 | |
| *** sdake has joined #openstack-infra | 12:03 | |
| *** dprince has joined #openstack-infra | 12:03 | |
| *** zeih has quit IRC | 12:04 | |
| *** sarob has joined #openstack-infra | 12:04 | |
| *** eezhova has joined #openstack-infra | 12:06 | |
| *** aysyd has joined #openstack-infra | 12:06 | |
| *** esikachev has joined #openstack-infra | 12:07 | |
| *** sarob has quit IRC | 12:09 | |
| *** raildo-afk is now known as raildo | 12:09 | |
| *** rhallisey has joined #openstack-infra | 12:10 | |
| *** sorantis has joined #openstack-infra | 12:12 | |
| *** infra-red has joined #openstack-infra | 12:13 | |
| *** zeih has joined #openstack-infra | 12:13 | |
| *** e0ne has quit IRC | 12:13 | |
| *** e0ne has joined #openstack-infra | 12:14 | |
| sdake | morning folks | 12:14 |
| sdake | need some ci/cd consulting from someone that would know | 12:14 |
| sdake | we want to upload our images to docker hub | 12:14 |
| sdake | I am pretty sure we can build them and upload them in the gate window of 60 minutes | 12:15 |
| sdake | what i'm not clear on is how to get the credentials in securely | 12:15 |
| sdake | docker hub requires a command "docker login" | 12:15 |
| sdake | this requires a password | 12:15 |
| *** pfallenop has joined #openstack-infra | 12:15 | |
| sdake | we dont want the password in our git repo | 12:15 |
| sdake | its probably a similar problem to pypi | 12:16 |
| *** coolsvap has joined #openstack-infra | 12:17 | |
| rcarrillocruz | sdake: so, the publish pattern is to have a post job that does the upload | 12:18 |
| rcarrillocruz | then the creds | 12:18 |
| rcarrillocruz | would be put on the privileged slave | 12:18 |
| rcarrillocruz | that's how pypi uploads work for example | 12:19 |
| sdake | the creds in the privileged slave | 12:19 |
| sdake | how do I get those there? | 12:19 |
| *** jtomasek_ has quit IRC | 12:19 | |
| sdake | would openstack create a ci/cd credential? | 12:19 |
| sdake | and I'd add them to our publish jobs? | 12:19 |
| rcarrillocruz | you need to check with fungi , but essentially they would create a hiera key for your creds | 12:19 |
| sdake | rather to our pubilsh account? | 12:19 |
| rcarrillocruz | that is just deployed on the privileged slave | 12:19 |
| sdake | rcarrillocruz ok i'll ping fungi when he awakes :) | 12:20 |
| sdake | its probably earlly for him iirc | 12:20 |
| *** jtyr has left #openstack-infra | 12:20 | |
| rcarrillocruz | yeah, you need a core to handle the creds on the hiera | 12:21 |
| rcarrillocruz | i assume they will follow the same pattern i depicted | 12:21 |
| *** tpsilva has joined #openstack-infra | 12:21 | |
| *** moravec1 has quit IRC | 12:22 | |
| *** Hal has joined #openstack-infra | 12:22 | |
| sdake | coolsvap you want to chsae that down | 12:22 |
| rcarrillocruz | gate jobs would create the artifacts and upload to tarballs | 12:22 |
| *** Hal is now known as Guest38349 | 12:22 | |
| coolsvap | sdake, sure i can | 12:22 |
| rcarrillocruz | post jobs would take the tarballs artifact and upload to $site, that run from privileged slave | 12:23 |
| sdake | the upload to tarballs is a problem | 12:23 |
| *** makowals has joined #openstack-infra | 12:23 | |
| rcarrillocruz | that's not just the pattern for pypi, it was for maven for example | 12:23 |
| sdake | docker push uploads directly to docker hub | 12:23 |
| *** pgadiya has quit IRC | 12:23 | |
| AJaeger | sdake: why is the upload to tarballs a problem? | 12:23 |
| rcarrillocruz | and you can't do a docker push from a file to hub | 12:23 |
| rcarrillocruz | ? | 12:23 |
| sdake | iguess we could capture /var/lib/docker | 12:23 |
| sdake | and use that as the artifact | 12:23 |
| AJaeger | sdake: yeah, capture the file there - and then have another job to the push. | 12:24 |
| sdake | its a whoel lot of files | 12:24 |
| sdake | in a btrfs filesystem | 12:24 |
| coolsvap | sdake, i think we can create a tarball of a docker image | 12:24 |
| sdake | i am not sure it can just be tarred | 12:24 |
| AJaeger | We want separation of these jobs to protect the keys | 12:24 |
| rcarrillocruz | yeah, gate jobs run on unprivileged, post on privileged | 12:24 |
| sdake | we build about 130 containers in parallel | 12:24 |
| AJaeger | coolsvap, sdake : You can upload any kind of file to tarballs.o.o | 12:24 |
| coolsvap | with docker export | 12:25 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Add gate-ironic-agent-tox-func to IPA gate https://review.openstack.org/332766 | 12:25 |
| *** d0ugal has quit IRC | 12:25 | |
| sdake | coolsvap docker export is not helpful - there is no way to docker push a docker export | 12:25 |
| sdake | we could tarball /var/lib/docker after shutting docker down | 12:25 |
| *** d0ugal has joined #openstack-infra | 12:25 | |
| sdake | and untar it on the build slave | 12:25 |
| sdake | still, feel a bit out of my element here :) | 12:26 |
| coolsvap | sdake, if we can regularly update the tarballs on tarballs.o.o | 12:26 |
| *** Jeffrey4l has joined #openstack-infra | 12:26 | |
| *** ddieterly has joined #openstack-infra | 12:26 | |
| *** rfolco has joined #openstack-infra | 12:26 | |
| openstackgerrit | Niall Bunting proposed openstack-infra/project-config: Remove tests on magnum liberty. https://review.openstack.org/332780 | 12:26 |
| sdake | i thought tarballs ws only to upload tarballs of release software | 12:27 |
| coolsvap | we can have an experimental job which imports images from tar and push to docker hub | 12:27 |
| AJaeger | sdake, coolsvap: figure out how to do it from the docker side with such a split setup, write it up, and then let's design the integration into the CI system. | 12:27 |
| sdake | not the actual artifacts produced by the build | 12:27 |
| *** xyang1 has joined #openstack-infra | 12:27 | |
| *** thiagop has joined #openstack-infra | 12:27 | |
| sdake | AJaeger cool sounds like a workable approach | 12:27 |
| sdake | AJaeger so we can uplod two different tarballs to too on a publish job? | 12:27 |
| AJaeger | sdake: we upload debian packages there and translation source like http://tarballs.openstack.org/translation-source/ceilometer/master/ceilometer/locale/ | 12:27 |
| AJaeger | sdake: you can upload different artefacts to t.o.o - whatever format works best for this | 12:28 |
| *** markusry has joined #openstack-infra | 12:28 | |
| sdake | AJaeger the secure gate job would have to run docker and docker push a whole lot ;) | 12:28 |
| AJaeger | sdake: docker itself as well? ;( Let's see what fungi and others think. It should be a simple upload if possible. | 12:29 |
| sdake | AJaeger docker does the uploading | 12:29 |
| AJaeger | sdake: I would rather not have the docker daemon running... | 12:30 |
| sdake | AJaeger docker is the only thing that understands the structure of /var/lib/docker | 12:30 |
| sdake | roger | 12:30 |
| AJaeger | sdake: are there no alternatives? A simple docker uploader? | 12:30 |
| sdake | i think that is why uploading to tarballs.oo is a proble m:) | 12:30 |
| sdake | AJaeger i'll look around but find it unlikely | 12:31 |
| sdake | the only python api i know of requires docker daemon running | 12:31 |
| AJaeger | sdake: write it up *high level*, so that you can reference it - and then let's discuss some more here once the rest of the infra team is awake. | 12:31 |
| sdake | coolsvap can you take that on then ? | 12:31 |
| coolsvap | sdake, yes | 12:31 |
| sdake | thanks AJaeger and coolsvap | 12:31 |
| sdake | docker's credential exchange isn't even written in python atm | 12:32 |
| sdake | there is no way I know of to get creds without using docker login | 12:32 |
| sdake | (which requires a daemon) | 12:32 |
| sdake | these are obviously just rest api calls | 12:32 |
| sdake | but undocumented i htink | 12:33 |
| AJaeger | ;( | 12:35 |
| *** chem|lunch is now known as chem | 12:35 | |
| sdake | docker login can get creds independently and then those creds can be used on a diffeerent system | 12:36 |
| sdake | but only the docker go code knows how to decode the credentials file | 12:37 |
| *** Na3iL has quit IRC | 12:37 | |
| *** virus has joined #openstack-infra | 12:37 | |
| *** pfallenop has quit IRC | 12:37 | |
| *** pfallenop has joined #openstack-infra | 12:38 | |
| *** virus has quit IRC | 12:38 | |
| AJaeger | sdake: let's see what clarkb, fungi, or jeblair think when they read backscroll of our discussion | 12:38 |
| sdake | sounds goodd | 12:38 |
| *** dfflanders has quit IRC | 12:38 | |
| *** virus has joined #openstack-infra | 12:38 | |
| AJaeger | otherwise, summarize and reference, please ;) | 12:39 |
| *** salv-orl_ has joined #openstack-infra | 12:39 | |
| sdake | ya coolsvap is going to do that | 12:39 |
| *** julim has quit IRC | 12:39 | |
| coolsvap | yes i will be back after sometime we can discuss that | 12:39 |
| *** burgerk has joined #openstack-infra | 12:40 | |
| *** zeih has quit IRC | 12:41 | |
| *** Srinu has quit IRC | 12:41 | |
| *** salv-orlando has quit IRC | 12:41 | |
| sdake | coolsvap write a email message about what is needed and send to openstack-infra | 12:41 |
| sdake | explain the idea aobut tarballing /var/lib/docker | 12:41 |
| *** ddieterly has quit IRC | 12:42 | |
| sdake | i dont even know if that would work tbh :) | 12:42 |
| coolsvap | :) | 12:42 |
| sdake | it would work without btrfs | 12:42 |
| sdake | dd the btrfs might be an option | 12:42 |
| sdake | ok back to bed - too early to think about such things :) | 12:43 |
| *** sdake has quit IRC | 12:43 | |
| *** edmondsw has joined #openstack-infra | 12:43 | |
| *** psilvad has joined #openstack-infra | 12:45 | |
| *** rlandy has joined #openstack-infra | 12:45 | |
| *** Na3iL has joined #openstack-infra | 12:49 | |
| *** zeih has joined #openstack-infra | 12:49 | |
| *** gouthamr has joined #openstack-infra | 12:49 | |
| *** binbincong has quit IRC | 12:50 | |
| *** virus has left #openstack-infra | 12:52 | |
| *** infra-red has quit IRC | 12:52 | |
| *** vhosakot has joined #openstack-infra | 12:53 | |
| *** ramishra has quit IRC | 12:53 | |
| *** ArxCruz has joined #openstack-infra | 12:55 | |
| *** sorantis has quit IRC | 12:56 | |
| *** xenogear has joined #openstack-infra | 12:56 | |
| *** amitgandhinz has joined #openstack-infra | 12:57 | |
| *** mriedem has joined #openstack-infra | 12:57 | |
| *** trown|outtypewww is now known as trown | 12:58 | |
| openstackgerrit | Alexander Evseev proposed openstack-infra/puppet-pip: Fix search of latest package version https://review.openstack.org/332793 | 12:58 |
| openstackgerrit | Alexander Evseev proposed openstack-infra/puppet-pip: Fix search of latest package version https://review.openstack.org/332793 | 13:01 |
| *** thiagop is now known as thiagop-afk | 13:01 | |
| *** amitgandhinz has quit IRC | 13:01 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config: python-saharaclient is required also in PROJECTS https://review.openstack.org/332797 | 13:01 |
| *** esikachev has quit IRC | 13:02 | |
| *** yamamoto has joined #openstack-infra | 13:02 | |
| pabelanger | morning | 13:02 |
| *** salv-orl_ has quit IRC | 13:03 | |
| *** lucasagomes is now known as lucas-hungry | 13:03 | |
| EmilienM | pabelanger: morning | 13:03 |
| EmilienM | pabelanger: ok today I see failures | 13:03 |
| EmilienM | http://logs.openstack.org/91/328591/26/check/gate-puppet-openstack-integration-3-scenario001-tempest-ubuntu-xenial/c28e4d4/console.html#_2016-06-22_11_50_24_233421 | 13:03 |
| pabelanger | jeblair: mordred: happy to report, the ansible-playbook to restart zuul-launchers worked as expected. | 13:03 |
| EmilienM | zuul killing the job in the middle | 13:04 |
| pabelanger | EmilienM: Yes, this is our async issue. We are still trying to address the issue. | 13:04 |
| pabelanger | EmilienM: we need to enable debug logs in ansible to see what is going on | 13:05 |
| pabelanger | EmilienM: for the moment, you can simply recheck | 13:05 |
| *** rhochmuth has joined #openstack-infra | 13:05 | |
| *** julim has joined #openstack-infra | 13:06 | |
| *** binbincong has joined #openstack-infra | 13:06 | |
| *** dizquierdo has joined #openstack-infra | 13:06 | |
| *** chlong has joined #openstack-infra | 13:06 | |
| *** dtardivel has quit IRC | 13:07 | |
| *** permalac has joined #openstack-infra | 13:07 | |
| EmilienM | pabelanger: ok | 13:07 |
| EmilienM | pabelanger: who has this issue? | 13:08 |
| EmilienM | pabelanger: only us and ansible or more? | 13:08 |
| *** kgiusti has joined #openstack-infra | 13:08 | |
| *** links has quit IRC | 13:09 | |
| *** vhosakot has quit IRC | 13:09 | |
| *** slagle_ is now known as slagle | 13:10 | |
| andreaf | I saw a couple of jobs failing with "POST_FAILURE" - anyone around who can explain me what does that mean? Is this something new in the no-jenkins world? | 13:11 |
| *** mdrabe has joined #openstack-infra | 13:11 | |
| *** woodster_ has joined #openstack-infra | 13:11 | |
| *** salv-orlando has joined #openstack-infra | 13:12 | |
| pabelanger | EmilienM: seems to be limited to ansible and puppet jobs, but not 100% | 13:12 |
| bkero | Does zuul keep a log of things like that? | 13:12 |
| *** garyk1 has quit IRC | 13:12 | |
| pabelanger | andreaf: POST_FAILURE means our upload tasks failed for some reason. Which review? | 13:12 |
| EmilienM | pabelanger: I can't understand why, we don't do special thing | 13:12 |
| andreaf | pabelanger: https://review.openstack.org/266398 | 13:13 |
| pabelanger | bkero: yes, but we need better (more verbose) logs to better understand the issue. | 13:13 |
| bkero | agreed | 13:13 |
| andreaf | pabelanger: or https://review.openstack.org/#/c/331062/ | 13:13 |
| *** rossella__ has joined #openstack-infra | 13:15 | |
| *** eezhova has quit IRC | 13:16 | |
| AJaeger | mmh, that's tempest. | 13:18 |
| pabelanger | andreaf: looks like an SSH error when we tried to copy off the logs: http://paste.openstack.org/show/521165/ | 13:18 |
| *** zeih has quit IRC | 13:18 | |
| pabelanger | So, job failed for some reason, then ansible could not SSH into the node because of public key failure | 13:19 |
| pabelanger | not sure why right now | 13:19 |
| EmilienM | pabelanger: puppet jobs do not ssh | 13:20 |
| EmilienM | pabelanger: except beaker jobs but they don't fail much | 13:20 |
| EmilienM | pabelanger: only 'p-o-i' | 13:20 |
| pabelanger | EmilienM: that is a different failure | 13:21 |
| *** eezhova has joined #openstack-infra | 13:21 | |
| bkero | Man, those ansible tasks need some better names | 13:21 |
| bkero | "msg": "Warning: Permanently added '104.130.209.148' (ECDSA) to the list of known hosts.\r\nPermission denied (publickey). | 13:22 |
| *** moravec has joined #openstack-infra | 13:22 | |
| bkero | I wouldn't expect that the jobs would alter ~jenkins/.ssh/authorized_keys | 13:23 |
| andreaf | pabelanger: ok - well I've seen a few of those, so I was wondering if that could signify a more general issue, perhaps on a provider or so | 13:23 |
| *** nwkarsten has joined #openstack-infra | 13:23 | |
| pabelanger | andreaf: it is _possible_ this is specific to rax, but need to review more failures | 13:23 |
| *** BigWillie has joined #openstack-infra | 13:24 | |
| EmilienM | pabelanger: also seeing lot of POST_FAILURE like some other folks | 13:25 |
| andreaf | pabelanger: hehe too bad that since logs files were not copied we cannot search for this in kiaba | 13:26 |
| EmilienM | pabelanger: ie: https://review.openstack.org/330498 | 13:26 |
| *** esikachev has joined #openstack-infra | 13:26 | |
| pabelanger | EmilienM: okay, lets start a new etherpad. https://etherpad.openstack.org/p/zuul-post-failures | 13:26 |
| pabelanger | do you mind adding the failures there? | 13:26 |
| andreaf | pabelanger: https://review.openstack.org/#/c/331065/, https://review.openstack.org/#/c/331064/6, https://review.openstack.org/#/c/331063/6 | 13:27 |
| *** tonytan4ever has joined #openstack-infra | 13:27 | |
| *** zeih has joined #openstack-infra | 13:27 | |
| AJaeger | looking at andreaf's failures, it's puppet as well - like gate gate-puppet-openstack-integration-3-scenario001-tempest-centos-7-nv | 13:28 |
| AJaeger | so, andreaf's problem and EmilienM's look related. | 13:28 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/irc-meetings: Change Kolla meetings to weekly https://review.openstack.org/332813 | 13:28 |
| *** hashar has quit IRC | 13:29 | |
| mordred | pabelanger: woot! | 13:30 |
| *** njohnston|afk is now known as njohnston | 13:30 | |
| *** hashar has joined #openstack-infra | 13:30 | |
| *** amitgandhinz has joined #openstack-infra | 13:30 | |
| *** zhurong has quit IRC | 13:31 | |
| mordred | sdague, coolsvap: sorry I wasn't active in channel when you were talking about docker push stuff ... | 13:31 |
| mordred | gah | 13:31 |
| mordred | sdague: sorry, wrong sda human | 13:31 |
| *** rossella__ has quit IRC | 13:32 | |
| *** esberglu has joined #openstack-infra | 13:32 | |
| mordred | coolsvap: loop me in on your thinking/writeup when you do - I would also like for us to be able to publish docker images and I have done at least some amount of thinking about how to do it | 13:32 |
| EmilienM | puppet jobs are really unstable, zuul is killing them | 13:32 |
| EmilienM | we can't land any patch, our gate is broken | 13:33 |
| bkero | :'( | 13:34 |
| *** ddieterly has joined #openstack-infra | 13:34 | |
| *** zhurong has joined #openstack-infra | 13:35 | |
| pabelanger | EmilienM: http://logs.openstack.org/20/328620/4/check/gate-puppet-keystone-puppet-beaker-rspec-centos-7/5379b91/_zuul_ansible/logs/ansible.txt | 13:35 |
| pabelanger | there is a problem with your log files | 13:36 |
| pabelanger | sudoers.txt looks to have the wrong permissions | 13:36 |
| EmilienM | e/gate-puppet-keystone-puppet-beaker-rspec-centos-7/logs/sudoers.txt\": Permission denied ( | 13:36 |
| EmilienM | ok looking at it | 13:36 |
| EmilienM | why is it random? | 13:36 |
| EmilienM | why is it only since recently and not before? | 13:36 |
| hrybacki | could someone help me (or point me in the right direction) merge two accounts on review.openstack.org by chance? | 13:37 |
| pabelanger | EmilienM: it seems like all the failures are related to centos | 13:37 |
| bkero | Maybe it's a bad nodepool image? | 13:37 |
| *** sdake has joined #openstack-infra | 13:37 | |
| pabelanger | so, something must have changed with sudo | 13:37 |
| EmilienM | pabelanger: I also saw some xenial jobs failures | 13:37 |
| *** amoralej|lunch is now known as amoralej | 13:38 | |
| *** inc0 has joined #openstack-infra | 13:38 | |
| EmilienM | pabelanger: but it was async thing for xenial | 13:38 |
| pabelanger | bkero: not sure bad, but maybe out of date | 13:38 |
| pabelanger | checking that now | 13:38 |
| *** savihou has joined #openstack-infra | 13:38 | |
| bkero | pabelanger: Perhaps something changed with sudo in the nodepool image and it isn't consistently deployed | 13:38 |
| *** pradk has joined #openstack-infra | 13:39 | |
| EmilienM | pabelanger: dropping sudoers thing, useless for us https://review.openstack.org/332818 | 13:41 |
| wznoinsk | hi infra | 13:41 |
| * krotscheck giggles. He hopes everyone minds their P's and Q's! | 13:41 | |
| pabelanger | EmilienM: a better fix would be to chown -R jenkins:jenkins logs, chmod -R 0644 logs as the last step in your copy_logs.sh file | 13:42 |
| pabelanger | this way rsync from zuul-launcher can read all the files | 13:42 |
| EmilienM | pabelanger: ok, doing another patch - we'll see | 13:43 |
| pabelanger | EmilienM: basically, rsync will fail on the first file it cannot read, which looks to be the cause of some of the POST_FAILURES for you. Why it is inconsistent, I do not know at this point | 13:44 |
| *** burgerk has quit IRC | 13:45 | |
| EmilienM | pabelanger: ok -- and for xenial jobs, do we know why this async thing happens? | 13:45 |
| *** ddieterly is now known as ddieterly[away] | 13:45 | |
| *** lucas-hungry is now known as lucasagomes | 13:45 | |
| pabelanger | EmilienM: have a review for that? was just looking for one | 13:45 |
| *** Nikolay_1t has quit IRC | 13:46 | |
| *** eharney has joined #openstack-infra | 13:47 | |
| EmilienM | pabelanger: http://logs.openstack.org/98/330498/10/check/gate-puppet-openstack-integration-3-scenario001-tempest-ubuntu-xenial/e49bcf6/console.html#_2016-06-22_11_59_28_533307 | 13:47 |
| EmilienM | pabelanger: please review https://review.openstack.org/332820 | 13:47 |
| pabelanger | EmilienM: that is our async timeout issue, that we are still trying to debug | 13:48 |
| *** sheeprine has quit IRC | 13:48 | |
| *** sheeprine has joined #openstack-infra | 13:48 | |
| EmilienM | pabelanger: ack | 13:48 |
| *** chlong has quit IRC | 13:49 | |
| pabelanger | I think (completely guessing) the async_status wrapper that runs on the node is getting an exception for some reason. Maybe OOM or something else, but right now we cannot see what is happening properly | 13:49 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config: python-saharaclient is required also in PROJECTS https://review.openstack.org/332797 | 13:49 |
| pabelanger | so, we need to increase our verbosity and hold a node so we can properly debug it | 13:49 |
| pabelanger | but, holding the node is the tricky part, since the failures somewhat random | 13:50 |
| mordred | krotscheck: there are some good names for both! | 13:50 |
| bkero | Start a recheck, hold node from one of the previously failed tests? | 13:51 |
| pabelanger | bkero: I've done that about 30 times now | 13:51 |
| bkero | Yuck :( | 13:51 |
| pabelanger | never caught a failure | 13:51 |
| *** PollyZ has joined #openstack-infra | 13:51 | |
| bkero | heisenbug | 13:52 |
| *** rbrndt has joined #openstack-infra | 13:53 | |
| *** erlon has joined #openstack-infra | 13:54 | |
| * bkero makes a note to make those task names a little more verbose | 13:54 | |
| *** ddieterly[away] is now known as ddieterly | 13:55 | |
| rcarrillocruz | pabelanger , mordred : https://review.openstack.org/#/c/332435/6 , to generate tmp keypair on the cloud launcher itself | 13:56 |
| krotscheck | mordred: I really want Podunk to win :D | 13:56 |
| *** derekh has joined #openstack-infra | 13:57 | |
| pabelanger | rcarrillocruz: ack, looking | 13:57 |
| rcarrillocruz | btw, i'm on and off all this week, at berlin opnfv | 13:57 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Add gate-ironic-agent-tox-func to IPA gate https://review.openstack.org/332766 | 13:57 |
| *** baoli has joined #openstack-infra | 13:59 | |
| *** isaacb has quit IRC | 13:59 | |
| pabelanger | rcarrillocruz: so, ansible-role-cloud-launcher is testing against devstack now? | 14:00 |
| rcarrillocruz | yeah, it's gating for a while now | 14:00 |
| *** baoli_ has joined #openstack-infra | 14:00 | |
| pabelanger | rcarrillocruz: nice! Ya, I haven't look in while. Going to do that now | 14:00 |
| rcarrillocruz | the reason why i had to tweak some tests on the change, for some reason ansible-lint was complaining now for things it didn't up till now | 14:00 |
| rcarrillocruz | and i abandoned the system-config role for generating tmp keypair | 14:01 |
| *** zeih has quit IRC | 14:01 | |
| *** baoli__ has joined #openstack-infra | 14:01 | |
| rcarrillocruz | as the cloud launcher can do it with above change with a flag | 14:01 |
| pabelanger | ack | 14:02 |
| *** sree has quit IRC | 14:02 | |
| fungi | sdake: AJaeger: right, so the way we've successfully handled privilege separation for other kinds of uploads is that we do the builds on single-use machines which are allowed to run untrusted code but have no access to credentials for third-party systems, then we have a second stage job on a trusted machine with access to the credentials which runs only statically defined commands to obtain the built | 14:03 |
| fungi | artifact from somewhere and push it to the remotely authenticated service | 14:03 |
| *** sree has joined #openstack-infra | 14:03 | |
| sdake | fungi so a daemon is out as an optoin o the ssecure server? | 14:03 |
| sdake | i don't know if coolsvap wrote the email yet - i've been sleeping | 14:04 |
| fungi | sdake: not sure what you mean by a daemon. it runs a variety of daemons (init, syslog, et cetera) | 14:04 |
| *** kushal has quit IRC | 14:04 | |
| sdake | it would need to run the docker daemon | 14:04 |
| sdake | to do the push | 14:04 |
| *** _ari_ has joined #openstack-infra | 14:04 | |
| sdake | we could script all the pushes | 14:04 |
| openstackgerrit | Merged openstack-infra/storyboard: Fix GET /v1/worklists/:id/filters/:id https://review.openstack.org/332210 | 14:04 |
| *** sree__ has joined #openstack-infra | 14:04 | |
| *** baoli has quit IRC | 14:04 | |
| sdake | only docker daemon knows how to push | 14:04 |
| *** savihou has quit IRC | 14:04 | |
| sdake | there is no non-daemon equivalent | 14:04 |
| *** jkraj has quit IRC | 14:04 | |
| sdake | (that I know of) | 14:05 |
| fungi | i don't know anything about docker other than it's a containery thing, so not sure what the "docker daemon" is but i'll go read up on it | 14:05 |
| *** baoli_ has quit IRC | 14:05 | |
| *** savihou has joined #openstack-infra | 14:05 | |
| sdake | fungi when you instaall docker, their a daemon which handles rqeuets | 14:05 |
| sdake | fungi from a local or remte client | 14:05 |
| sdake | typically local | 14:05 |
| * fungi hasn't used containers more sophisticated than freebsd jails up to now, and that was years ago | 14:05 | |
| sdake | so docker login gets credentials to login | 14:05 |
| *** hongbin has joined #openstack-infra | 14:05 | |
| sdake | docker push pushes to a registry | 14:05 |
| *** chlong has joined #openstack-infra | 14:06 | |
| sdake | docker push contacts the docker daemon to do the push work | 14:06 |
| *** gordc has joined #openstack-infra | 14:06 | |
| fungi | okay, so a persistent broker process to handle key management i guess? | 14:06 |
| dims | sdake : fungi : i worked out a way to start a docker daemon to run functional tests for python-k8sclient (gate-python-k8sclient-tox-with-sudo-py27-functional) | 14:06 |
| sdake | it also has a filesystem database which contains the files docker needs | 14:06 |
| fungi | sort of like ssh-agent or gpg-agent? | 14:06 |
| fungi | ahh | 14:07 |
| sdake | we were thinking to transfer this filesystem /var/lib/docker to tarballs.oo | 14:07 |
| sdake | then docker can read it at startup and do its push operation | 14:07 |
| fungi | i mean, if it being persistent is an issue, i suppose we could start the agent at the beginning of the job, do whatever interactions are needed with it, and then stop it again | 14:07 |
| sdake | one of the many complications is the /var/lib/docker fs is btrfs | 14:08 |
| sdake | so i don't know if a simple tarball will do | 14:08 |
| *** sree has quit IRC | 14:08 | |
| fungi | btrfs on a loop block device i guess? | 14:08 |
| sdake | /var/lib/docker contains the on disk database | 14:08 |
| sdake | yup loopback | 14:08 |
| sdake | actually not | 14:08 |
| fungi | i suppose you could just take an image of that instead if it's small | 14:08 |
| sdake | i think it usess th estorage on the nodes | 14:08 |
| sdake | it is not small | 14:08 |
| sdake | it is atleast 10gb | 14:08 |
| fungi | what is it storing in there? container images? | 14:09 |
| sdake | right | 14:09 |
| fungi | ahh | 14:09 |
| *** sree__ has quit IRC | 14:09 | |
| sdake | and other stuff too | 14:09 |
| sdake | but mostly container images | 14:09 |
| sdake | i have taken a /var/lib/docker from one system to another if it was lvm backed | 14:09 |
| sdake | but our gate jbos run btrfs backed because lvm doesn't work all that well | 14:10 |
| sdake | docker community hasdeprecated it iirc | 14:10 |
| fungi | i'm still a little confused how/where it creates its btrfs filesystem. it needs a dedicated partition, can't use a virtual block device? | 14:10 |
| sdake | we create the btrfs filesystem in our gate job | 14:10 |
| sdake | let me show you code | 14:10 |
| *** ajmiller has joined #openstack-infra | 14:10 | |
| *** rhochmuth has quit IRC | 14:10 | |
| derekh | pabelanger: did you have a chance to look at the afs server for rh2? | 14:10 |
| sdake | fungi https://github.com/openstack/kolla/blob/master/tools/setup_RedHat.sh#L21 | 14:11 |
| pabelanger | derekh: not yet, I can try launching a server this afternoon. For now, you can remove the configure_mirror.sh script in nodepool.yaml for rh2, that will allow the review to land. | 14:12 |
| pabelanger | then, once AFS is working, we can add it back | 14:12 |
| sdake | it is a loopback device with 20gb fungi | 14:12 |
| sdake | https://github.com/openstack/kolla/blob/master/tools/setup_RedHat.sh#L16 | 14:12 |
| derekh | pabelanger: ok | 14:13 |
| sdake | fwiw it probably compresses well (the loopback device) | 14:13 |
| fungi | okay, so on a virtual block device, not terrible | 14:13 |
| fungi | sdake: still, given the level of complexity and resource demands, i'm suspecting that at a minimum we'll need a separate machine which only handles this job, so that may give us a little additional leeway in what runs on it | 14:13 |
| sdake | AJaeger had issues with running docker on the publish machine | 14:13 |
| *** lvdongbing has joined #openstack-infra | 14:13 | |
| sdake | fungi we can use any gate machine to run the job | 14:14 |
| sdake | it doesn't need extra resources - our builds take 20 mins in the gate atm | 14:14 |
| fungi | sdake: not sure how we'd get the credentials onto it in that case | 14:14 |
| sdake | fungi right, that is where i am stuck | 14:14 |
| sdake | so we tlaked about uploading that loopbck file to tarballs.oo | 14:14 |
| sdake | and using the publish job to pull the loopback from there | 14:14 |
| fungi | our machines with privileged access to upload to third-party publication repositories are long-lived machines with the credentials installed on them via configuration management | 14:15 |
| sdake | AJaeger 's brainstorming | 14:15 |
| *** dizquierdo has quit IRC | 14:15 | |
| *** droyo has joined #openstack-infra | 14:15 | |
| sdake | fungi didnt know that | 14:15 |
| sdake | fungi makes sense though | 14:15 |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Add a temporary tripleo cloud "rh2" https://review.openstack.org/330514 | 14:15 |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Add a experimental tripleo job on rh2 https://review.openstack.org/330515 | 14:15 |
| sdake | fungi wecould add you an account to our kolla repo in dockerhub | 14:15 |
| derekh | pabelanger: thanks, ^^, I was't sure what you meant yesterday | 14:15 |
| *** ArxCruz has quit IRC | 14:15 | |
| sdake | or inra | 14:16 |
| sdake | or whoever | 14:16 |
| fungi | so nodepool doesn't have access to them, and they live in a completely separate tenant from our general use job nodes | 14:16 |
| sdake | ust as long s the creds dont leak | 14:16 |
| *** savihou has quit IRC | 14:16 | |
| sdake | fungi right this is where AJaeger came up with the idea of uploading to tarballs.oo | 14:16 |
| sdake | fungi i am less concerned with nightlyl images (although that would be great) - would prefer on tagging the release jobs release all the artifacts | 14:17 |
| *** savihou has joined #openstack-infra | 14:17 | |
| sdake | atm we are doing it manually | 14:17 |
| *** ArxCruz has joined #openstack-infra | 14:17 | |
| fungi | i'd be curious to know how much those would compress, and how much storage utilization growth we're talking about | 14:17 |
| sdake | making matters more complex sometimes the push fails for a whole lew of reasons | 14:17 |
| sdake | fungi we release a z stream every 45 days | 14:17 |
| fungi | not the least of which is probably "the internet" | 14:17 |
| sdake | fungi lol you goti t :) | 14:18 |
| *** zz_dimtruck is now known as dimtruck | 14:18 | |
| sdake | fungi so the growth is size of compressed images * number of stable branches * 365/45 | 14:18 |
| sdake | per year | 14:18 |
| *** tosky has quit IRC | 14:18 | |
| sdake | let me run a gate jobo and compress that loopback file and see how big it is | 14:19 |
| *** lvdongbing has quit IRC | 14:19 | |
| sdake | i keep telling people in our community nightly builds are not going to work because of the size constraints | 14:19 |
| sdake | but maybe tagged builds are a possiblity | 14:19 |
| fungi | between this and some of the packaging automation, i'm starting to think we need to come up with a more effective mechanism for sharing files between interrelated jobs that doesn't require distinct upload/download steps (though would still probably mean having it all transit the network since i don't think we could manage to solve it with shared block storage) | 14:19 |
| *** ociuhandu has joined #openstack-infra | 14:20 | |
| sdake | shared block storage would work in this case as long as there was only one accessor at a time | 14:20 |
| zaro | morning | 14:20 |
| *** tosky has joined #openstack-infra | 14:20 | |
| sdake | but the shared block would have to be semi-persistent | 14:20 |
| sdake | long enough for the publish job to run | 14:20 |
| *** rajinir has joined #openstack-infra | 14:21 | |
| sdake | AJaeger pretty much had tarballs.oo or bust :) | 14:21 |
| wznoinsk | are the 3rd party CI meetings still on? | 14:21 |
| wznoinsk | ignore pls, found them - Third Party meetings | 14:22 |
| *** signed8bit has joined #openstack-infra | 14:22 | |
| *** yamahata has joined #openstack-infra | 14:22 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:22 | |
| *** zeih has joined #openstack-infra | 14:23 | |
| fungi | i mean, if there was a mechanism by which we could delegate access tokens to the upload service we could have single-use workers doing the builds also upload. we worked out a pretty good solution for doing that with swift, but that's because zuul could grant one job node temporary access to a specific subtree of the container preventing it from writing wherever it wanted and making the credentials | 14:23 |
| fungi | mostly useless even if they leaked later | 14:23 |
| *** eharney has quit IRC | 14:24 | |
| *** mionkin has joined #openstack-infra | 14:25 | |
| sdake | fungi i'mm not sure i quite understand what you just said, but docker does allow you to have credentials in a group - similiar to how github works | 14:26 |
| sdake | so we could add an openstackkollaci user to our dockerhub account | 14:26 |
| wznoinsk | btw. I see some inconsistency on eavesdrop.o.o, http://eavesdrop.openstack.org/calendars/sriov-meeting.ics - opens nice calendar entry with the schedule while http://eavesdrop.openstack.org/calendars/third-party-meeting.ics opens a completely new calendar with an entry in it - not a biggie but inconsisten | 14:27 |
| fungi | sdake: you would need to be able to generate one-time-access (or short-expiry) credentials unique to each job run which only granted access to the exact location you wanted the file uploaded and not the entire organization or whatever | 14:27 |
| sdake | fungi what would happen there is the credentials would need to be inserted into the worker | 14:27 |
| jeblair | fungi: are you saying that you don't want the build/upload job split? | 14:27 |
| *** sorantis has joined #openstack-infra | 14:27 | |
| fungi | jeblair: i'm saying we may not effectively be able to split build/upload if there is 10gb worth of data that needs to get shuffled between job workers | 14:28 |
| sdake | fungi we upload containers to the entire organization (unless you mean making a n openstack organization) | 14:28 |
| sdake | note 10gb is just a guess ;) | 14:28 |
| fungi | sdake: yeah, just wondering if you could have an account temporarily granted access to only upload one container with a given name/path | 14:29 |
| *** isaacb has joined #openstack-infra | 14:29 | |
| sdake | fungi if docker provided an api to their registry yes, but they dont, so no :( | 14:29 |
| *** PollyZ has quit IRC | 14:29 | |
| fungi | but implementing the mechanics for managing that access (even if dockerhub has a solution for it) could also be complexity we wouldn't want to intorduce into, e.g., zuul | 14:29 |
| sdake | we also push multiple containers at once to speed thiigns up | 14:30 |
| openstackgerrit | Merged openstack-infra/system-config: Add missing source.list for xenial https://review.openstack.org/331444 | 14:30 |
| sdake | doing this serially takes 4-6 hours | 14:30 |
| *** ddieterly is now known as ddieterly[away] | 14:30 | |
| rcarrillocruz | pabelanger: btw, https://review.openstack.org/#/c/320466/13/playbooks/launch_node.yml , that's the modified launch node by using the feature of generating keypairs on the role itself | 14:30 |
| *** claudiub has quit IRC | 14:30 | |
| *** PollyZ has joined #openstack-infra | 14:30 | |
| rcarrillocruz | it needs as dependencies merging of the role change ofc, plus the other utility roles , topic: cloud-launcher | 14:30 |
| *** sree has joined #openstack-infra | 14:30 | |
| *** ddieterly[away] is now known as ddieterly | 14:30 | |
| jeblair | fungi: eleminating the build/upload split means that the credentials need to be scoped for only the area that job should be uploading to. so in pypi, that means an account for each project, and, i'd guess, something similar with docker. | 14:31 |
| rcarrillocruz | in case you want to quickly test it, the launch playbook is easy to follow | 14:31 |
| *** ddieterly is now known as ddieterly[away] | 14:31 | |
| fungi | jeblair: right, that's what i was getting at | 14:31 |
| *** ddieterly[away] is now known as ddieterly | 14:31 | |
| jeblair | i also don't know if pypi has an api for something like that | 14:31 |
| fungi | jeblair: ideally they would be scoped to a specific file name so that the worker for the novaclient 4.1 release can't publish some other releases too while at it | 14:32 |
| sdake | i get consistency but does it really need to be done consistently for everything? :) | 14:32 |
| *** hosanai has joined #openstack-infra | 14:32 | |
| sdake | that seems like a whole lot of pain - pypi in particular | 14:32 |
| *** yolanda has joined #openstack-infra | 14:32 | |
| fungi | sdake: so i guess we should probably first try something close to what we do now (build on single-use node, compress artifact, have zuul collect and publish that to tarballs.o.o, then have a second job (probably on a worker dedicated to docker publication running the "docker daemon") retrieve that from tarballs.o.o and try to push it up to dockerhub | 14:33 |
| jeblair | sdake: as in, if we do this for docker, do we need to do it for pypi? probably not. just brainstorming, and pypi is very simple to think about. | 14:33 |
| mordred | oh. sdake is back | 14:33 |
| thiagop-afk | Hello infra folks, good morning. This patch was approved some days ago, but apprently didn't merged somehow: https://review.openstack.org/#/c/301877/ | 14:33 |
| sdake | mordred ya it was 5am went bck to bed ;) | 14:33 |
| mordred | jeblair, fungi, sdake: sorry I missed the conversation - give me a sec to read the scrollback, I have many thoughts on this subject and had hoped to be pinged when it started up again :) | 14:33 |
| thiagop-afk | is there something that I can do to get it to the merge queue again? | 14:33 |
| fungi | sdake: the reason we don't really consider it for pypi is that the artifacts published there are a few megabytes in size | 14:33 |
| sdake | fungi understood | 14:34 |
| fungi | so having them traverse the network a few times isn't particularly onerous | 14:34 |
| jeblair | the docker registery does have an api | 14:34 |
| jeblair | https://docs.docker.com/registry/spec/api/ | 14:34 |
| dstufft | PyPI does not really have an aPI for munging with roles or the like | 14:34 |
| mordred | right. so - docker images are like git repos - each image is comprised of layers of things | 14:34 |
| *** sree__ has joined #openstack-infra | 14:34 | |
| SamYaple | for pushing to docker hub we can do this with bearer tokens and seperate repos | 14:34 |
| vgridnev | seems like several jobs again are failing with POST_FAILURE, is everything ok? | 14:34 |
| *** rhochmuth has joined #openstack-infra | 14:35 | |
| mordred | if we just do a tarball export, publish to tarballs.o.o then an import then a push | 14:35 |
| *** sree has quit IRC | 14:35 | |
| mordred | we'll be squashing the layers and making the images very bad | 14:35 |
| dstufft | If you want something like capability urls you'll need to open an issue on warehouse | 14:35 |
| sdake | jeblair all that api is tied into docker's implementation - its not super reusable as a result | 14:35 |
| SamYaple | fungi: job would request a token from a service we are running which fetchs it with the store (non-public) credentials and use that token to talk to dockerhub. it can have experation and all | 14:35 |
| mordred | once the image has been made on the untrusted host, we need to push it using docker off of the untrusted host | 14:35 |
| *** jamesmcarthur has joined #openstack-infra | 14:36 | |
| mordred | *somehow* - let's come back to that | 14:36 |
| jeblair | dstufft: ok, thanks | 14:36 |
| *** amotoki has quit IRC | 14:36 | |
| mordred | and then to publish to docker hub itself, we will either have to write a LOT of code, or use docker itself to pull from the untrusted location and push to docker hub | 14:36 |
| *** gongysh has joined #openstack-infra | 14:36 | |
| jeblair | mordred: can you docker pull from another docker daemon? | 14:37 |
| mordred | this will be much easier with zuul v3 - and it's possible the answer is going to nede to be "we just need to wait for v3" | 14:37 |
| mordred | jeblair: you can docker pull from a docker registry | 14:37 |
| mordred | which itself can run in docker | 14:37 |
| *** cody-somerville has joined #openstack-infra | 14:37 | |
| *** sree has joined #openstack-infra | 14:37 | |
| mordred | jeblair: so we could use the docker on the untrusted host to runa registry, then have a docker on the zuul launcher node pull from that registry and push to a different registry | 14:38 |
| mordred | similar to how we pull then publish tarballs | 14:38 |
| jeblair | mordred: right, as long as the untrusted host lives -- but today it doesn't | 14:38 |
| fungi | but our traditional publication model applied to docker images (say the compressed artifact is 2gb?) means that it will traverse the network once when zuul-launcher pulls that 2gb from the single-use node where it was built, finds temporary storage space for 2gb, then pushes that 2gb to tarballs.o.o (which will be accumulating these pretty rapidly, it's only a 200gb partition right now and is 1/3 | 14:38 |
| sdake | ya there are two ways to transfer the data, either transfer the loopback device containing the filesystem or use docker pull then docker push | 14:38 |
| fungi | full so would probably need expanding). then the second job would pull that 2gb over the network from tarballs.o.o to the dedicated docker upload worker, make sure to have 2gb worth of space to temporarily store it, and attempt to push the 2gb over the network again to dockerhub. so 3 non-disposable systems which need to store that 2gb (one probably indefinitely, the other two temporarily), and 4 | 14:38 |
| fungi | times that 2gb traverses the network | 14:38 |
| jeblair | mordred: but a long-lived docker registry server that acts like tarballs.o.o could fit the bill | 14:38 |
| mordred | jeblair: right. I'm suggesting both | 14:38 |
| *** sree__ has quit IRC | 14:39 | |
| sdake | jeblair thats a winning solution to me :) | 14:39 |
| rcarrillocruz | :D | 14:39 |
| mordred | jeblair: what I'm suggesting is that we have a publisher that reaches into the node at the end of the job, runs a docker registry on the host and docker pushes the image into the local registry, then have the publisher pull images from that per-host registry and push to the infra-registry | 14:39 |
| sdake | docker.openstack.org ;-) | 14:39 |
| mordred | jeblair: then we can have another job on a trusted host with credentials pull from the infra registry and push to the public dockerhum | 14:39 |
| mordred | I dont think it would be terribly hard to do nor insecure - but it would involve adding a docker to the zuul launchers so that we have a place to pull into and push from | 14:40 |
| sdake | mordred we could put a docker registry in our gate jobs and push to them - thats a 2 liner or a few more | 14:40 |
| jeblair | mordred: why does it have to run on the zuul launcher? why can't the untrusted build job just push to docker.o.o? | 14:41 |
| sdake | the issue is the workers go byebye | 14:41 |
| mordred | sdake: I think it would be better for the publisher macro to do that | 14:41 |
| jeblair | mordred: within the job itself | 14:41 |
| mordred | jeblair: it would need credentials | 14:41 |
| lucasagomes | hi all, can I have some input here https://review.openstack.org/#/c/332766/ ? I thought the "tox-func" jobs followed a template gate-{project-name}-tox-func, but the layout check is failing saying the job is not defined | 14:41 |
| mordred | jeblair: to push to docker.o.o | 14:41 |
| lucasagomes | where should I define it? | 14:41 |
| jeblair | mordred: right -- but credentials for our untrusted dockerhub | 14:41 |
| lucasagomes | it seems to work for ironic-inspector and swift | 14:41 |
| mordred | right | 14:41 |
| sdake | mordred I wouldn't know where the best place in infra is for the work | 14:41 |
| *** sree__ has joined #openstack-infra | 14:41 | |
| fungi | we would need to assure ourselves that the risk the single-use slave running the docker build job won't leak the upload credentials for docker.openstack.org, right? | 14:41 |
| *** sree has quit IRC | 14:41 | |
| mordred | jeblair: if we are comfortable with there being credentials for docker.o.o on single-use slaves, then we can totally do that | 14:42 |
| sdake | any single use slave can leak credentials via git review | 14:42 |
| jeblair | mordred: i'm not yet -- i'm wondering how constrained we can make those credentials | 14:42 |
| fungi | sdake: huh? can you elaborate on that? | 14:42 |
| mordred | if we're not, then having a z-l intermediary hub for pull from/ push to is a thing we can do | 14:42 |
| sdake | i could easily create a git review to print out the filesystem | 14:42 |
| sdake | the filesystem would contain the credentials i assuem | 14:42 |
| mordred | sdake: yah- we know - that's the thing we're talking about :) | 14:42 |
| fungi | sdake: oh, you mean if we put the docker.o.o creds on it, yes i agree that would be a risk | 14:43 |
| vgridnev | seems like several jobs again are failing with POST_FAILURE, is everything ok? sorry for repeating | 14:43 |
| mordred | jeblair: you're thinking perhaps having the z-l talk to docker.o.o and make a single-use credential? | 14:43 |
| jeblair | mordred: no, i'm wondering if there's a way to constrain the docker.o.o credentials in any way; or are we in the same boat as dockerhub itself there? | 14:43 |
| sdake | the credentials are in the orm of ca certs | 14:44 |
| sdake | so they coudl be rotated | 14:44 |
| *** ramishra has joined #openstack-infra | 14:44 | |
| sdake | jeblair worse off in fact - because docker's hub is more advanced then their open source version | 14:44 |
| jeblair | can we have per-project ca certs? | 14:44 |
| mordred | yes | 14:44 |
| sdake | jeblair not certain | 14:45 |
| jeblair | so in the zuulv3 world where we can pass credentials to specific jobs, we could allow kolla jobs to push to the kolla area of docker.o.o ? | 14:45 |
| *** xarses has quit IRC | 14:45 | |
| mordred | jeblair: oh - I say that | 14:45 |
| *** sree has joined #openstack-infra | 14:45 | |
| mordred | jeblair: I need to learn more | 14:45 |
| roaet | hello! we have a new project set up and I am currently unclear as to why jenkins is -1 on a change that appears to be passing otherwise: https://review.openstack.org/#/c/331906/8 | 14:45 |
| sdake | the issue is I dont want OTHER people printing out our creds via a git review and uploading to docker.oo | 14:45 |
| *** sree__ has quit IRC | 14:46 | |
| jeblair | sdake: yes, i don't want that either | 14:46 |
| pabelanger | derekh: http://8.43.87.229/ | 14:46 |
| pabelanger | derekh: that is the AFS mirror | 14:46 |
| sdake | roaet http://logs.openstack.org/06/331906/8/check/gate-quark-python34/4fed52d/console.html | 14:46 |
| jeblair | sdake: i'm still very much in a brainstorming place where i'm trying to see both what we could do now, and what we might be able to do later when we have zuulv3 available | 14:46 |
| sdake | click "toggle ci" | 14:46 |
| sdake | jeblair cool thanks :) | 14:47 |
| *** jaosorior has quit IRC | 14:47 | |
| *** david-lyle_ has quit IRC | 14:47 | |
| sdake | jeblair here is the deal - I think this problem is too hard to solve, but if you can solve it our community would love yu for it :) | 14:47 |
| roaet | sdake: but it's non-voting right? | 14:47 |
| *** zeih has quit IRC | 14:47 | |
| sdake | although we already love infra | 14:47 |
| *** jaosorior has joined #openstack-infra | 14:47 | |
| roaet | OH | 14:47 |
| roaet | sdake: toggle CI is waaaaaaaaay at the bottom | 14:47 |
| mordred | sdake: I don't think it's too hard to solve | 14:47 |
| mordred | sdake: I think we can TOTALLY do it | 14:47 |
| roaet | sdake: thanks you much. I appreciate it | 14:47 |
| jeblair | it is hard. we've done harder. :) | 14:48 |
| sdake | roet ya not super intuitive huh :) and no python34 is voting | 14:48 |
| mordred | we might argue for a few weeks about the best way to do it | 14:48 |
| clarkb | if we are already talking about a dedicated slave running docker daemon, just run all the bits there | 14:48 |
| fungi | clarkb: including the image build? | 14:48 |
| clarkb | especially since this happens periodically and not for every change you can be fairly confident in the stuff running | 14:48 |
| clarkb | fungi ya | 14:48 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: fix release notes script for new reno https://review.openstack.org/332852 | 14:48 |
| roaet | sdake: I think we set up python34 to be non voting on our project | 14:48 |
| clarkb | none of our other secrets are exposed | 14:48 |
| sdake | roaet it is not - or it would say (non-voting) next to it | 14:49 |
| fungi | clarkb: i guess the question is how much we trust the docker build process not to be perverted into a mechanism to leak the upload credentials | 14:49 |
| *** hashar has quit IRC | 14:49 | |
| roaet | sdake: gate-quark-python34 FAILURE in 3m 05s (non-voting) | 14:49 |
| mordred | clarkb: I do not think we shoudl do the image build there | 14:49 |
| jeblair | clarkb: any project that claims to use docker would have access to all of the docker creds | 14:49 |
| mordred | the image build is part of the ci job | 14:49 |
| mordred | HOWEVER | 14:49 |
| clarkb | jeblair true | 14:49 |
| *** sree has quit IRC | 14:49 | |
| mordred | rather than adding a docker daemon to z-l to be able to pull from the registry on the single-use node | 14:49 |
| *** hdaniel has quit IRC | 14:50 | |
| *** hashar has joined #openstack-infra | 14:50 | |
| mordred | we could potentially have z-l use ansible to run the pull directly from the docker registry host | 14:50 |
| mordred | so that we have one less docker daemon push/pull operation | 14:50 |
| *** tonytan4ever has quit IRC | 14:50 | |
| jeblair | mordred: now you're getting somewhere | 14:50 |
| derekh | pabelanger: ack, thanks | 14:50 |
| sdake | what is z-i | 14:50 |
| mordred | sdake: zuul-launcher | 14:50 |
| fungi | mordred: yeah, would cut down two network transits and stop us needing places to stash the files on teh zuul-launcher and atrballs site | 14:51 |
| jeblair | sdake: you need a new font! :) (it's Z-L) | 14:51 |
| sdake | jeblair need new eyeballs actually :) | 14:51 |
| mordred | jeblair: if docker.o.o host is asked to do a pull from the single-use-slave, and the pushes to the registry it runs | 14:51 |
| mordred | I think we're in good shape | 14:51 |
| openstackgerrit | Jiri Tyr proposed openstack-infra/jenkins-job-builder: Adding property-file into the pipeline publisher https://review.openstack.org/311150 | 14:51 |
| pabelanger | derekh: -1 on the code review, only because AFS mirror is online now | 14:52 |
| jeblair | mordred: should be totally doable in zuulv3 :) | 14:52 |
| sdake | how do we set which images should be loaded into the docker registry on the push? | 14:52 |
| SamYaple | mordred: at some point is there a push to hub.docker.com? | 14:52 |
| mordred | yup | 14:52 |
| mordred | SamYaple: totally. that then becomes quite easy | 14:52 |
| fungi | mordred: in that case why have that machine run a docker registry, why not just have it push the image it retrieved to dockerhub? | 14:52 |
| derekh | pabelanger: ok, updating | 14:52 |
| mordred | fungi: well, having locally published artifacts is also a good thing | 14:52 |
| *** knikolla has left #openstack-infra | 14:52 | |
| mordred | fungi: like, I wouldn't want dockerhub to be the only source that owns the artifacts we built | 14:52 |
| *** eharney has joined #openstack-infra | 14:52 | |
| fungi | mordred: true, especially if the push to dockerhub is as failure-prone as sdake suggests | 14:53 |
| mordred | also, other jobs running in infra might want to use iamges built | 14:53 |
| *** hashar has quit IRC | 14:53 | |
| mordred | so if we publish to a local dockerhub, we get those benefits | 14:53 |
| sdake | fungi its actually the build that is error prone | 14:53 |
| SamYaple | mordred: why not directly? could easily have a collaborater on a repo called kolla/nova-dev, collaboraters can be added and removed | 14:53 |
| *** tonytan4ever has joined #openstack-infra | 14:53 | |
| mordred | and we can then push to docker hub similar to github replicatin | 14:53 |
| sdake | fungi although push is a little flakey occasionally | 14:53 |
| mordred | SamYaple: because of all of the reasons I just said | 14:53 |
| mordred | SamYaple: we do not want to _Depend_ on dockerhub | 14:53 |
| *** vhosakot has joined #openstack-infra | 14:54 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: do not show empty change lists https://review.openstack.org/332856 | 14:54 |
| *** zul has quit IRC | 14:54 | |
| mordred | replicating things to an external service such as docker hub si a great idea | 14:54 |
| sdake | mordred fwiw dockerhub depends on aws storage :) | 14:54 |
| mordred | but we should also have our own copies of the things | 14:54 |
| EmilienM | pabelanger: https://review.openstack.org/#/c/332820/ | 14:54 |
| mordred | sdake: another good reason to not have it be the only source of our docker artifacts | 14:54 |
| fungi | in the same way that tarballs.openstack.org stores and publishes everything that we also publish on pypi.python.org | 14:54 |
| sdake | fungi makes sense | 14:54 |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Add a temporary tripleo cloud "rh2" https://review.openstack.org/330514 | 14:54 |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Add a experimental tripleo job on rh2 https://review.openstack.org/330515 | 14:54 |
| EmilienM | pabelanger: sounds like chown/chmod is not enough | 14:54 |
| mordred | sdake: local docker registries do not depend on aws | 14:54 |
| sdake | keep a local record | 14:54 |
| sdake | and replicate it | 14:54 |
| derekh | pabelanger: updated patch | 14:55 |
| mordred | ++ | 14:55 |
| sdake | mordred right - i meant dockerhub | 14:55 |
| SamYaple | mordred: fair enough but there needs to be only one push. you can add a one-time user per build job and push from the node to the docker.openstack.org registery directly | 14:55 |
| sdake | although local registries can depend on aws storage (by setup) | 14:55 |
| rdx_matt | I get a failure in gate-cinder-releasenotes: " AttributeError: 'dict' object has no attribute 'splitlines'" in reno's formatter.py. Does anyone got a clue? | 14:55 |
| mordred | SamYaple: no, we're not going t push from the build node to anywhere | 14:55 |
| *** zul has joined #openstack-infra | 14:55 | |
| mordred | SamYaple: because of credential leakage | 14:55 |
| cinerama | hi folks, we're using the conference system for the midcycle and the pstn line is giving a busy signal | 14:55 |
| dhellmann | rdx_matt : have a link to the patch? | 14:55 |
| mordred | SamYaple: we have a 2-stage process for all artifact publication - doing one for docker will not be hard | 14:55 |
| SamYaple | mordred: right thats not a problem though if you generate the user creds per build job and remove that the end | 14:55 |
| rdx_matt | dhellmann: https://review.openstack.org/#/c/332032/ | 14:55 |
| jeblair | cinerama, clarkb, fungi: do i need to pay the phone bill again? | 14:55 |
| mordred | SamYaple: we can do it more easily than that | 14:56 |
| fungi | jeblair: oh, i guess you didn't see when i asked you about that last night | 14:56 |
| cinerama | jeblair, i'm guessing so... | 14:56 |
| *** kushal has joined #openstack-infra | 14:56 | |
| dhellmann | rdx_matt : my guess is the : characters on line 7 and 8 are making the yaml parser pass a dict to reno instead of a string | 14:56 |
| jeblair | fungi: nope | 14:56 |
| dhellmann | rdx_matt : there's an example of how to deal with that in http://docs.openstack.org/developer/reno/usage.html#formatting | 14:57 |
| jeblair | Current Balance | 14:57 |
| sdake | so folks i'm not sure what the next step is | 14:57 |
| jeblair | $ -8.39 (Add Funds) | 14:57 |
| sdake | something on kolla end | 14:57 |
| sdake | something on infra end | 14:57 |
| sdake | someething on both? | 14:57 |
| mat128 | jeblair: ouch | 14:57 |
| jeblair | fungi: where do i send the bill? | 14:57 |
| *** pilgrimstack has joined #openstack-infra | 14:57 | |
| mordred | sdake: next step is one of us (I guess me) will write up a plan | 14:57 |
| *** yamahata has quit IRC | 14:57 | |
| mordred | sdake: I'll make sure you get copied on it | 14:58 |
| SamYaple | mordred: i am going to have to disagree with what you have said is "easier". generating a repo and credentials per build and then when the build is done tag it as the appropriate name of the image | 14:58 |
| rdx_matt | dhellmann: Thanks! | 14:58 |
| *** jistr is now known as jistr|mtg | 14:58 | |
| SamYaple | mordred: no cred leakage, no priv issues | 14:58 |
| *** tonytan4ever has quit IRC | 14:58 | |
| SamYaple | mordred: one push from the build node to openstack registry | 14:58 |
| sdake | that could be hacked | 14:58 |
| *** yamahata has joined #openstack-infra | 14:58 | |
| pabelanger | EmilienM: http://logs.openstack.org/20/332820/1/check/gate-puppet-openstack-integration-3-scenario001-tempest-ubuntu-xenial/601ffc0/console.html#_2016-06-22_14_23_18_789681 | 14:58 |
| jeblair | fungi: btw, the "balance alert" emails go to jeblair@openstack.org, which no longer exists | 14:58 |
| sdake | someone could create a git review, get the one time credential, upload a bunch of bad images to docker.oo | 14:58 |
| SamYaple | sure. to mess with a single repo that is empty sdake | 14:58 |
| fungi | jeblair: i thought clarkb was working out something with scott raschke (finance at the foundation) to set up some sort of directly bulled voip.ms account instead but i don't know how far he got | 14:58 |
| pabelanger | EmilienM: that is why it is failing | 14:58 |
| mordred | SamYaple: we'd have to orchestrate the creation and deletion of the credentials | 14:58 |
| SamYaple | i said generate per build | 14:58 |
| clarkb | fungi: I never got a central billing typw answer | 14:59 |
| clarkb | I can bug again if we want | 14:59 |
| SamYaple | mordred: there is an api for that. also it supports a whole lotta backends | 14:59 |
| *** ramishra has quit IRC | 14:59 | |
| jeblair | fungi: i would be happy to pay now, and send a copy of the invoice to the foundation and they can cut me a check. how does that sound? and how much? (i'd recommend at least $100 if we have a midcycle going on) | 14:59 |
| *** ramishra has joined #openstack-infra | 14:59 | |
| SamYaple | mordred: actually, it could be open for all that matters, its a temporary repo that is goign to get destroyed at the end of the job. no passwords at all | 15:00 |
| EmilienM | pabelanger: ok, /me continue to look at it | 15:00 |
| pabelanger | clarkb: jeblair: I was able to deploy our AFS mirrors correctly using tripleo-test-cloud-rh2: http://mirror.regionone.tripleo-test-cloud-rh2.openstack.org/ | 15:00 |
| mordred | SamYaple: I think we're still talking past each other | 15:00 |
| mordred | SamYaple: so I'm going to need to ask you to be a bit more annoying and pedantic with words | 15:00 |
| nibalizer | good morning | 15:00 |
| mordred | when you say "create a repo" - who creates it. what type of repo is it. where is it | 15:00 |
| fungi | jeblair: that seems fine to me (though i don't handle finance there). i'll see if they can tell me where to have you send that | 15:00 |
| *** abregman_ has joined #openstack-infra | 15:00 | |
| cinerama | jeblair, fungi: this is the last day of our midcycle but it'd be nice to have as a few folks are relying on pstn | 15:00 |
| *** rbrndt has quit IRC | 15:01 | |
| cinerama | jeblair, fungi: we're also meant to be starting for the day now :) | 15:01 |
| SamYaple | mordred: if the image is 'kolla/nova:latest' the repo is 'kolla/nova' | 15:01 |
| jeblair | cinerama: it should be restored in a few mins | 15:01 |
| cinerama | jeblair, fungi: oh and many thanks for attending to this | 15:01 |
| sdake | jeblair i hate out of money errors :) | 15:01 |
| *** ramishra has quit IRC | 15:01 | |
| SamYaple | mordred: build job starts, generate 'kolla/build-job-123' repo and allow world write to it. inform build node of 'kolla/build-job-123' name and thats it | 15:02 |
| *** ramishra has joined #openstack-infra | 15:02 | |
| roaet | does gate-*-requirements fail on the first one, or will it show me all the bad requirements? | 15:02 |
| *** jaosorior has quit IRC | 15:02 | |
| mordred | SamYaple: generate kolla/build-job-123 where? | 15:02 |
| SamYaple | mordred: docker registry/distribution at docker.openstack.org | 15:02 |
| *** ramishra has quit IRC | 15:02 | |
| *** sorantis has quit IRC | 15:02 | |
| *** weshay has quit IRC | 15:03 | |
| *** ramishra has joined #openstack-infra | 15:03 | |
| mordred | SamYaple: ok. so if kolla then pushes to kolla/build-job-123 on docker.o.o, how does that content persist once the job is done and we want to delete that repo | 15:03 |
| *** rbrndt has joined #openstack-infra | 15:03 | |
| *** pilgrimstack has quit IRC | 15:03 | |
| sdake | atm we push in parallel to speed things up | 15:03 |
| sdake | actually we may push serially - not certain | 15:03 |
| SamYaple | mordred: you can instant CoW clone it into 'kolla/nova:lastest' | 15:03 |
| mordred | SamYaple: so the thing that creates and deletes repos would also know to clone it to kolla/nova:latest before it deletes the repo, yeah? | 15:04 |
| *** weshay has joined #openstack-infra | 15:04 | |
| SamYaple | mordred: correct | 15:04 |
| jeblair | cinerama, fungi: paid; it's working now | 15:04 |
| *** eezhova has quit IRC | 15:04 | |
| mordred | SamYaple: awesome. thank you - I understand your words now :) | 15:04 |
| mordred | SamYaple: I will write up both options | 15:04 |
| mordred | SamYaple: so that we can look at both long-form | 15:05 |
| SamYaple | mordred: awesome. i have implemented the second option (the one i just suggested) here at my current job fyi | 15:05 |
| mordred | cool. good to know! | 15:05 |
| *** ArxCruz has quit IRC | 15:05 | |
| *** mdrabe has quit IRC | 15:05 | |
| fungi | jeblair: thanks, i've asked where to have you send that receipt | 15:06 |
| *** sree_ has joined #openstack-infra | 15:06 | |
| nibalizer | for gemfiles, we're down to just these 3 https://review.openstack.org/#/q/topic:gemfile-sync+status:open | 15:06 |
| *** mdrabe has joined #openstack-infra | 15:06 | |
| fungi | jeblair: i will hopefully let you know shortly | 15:06 |
| SamYaple | mordred: i do want to point out hub.docker.com runs very tweaked code from https://github.com/docker/distribution | 15:06 |
| *** sree_ is now known as Guest66994 | 15:06 | |
| nibalizer | we don't actually have acceptance jobs for refstack, so i have a project-config change up to remove that | 15:06 |
| SamYaple | mordred: setting up a repo we host at docker.openstack.org can be hard to get right from performance perspective | 15:06 |
| nibalizer | the other two look like legitimate test failures and will need Real Investigation (TM) | 15:06 |
| cinerama | jeblair: thank you so much for getting that sorted for us | 15:06 |
| *** nmagnezi has quit IRC | 15:07 | |
| jeblair | sdake, SamYaple, mordred: regarding timing -- some or both of those options may be a better fit for zuulv3, which means we may not get started on implementing them right away (though we are getting started on zuulv3 right away) | 15:07 |
| mordred | jeblair: agree | 15:07 |
| fungi | SamYaple: from an upload performance standpoint? or you mean publication on the assumption that lots of people will pull from there instead of dockerhub? or assuming that we'll pull from there to run ci jobs? | 15:07 |
| jeblair | (er, one or both, which i guess is the same as 'some') | 15:07 |
| *** pilgrimstack has joined #openstack-infra | 15:07 | |
| mordred | jeblair: :) | 15:08 |
| jeblair | cinerama: you're welcome! enjoy conferencing! | 15:08 |
| *** eantyshev1 is now known as eantyshev | 15:08 | |
| cinerama | (now if only the glean release fairy would come and grant the rest of my wishes... :) | 15:08 |
| *** permalac has quit IRC | 15:08 | |
| *** burgerk has joined #openstack-infra | 15:08 | |
| SamYaple | fungi: all of the above. but also things like old layers (storage space being consumed) may stick around. forever. with the only option to rekick the registry (or do non-documented stuff). | 15:09 |
| *** rcernin has quit IRC | 15:09 | |
| SamYaple | fungi: so there is a maintenance overhead as well | 15:09 |
| *** Guest66994 has quit IRC | 15:10 | |
| mordred | fungi, jeblair: fwiw, docker registry does have a swift storage backend | 15:10 |
| *** cody-somerville has quit IRC | 15:10 | |
| *** yolanda has quit IRC | 15:10 | |
| *** sdake has quit IRC | 15:10 | |
| *** coolsvap has quit IRC | 15:10 | |
| *** bapalm has quit IRC | 15:10 | |
| SamYaple | fungi: in not trying to FUD this option, im just saying its not super straight forward all the time and may be a huge PITA | 15:10 |
| *** robcresswell_ has joined #openstack-infra | 15:10 | |
| *** bapalm has joined #openstack-infra | 15:10 | |
| *** yarkot1 has quit IRC | 15:10 | |
| *** csatari has quit IRC | 15:10 | |
| *** hogepodge has quit IRC | 15:10 | |
| *** andymccr_ has quit IRC | 15:10 | |
| *** robcresswell has quit IRC | 15:10 | |
| *** andymccr has joined #openstack-infra | 15:10 | |
| *** yolanda has joined #openstack-infra | 15:10 | |
| *** robcresswell_ is now known as robcresswell | 15:10 | |
| *** tesseract- has quit IRC | 15:10 | |
| *** cody-somerville has joined #openstack-infra | 15:10 | |
| *** cody-somerville has joined #openstack-infra | 15:10 | |
| *** coolsvap has joined #openstack-infra | 15:10 | |
| mordred | SamYaple: that's fine | 15:10 |
| *** csatari has joined #openstack-infra | 15:10 | |
| *** yarkot1 has joined #openstack-infra | 15:11 | |
| *** hogepodge has joined #openstack-infra | 15:11 | |
| mordred | SamYaple: we do tons of things that are a huge PITA | 15:11 |
| mordred | :) | 15:11 |
| *** sdake has joined #openstack-infra | 15:11 | |
| *** sdake_ has joined #openstack-infra | 15:11 | |
| mordred | the work will be worth it if it means that our content is not dependent on an external party for existence | 15:11 |
| SamYaple | heh. well just be prepared. running a registry is a huge pain. we abandonded it and just push to dockerhub and quay | 15:11 |
| mordred | SamYaple: yup. I imagine it's a PITA on purpose so that you'll make that choice :) | 15:12 |
| jeblair | SamYaple: quayke with fear? | 15:12 |
| roaet | is there a way to make zuul.openstack.org show only a particular queue? (other than the filter at the top)? | 15:12 |
| *** isaacb has quit IRC | 15:12 | |
| *** sree has joined #openstack-infra | 15:12 | |
| mordred | SamYaple: however, that's never stopped us before | 15:12 |
| openstackgerrit | Merged openstack-infra/storyboard: Make `/v1/worklists/:id/filters` resolve criteria https://review.openstack.org/332764 | 15:13 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Add gate-ironic-agent-tox-func to IPA gate https://review.openstack.org/332766 | 15:14 |
| coolsvap | sdake, sorry i was away from my desk read through most of the scroll back | 15:14 |
| openstackgerrit | Vitaly Kramskikh proposed openstack-infra/irc-meetings: Add vkramskikh to chair of javascript-meeting https://review.openstack.org/332863 | 15:14 |
| *** sarob has joined #openstack-infra | 15:14 | |
| *** sdake has quit IRC | 15:15 | |
| *** sarob has quit IRC | 15:15 | |
| *** sarob has joined #openstack-infra | 15:16 | |
| *** sree has quit IRC | 15:17 | |
| *** _nadya_ has quit IRC | 15:17 | |
| *** itisha has joined #openstack-infra | 15:17 | |
| *** sdake has joined #openstack-infra | 15:18 | |
| *** sdake_ has quit IRC | 15:19 | |
| coolsvap | mordred, sorry i was a bit away, i think having docker.o.o is a great idea | 15:19 |
| coolsvap | the solution i was looking for since it dynamically reduces the docker push time compared to dockerhub | 15:20 |
| *** sarob has quit IRC | 15:20 | |
| *** sarob has joined #openstack-infra | 15:21 | |
| *** pcaruana has quit IRC | 15:22 | |
| *** gongysh has quit IRC | 15:22 | |
| openstackgerrit | Bernard Cafarelli proposed openstack/diskimage-builder: Run RHEL system unregister element earlier https://review.openstack.org/332866 | 15:22 |
| *** xarses has joined #openstack-infra | 15:22 | |
| sdake | mordred coolsvap wants to take this project on from the kolla side | 15:24 |
| sdake | mordred if you need anything from me let me know, but coolsvap should likely be our primary go to guy :) | 15:24 |
| *** rcernin has joined #openstack-infra | 15:24 | |
| sdake | mordred its not because i dont care, its ecause i'm swamped | 15:25 |
| nibalizer | having run a docker registry in the past.. its pretty difficult to debug when it isn't happy | 15:25 |
| *** gouthamr has quit IRC | 15:25 | |
| nibalizer | it also wants to run /in/ docker exclusively (as of last year) and that's not really how we roll | 15:25 |
| *** psachin has quit IRC | 15:26 | |
| *** gouthamr has joined #openstack-infra | 15:27 | |
| *** sorantis has joined #openstack-infra | 15:27 | |
| *** rossella__ has joined #openstack-infra | 15:28 | |
| *** pt_15 has joined #openstack-infra | 15:28 | |
| *** andreas_s has quit IRC | 15:29 | |
| *** ociuhandu has quit IRC | 15:29 | |
| mordred | nibalizer: it's not - but if that's how it runs then that's how it runs | 15:29 |
| *** Na3iL has quit IRC | 15:30 | |
| mordred | nibalizer: but I agree, there is a decent amount of learning we're going to need to do in order to think about this in infra terms | 15:30 |
| *** rossella_ has joined #openstack-infra | 15:31 | |
| *** mib_0ye has joined #openstack-infra | 15:31 | |
| *** jistr|mtg is now known as jistr | 15:32 | |
| *** roxanaghe has joined #openstack-infra | 15:32 | |
| *** sree has joined #openstack-infra | 15:32 | |
| clarkb | it seems funny that a webserver that proxies storage to s3/swift via a rest api cares where it runa | 15:33 |
| *** droyo has quit IRC | 15:33 | |
| *** pilgrimstack has quit IRC | 15:33 | |
| nibalizer | jesusaur: undockerized the docker hub server we had at hp, he knows more | 15:34 |
| *** rossella__ has quit IRC | 15:34 | |
| *** mib_0ye has quit IRC | 15:35 | |
| jesusaur | nibalizer: no, i never managed to undockerize it, but i did manage to turn it off and never think about it again... | 15:35 |
| *** amitgandhinz has quit IRC | 15:35 | |
| *** abregman_ has quit IRC | 15:35 | |
| nibalizer | jesusaur: oh thats the best | 15:35 |
| *** amitgandhinz has joined #openstack-infra | 15:36 | |
| *** roxanaghe has quit IRC | 15:36 | |
| *** pcaruana has joined #openstack-infra | 15:36 | |
| jesusaur | clarkb: the wanting to run in docker aspect has to do with configuration and lack of docs | 15:37 |
| *** sree has quit IRC | 15:37 | |
| *** zhurong has quit IRC | 15:37 | |
| *** csomerville has joined #openstack-infra | 15:37 | |
| *** esikachev has quit IRC | 15:38 | |
| jesusaur | clarkb: as of a year ago, the only documentation I could find on configuring the registry described configuration in terms of docker run parameters, and i had a lot of difficulty configuring a cloud instance to do the same things | 15:38 |
| jesusaur | there are likely magic bits baked into their docker image that they don't document | 15:38 |
| clarkb | gotcha | 15:40 |
| jesusaur | nibalizer: what's the purpose of infra_spec_helper? | 15:40 |
| *** savihou has quit IRC | 15:41 | |
| *** tonytan4ever has joined #openstack-infra | 15:41 | |
| *** cody-somerville has quit IRC | 15:41 | |
| *** eezhova has joined #openstack-infra | 15:42 | |
| *** thorst_ has quit IRC | 15:42 | |
| *** sree has joined #openstack-infra | 15:42 | |
| *** sarob has quit IRC | 15:43 | |
| *** sorantis has quit IRC | 15:43 | |
| *** thorst_ has joined #openstack-infra | 15:43 | |
| *** pilgrimstack has joined #openstack-infra | 15:44 | |
| *** vincentll has quit IRC | 15:44 | |
| *** sree has quit IRC | 15:44 | |
| *** stevemar has joined #openstack-infra | 15:44 | |
| *** sree has joined #openstack-infra | 15:44 | |
| *** thorst__ has joined #openstack-infra | 15:44 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/system-config: Add #openstack-pkg to statusbot & meetbot_channels https://review.openstack.org/332881 | 15:45 |
| *** bhavik has quit IRC | 15:47 | |
| *** thorst_ has quit IRC | 15:47 | |
| nibalizer | jesusaur: its a single gem that all out modules depebd on | 15:47 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: do not show empty change lists https://review.openstack.org/332856 | 15:47 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: fix release notes script for new reno https://review.openstack.org/332852 | 15:47 |
| nibalizer | then we put dependencies and pins on that gem | 15:47 |
| nibalizer | instead of 60+ patches to everything | 15:47 |
| *** Sukhdev has joined #openstack-infra | 15:47 | |
| jesusaur | ah, ok, that makes sense | 15:48 |
| *** thorst__ has quit IRC | 15:49 | |
| *** sree has quit IRC | 15:49 | |
| *** chlong has quit IRC | 15:50 | |
| *** harlowja_at_home has joined #openstack-infra | 15:50 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 15:52 |
| *** makowals has quit IRC | 15:53 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/puppet-log_processor: Switch to using the new log_processor project https://review.openstack.org/254992 | 15:54 |
| *** roxanaghe has joined #openstack-infra | 15:54 | |
| *** esikachev has joined #openstack-infra | 15:54 | |
| fungi | nibalizer: nice, we're down to just 3 patches for that not passing yet | 15:55 |
| nibalizer | yep | 15:56 |
| nibalizer | fungi: one (refstack) is fixed by a prohect config patch i have up to remove the test though | 15:56 |
| fungi | different review topic? | 15:57 |
| *** javeriak has joined #openstack-infra | 15:58 | |
| *** armax has joined #openstack-infra | 15:58 | |
| nibalizer | ah yes, im still terrible at topics | 15:58 |
| nibalizer | linked in a comment on the refstack one though | 15:59 |
| *** mriedem has quit IRC | 15:59 | |
| fungi | i am just a terrible topic | 15:59 |
| *** daemontool has joined #openstack-infra | 16:00 | |
| *** banix has joined #openstack-infra | 16:00 | |
| njohnston | Hi! I was wondering if the issue related to hitting the maximum number of channels had been fixed, allowing accessbot to listen to more channels and letting https://review.openstack.org/#/c/329570/ to proceed. | 16:00 |
| *** mriedem has joined #openstack-infra | 16:01 | |
| *** rossella__ has joined #openstack-infra | 16:01 | |
| zigo | mordred: pabelanger: Maybe it'd be nicer to discuss packaging-deb stuff within the dedicated channel. Could you join #openstack-pkg ? | 16:01 |
| *** fabo has quit IRC | 16:01 | |
| *** fabo has joined #openstack-infra | 16:01 | |
| *** dmk0202 has quit IRC | 16:02 | |
| *** mriedem1 has joined #openstack-infra | 16:03 | |
| *** rossella_ has quit IRC | 16:04 | |
| *** rhochmuth has quit IRC | 16:04 | |
| *** pots has joined #openstack-infra | 16:05 | |
| *** esikachev has quit IRC | 16:05 | |
| *** mriedem has quit IRC | 16:05 | |
| *** jamesmcarthur has quit IRC | 16:05 | |
| *** jlanoux has quit IRC | 16:06 | |
| *** nwkarsten has quit IRC | 16:07 | |
| *** rhochmuth has joined #openstack-infra | 16:07 | |
| *** raunak has joined #openstack-infra | 16:07 | |
| *** nwkarsten has joined #openstack-infra | 16:07 | |
| *** Rockyg has joined #openstack-infra | 16:08 | |
| *** roxanaghe has quit IRC | 16:08 | |
| *** ArxCruz has joined #openstack-infra | 16:09 | |
| *** xyang1 has quit IRC | 16:09 | |
| *** ggnel_t has quit IRC | 16:10 | |
| *** xyang1_ has joined #openstack-infra | 16:10 | |
| *** ggnel_t has joined #openstack-infra | 16:10 | |
| *** tphummel has joined #openstack-infra | 16:11 | |
| *** Nikolay_St has joined #openstack-infra | 16:11 | |
| *** roxanaghe has joined #openstack-infra | 16:11 | |
| nibalizer | jesusaur: others puppet-lint 2.0.0 released today, so thats cool | 16:11 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: DNM playing with tempest run https://review.openstack.org/327291 | 16:11 |
| *** nwkarsten has quit IRC | 16:11 | |
| jesusaur | nibalizer: yeah, I'm discovering that I'm not disabling the 140 char limit | 16:12 |
| *** harlowja_at_home has quit IRC | 16:12 | |
| nibalizer | how dare you | 16:12 |
| jesusaur | (which used to be an 80 char limit that I *am* disabling?) | 16:12 |
| *** mdrabe has quit IRC | 16:12 | |
| *** trown is now known as trown|lunch | 16:13 | |
| *** gyee has joined #openstack-infra | 16:13 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update Gerrit Trigger to use convert xml https://review.openstack.org/332893 | 16:13 |
| *** javeriak_ has joined #openstack-infra | 16:13 | |
| *** javeriak has quit IRC | 16:13 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update Gerrit Trigger to use convert xml https://review.openstack.org/332893 | 16:14 |
| *** ddieterly is now known as ddieterly[away] | 16:14 | |
| *** tonytan4ever has quit IRC | 16:15 | |
| *** eezhova has quit IRC | 16:16 | |
| *** javeriak_ has quit IRC | 16:16 | |
| *** links has joined #openstack-infra | 16:16 | |
| *** baoli__ has quit IRC | 16:16 | |
| *** baoli has joined #openstack-infra | 16:17 | |
| krotscheck | fungi: When you added me as core to js-openstack-lib, I didn't mean for that to live under the infra banner rather to be on-par with the python-openstacksdk. Did I put the ACL file in the wrong place? | 16:17 |
| *** jamesmcarthur has joined #openstack-infra | 16:18 | |
| *** bhavik has joined #openstack-infra | 16:18 | |
| *** eezhova has joined #openstack-infra | 16:18 | |
| AJaeger | njohnston: no, it's not fixed ;( | 16:20 |
| AJaeger | njohnston: we need a second bot setup. Do you want to help? | 16:21 |
| *** claudiub has joined #openstack-infra | 16:21 | |
| *** rossella_ has joined #openstack-infra | 16:21 | |
| *** ddieterly[away] is now known as ddieterly | 16:22 | |
| fungi | krotscheck: looks like it's only referenced in gerrit/acls/openstack/js-openstack-lib.config which is only used by openstack/js-openstack-lib _but_ https://review.openstack.org/314313 proposed it as an infra project, not an sdks project | 16:22 |
| *** ArxCruz has quit IRC | 16:22 | |
| fungi | krotscheck: (also i don't think the sdk team is part of the big tent yet?) | 16:22 |
| krotscheck | fungi: Oh, drat | 16:23 |
| * krotscheck sees it now. | 16:23 | |
| krotscheck | fungi: Well, it's as good a place as any. | 16:23 |
| AJaeger | thiagop-afk: are you still here? Your change was approved and then we had to restart zuul - see https://wiki.openstack.org/wiki/Infrastructure_Status | 16:23 |
| njohnston | AJaeger: Yes, I'd be happy to help! How can I assist? | 16:23 |
| AJaeger | thiagop-afk: you need to toggle the approval, remove it and add it again | 16:23 |
| krotscheck | fungi: Would you mind adding vkramskikh as a core then? Also, betherly's agreed to be core on js-generator-openstack (We have two more nominees, but they haven't accepted). | 16:24 |
| *** ArxCruz has joined #openstack-infra | 16:24 | |
| thiagop-afk | AJaeger: was about to get out for lunch, but ok. I can do that. | 16:24 |
| fungi | krotscheck: since it seemed like it was used by/related to at least a couple of your infra js repos, i didn't see any concern adding it to infra. also sure, i'll add them | 16:24 |
| AJaeger | njohnston: we need to run a second gerritbot - I'm not the right person to ask how that works | 16:24 |
| zaro | fungi: was wondering if you would be willing to a+ the git gc change? https://review.openstack.org/#/c/329566/ | 16:24 |
| zaro | already two +2s | 16:24 |
| krotscheck | fungi: Once we get to a place where it makes sense to live under our own umbrella, I'll ping you :) | 16:24 |
| fungi | krotscheck: no worries, it's a simple governance patch and some group configuration changes when that happens | 16:25 |
| *** rossella__ has quit IRC | 16:25 | |
| njohnston | AJaeger: Any nominations on who I could talk to in order to make sure I get started in the right spot? | 16:25 |
| thiagop-afk | AJaeger: worked, it's back on the gate. Thanks. | 16:26 |
| AJaeger | njohnston: just ask here in the channel. fungi, do you have pointers for setting up second gerritbot? | 16:26 |
| AJaeger | thiagop-afk: great! | 16:26 |
| fungi | betherly: what's your name in gerrit? | 16:26 |
| krotscheck | fungi: Much obliged :) | 16:27 |
| betherly | fungi: bethelwell thanks :) | 16:27 |
| krotscheck | fungi e.r.elwell | 16:27 |
| fungi | AJaeger: i might at some point, but i'm juggling far too many conversations/requests in half a dozen channels right now | 16:27 |
| krotscheck | Or that | 16:27 |
| krotscheck | ugh lag | 16:27 |
| betherly | lol! | 16:27 |
| clarkb | does it make sense to setup a notifications channel and have people filter for the projects they care about | 16:28 |
| *** vhosakot_ has joined #openstack-infra | 16:28 | |
| clarkb | as an alternative to running multiple bots? | 16:28 |
| fungi | clarkb: we could pick some new and obscure name for that channel... like #openstack-dev maybe? | 16:28 |
| clarkb | fungi: indeed | 16:28 |
| krotscheck | clarkb: We could make use of the zuul thing that mordred is doing and provide a way for people to build notification things off of that. | 16:29 |
| * krotscheck would love to get OSX notifications on things. | 16:29 | |
| clarkb | krotscheck: this is specifically for irc | 16:29 |
| greghaynes | there might be some rate limiting issues that come in to play if we did one big channel | 16:29 |
| greghaynes | depending on how timely people like their notices | 16:29 |
| njohnston | clarkb: I am not sure your suggestion would help me; what I really want is my channel to be logged | 16:29 |
| krotscheck | clarkb: A MAN CAN DREAM! :p | 16:30 |
| clarkb | njohnston: channel logging is indeoendent of the gerritbot | 16:30 |
| *** vhosakot has quit IRC | 16:30 | |
| clarkb | but maybe we have the same problem there | 16:30 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: DNM: test commit to see if puppet-lint 2.0 broke us https://review.openstack.org/332903 | 16:30 |
| clarkb | in which case a single notifications channel wont help | 16:30 |
| njohnston | Yes, my change - which seems to have precipitated this discussion in some way - is about accessbot: https://review.openstack.org/#/c/329570/ | 16:30 |
| *** Nikolay_St has quit IRC | 16:30 | |
| clarkb | ah ok, this came uo with gerritbot alerts earlier (last week) | 16:31 |
| AJaeger | njohnston: just to be logged? Thjen you don't need gerritbot | 16:31 |
| jesusaur | nibalizer: good news, unquoted_node_name check now supports multiple node names in a comma separated list in a single node block. | 16:31 |
| jesusaur | qua? | 16:31 |
| AJaeger | njohnston: let me review your chang eagain... | 16:31 |
| * njohnston is confused. I only ever wanted to get my channel logged. | 16:32 | |
| jesusaur | nibalizer: good news, https://review.openstack.org/254992 passed tests | 16:32 |
| jesusaur | thats the right copy buffer | 16:32 |
| AJaeger | njohnston: that change is fine as is and can be merged - whenever we have no meetings going on in parallel... | 16:32 |
| njohnston | AJaeger: OK, so my change is not blocked by the max-channels situation? | 16:32 |
| fungi | though we should start taking count of how many channels we're listing in the meetbot and statusbot configs | 16:33 |
| *** baoli has quit IRC | 16:33 | |
| fungi | presumably still fewer than gerritbot, for now at least | 16:33 |
| *** baoli has joined #openstack-infra | 16:33 | |
| pabelanger | Hmm, zl04.o.o is tearing it up! 162 jobs it is currently running | 16:34 |
| njohnston | thanks all | 16:34 |
| pabelanger | next closest is 63 | 16:34 |
| *** mdrabe has joined #openstack-infra | 16:34 | |
| mordred | pabelanger: we shoudl declare a daily winner | 16:34 |
| pabelanger | sorry 142 not 162 | 16:34 |
| clarkb | its probably not a bad idea to at least co sider co solidation. 120 silos is not good for communication | 16:35 |
| clarkb | wow no 'n' key this morning | 16:35 |
| pabelanger | mordred: place your bets now! | 16:35 |
| greghaynes | did anyone ask freenode staff if they could raise the limit for us? | 16:35 |
| mordred | greghaynes: yes | 16:35 |
| greghaynes | aw | 16:35 |
| mordred | greghaynes: it is not possible | 16:35 |
| greghaynes | oh well | 16:35 |
| *** jamesmcarthur has quit IRC | 16:36 | |
| *** e0ne has quit IRC | 16:37 | |
| *** yamahata has quit IRC | 16:37 | |
| *** sdague has quit IRC | 16:37 | |
| *** daemontool has quit IRC | 16:37 | |
| *** mriedem1 has quit IRC | 16:38 | |
| AJaeger | njohnston: correct. Sorry, if there was confusion | 16:38 |
| *** mionkin has quit IRC | 16:39 | |
| *** links has quit IRC | 16:40 | |
| *** rcernin has quit IRC | 16:40 | |
| openstackgerrit | Merged openstack-infra/system-config: run git gc instead of repack for git repos https://review.openstack.org/329566 | 16:41 |
| *** rossella__ has joined #openstack-infra | 16:41 | |
| krotscheck | fungi: both bmorel@internap.com and zhang.yujunz@zte.com.cn have also accepted their nominations. | 16:41 |
| krotscheck | (for js-generator-openstack) | 16:41 |
| *** ramishra has quit IRC | 16:41 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Add gate-ironic-agent-tox-func to IPA gate https://review.openstack.org/332766 | 16:42 |
| *** baoli_ has joined #openstack-infra | 16:42 | |
| *** baoli has quit IRC | 16:43 | |
| *** pcaruana has quit IRC | 16:43 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove requirements job from Rally https://review.openstack.org/328682 | 16:43 |
| openstackgerrit | Merged openstack-infra/project-config: Remove projects tox environment https://review.openstack.org/332649 | 16:43 |
| fungi | krotscheck: i've added bmorel@internap.com to infra-javascript-core now as well, but zhang.yujunz@zte.com.cn isn't an address known to gerrit | 16:43 |
| *** baoli has joined #openstack-infra | 16:44 | |
| *** pgadiya has joined #openstack-infra | 16:44 | |
| fungi | krotscheck: no, strike that, it's an address belonging to two accounts in gerrit | 16:44 |
| krotscheck | fungi: Ah, gotcha | 16:44 |
| *** pgadiya has quit IRC | 16:44 | |
| * krotscheck was confused for a sec. | 16:44 | |
| fungi | there's a "Zhang Yujunz" and a "Yujunz Zhang" in gerrit, both with that address | 16:45 |
| *** mescanef has quit IRC | 16:45 | |
| fungi | i'll see if i can figure out which one is active | 16:45 |
| *** rossella_ has quit IRC | 16:45 | |
| *** hosanai has quit IRC | 16:46 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix Monasca Analytics python27 job https://review.openstack.org/331535 | 16:46 |
| *** jamesmcarthur has joined #openstack-infra | 16:46 | |
| nibalizer | jesusaur: fantastic | 16:47 |
| *** baoli_ has quit IRC | 16:47 | |
| *** Nikolay_St has joined #openstack-infra | 16:47 | |
| AJaeger | fungi, clarkb, mordred, jeblair, project-config cores: could you look later at https://review.openstack.org/#/c/331535/ , please? This sets up a python job with custom package install - using devstack. I'm not sure this is the best way of doing it and would like to have some review help on it. | 16:47 |
| *** ramishra has joined #openstack-infra | 16:47 | |
| nibalizer | jesusaur: im interested to see if https://review.openstack.org/#/c/332903/ passes | 16:48 |
| jesusaur | nibalizer: they run the same integration tests, i would be surprised if the apply tests were vastly different on the two changes | 16:50 |
| clarkb | AJaeger: I think that will run tempest and stuff, probably huge overkill for what they need | 16:50 |
| clarkb | AJaeger: my suggestion would be to just add in a builder to install the deps then run the python27 macro | 16:51 |
| *** esikachev has joined #openstack-infra | 16:51 | |
| pabelanger | I posted this to our ML over the weekend, it is regarding our current usage of ::jenkins::cgroups and how they apply to zuul-launcher and ubuntu-xenial. If people have a few minutes to read: http://lists.openstack.org/pipermail/openstack-infra/2016-June/004449.html | 16:51 |
| AJaeger | clarkb: my impression as well ;(. Apparently that was done on advise given here a few days ago, they might have misunderstood it... | 16:51 |
| AJaeger | clarkb: can you comment on the review, please? | 16:51 |
| pabelanger | tl;dr: was able to launch ubuntu-xenial wheel over the weekend, but puppet fails to apply corretly (-6 error code) | 16:52 |
| clarkb | AJaeger: ya I am writing a response now | 16:52 |
| AJaeger | thanks, clarkb | 16:52 |
| *** ramishra has quit IRC | 16:53 | |
| *** PollyZ has quit IRC | 16:53 | |
| *** dizquierdo has joined #openstack-infra | 16:54 | |
| openstackgerrit | James Page proposed openstack-infra/project-config: Add openstack-charm-guide repository https://review.openstack.org/332916 | 16:54 |
| *** derekh has quit IRC | 16:55 | |
| pabelanger | jeblair: mordred: notmorgan: This looks new, [Zuul] standard output/error still open after child exited: http://logs.openstack.org/20/332820/3/check/gate-puppet-openstack-integration-3-scenario001-tempest-centos-7/ba5c921/console.html#_2016-06-22_16_28_32_205397 | 16:55 |
| *** shardy has quit IRC | 16:56 | |
| *** esikachev has quit IRC | 16:56 | |
| *** gfreeman has joined #openstack-infra | 16:56 | |
| AJaeger | pabelanger, jeblair: I updated today the files conditions for project-config (merged already) and think that everything is fine now. We *always* run the zuul layout check. A double check of current conditions would be nice in case I missed something. | 16:56 |
| pabelanger | AJaeger: okay | 16:57 |
| fungi | krotscheck: okay, i got rid of the duplicate account which wasn't set up to contribute code (no username, no ssh keys, no cla/contact submission) and reassociated its openid with the other account now | 16:58 |
| fungi | krotscheck: and added it to the infra-javascript-core group as well | 16:58 |
| krotscheck | fungi: Thanks for doing the cleanup. | 16:58 |
| * krotscheck reminds himself to buy fungi a donut | 16:58 | |
| *** PollyZ has joined #openstack-infra | 16:58 | |
| clarkb | gerrit seems unhappy /me checks melody | 16:58 |
| fungi | one of those with pink frosting and multi-colored sprinkles from lard lad | 16:59 |
| *** Apoorva has joined #openstack-infra | 16:59 | |
| *** jcoufal has joined #openstack-infra | 16:59 | |
| fungi | clarkb: that would be unfortunate if so, i restarted it only sunday i think? | 16:59 |
| *** javeriak has joined #openstack-infra | 16:59 | |
| *** Apoorva has quit IRC | 16:59 | |
| jesusaur | wasnt there an emergency restart yesterday? | 16:59 |
| fungi | clarkb: i did just flush-caches a few minutes ago after fixing that dupe account in the db, so that might be it dealing with a cold cache for a bit | 17:00 |
| gfreeman | hi all; my CI zuul seems to be stuck. jenkins has finished executing the job, but zuul doesn't think that it's done yet. is there a way to resolve this w/o restarting zuul and losing my job queue? i've just tried restarting jenkins, and that was a no-go. | 17:00 |
| *** rossella_ has joined #openstack-infra | 17:00 | |
| fungi | jesusaur: oh, yes, i also restarted it yesterday for non-performance reasons | 17:00 |
| clarkb | it doesn't seem to be GC related | 17:00 |
| clarkb | fungi: ya maybe cold cache is resulting in slow responses | 17:00 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-openstackci: Add other-requirements.txt for bindep support https://review.openstack.org/332921 | 17:00 |
| *** tphummel has quit IRC | 17:01 | |
| *** moravec has quit IRC | 17:02 | |
| *** PollyZ has quit IRC | 17:03 | |
| notmorgan | pabelanger: hmmm | 17:03 |
| *** ri0 has joined #openstack-infra | 17:03 | |
| zaro | clarkb: gerrit seems happy for me | 17:03 |
| clarkb | zaro: it is working, but I got at least one server is unavailable and posting comments was really slow | 17:04 |
| clarkb | AJaeger: comments posted | 17:04 |
| *** rossella__ has quit IRC | 17:04 | |
| jesusaur | fungi: clarkb: how often do you end up restarting gerrit for performance? | 17:04 |
| *** bhavik1 has joined #openstack-infra | 17:04 | |
| clarkb | it was about once a week or so before we bumped the server size, now its less frequent, once a month? every other month? | 17:05 |
| *** sree has joined #openstack-infra | 17:05 | |
| notmorgan | pabelanger: i;ve not seen that before | 17:06 |
| fungi | i would guess ~monthly | 17:06 |
| *** bhavik has quit IRC | 17:06 | |
| *** bhavik1 is now known as bhavik | 17:06 | |
| clarkb | we definitely reduced the frequency by throwing more ram at it | 17:06 |
| AJaeger | clarkb, thanks | 17:06 |
| EmilienM | pabelanger: quick update before /me afk most of afternoon -> I did https://review.openstack.org/#/c/332820 and it seems to work for us (exccept the random async thing). Please review it, so maybe we can land it this afternoon. Thanks | 17:07 |
| AJaeger | clarkb: Gerrit was restared yesterday and also last Sunday - and before on the 4th according to https://wiki.openstack.org/wiki/Infrastructure_Status | 17:07 |
| clarkb | AJaeger: ya, its definitely not memory pressure this time according to melody. Fungi's cache theory is probably correct | 17:08 |
| *** Jeffrey4l has quit IRC | 17:08 | |
| AJaeger | clarkb: let's see whether we hit further problems... | 17:09 |
| *** sdake_ has joined #openstack-infra | 17:09 | |
| *** javeriak_ has joined #openstack-infra | 17:09 | |
| *** shashank_hegde has joined #openstack-infra | 17:09 | |
| *** sree has quit IRC | 17:09 | |
| *** electrofelix has quit IRC | 17:10 | |
| *** sdake has quit IRC | 17:10 | |
| *** javeriak has quit IRC | 17:10 | |
| *** bhavik has quit IRC | 17:11 | |
| *** Apoorva has joined #openstack-infra | 17:11 | |
| *** bhavik has joined #openstack-infra | 17:11 | |
| *** watanabe_isao has joined #openstack-infra | 17:12 | |
| *** gongysh has joined #openstack-infra | 17:13 | |
| *** rossella__ has joined #openstack-infra | 17:13 | |
| *** watanabe_isao has quit IRC | 17:14 | |
| *** nwkarsten has joined #openstack-infra | 17:15 | |
| *** yamahata has joined #openstack-infra | 17:16 | |
| jeblair | pabelanger: yeah, the stdout/err thing is basically a warning -- it means that something in the job was still holding a stdout/stderr file descriptor open after it had (theoretically) completed. | 17:16 |
| jeblair | pabelanger: it won't affect zuul, but it means something is messy in the job itself. | 17:16 |
| *** tphummel has joined #openstack-infra | 17:16 | |
| *** rossella_ has quit IRC | 17:17 | |
| *** rossella_ has joined #openstack-infra | 17:17 | |
| *** Sukhdev has quit IRC | 17:17 | |
| jeblair | mordred, notmorgan: ^ fyi (closing the ping loop) | 17:19 |
| *** eezhova has quit IRC | 17:20 | |
| *** rcernin has joined #openstack-infra | 17:20 | |
| notmorgan | jeblair: see it thnx :) | 17:21 |
| *** rossella__ has quit IRC | 17:21 | |
| *** pilgrimstack has quit IRC | 17:21 | |
| *** eezhova has joined #openstack-infra | 17:21 | |
| *** nwkarsten has quit IRC | 17:23 | |
| *** ri0 has quit IRC | 17:25 | |
| jeblair | pabelanger, odyssey4me: so i think all of the restarts to pick up the 'keep files on remote' change finished last night -- have we seen erroneous timeouts or other such errors since then? | 17:25 |
| *** ri0 has joined #openstack-infra | 17:25 | |
| *** PollyZ has joined #openstack-infra | 17:25 | |
| pabelanger | jeblair: okay, I'll follow up with EmilienM so see what is happening | 17:27 |
| *** shashank_hegde has quit IRC | 17:27 | |
| pabelanger | jeblair: odyssey4me: Restarts did happen, and still see random timeout issue around puppet jobs. I haven't looked into openstack-ansible yet. | 17:28 |
| jeblair | pabelanger: maybe running an 'lsof' at the end of the job could help shed light on the fd-still-open issue? i'm not sure. | 17:28 |
| pabelanger | I plan to try and hold a node later this afternoon, but that is a pretty random process | 17:29 |
| pabelanger | jeblair: good idea | 17:29 |
| EmilienM | right it's super random | 17:29 |
| *** sdake_ has quit IRC | 17:29 | |
| *** sree_ has joined #openstack-infra | 17:29 | |
| *** banix has quit IRC | 17:29 | |
| *** sree_ is now known as Guest58637 | 17:30 | |
| *** ri0 has quit IRC | 17:30 | |
| *** coolsvap has quit IRC | 17:30 | |
| *** _nadya_ has joined #openstack-infra | 17:30 | |
| *** armax has quit IRC | 17:30 | |
| *** PollyZ has quit IRC | 17:31 | |
| jeblair | pabelanger, EmilienM: tell you what, i'm going to write a change to nodepool to help us automatically hold nodes that fail certain jobs. i think we need that. | 17:33 |
| EmilienM | jeblair++ | 17:33 |
| *** ri0 has joined #openstack-infra | 17:33 | |
| *** Guest58637 has quit IRC | 17:34 | |
| AJaeger | jeblair: please make it easy to switch on and off ;) not that it holds all nodes while you're asleep ;) | 17:34 |
| EmilienM | ahah | 17:35 |
| *** ddieterly is now known as ddieterly[away] | 17:36 | |
| jeblair | AJaeger: yeah, it will be by job name, so fairly targeted, however, it will still probably hold a lot of nodes, so is not something we want to leave turned on a lot. | 17:36 |
| *** tosky has quit IRC | 17:36 | |
| pabelanger | jeblair: great, thanks | 17:36 |
| *** tbarron is now known as tbarron|errand | 17:36 | |
| notmorgan | jeblair: maybe just have a limit on nodes that can be held as well ? | 17:36 |
| *** amitgandhinz has quit IRC | 17:37 | |
| jeblair | notmorgan: good idea and easy to do | 17:37 |
| notmorgan | yeah :) | 17:37 |
| *** amitgandhinz has joined #openstack-infra | 17:37 | |
| *** yaume has quit IRC | 17:39 | |
| *** jamesmcarthur has quit IRC | 17:40 | |
| *** rossella__ has joined #openstack-infra | 17:40 | |
| *** daemontool has joined #openstack-infra | 17:41 | |
| jeblair | okay, it's actually hard to do *well* but it might be easy to do *well enough*. we'll see :) | 17:42 |
| fungi | so, going to be submitting a change forthwith to create a new long-lived job node (where our signing subkey will be puppeted for signing tarballs, tags, et cetera). do we want to stick with the "slave" subdomain on it, or now that we're no longer using jenkins should we migrate away from its terminology via attrition? | 17:42 |
| jeblair | fungi: i would like to attrite | 17:42 |
| fungi | and if we do, what sort of new naming pattern do we want? | 17:43 |
| fungi | just name it like any of our other normal servers? | 17:43 |
| jeblair | strictly speaking, we probably don't need any subdomain do we? yeah, maybe so? | 17:43 |
| EmilienM | pabelanger: https://review.openstack.org/#/c/332820/ is about to be green (watching zuul) | 17:43 |
| EmilienM | pabelanger: can we land it? | 17:43 |
| clarkb | subdomains are useful with the 500 record limit on rax dns | 17:43 |
| *** daemontool has quit IRC | 17:44 | |
| jeblair | are we the biggest customer of every service we use? | 17:44 |
| *** rossella_ has quit IRC | 17:44 | |
| pleia2 | heh | 17:44 |
| fungi | i'll be surprised if we get beyond maybe half a dozen long-lived job nodes | 17:44 |
| notmorgan | jeblair: perhaps | 17:44 |
| *** rbrndt has quit IRC | 17:44 | |
| fungi | right now we have two i think, i'm adding a third for this | 17:45 |
| *** jkraj has joined #openstack-infra | 17:45 | |
| clarkb | fungi: ya just thinking more about fewer records in top level domain results in happiness | 17:45 |
| jeblair | fungi: we have 4 now | 17:45 |
| *** esikachev has joined #openstack-infra | 17:45 | |
| fungi | proposal, release, ... | 17:45 |
| jeblair | fungi: (proposal, publish, and 2 wheel builders) | 17:45 |
| fungi | wheel builders! | 17:45 |
| fungi | right, i forgot those | 17:45 |
| clarkb | granted they aren't configured as a proper subdomian anywhere (but its possible to make that switch later without much fuss this way) | 17:45 |
| fungi | we might end up with more wheel builders too i guess | 17:45 |
| EmilienM | pabelanger, jeblair: see 332820 in zuul, job is finished but still in progress in the telnet | 17:46 |
| *** mhickey has quit IRC | 17:46 | |
| *** Guest38349 has quit IRC | 17:46 | |
| EmilienM | 2016-06-22 17:43:25.288713 | [Zuul] standard output/error still open after child exited2016-06-22 17:43:25.288806 | [Zuul] Task exit code: 0 | 17:46 |
| EmilienM | 2016-06-22 17:43:35.357875 | [Zuul] Job complete, result: SUCCESS | 17:46 |
| EmilienM | it's hanging on ^ | 17:46 |
| EmilienM | is it waiting for something? | 17:46 |
| fungi | clarkb: are you proposing a nodes.openstack.org subdomain/zone then? | 17:46 |
| *** rhochmuth has quit IRC | 17:46 | |
| jeblair | EmilienM: zuul will now proceed (and should report the job as success), but something is still running and still outputing data to the console | 17:46 |
| EmilienM | jeblair: rsync? | 17:47 |
| *** rossella_ has joined #openstack-infra | 17:47 | |
| jeblair | EmilienM: no, i mean something in the job | 17:47 |
| *** flepied has quit IRC | 17:47 | |
| clarkb | fungi: I think we want a proper subdomain for infracloud, and another one for our less public facing hosts makes sense to me as well | 17:47 |
| EmilienM | jeblair: job is finished! | 17:47 |
| clarkb | fungi: I really don't want to be in another situation where "we can't deploy anything because dns" | 17:47 |
| EmilienM | well, it's done now | 17:47 |
| jeblair | EmilienM: like the job started a daemon, and the deamon forked but did not close the standard file descriptor | 17:47 |
| EmilienM | mhh ok | 17:47 |
| clarkb | we only had an out there because we could delete records for infracloud | 17:47 |
| fungi | i guess we could make ci.openstack.org more than just the name of a web redirect | 17:47 |
| fungi | it's nice and short, and works well for ci-specific bits | 17:48 |
| jeblair | we could do that, and start moving zuul servers into it too | 17:48 |
| jeblair | that's currently 17 servers | 17:48 |
| jeblair | 18 | 17:48 |
| fungi | or make an infra.openstack.org subdomain and move lots more stuff there over time | 17:48 |
| jeblair | idunno, someteen | 17:48 |
| jeblair | infra.o.o sounds confusing | 17:49 |
| *** nwkarsten has joined #openstack-infra | 17:49 | |
| fungi | all things sound confusing to me. computers! | 17:49 |
| *** esikachev has quit IRC | 17:49 | |
| fungi | ci.o.o makes sense only if we don't put stuff besides our ci systems in it | 17:50 |
| fungi | but that's still a ton of stuff | 17:50 |
| *** rbrndt has joined #openstack-infra | 17:50 | |
| *** rossella__ has quit IRC | 17:50 | |
| jeblair | or we could use a proper dns server | 17:50 |
| fungi | if we used a subdomain, we could _easily_ delegate it to a proper dns server | 17:51 |
| fungi | however we still wouldn't be able to chain dnssec back up to the .org tld without support from the servers hosting openstack.org unfortunately | 17:52 |
| fungi | weakest link | 17:52 |
| jeblair | fungi: oh, i was suggesting that we use a proper dns server for openstack.org | 17:52 |
| jeblair | i'm not really interested in working on something *other* than the openstack project :) | 17:52 |
| fungi | that becomes a hard conversation with non-infra people at the foundation who like to have an interface to quickly add and remove arbitrary records in that domain | 17:53 |
| *** shashank_hegde has joined #openstack-infra | 17:53 | |
| *** PollyZ has joined #openstack-infra | 17:53 | |
| fungi | but maybe it's a hard conversation worth having anyway | 17:53 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: move the release tools that need to run on secure nodes https://review.openstack.org/332448 | 17:54 |
| jeblair | i also like being able to quickly add and remove records from that domain, so maybe it's not that hard :) | 17:54 |
| fungi | if it weren't for the shared jurisdiction problem, this would be pretty simple to pull off | 17:54 |
| *** trown|lunch is now known as trown | 17:54 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: start a readme describing the release tools https://review.openstack.org/332945 | 17:54 |
| *** nwkarsten has quit IRC | 17:54 | |
| fungi | jeblair: well, if we're not married to putting dns records in git via gerrit, there are plenty of web front-ends for dns manglement | 17:54 |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Story api https://review.openstack.org/332946 | 17:55 |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Story api https://review.openstack.org/332946 | 17:57 |
| *** PollyZ has quit IRC | 17:57 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add comment field to node and reason to hold https://review.openstack.org/225428 | 18:01 |
| *** jamesmcarthur has joined #openstack-infra | 18:01 | |
| jeblair | mordred, notmorgan, pabelanger: ^ updated that patch in prep for the auto-hold thing | 18:01 |
| notmorgan | loolinh | 18:01 |
| notmorgan | looking* | 18:01 |
| fungi | was wracking my brain to remember the multi-tenant frontend i added for authoritative dns management at the isp i used to work for... it was http://www.nictool.com/ | 18:01 |
| *** rhochmuth has joined #openstack-infra | 18:02 | |
| fungi | sort of annoying that it's under the affero gpl | 18:02 |
| EmilienM | jeblair: what has been blocking zuul for 15 min -> Queue lengths: 3946 events, 307 results | 18:02 |
| fungi | also chances are there are much better solutions around these daus | 18:02 |
| fungi | days | 18:02 |
| *** salv-orlando has quit IRC | 18:02 | |
| *** sdague has joined #openstack-infra | 18:02 | |
| *** salv-orlando has joined #openstack-infra | 18:03 | |
| jeblair | EmilienM: i don't know. it would take me some time to find out. i'd prefer not to look until it passes 10k events, which is about as high as we've seen it go. | 18:03 |
| *** degorenko is now known as _degorenko|afk | 18:03 | |
| jeblair | EmilienM: there are a number of potential causes, most of which we plan to address in zuulv3 | 18:04 |
| jeblair | few of which we can do much about now | 18:04 |
| EmilienM | ack | 18:04 |
| EmilienM | fungi, jeblair: can I exceptionally ask to promote 332820 in gate please? | 18:05 |
| clarkb | jeblair: mordred 225428 reminds me it would be good to resurrect the db migrations change for nodepool | 18:05 |
| *** jpich has quit IRC | 18:05 | |
| EmilienM | fungi: it's my weekly joker :P | 18:05 |
| clarkb | at the time I was the only person reviewing it and I think it has merge conflicted its way into pain now (and likely needs new db foo for interim db changes) | 18:05 |
| EmilienM | fungi: oh nevermind, it's in gate now | 18:05 |
| *** PollyZ has joined #openstack-infra | 18:06 | |
| jeblair | clarkb: yes. perhaps we will land it right before we switch everything to zookeeper. :) | 18:07 |
| *** salv-orlando has quit IRC | 18:07 | |
| *** gongysh has quit IRC | 18:08 | |
| *** rhochmuth has quit IRC | 18:10 | |
| *** sree has joined #openstack-infra | 18:12 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Add oslosphinx to a docs requirements.txt https://review.openstack.org/332953 | 18:12 |
| *** kzaitsev_mb has quit IRC | 18:13 | |
| *** maishsk has joined #openstack-infra | 18:13 | |
| *** kzaitsev_mb has joined #openstack-infra | 18:13 | |
| *** rbrndt has quit IRC | 18:14 | |
| jeblair | clarkb, greghaynes: do we have any testing in nodepool that touches the zmq stuff ? | 18:15 |
| jeblair | clarkb, greghaynes: like "test that nodepool deletes a node when onFinalized is received over zmq"? | 18:15 |
| greghaynes | jeblair: its been a while, I think no but not totally sure | 18:16 |
| clarkb | jeblair: https://review.openstack.org/#/c/230023/ | 18:16 |
| *** sree has quit IRC | 18:16 | |
| greghaynes | aha | 18:16 |
| clarkb | so nothing is merged, and that conflicts now | 18:16 |
| jeblair | clarkb: cool, thanks, i'll fix it up | 18:16 |
| *** Apoorva_ has joined #openstack-infra | 18:17 | |
| *** rhochmuth has joined #openstack-infra | 18:17 | |
| *** rbrndt has joined #openstack-infra | 18:17 | |
| *** Apoorva has quit IRC | 18:17 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Story api https://review.openstack.org/332946 | 18:17 |
| *** dimtruck is now known as zz_dimtruck | 18:18 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Test ZMQ event handler https://review.openstack.org/230023 | 18:18 |
| jeblair | clarkb, greghaynes: ^ | 18:18 |
| *** david-lyle has joined #openstack-infra | 18:19 | |
| clarkb | jeblair: I think you need to add the usebuilder stuff to those tests as they build images then boot nodes | 18:20 |
| clarkb | not sure if there are other things that need updating to match current test framework | 18:20 |
| *** sree has joined #openstack-infra | 18:20 | |
| jeblair | clarkb: good point. i'll actually run the tests | 18:21 |
| *** amoralej is now known as amoralej|off | 18:21 | |
| fungi | jeblair: clarkb: so, to beat this horse just a little more... preferences from you between signing01.slave.o.o vs signing01.ci.o.o vs just signing01.o.o? | 18:22 |
| fungi | we can always change it down the road, but would at least like to test drive a mostly agreeable solution | 18:23 |
| *** yamamoto has quit IRC | 18:23 | |
| *** ri0 has quit IRC | 18:24 | |
| fungi | (01 because we said we'd start adding ordinal suffixes to new hostnames) | 18:24 |
| jeblair | fungi: slave.o.o:-2, ci.o.o:+1, .o.o:+0.5 | 18:24 |
| clarkb | fungi: signing01.ci.o.o and signing01.o.o are both fine. I agree we should move away from slave. particularly since we aren't using jenkins anymore | 18:24 |
| *** sree has quit IRC | 18:25 | |
| fungi | okay, cool. i'm 50/50 on ci.o.o or just o.o but also not a fan od continuing slave.o.o so i'll go with ci.o.o as the most agreeable | 18:25 |
| jeblair | clarkb: those tests pass as-is | 18:25 |
| *** amitgandhinz has quit IRC | 18:26 | |
| *** amitgandhinz has joined #openstack-infra | 18:26 | |
| jeblair | clarkb: because they don't use dib images? | 18:26 |
| clarkb | jeblair: oh that could be if its all snapshots | 18:26 |
| *** Apoorva_ has quit IRC | 18:27 | |
| *** Apoorva has joined #openstack-infra | 18:27 | |
| *** sree_ has joined #openstack-infra | 18:28 | |
| *** dizquierdo has quit IRC | 18:28 | |
| *** sree_ is now known as Guest2720 | 18:28 | |
| *** Apoorva has quit IRC | 18:29 | |
| *** e0ne has joined #openstack-infra | 18:30 | |
| *** Apoorva has joined #openstack-infra | 18:30 | |
| *** sree has joined #openstack-infra | 18:32 | |
| *** Guest2720 has quit IRC | 18:32 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Story api https://review.openstack.org/332946 | 18:33 |
| *** lucasagomes is now known as lucas-afk | 18:34 | |
| *** jamesmcarthur has quit IRC | 18:34 | |
| *** eezhova has quit IRC | 18:35 | |
| *** amitgandhinz has quit IRC | 18:35 | |
| *** jamesmcarthur has joined #openstack-infra | 18:35 | |
| *** amitgandhinz has joined #openstack-infra | 18:35 | |
| *** ddieterly[away] has quit IRC | 18:36 | |
| *** sree has quit IRC | 18:37 | |
| *** rossella__ has joined #openstack-infra | 18:39 | |
| *** sree has joined #openstack-infra | 18:39 | |
| *** esikachev has joined #openstack-infra | 18:39 | |
| *** abregman has joined #openstack-infra | 18:40 | |
| jeblair | clarkb, greghaynes: how do i see nodepool debug logs when running a test? | 18:40 |
| clarkb | jeblair: they are written to the subunit log for that particular test using the log capture stuff | 18:41 |
| *** ifarkas has quit IRC | 18:42 | |
| *** jkraj has quit IRC | 18:42 | |
| clarkb | will be in .testrepository/$testrrunnumber | 18:42 |
| *** dtantsur is now known as dtantsur|afk | 18:42 | |
| jeblair | clarkb: i'm running "python -m testtools.run nodepool.tests.test_nodepool.TestNodepool.test_foo" | 18:42 |
| jeblair | clarkb: is that still true? | 18:42 |
| *** rossella_ has quit IRC | 18:42 | |
| clarkb | if you do that then there is no subunit, not sure how the logging fixture deals with that | 18:42 |
| jeblair | hrm | 18:43 |
| jeblair | clarkb: is there a better way to run a single test and see its output then? | 18:43 |
| *** matrohon has joined #openstack-infra | 18:43 | |
| *** sree has quit IRC | 18:43 | |
| clarkb | jeblair: I would do tox -epy27 -- test_foo | 18:43 |
| *** sree has joined #openstack-infra | 18:44 | |
| clarkb | jeblair: you can likely disable the log capturing with an env var and it will go to stdout using your command | 18:44 |
| jeblair | clarkb: how do i see the output after doing that? | 18:44 |
| *** _nadya_ has quit IRC | 18:44 | |
| *** r-mibu has quit IRC | 18:45 | |
| clarkb | OS_LOG_CAPTURE=0 python -m... | 18:45 |
| jeblair | clarkb: well, i don't have any of the OS_ vars set in the environment where i was running python -m. ... | 18:45 |
| *** ddieterly has joined #openstack-infra | 18:45 | |
| *** tonytan4ever has joined #openstack-infra | 18:45 | |
| *** r-mibu has joined #openstack-infra | 18:45 | |
| clarkb | oh right we set those in the testr conf | 18:45 |
| clarkb | jeblair: look in the subunit log file if you used my command | 18:45 |
| jeblair | clarkb: i confirmed that when i run python -m, it runs logging.basicConfig(level=logging.DEBUG) | 18:45 |
| clarkb | .testrepository/$testrrunnumber | 18:45 |
| jeblair | clarkb: how do i know the testrunnumber? | 18:46 |
| clarkb | jeblair: it tells you at the end of the run, | 18:46 |
| * clarkb finds example | 18:46 | |
| *** sdake has joined #openstack-infra | 18:47 | |
| jeblair | clarkb: yep, it's there | 18:47 |
| jeblair | this is not easy | 18:47 |
| clarkb | http://logs.openstack.org/28/225428/3/check/gate-nodepool-python27-db/be56e8b/console.html#_2016-06-22_18_20_21_830963 the id=0 identifies the testrrunnumber | 18:47 |
| jeblair | somehow this dtrt in zuul | 18:47 |
| jeblair | and it's weird because *some* output goes to stdout | 18:48 |
| *** sree has quit IRC | 18:48 | |
| jeblair | just not the nodepool log | 18:48 |
| jeblair | (to be clear, the subunit log has everything, i'm just still confused why running it in the foreground only has some stuff) | 18:48 |
| *** sree has joined #openstack-infra | 18:48 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/infra-specs: Update Artifact Signing details https://review.openstack.org/332968 | 18:49 |
| mtreinish | jeblair: give ostestr or subunit-trace a try (ostestr just uses it by default) | 18:50 |
| mtreinish | but it will print the captured artifacts to stdout | 18:50 |
| mtreinish | (it might just be stdout and stderr) | 18:50 |
| mtreinish | on failures it will dump everything though | 18:50 |
| clarkb | jeblair: I do not immediately see why that would happen the log config happens in one place so I dout its a race with something else configuring logs (though maybe it is?) | 18:51 |
| clarkb | in theory setup is going to run before any other nodepool code and that should set the logging t odebug | 18:52 |
| *** Sukhdev has joined #openstack-infra | 18:52 | |
| *** javeriak_ has quit IRC | 18:53 | |
| *** sree has quit IRC | 18:53 | |
| mtreinish | clarkb: do you have the logging fixture configured with the nuke option? | 18:53 |
| mtreinish | clarkb: because you should be able to write to a log file and capture things in the subunit too (that's how tempest does it) | 18:54 |
| clarkb | mtreinish: I don't think so I hve no idea what the nuke option is. But in jeblairs case the logging fixture should not be used since the OS_* vars are not set | 18:54 |
| clarkb | it should just be default python logging dumping to stdout or stderr | 18:54 |
| *** ajmiller has quit IRC | 18:54 | |
| mtreinish | clarkb: https://github.com/testing-cabal/fixtures/blob/master/fixtures/_fixtures/logger.py#L42-L43 | 18:55 |
| *** sdake_ has joined #openstack-infra | 18:55 | |
| mtreinish | clarkb: oh, ok. But, by default stdout and stderr get gobbled up by subunit | 18:55 |
| clarkb | zuuls tests do the basic logging thing at the module level in tests/__init__.py | 18:55 |
| clarkb | jeblair: ^ rather than in the setup method | 18:56 |
| clarkb | mtreinish: not in this case because there are no OS_* vars | 18:56 |
| clarkb | mtreinish: he isn't using testr which turns all that on | 18:56 |
| *** sree has joined #openstack-infra | 18:56 | |
| mtreinish | clarkb: it does, just not captured (which what those vars do) | 18:56 |
| mtreinish | it just doesn't passthrough | 18:57 |
| clarkb | mtreinish: there is no subunit... | 18:57 |
| mtreinish | oh, I thought he was running with subunit. Nm then | 18:57 |
| clarkb | jeblair: I think it must be a difference between configuring logging at import time and execution time | 18:57 |
| *** kzaitsev_mb has quit IRC | 18:58 | |
| *** sdake has quit IRC | 18:58 | |
| *** achanda has joined #openstack-infra | 18:58 | |
| clarkb | and whatever the logging fixture does is smart enough to do it at run time | 18:58 |
| *** ajmiller has joined #openstack-infra | 18:59 | |
| *** zz_dimtruck is now known as dimtruck | 19:00 | |
| *** roxanaghe has quit IRC | 19:00 | |
| *** sarob has joined #openstack-infra | 19:01 | |
| *** sree has quit IRC | 19:01 | |
| *** roxanaghe has joined #openstack-infra | 19:02 | |
| *** achanda has quit IRC | 19:02 | |
| *** sree has joined #openstack-infra | 19:03 | |
| *** rossella_ has joined #openstack-infra | 19:03 | |
| openstackgerrit | Merged openstack-infra/project-config: Add partition image src jobs for agent driver https://review.openstack.org/330738 | 19:04 |
| *** hemanthm|afk is now known as hemanthm | 19:06 | |
| *** sree_ has joined #openstack-infra | 19:07 | |
| *** flepied has joined #openstack-infra | 19:07 | |
| *** rossella__ has quit IRC | 19:07 | |
| *** jordan_ has joined #openstack-infra | 19:07 | |
| *** yolanda has left #openstack-infra | 19:07 | |
| *** yolanda has joined #openstack-infra | 19:07 | |
| *** sree_ is now known as Guest33406 | 19:07 | |
| *** sree has quit IRC | 19:07 | |
| *** jordan_ is now known as jordanP | 19:07 | |
| *** rbrndt has quit IRC | 19:08 | |
| *** sshnaidm_ has quit IRC | 19:08 | |
| *** rhochmuth has quit IRC | 19:09 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Story api https://review.openstack.org/332946 | 19:11 |
| *** Guest33406 has quit IRC | 19:11 | |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Add configurations for gerrit its plugins https://review.openstack.org/330922 | 19:11 |
| *** e0ne has quit IRC | 19:12 | |
| *** jordanP has quit IRC | 19:13 | |
| *** jordanP has joined #openstack-infra | 19:13 | |
| *** ddieterly is now known as ddieterly[away] | 19:13 | |
| *** jordanP has quit IRC | 19:15 | |
| *** rbrndt has joined #openstack-infra | 19:15 | |
| *** jordanP_ has joined #openstack-infra | 19:15 | |
| *** sree has joined #openstack-infra | 19:16 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add auto-hold feature https://review.openstack.org/332976 | 19:17 |
| jeblair | clarkb, mordred, pabelanger, notmorgan, EmilienM: ^ | 19:17 |
| *** baoli has quit IRC | 19:17 | |
| jeblair | mtreinish: ah, thanks, if i do end up having to use subunit, ostestr sounds like it may do what i want (yay!) | 19:18 |
| *** baoli has joined #openstack-infra | 19:18 | |
| *** jordanP has joined #openstack-infra | 19:18 | |
| *** jordanP_ has quit IRC | 19:18 | |
| *** tqtran has joined #openstack-infra | 19:19 | |
| *** eezhova has joined #openstack-infra | 19:19 | |
| *** jordanP has quit IRC | 19:19 | |
| *** jordanP has joined #openstack-infra | 19:19 | |
| *** sree has quit IRC | 19:20 | |
| *** jamielennox is now known as jamielennox|away | 19:21 | |
| *** HeOS has quit IRC | 19:22 | |
| *** tonytan4ever has quit IRC | 19:22 | |
| *** jordanP has quit IRC | 19:23 | |
| *** javeriak has joined #openstack-infra | 19:24 | |
| *** yamamoto has joined #openstack-infra | 19:24 | |
| *** eezhova has quit IRC | 19:24 | |
| pabelanger | jeblair: looking | 19:25 |
| mtreinish | jeblair: just an fyi, I copied a a bit of the output from ostestr/subunit-trace: http://paste.openstack.org/show/521266/ I think that's what you're looking for | 19:27 |
| *** pvaneck has joined #openstack-infra | 19:28 | |
| *** sree has joined #openstack-infra | 19:28 | |
| jeblair | mtreinish: ya. though i am just reminded of why i like to run in the foreground -- when i write something that hangs the program or is slow, it's nice to see that in real time | 19:29 |
| mordred | jeblair, pabelanger: the first patch in the series (my very old held-comment patch) failed coverage. I popped a recheck, it looked like a spurious timeout | 19:29 |
| *** rossella__ has joined #openstack-infra | 19:30 | |
| jeblair | mordred: like, the thing this whole patch series is designed to help us track down? | 19:30 |
| *** kgiusti has quit IRC | 19:30 | |
| *** sdake has joined #openstack-infra | 19:31 | |
| *** sarob has quit IRC | 19:31 | |
| mordred | jeblair: yup | 19:31 |
| mordred | jeblair: funny that | 19:31 |
| *** sdake_ has quit IRC | 19:33 | |
| *** sree has quit IRC | 19:33 | |
| *** rossella_ has quit IRC | 19:33 | |
| *** javeriak_ has joined #openstack-infra | 19:33 | |
| *** javeriak has quit IRC | 19:34 | |
| clarkb | this stack will need manual DB munging after it merges | 19:34 |
| *** yamamoto has quit IRC | 19:34 | |
| *** baoli has quit IRC | 19:35 | |
| *** baoli has joined #openstack-infra | 19:35 | |
| *** ddieterly[away] is now known as ddieterly | 19:37 | |
| jeblair | clarkb: yes -- the comment field will need manual db munge; though the new table will be automatic | 19:38 |
| *** ddieterly is now known as ddieterly[away] | 19:38 | |
| jeblair | i am going to burrito | 19:38 |
| openstackgerrit | Merged openstack-infra/nodepool: Revoke nodes via gearman on deletion https://review.openstack.org/331274 | 19:39 |
| *** javeriak has joined #openstack-infra | 19:40 | |
| *** javeriak_ has quit IRC | 19:41 | |
| craige | o/ | 19:43 |
| *** eezhova has joined #openstack-infra | 19:44 | |
| *** sarob has joined #openstack-infra | 19:44 | |
| clarkb | jeblair: found a couple small things on the hold failed jobs change | 19:44 |
| *** _nadya_ has joined #openstack-infra | 19:45 | |
| *** sree has joined #openstack-infra | 19:45 | |
| *** markusry has quit IRC | 19:48 | |
| *** sarob has quit IRC | 19:50 | |
| *** sree has quit IRC | 19:50 | |
| *** _nadya_ has quit IRC | 19:50 | |
| *** sree has joined #openstack-infra | 19:51 | |
| *** baoli has quit IRC | 19:53 | |
| *** baoli has joined #openstack-infra | 19:54 | |
| *** abregman has quit IRC | 19:55 | |
| fungi | anybody happen to know why release.slave.o.o is in the afsadmin group in our global site manifestand an admin_keytab? | 19:55 |
| *** sree has quit IRC | 19:56 | |
| fungi | is it running vos release? | 19:56 |
| notmorgan | jeblair: burrito is a good plan. | 19:56 |
| fungi | i guess i can figure that out pretty easily | 19:56 |
| *** pradk_ has joined #openstack-infra | 19:56 | |
| openstackgerrit | James Page proposed openstack-infra/project-config: Add openstack-charm-guide repository https://review.openstack.org/332916 | 19:58 |
| fungi | ahh, yep. so it is | 19:58 |
| *** sree has joined #openstack-infra | 19:59 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Story api https://review.openstack.org/332946 | 20:01 |
| *** sarob has joined #openstack-infra | 20:01 | |
| *** sdake_ has joined #openstack-infra | 20:01 | |
| *** kgiusti has joined #openstack-infra | 20:02 | |
| *** sree has quit IRC | 20:03 | |
| *** salv-orlando has joined #openstack-infra | 20:04 | |
| *** sdake has quit IRC | 20:04 | |
| *** sree has joined #openstack-infra | 20:05 | |
| *** amitgandhinz has quit IRC | 20:06 | |
| *** sarob has quit IRC | 20:07 | |
| *** amitgandhinz has joined #openstack-infra | 20:07 | |
| *** matt-borland has joined #openstack-infra | 20:08 | |
| *** sree has quit IRC | 20:10 | |
| *** ddieterly[away] is now known as ddieterly | 20:10 | |
| *** sree has joined #openstack-infra | 20:11 | |
| *** vhosakot_ has quit IRC | 20:11 | |
| *** knikolla has joined #openstack-infra | 20:13 | |
| *** stevemar has quit IRC | 20:14 | |
| pabelanger | EmilienM: improvement in puppet jobs after merging the copy_logs.sh fix? | 20:14 |
| *** vhosakot has joined #openstack-infra | 20:14 | |
| *** stevemar has joined #openstack-infra | 20:14 | |
| *** baoli has quit IRC | 20:14 | |
| *** baoli_ has joined #openstack-infra | 20:14 | |
| *** sree has quit IRC | 20:16 | |
| *** nwkarsten has joined #openstack-infra | 20:16 | |
| *** pradk has quit IRC | 20:17 | |
| *** pradk_ is now known as pradk | 20:17 | |
| *** sree has joined #openstack-infra | 20:19 | |
| *** stevemar has quit IRC | 20:19 | |
| *** banix has joined #openstack-infra | 20:19 | |
| *** Apoorva has quit IRC | 20:21 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard: Add example commands for the Story api https://review.openstack.org/332946 | 20:23 |
| *** sree has quit IRC | 20:23 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add auto-hold feature https://review.openstack.org/332976 | 20:24 |
| *** aorourke__ is now known as aorourke | 20:24 | |
| *** Apoorva has joined #openstack-infra | 20:24 | |
| jeblair | clarkb: thanks ^ mordred, pabelanger: ^ updated | 20:24 |
| *** sree has joined #openstack-infra | 20:25 | |
| *** rossella_ has joined #openstack-infra | 20:27 | |
| *** ajmiller has quit IRC | 20:27 | |
| *** salv-orlando has quit IRC | 20:27 | |
| *** vhosakot has quit IRC | 20:28 | |
| *** salv-orlando has joined #openstack-infra | 20:28 | |
| *** matrohon has quit IRC | 20:28 | |
| *** rbrndt has quit IRC | 20:29 | |
| *** sree has quit IRC | 20:29 | |
| pabelanger | jeblair: +2 hopefully we don't need to rely on it much | 20:30 |
| *** ajmiller has joined #openstack-infra | 20:30 | |
| *** sree has joined #openstack-infra | 20:30 | |
| jeblair | pabelanger: do you want to come up with a list of job names we should enable there? | 20:31 |
| pabelanger | jeblair: sure | 20:31 |
| *** rossella__ has quit IRC | 20:31 | |
| mordred | jeblair: sigh. the first patch in the series failed again | 20:31 |
| *** vhosakot has joined #openstack-infra | 20:31 | |
| *** sarob has joined #openstack-infra | 20:32 | |
| *** Rockyg has quit IRC | 20:33 | |
| *** sdake_ has quit IRC | 20:34 | |
| *** asselin has joined #openstack-infra | 20:34 | |
| *** amitgandhinz has quit IRC | 20:35 | |
| *** dprince has quit IRC | 20:35 | |
| *** amitgandhinz has joined #openstack-infra | 20:35 | |
| *** sree has quit IRC | 20:35 | |
| *** sarob_ has joined #openstack-infra | 20:36 | |
| *** sree has joined #openstack-infra | 20:37 | |
| jeblair | actually http://logs.openstack.org/28/225428/3/check/gate-nodepool-python27-db/201588f/console.html is a good example of a failure that is difficult (impossible?) to debug without timestamps | 20:37 |
| jeblair | so i think we need to add timestamps to the nodepool unit test logging (in the subunit context) | 20:38 |
| *** ArxCruz has quit IRC | 20:38 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:39 | |
| *** baoli_ has quit IRC | 20:39 | |
| *** sarob has quit IRC | 20:40 | |
| *** hamzy_ is now known as hamzy | 20:40 | |
| *** sree has quit IRC | 20:41 | |
| *** sree has joined #openstack-infra | 20:42 | |
| *** nwkarsten has quit IRC | 20:42 | |
| *** nwkarsten has joined #openstack-infra | 20:43 | |
| *** kzaitsev_mb has quit IRC | 20:46 | |
| *** eezhova has quit IRC | 20:46 | |
| *** nwkarsten has quit IRC | 20:47 | |
| *** sree has quit IRC | 20:47 | |
| *** sree has joined #openstack-infra | 20:48 | |
| openstackgerrit | Merged openstack/os-testr: Updated from global requirements https://review.openstack.org/332335 | 20:49 |
| *** _ari_ has quit IRC | 20:49 | |
| *** rook has quit IRC | 20:51 | |
| *** rbrndt has joined #openstack-infra | 20:51 | |
| *** rook has joined #openstack-infra | 20:51 | |
| *** sree has quit IRC | 20:52 | |
| *** matrohon has joined #openstack-infra | 20:52 | |
| openstackgerrit | Merged openstack-infra/release-tools: fix release notes script for new reno https://review.openstack.org/332852 | 20:52 |
| *** gouthamr has quit IRC | 20:54 | |
| *** stevemar has joined #openstack-infra | 20:54 | |
| *** kgiusti has quit IRC | 20:54 | |
| *** BigWillie has quit IRC | 20:54 | |
| notmorgan | jeblair: ++ we should | 20:55 |
| *** armax has joined #openstack-infra | 20:55 | |
| notmorgan | jeblair: i was noticing that gap when doing some recent looking at nodepool | 20:55 |
| *** akshai has joined #openstack-infra | 20:57 | |
| *** sree has joined #openstack-infra | 20:57 | |
| *** jamesmcarthur has quit IRC | 20:58 | |
| openstackgerrit | Merged openstack-infra/release-tools: do not show empty change lists https://review.openstack.org/332856 | 20:59 |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Story api https://review.openstack.org/332946 | 21:00 |
| *** javeriak has quit IRC | 21:01 | |
| *** rook has quit IRC | 21:01 | |
| pabelanger | jeblair: gate-puppet-openstack-integration-3-scenario001-tempest-ubuntu-xenial is our job for today | 21:01 |
| *** ddieterly is now known as ddieterly[away] | 21:01 | |
| *** jamesmcarthur has joined #openstack-infra | 21:01 | |
| *** rook has joined #openstack-infra | 21:01 | |
| pabelanger | jeblair: I haven't see anything from openstack-ansible today, but will follow up | 21:01 |
| *** sree has quit IRC | 21:02 | |
| *** eezhova has joined #openstack-infra | 21:02 | |
| *** eezhova has quit IRC | 21:02 | |
| *** moravec has joined #openstack-infra | 21:02 | |
| *** sree has joined #openstack-infra | 21:02 | |
| *** sree_ has joined #openstack-infra | 21:04 | |
| *** trown is now known as trown|outtypewww | 21:04 | |
| *** sree_ is now known as Guest79737 | 21:04 | |
| pabelanger | jeblair: and gate-openstack-ansible-rsyslog_server-ansible-func-centos-7-nv from openstack ansible | 21:05 |
| *** acoles is now known as acoles_ | 21:05 | |
| *** thorst_ has joined #openstack-infra | 21:05 | |
| *** psilvad has quit IRC | 21:05 | |
| clarkb | pabelanger: have a link to some example fails for those? or do we not even get logs? | 21:05 |
| *** kzaitsev_mb has joined #openstack-infra | 21:05 | |
| *** ddieterly[away] is now known as ddieterly | 21:05 | |
| *** julim has quit IRC | 21:05 | |
| *** yolanda has quit IRC | 21:06 | |
| pabelanger | clarkb: http://logs.openstack.org/29/332229/3/check/gate-openstack-ansible-rsyslog_server-ansible-func-centos-7-nv/b9f95ce/ | 21:06 |
| *** matrohon has quit IRC | 21:06 | |
| pabelanger | http://logs.openstack.org/98/330498/10/check/gate-puppet-openstack-integration-3-scenario001-tempest-ubuntu-xenial/e49bcf6/ | 21:06 |
| pabelanger | are examples of the failures | 21:06 |
| openstackgerrit | Steve Baker proposed openstack-infra/shade: Treat DELETE_COMPLETE stacks as NotFound https://review.openstack.org/333026 | 21:07 |
| *** rhallisey has quit IRC | 21:07 | |
| clarkb | neat, the puppet one disappears during tempest | 21:07 |
| *** acoles_ is now known as acoles | 21:07 | |
| clarkb | the other looks like it runs successfully then zuul launcher reports fail back? | 21:07 |
| *** sree has quit IRC | 21:07 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Fix race condition in disconnect test https://review.openstack.org/333028 | 21:07 |
| *** sree has joined #openstack-infra | 21:08 | |
| jeblair | clarkb, greghaynes, notmorgan, fungi: ^ that should fix the teast_handle_disconnect test in nodepool | 21:08 |
| mordred | jeblair: nice | 21:08 |
| *** Guest79737 has quit IRC | 21:08 | |
| jeblair | fungi: i particularly want your opinion on the commit message. | 21:08 |
| pabelanger | clarkb: right, our async handler in ansible reports a timeout, which I suspect is actually an exception on our worker. Then zuul-launcher brings down the job as a failure | 21:09 |
| pabelanger | clarkb: in both cases, not enough time as passed for ansible to trigger a timeout. | 21:10 |
| pabelanger | clarkb: we have some openstack ansible linters job that would fail pretty quickly, but I don't see any of them today | 21:10 |
| jeblair | perhaps we're hitting more than one issue with async, and the keep_remote helped with one of them? | 21:11 |
| fungi | jeblair: was that suspected to be the main source of instability we had in nodepool unit tests? | 21:11 |
| clarkb | fungi: there is another one where the mysqldb fixture can't delete the db because the db is in use | 21:11 |
| fungi | seems plausible | 21:11 |
| pabelanger | jeblair: possible. Since restarts last night, I haven't see linters failing | 21:11 |
| clarkb | but my attempts at debugging that particular issue haven't fixed it | 21:11 |
| jeblair | fungi: that addresses instability in a single test | 21:11 |
| pabelanger | which includes our keep_remote_files | 21:12 |
| *** sree has quit IRC | 21:12 | |
| fungi | jeblair: yeah, i just don't remember if that's the particular test i saw failing regularly. i'm guessing it is | 21:12 |
| jeblair | clarkb: you have a change to dump state for that, right? | 21:12 |
| jeblair | fungi: it rings a bell with me at least :) | 21:12 |
| fungi | my bells are pretty wrung out | 21:12 |
| clarkb | pabelanger: thanks I see that now in the ansible logs | 21:13 |
| clarkb | jeblair: I do, there are a pair iirc | 21:13 |
| fungi | i'm eyeballs-deep in openpgp/gnupg literature so brain turning to goop | 21:13 |
| mordred | fungi: you still have one at this point doing that? | 21:13 |
| *** ffrank has joined #openstack-infra | 21:13 | |
| fungi | dunno, i'll have to ask it | 21:13 |
| clarkb | jeblair: https://review.openstack.org/#/c/280989/ and its parent | 21:13 |
| jeblair | clarkb: is this the symptom or something else? http://logs.openstack.org/28/225428/3/check/nodepool-coverage-db/b388e62/console.html | 21:14 |
| *** PollyZ has quit IRC | 21:14 | |
| clarkb | jeblair: thats something else (but possibly related) | 21:15 |
| *** HeOS has joined #openstack-infra | 21:15 | |
| clarkb | jeblair: the fail I am talking about tracebacks in the cleanup for the db fixture on the drop dbname execution | 21:15 |
| jeblair | gotcha | 21:15 |
| clarkb | they both fail on a timeout exception though | 21:15 |
| *** aeng has joined #openstack-infra | 21:15 | |
| clarkb | perhaps the same persistent db locker is locking in both cases? | 21:16 |
| *** PollyZ has joined #openstack-infra | 21:16 | |
| *** amitgandhinz has quit IRC | 21:17 | |
| *** jamesmcarthur has quit IRC | 21:17 | |
| jeblair | oh my -- dougwig has made an excellent point in https://review.openstack.org/330698 | 21:17 |
| *** ffrank has quit IRC | 21:18 | |
| fungi | hah | 21:19 |
| mordred | dougwig wins | 21:19 |
| *** ldnunes has quit IRC | 21:20 | |
| fungi | though it will get confusing when we have to keep reminding users that there is no dana | 21:20 |
| *** acoles is now known as acoles_ | 21:20 | |
| mordred | fungi: but we would have an unending source of reasons to say that | 21:21 |
| dmsimard | http://knowyourmeme.com/memes/there-is-no-dana-only-zuul | 21:21 |
| *** acoles_ is now known as acoles | 21:21 | |
| *** Sukhdev has quit IRC | 21:21 | |
| *** acoles is now known as acoles_ | 21:22 | |
| mordred | dmsimard: I have learned many new informations from that informative article | 21:23 |
| *** acoles_ is now known as acoles | 21:24 | |
| *** ffrank has joined #openstack-infra | 21:24 | |
| dougwig | lol | 21:24 |
| *** _Fdaisuke_ has joined #openstack-infra | 21:25 | |
| *** Fdaisuke has quit IRC | 21:26 | |
| *** jamesmcarthur has joined #openstack-infra | 21:26 | |
| *** thiagop-afk has quit IRC | 21:28 | |
| *** sdake has joined #openstack-infra | 21:28 | |
| *** ffrank has quit IRC | 21:28 | |
| *** sree has joined #openstack-infra | 21:29 | |
| *** yolanda has joined #openstack-infra | 21:29 | |
| pabelanger | pleia2: I'm going to start fleshing out more of the infracloud talk later tonight: https://etherpad.openstack.org/p/barcelona-upstream-openstack-infa | 21:29 |
| *** acoles is now known as acoles_ | 21:30 | |
| pabelanger | jeblair: mordred: would be also interested in thoughts about talking about zuulv2.5^ If you think it is appropriate | 21:30 |
| *** mhickey has joined #openstack-infra | 21:30 | |
| *** acoles_ is now known as acoles | 21:31 | |
| pabelanger | thingee: also, do you think we'll be having a dedicated lightning talks slot again in Barcelona? | 21:32 |
| mtreinish | nibalizer: did your thing change the way lint enforcement is done: http://logs.openstack.org/67/328467/2/gate/gate-puppet-openstack_health-puppet-lint/44d3d85/console.html#_2016-06-22_21_14_06_370900 | 21:32 |
| *** sree has quit IRC | 21:33 | |
| *** burgerk has quit IRC | 21:33 | |
| thingee | pabelanger: oh yes! that was a great - I'll submit something | 21:33 |
| SamYaple | say I wanted to duplicate the setup here in infra for a private gerrit/jenkins/zuul setup... is there docs sfor that? and is it recommended/not recommended? | 21:33 |
| mordred | pabelanger: I'd actually like to talk about zuul v3 in barcelona | 21:33 |
| thingee | mordred: +1 | 21:33 |
| mordred | SamYaple: there are docs, and even a puppet module just for you | 21:33 |
| *** sree has joined #openstack-infra | 21:34 | |
| thingee | "Look mom, no jenkins" | 21:34 |
| mordred | SamYaple: http://git.openstack.org/cgit/openstack-infra/puppet-openstackci is the puppet module that does the things that are not gerrit | 21:34 |
| *** roxanaghe has quit IRC | 21:34 | |
| SamYaple | mordred: and there are docs you say? | 21:34 |
| SamYaple | my lucky day | 21:35 |
| mordred | SamYaple: I don't remember where the docs are ... pabelanger ? | 21:35 |
| *** roxanaghe has joined #openstack-infra | 21:35 | |
| SamYaple | i did GTS before asking, but came up empty | 21:35 |
| *** pt_15 has quit IRC | 21:36 | |
| clarkb | mordred: SamYaple that repo has in tree docs http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/doc/source/third_party_ci.rst | 21:36 |
| *** yolanda has quit IRC | 21:36 | |
| mordred | clarkb: awesome. that's what I was missing | 21:36 |
| SamYaple | clarkb: always coming through. da best | 21:36 |
| clarkb | there is also a running your own at http://docs.openstack.org/infra/system-config/running-your-own.html with an out of date warning | 21:36 |
| pabelanger | right, I only knew of the 2nd one clarkb linked | 21:37 |
| *** ffrank has joined #openstack-infra | 21:37 | |
| *** sdake_ has joined #openstack-infra | 21:38 | |
| EmilienM | pabelanger: back. All seems good for us now | 21:38 |
| *** sree has quit IRC | 21:38 | |
| *** eezhova has joined #openstack-infra | 21:39 | |
| *** esikachev has quit IRC | 21:39 | |
| *** sdake has quit IRC | 21:39 | |
| *** eezhova has quit IRC | 21:40 | |
| *** itisha has quit IRC | 21:40 | |
| *** derekh has joined #openstack-infra | 21:40 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 21:40 | |
| pabelanger | rcarrillocruz: crinkle: thoughts above for an infracloud talk^? Giving an update on our progress so far, things we found, etc? | 21:40 |
| rcarrillocruz | pabelanger: if they move the racks on time for Barcelona, sure! ;-) | 21:41 |
| rcarrillocruz | j/k | 21:41 |
| rcarrillocruz | happy to talk about it , sure | 21:41 |
| pabelanger | rcarrillocruz: Ya, maybe tomorrow we can do a quick status run down see where we are with the racks, see if we can get a controller backonline | 21:41 |
| rcarrillocruz | yeah, i've been on/off this week, on Berlin | 21:42 |
| rcarrillocruz | i'll check with Lab on Monday if we have a more definite date for the move | 21:42 |
| mordred | rcarrillocruz: if you're still in Berlin - you should go eat/drink at White Trash Fast Food | 21:42 |
| derekh | the tripleo geard that brokers test environments is seeing a load of tcp connections being left open, as if the clients just disappeared. | 21:43 |
| derekh | this is a long shot but by any chance was anything restarted around 15:09 UTC today? zuul or nodepool maybe? and again at 16:21? | 21:43 |
| rcarrillocruz | gah, leaving tomorrow at noon-ish | 21:43 |
| *** banix has quit IRC | 21:43 | |
| rcarrillocruz | pabelanger: i'll also need you/other core to make amendments to hiera | 21:44 |
| rcarrillocruz | since i asked for a pw reset on a lot of servers | 21:44 |
| mordred | rcarrillocruz: blerg. well, next time you're there: http://whitetrashfastfood.com/ | 21:44 |
| clarkb | rcarrillocruz: this is ilo password updates? | 21:44 |
| rcarrillocruz | yeah, checking the site, looking awesome | 21:44 |
| *** sree has joined #openstack-infra | 21:44 | |
| mordred | derekh: I don't see any mention in IRC of us having done that at 15:09 | 21:44 |
| rcarrillocruz | clarkb: yes | 21:44 |
| mordred | derekh: which doesn't mean it didn't happen | 21:45 |
| *** eharney has quit IRC | 21:45 | |
| clarkb | mordred: I am not sure that organic patties and buns is a proper white trash burger :P | 21:45 |
| derekh | mordred: ya I didn't see anything obvious in irc either | 21:45 |
| pabelanger | rcarrillocruz: sure, happy to do that. If you want to send me the info | 21:45 |
| *** mhickey has quit IRC | 21:46 | |
| pabelanger | rcarrillocruz: tomorrow?\ | 21:46 |
| *** sambetts is now known as sambetts|afk | 21:46 | |
| rcarrillocruz | let's better do on Friday | 21:46 |
| rcarrillocruz | tomorrow i have flight and drive back home, don't expect to arrive till 8PM my time | 21:47 |
| clarkb | pabelanger: rcarrillocruz as far as deploying a new controller we need to build new dib images right? have we done that? might be worthwhile if not just to make sure that piece is workin | 21:47 |
| clarkb | I know the old image had the unworking ssh keys setup so we don't want toreuse them | 21:47 |
| rcarrillocruz | i have not, not sure if crinkle did | 21:48 |
| rcarrillocruz | don't think | 21:48 |
| *** jamielennox|away is now known as jamielennox | 21:48 | |
| *** cgalan has joined #openstack-infra | 21:48 | |
| rcarrillocruz | but yeah, we should build dibs | 21:48 |
| mordred | clarkb: I promise. if you ever go there, you will not be diappointed by the level of whitetrashness | 21:48 |
| rcarrillocruz | mordred: btw, i got pinged by dflanders about giving a talk (unsure if lightning or a longer one) about shade in EuroPython | 21:49 |
| rcarrillocruz | still waiting for getting approval to attend, will let you know if it goes thru | 21:49 |
| rcarrillocruz | Shrews: ^ | 21:49 |
| mordred | rcarrillocruz: woot. when is EuroPython? | 21:49 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Prefix package upload folder by /packaging-deb https://review.openstack.org/333045 | 21:49 |
| rcarrillocruz | next month | 21:49 |
| mordred | cool | 21:49 |
| *** sree has quit IRC | 21:49 | |
| rcarrillocruz | at Bilbao | 21:49 |
| rcarrillocruz | which means | 21:49 |
| rcarrillocruz | AMAZING FOOD | 21:49 |
| *** sree has joined #openstack-infra | 21:49 | |
| rcarrillocruz | /cc greghaynes | 21:49 |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Story api https://review.openstack.org/332946 | 21:50 |
| rcarrillocruz | btw | 21:50 |
| mordred | rcarrillocruz: zomg. the food. the food | 21:50 |
| rcarrillocruz | opnfv folks | 21:50 |
| rcarrillocruz | are willing to use puppet infracloud stuff | 21:50 |
| rcarrillocruz | and bifrost | 21:50 |
| rcarrillocruz | so i'd like to refactor it a bit at some point | 21:50 |
| mordred | rcarrillocruz: I look forward to working with our opnfv friends more | 21:51 |
| rcarrillocruz | remove hardcoded things etc | 21:51 |
| rcarrillocruz | yolanda will look into it too, since she's doing nfv stuff now | 21:51 |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Story api https://review.openstack.org/332946 | 21:51 |
| *** sree_ has joined #openstack-infra | 21:52 | |
| *** sree_ is now known as Guest25571 | 21:52 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Story api https://review.openstack.org/332946 | 21:54 |
| *** sree has quit IRC | 21:54 | |
| *** sree has joined #openstack-infra | 21:55 | |
| *** Guest25571 has quit IRC | 21:56 | |
| *** tqtran has quit IRC | 21:56 | |
| *** cgalan has left #openstack-infra | 21:57 | |
| *** maishsk has quit IRC | 21:58 | |
| *** esberglu has quit IRC | 21:58 | |
| *** eezhova has joined #openstack-infra | 21:59 | |
| *** eezhova has quit IRC | 21:59 | |
| *** sree has quit IRC | 21:59 | |
| *** weshay has quit IRC | 22:00 | |
| *** sree has joined #openstack-infra | 22:00 | |
| *** eezhova has joined #openstack-infra | 22:00 | |
| *** Sukhdev has joined #openstack-infra | 22:00 | |
| *** maishsk has joined #openstack-infra | 22:00 | |
| openstackgerrit | Merged openstack-infra/nodepool: Add comment field to node and reason to hold https://review.openstack.org/225428 | 22:01 |
| *** gouthamr has joined #openstack-infra | 22:01 | |
| openstackgerrit | Merged openstack-infra/nodepool: Test ZMQ event handler https://review.openstack.org/230023 | 22:01 |
| mordred | woot! | 22:01 |
| openstackgerrit | Merged openstack-infra/nodepool: Add auto-hold feature https://review.openstack.org/332976 | 22:02 |
| mordred | jeblair: well done. I think auto-hold is gonna be quite nice | 22:02 |
| *** eezhova has quit IRC | 22:02 | |
| jeblair | mordred: thanks. so will comments on hold :) | 22:02 |
| *** eezhova has joined #openstack-infra | 22:02 | |
| jeblair | i'm doing the alter table for that now | 22:02 |
| dmsimard | pabelanger: do you have the link to the chrome telnet etherpad handy ? | 22:02 |
| greghaynes | rcarrillocruz: haha, I don't think i'll be there this time :( | 22:03 |
| jeblair | though, i think that's running into the typical inability to grab a whole table lock | 22:03 |
| greghaynes | rcarrillocruz: but that is where I have had the best steak of my life | 22:03 |
| *** signed8bit is now known as signed8bit_Zzz | 22:03 | |
| jeblair | so we may need to do that during the restart | 22:03 |
| *** eezhova has quit IRC | 22:03 | |
| *** sree has quit IRC | 22:04 | |
| nibalizer | mtreinish: lint 2.0 released, all hail | 22:04 |
| nibalizer | (an mich cleanup patch) | 22:04 |
| *** sree has joined #openstack-infra | 22:04 | |
| *** Sukhdev has quit IRC | 22:05 | |
| *** inc0 has quit IRC | 22:05 | |
| openstackgerrit | Merged openstack-infra/nodepool: Fix race condition in disconnect test https://review.openstack.org/333028 | 22:06 |
| mtreinish | nibalizer: sigh, doesn't help my cursed patch though :) | 22:07 |
| jeblair | | ID | Name | Hold on Failure | | 22:08 |
| jeblair | | 1 | gate-puppet-openstack-integration-3-scenario001-tempest-ubuntu-xenial | 3 | | 22:08 |
| jeblair | | 2 | gate-openstack-ansible-rsyslog_server-ansible-func-centos-7-nv | 3 | | 22:08 |
| jeblair | pabelanger, mordred: ^ | 22:08 |
| jeblair | i restarted nodepool with those changes | 22:08 |
| mordred | jeblair: sweet | 22:08 |
| mordred | EmilienM: ^^ | 22:09 |
| jeblair | #status log nodepool now support adding a reason when holding a node "--reason <foo>" please use that so that we can remember why they are held :) | 22:09 |
| openstackstatus | jeblair: finished logging | 22:09 |
| *** sree has quit IRC | 22:09 | |
| jeblair | #status log nodepool also supports auto-holding nodes for specific failed jobs (it will set the reason appropriately) | 22:09 |
| openstackstatus | jeblair: finished logging | 22:09 |
| jeblair | infra-root: ^ | 22:09 |
| fungi | jeblair: awesome addition--thanks! | 22:10 |
| fungi | always wanted to be able to track why i held something | 22:10 |
| fungi | even if people just put their names there so we know to come hunt them down | 22:11 |
| nibalizer | mtreinish: fir that id just fix the lint errors | 22:11 |
| nibalizer | probably my fix added mire lint checks that you didnt used to have to foloow | 22:11 |
| *** ddieterly is now known as ddieterly[away] | 22:11 | |
| *** sree has joined #openstack-infra | 22:11 | |
| *** rlandy has quit IRC | 22:12 | |
| mtreinish | nibalizer: sure, I realize that. I already start the patch to do that. It's just another thing | 22:12 |
| nibalizer | yep | 22:12 |
| nibalizer | sorry i cant do it for you | 22:12 |
| nibalizer | my laptop crapped out | 22:12 |
| jeblair | | 1904057 | internap-nyj01 | None | ubuntu-xenial | zuul | zl02 | ubuntu-xenial-internap-nyj01-1904057 | ubuntu-xenial-internap-nyj01-1904057 | 511a6896-6acb-4e9d-b398-7555f0fc2578 | 63.251.114.208 | hold | 00:00:05:08 | Automatically held after failing gate-puppet-openstack-integration-3-scenario001-tempest-ubuntu-xenial | | 22:14 |
| jeblair | EmilienM, pabelanger, mordred: ^ | 22:14 |
| jeblair | that was fast | 22:14 |
| *** acoles is now known as acoles_ | 22:14 | |
| *** Sukhdev has joined #openstack-infra | 22:15 | |
| mordred | jeblair: wow | 22:15 |
| jeblair | http://logs.openstack.org/49/333049/1/check/gate-puppet-openstack-integration-3-scenario001-tempest-ubuntu-xenial/cb84fe9/ | 22:15 |
| *** sree has quit IRC | 22:16 | |
| clarkb | one possibility is if ansible is just using time.sleep() and otherwise waiting for a signal to wake it sleep isn't garunteed to sleep the complete time specified since any caught signal will cause it to return | 22:17 |
| EmilienM | I'm currently afk (on my phone), I'll come back later. Sorry | 22:17 |
| *** dangers_` is now known as dangers_away | 22:17 | |
| jeblair | it does look like the timeout issue hit that job | 22:18 |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Story api https://review.openstack.org/332946 | 22:18 |
| openstackgerrit | Merged openstack-infra/project-config: Prefix package upload folder by /packaging-deb https://review.openstack.org/333045 | 22:18 |
| *** sree has joined #openstack-infra | 22:18 | |
| jeblair | ran for 12 minutes then kaput | 22:19 |
| *** mdrabe has quit IRC | 22:19 | |
| *** sdake has joined #openstack-infra | 22:21 | |
| *** roxanaghe has quit IRC | 22:21 | |
| jeblair | Jun 22 22:08:08 ubuntu ansible-async_wrapper: 1635 still running (2840) | 22:22 |
| *** roxanaghe has joined #openstack-infra | 22:22 | |
| jeblair | the async wrapper continued to print that it was still running until it apparently normally terminated | 22:22 |
| *** sree has quit IRC | 22:22 | |
| *** sree_ has joined #openstack-infra | 22:22 | |
| *** sree_ is now known as Guest83069 | 22:23 | |
| *** sree has joined #openstack-infra | 22:23 | |
| *** sdake_ has quit IRC | 22:24 | |
| jeblair | according to syslog, a bunch of neutron stuff is happening around the time ansible declared the async timeout | 22:25 |
| *** dimtruck is now known as zz_dimtruck | 22:25 | |
| jeblair | i wonder if it could have impacted connectivity... but as usual, i also wonder what impact controlmaster has on this | 22:26 |
| *** Guest83069 has quit IRC | 22:27 | |
| *** eezhova has joined #openstack-infra | 22:27 | |
| *** sree has quit IRC | 22:28 | |
| *** achanda_ has joined #openstack-infra | 22:29 | |
| *** jcoufal has quit IRC | 22:29 | |
| *** sdake_ has joined #openstack-infra | 22:31 | |
| jeblair | according to sshd on the worker, the connection ansible made from zl to the worker terminated exactly 61 seconds after the async timeout was declared. ansible sets controlmaster to stick around for 60 seconds, which makes me think that the ssh connection that ansible was using stayed open for the entire time it was supposed to (and beyond the timeout) | 22:32 |
| *** eezhova has quit IRC | 22:33 | |
| *** sdake has quit IRC | 22:34 | |
| *** pvaneck has quit IRC | 22:34 | |
| *** ddieterly[away] is now known as ddieterly | 22:34 | |
| *** xyang1_ has quit IRC | 22:37 | |
| jeblair | the async output file for that task is present and says "changed": true, "rc": 0 | 22:39 |
| *** akshai has quit IRC | 22:40 | |
| *** akshai has joined #openstack-infra | 22:42 | |
| jeblair | /home/jenkins/.ansible/tmp/ansible-tmp-1466633286.77-167534110842955/async_status is the status check that ran 1 second before the timeout | 22:43 |
| jeblair | pabelanger, Shrews: so yeah, i think we might be at a point where we may need to run with -vvv, assuming that will give us the raw output of the async_status pseudo-task | 22:44 |
| *** jamesmcarthur has quit IRC | 22:44 | |
| *** jamielennox is now known as jamielennox|away | 22:44 | |
| clarkb | jeblair: ya at the very least https://github.com/ansible/ansible-modules-core/blob/a1e6311197a7b887adaf60725b01562883b8e72f/utilities/logic/async_wrapper.py#L182-L183 is buggy I think, not sure if causing the problem here though | 22:46 |
| *** ffrank has quit IRC | 22:46 | |
| clarkb | jeblair: basically that code assumes step was the period slept for when it may not have been, it needs to actually check wall time and substract the difference between then and when it slept rather than rely on step | 22:47 |
| clarkb | the code would be something like last_sleep = time.time() ; time.sleep(step) ; now = time.time() ; remaining = remaining - (now - last_sleep)) | 22:48 |
| clarkb | (also it takes a long time to find that code ya subrepos) | 22:49 |
| *** sdake_ has quit IRC | 22:49 | |
| clarkb | also I think the timeout may always be off by up to one step since there is a sleep(step) before the while loop? | 22:50 |
| *** jamesmcarthur has joined #openstack-infra | 22:51 | |
| *** eezhova has joined #openstack-infra | 22:51 | |
| *** tphummel has quit IRC | 22:51 | |
| *** eezhova has quit IRC | 22:51 | |
| *** jamielennox|away is now known as jamielennox | 22:53 | |
| *** derekh has quit IRC | 22:54 | |
| *** tqtran has joined #openstack-infra | 22:54 | |
| *** yamahata has quit IRC | 22:54 | |
| *** vhosakot has quit IRC | 22:54 | |
| *** matt-borland has quit IRC | 22:54 | |
| jeblair | clarkb: hrm, but that's what runs on the host, which *appears* to be functioning | 22:55 |
| jeblair | clarkb: at least, it got all the way down to "Done in kid B" long after the 'timeout' happened | 22:55 |
| clarkb | jeblair: that code appears to be what determines if the timeout has expired by decrementing remaining | 22:55 |
| jeblair | clarkb: i agree that it suffers from the bug you describe | 22:55 |
| jeblair | clarkb: but i don't think that's what's happened here, since "Done in kid B" happened like 20 minutes after our erroneous timeout | 22:56 |
| *** ddieterly is now known as ddieterly[away] | 22:57 | |
| clarkb | jeblair: I am looking at the older logs pabelanger provided earlier today http://logs.openstack.org/98/330498/10/check/gate-puppet-openstack-integration-3-scenario001-tempest-ubuntu-xenial/e49bcf6/_zuul_ansible/logs/ansible.txt and at least there it says I have >3k seconds then times out after 15 minutes | 22:57 |
| *** gfreeman has quit IRC | 22:57 | |
| clarkb | 15 minutes == 900 seconds (I had to maths this to make sure I wasn't reading it wrong | 22:58 |
| jeblair | clarkb: right, there are two components here -- there's an async wrapper which runs on the worker and is the parent of the process we actuall want to run. that's the source code you are pointing to | 22:58 |
| *** salv-orl_ has joined #openstack-infra | 22:58 | |
| *** tqtran has quit IRC | 22:58 | |
| jeblair | clarkb: then there's ansible running on the launcher. it periodically checks in on the async_wrapper process to see if it's still running or if it finished | 22:58 |
| clarkb | right the second bit is what the poll parameter configures aiui. defaults to polling every 5 seconds | 22:59 |
| jeblair | clarkb: from examining the logs on the auto-held worker, i believe the async_wrapper process is not timing out -- it's running to normal completion | 22:59 |
| *** gouthamr has quit IRC | 22:59 | |
| jhesketh | Morning | 22:59 |
| jeblair | clarkb: but the poller is miscommunicating with it at some point and *thinks* it timed out | 22:59 |
| clarkb | jeblair: but the launcher is sayin it timed out? | 22:59 |
| clarkb | jeblair: gotcha | 22:59 |
| *** rcernin has quit IRC | 23:00 | |
| jeblair | clarkb: you should totally send in that patch though :) | 23:00 |
| mordred | clarkb: let me konw if you do and I'll +2 it | 23:00 |
| jeblair | it's a good catch | 23:00 |
| *** salv-orlando has quit IRC | 23:00 | |
| *** Sukhdev has quit IRC | 23:01 | |
| jeblair | (could make a big difference if someone sets a large poll interval) | 23:01 |
| clarkb | or a really small async timeout | 23:01 |
| jeblair | ya | 23:01 |
| pabelanger | jeblair: oh, nice. Looking in a few minutes | 23:03 |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Story api https://review.openstack.org/332946 | 23:03 |
| pabelanger | dmsimard: https://etherpad.openstack.org/p/google-telnet https://etherpad.openstack.org/p/firefox-telnet | 23:04 |
| *** PollyZ has quit IRC | 23:04 | |
| *** rbrndt has quit IRC | 23:04 | |
| pabelanger | jeblair: I believe syslog on the held nodes will provide some info too. That is what I plan on checking. | 23:04 |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Story api https://review.openstack.org/332946 | 23:05 |
| *** jamielennox is now known as jamielennox|away | 23:05 | |
| *** stevemar has quit IRC | 23:06 | |
| *** salv-orl_ has quit IRC | 23:06 | |
| jeblair | pabelanger: i've been reading syslog, authlog, and poking at various .ansible directories... you can find some of my analysis in scrollback | 23:06 |
| pabelanger | and caught up on backscroll | 23:06 |
| *** stevemar has joined #openstack-infra | 23:06 | |
| pabelanger | jeblair: ack, and thanks for the info | 23:07 |
| *** hongbin has quit IRC | 23:07 | |
| anteaya | morning jhesketh | 23:08 |
| pleia2 | pabelanger: as I was going through the etherpad earlier today I was thinking we probably want to nudge some of the key players for infra-cloud and zuul things directly to see about submitting talks | 23:08 |
| *** bhavik has quit IRC | 23:10 | |
| *** sree has joined #openstack-infra | 23:10 | |
| *** zhenguo_ has quit IRC | 23:11 | |
| *** stevemar has quit IRC | 23:11 | |
| *** gyee has quit IRC | 23:11 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: add verbose command https://review.openstack.org/333059 | 23:13 |
| jeblair | pabelanger, mordred, clarkb: ^ | 23:13 |
| *** jamesmcarthur has quit IRC | 23:14 | |
| *** edmondsw has quit IRC | 23:15 | |
| *** sree has quit IRC | 23:15 | |
| jeblair | i'd like to land that now and run the restart playbook overnight so we can use it tomorrow | 23:15 |
| mordred | lgtm | 23:15 |
| jeblair | also, because i didn't reset my repo, it's stacked on this silly little change: https://review.openstack.org/332263 | 23:16 |
| jeblair | but i would like to land that too :) | 23:17 |
| *** pradk has quit IRC | 23:17 | |
| *** gouthamr has joined #openstack-infra | 23:17 | |
| *** amitgandhinz has joined #openstack-infra | 23:17 | |
| *** jamesmcarthur has joined #openstack-infra | 23:18 | |
| calebb | mordred: any idea when you're going to release a new version of shade? | 23:20 |
| * jeblair watches mordred shake the 8 ball | 23:20 | |
| calebb | haha | 23:20 |
| mordred | calebb: oh yeah- there's a bunch of stuff in there | 23:21 |
| pabelanger | pleia2: agreed. I have calls out to both groups to look at our etherpads | 23:21 |
| pabelanger | pleia2: I am trying to figure out who suggested diskimage-builder | 23:22 |
| mordred | calebb: I'll poke Shrews in the morning and see how he feels - but I think I could cut one tomorrow | 23:22 |
| pleia2 | pabelanger: yeah, not sure about dib | 23:22 |
| pabelanger | jeblair: +2 for verbose! | 23:23 |
| *** amitgandhinz has quit IRC | 23:23 | |
| *** sdague has quit IRC | 23:23 | |
| calebb | mordred: heh, ok, I'm mostly asking because there's some docs in the firstapp for shade that use some features that havent been released yet, but Im not sure how much hurry there is to post those docs | 23:23 |
| calebb | I'll keep an eye out though, thanks | 23:23 |
| *** sdake has joined #openstack-infra | 23:24 | |
| *** jamesmcarthur has quit IRC | 23:24 | |
| *** jamesmcarthur has joined #openstack-infra | 23:25 | |
| jeblair | pabelanger: are you okay with https://review.openstack.org/332263 ? | 23:26 |
| *** gouthamr_ has joined #openstack-infra | 23:27 | |
| *** dfflanders has joined #openstack-infra | 23:28 | |
| *** jamesmcarthur has quit IRC | 23:29 | |
| jeblair | pabelanger: revert it if you are not. :) | 23:30 |
| jeblair | oh heh, there's a vote :) | 23:30 |
| pabelanger | jeblair: I don't object enough to make a stand :) | 23:30 |
| *** gouthamr has quit IRC | 23:30 | |
| jeblair | pabelanger: aw, i was hoping you'd like it. i've been doing a lot of extra clicking to open a log and then compare it to a playbook, etc. | 23:32 |
| jeblair | figured it'd be nice to have them closer together | 23:32 |
| *** sree has joined #openstack-infra | 23:33 | |
| pabelanger | jeblair: agreed, having them in the same folder is nice. | 23:34 |
| *** jamesmcarthur has joined #openstack-infra | 23:34 | |
| openstackgerrit | Merged openstack-infra/zuul: Ansible launcher: move ansible log to ansible root https://review.openstack.org/332263 | 23:35 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add a node for artifact signing jobs https://review.openstack.org/333065 | 23:35 |
| fungi | OKAY! that's ^ behind me | 23:35 |
| openstackgerrit | Merged openstack-infra/zuul: Ansible launcher: add verbose command https://review.openstack.org/333059 | 23:35 |
| pleia2 | fungi: I'm so excited! | 23:36 |
| pabelanger | jeblair: only outstanding thing I have for v25 is 331499 but we can defer that until we address our async issue | 23:36 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-openstack_health: Fix puppet-lint issues caused by new release https://review.openstack.org/333066 | 23:36 |
| fungi | a lot more heads-down time than i anticipated | 23:36 |
| mtreinish | nibalizer: ^^^ sry, got distracted | 23:36 |
| *** kzaitsev_mb has quit IRC | 23:36 | |
| pabelanger | fungi: wow, nice work! | 23:36 |
| *** zhurong has joined #openstack-infra | 23:36 | |
| clarkb | mordred: you should totally test this https://github.com/ansible/ansible-modules-core/pull/4017 | 23:36 |
| mtreinish | nibalizer: I'm just assuming that's right. I actually couldn't reproduce the warning from the gate locally | 23:36 |
| *** esikachev has joined #openstack-infra | 23:36 | |
| clarkb | mordred: and/or help write proper tests for it? I dunno what I am doing but I think the commit message and diff itself capture the goals well | 23:37 |
| fungi | most spent testing out how many files i could get by with stashing in hiera to ferry the subkey and what gpg would create on its own when missing | 23:37 |
| *** sree has quit IRC | 23:37 | |
| *** sree has joined #openstack-infra | 23:37 | |
| jeblair | pabelanger: yeah, let's defer that until later to avoid changing unecessary variables | 23:37 |
| nibalizer | mtreinish: haha valiant effort! | 23:38 |
| nibalizer | not correct though, ill see what i can do hold on | 23:38 |
| nibalizer | mind if i update your patch? | 23:38 |
| mtreinish | nibalizer: go for it | 23:39 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Configure its-storyboard plugin https://review.openstack.org/330925 | 23:39 |
| *** bhavik has joined #openstack-infra | 23:39 | |
| *** thorst_ has quit IRC | 23:41 | |
| *** sree has quit IRC | 23:41 | |
| *** thorst has joined #openstack-infra | 23:42 | |
| *** sdake has quit IRC | 23:42 | |
| mtreinish | nibalizer: wait, adding a double colon in front of everything doesn't magically make everything work? I thought that's how puppet worked :) | 23:43 |
| nibalizer | mtreinish: a common misconception! | 23:44 |
| *** esikachev has quit IRC | 23:45 | |
| *** sree has joined #openstack-infra | 23:45 | |
| *** baoli has joined #openstack-infra | 23:47 | |
| *** tpsilva has quit IRC | 23:47 | |
| *** ddieterly has joined #openstack-infra | 23:48 | |
| fungi | you're thinking of triple-colon | 23:49 |
| *** gongysh has joined #openstack-infra | 23:50 | |
| *** sree has quit IRC | 23:50 | |
| mtreinish | fungi: lol | 23:51 |
| *** thorst has quit IRC | 23:51 | |
| *** sree has joined #openstack-infra | 23:51 | |
| ianw | prometheanfire: i don't know why, but the gentoo dib element seems to have started to try to build gcc (and fail) -> http://logs.openstack.org/66/332866/1/check/gate-dib-dsvm-functests-centos-7-extras-nv/9375d7c/console.html | 23:51 |
| fungi | pabelanger: your comments make me think we've gotten further in unjenkinsifying our job nodes than i realized | 23:52 |
| fungi | pabelanger: are we creating "zuul" accounts on them with puppet now, and does zuul-launcher know to use those? | 23:52 |
| fungi | if so, i'm happy to accommodate | 23:52 |
| *** baoli has quit IRC | 23:54 | |
| *** armax has quit IRC | 23:54 | |
| *** sree has quit IRC | 23:55 | |
| *** erlon has quit IRC | 23:56 | |
| *** yamahata has joined #openstack-infra | 23:57 | |
| *** jamielennox|away is now known as jamielennox | 23:57 | |
| *** baoli has joined #openstack-infra | 23:57 | |
| *** baoli has quit IRC | 23:58 | |
| *** stevemar has joined #openstack-infra | 23:59 | |
| *** zhurong has quit IRC | 23:59 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Configure its-storyboard plugin https://review.openstack.org/330925 | 23:59 |
| *** tqtran has joined #openstack-infra | 23:59 | |
| *** lmiccini has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!