| *** kzaitsev_mb has quit IRC | 00:01 | |
| *** john-davidge has joined #openstack-infra | 00:02 | |
| jeblair | let's all go to the lobby... while gerrit grabs a snack | 00:03 |
|---|---|---|
| mordred | I really need to look in to that | 00:03 |
| mordred | it makes no sense to me | 00:03 |
| *** jtomasek has quit IRC | 00:04 | |
| clarkb | I actually did peruse it a bit and if we wanted to easy mode I think we can have a slave with trove | 00:04 |
| clarkb | and pull backups off that | 00:04 |
| *** askb has quit IRC | 00:05 | |
| *** erikmwilson has quit IRC | 00:05 | |
| clarkb | or create a one off replica once a day (though thats likely to have a high cost) | 00:06 |
| *** Sukhdev has joined #openstack-infra | 00:06 | |
| *** john-davidge has quit IRC | 00:07 | |
| *** erikmwilson has joined #openstack-infra | 00:08 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add zuulv3-dev.openstack.org server https://review.openstack.org/430418 | 00:08 |
| pabelanger | clarkb: updated, also addressed your comment^ | 00:09 |
| *** rlandy has quit IRC | 00:11 | |
| *** Sukhdev has quit IRC | 00:11 | |
| *** bswartz has quit IRC | 00:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add action plugins to restrict untrusted execution https://review.openstack.org/428798 | 00:12 |
| *** askb has joined #openstack-infra | 00:13 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add __repr__ to jobtree https://review.openstack.org/430493 | 00:15 |
| *** andreww has quit IRC | 00:15 | |
| *** mtanino has quit IRC | 00:16 | |
| *** wolverineav has joined #openstack-infra | 00:17 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Correct playbook requirement logic https://review.openstack.org/434564 | 00:17 |
| *** sdague has quit IRC | 00:18 | |
| *** thorst_ has joined #openstack-infra | 00:19 | |
| *** thorst_ has quit IRC | 00:19 | |
| *** Sukhdev has joined #openstack-infra | 00:20 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-mysql_backup master: Use --single-transaction for mysqldump https://review.openstack.org/434566 | 00:21 |
| mordred | clarkb, jeblair: ^^ I got tired of gerrit pausing once a day, since there is literally no reason for such a thing to occur | 00:21 |
| *** wolverineav has quit IRC | 00:21 | |
| NobodyCam | Good after noon Infra folk | 00:22 |
| clarkb | mordred: re DDL, is that also true of current situation with bigly locks? | 00:22 |
| clarkb | NobodyCam: hello there | 00:22 |
| NobodyCam | off the wall question: is there a good reference I can point someone to to learn about the best way to keep a local repo in sync with OO.org? | 00:22 |
| jeblair | mordred: into the future! | 00:22 |
| mordred | clarkb: yes | 00:24 |
| clarkb | NobodyCam: if its your local dev repo on your laptop, just run `git remote update` when you sit down to do new work | 00:24 |
| NobodyCam | hey there clarkb and mordred | 00:24 |
| mordred | hey NobodyCam ! | 00:24 |
| NobodyCam | :) | 00:24 |
| clarkb | NobodyCam: can you maybe expand on the use case/problem you are trying to solve a bit more? | 00:24 |
| NobodyCam | clarkb: its a local gerrit | 00:25 |
| clarkb | mordred: jeblair and I +2'd mysql backup change I did not approve it because it would be good if you could ababysit it | 00:25 |
| mordred | clarkb: ++ | 00:25 |
| *** esikachev has joined #openstack-infra | 00:25 | |
| clarkb | NobodyCam: so you have a local gerrit with a fork of $openstackgerrit repo and want to keep it up to date? | 00:25 |
| mordred | clarkb: remind me tomorrow before the 00:00 backup | 00:25 |
| clarkb | mordred: I shall try | 00:25 |
| NobodyCam | I have a internal gerrit/jenkins/zuul setup and I would like to point the person running it to a best practices type doc | 00:25 |
| NobodyCam | clarkb: yep :) | 00:26 |
| clarkb | NobodyCam: I don't know that ther is a best practices doc. BUT jeepyb does support doing this for you | 00:26 |
| clarkb | NobodyCam: currently the manage-projects command can sycn upstreams when it runs (but mordred is working to make it its own command) | 00:26 |
| mordred | yah - hopefully that patch will land real soon | 00:27 |
| clarkb | NobodyCam: so if you wait a bit there may be a command called `track-upstream` that you get out of jeepyb | 00:27 |
| NobodyCam | awesome! | 00:27 |
| mordred | then the command will be called "track-upstream" | 00:27 |
| *** sabayonuser has quit IRC | 00:27 | |
| mordred | that will put branches from upstream into branches in the local gerrit prefixed with upstream/ | 00:27 |
| NobodyCam | can that be adjusted to openstack/ ?? | 00:28 |
| clarkb | mordred: so I just raelized a potential problem with the cache stuff and ^ | 00:28 |
| mordred | I think | 00:28 |
| clarkb | mordred: if you weren't already using manage-projects then th ecache would be wrong and first run we'd force push to repo? | 00:28 |
| mordred | NobodyCam: I don't remember - I'd need to look | 00:28 |
| clarkb | maybe thats fixed by splitting track-upstream out to its own tool | 00:28 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Move ProviderManagers into main NodePool thread https://review.openstack.org/434567 | 00:28 |
| mordred | clarkb: yah - I think it is | 00:28 |
| *** adrian_otto has quit IRC | 00:30 | |
| *** gongysh has joined #openstack-infra | 00:30 | |
| *** esikachev has quit IRC | 00:30 | |
| ianw | clarkb: after a *lot* of fiddling I think i've come up with a small python script that can basically insert the coverage output as an attachment into the testrepository -> http://paste.openstack.org/show/599129/. run that and | testr load | 00:30 |
| ianw | clarkb: my next question is ... will that make it through subunit2sql and into the db so i can query it | 00:31 |
| clarkb | ianw: it should make it through if you inject it into the normal per job run subunit file | 00:31 |
| *** chlong has joined #openstack-infra | 00:32 | |
| ianw | clarkb: so you mean it's just one of the entries in .testrepository ? | 00:32 |
| clarkb | mordred: thinking more about backups change if we really want to be paranoid we could apply it to etherpad first (since that also locks up for a few minutes) and check it separately then apply everywhere | 00:32 |
| clarkb | mordred: since etherpad has "ether" in its name we probably care less about impacting it :) | 00:33 |
| clarkb | ianw: no I think it has to be appended to .testrepository/0 | 00:33 |
| *** hongbin has quit IRC | 00:33 | |
| *** Swami_ has joined #openstack-infra | 00:33 | |
| clarkb | ianw: but I'd have to double check the publisher macros stuff | 00:33 |
| *** aeng has quit IRC | 00:33 | |
| *** Goneri has quit IRC | 00:35 | |
| *** spzala has joined #openstack-infra | 00:35 | |
| clarkb | ianw: https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/logstash/jenkins-log-client.yaml#n14 is current config for subunit things | 00:36 |
| clarkb | so its looking for the testrepository.subunit files in most jobs | 00:36 |
| *** wolverineav has joined #openstack-infra | 00:37 | |
| clarkb | ianw: and for unittest/coverage we run $bin_path/testr last --subunit > ./testrepository.subunit to generate that file in project-config/jenkins/scripts/run-tox.sh | 00:37 |
| *** mriedem has joined #openstack-infra | 00:37 | |
| clarkb | so we'd need to stitch together the unittest run output and the coverage data output | 00:38 |
| ianw | clarkb: ahhh, ok ... very helpful, thanks | 00:38 |
| *** Swami_ has quit IRC | 00:38 | |
| ianw | i wonder if you can just cat subunit files together | 00:39 |
| mordred | clarkb: want me to do a quick check of that? | 00:39 |
| clarkb | ianw: not sure but you can probably use testr load to find out | 00:40 |
| clarkb | mordred: sure | 00:40 |
| *** spzala has quit IRC | 00:40 | |
| clarkb | mordred: we can see if https://etherpad.openstack.org/p/clarkb-test stop loading or not | 00:40 |
| mordred | clarkb: ok -I'm about to pull the trigger - ready? | 00:41 |
| clarkb | yup | 00:41 |
| clarkb | say when | 00:41 |
| mordred | it's running | 00:41 |
| ianw | clarkb: i think you can http://paste.openstack.org/show/599131/ | 00:41 |
| *** Julien-zte has joined #openstack-infra | 00:41 | |
| clarkb | mordred: seems happy | 00:41 |
| mordred | clarkb: yup | 00:41 |
| *** wolverineav has quit IRC | 00:41 | |
| mordred | and the backup is still running | 00:41 |
| clarkb | ianw: neat! | 00:42 |
| clarkb | mordred: yay | 00:42 |
| openstackgerrit | Merged openstack-infra/zuul master: Add support for sqlalchemy reporter https://review.openstack.org/223333 | 00:42 |
| mordred | clarkb: I apologize that it took me $years to make that patch | 00:42 |
| jeblair | jhesketh, mtreinish: ^ | 00:42 |
| clarkb | mordred: eh well | 00:42 |
| mordred | \o/ | 00:42 |
| mordred | jeblair, mtreinish, jhesketh: yay! | 00:42 |
| ianw | clarkb: so i'm starting to think adding "if [ -f cover/coverage.xml ]; then $(python to blag coverage.xml as a subunit output) >> testrepository.subunit" and maybe it "just works" ... | 00:43 |
| clarkb | ianw: sounds good | 00:43 |
| clarkb | its all non voting jobs now anyways | 00:43 |
| clarkb | so low on danger scale | 00:43 |
| ianw | and with a change to pbr's "--coverage" argument to auto create the .xml file, it should also "just work" | 00:43 |
| mtreinish | jeblair: ooh | 00:45 |
| *** Sukhdev has quit IRC | 00:45 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Re-enable devstack test job https://review.openstack.org/431649 | 00:45 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add lock state to node listing https://review.openstack.org/434397 | 00:45 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Implement node cleanup https://review.openstack.org/433736 | 00:45 |
| mtreinish | jeblair: I guess that means I'll have to start looking at how to integrate that data into openstack-health :) | 00:45 |
| jeblair | mtreinish: ++ | 00:46 |
| mtreinish | ianw: yes, you can just append subunit files together | 00:46 |
| *** thorst_ has joined #openstack-infra | 00:46 | |
| *** thorst_ has quit IRC | 00:46 | |
| NobodyCam | Thank you @everyone for the help!! | 00:47 |
| jeblair | zomg. "Project Management Combat Operations" | 00:47 |
| ianw | mtreinish: that's quite cool. so basically if i have the coverage.xml as an artifact of "fake" test reported in testrepository.subunit, you don't see any reason why we won't be able to pull it out of sql? | 00:47 |
| mtreinish | ianw: that's how we deal with devstack's subunit output in dsvm tempest runs. The 2 subunit streams are just combined | 00:47 |
| *** vhosakot has quit IRC | 00:47 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Re-enable devstack test job https://review.openstack.org/431649 | 00:48 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add lock state to node listing https://review.openstack.org/434397 | 00:48 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Implement node cleanup https://review.openstack.org/433736 | 00:48 |
| mtreinish | ianw: well it'll look a little confusing on openstack-health, but that should work | 00:48 |
| NobodyCam | Project Management Combat Operations - Two Projects enter one project leaves! | 00:48 |
| mtreinish | ianw: fwiw, I think masayukig_ was working on aggergating the coverage data at one point | 00:48 |
| ianw | yeah, i have dejavu about that, i think git blame might have me touching that at some point :) | 00:48 |
| ianw | mtreinish: that's true. if it's a constant fake test name, we could just filter it out for clarity? | 00:48 |
| mtreinish | ianw: http://git.openstack.org/cgit/openstack/coverage2sql/ | 00:49 |
| mtreinish | ianw: yeah we can do that, it's just something we'd have to add to openstack-health | 00:49 |
| *** masayukig_ is now known as masayukig | 00:49 | |
| masayukig | ianw , mtreinish : I'm sorry for the slow progress for that ... :-p | 00:51 |
| *** aeng has joined #openstack-infra | 00:52 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Exclude /var/log/journal from devstack-logs https://review.openstack.org/434467 | 00:52 |
| clarkb | we are up to 55GB ish free on logs volume | 00:52 |
| clarkb | so we are winnning the race now | 00:52 |
| *** wolverineav has joined #openstack-infra | 00:53 | |
| openstackgerrit | Hiroaki Kobayashi proposed openstack-infra/project-config master: Add release notes jobs for blazar https://review.openstack.org/431274 | 00:53 |
| *** wolverin_ has joined #openstack-infra | 00:53 | |
| masayukig | mtreinish, ianw: I'll make a progress for integration of coverage2sql into openstack-health during the PTG.. | 00:55 |
| ianw | masayukig: thanks for working on it. i think having the xml result available will be a quick way to just show a overall up/down graph without too much integration. i think coverage2sql would be good for a much deeper view of what's going on, especially drilling down by file | 00:55 |
| *** esikachev has joined #openstack-infra | 00:56 | |
| masayukig | ianw : oh, the xml can store the history of coverages? | 00:56 |
| *** dave-mccowan has joined #openstack-infra | 00:57 | |
| *** wolverineav has quit IRC | 00:57 | |
| *** wolverin_ has quit IRC | 00:57 | |
| ianw | masayukig: not exactly ... my thought at the moment is to attach the XML coverage report just as another test to the existing subunit output put into the subunit2sql db | 00:59 |
| ianw | it should be quite easy to just pull the "overall" coverage number out of that (and link to the html report) | 01:00 |
| masayukig | ianw : ah, understood. yeah, it should be easy :) | 01:00 |
| masayukig | it's easier than coverage2sql :) | 01:00 |
| *** esikachev has quit IRC | 01:01 | |
| ianw | masayukig: i hope to have a review out this afternoon, i'll loop you in. really this just came out of me trying to get some standardisation in the coverage jobs. if we have this, and just have a rule that they're in the check queue (so you see if its working) and post queue (for sending to health for basic tracking) i think that's progress | 01:04 |
| ianw | because the logs seem to suggest that nobody is looking at the HTML reports | 01:04 |
| *** erikmwilson has quit IRC | 01:04 | |
| masayukig | ianw : ++ | 01:05 |
| masayukig | ianw: yeah, agree, only a few people care about that.. | 01:05 |
| *** erikmwilson has joined #openstack-infra | 01:06 | |
| mtreinish | ianw: fwiw, SpamapS was doing something similar with counting data. You don't actually need to make a fake test you can just add an attachment to the stream | 01:06 |
| mtreinish | s/counting/counters | 01:07 |
| masayukig | mtreinish : btw, I noticed that gate-elastic-recheck-linters-ubuntu-xenial jobs failing https://review.openstack.org/#/c/434519/ | 01:07 |
| mtreinish | it won't get dumped in the db, directly but you could easily process it independently in the subunit workers | 01:07 |
| mtreinish | ianw: I would ask him about it though, because I'm a bit hazy on the details | 01:08 |
| mtreinish | masayukig: tbh, I don't know what that job is, I didn't add it | 01:08 |
| *** chlong has quit IRC | 01:08 | |
| mtreinish | and it's failing because there is no linters tox job definition :) | 01:08 |
| masayukig | yeah, should we remove the job? | 01:08 |
| mtreinish | masayukig: yeah, lets just remove it | 01:09 |
| masayukig | mtreinish : sure :) | 01:09 |
| *** liujiong has joined #openstack-infra | 01:11 | |
| *** wolverineav has joined #openstack-infra | 01:12 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix missing mutex release when aborting builds https://review.openstack.org/432211 | 01:13 |
| *** esikachev has joined #openstack-infra | 01:14 | |
| *** claudiub has quit IRC | 01:15 | |
| *** fandi has joined #openstack-infra | 01:15 | |
| *** spzala has joined #openstack-infra | 01:16 | |
| *** sabayonuser has joined #openstack-infra | 01:17 | |
| *** wolverineav has quit IRC | 01:17 | |
| *** MasterOfBugs has quit IRC | 01:18 | |
| *** ZZelle_ has quit IRC | 01:20 | |
| *** esikachev has quit IRC | 01:21 | |
| *** spzala has quit IRC | 01:21 | |
| *** funzo_ has quit IRC | 01:21 | |
| *** dave-mccowan has quit IRC | 01:23 | |
| *** yanyanhu has joined #openstack-infra | 01:23 | |
| *** rbuzatu has joined #openstack-infra | 01:24 | |
| *** spzala has joined #openstack-infra | 01:26 | |
| *** thorst_ has joined #openstack-infra | 01:29 | |
| *** thorst_ has quit IRC | 01:32 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/project-config master: Remove linters job of elastic-recheck https://review.openstack.org/434573 | 01:35 |
| *** wolverineav has joined #openstack-infra | 01:36 | |
| *** fandi has quit IRC | 01:39 | |
| *** wolverineav has quit IRC | 01:41 | |
| *** sree has quit IRC | 01:42 | |
| *** sree has joined #openstack-infra | 01:42 | |
| *** wolverineav has joined #openstack-infra | 01:44 | |
| *** gyee has quit IRC | 01:46 | |
| *** sree has quit IRC | 01:46 | |
| *** salv-orlando has joined #openstack-infra | 01:47 | |
| *** jamesdenton has joined #openstack-infra | 01:50 | |
| *** salv-orlando has quit IRC | 01:52 | |
| *** Apoorva has quit IRC | 01:52 | |
| *** wolverineav has quit IRC | 01:53 | |
| *** esikachev has joined #openstack-infra | 01:54 | |
| *** jamesdenton has quit IRC | 01:54 | |
| *** dave-mccowan has joined #openstack-infra | 01:57 | |
| *** wolverineav has joined #openstack-infra | 01:58 | |
| *** jamesdenton has joined #openstack-infra | 01:58 | |
| *** rbuzatu has quit IRC | 01:58 | |
| *** esikachev has quit IRC | 01:59 | |
| *** tqtran has quit IRC | 01:59 | |
| *** john-davidge has joined #openstack-infra | 02:03 | |
| *** wolverineav has quit IRC | 02:03 | |
| *** esikachev has joined #openstack-infra | 02:05 | |
| *** wolverineav has joined #openstack-infra | 02:06 | |
| *** john-davidge has quit IRC | 02:07 | |
| *** esikachev has quit IRC | 02:09 | |
| *** gongysh has quit IRC | 02:09 | |
| *** eliqiao has quit IRC | 02:09 | |
| *** wolverineav has quit IRC | 02:10 | |
| *** rbuzatu has joined #openstack-infra | 02:15 | |
| *** pahuang has quit IRC | 02:15 | |
| *** yamahata has quit IRC | 02:16 | |
| *** esikachev has joined #openstack-infra | 02:16 | |
| *** thorst_ has joined #openstack-infra | 02:17 | |
| ianw | mtreinish: hmm, i thought an attachment was related to a job? | 02:18 |
| *** thorst_ has quit IRC | 02:18 | |
| *** rbuzatu has quit IRC | 02:19 | |
| *** Julien-z_ has joined #openstack-infra | 02:20 | |
| *** Julien-zte has quit IRC | 02:20 | |
| *** esikachev has quit IRC | 02:21 | |
| *** funzo has joined #openstack-infra | 02:22 | |
| *** wolverineav has joined #openstack-infra | 02:23 | |
| *** esikachev has joined #openstack-infra | 02:24 | |
| *** yamahata has joined #openstack-infra | 02:25 | |
| *** funzo has quit IRC | 02:26 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Remove test-environments/multinode_major_upgrade.yaml https://review.openstack.org/426375 | 02:31 |
| *** esikachev has quit IRC | 02:33 | |
| *** wolverineav has quit IRC | 02:35 | |
| *** spzala has quit IRC | 02:35 | |
| *** thorst_ has joined #openstack-infra | 02:38 | |
| *** dhinesh has quit IRC | 02:39 | |
| *** thorst_ has quit IRC | 02:40 | |
| *** thorst_ has joined #openstack-infra | 02:43 | |
| *** thorst_ has quit IRC | 02:43 | |
| *** xyang1 has joined #openstack-infra | 02:44 | |
| *** wolverineav has joined #openstack-infra | 02:45 | |
| *** thorst_ has joined #openstack-infra | 02:46 | |
| *** wolverineav has quit IRC | 02:50 | |
| *** thorst_ has quit IRC | 02:51 | |
| *** xyang1 has quit IRC | 02:52 | |
| *** amotoki has joined #openstack-infra | 02:53 | |
| *** ijw has quit IRC | 02:55 | |
| *** hongbin has joined #openstack-infra | 02:57 | |
| *** yamahata has quit IRC | 02:57 | |
| *** thorst_ has joined #openstack-infra | 03:01 | |
| *** thorst_ has quit IRC | 03:03 | |
| *** ramishra has joined #openstack-infra | 03:04 | |
| *** gongysh has joined #openstack-infra | 03:06 | |
| *** winggundamth has joined #openstack-infra | 03:07 | |
| *** mdrabe has joined #openstack-infra | 03:08 | |
| *** rbuzatu has joined #openstack-infra | 03:10 | |
| openstackgerrit | Xu Ao proposed openstack/diskimage-builder master: DOC:Delete a obsolete note in the doc of python-brickclient https://review.openstack.org/434600 | 03:12 |
| *** mdrabe has quit IRC | 03:13 | |
| *** rbuzatu has quit IRC | 03:14 | |
| *** wolverineav has joined #openstack-infra | 03:17 | |
| *** panda|zZ has quit IRC | 03:18 | |
| *** TuHV has joined #openstack-infra | 03:20 | |
| TuHV | Hi all, | 03:21 |
| TuHV | I would like to make a functional test on neutron-fwaas with sudo priviledge | 03:21 |
| *** funzo has joined #openstack-infra | 03:21 | |
| TuHV | I have tried with https://review.openstack.org/#/c/432183/, but it fails in gate | 03:22 |
| TuHV | in local, it runs fine | 03:22 |
| *** wolverineav has quit IRC | 03:22 | |
| TuHV | Can somebody take a look at this? | 03:22 |
| TuHV | here is the log: http://logs.openstack.org/83/432183/2/check/gate-neutron-fwaas-dsvm-functional/ddc7a5f/console.html | 03:23 |
| clarkb | sudo is likely disabled to force rootwrap/privsep to be used | 03:24 |
| clarkb | why do you need to chmod? | 03:24 |
| TuHV | clarkb, Is there file failed with chmod? | 03:25 |
| clarkb | it looks like that is the sudo that failed? | 03:25 |
| *** funzo has quit IRC | 03:26 | |
| TuHV | clarkb, is that https://review.openstack.org/#/c/432183/2/tools/deploy_rootwrap.sh? | 03:26 |
| clarkb | maybe thats after the fail hard to read on phobe | 03:26 |
| clarkb | in any case lack of sudo likely intentional | 03:26 |
| *** wolverineav has joined #openstack-infra | 03:27 | |
| TuHV | clarkb, log fail with ERROR: InvocationError: '/opt/stack/new/neutron-fwaas/tools/deploy_rootwrap.sh /opt/stack/new/neutron-fwaas /opt/stack/new/neutron-fwaas/.tox/dsvm-functional' but I run in local, it is fine | 03:28 |
| clarkb | probavly because sudo is enabled for your local user? | 03:29 |
| TuHV | clarkb, yes, I run with "sudo -E" | 03:29 |
| clarkb | right its not allowed at that point in a devstack gate job iirc | 03:29 |
| TuHV | clarkb, in case no sudo privileges in gate, how can we run functional tests that needs sudo privileges? | 03:30 |
| clarkb | its allowed during devstack run then disabled then post_test_hook runs and it fails | 03:30 |
| clarkb | TuHV: you have to do anything requiring sudo when it is allowed in the setup phase | 03:30 |
| *** wolverineav has quit IRC | 03:31 | |
| TuHV | clarkb, you mean I should edit in post_test_hook.sh? | 03:31 |
| clarkb | no, sudo is not allowed in the post test hook | 03:32 |
| clarkb | you need to perform anything that requires sudo beforehand | 03:32 |
| TuHV | clarkb, It's related to my code, or related to devstack gate configuration? | 03:33 |
| clarkb | both | 03:33 |
| clarkb | devstack-gate disables sudo between the gate hook and the post test hook | 03:34 |
| clarkb | the change attempts to sudo in the post test hook but sudo is disabled there | 03:34 |
| TuHV | clarkb, Can I change the devstack gate configuration or need to ping someone else? | 03:38 |
| clarkb | TuHV: I think you should run the sudo pieces in the gate hook | 03:39 |
| TuHV | clarkb, Do you mean https://github.com/openstack/neutron-fwaas/blob/8af9d55670bbf9571b870915ceda2808330736e8/neutron_fwaas/tests/contrib/gate_hook_tempest.sh | 03:40 |
| clarkb | yes I think sudo should work there | 03:41 |
| clarkb | well before the last line | 03:41 |
| *** amotoki has quit IRC | 03:41 | |
| clarkb | the last line us what disables sudo | 03:41 |
| *** nicolasbock has quit IRC | 03:42 | |
| *** wolverineav has joined #openstack-infra | 03:42 | |
| TuHV | clarkb, How can I enable it? | 03:43 |
| *** gouthamr has quit IRC | 03:44 | |
| *** esikachev has joined #openstack-infra | 03:44 | |
| clarkb | TuHV: https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n746 you can change the default value of that var however I think it would be better to do your setup before hand and then disable sudo so that rootwrap/privsep are tested | 03:45 |
| clarkb | that is why we remove audo there | 03:45 |
| *** annp has joined #openstack-infra | 03:47 | |
| *** wolverineav has quit IRC | 03:48 | |
| *** mriedem is now known as mriedem_afk | 03:48 | |
| *** salv-orlando has joined #openstack-infra | 03:48 | |
| *** esikachev has quit IRC | 03:49 | |
| TuHV | clarb, Actually, I don't have much experience with this configuration. Could you please show me how I can do that? | 03:51 |
| TuHV | clarkb, Actually, I don't have much experience with this configuration. Could you please show me how I can do that? | 03:51 |
| *** yamahata has joined #openstack-infra | 03:53 | |
| clarkb | I'm not familiar with the fwaas gate hook either | 03:53 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Coverage: add and store subunit stream for coverage stats https://review.openstack.org/434625 | 03:53 |
| *** salv-orlando has quit IRC | 03:53 | |
| clarkb | it can run before the devstack gate script or as part of devstack (via a plugin) amd would run when sudo is allowed | 03:53 |
| *** wolverineav has joined #openstack-infra | 03:55 | |
| *** armax has quit IRC | 03:59 | |
| *** ijw has joined #openstack-infra | 04:01 | |
| *** links has joined #openstack-infra | 04:02 | |
| *** john-davidge has joined #openstack-infra | 04:03 | |
| *** wolverineav has quit IRC | 04:06 | |
| *** nmathew has joined #openstack-infra | 04:07 | |
| *** tphummel has quit IRC | 04:08 | |
| *** john-davidge has quit IRC | 04:09 | |
| *** yamamoto has joined #openstack-infra | 04:10 | |
| *** rbuzatu has joined #openstack-infra | 04:11 | |
| *** Julien-z_ has quit IRC | 04:15 | |
| *** erikmwilson has quit IRC | 04:15 | |
| *** erikmwilson has joined #openstack-infra | 04:16 | |
| *** dave-mccowan has quit IRC | 04:19 | |
| *** adrian_otto has joined #openstack-infra | 04:26 | |
| *** esikachev has joined #openstack-infra | 04:26 | |
| *** funzo has joined #openstack-infra | 04:27 | |
| *** amotoki has joined #openstack-infra | 04:29 | |
| *** esikachev has quit IRC | 04:30 | |
| TuHV | clarkb, I will check that | 04:31 |
| TuHV | clarkb, thank you for your time :) | 04:31 |
| *** funzo has quit IRC | 04:32 | |
| *** sree has joined #openstack-infra | 04:32 | |
| *** tlian has quit IRC | 04:34 | |
| *** adriant has quit IRC | 04:34 | |
| *** sree_ has joined #openstack-infra | 04:36 | |
| *** sree_ is now known as Guest58863 | 04:36 | |
| *** sree has quit IRC | 04:36 | |
| *** wolverineav has joined #openstack-infra | 04:36 | |
| *** esikachev has joined #openstack-infra | 04:37 | |
| *** hongbin has quit IRC | 04:38 | |
| *** amotoki has quit IRC | 04:40 | |
| *** knangia has quit IRC | 04:40 | |
| *** esikachev has quit IRC | 04:44 | |
| *** rbuzatu has quit IRC | 04:45 | |
| *** Julien-zte has joined #openstack-infra | 04:47 | |
| *** wolverineav has quit IRC | 04:47 | |
| *** Julien-zte has quit IRC | 04:47 | |
| *** amotoki has joined #openstack-infra | 04:48 | |
| *** Julien-zte has joined #openstack-infra | 04:49 | |
| *** tqtran has joined #openstack-infra | 04:57 | |
| *** jamesdenton has quit IRC | 04:58 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Coverage: add and store subunit stream for coverage stats https://review.openstack.org/434625 | 05:00 |
| *** tqtran has quit IRC | 05:02 | |
| *** thorst_ has joined #openstack-infra | 05:04 | |
| *** erlon has quit IRC | 05:05 | |
| *** nadya has joined #openstack-infra | 05:06 | |
| *** thorst_ has quit IRC | 05:08 | |
| *** sdake has joined #openstack-infra | 05:09 | |
| *** sdake_ has joined #openstack-infra | 05:12 | |
| *** gongysh has quit IRC | 05:13 | |
| *** sdake has quit IRC | 05:14 | |
| *** adrian_otto has quit IRC | 05:15 | |
| *** gongysh has joined #openstack-infra | 05:19 | |
| *** csomerville has quit IRC | 05:20 | |
| *** csomerville has joined #openstack-infra | 05:20 | |
| *** wolverineav has joined #openstack-infra | 05:20 | |
| *** gildub has quit IRC | 05:21 | |
| *** rkukura has quit IRC | 05:21 | |
| *** TuHV has quit IRC | 05:21 | |
| *** ijw has quit IRC | 05:22 | |
| *** dimtruck is now known as zz_dimtruck | 05:24 | |
| *** esikachev has joined #openstack-infra | 05:26 | |
| *** gildub has joined #openstack-infra | 05:26 | |
| *** TuHV has joined #openstack-infra | 05:27 | |
| *** funzo has joined #openstack-infra | 05:28 | |
| *** salv-orlando has joined #openstack-infra | 05:28 | |
| *** wolverineav has quit IRC | 05:30 | |
| *** esikachev has quit IRC | 05:31 | |
| *** jaosorior has joined #openstack-infra | 05:31 | |
| *** funzo has quit IRC | 05:32 | |
| *** salv-orlando has quit IRC | 05:33 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job https://review.openstack.org/426863 | 05:34 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job https://review.openstack.org/426863 | 05:34 |
| *** esikachev has joined #openstack-infra | 05:35 | |
| *** nadya has quit IRC | 05:40 | |
| *** admcleod has quit IRC | 05:43 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy novajoin in the TLS-everywhere job https://review.openstack.org/427481 | 05:43 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job (novajoin) https://review.openstack.org/427482 | 05:43 |
| openstackgerrit | Rabi Mishra proposed openstack-infra/project-config master: Run amqp1 non-voting job only on master https://review.openstack.org/434662 | 05:51 |
| *** adrian_otto has joined #openstack-infra | 05:52 | |
| *** masber has quit IRC | 05:53 | |
| *** salv-orlando has joined #openstack-infra | 05:53 | |
| *** gildub has quit IRC | 05:54 | |
| *** sdake_ has quit IRC | 05:54 | |
| *** adrian_otto has quit IRC | 05:54 | |
| *** sandanar has joined #openstack-infra | 05:58 | |
| *** tnovacik has joined #openstack-infra | 05:58 | |
| openstackgerrit | Matthew Oliver proposed openstack-infra/project-config master: Stop swifts func-post-as-copy job running on stable https://review.openstack.org/434664 | 06:02 |
| *** salv-orl_ has joined #openstack-infra | 06:04 | |
| *** john-davidge has joined #openstack-infra | 06:04 | |
| *** salv-orlando has quit IRC | 06:07 | |
| *** john-davidge has quit IRC | 06:09 | |
| *** rzetikx has joined #openstack-infra | 06:10 | |
| *** apetrich has quit IRC | 06:12 | |
| *** wolverineav has joined #openstack-infra | 06:15 | |
| *** esikachev has quit IRC | 06:19 | |
| *** tuanluong has joined #openstack-infra | 06:20 | |
| *** rbuzatu has joined #openstack-infra | 06:20 | |
| *** wolverineav has quit IRC | 06:21 | |
| *** claudiub has joined #openstack-infra | 06:25 | |
| *** funzo has joined #openstack-infra | 06:28 | |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config master: Enable coverage jobs for monasca projects https://review.openstack.org/432845 | 06:33 |
| *** funzo has quit IRC | 06:33 | |
| *** wolverineav has joined #openstack-infra | 06:33 | |
| *** esikachev has joined #openstack-infra | 06:38 | |
| *** adrian_otto has joined #openstack-infra | 06:40 | |
| *** wolverineav has quit IRC | 06:42 | |
| *** sree has joined #openstack-infra | 06:44 | |
| *** e0ne has joined #openstack-infra | 06:46 | |
| *** tnovacik has quit IRC | 06:47 | |
| *** Guest58863 has quit IRC | 06:47 | |
| *** tnovacik has joined #openstack-infra | 06:50 | |
| *** wolverineav has joined #openstack-infra | 06:52 | |
| *** rbuzatu has quit IRC | 06:55 | |
| *** abregman has joined #openstack-infra | 06:55 | |
| *** apetrich has joined #openstack-infra | 06:56 | |
| *** ijw has joined #openstack-infra | 06:57 | |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config master: Enable coverage jobs for monasca projects https://review.openstack.org/432845 | 06:58 |
| *** wolverineav has quit IRC | 07:02 | |
| *** rcernin has joined #openstack-infra | 07:03 | |
| *** andreas_s has joined #openstack-infra | 07:04 | |
| *** sandanar_ has joined #openstack-infra | 07:04 | |
| *** thorst_ has joined #openstack-infra | 07:05 | |
| *** sandanar has quit IRC | 07:06 | |
| *** sandanar__ has joined #openstack-infra | 07:07 | |
| *** thorst_ has quit IRC | 07:10 | |
| *** sandanar_ has quit IRC | 07:11 | |
| *** jtomasek has joined #openstack-infra | 07:12 | |
| *** tesseract has joined #openstack-infra | 07:13 | |
| *** cshastri has joined #openstack-infra | 07:15 | |
| *** yuval has joined #openstack-infra | 07:16 | |
| *** jbryce has quit IRC | 07:18 | |
| *** jbryce has joined #openstack-infra | 07:20 | |
| *** e0ne has quit IRC | 07:24 | |
| *** TuHV has quit IRC | 07:25 | |
| *** wolverineav has joined #openstack-infra | 07:26 | |
| openstackgerrit | Rabi Mishra proposed openstack-infra/project-config master: Run amqp1 non-voting job only on master https://review.openstack.org/434662 | 07:27 |
| *** yamamoto has quit IRC | 07:27 | |
| *** funzo has joined #openstack-infra | 07:29 | |
| *** amotoki_ has joined #openstack-infra | 07:29 | |
| *** Julien-zte has quit IRC | 07:31 | |
| *** wolverineav has quit IRC | 07:32 | |
| *** amotoki has quit IRC | 07:32 | |
| *** gus_ is now known as gus | 07:32 | |
| *** markus_z has joined #openstack-infra | 07:32 | |
| *** eroux has joined #openstack-infra | 07:33 | |
| *** rbuzatu_ has joined #openstack-infra | 07:35 | |
| *** funzo has quit IRC | 07:35 | |
| openstackgerrit | Michele Baldessari proposed openstack-infra/tripleo-ci master: Collect /var/lib/pacemaker/cib in the tar files from nodes https://review.openstack.org/425151 | 07:41 |
| *** tnovacik has quit IRC | 07:43 | |
| *** TuHV has joined #openstack-infra | 07:45 | |
| *** eroux has quit IRC | 07:47 | |
| *** flepied has quit IRC | 07:50 | |
| *** eroux has joined #openstack-infra | 07:50 | |
| *** Cibo has quit IRC | 07:51 | |
| *** adrian_otto has quit IRC | 07:55 | |
| *** Julien-zte has joined #openstack-infra | 07:55 | |
| *** cshastri has quit IRC | 07:56 | |
| *** tqtran has joined #openstack-infra | 07:59 | |
| *** Hal has quit IRC | 08:01 | |
| *** vsaienko has joined #openstack-infra | 08:02 | |
| *** cody-somerville has joined #openstack-infra | 08:02 | |
| *** yamamoto has joined #openstack-infra | 08:03 | |
| *** tnovacik has joined #openstack-infra | 08:03 | |
| *** shardy has joined #openstack-infra | 08:03 | |
| *** tqtran has quit IRC | 08:03 | |
| *** Cibo has joined #openstack-infra | 08:04 | |
| *** csomerville has quit IRC | 08:05 | |
| *** john-davidge has joined #openstack-infra | 08:05 | |
| *** tnovacik has quit IRC | 08:08 | |
| *** adrian_otto has joined #openstack-infra | 08:08 | |
| *** jamespag` is now known as jamespage | 08:09 | |
| *** john-davidge has quit IRC | 08:10 | |
| yolanda | good morning | 08:10 |
| *** pcaruana has joined #openstack-infra | 08:14 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job (novajoin) https://review.openstack.org/427482 | 08:16 |
| *** Hal has joined #openstack-infra | 08:17 | |
| *** pcaruana has quit IRC | 08:20 | |
| *** adrian_otto has quit IRC | 08:21 | |
| *** pcaruana has joined #openstack-infra | 08:22 | |
| *** d0ugal has joined #openstack-infra | 08:23 | |
| *** ijw has quit IRC | 08:23 | |
| *** flepied has joined #openstack-infra | 08:24 | |
| *** bhavik1 has joined #openstack-infra | 08:24 | |
| *** thorst_ has joined #openstack-infra | 08:25 | |
| *** martinkopec has joined #openstack-infra | 08:26 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job (novajoin) https://review.openstack.org/427482 | 08:27 |
| *** thorst_ has quit IRC | 08:30 | |
| *** dingyichen has quit IRC | 08:31 | |
| *** ssbarnea has joined #openstack-infra | 08:31 | |
| *** funzo has joined #openstack-infra | 08:31 | |
| *** yamamoto has quit IRC | 08:32 | |
| *** yamamoto has joined #openstack-infra | 08:33 | |
| *** sflanigan has quit IRC | 08:34 | |
| *** funzo has quit IRC | 08:36 | |
| *** yamamoto has quit IRC | 08:37 | |
| *** huanxie has joined #openstack-infra | 08:38 | |
| *** winggundamth has quit IRC | 08:42 | |
| *** winggundamth has joined #openstack-infra | 08:44 | |
| *** rossella_s has joined #openstack-infra | 08:46 | |
| *** ralonsoh has joined #openstack-infra | 08:49 | |
| *** isaacb has joined #openstack-infra | 08:51 | |
| *** jpich has joined #openstack-infra | 08:51 | |
| *** ccamacho has joined #openstack-infra | 08:52 | |
| *** nadya has joined #openstack-infra | 08:54 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove linters job of elastic-recheck https://review.openstack.org/434573 | 08:56 |
| *** nadya has quit IRC | 08:57 | |
| *** nadya has joined #openstack-infra | 08:57 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add release notes jobs for blazar https://review.openstack.org/431274 | 08:58 |
| *** vsaienko has quit IRC | 08:59 | |
| *** vsaienko has joined #openstack-infra | 08:59 | |
| *** vsaienko has quit IRC | 08:59 | |
| *** zzzeek has quit IRC | 09:00 | |
| *** zzzeek has joined #openstack-infra | 09:00 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add functional testing to Craton's check and gate https://review.openstack.org/434399 | 09:01 |
| *** cshastri has joined #openstack-infra | 09:03 | |
| *** yamahata has quit IRC | 09:06 | |
| openstackgerrit | Merged openstack-infra/project-config master: Retiring python-wsmanclient: stop gating on changes https://review.openstack.org/433653 | 09:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Add experimental job for ironic's ansible-deploy https://review.openstack.org/415024 | 09:09 |
| openstackgerrit | Merged openstack-infra/project-config master: add tripleo non-voting job into zaqar gate https://review.openstack.org/433749 | 09:09 |
| openstackgerrit | Merged openstack-infra/project-config master: add tripleo non-voting job into congress gate https://review.openstack.org/433739 | 09:09 |
| *** aarefiev_afk is now known as aarefiev | 09:09 | |
| openstackgerrit | Merged openstack-infra/project-config master: Kill gate-tempest-dsvm-platform-ubuntu-trusty https://review.openstack.org/433274 | 09:09 |
| openstackgerrit | Merged openstack-infra/project-config master: Add IRC notifications from Gerrit for #craton https://review.openstack.org/434391 | 09:09 |
| openstackgerrit | Oksana Voshchana proposed openstack-infra/devstack-gate master: TEST for multinode grenade devstack https://review.openstack.org/396679 | 09:09 |
| openstackgerrit | Merged openstack-infra/project-config master: grafana: track new rally py35 job for neutron https://review.openstack.org/434402 | 09:10 |
| *** hashar has joined #openstack-infra | 09:10 | |
| *** abregman has quit IRC | 09:10 | |
| *** derekh has joined #openstack-infra | 09:15 | |
| *** abregman has joined #openstack-infra | 09:16 | |
| *** rzetikx_ has joined #openstack-infra | 09:17 | |
| *** yamamoto has joined #openstack-infra | 09:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: tripleo-ci move multinode-upgrades-nv to check queue https://review.openstack.org/434247 | 09:18 |
| openstackgerrit | Merged openstack-infra/project-config master: Notify #openstack-dns of stable/ocata changes https://review.openstack.org/434221 | 09:18 |
| *** yamamoto has quit IRC | 09:19 | |
| *** yamamoto has joined #openstack-infra | 09:20 | |
| *** rzetikx has quit IRC | 09:20 | |
| *** yamahata has joined #openstack-infra | 09:21 | |
| *** dtantsur|afk is now known as dtantsur | 09:23 | |
| *** yamamoto has quit IRC | 09:25 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy novajoin in the TLS-everywhere job https://review.openstack.org/427481 | 09:26 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job (novajoin) https://review.openstack.org/427482 | 09:26 |
| *** mhickey has joined #openstack-infra | 09:29 | |
| *** yamahata has quit IRC | 09:30 | |
| *** openstackgerrit has quit IRC | 09:32 | |
| *** rbuzatu_ has quit IRC | 09:32 | |
| *** Julien-zte has quit IRC | 09:34 | |
| *** lucas-afk is now known as lucasagomes | 09:35 | |
| *** rbuzatu has joined #openstack-infra | 09:35 | |
| *** ssbarnea has quit IRC | 09:51 | |
| *** ssbarnea has joined #openstack-infra | 09:51 | |
| *** saibarspeis has joined #openstack-infra | 09:51 | |
| *** openstackgerrit has joined #openstack-infra | 09:52 | |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config master: Retiring python-wsmanclient: remove all jobs and set project read-only https://review.openstack.org/434779 | 09:52 |
| *** claudiub has quit IRC | 09:53 | |
| *** TuHV has quit IRC | 09:54 | |
| *** e0ne has joined #openstack-infra | 09:56 | |
| *** sshnaidm|off is now known as sshnaidm | 09:57 | |
| *** Julien-zte has joined #openstack-infra | 09:59 | |
| *** TuHV has joined #openstack-infra | 09:59 | |
| *** nadya has quit IRC | 10:00 | |
| *** tqtran has joined #openstack-infra | 10:01 | |
| *** amotoki_ has quit IRC | 10:01 | |
| *** ralonsoh_ has joined #openstack-infra | 10:04 | |
| *** gildub has joined #openstack-infra | 10:04 | |
| *** isaacb has quit IRC | 10:05 | |
| *** tqtran has quit IRC | 10:05 | |
| *** john-davidge has joined #openstack-infra | 10:06 | |
| *** flepied has quit IRC | 10:07 | |
| *** ralonsoh has quit IRC | 10:07 | |
| *** Julien-zte has quit IRC | 10:07 | |
| *** esikachev has quit IRC | 10:08 | |
| *** sdake has joined #openstack-infra | 10:08 | |
| *** Cibo has quit IRC | 10:08 | |
| *** sambetts|afk is now known as sambetts | 10:09 | |
| *** john-davidge has quit IRC | 10:11 | |
| *** Julien-zte has joined #openstack-infra | 10:12 | |
| *** isaacb has joined #openstack-infra | 10:13 | |
| *** amotoki has joined #openstack-infra | 10:13 | |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config master: Retiring python-wsmanclient: remove all jobs and set project read-only https://review.openstack.org/434779 | 10:14 |
| openstackgerrit | Béla Vancsics proposed openstack-dev/hacking master: Add a check to make sure the right assert* method is used https://review.openstack.org/434366 | 10:15 |
| *** Julien-z_ has joined #openstack-infra | 10:17 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 10:17 |
| *** Julien-zte has quit IRC | 10:17 | |
| *** cshastri has quit IRC | 10:17 | |
| *** denisra has joined #openstack-infra | 10:18 | |
| *** erlon has joined #openstack-infra | 10:18 | |
| *** admcleod has joined #openstack-infra | 10:20 | |
| *** amotoki has quit IRC | 10:20 | |
| *** Cibo has joined #openstack-infra | 10:22 | |
| *** yamamoto has joined #openstack-infra | 10:22 | |
| *** ralonsoh_ is now known as ralonsoh | 10:23 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:23 | |
| *** flepied has joined #openstack-infra | 10:24 | |
| openstackgerrit | Oksana Voshchana proposed openstack-infra/devstack-gate master: WIP: test grenade ironic https://review.openstack.org/414986 | 10:26 |
| *** admcleod has quit IRC | 10:26 | |
| *** thorst_ has joined #openstack-infra | 10:26 | |
| *** admcleod has joined #openstack-infra | 10:27 | |
| *** admcleod has joined #openstack-infra | 10:27 | |
| *** sdake has quit IRC | 10:28 | |
| *** jordanP has joined #openstack-infra | 10:30 | |
| *** thorst_ has quit IRC | 10:31 | |
| openstackgerrit | Merged openstack-infra/project-config master: fuel-web: Use normal python jobs https://review.openstack.org/432436 | 10:32 |
| *** funzo has joined #openstack-infra | 10:33 | |
| *** tosky has joined #openstack-infra | 10:33 | |
| *** liujiong has quit IRC | 10:34 | |
| *** tuanluong has quit IRC | 10:34 | |
| *** nadya has joined #openstack-infra | 10:35 | |
| *** amotoki has joined #openstack-infra | 10:35 | |
| *** funzo has quit IRC | 10:37 | |
| *** ldnunes has joined #openstack-infra | 10:38 | |
| *** nadya has quit IRC | 10:40 | |
| *** yamamoto has quit IRC | 10:40 | |
| *** esikachev has joined #openstack-infra | 10:41 | |
| *** amotoki has quit IRC | 10:42 | |
| *** markus_z has quit IRC | 10:42 | |
| *** winggundamth has quit IRC | 10:45 | |
| *** askb has quit IRC | 10:45 | |
| openstackgerrit | jeck proposed openstack-infra/project-config master: Add Apache 2.0 license to source file https://review.openstack.org/418187 | 10:46 |
| frickler | infra-root: looks like there is something broken with pypi mirroring? http://logs.openstack.org/49/429949/11/check/gate-tempest-dsvm-py35-ubuntu-xenial-nv/0b42e02/logs/devstacklog.txt.gz#_2017-02-16_09_42_15_234 | 10:46 |
| ramishra | AJaeger: Hi, can we discuss https://review.openstack.org/#/c/434662/ ? | 10:47 |
| *** bhavik1 has quit IRC | 10:48 | |
| *** isaacb has quit IRC | 10:48 | |
| ramishra | AJaeger: I don't know why both the amqp1 and py35 jobs are added as nv jobs and not experimental to start with | 10:48 |
| ramishra | I wanted to avoid running it in stable gate unnecessarily | 10:50 |
| *** nadya has joined #openstack-infra | 10:50 | |
| ramishra | AJaeger: I assume we had a plan to fix it in Ocata but could not do it | 10:50 |
| *** annp has quit IRC | 10:51 | |
| AJaeger | ramishra: I'm happy to take the change to not run it for stable branches. | 10:52 |
| AJaeger | ramishra: Do you want to move both to experimental for now? Or do you expect to get them working quickly? | 10:52 |
| ramishra | AJaeger: Thanks. I think we would get them working soon. | 10:53 |
| *** rossella_s has quit IRC | 10:53 | |
| AJaeger | ramishra: your (=heat's) call | 10:53 |
| AJaeger | ok, then let me +2 now | 10:53 |
| *** rossella_s has joined #openstack-infra | 10:54 | |
| ramishra | AJaeger: thanks! | 10:54 |
| *** yanyanhu has quit IRC | 10:55 | |
| *** winggundamth has joined #openstack-infra | 10:57 | |
| *** szaher_ has quit IRC | 10:58 | |
| *** szaher has joined #openstack-infra | 10:59 | |
| *** esikache1 has joined #openstack-infra | 11:00 | |
| *** dtantsur is now known as dtantsur|bbl | 11:01 | |
| *** esikachev has quit IRC | 11:03 | |
| *** sree has quit IRC | 11:04 | |
| *** nicolasbock has joined #openstack-infra | 11:05 | |
| *** sdague has joined #openstack-infra | 11:06 | |
| *** isaacb has joined #openstack-infra | 11:06 | |
| *** TuHV has quit IRC | 11:07 | |
| *** claudiub has joined #openstack-infra | 11:14 | |
| *** flepied has quit IRC | 11:15 | |
| openstackgerrit | Isaac Beckman proposed openstack-infra/puppet-openstackci master: Pass Java arguments to Jenkins master https://review.openstack.org/263021 | 11:15 |
| *** jaosorior has quit IRC | 11:16 | |
| *** jaosorior has joined #openstack-infra | 11:17 | |
| *** wolverineav has joined #openstack-infra | 11:23 | |
| *** lseki has joined #openstack-infra | 11:23 | |
| *** sandanar__ has quit IRC | 11:25 | |
| *** lucasagomes is now known as lucas-bbl | 11:26 | |
| *** eranrom has quit IRC | 11:26 | |
| *** wolverineav has quit IRC | 11:27 | |
| *** eranrom has joined #openstack-infra | 11:28 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci master: Enable Ceph in the updates job https://review.openstack.org/434832 | 11:28 |
| *** matrohon has quit IRC | 11:29 | |
| *** rfolco has joined #openstack-infra | 11:30 | |
| *** flepied has joined #openstack-infra | 11:31 | |
| *** ralonsoh has quit IRC | 11:31 | |
| *** funzo has joined #openstack-infra | 11:34 | |
| *** nmathew has quit IRC | 11:37 | |
| *** yamamoto has joined #openstack-infra | 11:38 | |
| *** funzo has quit IRC | 11:38 | |
| *** jkilpatr_ has quit IRC | 11:39 | |
| *** markus_z has joined #openstack-infra | 11:45 | |
| *** yamamoto has quit IRC | 11:47 | |
| *** gildub has quit IRC | 11:47 | |
| *** jkilpatr has joined #openstack-infra | 12:00 | |
| *** sdake has joined #openstack-infra | 12:02 | |
| openstackgerrit | Kirill Zaitsev proposed openstack-infra/project-config master: Remove python34-db-jobs https://review.openstack.org/434851 | 12:02 |
| *** tqtran has joined #openstack-infra | 12:03 | |
| *** salv-orlando has joined #openstack-infra | 12:04 | |
| *** fandi has joined #openstack-infra | 12:04 | |
| *** dave-mccowan has joined #openstack-infra | 12:05 | |
| *** psachin has joined #openstack-infra | 12:06 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci master: Enable Ceph in multinode upgrades job (via HCI). https://review.openstack.org/434855 | 12:06 |
| *** raildo has joined #openstack-infra | 12:06 | |
| *** john-davidge has joined #openstack-infra | 12:07 | |
| *** tqtran has quit IRC | 12:07 | |
| *** Anticime1 is now known as Anticimex | 12:07 | |
| *** salv-orl_ has quit IRC | 12:07 | |
| *** fandi has quit IRC | 12:09 | |
| *** fandi has joined #openstack-infra | 12:09 | |
| *** john-davidge has quit IRC | 12:12 | |
| *** Julien-z_ has quit IRC | 12:14 | |
| *** Julien-zte has joined #openstack-infra | 12:15 | |
| *** john-davidge has joined #openstack-infra | 12:16 | |
| *** askb has joined #openstack-infra | 12:19 | |
| *** askb has quit IRC | 12:19 | |
| *** Julien-zte has quit IRC | 12:19 | |
| *** esikache1 has quit IRC | 12:20 | |
| *** isaacb has quit IRC | 12:24 | |
| *** shardy is now known as shardy_lunch | 12:24 | |
| *** esikache1 has joined #openstack-infra | 12:24 | |
| *** yamamoto has joined #openstack-infra | 12:25 | |
| *** isaacb has joined #openstack-infra | 12:26 | |
| openstackgerrit | Oksana Voshchana proposed openstack-infra/devstack-gate master: TEST for multinode grenade devstack https://review.openstack.org/396679 | 12:27 |
| *** esikache1 has quit IRC | 12:28 | |
| *** gongysh has quit IRC | 12:31 | |
| sdague | anyone able to get this bump through - https://review.openstack.org/#/c/433093/ | 12:32 |
| *** funzo has joined #openstack-infra | 12:34 | |
| *** gordc has joined #openstack-infra | 12:36 | |
| mordred | sdague: +2 | 12:37 |
| *** funzo has quit IRC | 12:39 | |
| *** sdake_ has joined #openstack-infra | 12:39 | |
| *** esikache1 has joined #openstack-infra | 12:40 | |
| *** wolverineav has joined #openstack-infra | 12:41 | |
| *** sdake has quit IRC | 12:42 | |
| *** thorst_ has joined #openstack-infra | 12:44 | |
| openstackgerrit | Jakub Libosvar proposed openstack-infra/project-config master: neutron: Run in-tree Neutron tempest tests with concurrency 2 https://review.openstack.org/434866 | 12:44 |
| *** esikache1 has quit IRC | 12:44 | |
| *** esikache1 has joined #openstack-infra | 12:50 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use reno for release notes https://review.openstack.org/262597 | 12:55 |
| *** chlong has joined #openstack-infra | 12:56 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy novajoin in the TLS-everywhere job https://review.openstack.org/427481 | 12:56 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job (novajoin) https://review.openstack.org/427482 | 12:56 |
| *** mptacekx has joined #openstack-infra | 12:57 | |
| *** eroux has quit IRC | 12:58 | |
| *** kzaitsev_mb has quit IRC | 13:00 | |
| *** lucas-bbl is now known as lucasagomes | 13:01 | |
| *** kzaitsev_mb has joined #openstack-infra | 13:02 | |
| *** EricGonczer_ has joined #openstack-infra | 13:03 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add Apache 2.0 license to source file https://review.openstack.org/418187 | 13:05 |
| *** fandi has quit IRC | 13:09 | |
| *** fandi has joined #openstack-infra | 13:10 | |
| *** ccamacho is now known as ccamacho|lunch | 13:10 | |
| *** shardy_lunch is now known as shardy | 13:12 | |
| *** lucasagomes is now known as lucas-brb | 13:13 | |
| *** mtanino has joined #openstack-infra | 13:14 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy novajoin in the TLS-everywhere job https://review.openstack.org/427481 | 13:14 |
| *** rvba` is now known as rvba | 13:14 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job (novajoin) https://review.openstack.org/427482 | 13:14 |
| *** gouthamr has joined #openstack-infra | 13:15 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 13:16 |
| *** rhallisey has joined #openstack-infra | 13:17 | |
| *** dprince has joined #openstack-infra | 13:19 | |
| *** dave-mccowan has quit IRC | 13:19 | |
| *** trown|outtypewww is now known as trown | 13:20 | |
| *** nadya has quit IRC | 13:24 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 13:26 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci master: Enable Ceph in multinode upgrades job (via HCI). https://review.openstack.org/434855 | 13:28 |
| *** mriedem_afk is now known as mriedem | 13:29 | |
| *** vi01et03 has joined #openstack-infra | 13:30 | |
| *** Jeffrey4l has quit IRC | 13:30 | |
| *** wolverineav has quit IRC | 13:30 | |
| *** EricGonc_ has joined #openstack-infra | 13:30 | |
| *** Jeffrey4l has joined #openstack-infra | 13:31 | |
| *** wolverineav has joined #openstack-infra | 13:31 | |
| *** EricGonczer_ has quit IRC | 13:32 | |
| *** abregman is now known as abregman|mtg | 13:34 | |
| *** abregman|mtg has quit IRC | 13:34 | |
| *** wolverineav has quit IRC | 13:35 | |
| *** wolverineav has joined #openstack-infra | 13:35 | |
| *** salv-orlando has quit IRC | 13:36 | |
| *** rbuzatu_ has joined #openstack-infra | 13:38 | |
| *** psachin has quit IRC | 13:39 | |
| *** links has quit IRC | 13:39 | |
| *** rbuzatu has quit IRC | 13:40 | |
| *** rlandy has joined #openstack-infra | 13:40 | |
| *** ralonsoh has joined #openstack-infra | 13:40 | |
| *** e0ne has quit IRC | 13:41 | |
| *** psachin has joined #openstack-infra | 13:41 | |
| *** e0ne has joined #openstack-infra | 13:42 | |
| *** gouthamr has quit IRC | 13:42 | |
| *** yamamoto has quit IRC | 13:43 | |
| pabelanger | frickler: have you been able to reproduce the issue? | 13:43 |
| *** gouthamr has joined #openstack-infra | 13:43 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Change request_id logging to match nova format https://review.openstack.org/434916 | 13:44 |
| *** fandi has quit IRC | 13:45 | |
| pabelanger | /dev/mapper/main-logs 12T 12T 72G 100% /srv/static/logs | 13:45 |
| pabelanger | up to 75GB free now | 13:46 |
| AJaeger | not much ;( | 13:46 |
| frickler | pabelanger: no, went away after a recheck, but may have hit a different cloud then. might also have been cause by a mirror update running in parallel I guess | 13:46 |
| pabelanger | frickler: yes, lets see how the day continues. Thanks for reporting | 13:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Change request_id logging to match nova format https://review.openstack.org/434916 | 13:48 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Actually normalize nova usage data https://review.openstack.org/434481 | 13:48 |
| *** sdake_ is now known as sdake | 13:48 | |
| *** yamamoto has joined #openstack-infra | 13:49 | |
| *** lucas-brb is now known as lucasagomes | 13:49 | |
| *** xyang1 has joined #openstack-infra | 13:50 | |
| *** bhavik1 has joined #openstack-infra | 13:51 | |
| *** mtanino has quit IRC | 13:51 | |
| *** tosky has quit IRC | 13:52 | |
| *** xyang1 has quit IRC | 13:53 | |
| *** yamamoto has quit IRC | 13:53 | |
| *** bhavik1 has quit IRC | 13:53 | |
| *** gongysh has joined #openstack-infra | 13:55 | |
| *** E_Y has quit IRC | 13:55 | |
| *** e0ne has quit IRC | 13:56 | |
| *** vi01et03_ has joined #openstack-infra | 13:56 | |
| *** E_Y has joined #openstack-infra | 13:56 | |
| *** mdrabe has joined #openstack-infra | 13:56 | |
| *** bh526r has joined #openstack-infra | 13:57 | |
| *** tlian has joined #openstack-infra | 13:58 | |
| *** zz_dimtruck is now known as dimtruck | 13:58 | |
| AJaeger | pabelanger: could you put https://review.openstack.org/431134 (another python-db change) and https://review.openstack.org/434851 on your review list - if logs allow ;) Thanks! | 13:59 |
| *** vi01et03 has quit IRC | 13:59 | |
| *** eharney has joined #openstack-infra | 14:01 | |
| *** winggundamth has quit IRC | 14:01 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix quickstart issues https://review.openstack.org/425459 | 14:01 |
| *** esikache1 has quit IRC | 14:02 | |
| *** burgerk has joined #openstack-infra | 14:03 | |
| *** markus_z has quit IRC | 14:06 | |
| *** sree has joined #openstack-infra | 14:09 | |
| *** abregman has joined #openstack-infra | 14:09 | |
| *** wolverineav has quit IRC | 14:11 | |
| *** wolverineav has joined #openstack-infra | 14:12 | |
| *** mriedem has quit IRC | 14:13 | |
| *** baoli has joined #openstack-infra | 14:15 | |
| *** wolverin_ has joined #openstack-infra | 14:16 | |
| pabelanger | AJaeger: done | 14:16 |
| *** wolverin_ has quit IRC | 14:16 | |
| *** wolverineav has quit IRC | 14:17 | |
| *** _ari_ has quit IRC | 14:20 | |
| *** kgiusti has joined #openstack-infra | 14:20 | |
| *** ccamacho|lunch is now known as ccamacho | 14:20 | |
| *** mhickey has quit IRC | 14:20 | |
| *** matt-borland has joined #openstack-infra | 14:21 | |
| *** adrian_otto has joined #openstack-infra | 14:21 | |
| *** markus_z has joined #openstack-infra | 14:21 | |
| *** _ari_ has joined #openstack-infra | 14:22 | |
| AJaeger | thanks, pabelanger ! | 14:22 |
| openstackgerrit | Radoslav Gerganov proposed openstack-infra/project-config master: Add non-voting py35 job for nova-mksproxy https://review.openstack.org/434932 | 14:22 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove python34-db-jobs https://review.openstack.org/434851 | 14:22 |
| *** mhickey has joined #openstack-infra | 14:23 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Fix potential race in image upload cleanup https://review.openstack.org/434934 | 14:24 |
| Zara | hey, so this is a bit out there-- at the moment I'm looking into making a python notedb parser and potentially a CLI for it down the road (one day!); I'm currently hacking up the initial skeleton on my personal github but I'm wondering if down the line this would be something that made sense as an openstack(-infra?) project. | 14:24 |
| *** funzo has joined #openstack-infra | 14:24 | |
| *** adrian_otto has quit IRC | 14:24 | |
| *** wolverineav has joined #openstack-infra | 14:26 | |
| fungi | frickler: when i download http://mirror.regionone.osic-cloud1.openstack.org/pypi/packages/32/0a/9fa2a1cbe00656020f83d8c331756b325a75203f647aad5535c5718b4c1b/lxml-3.7.2-cp35-cp35m-manylinux1_x86_64.whl#md5=82c56453ae9a855469c36733afe162b7 i get a file with an md5sum of 82c56453ae9a855469c36733afe162b7 so odds are the download got truncated somehow, probably just blame cosmic rays or solar flares | 14:27 |
| fungi | unless it continues | 14:27 |
| *** kgiusti has quit IRC | 14:27 | |
| Zara | (partly asking because I like the idea of developing a notedb parser while using notedb, tbh) | 14:27 |
| *** isaacb has quit IRC | 14:28 | |
| mordred | Zara: nodedb as in the gerrit notedb yeah? | 14:29 |
| fungi | Zara: you mean the new storage backend for gerrit using git notee? | 14:29 |
| fungi | er, notes | 14:29 |
| *** esikache1 has joined #openstack-infra | 14:29 | |
| Zara | that's the one! | 14:29 |
| fungi | odds are we would have a use for it once we upgrade to a newer gerrit release (soon! i hope) | 14:29 |
| *** pblaho has quit IRC | 14:30 | |
| *** nadya has joined #openstack-infra | 14:30 | |
| fungi | right now we still tend to fiddle with things in mysqlclient that may no longer be there in newer releases | 14:30 |
| mordred | yup. so I'd say that sounds like a lovely infra library to me | 14:31 |
| *** Goneri has joined #openstack-infra | 14:31 | |
| sdague | fungi: can I get an approve on this? the ironic folks need it to get some grenade bits working - https://review.openstack.org/#/c/433093/ | 14:32 |
| Zara | ace :) right now I'm wandering around consuming docs and trying to understand how it all works, partly by translating an existing perl notedb parser | 14:32 |
| mordred | oy | 14:32 |
| Zara | (which is why I've been rather quiet; perl is exciting new territory) | 14:33 |
| Zara | I'll try to get a story up on it sometime then | 14:33 |
| *** wolverineav has quit IRC | 14:34 | |
| mordred | Zara: if you run into any perl-understanding challenges, mmmpork is a great resource | 14:35 |
| *** sree has quit IRC | 14:35 | |
| *** Ike has joined #openstack-infra | 14:37 | |
| *** winggundamth has joined #openstack-infra | 14:38 | |
| Ike | Hi all. What's the best place to ask for help for problems creating an ICLA Agreement on Gerrit? | 14:38 |
| *** isaacb has joined #openstack-infra | 14:39 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: murano: Use normal python jobs https://review.openstack.org/429050 | 14:40 |
| Zara | mordred: hah, I was about to say 'thanks for the tip', searched them, and oh, it's auggy! :D thanks | 14:40 |
| AJaeger | Ike: here;) This needs your gerrit preferred e-mail address to match a primary e-mail address for a foundation individual member account. | 14:40 |
| *** eroux has joined #openstack-infra | 14:40 | |
| AJaeger | Ike: If you already followed the instructions at http://docs.openstack.org/infra/manual/developers.html#account-setup and still get that, see https://ask.openstack.org/question/56720 for additional troubleshooting tips. | 14:41 |
| AJaeger | mordred, fungi: could either of you put https://review.openstack.org/431134 (another python-db change) on your review list, please? | 14:41 |
| openstackgerrit | Jeffrey Zhang proposed openstack-infra/project-config master: Add debian relate jobs for kolla and kolla-ansible project https://review.openstack.org/434946 | 14:42 |
| *** kgiusti has joined #openstack-infra | 14:42 | |
| *** hashar has quit IRC | 14:42 | |
| Ike | Thanks AJaeger | 14:43 |
| *** hashar has joined #openstack-infra | 14:43 | |
| Ike | It was the Foundation Member association thing | 14:43 |
| *** wolverineav has joined #openstack-infra | 14:45 | |
| *** burgerk has quit IRC | 14:45 | |
| *** mkoderer has joined #openstack-infra | 14:45 | |
| eantyshev | Hello, guys! I'm struggling to get rid of Jenkins and switch to use Zuul launcher in our 3rd-party CI. What versions of nodepool/zuul/jjb do I need? | 14:45 |
| *** apetrich has quit IRC | 14:46 | |
| *** apetrich has joined #openstack-infra | 14:47 | |
| *** tosky has joined #openstack-infra | 14:47 | |
| isaacb | eantyshev: you can post 3rd-party ci questions to #openstack-third-party-ci | 14:48 |
| *** hongbin has joined #openstack-infra | 14:48 | |
| eantyshev | isaacb: good advice, I'll try | 14:49 |
| *** wolverineav has quit IRC | 14:49 | |
| *** wolverineav has joined #openstack-infra | 14:50 | |
| AJaeger | thanks, fungi | 14:50 |
| *** yamamoto has joined #openstack-infra | 14:50 | |
| openstackgerrit | Merged openstack-infra/project-config master: taskflow: Use normal python jobs https://review.openstack.org/431134 | 14:51 |
| *** rossella_s has quit IRC | 14:53 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/infra-specs master: Stackalitycs Persistent Cache https://review.openstack.org/434951 | 14:53 |
| *** rossella_s has joined #openstack-infra | 14:53 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/infra-specs master: Stackalytics Persistent Cache https://review.openstack.org/434951 | 14:54 |
| *** Ike has quit IRC | 14:54 | |
| *** dave-mccowan has joined #openstack-infra | 14:54 | |
| *** wolverineav has quit IRC | 14:54 | |
| isaacb | infra core members: I appreciate your review on https://review.openstack.org/#/c/407571 | 14:56 |
| mrmartin | fungi: I made a spec for the stackalytics persistent cache issue, I did an investigation and some preliminary testing, and Redis Server seems to be a good alternative. I just need to add a new class and some minor changes on Stackalytics code, and some minimal puppet changes to disable memcached server and deploy redis. https://review.openstack.org/434951 | 14:57 |
| *** yamamoto has quit IRC | 14:57 | |
| *** tphummel has joined #openstack-infra | 14:58 | |
| *** xyang1 has joined #openstack-infra | 14:58 | |
| *** mriedem has joined #openstack-infra | 14:59 | |
| mordred | mrmartin: nice | 15:00 |
| *** markmcclain has quit IRC | 15:00 | |
| *** annegentle has joined #openstack-infra | 15:00 | |
| *** markmcclain has joined #openstack-infra | 15:00 | |
| mrmartin | thnx | 15:01 |
| *** wolverineav has joined #openstack-infra | 15:04 | |
| *** mdbooth_ is now known as mdbooth | 15:04 | |
| *** tqtran has joined #openstack-infra | 15:04 | |
| *** annegentle has quit IRC | 15:05 | |
| *** chlong has quit IRC | 15:05 | |
| *** rkukura has joined #openstack-infra | 15:06 | |
| *** markus_z has quit IRC | 15:06 | |
| *** annegentle has joined #openstack-infra | 15:06 | |
| *** spzala has joined #openstack-infra | 15:07 | |
| *** mtanino has joined #openstack-infra | 15:07 | |
| *** rbrndt has joined #openstack-infra | 15:07 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci master: Enable Ceph in multinode upgrades job (via HCI). https://review.openstack.org/434855 | 15:08 |
| *** wolverineav has quit IRC | 15:08 | |
| *** jkilpatr has quit IRC | 15:09 | |
| *** tqtran has quit IRC | 15:09 | |
| *** jkilpatr has joined #openstack-infra | 15:09 | |
| openstackgerrit | Merged openstack-infra/project-config master: Zun: Move gate_hook.sh to the right location https://review.openstack.org/432844 | 15:10 |
| *** E_Y has quit IRC | 15:10 | |
| *** vi01et03_ has quit IRC | 15:12 | |
| openstackgerrit | Merged openstack-infra/shade master: Fix several concurrent shade gate issues https://review.openstack.org/434428 | 15:14 |
| AJaeger | team, https://review.openstack.org/#/c/429868/ (python-db for tacker) is now ready for review, please put it on your list. | 15:15 |
| openstackgerrit | Jordan Pittier proposed openstack-infra/shade master: attach_volume should always return a vol attachment. https://review.openstack.org/434964 | 15:16 |
| fungi | eantyshev: be aware that a big part of why zuul-launcher is poorly (or un-) documented is that it was a stop-gap for us while we complete work on zuul v3. the zuul v2.5 ansible launcher isn't the same as how v3 will work, nor do we intend to support it much (if at all) once v3 is released. the current recommendation is to continue using zuul v2.x with jenkins and the jenkins gearman plug-in | 15:17 |
| fungi | that said, if you want to try anyway, we puppet the launchers from class definitions in the openstack-infra/system-config repo | 15:18 |
| *** burgerk has joined #openstack-infra | 15:18 | |
| fungi | mrmartin: wow, thanks! i was just looking at our stackalytics.o.o deployment last week and realized that it's months behind in data because it hasn't been restarted recently | 15:19 |
| mrmartin | yeah. | 15:19 |
| mrmartin | I hope this change will solve it. | 15:19 |
| pabelanger | ++ for persistent cache | 15:20 |
| mrmartin | Unfortunately I cannot attend Atlanta next week. | 15:20 |
| mrmartin | Would be great to participate in PTG but it collides with family ski trip with children. | 15:20 |
| eantyshev | fungi: Zuul 2.5.1 doesn't include https://review.openstack.org/238988 and https://review.openstack.org/430872 which I need | 15:20 |
| fungi | mrmartin: we'll miss you, but family first of course | 15:20 |
| *** marst has quit IRC | 15:20 | |
| fungi | eantyshev: we continuously deploy zuul from the master branch tip, so that isn't surprising | 15:22 |
| *** kzaitsev_mb has quit IRC | 15:22 | |
| eantyshev | fungi: and now it is time I create another CI, so I'm trying to switch from Zuul & Jenkins combo | 15:22 |
| fungi | eantyshev: but point taken, it's probably time to tag another 2.x.x release on master | 15:23 |
| fungi | jeblair: ^ what do you think? | 15:23 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add action plugins to restrict untrusted execution https://review.openstack.org/428798 | 15:23 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add zuulv3-dev.openstack.org server https://review.openstack.org/430418 | 15:24 |
| jeblair | fungi, eantyshev: i'd like to restart zuul.o.o first; some of those changes we won't have actually seen in production yet since we haven't restarted in a while | 15:26 |
| jeblair | fungi, eantyshev: (and i'd feel better about making a release if we've actually run them) | 15:26 |
| fungi | jeblair: excellent point | 15:26 |
| jeblair | fungi, eantyshev: but afterwords, sure :) | 15:26 |
| *** saibarspeis has quit IRC | 15:26 | |
| *** knangia has joined #openstack-infra | 15:26 | |
| *** nadya has quit IRC | 15:27 | |
| jeblair | maybe we can restart tomorrow? | 15:27 |
| fungi | as long as we're not too concerned of risk of complication for the ocata openstack release (happening in 6 days from today) | 15:28 |
| *** marst has joined #openstack-infra | 15:30 | |
| *** sree has joined #openstack-infra | 15:32 | |
| *** kzaitsev_mb has joined #openstack-infra | 15:32 | |
| *** markus_z has joined #openstack-infra | 15:33 | |
| *** tphummel has quit IRC | 15:38 | |
| *** martinkopec has quit IRC | 15:40 | |
| pabelanger | /dev/mapper/main-logs 12T 12T 63G 100% /srv/static/logs | 15:42 |
| pabelanger | just dropped 12GB, in a hour | 15:42 |
| pabelanger | checking again | 15:42 |
| *** nadya has joined #openstack-infra | 15:42 | |
| *** eroux has quit IRC | 15:42 | |
| *** martinkopec has joined #openstack-infra | 15:42 | |
| *** gongysh has quit IRC | 15:43 | |
| *** kzaitsev_mb has quit IRC | 15:43 | |
| *** e0ne has joined #openstack-infra | 15:44 | |
| pabelanger | jeblair: fungi: clarkb: I think we found the issue: http://paste.openstack.org/show/599265/ | 15:45 |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config master: Make craton functional tests non-voting https://review.openstack.org/434979 | 15:45 |
| pabelanger | https://review.openstack.org/#/c/425459 alone is using 130GB | 15:45 |
| *** faizy has joined #openstack-infra | 15:45 | |
| *** jsho has joined #openstack-infra | 15:46 | |
| pabelanger | EmilienM: ^ we have another tripleo job that is using excessive HDD space on logs.o.o | 15:47 |
| clarkb | pabelanger: wow nice find | 15:48 |
| *** jsho has quit IRC | 15:48 | |
| *** isaacb has quit IRC | 15:49 | |
| *** jsho has joined #openstack-infra | 15:49 | |
| pabelanger | going to start purging data on that review | 15:49 |
| pabelanger | I'll leave patchset 9 logs | 15:50 |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config master: Make craton functional tests non-voting https://review.openstack.org/434979 | 15:50 |
| EmilienM | pabelanger: let me look | 15:50 |
| EmilienM | pabelanger: we are currently in tripleo ci meeting, I'll talk to sshnaidm about it | 15:50 |
| sigmavirus | Is it possible for a separate project's change to be dependent on a project-config change? i.e., would that be a good way to test what I just submitted? | 15:51 |
| pabelanger | most of this is the same issue as yesterday, the full directory of /etc/ is getting uploaded to logs.o.o, includes 4.3GB of /etc/alternatives | 15:52 |
| sshnaidm | pabelanger, can you please make ls -l and du on everything there before you remove it? | 15:52 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci master: Enable Ceph in multinode upgrades job https://review.openstack.org/434855 | 15:52 |
| pabelanger | sshnaidm: I will be leaving the latest patchset logs | 15:52 |
| jeblair | sigmavirus: you can make a change dependent on project-config, but the new content in project-config won't be used until that change lands. | 15:53 |
| jeblair | sigmavirus: (this is something we are working to change in zuulv3) | 15:53 |
| * sigmavirus nods | 15:54 | |
| sigmavirus | I suspect I know why our gate isn't working but I can't prove it | 15:54 |
| pabelanger | sshnaidm: http://logs.openstack.org/59/425459/9/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq-nv/e2ff8c0/logs/oooq/undercloud/ is one problem folder, you shouldn't be uploading the how filesystem for undercloud, only the log files. | 15:55 |
| pabelanger | /dev/mapper/main-logs 12T 12T 182G 99% /srv/static/logs | 15:55 |
| pabelanger | back up to 182GB | 15:55 |
| fungi | pabelanger: awesome, thanks! | 15:55 |
| fungi | that freed up a good chunk | 15:56 |
| fungi | the 45-day expiration pass, as expected, is still underway | 15:56 |
| sshnaidm | pabelanger, sorry, I don't get, what is with this folder? | 15:57 |
| *** yuval has quit IRC | 15:57 | |
| *** winggundamth has quit IRC | 15:58 | |
| *** tphummel has joined #openstack-infra | 15:59 | |
| *** xarses_ has joined #openstack-infra | 15:59 | |
| *** martinkopec has quit IRC | 16:00 | |
| pabelanger | sshnaidm: http://logs.openstack.org/59/425459/du.txt it is currently using 8.9GB of HDD | 16:00 |
| *** salv-orlando has joined #openstack-infra | 16:00 | |
| *** martinkopec has joined #openstack-infra | 16:01 | |
| *** eroux has joined #openstack-infra | 16:01 | |
| pabelanger | sshnaidm: gate-tripleo-ci-centos-7-nonha-multinode-oooq-nv is the job in question | 16:01 |
| *** dtantsur|bbl is now known as dtantsur | 16:02 | |
| *** andreww has joined #openstack-infra | 16:02 | |
| *** rcernin has quit IRC | 16:03 | |
| *** xarses_ has quit IRC | 16:03 | |
| *** xyang1 has quit IRC | 16:04 | |
| *** tqtran has joined #openstack-infra | 16:06 | |
| *** mriedem has quit IRC | 16:07 | |
| *** annegentle has quit IRC | 16:07 | |
| *** saibarspeis has joined #openstack-infra | 16:08 | |
| zxiiro | Has atc codes come out for the Boston summit yet? | 16:08 |
| *** mriedem has joined #openstack-infra | 16:08 | |
| pabelanger | zxiiro: IIRC, they are only handed out to PTG members now | 16:09 |
| pabelanger | s/members/attends | 16:09 |
| sshnaidm | pabelanger, and all other jobs are ok? | 16:09 |
| *** kzaitsev_mb has joined #openstack-infra | 16:09 | |
| zxiiro | Oh really, wasn't aware of that change. Thanks! | 16:09 |
| pabelanger | sshnaidm: not sure, but that one is the main issue ATM | 16:10 |
| openstackgerrit | Jordan Pittier proposed openstack-infra/shade master: attach_volume should always return a vol attachment. https://review.openstack.org/434964 | 16:10 |
| *** tqtran has quit IRC | 16:10 | |
| *** pcaruana has quit IRC | 16:11 | |
| *** xyang1 has joined #openstack-infra | 16:11 | |
| *** derekh has quit IRC | 16:12 | |
| *** ihrachys has joined #openstack-infra | 16:12 | |
| *** abregman has quit IRC | 16:13 | |
| pabelanger | zxiiro: https://www.openstack.org/blog/2016/05/faq-evolving-the-openstack-design-summit/ has some info about ATC, maybe fungi has more details | 16:13 |
| *** trown is now known as trown|outtypewww | 16:13 | |
| *** tesseract has quit IRC | 16:14 | |
| pabelanger | https://www.openstack.org/summit/boston-2017/faq/ | 16:14 |
| pabelanger | has a section about ATC | 16:14 |
| *** faizy has quit IRC | 16:15 | |
| *** derekh has joined #openstack-infra | 16:15 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix quickstart issues https://review.openstack.org/425459 | 16:15 |
| sshnaidm | pabelanger, thanks, I excluded /etc/alternatives, watching it ^^ | 16:15 |
| pabelanger | sshnaidm: k, lets see what happens on the next patch. That logic should like be inverted, blacklist everything, and only whitelist the things you actually need. That is how we do it for devstack-logs macro in JJB | 16:17 |
| sshnaidm | pabelanger, actually it was always so in tripleo-ci, we were collecting "everything except of" | 16:18 |
| pabelanger | sshnaidm: right, I am saying invert it like other openstack projects | 16:19 |
| sshnaidm | pabelanger, yeah, I see | 16:19 |
| fungi | zxiiro: as i understand it, ptg attendees will get summit registration codes on-site (when checking in at the ptg registration booth, so presumably as soon as this sunday afternoon) | 16:19 |
| *** dimtruck is now known as zz_dimtruck | 16:19 | |
| fungi | zxiiro: and speakers for the summit will get registration codes when their talks are accepted | 16:20 |
| fungi | zxiiro: something is also being done i think for those active in the operators community and the user committee | 16:20 |
| *** annegentle has joined #openstack-infra | 16:21 | |
| *** psachin has quit IRC | 16:21 | |
| *** Apoorva has joined #openstack-infra | 16:22 | |
| *** sdake_ has joined #openstack-infra | 16:24 | |
| *** sdake has quit IRC | 16:26 | |
| *** yamahata has joined #openstack-infra | 16:27 | |
| *** Hal has quit IRC | 16:29 | |
| *** tonytan4ever has joined #openstack-infra | 16:30 | |
| *** yamahata has quit IRC | 16:31 | |
| openstackgerrit | Jens Rosenboom proposed openstack-infra/project-config master: Set NOVA_VNC_ENABLED for dsvm-neutron-multinode-full job https://review.openstack.org/434996 | 16:32 |
| *** jcoufal has joined #openstack-infra | 16:32 | |
| *** ihrachys has quit IRC | 16:33 | |
| *** ihrachys has joined #openstack-infra | 16:33 | |
| *** ihrachys has quit IRC | 16:34 | |
| openstackgerrit | Kaitlin Farr proposed openstack-infra/project-config master: Enable signature verification for barbican-tempest-plugin gate https://review.openstack.org/434384 | 16:34 |
| *** egonzalez has joined #openstack-infra | 16:35 | |
| Jeffrey4l | cloud anybody explain why this patch https://review.openstack.org/#/c/388544/ does not trigger gate after Jan 26? | 16:35 |
| Jeffrey4l | neither "recheck" nor "rebase" works. | 16:36 |
| *** Hal has joined #openstack-infra | 16:37 | |
| *** zz_dimtruck is now known as dimtruck | 16:37 | |
| Jeffrey4l | the only special of that patch is: he share the same change-id across two projects. is that matter? | 16:38 |
| pabelanger | Jeffrey4l: I can only guess you are not matching a skip-if setting | 16:38 |
| *** sree has quit IRC | 16:38 | |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config master: Make craton functional tests non-voting https://review.openstack.org/434979 | 16:38 |
| pabelanger | I'll have to look at zuul debug logs | 16:38 |
| Jeffrey4l | pabelanger, it is normal patch as others. so i think it should trigger some jos. | 16:39 |
| Jeffrey4l | thanks. | 16:39 |
| *** ihrachys has joined #openstack-infra | 16:39 | |
| pabelanger | http://paste.openstack.org/show/599285/ | 16:40 |
| pabelanger | looks like an issue with the change-id? | 16:41 |
| pabelanger | 2 different patches are using it | 16:41 |
| pabelanger | https://review.openstack.org/#/c/405215/ | 16:41 |
| clarkb | interesting | 16:42 |
| clarkb | abandoned change depends on itself | 16:42 |
| clarkb | second change has same change id so can't merge | 16:42 |
| clarkb | neat | 16:42 |
| pabelanger | ya, 405215 is using the change-id for Depends-On and Change-id | 16:42 |
| *** nadya has quit IRC | 16:42 | |
| sigmavirus | Is there a trick to using docker in a CI job? #craton uses docker to spin up it service for testing, but that doesn't seem to work in the functional gate. Searching makes me think we need a new job template with OVERRIDE_ENABLED_SERVICES=docker-engine,... but I can't find much documentation around this :( | 16:42 |
| *** annegentle has quit IRC | 16:42 | |
| clarkb | quick fix there would be to unabandon other change, remove depends on, abandon it again | 16:43 |
| *** andreas_s has quit IRC | 16:44 | |
| *** annegentle has joined #openstack-infra | 16:44 | |
| clarkb | sigmavirus: there should be nothing special required to docker, many projects use it successfully | 16:46 |
| Jeffrey4l | pabelanger, i am rechecking 3885444, why it detected 405215 has Dependency cycle. | 16:48 |
| clarkb | sigmavirus: can you be more specific on how it doesn't work? have job logs or similar? | 16:48 |
| clarkb | Jeffrey4l: because they have the same change id and the second change depends on itself | 16:48 |
| pabelanger | yes | 16:48 |
| sigmavirus | clarkb: I do. Hold for 1 s | 16:48 |
| clarkb | Jeffrey4l: so the cycle is change A -> change A | 16:48 |
| sigmavirus | http://logs.openstack.org/06/434506/3/check/gate-craton-tox-functional-ubuntu-xenial/2b1ccf3/console.html#_2017-02-16_16_02_44_870561 | 16:48 |
| Jeffrey4l | aha, got it. thanks. | 16:49 |
| clarkb | Jeffrey4l: so just unabanson the abandoned change, remove the depends on, then abandon it again and it should work again | 16:49 |
| Jeffrey4l | roger. thanks. | 16:49 |
| clarkb | sigmavirus: seems like you need to set a config of some sort? | 16:50 |
| sigmavirus | clarkb: yeah, this works fine locally (on a xenial vm) | 16:50 |
| clarkb | sigmavirus: as a guess its because we don't allow sudo in thosejobs | 16:51 |
| clarkb | sigmavirus: the idea being anything running "simple" tox invocation should run as arbitrary users | 16:51 |
| clarkb | and not own your laptop | 16:51 |
| sigmavirus | ah, is this why kolla does https://github.com/openstack/kolla/blob/master/tools/setup_gate.sh#L17 | 16:52 |
| clarkb | sigmavirus: so you may need to modify the job to add jenkins to the right gruop in order to talk to docker | 16:52 |
| *** wolverineav has joined #openstack-infra | 16:52 | |
| mordred | clarkb: I wonder if we shouldn't add jenkins to the docker group in general, even if docker isn't installed | 16:52 |
| mordred | or have a macro that installs docker and adds jenkins to the group that people can use | 16:53 |
| clarkb | mordred: a more generic group addition might help too. For kvm/docker/etc | 16:53 |
| mordred | yah | 16:53 |
| *** egonzalez has left #openstack-infra | 16:53 | |
| mordred | sigmavirus: I think there are a bunch of places in which we could have more glue/help for folks doing docker - so far a lot of it hasn't quite hit the tipping point where it's been implemented just yet | 16:54 |
| *** wolverin_ has joined #openstack-infra | 16:54 | |
| pabelanger | ++ | 16:54 |
| pabelanger | it would be great if we get an ansible-playbook in place once we get zuulv3 | 16:54 |
| *** martinkopec has quit IRC | 16:55 | |
| *** mdrabe has quit IRC | 16:55 | |
| sigmavirus | so, tl;dr, for now I should create a jenkins/jobs/craton.yaml and add jenkins to the docker group via that? | 16:55 |
| mordred | pabelanger: ++ | 16:55 |
| *** markus_z has quit IRC | 16:56 | |
| sigmavirus | or should I look for where to modify our disk images so that jenkins user is always part of teh docker group? | 16:56 |
| pabelanger | I'd say JJB macro for now | 16:56 |
| clarkb | sigmavirus: I think straightfoward fix is the first one. Define a job for your functional tests that has a step in it that adds jenkins to docker group be fore sudo goes away | 16:56 |
| pabelanger | over a DIB change | 16:57 |
| clarkb | or just don't remove sudio | 16:57 |
| *** wolverineav has quit IRC | 16:57 | |
| clarkb | and yes I don't think this should modify the disk images, should be handled at runtime around when we install packages | 16:57 |
| *** kzaitsev_mb has quit IRC | 16:57 | |
| *** kiltzman has joined #openstack-infra | 16:57 | |
| *** jordanP has quit IRC | 16:57 | |
| clarkb | we could do something generic and silly like diff old version of /etc/groups before package installs against new version after package installs and add jenkins to every new group | 16:58 |
| clarkb | to cover the kvm and other group cases | 16:58 |
| zxiiro | fungi: what about active contributors and reviewers who can't attend the ptg? | 16:58 |
| clarkb | (but also probably fine to explicitly check for and add to a known set of useful ones) | 16:58 |
| pabelanger | clarkb: mind a review again: https://review.openstack.org/#/c/430418/ | 16:59 |
| pabelanger | for zuulv3-dev | 16:59 |
| *** chlong has joined #openstack-infra | 16:59 | |
| *** mhickey has quit IRC | 17:00 | |
| clarkb | pabelanger: ya looking | 17:00 |
| *** mdrabe has joined #openstack-infra | 17:01 | |
| *** baoli has quit IRC | 17:01 | |
| *** derekh has quit IRC | 17:01 | |
| pabelanger | Hmm | 17:01 |
| pabelanger | we are building a large amount of nodes atm | 17:01 |
| pabelanger | upwards of 200+ | 17:01 |
| pabelanger | looks like a problem with centos-7 nodes | 17:02 |
| *** liusheng has quit IRC | 17:02 | |
| pabelanger | Hmm, DNS issues on osic-cloud1? | 17:03 |
| pabelanger | ;; connection timed out; trying next origin^ | 17:04 |
| *** ihrachys_ has joined #openstack-infra | 17:04 | |
| pabelanger | cloudnull: ^ is there any outages in osic-cloud1 ATM? | 17:04 |
| *** ihrachys has quit IRC | 17:04 | |
| pabelanger | looks like we are failing to resolve DNS | 17:04 |
| *** thorst_ is now known as thorst_afk | 17:05 | |
| cloudnull | pabelanger: not that i'm aware of. | 17:05 |
| *** derekh has joined #openstack-infra | 17:05 | |
| pabelanger | http://paste.openstack.org/show/599289/ | 17:05 |
| pabelanger | let me see when the issue started | 17:05 |
| pabelanger | maybe we have a bad image | 17:05 |
| clarkb | pabelanger: I don't think its dns | 17:05 |
| clarkb | but I can't hit cloud1.osic.org | 17:06 |
| clarkb | "The Connection Was Reset" says firefox | 17:06 |
| *** tqtran has joined #openstack-infra | 17:06 | |
| cloudnull | its working for me. | 17:06 |
| * cloudnull trying from elsewhere. | 17:06 | |
| *** esikache1 has quit IRC | 17:06 | |
| pabelanger | 2017-02-16 15:21:04,800 INFO nodepool.NodeLauncher: Creating server with hostname ubuntu-xenial-osic-cloud1-disk-7336874 in osic-cloud1-disk from image ubuntu-xenial for node id: 7336874 | 17:07 |
| pabelanger | is first log entry of failure | 17:07 |
| fungi | zxiiro: the travel support program is supposed to be one of the solutions to that (it's supposed to also be able to cover registration costs, i believe), and the other suggested mechanism is to get a speaker with an accepted talk to include you as a co-presenter | 17:08 |
| pabelanger | 2017-02-16 14:29:10 is the first failures of the day we seen in osic-cloud1 | 17:08 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-osic | 17:08 |
| clarkb | cloudnull: pabelanger ya DNS resolved for me, it was actual tcp connectivity that seemed to fail | 17:08 |
| clarkb | oh now it works | 17:08 |
| clarkb | maybe load balancer with bad backend only affecting percentage of connections? | 17:08 |
| pabelanger | http://logs.openstack.org/18/430418/5/check/gate-infra-puppet-apply-3-debian-jessie/e3910f9/console.html | 17:08 |
| cloudnull | maybe folks are doing somehthing within the env | 17:08 |
| pabelanger | ya, something network related | 17:08 |
| cloudnull | asking around. | 17:08 |
| pabelanger | I see some route failures in ^ | 17:09 |
| fungi | zxiiro: also i believe the plan is for future summits (after the pike summit at least) to provide passes to people who attended one of the past two ptgs, or two of the past four, or something along those lines... so not requiring that you go to every ptg to be able to get discounted admission to the summit/forum | 17:09 |
| *** dimtruck is now known as zz_dimtruck | 17:09 | |
| *** baoli has joined #openstack-infra | 17:10 | |
| fungi | zxiiro: that said, this is all very new and i'm not sure the events team has completely planned out how discounted admission is going to work. the reason we gave it to atcs in the past is because we wanted to encourage them to be present at the design summit. now that the meat of the design summit has been shifted to the ptg the idea there was that we use those funds to make it easier for | 17:11 |
| fungi | contributors to be present at the ptg | 17:11 |
| *** rbuzatu_ has quit IRC | 17:12 | |
| *** rbuzatu has joined #openstack-infra | 17:12 | |
| clarkb | pabelanger: found one thing on the zuulv3 change | 17:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Temporarily disable osic-cloud1 https://review.openstack.org/435028 | 17:14 |
| pabelanger | clarkb: cloudnull: in-case we need to ^ | 17:14 |
| pabelanger | clarkb: looking | 17:14 |
| pabelanger | clarkb: thanks, fixing | 17:14 |
| *** mudpuppy has joined #openstack-infra | 17:15 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add zuulv3-dev.openstack.org server https://review.openstack.org/430418 | 17:15 |
| *** rbuzatu_ has joined #openstack-infra | 17:16 | |
| pabelanger | clarkb: jeblair: ^ updates as requested | 17:16 |
| *** rbuzatu__ has joined #openstack-infra | 17:16 | |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config master: Add a new macro for adding the jenkins user to docker https://review.openstack.org/435033 | 17:17 |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config master: User new with-docker template for craton jobs https://review.openstack.org/435034 | 17:17 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate master: Ansibly the network sanity check https://review.openstack.org/404243 | 17:17 |
| sigmavirus | mordred: pabelanger clarkb ^ I added a new macro and made a new python-job template while I was at it | 17:17 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate master: Ansibly the network sanity check https://review.openstack.org/404243 | 17:17 |
| *** rbuzat___ has joined #openstack-infra | 17:18 | |
| *** rbuzatu has quit IRC | 17:19 | |
| cloudnull | pabelanger: mudpuppy: is looking into it. | 17:19 |
| mordred | sigmavirus: woot! | 17:19 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate master: Ansibly the network sanity check https://review.openstack.org/404243 | 17:19 |
| fungi | sigmavirus: mordred: clarkb: what about putting the `adduser jenkins docker` call into tools/test-setup.sh in the repo? many of our jobs already have a test-setup builder macro that calls that before sudo is revoked | 17:19 |
| *** rbuzatu_ has quit IRC | 17:20 | |
| cloudnull | sadly my day to day has changed such that I'm no longer working on the osic (for the most part). | 17:20 |
| *** rbuzatu has joined #openstack-infra | 17:20 | |
| fungi | that way it also provides some indication to local devs as to what changes need to be performed as root to correctly set up the test environment | 17:20 |
| *** jpich has quit IRC | 17:20 | |
| sigmavirus | fungi: in addition to the macro or instead of it? | 17:20 |
| *** rbuzatu__ has quit IRC | 17:21 | |
| fungi | (whether that's adding an admin user to mysql/postgres for nova's unit tests or mounting an xfs partition in a specific location for swift unit tests) | 17:21 |
| pabelanger | fungi: ++ I was just going to ask AJaeger about that | 17:21 |
| mordred | sigmavirus: TIL about gpasswd | 17:21 |
| *** e0ne has quit IRC | 17:22 | |
| fungi | sigmavirus: calling this macro in your job definition before revoke-sudo: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n947 | 17:22 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 17:22 |
| sigmavirus | mordred: found that solution on askubuntu franlky | 17:22 |
| *** rbuzat___ has quit IRC | 17:22 | |
| fungi | sigmavirus: and then putting special things your project's tests need as root-level prerequisites in that script in your repo | 17:22 |
| sigmavirus | ooooh | 17:23 |
| sigmavirus | i see | 17:23 |
| *** rbuzatu_ has joined #openstack-infra | 17:23 | |
| pabelanger | cloudnull: mudpuppy: thanks | 17:23 |
| fungi | sigmavirus: it's part of an ongoing effort to standardize our jobs in such a way as to also make it simpler for developers to repeat the same tests locally on their own systems | 17:23 |
| *** nadya has joined #openstack-infra | 17:24 | |
| *** kzaitsev_mb has joined #openstack-infra | 17:24 | |
| *** rbuzatu has quit IRC | 17:24 | |
| fungi | sigmavirus: nova has this, for example: http://git.openstack.org/cgit/openstack/nova/tree/tools/test-setup.sh | 17:24 |
| sigmavirus | fungi: got it, thaks | 17:24 |
| sigmavirus | *thanks | 17:24 |
| *** rbuzatu has joined #openstack-infra | 17:24 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: jjb: creates tripleo-puppet-scenario003-nv job https://review.openstack.org/433750 | 17:25 |
| clarkb | fungi: sigmavirus I like that approach if it means we don't need to edit/create jobs | 17:25 |
| sigmavirus | clarkb: seems that way | 17:25 |
| *** rbuzatu__ has joined #openstack-infra | 17:26 | |
| fungi | also some of this probably can turn into ansible playbooks in repos after zuul v3 is in production | 17:26 |
| clarkb | fungi: sigmavirus one potential downside is that it forces craton et al to know they need to do these things, BUT the upside of that is by writing it down they document it and should make it easier for other users down the road | 17:26 |
| *** nadya has quit IRC | 17:26 | |
| * sigmavirus agrees | 17:26 | |
| fungi | clarkb: another big up-side is that it makes changes to those prereqa self-testing | 17:26 |
| *** ihrachys_ has quit IRC | 17:26 | |
| fungi | er, prereqs | 17:26 |
| clarkb | fungi: ++ | 17:26 |
| *** rbuzat___ has joined #openstack-infra | 17:26 | |
| *** rbuzatu_ has quit IRC | 17:27 | |
| fungi | clarkb: really, the biggest downside (i agree with sdague on this one) is that it encourages copying and subsequent divergence of previously centralized/common functions like database setup, but i still feel like the pros outweigh the cons on that | 17:27 |
| *** Apoorva_ has joined #openstack-infra | 17:28 | |
| *** derekh has quit IRC | 17:28 | |
| *** kzaitsev_mb has quit IRC | 17:28 | |
| *** rbuzatu_ has joined #openstack-infra | 17:28 | |
| fungi | also, that ship is sailing with zuul v3 as well (we can and will still have central playbooks, but projects will also have their own in many cases which they may cargo-cult from other projects) | 17:28 |
| *** rbuzatu has quit IRC | 17:29 | |
| *** Apoorva has quit IRC | 17:29 | |
| *** kzaitsev_mb has joined #openstack-infra | 17:29 | |
| sdague | fungi: yeh, honestly, moving towards a more self service model here makes this all make more sense to me | 17:30 |
| jeblair | no need to cargo cult when you can include directly | 17:30 |
| *** rbuzatu has joined #openstack-infra | 17:30 | |
| fungi | and we already fail to prevent this in tox-based jobs with so many projects have custom pip-wrapper scripts they call out to from their tox.ini files | 17:30 |
| sdague | I'm honestly not super anti copy/paste, I just had no initial context | 17:30 |
| *** rbuzatu__ has quit IRC | 17:30 | |
| fungi | jeblair: right, ideally zuul v3 even reighns some of this back in | 17:30 |
| sdague | I mean, we all learn to code a new language by copy and pasting things | 17:30 |
| fungi | since it gives us an easier mechanism for users to locally source centralized job configuration | 17:31 |
| *** rbuzat___ has quit IRC | 17:31 | |
| *** rbuzatu__ has joined #openstack-infra | 17:31 | |
| fungi | making it less challenging for them to closely mimic what the ci system will run | 17:31 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate master: Ansibly the network sanity check https://review.openstack.org/404243 | 17:32 |
| *** rbuzat___ has joined #openstack-infra | 17:32 | |
| *** rbuzatu_ has quit IRC | 17:33 | |
| *** rbuzatu_ has joined #openstack-infra | 17:33 | |
| *** rbuzatu has quit IRC | 17:34 | |
| *** rbuzatu has joined #openstack-infra | 17:34 | |
| *** hashar has quit IRC | 17:35 | |
| *** rbuzatu__ has quit IRC | 17:35 | |
| *** rbuzatu__ has joined #openstack-infra | 17:35 | |
| *** jamesdenton has joined #openstack-infra | 17:36 | |
| *** rbuzat___ has quit IRC | 17:36 | |
| *** rbuzat___ has joined #openstack-infra | 17:37 | |
| AJaeger | sigmavirus, please see https://review.openstack.org/432786 - fungi, pabelanger, please review. That is an infra-manual change to document test-setup.sh | 17:37 |
| *** rbuzatu_ has quit IRC | 17:38 | |
| zxiiro | fungi: thanks for the explanation, that makes sense. | 17:38 |
| *** rbuzatu_ has joined #openstack-infra | 17:38 | |
| *** rbuzatu has quit IRC | 17:39 | |
| *** rbuzatu has joined #openstack-infra | 17:39 | |
| pabelanger | fungi: any thoughts on https://review.openstack.org/#/c/435028/ ? osic-cloud1 is looking into the outage, but we have jobs in the gate pipeline failing ATM. Were about 90mins into issues and have lots of cloud resources to spare | 17:39 |
| *** shardy is now known as shardy_afk | 17:40 | |
| *** rbuzatu__ has quit IRC | 17:40 | |
| *** jamesdenton has quit IRC | 17:40 | |
| *** rbuzatu__ has joined #openstack-infra | 17:40 | |
| *** rbuza____ has joined #openstack-infra | 17:41 | |
| *** kzaitsev_mb has quit IRC | 17:42 | |
| sigmavirus | hm, http://logs.openstack.org/38/435038/2/check/gate-craton-python35/21bd5f1/console.html#_2017-02-16_17_40_41_163651 | 17:42 |
| *** rbuzat___ has quit IRC | 17:42 | |
| pabelanger | docker isn't installed | 17:42 |
| *** rbuzat___ has joined #openstack-infra | 17:42 | |
| pabelanger | you'd need to add it to bindep.txt | 17:42 |
| sigmavirus | pabelanger: okay, I didn't see any other projects with it in bindep | 17:42 |
| *** rbuzatu_ has quit IRC | 17:42 | |
| pabelanger | sigmavirus: ya, its possible there is more work you need to do. Where do you want to install docker from? | 17:43 |
| fungi | sigmavirus: they're probably using very specially customized jobs instead | 17:43 |
| pabelanger | I know they publish their own apt repos | 17:43 |
| *** rbuzatu_ has joined #openstack-infra | 17:43 | |
| sigmavirus | I think xenial's docker is fine for us | 17:43 |
| *** rbuzatu has quit IRC | 17:43 | |
| fungi | pabelanger: fast-approved | 17:44 |
| fungi | pabelanger: do you have a revery already proposed too so we can open it back up once they've got a diagnosis/cure? | 17:44 |
| fungi | s/revery/revert/ | 17:44 |
| *** rbuzatu__ has quit IRC | 17:45 | |
| *** rbuzatu has joined #openstack-infra | 17:45 | |
| *** rbuzatu__ has joined #openstack-infra | 17:46 | |
| pabelanger | fungi: not yet, I can do that now | 17:46 |
| *** rbuza____ has quit IRC | 17:46 | |
| *** rbuzat___ has quit IRC | 17:47 | |
| *** rbuzat___ has joined #openstack-infra | 17:47 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Revert "Temporarily disable osic-cloud1" https://review.openstack.org/435047 | 17:47 |
| jeblair | pabelanger: what do you mean 'jobs in the gate pipeline failing' ? | 17:47 |
| jeblair | pabelanger: (what's causing the failure?) | 17:48 |
| pabelanger | jeblair: is an example of the networking issues: http://logs.openstack.org/20/434920/2/gate/gate-puppet-panko-puppet-lint/5484ded/console.html#_2017-02-16_17_01_28_535416 | 17:48 |
| *** rbuza____ has joined #openstack-infra | 17:48 | |
| *** rbuzatu_ has quit IRC | 17:48 | |
| fungi | we hit a hung 'npm install' call in osic on a release job as one example http://logs.openstack.org/45/452b4cb427a86e4b097b9711fb045a211b5e7039/release/tripleo-ui-nodejs6-npm-publish-tarball/6ec2420/console.html#_2017-02-16_16_51_35_319015 | 17:48 |
| pabelanger | jeblair: we think a networking outage, but waiting for confirmation | 17:48 |
| pabelanger | osic-cloud1 is looking into it | 17:48 |
| jeblair | oh, got it. i thought the problems were cloud api (eg, nodepool launch) issues | 17:48 |
| jeblair | i'm caught up now | 17:48 |
| fungi | seems it's both | 17:49 |
| *** rbuzatu_ has joined #openstack-infra | 17:49 | |
| fungi | or rather possibly one problem impacting both cloud networking and api networking there | 17:49 |
| pabelanger | Ya, we don't have a root cause ATM | 17:49 |
| *** rbuzatu has quit IRC | 17:49 | |
| *** rbuzatu has joined #openstack-infra | 17:50 | |
| *** rbuzatu__ has quit IRC | 17:50 | |
| openstackgerrit | Merged openstack-infra/project-config master: Temporarily disable osic-cloud1 https://review.openstack.org/435028 | 17:50 |
| *** zz_dimtruck is now known as dimtruck | 17:50 | |
| *** rbuzat___ has quit IRC | 17:51 | |
| *** rbuzatu__ has joined #openstack-infra | 17:51 | |
| AJaeger | mtreinish: could you check https://review.openstack.org/434625 , please? ianw is adding subunit for coverage. | 17:51 |
| *** rbuza____ has quit IRC | 17:52 | |
| *** rbuzat___ has joined #openstack-infra | 17:52 | |
| *** mriedem1 has joined #openstack-infra | 17:53 | |
| *** ihrachys_ has joined #openstack-infra | 17:53 | |
| *** rbuzatu_ has quit IRC | 17:53 | |
| mtreinish | AJaeger: sure I'll take a look | 17:53 |
| *** rbuzatu_ has joined #openstack-infra | 17:54 | |
| *** rbuzatu has quit IRC | 17:54 | |
| *** rhallisey has quit IRC | 17:55 | |
| *** rwsu has joined #openstack-infra | 17:55 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:55 | |
| *** rbuzatu has joined #openstack-infra | 17:55 | |
| *** mriedem has quit IRC | 17:55 | |
| *** rhallisey has joined #openstack-infra | 17:55 | |
| *** rbuzatu__ has quit IRC | 17:56 | |
| *** dtantsur is now known as dtantsur|afk | 17:56 | |
| *** rbuzatu__ has joined #openstack-infra | 17:56 | |
| *** rbuzat___ has quit IRC | 17:56 | |
| *** ihrachys_ has quit IRC | 17:57 | |
| *** rbuzat___ has joined #openstack-infra | 17:58 | |
| *** rbuzatu_ has quit IRC | 17:59 | |
| *** rbuzatu_ has joined #openstack-infra | 17:59 | |
| *** rbuzatu has quit IRC | 17:59 | |
| *** esikache1 has joined #openstack-infra | 17:59 | |
| *** Apoorva_ has quit IRC | 17:59 | |
| AJaeger | thanks, mtreinish | 18:00 |
| *** rbuzatu has joined #openstack-infra | 18:00 | |
| *** rbuzatu__ has quit IRC | 18:01 | |
| *** ihrachys has joined #openstack-infra | 18:01 | |
| *** rbuzatu__ has joined #openstack-infra | 18:01 | |
| *** rbuzat___ has quit IRC | 18:02 | |
| *** armax has joined #openstack-infra | 18:02 | |
| *** flepied has quit IRC | 18:02 | |
| *** rbuzat___ has joined #openstack-infra | 18:02 | |
| *** rbuza____ has joined #openstack-infra | 18:03 | |
| *** rbuzatu_ has quit IRC | 18:03 | |
| *** esikache1 has quit IRC | 18:04 | |
| *** salv-orl_ has joined #openstack-infra | 18:04 | |
| *** rbuzatu_ has joined #openstack-infra | 18:04 | |
| *** rossella_s has quit IRC | 18:04 | |
| *** rbuzatu has quit IRC | 18:05 | |
| clarkb | EmilienM: sshnaidm: digging more on the logs front tripleo changes do seem to pop up frequently in the large users of disk. Some of that is quite a few jobs can be run per patchset, but even then a single job is capable of easily generating 400MB on its own. This seems largely due to the collection of all the things on the hosts. Out of curiousity has that been valuable? What are we trying to achieve | 18:05 |
| clarkb | by doing that? Wondering if there is some deeper issue we could be addressing | 18:05 |
| *** rbuzatu__ has quit IRC | 18:06 | |
| *** salv-orlando has quit IRC | 18:07 | |
| *** rbuzatu has joined #openstack-infra | 18:07 | |
| *** rbuzat___ has quit IRC | 18:07 | |
| mwhahaha | it looks like the osic cloud is having issues talking to rubygems, http://logs.openstack.org/65/434265/1/gate/gate-puppet-openstack-integration-4-scenario003-tempest-ubuntu-xenial/66bbd79/console.html#_2017-02-16_17_03_18_558061 | 18:07 |
| mwhahaha | we've got a bunch of failures on that | 18:07 |
| mwhahaha | not sure who to poke | 18:07 |
| EmilienM | mwhahaha: it's already discussed, | 18:07 |
| EmilienM | pabelanger | osic-cloud1 is looking into it | 18:07 |
| *** rbuza____ has quit IRC | 18:07 | |
| * mwhahaha learns to scroll | 18:07 | |
| EmilienM | they have a networking outage it seems | 18:07 |
| EmilienM | clarkb: I agree, we talked about it this morning during our weekly meeting | 18:08 |
| EmilienM | clarkb: 400 MB is way too much and we'll reduce it to maximum | 18:08 |
| *** leifmadsen_ has joined #openstack-infra | 18:08 | |
| sshnaidm | clarkb, I'm aware of this issue, I try to address it and reduce amount of collected logs in patch: https://review.openstack.org/#/c/425459/ | 18:08 |
| *** rustlebee has joined #openstack-infra | 18:08 | |
| *** rbuzatu_ has quit IRC | 18:09 | |
| sshnaidm | clarkb, if it won't go well, we will collect specific folders only, but I hope it'll be enough | 18:10 |
| *** leifmadsen has quit IRC | 18:11 | |
| sshnaidm | clarkb, do you see a particular job that takes a lot space? or does it happen with all jobs? | 18:11 |
| clarkb | sshnaidm: well mostly wondering what the intent or goal is behind it | 18:11 |
| *** russellb has quit IRC | 18:12 | |
| clarkb | sshnaidm: do you end up needing all those logs regularly to debug issues? | 18:12 |
| sshnaidm | clarkb, we take partially /etc and /var/log from each of servers (from 3 to 5) | 18:12 |
| sshnaidm | clarkb, I'm afraid yes, we do | 18:12 |
| sshnaidm | clarkb, it's just a lot of servers, that's why so many logs.. | 18:13 |
| *** dhinesh has joined #openstack-infra | 18:13 | |
| fungi | sshnaidm: was following symlinks when copying intentional? or did you just need to know what paths those symlinks target? | 18:13 |
| clarkb | well not even devstack gate runs collect that much though | 18:13 |
| *** lucasagomes is now known as lucas-afk | 18:14 | |
| clarkb | and I think we collect a useable subset to debug most problems there | 18:14 |
| pabelanger | Temporarily disable osic-cloud1 is now live | 18:14 |
| sshnaidm | fungi, yes, it's intentional, but it's only few symlinks there, I remember neutorn configs symlinks only | 18:14 |
| fungi | sshnaidm: i agree copying /etc is generally not risky, but copying all files outside of /etc that may be the targets of symlinks in /etc can quickly go very wrong | 18:15 |
| AJaeger | pabelanger, clarkb, fungi : Could you review https://review.openstack.org/429868 - tacker python-db change, please? | 18:15 |
| *** esikache1 has joined #openstack-infra | 18:15 | |
| sshnaidm | fungi, we don't copy everything in etc, usually excluding all big folders, puppet modules, etc | 18:15 |
| *** jamesdenton has joined #openstack-infra | 18:16 | |
| fungi | i still think that a blacklist approach instead of a whitelist is dangerous, but i guess once we get size limits implemented in zuul we'll just refuse to copy logs for your jobs if they make poor assumptions there | 18:17 |
| sshnaidm | clarkb, devstack copies only limited set of /etc folders iirc, not including those we need, ironic logs for example | 18:17 |
| pabelanger | fungi: Ya, I think that is the safe approach moving forward | 18:17 |
| *** sambetts is now known as sambetts|afk | 18:18 | |
| sshnaidm | fungi, we'll switch to whitelist if problem will continue, but blacklist was always the approach, although when we rework all our ci now it's possible to happen sometimes that some defaults is changes and we are stuck with GBs of logs | 18:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Reload tenant configuration on change merged https://review.openstack.org/426433 | 18:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Correct playbook requirement logic https://review.openstack.org/434564 | 18:18 |
| clarkb | sshnaidm: devstack-gate collects ironic logs when we test ironic | 18:19 |
| pabelanger | #status log osic-cloud1 temporarily disable. Currently waiting for root cause of networking issues. | 18:19 |
| openstackstatus | pabelanger: finished logging | 18:19 |
| sshnaidm | fungi, I mean when we are in heavy CI development it could be an issue *sometimes* that we'll address, it's not going to be always | 18:19 |
| *** esikache1 has quit IRC | 18:19 | |
| *** mptacekx has quit IRC | 18:20 | |
| * clarkb is working on pulling up a specific example as that may be easier to talk about | 18:20 | |
| clarkb | https://review.openstack.org/#/c/434250/ used 9GB of log storage | 18:20 |
| *** nadya has joined #openstack-infra | 18:20 | |
| clarkb | 4.4GB are from patchset 3 | 18:20 |
| *** jamesdenton has quit IRC | 18:21 | |
| clarkb | 1.7 of that is from check-tripleo, 1.5 from experimental | 18:21 |
| clarkb | gate-tripleo-ci-centos-7-ovb-ha-oooq-nv used 620MB from two job runs | 18:21 |
| jimbaker | hopefully we can have someone look at https://review.openstack.org/#/c/434979/ - the recent addition to our tests of a functional testing gate is not working properly | 18:22 |
| *** jkilpatr has quit IRC | 18:22 | |
| clarkb | quickstarts log alone appears to be 24MB compressed | 18:22 |
| clarkb | with the bulk of log use in oooq/collected_logs | 18:22 |
| *** jkilpatr has joined #openstack-infra | 18:22 | |
| jimbaker | so as the project lead for craton, i do want the project to be able to start merging in code again :) | 18:22 |
| AJaeger | jimbaker: sigmavirus was fixing that with tools/test-setup.sh . Is that not working? | 18:23 |
| jimbaker | AJaeger, not yet | 18:23 |
| sshnaidm | clarkb, in devstack I see only those:http://logs.openstack.org/48/422448/12/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/1c83062/logs/etc/ | 18:23 |
| jimbaker | so making this non voting for the moment seems to be the best course of action | 18:23 |
| *** ihrachys has quit IRC | 18:23 | |
| sshnaidm | clarkb, yeah, it's a problem with all those logs, we try to resolve it right now | 18:24 |
| clarkb | jimbaker: I believe sigmavirus is fixing that | 18:24 |
| clarkb | jimbaker: using just a change to craton | 18:24 |
| clarkb | sshnaidm: if ironic etc config is missing thats probably a bug that could be fixed | 18:25 |
| clarkb | looks like 16MB is just ceilometer | 18:25 |
| jimbaker | clarkb, that was the hope, and sigmavirus did initially -1 workflow on https://review.openstack.org/#/c/434979/ - but so far the debugging has not worked out | 18:25 |
| *** mriedem1 is now known as mriedem | 18:25 | |
| AJaeger | jimbaker: do you have a link to sigmavirus change for craton? | 18:25 |
| jimbaker | in our meeting that just concluded, we agreed that we should make nonvoting | 18:25 |
| jimbaker | AJaeger, it's here: https://review.openstack.org/#/c/435038/ | 18:26 |
| *** esikache1 has joined #openstack-infra | 18:26 | |
| clarkb | jimbaker: sigmavirus can you explain why the fix wouldn't work, my understanding from the conversation we just had was woo simple fix do it | 18:26 |
| *** Swami_ has joined #openstack-infra | 18:26 | |
| clarkb | sshnaidm: maybe reduce the verbosity of ceilometer logging (not sure how often you need full debug) | 18:26 |
| clarkb | jimbaker: sigmavirus add docker to bindep, use test-setup.sh to add jenkins user to docker group? | 18:27 |
| *** nadya has quit IRC | 18:27 | |
| jimbaker | clarkb, i think this may be a helpful fix, but my current theory is that we may be seeing some sort of race that was previously not seen on developer's personal machines | 18:27 |
| clarkb | as an alternative skip your tests that fail | 18:27 |
| *** rbuzatu has quit IRC | 18:27 | |
| clarkb | or mark them as expected fails | 18:27 |
| AJaeger | jimbaker: I think you install too few files now - the bindep file overrides the default file. | 18:28 |
| AJaeger | jimbaker: see project-config file jenkins/data/bindep-fallback.txt for the default | 18:28 |
| jimbaker | AJaeger, ok, so that's something we can try as well | 18:29 |
| jimbaker | sigmavirus, ^^^ | 18:29 |
| *** jamesdenton has joined #openstack-infra | 18:30 | |
| sshnaidm | clarkb, do you see it taking space? my usual undercloud /var/log looks like: http://paste.openstack.org/show/599298/ | 18:30 |
| *** esikache1 has quit IRC | 18:30 | |
| pabelanger | sshnaidm: var/log/journal can be dropped | 18:32 |
| pabelanger | we just did that in tripleo-ci | 18:32 |
| *** esikache1 has joined #openstack-infra | 18:32 | |
| sshnaidm | pabelanger, afaik we convert it to text and drop binary (at least it should be so) | 18:32 |
| *** claudiub has quit IRC | 18:33 | |
| sshnaidm | pabelanger, it's from some old job run, so maybe it didn't reflect last changes.. | 18:33 |
| pabelanger | sshnaidm: possible, the change just merged yesterday | 18:33 |
| sshnaidm | pabelanger, yeah, this is much before | 18:34 |
| clarkb | sshnaidm: ya trying to get a full du -ah listing shared out but its 1.2MB large so I think paste.o.o will be mad at it | 18:34 |
| *** jamesdenton has quit IRC | 18:34 | |
| sshnaidm | clarkb, gist on github? | 18:34 |
| clarkb | sshnaidm: will that do 1.2MB /me tries | 18:34 |
| *** jaosorior has quit IRC | 18:34 | |
| *** esikache1 has quit IRC | 18:36 | |
| *** Swami has joined #openstack-infra | 18:36 | |
| *** Swami_ has quit IRC | 18:38 | |
| clarkb | sshnaidm: https://gist.github.com/cboylan/254a072910b0559cf2cf4b5b25c50a1b | 18:39 |
| sshnaidm | clarkb, thanks, looking | 18:39 |
| clarkb | sshnaidm: I think you can likely stop collecting all of that /var/lib/mysql stuff | 18:39 |
| clarkb | we aren't testing mysql | 18:39 |
| clarkb | (though thats not a huge amount of disk) | 18:40 |
| sshnaidm | clarkb, yes, I've excluded it in the patch | 18:40 |
| clarkb | I would also argue you don't need all of the tripleo heat templates to be logged | 18:40 |
| clarkb | you have a git sha1 for that | 18:40 |
| sshnaidm | clarkb, yeah, I think it could be dropped too.. | 18:41 |
| clarkb | what is an ib_logfile? | 18:42 |
| clarkb | ./logs/oooq/quickstart.tar.gz is bigish too | 18:43 |
| *** SumitNaiksatam has quit IRC | 18:43 | |
| *** ssbarnea has quit IRC | 18:43 | |
| *** sabayonuser has quit IRC | 18:43 | |
| *** Swami has quit IRC | 18:43 | |
| *** esikache1 has joined #openstack-infra | 18:44 | |
| *** dharinic is now known as dharinic|lunch | 18:44 | |
| *** kzaitsev_mb has joined #openstack-infra | 18:44 | |
| clarkb | I'm also kind of impressed that the zuul ansible dir is so large | 18:45 |
| clarkb | jeblair: maybe we can get zuul to gzip ^ | 18:45 |
| sshnaidm | clarkb, yeah, will drop it too.. | 18:45 |
| *** tosky has quit IRC | 18:46 | |
| clarkb | sshnaidm: is quickstart.tar.gz a tarballs of quickstart repo? | 18:46 |
| clarkb | if so ya I think you can drop that | 18:46 |
| sshnaidm | clarkb, it's ansible venv | 18:46 |
| openstackgerrit | Honza Pokorny proposed openstack-infra/project-config master: Be verbose when running npm commands https://review.openstack.org/435063 | 18:46 |
| jeblair | clarkb: sorry, how big is it? | 18:46 |
| clarkb | jeblair: 5MB for this one job (so overall small just bigger than I expected) | 18:47 |
| pabelanger | looks like ansible_log.txt is the issue | 18:47 |
| pabelanger | 7.7MB in this folder | 18:47 |
| jeblair | heh, probably because it's listing all the files it's copying. | 18:47 |
| clarkb | sshnaidm: oh definitely I would drop any build artifacts like that, you should be able to rebuild that venv using constraints later if necessary | 18:47 |
| pabelanger | jeblair: yes | 18:48 |
| clarkb | jeblair: that would explain 1.2MB ish of it yes :) | 18:48 |
| jeblair | clarkb: probably twice | 18:48 |
| jeblair | node->launcher->static | 18:49 |
| clarkb | oh right so 2.4 ish :) | 18:49 |
| sshnaidm | clarkb, usually yes, but when we are in transition to ansible now, everything is being changed and developed, so we need sometimes artifacts there for debug, like generated hosts file, ssh configs, and all other generated config | 18:49 |
| jeblair | clarkb: a more typical ansible_log file from a devstack-gate job is 24k | 18:50 |
| sshnaidm | clarkb, I mean we are in transition stage now, so it won't be always the case, fixing issues as they come asap | 18:50 |
| clarkb | sshnaidm: pabelanger even compressed the journal file is still fairly big, I wonder if it would be smaller if you did something similar to devstack-gate where only the journal logs starting from when the job started are collected as text then compressed | 18:50 |
| clarkb | jeblair: ya thats why it jumped out at me | 18:50 |
| jeblair | clarkb: sorry maybo 200k | 18:50 |
| jeblair | (i misread a compressed version) | 18:50 |
| *** ralonsoh has quit IRC | 18:51 | |
| clarkb | sshnaidm: also galera caches are pretty big those are in /var/lib/mysql so if you drop that it would be good | 18:51 |
| pabelanger | clarkb: yes, I patched tripleo-ci yesterday to exclude var/log/journal | 18:51 |
| pabelanger | patch should be live now | 18:51 |
| clarkb | pabelanger: to exclude or compress it? I see it compressed | 18:52 |
| sshnaidm | clarkb, mysql is already dropped in my patch | 18:52 |
| pabelanger | exclude, where are you looking? | 18:52 |
| clarkb | pabelanger: in the gisted du -ah above | 18:52 |
| *** Apoorva has joined #openstack-infra | 18:52 | |
| sshnaidm | clarkb, yes, the approach for journal logs should be as you said, I'll check if it's already so.. | 18:52 |
| clarkb | sshnaidm: is that merged? | 18:52 |
| clarkb | sshnaidm: sorry is the mysql change merged? | 18:52 |
| pabelanger | looking | 18:53 |
| sshnaidm | clarkb, it's waiting for gate pass https://review.openstack.org/#/c/425459/ | 18:53 |
| clarkb | ok | 18:53 |
| *** esikache1 has quit IRC | 18:53 | |
| clarkb | also there are a bunch of extra/lsof.txt.gz files that are pretty big too, what are we lsofing and do we need to be doing that? | 18:53 |
| *** matrohon has joined #openstack-infra | 18:54 | |
| clarkb | ./logs/undercloud/var/log/messages should be compressed too | 18:54 |
| sshnaidm | pabelanger, clarkb, I think it's unexpected /etc/alternatives on nodepool image that could take a few GBs made problems | 18:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Change request_id logging to match nova format https://review.openstack.org/434916 | 18:54 |
| *** e0ne has joined #openstack-infra | 18:54 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Actually normalize nova usage data https://review.openstack.org/434481 | 18:54 |
| sshnaidm | clarkb, it's already compressed as all other text files | 18:55 |
| *** jsho has quit IRC | 18:55 | |
| *** matrohon has quit IRC | 18:55 | |
| clarkb | sshnaidm: ok so I'm just looking at an older run? | 18:55 |
| pabelanger | clarkb: maybe this is limited to quickstart logs, sshnaidm would know more. The change I made was https://review.openstack.org/#/c/434467/ | 18:55 |
| sshnaidm | clarkb, yeah, maybe | 18:55 |
| clarkb | sshnaidm: can you point me at a job run log that you expect to be better? I can grab the du out put from that to confirm | 18:55 |
| clarkb | audit logs are also large in the current example | 18:55 |
| clarkb | (are you running with selinux enabled?) | 18:55 |
| sshnaidm | clarkb, I suggest to wait for https://review.openstack.org/#/c/425459/ to finish jobs, it should be the ideal one :) | 18:56 |
| mordred | in case anyone is wondering: "reckec" is not valid recheck syntax | 18:56 |
| clarkb | also is logs/undercloud.tar.xz a tarball of all the files under /logs/undercloud/ ? | 18:56 |
| clarkb | sshnaidm: ok I can check that one | 18:56 |
| clarkb | sshnaidm: if undercloud.tar.xz is a tarball of /logs/undercloud/ can we delete one of them? | 18:57 |
| clarkb | we don't need duplicate data | 18:57 |
| sshnaidm | clarkb, it's both with /etc and /var/log | 18:57 |
| *** saibarspeis has quit IRC | 18:57 | |
| clarkb | sshnaidm: which is also included as flat files and not a tarball so my suggestion would be to remove one or the other | 18:57 |
| sshnaidm | clarkb, sometimes it's better to point to particular file in web or index it in ELK | 18:58 |
| clarkb | sshnaidm: then delete the tarball? | 18:58 |
| clarkb | my point is we shouldn't be copying everything twice | 18:58 |
| pabelanger | ++ | 18:58 |
| sshnaidm | clarkb, but sometimes you can find something only with grep -r | 18:58 |
| *** beslemon has joined #openstack-infra | 18:58 | |
| sshnaidm | clarkb, downloading with crawler could be a bit challenging.. | 18:58 |
| clarkb | sshnaidm: but is possible | 18:58 |
| clarkb | (I've done it before) | 18:59 |
| *** jsho has joined #openstack-infra | 18:59 | |
| clarkb | sshnaidm: we don't have infinite disk space | 18:59 |
| clarkb | sshnaidm: we used to be able to have 6 months of logs now we have 6 weeks | 18:59 |
| clarkb | I'm sorry but we shouldn't be copying things twice | 18:59 |
| fungi | in fact, we have precisely 12tib of disk space, which lately seems to be enough for 6-8 weeks of job logs across all our projects | 18:59 |
| *** johndperkins has joined #openstack-infra | 18:59 | |
| sshnaidm | pabelanger, maybe we can host logs on rh1 or other place? | 19:01 |
| beslemon | hi all | 19:01 |
| beslemon | Hello My name is Besan and I am a performance engineer at RAX. I had a few qs about a gating jobs run for upstream openstack if you had the time | 19:01 |
| clarkb | beslemon: go for it | 19:01 |
| beslemon | So my question is is there an easy way that you know of to point the gating jobs to run on a specific server or an ansible script that has the tests/tools packaged for easy execution | 19:02 |
| pabelanger | if logging is an issue, I would suggest maybe we revisit the move to 3rd party CI | 19:02 |
| clarkb | beslemon: I don't know if it is easy, but every job now publishes the ansible playbooks used to run the job which could in theory be used to run the job elsewhere | 19:02 |
| clarkb | beslemon: I don't think we have wanted to support that as a real thing until zuul v3 though. | 19:03 |
| pabelanger | but, I would not upload logs to tripleo-test-cloud-rh1, it is already running private infrastructure I'd like to see move to public | 19:03 |
| sshnaidm | pabelanger, I'm trying to explain that it's transition errors while we change our CI. | 19:03 |
| clarkb | beslemon: most other jobs have an easy way to run them though. For example pep8 and python unittests should mostly just be tox -e$tox_env and devstack-gate jobs publish a reproduce.sh script that you can run | 19:03 |
| beslemon | yeah exactly | 19:04 |
| beslemon | ^ clarkb | 19:04 |
| beslemon | so is anyone working on this task clarkb | 19:04 |
| *** mdurrant has quit IRC | 19:04 | |
| pabelanger | sshnaidm: Right, I think we understand that. I'm not sure I understand the objections to clarkb and fungi requests | 19:04 |
| sshnaidm | pabelanger, if I knew way to switch CI to ansible in one moment without any error and in ideal way - I would do so. | 19:04 |
| fungi | beslemon: see http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html | 19:04 |
| pabelanger | I agree, we shouldn't have duplicate data in logs.o.o | 19:04 |
| sigmavirus | AJaeger: so not having the defaults wasn't the problem =/ | 19:05 |
| clarkb | sshnaidm: I understand that while we may be in a transition period for those jobs they are also consuming large amounts of disk for which we have a finite amount. So we'll need to be more careful about what we log | 19:05 |
| beslemon | clarkb: in regards to the job publishing their playbooks, where is this? | 19:05 |
| pabelanger | sshnaidm: keep in mind, we as the openstack-infra team, will also be doing that shortly once zuulv3 is production ready. We'll then have a collection of roles / playbooks for more native ansible things | 19:06 |
| fungi | beslemon: it's a pretty lengthy specification, but details the currentlt ongoing zuul v3 development work which in part implements what you're seeking | 19:06 |
| clarkb | beslemon: in the _zuul_ansible dir example: http://logs.openstack.org/48/422448/12/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/1c83062/_zuul_ansible/ | 19:06 |
| clarkb | sshnaidm: pabelanger my personal concern is that rather than add specific things we need to debug issues we seem to have decided we just need everything and then in some cases made multiple copies of everything | 19:07 |
| clarkb | sshnaidm: pabelanger a better appraoch imo would be to start with what you know you need and grow from there and only keep a single copy | 19:07 |
| pabelanger | clarkb: yes, I agree with at approach too | 19:07 |
| beslemon | clarkb: let me see | 19:07 |
| *** bh526r has quit IRC | 19:08 | |
| AJaeger | sigmavirus: at least not the only one ;) | 19:08 |
| *** esikache1 has joined #openstack-infra | 19:09 | |
| fungi | sshnaidm: which is what we have done with other jobs. we start out just collecting a few files we know are useful, we hit a problem in one change/job where we have insufficient information to debug it, then we adjust teh job to collect the files which might have been helpful to track down that specific issue and try again. over time you end up with the set of files useful for diagnosing actual issues | 19:09 |
| fungi | you've encountered rather than speculating you might need some files in the future with no basis in past experience | 19:09 |
| openstackgerrit | Russell Bryant proposed openstack-infra/project-config master: networking-ovn: Reorganize CI jobs. https://review.openstack.org/394684 | 19:09 |
| AJaeger | sigmavirus: move the job to experimental for now then... | 19:09 |
| sigmavirus | AJaeger: is that different from non-voting? | 19:10 |
| *** ZZelle_ has joined #openstack-infra | 19:10 | |
| sigmavirus | I'm sorry for the naive questions, I'll go find the manual :) | 19:10 |
| AJaeger | sigmavirus: it runs only on demand - when you type "check experimental" | 19:10 |
| *** jamesdenton has joined #openstack-infra | 19:10 | |
| *** SumitNaiksatam has joined #openstack-infra | 19:10 | |
| sshnaidm | fungi, clarkb pabelanger I just want to emphasize that OOO integrates a lot of openstack services and projects, so the error could be in one of them and investigating CI failure without all logs available is a nightmare | 19:11 |
| beslemon | clarkb: ill take a look thank you | 19:11 |
| AJaeger | sigmavirus: https://docs.openstack.org/infra/manual/creators.html#adding-a-new-job | 19:11 |
| sshnaidm | it's a bit different from one-service-job | 19:11 |
| clarkb | sshnaidm: the problem is its more than just all logs | 19:11 |
| clarkb | sshnaidm: you are collecting build artifacts, database tables as written to disk, virtualenvs, etc and in some cases have multiple copies of said items | 19:12 |
| clarkb | sshnaidm: definitely collect the openstack configs and logs. Possibly adjust their verbosity as necessary | 19:12 |
| fungi | sshnaidm: also you just described devstack-gate which collects specific files known to be useful in debugging the issues experienced in jobs rather than copying everything. and what i described is how we iterated on determining what logs it should archive | 19:12 |
| clarkb | then run the tests, when you can't debug an issue because something is missing add it in and go from there | 19:12 |
| sshnaidm | clarkb, about you mentioned I agree to remove either it was already removed, although while developing CI we may need some artifacts which we don't need in usual way | 19:13 |
| pabelanger | sshnaidm: I am still struggling to find the reason for compressed logs (tar.gz file) and extracted file. What is the reason for both? | 19:13 |
| clarkb | sshnaidm: right but don't assert you need them until you need them | 19:13 |
| clarkb | sshnaidm: add them in once they become useful | 19:13 |
| fungi | sshnaidm: but also i agree with pabelanger that special needs like archiving the majority of your build platform on every run is probably better tackled in a third-party ci system and not upstream | 19:13 |
| sshnaidm | fungi, pabelanger I'd happy to host logs anywhere else, actually I don't mind where developers can look at them while they can | 19:14 |
| fungi | sshnaidm: cool. a third-party ci system can absolutely put logs wherever it wants | 19:15 |
| *** esikache1 has quit IRC | 19:15 | |
| pabelanger | sshnaidm: fungi: yes, something we've been trying to implement for a while | 19:15 |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config master: Make craton functional tests experimental https://review.openstack.org/434979 | 19:15 |
| openstackgerrit | Merged openstack-infra/system-config master: Add zuulv3-dev.openstack.org server https://review.openstack.org/430418 | 19:15 |
| sshnaidm | pabelanger, I suggest to take it to tripleo weekly meeting | 19:16 |
| sshnaidm | EmilienM, ^^ | 19:16 |
| pabelanger | sshnaidm: I have many times before :) | 19:16 |
| sigmavirus | Thanks for all of your help AJaeger clarkb pabelanger and fungi | 19:16 |
| fungi | sigmavirus: any time! | 19:16 |
| *** tnovacik has joined #openstack-infra | 19:16 | |
| pabelanger | sigmavirus: current plan is to use zuul.rdoproject.org however, the software-factory team needs to do work. | 19:16 |
| pabelanger | sigmavirus: sorry | 19:16 |
| pabelanger | sshnaidm: : current plan is to use zuul.rdoproject.org however, the software-factory team needs to do work. | 19:17 |
| sigmavirus | :) no worries pabelanger | 19:17 |
| pabelanger | sshnaidm: what I would suggest, we can use the issue of large log files as a new reason to restart the effort | 19:17 |
| EmilienM | fungi, pabelanger, sshnaidm : my main concern is the fact we need to have logstash able to read these logs | 19:17 |
| AJaeger | anybody wants to +2A sigmavirus ' change to make his job experimental? https://review.openstack.org/#/c/434979/ | 19:17 |
| fungi | EmilienM: we have puppet modules for deploying that too | 19:17 |
| clarkb | looks like https://review.openstack.org/#/c/425459/10 shaved ~125MB of logs per oooq job | 19:18 |
| clarkb | workign on getting a du -ah posted | 19:18 |
| EmilienM | fungi: we could use them for sure | 19:18 |
| *** Apoorva has quit IRC | 19:19 | |
| pabelanger | EmilienM: yes, there is some infrastructure that would need to be managed | 19:19 |
| *** Apoorva has joined #openstack-infra | 19:19 | |
| fungi | the flip side is that the vast majority of what these jobs are collecting isn't (and can't be) indexed in logstash anyway | 19:19 |
| *** dprince has quit IRC | 19:19 | |
| pabelanger | okay, going to start work on zuulv3-dev.o.o | 19:19 |
| fungi | thanks for getting that going pabelanger! this is going to be the best ptg ever ;) | 19:20 |
| AJaeger | I'm done to three repos using python-db in project-config. Getting this to 2 with https://review.openstack.org/429868 - tacker is ready. Please review | 19:20 |
| AJaeger | my English ;( I meant: I'm down to... | 19:21 |
| fungi | your english is fine. also change lgtm | 19:21 |
| clarkb | sshnaidm: pabelanger fungi http://paste.openstack.org/show/599301/ is truncated du output for http://logs.openstack.org/59/425459/10/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-nv/d5cab22/ | 19:21 |
| AJaeger | ;) thanks, fungi | 19:21 |
| AJaeger | this leaves trove (broken newton branch) and murano (Broken mitaka branch) | 19:21 |
| fungi | AJaeger: i'm surprised so few projects actually bitrotted on stable branches for this. upper constraints has made a huge improvement | 19:22 |
| clarkb | sshnaidm: looks like removing the quickstart venv would be a relatively large chunk there ~10% | 19:22 |
| clarkb | sshnaidm: also http://logs.openstack.org/59/425459/10/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-nv/d5cab22/logs/oooq/full_logs.html is a redirect to nowhere and can be removed | 19:23 |
| sshnaidm | clarkb, yeah, I remove it now | 19:23 |
| fungi | AJaeger: and i'm guessing most of the stable branch issues you fixed were just backporting constraints usage? | 19:23 |
| AJaeger | fungi: there were more - I pushed quite a few projects to fix their mitaka branches - 2 fuel, tacker, taskflow,... | 19:23 |
| AJaeger | fungi: taskflow was the only one with constraints - and I didn't backport myself | 19:23 |
| fungi | AJaeger: yep, i saw side discussions of some of that, excellent work there | 19:23 |
| *** Goneri has quit IRC | 19:24 | |
| sshnaidm | clarkb, without quickstart.tar.gz I see there problem with ceiplometer only.. | 19:24 |
| AJaeger | thanks | 19:24 |
| clarkb | sshnaidm: also its still grabbing tripleo heat templates | 19:24 |
| clarkb | sshnaidm: ya ceilometer is another big one, is it possible that maybe you don't need full debug logs from ceilometer? | 19:24 |
| sshnaidm | clarkb, it may be a bug with such crazy logging.. | 19:25 |
| *** asselin has quit IRC | 19:25 | |
| clarkb | sshnaidm: well I think it logs everything at debug level | 19:25 |
| clarkb | sshnaidm: so every single poll request and response gets logged of which there are many | 19:26 |
| clarkb | sshnaidm: pabelanger also I confirmed that messages and audit logs are being compressed now and are much smaller | 19:26 |
| sshnaidm | clarkb, yeah, I'll look how to disable it there | 19:26 |
| AJaeger | fungi, trove and murano use already constraints on the failing branches, so no easy fix there ;( | 19:26 |
| clarkb | sshnaidm: I think you can also stop collecting /etc/X11 since these are headless | 19:27 |
| *** hashar has joined #openstack-infra | 19:27 | |
| fungi | AJaeger: a couple years ago, it seemed to be the case that every single security advisory got delayed by having to fix bitrot breakage on stable branches, so still a definite improvement | 19:27 |
| sshnaidm | clarkb, yeah.. although it's 612K only | 19:28 |
| clarkb | sshnaidm: yes I know but it serves little purpose :) | 19:28 |
| *** rbuzatu has joined #openstack-infra | 19:28 | |
| AJaeger | fungi, wow - ok, I consider myself lucky now ;) | 19:29 |
| fungi | we're actually over 1% free still on the logs volume (and the 45-day expiration pass hasn't even completed yet) | 19:29 |
| *** rbuzatu_ has joined #openstack-infra | 19:29 | |
| *** e0ne has quit IRC | 19:32 | |
| *** rbuzatu has quit IRC | 19:33 | |
| clarkb | sshnaidm: pabelanger any idea how the syslog.txt file is different than messages.txt? | 19:33 |
| clarkb | ./logs/syslog.txt.gz vs ./logs/oooq/undercloud/var/log/messages.txt.gz | 19:34 |
| *** rbuzatu_ has quit IRC | 19:34 | |
| sshnaidm | clarkb, no idea.. we don't need this syslog and logs/etc at all | 19:37 |
| clarkb | sshnaidm: digging more, do you need to build and publish docs artifacts for every job run? | 19:38 |
| clarkb | couldn't you publish those on docs.o.o like most projects instead? | 19:38 |
| sshnaidm | clarkb, not critical | 19:38 |
| clarkb | sshnaidm: because thats another megabyte. Anyawys by having everything possible in every job it adds up really quickly | 19:38 |
| sshnaidm | clarkb, I didn't know about docs.o.o.. | 19:38 |
| *** jsho has quit IRC | 19:39 | |
| sshnaidm | clarkb, yeah, I think we can remove docs in CI | 19:39 |
| clarkb | also looks like we have two ara "intallations" which doubles up any of the static resources there | 19:39 |
| clarkb | sshnaidm: most projects have a docs build job that builds and publishes the docs for review, but it does that once per patchset not once epr job. Then when changes to th project merge publishing jobs run and publish to docs.openstack.org proper | 19:40 |
| sshnaidm | the logs/ara is zuul stuff afaik | 19:40 |
| clarkb | oh is that from devstack-gate? | 19:40 |
| sshnaidm | clarkb, I see.. no, it's leftover from quickstart that should run locally and generate docs for users, could be removed | 19:41 |
| sshnaidm | clarkb, yep | 19:41 |
| sshnaidm | clarkb, our is in logs/oooq/ara | 19:41 |
| openstackgerrit | Merged openstack-infra/project-config master: Make craton functional tests experimental https://review.openstack.org/434979 | 19:41 |
| clarkb | ya confirmed its the setup host ara stuff | 19:41 |
| *** esikache1 has joined #openstack-infra | 19:42 | |
| *** jkilpatr has quit IRC | 19:42 | |
| clarkb | sshnaidm: so I think part of the issue here is also that devstack-gate collects logs and your stuff collects logs | 19:42 |
| *** jkilpatr has joined #openstack-infra | 19:42 | |
| clarkb | and in some places they overlap like logs/etc/glance and logs/oooq/$host/etc/glance | 19:42 |
| clarkb | so that is possibly another place where we can avoid duplication | 19:43 |
| clarkb | I'm guessing its the undercloud /etc that devstack-gate is also collecting? | 19:43 |
| clarkb | but thats just a guess. In any case should remove that duplication too | 19:43 |
| sshnaidm | clarkb, yes, as I said we don't need this.. | 19:44 |
| sshnaidm | clarkb, but the code that collects it runs after the job, in devstack-gate | 19:44 |
| *** dharinic|lunch is now known as dharinic | 19:44 | |
| clarkb | right so something will likely need changing. Either rely on devstack-gate to collect that for you or get it to stop somehow | 19:45 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Honor NODE_CLEANUP time again https://review.openstack.org/435083 | 19:45 |
| *** jsho has joined #openstack-infra | 19:46 | |
| *** jkilpatr has quit IRC | 19:46 | |
| *** jkilpatr has joined #openstack-infra | 19:46 | |
| clarkb | sshnaidm: looks like we can also exclude /etc/bash_completion since this isn't interactive | 19:51 |
| sshnaidm | clarkb, yeah | 19:51 |
| clarkb | sshnaidm: and crontab things in /etc and /etc/csh* | 19:51 |
| clarkb | and /etc/fonts | 19:52 |
| clarkb | sshnaidm: let me write a list :) | 19:52 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Honor NODE_CLEANUP time again https://review.openstack.org/435083 | 19:54 |
| sshnaidm | clarkb, ok :)h ttps://etherpad.openstack.org/p/tripleo-logs-actions | 19:54 |
| *** eroux has quit IRC | 19:55 | |
| *** chlong has quit IRC | 19:56 | |
| *** mkoderer has quit IRC | 19:57 | |
| *** john-davidge has quit IRC | 19:57 | |
| *** krtaylor has quit IRC | 19:59 | |
| *** cody-somerville has quit IRC | 20:01 | |
| *** csomerville has joined #openstack-infra | 20:02 | |
| clarkb | sshnaidm: ok I just put a bunch of stuff in there, probably not complete but should give us plenty to do and then check the listings again and go from there | 20:02 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Wait for server deletion before deleting ZK node https://review.openstack.org/435091 | 20:05 |
| *** rbuzatu has joined #openstack-infra | 20:06 | |
| *** kzaitsev_mb has quit IRC | 20:09 | |
| mordred | infra-root: I'm running a find on logs.o.o to collect size numbers for all of the job runs we have so that we can do math on them to figure out aveage/median/min/max numbers and whatnot | 20:10 |
| *** sdake_ has quit IRC | 20:10 | |
| mordred | infra-root: so if you see a file called "size_analysis" in /srv/static/logs - that's what it is | 20:11 |
| *** sdake has joined #openstack-infra | 20:11 | |
| mordred | oh - wait - fungi's 45-day expiration is still running - I will wait until that is done | 20:12 |
| fungi | mordred: thanks, and yeah any parallel analysis we're doing there does probably slow down that deletion process | 20:12 |
| *** Sukhdev has joined #openstack-infra | 20:14 | |
| *** Goneri has joined #openstack-infra | 20:15 | |
| *** ldnunes has quit IRC | 20:15 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Partial dead code deletion https://review.openstack.org/435095 | 20:16 |
| *** salv-orl_ has quit IRC | 20:16 | |
| fungi | unfortunately i have no idea how close it is to being done (i still hope it will pick up steam since reducing our retention period by 25% should reduce our utilization by more than 1% anyway, which is where we're at right now) | 20:18 |
| openstackgerrit | Chris St. Pierre proposed openstack-infra/zuul master: Add optional basic auth for merge check https://review.openstack.org/435098 | 20:19 |
| *** spzala has quit IRC | 20:19 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:20 | |
| fungi | strace indicates it's definitely still going | 20:22 |
| *** salv-orlando has joined #openstack-infra | 20:23 | |
| *** sshnaidm is now known as sshnaidm|off | 20:23 | |
| clarkb | apparently both stackviz and ara vendor fonts. I guess thats so everything looks right when loaded | 20:23 |
| clarkb | (they aren't very large so likely not worth having a "light" build without embedded fonts) | 20:24 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Remove openstack-heat-templates from bootstrap elements path https://review.openstack.org/435102 | 20:24 |
| *** gouthamr has quit IRC | 20:24 | |
| *** jsho has quit IRC | 20:24 | |
| mordred | clarkb: how big? | 20:26 |
| mordred | clarkb: becuase multiplied times a billion job dirs | 20:26 |
| pabelanger | I did notice stackviz folder a little large | 20:27 |
| *** rfolco has quit IRC | 20:27 | |
| pabelanger | will have to find the value again | 20:27 |
| *** rfolco has joined #openstack-infra | 20:27 | |
| clarkb | mordred: 800kb and 600kb ish for stackviz and ara respectively | 20:27 |
| *** salv-orlando has quit IRC | 20:28 | |
| *** kzaitsev_mb has quit IRC | 20:28 | |
| mordred | yah - I mean, it's not tiny multiplied by number of dirs | 20:28 |
| *** rfolco has quit IRC | 20:28 | |
| *** baoli has quit IRC | 20:29 | |
| *** tnovacik has quit IRC | 20:30 | |
| openstackgerrit | Merged openstack-infra/shade master: Wait for volumes to detach before deleting them https://review.openstack.org/432770 | 20:31 |
| * clarkb remembers that he needs to remind people things | 20:31 | |
| clarkb | mordred: don't forget about mysql backup change getting in before 0000UTC | 20:31 |
| clarkb | pabelanger: and I think you were going to try a backup restore at some point this week? | 20:31 |
| mordred | clarkb: +A | 20:32 |
| * jeblair wonders if he has dry cleaning to pick up | 20:32 | |
| mordred | jeblair: my dry cleaning has already been picked up | 20:32 |
| fungi | dry cleaning notification as a service | 20:32 |
| clarkb | I was not notified of any need to remind jeblair of dry cleaning pick up :) | 20:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Fix syntax errors with zuulv3-dev.o.o https://review.openstack.org/435103 | 20:34 |
| pabelanger | clarkb: ya, I was hoping to. If I get get zuulv3-dev.o.o online today, I'll set aside time tomorrow for that | 20:34 |
| *** kzaitsev_mb has joined #openstack-infra | 20:34 | |
| jeblair | ETOOMANYROCKETSHIPS | 20:35 |
| clarkb | pabelanger: cool | 20:35 |
| clarkb | speaking of rocket ships. space x is supposed to be launching off a reused first stage sometime soon. I should check when that is | 20:35 |
| pabelanger | yup | 20:35 |
| pabelanger | not sure why the gate didn't catch that | 20:35 |
| pabelanger | jeblair: AH | 20:35 |
| jeblair | pabelanger: -1 on that one so you can add the thing that will let the gate catch those errors :) | 20:35 |
| pabelanger | thanks | 20:35 |
| pabelanger | yay for code review | 20:36 |
| jeblair | yeah, i'm um, not sure why code review didn't catch that. ;) | 20:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add Node-OS: trusty for zuulv3-dev.o.o https://review.openstack.org/435103 | 20:37 |
| pabelanger | lets see what else fails | 20:37 |
| clarkb | is it bad that I now have a rough idea of how many tabs I can have open at once based on the size of my tab scrollbar? | 20:38 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add Node-OS: trusty for zuulv3-dev.o.o https://review.openstack.org/435103 | 20:39 |
| AJaeger | clarkb: it sounds bad - but what's the number? | 20:40 |
| clarkb | AJaeger: the number is like 150 | 20:40 |
| clarkb | I have a problem | 20:40 |
| *** baoli has joined #openstack-infra | 20:40 | |
| AJaeger | wow ;( | 20:40 |
| *** gouthamr has joined #openstack-infra | 20:41 | |
| *** armax has quit IRC | 20:41 | |
| *** armax has joined #openstack-infra | 20:41 | |
| openstackgerrit | Merged openstack-infra/puppet-mysql_backup master: Use --single-transaction for mysqldump https://review.openstack.org/434566 | 20:42 |
| *** armax has quit IRC | 20:44 | |
| AJaeger | fungi, found a patch to backport for murano and that one is merging. So, https://review.openstack.org/429050 can be reviewed now (not yet approved since dependency is not merged yet), please | 20:47 |
| jeblair | pabelanger: yay! failures! | 20:49 |
| mordred | ok - so gerrit should NOT go on vacation at 00:00UTC today | 20:50 |
| jeblair | when are we going to get our snacks? | 20:50 |
| AJaeger | mordred: so what time will it be then? ;) | 20:50 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add Node-OS: trusty for zuulv3-dev.o.o https://review.openstack.org/435103 | 20:50 |
| jeblair | pabelanger: http://logs.openstack.org/03/435103/3/check/gate-infra-puppet-apply-3-ubuntu-trusty/1936df5/puppetapplytest45.final.out.FAILED | 20:50 |
| AJaeger | mordred: everybody needs a vacation! | 20:51 |
| pabelanger | jeblair: yup, I am slightly embarrassed we merged broken code :( | 20:51 |
| *** ccamacho has left #openstack-infra | 20:52 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/shade master: DNM- Pretty Tox addition to debug memory issues in py35 https://review.openstack.org/435107 | 20:52 |
| timothyb89 | pabelanger, clarkb: for what it's worth there are some long-term plans to to greatly reduce disk usage in stackviz, unfortunately they've been stalled for a while but I hope to pick that back up soon | 20:52 |
| clarkb | timothyb89: good to know. Was just thinking that for many of us the specific font likely doesn't amtter and are happy with browser fallbacks for whatever which amy mean a "light" build could drop the built in fonts | 20:53 |
| pabelanger | clarkb: mind +3 https://review.openstack.org/#/c/434550/ | 20:53 |
| pabelanger | open firewall for nl01.o.o to zookeeper | 20:53 |
| timothyb89 | clarkb: among other things I'd actually like to avoid putting a stackviz in each job build in the first place :) | 20:54 |
| clarkb | pabelanger: ya looking now | 20:54 |
| *** aleph1 is now known as agarner_away | 20:54 | |
| *** maishsk has joined #openstack-infra | 20:54 | |
| mordred | timothyb89, clarkb: or a 'light' build which assumes the fonts and whatnot will be at a static location on the webserver already | 20:56 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Coverage: add and store subunit stream for coverage stats https://review.openstack.org/434625 | 20:56 |
| *** esikache1 has quit IRC | 20:56 | |
| mordred | timothyb89: but I welcome any and all improvements you have planned for our stackviz overlords | 20:56 |
| *** kgiusti has left #openstack-infra | 20:57 | |
| timothyb89 | mordred, clarkb: the 2 main goals are to reduce the overall footprint by a ton (targeting max of 300kb total) and then making the entire thing 1 single static instance | 20:57 |
| timothyb89 | and for bonus points to remove the processing step we have to run in the devstack scripts | 20:58 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: remove test-environments/multinode_major_upgrade.yaml https://review.openstack.org/426375 | 20:58 |
| mordred | timothyb89: oh fancy | 20:58 |
| *** john-davidge has joined #openstack-infra | 20:58 | |
| ianw | hey, can i get https://review.openstack.org/#/c/428500 quickly cleared? it just adds apache redirects to the nodepool .json outputs to puppet-nodepool. they're just partners to the text output | 20:58 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: upgrades: switch 'newton' to $UPGRADE_RELEASE https://review.openstack.org/433209 | 20:59 |
| pabelanger | jeblair: Yah, green: https://review.openstack.org/#/c/435103/ | 21:00 |
| jeblair | +3 | 21:01 |
| openstackgerrit | Merged openstack-infra/system-config master: Allow nl01.o.o to access zookeeper https://review.openstack.org/434550 | 21:02 |
| *** krtaylor has joined #openstack-infra | 21:03 | |
| *** john-davidge has quit IRC | 21:03 | |
| openstackgerrit | Tim Burke proposed openstack-infra/project-config master: Remove link to dead system-config/jenkins documentation https://review.openstack.org/435113 | 21:05 |
| EmilienM | we have noticed this thing: http://logs.openstack.org/24/433824/4/check/gate-tripleo-ci-centos-7-nonha-multinode/05d3e89/console.html#_2017-02-16_18_31_35_979054 | 21:06 |
| EmilienM | "... this takes 3 - 5 minutes (logs at logs/devstack-gate-setup-workspace-new.txt.gz)" but it actually takes 14 minutes | 21:06 |
| *** sdake has quit IRC | 21:07 | |
| clarkb | EmilienM: http://logs.openstack.org/24/433824/4/check/gate-tripleo-ci-centos-7-nonha-multinode/05d3e89/logs/devstack-gate-setup-workspace-new.txt.gz#_2017-02-16_18_31_47_211 a decent chunk of it is right at the beginning shuffling /opt around | 21:07 |
| fungi | EmilienM: could probably stand to update that message if it's ceased to be accurate... i think it's just hard-coded in devstack-gate | 21:08 |
| EmilienM | ok I was just wondering if it was "normal" | 21:08 |
| openstackgerrit | Merged openstack-infra/puppet-nodepool master: Add redirects to json status output https://review.openstack.org/428500 | 21:08 |
| clarkb | EmilienM: workspace setup time is also directly related to the total number of PROJECTS in the PROJECTS var | 21:08 |
| fungi | EmilienM: note the message immediately after it "setup workspace took > 10 minutes, this is a very slow node" | 21:08 |
| clarkb | since each one has to be configured. Its why I have been slowly trying to trim the list by default | 21:09 |
| clarkb | (I should pick that back up again) | 21:09 |
| fungi | it's possible that measurement also needs updating, no clue without better statistics | 21:09 |
| clarkb | EmilienM: that job has quite the list http://logs.openstack.org/24/433824/4/check/gate-tripleo-ci-centos-7-nonha-multinode/05d3e89/logs/devstack-gate-setup-host.txt.gz | 21:09 |
| pabelanger | in a zuulv3 world, would jobs show that process? since it is a pre_playbook? Or would job attach after that step? | 21:10 |
| clarkb | EmilienM: but further reducing it would help since I think a good chunk of that you are picking up by default | 21:10 |
| fungi | ahh, yeah so this is likely a case of the hard-coded message and "slowness" threshold not dynamically updated by the optional amount of work some variations of the job might specify | 21:10 |
| clarkb | fungi: yes | 21:11 |
| fungi | maybe those could become an optional envvar | 21:11 |
| openstackgerrit | Chris Spencer proposed openstack-infra/project-config master: Enable watcher tempest gate tests for watcher-dashboard. Enable same checks as watcher project. https://review.openstack.org/435119 | 21:12 |
| clarkb | well we need to pick up work again to stop setting up projects like sahara on every run | 21:12 |
| fungi | so that some jobs could actually assert the length of time they expect that step to take and then warn when a certain percent over that number is exceeded | 21:12 |
| clarkb | since the total number of projects that need sahara from source are quite small | 21:12 |
| clarkb | its just been low on priority since things mostly work right now | 21:12 |
| openstackgerrit | Merged openstack-infra/system-config master: Add Node-OS: trusty for zuulv3-dev.o.o https://review.openstack.org/435103 | 21:12 |
| pabelanger | yay | 21:13 |
| clarkb | sahara, trove, manila, all the oslo libs could all stand to be removed from the default list and added in where necessary | 21:13 |
| clarkb | trouble is it takes time and coordination to avoid breaking a bunch of projects and their jobs | 21:13 |
| *** dimtruck is now known as zz_dimtruck | 21:14 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/shade master: Convert use of .register_uri to .register_uris https://review.openstack.org/434516 | 21:15 |
| clarkb | mtreinish: is horizon enabled by default in the base tempest jobs still? or did we stop testing there now that horizon has their own functional testing | 21:15 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/shade master: Convert test_object to use .register_uris https://review.openstack.org/434532 | 21:16 |
| mtreinish | clarkb: david-lyle and I split that 1 tempest test out into a tempest plugin | 21:16 |
| mtreinish | IIRC the dsvm jobs still enable horizon, but tempest doesn't hit it anymore | 21:17 |
| *** yamamoto has joined #openstack-infra | 21:17 | |
| clarkb | should we maybe stop enabling it by default? | 21:17 |
| *** yamamoto has quit IRC | 21:19 | |
| *** dave-mccowan has quit IRC | 21:21 | |
| *** armax has joined #openstack-infra | 21:22 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Upgrade devstack-tools to 0.2.1 https://review.openstack.org/433093 | 21:22 |
| *** salv-orlando has joined #openstack-infra | 21:24 | |
| *** eharney has quit IRC | 21:25 | |
| *** flepied has joined #openstack-infra | 21:27 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Use thread name in launcher ID https://review.openstack.org/435125 | 21:28 |
| *** salv-orlando has quit IRC | 21:29 | |
| *** armax has quit IRC | 21:30 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Remove openstack-heat-templates from bootstrap elements path https://review.openstack.org/435102 | 21:30 |
| openstackgerrit | Matthew Oliver proposed openstack-infra/project-config master: Stop swifts func-post-as-copy job running on stable https://review.openstack.org/434664 | 21:31 |
| *** sdake has joined #openstack-infra | 21:33 | |
| *** MasterOfBugs has joined #openstack-infra | 21:33 | |
| *** jkilpatr has quit IRC | 21:34 | |
| *** ccamacho has joined #openstack-infra | 21:34 | |
| *** Jeffrey4l has quit IRC | 21:35 | |
| *** Jeffrey4l has joined #openstack-infra | 21:36 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/shade master: Require `complete_qs` if a query_string is in the mocked uri https://review.openstack.org/435127 | 21:36 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Fix possible race with node request handling https://review.openstack.org/435128 | 21:36 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci master: Use bootstrap subnodes environment in CI https://review.openstack.org/425888 | 21:37 |
| *** ihrachys has joined #openstack-infra | 21:37 | |
| pabelanger | #status log zuulv3-dev.o.o is now online. Zuul services are currently stopped. | 21:37 |
| openstackstatus | pabelanger: finished logging | 21:37 |
| *** beslemon has quit IRC | 21:38 | |
| *** salv-orlando has joined #openstack-infra | 21:38 | |
| openstackgerrit | Tim Burke proposed openstack-infra/project-config master: Remove dead ubuntu-trusty jobs for swift https://review.openstack.org/435129 | 21:39 |
| *** zz_dimtruck is now known as dimtruck | 21:39 | |
| *** wolverin_ has quit IRC | 21:40 | |
| pabelanger | cloudnull: have you heard any updates to the networking issues for osic-cloud1? | 21:40 |
| *** wolverineav has joined #openstack-infra | 21:40 | |
| openstackgerrit | Chris Spencer proposed openstack-infra/project-config master: Enable watcher tempest gate tests for watcher-dashboard. Enable same checks as watcher project. https://review.openstack.org/435119 | 21:42 |
| cloudnull | pabelanger: I was just talking to mudpuppy about that. | 21:43 |
| cloudnull | seems there was a massive spike in networking | 21:43 |
| cloudnull | i'm not sure of the what/why though | 21:43 |
| cloudnull | mudpuppy: ? | 21:43 |
| mudpuppy | We are trying to track it down. I"m not sure yet, but asked our networking folks to check it out. | 21:44 |
| pabelanger | cloudnull: mudpuppy: okay, cool. No rush, just wanted to see how you are making out. | 21:44 |
| *** Sukhdev has quit IRC | 21:45 | |
| cloudnull | pabelanger: -- https://cloud1.osic.org/grafana/dashboard/db/openstack-system-metrics?from=1487264574272&to=1487269088743&panelId=42026&fullscreen -- from the infra hosts within Cloud1. | 21:47 |
| cloudnull | which was only on the api systems | 21:47 |
| cloudnull | so it looks a little odd. | 21:47 |
| cloudnull | https://cloud1.osic.org/grafana/dashboard/db/openstack-system-metrics?from=1487264574272&to=1487269088743&panelId=42026&fullscreen&var-datasource=OSIC%20InfluxDB&var-server=725155-network03&var-server=725156-network01&var-server=727112-network02&var-inter=$__auto_interval - - from the network nodes, so l3 traffic looks like it was ok. | 21:47 |
| cloudnull | but anyway mudpuppy is on the case :) | 21:47 |
| rm_work | hey, is the build system still suffering from disk space issues? | 21:48 |
| rm_work | or was that resolved | 21:48 |
| clarkb | rm_work: it should be resolved enough that jobs are happy | 21:48 |
| rm_work | ok | 21:49 |
| rm_work | we're seeing some odd stuff with our images, trying to rule things out | 21:49 |
| rm_work | seems like our octavia amphora images can't mount their root block device | 21:49 |
| *** wolverineav has quit IRC | 21:49 | |
| rm_work | and the cirros images aren't booting either, though we have less insight into why that is | 21:49 |
| rm_work | not sure if it could have been disk space related | 21:50 |
| clarkb | rm_work: that wouldn't have been affected by disk space issues anyways | 21:50 |
| clarkb | since disk was only for the log server | 21:50 |
| rm_work | kk | 21:50 |
| rm_work | thanks | 21:50 |
| *** tonytan4ever has quit IRC | 21:51 | |
| johnsom | I think it's the switch to Cirros 0.3.5: https://review.openstack.org/#/c/432793/ | 21:51 |
| johnsom | The timing lines up | 21:51 |
| *** SumitNaiksatam has quit IRC | 21:52 | |
| *** beslemon has joined #openstack-infra | 21:53 | |
| johnsom | It is failing to boot on both RAX, OSIC, and infracloud hosts, so I doubt it is disk space related | 21:54 |
| clarkb | do you have an example of a failure? | 21:57 |
| rm_work | http://logs.openstack.org/18/434518/2/check/gate-octavia-v1-dsvm-py3x-scenario-ubuntu-xenial-nv/b46c5f2/console.html#_2017-02-16_19_57_06_720587 | 21:57 |
| clarkb | is that how cirros fails too? | 21:58 |
| clarkb | I think you need root=vda1 | 21:59 |
| clarkb | maybe? | 21:59 |
| clarkb | I'm not sure where cloudimg-rootfs as a label name is coming from | 21:59 |
| *** lseki has quit IRC | 22:00 | |
| *** dfflanders has joined #openstack-infra | 22:01 | |
| clarkb | rm_work: johnsom there was a recent switch from ami to qcow2 as well | 22:02 |
| clarkb | perhaps you are trying to boot with flags for an ami? | 22:02 |
| jhesketh | Morning | 22:03 |
| mudpuppy | pabelanger: Can you revert that PR? I think cloud1 is behaving better now | 22:03 |
| johnsom | It looks like devstack is loading http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img | 22:03 |
| *** leifmadsen_ is now known as leifmadsen | 22:04 | |
| rm_work | https://gist.github.com/rm-you/d8acb9e63ea098efddbaa84c37352c9a | 22:04 |
| rm_work | I think that's how we booted it? | 22:04 |
| rm_work | err wait | 22:04 |
| rm_work | that's the response, crap | 22:04 |
| *** chlong has joined #openstack-infra | 22:05 | |
| clarkb | what does "OS-DCF:diskConfig": "MANUAL", mean? | 22:05 |
| rm_work | sorry this is the request: https://gist.github.com/rm-you/1b8f2862413441b061d54bc4e676150c | 22:06 |
| rm_work | nothing interesting there i don't think? | 22:06 |
| clarkb | ya that seems fine | 22:06 |
| clarkb | is that heat? perhaps heat assumed ami under the covers or something? | 22:07 |
| clarkb | I dunno | 22:07 |
| rm_work | POST http://15.184.68.20:8774/v2.1/servers | 22:07 |
| rm_work | looks like straight to nova? | 22:07 |
| *** Julien-zte has joined #openstack-infra | 22:08 | |
| rm_work | yeah I think maybe johnsom is right, it could be a 0.3.5 issue | 22:08 |
| rm_work | though the timing... | 22:08 |
| rm_work | clarkb: do you know WHEN the ami->qcow2 switch happened? | 22:08 |
| clarkb | rm_work: not exactly, was this week though | 22:09 |
| fungi | pabelanger: i removed your workflow -1 from 435047 so we can land it, hope that was okay | 22:09 |
| fungi | mudpuppy: cloudnull: i've approved the change which turns osic back on | 22:09 |
| clarkb | rm_work: I'd run devstack locally and then try booting that same image and go from there | 22:09 |
| mudpuppy | fungi: cool | 22:10 |
| fungi | mudpuppy: cloudnull: as for a spike to the api from one of our servers, that was likely a symptom of connectivity issues rather than a cause... nodepool will retry api requests fairly rapidly when it gets back failures/errors (though we can throttle that with a configurable delay per provider if you need) | 22:11 |
| *** chlong has quit IRC | 22:11 | |
| *** jkilpatr has joined #openstack-infra | 22:11 | |
| rm_work | k i'll give that a shot | 22:11 |
| johnsom | clarkb Working on that now. Booting good with 0.3.4, checking 0.3.5 now. I do see that the image devstack is using is in qcow format | 22:11 |
| fungi | mudpuppy: cloudnull: normally with a low volume of errors that shouldn't be noticeable, but if a majority of calls result in failure you're going to see a pretty significant additional request volume | 22:12 |
| pabelanger | fungi: ++ | 22:12 |
| *** wolverineav has joined #openstack-infra | 22:13 | |
| *** bknudson has left #openstack-infra | 22:14 | |
| *** wolverineav has quit IRC | 22:14 | |
| *** wolverin_ has joined #openstack-infra | 22:14 | |
| *** bknudson has joined #openstack-infra | 22:14 | |
| *** dave-mccowan has joined #openstack-infra | 22:14 | |
| *** gouthamr has quit IRC | 22:15 | |
| openstackgerrit | Tim Burke proposed openstack-infra/project-config master: Remove dead ubuntu-trusty jobs for swift and swiftclient https://review.openstack.org/435129 | 22:17 |
| *** portdirect is now known as intlabs | 22:17 | |
| *** maishsk has quit IRC | 22:20 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Move ProviderManagers into main NodePool thread https://review.openstack.org/434567 | 22:21 |
| johnsom | This is supper strange. Locally I can boot the cirros 0.3.5 image | 22:21 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Temporarily disable osic-cloud1" https://review.openstack.org/435047 | 22:22 |
| clarkb | johnsom: using the same setuo via reproduce.sh or similar? | 22:23 |
| johnsom | I used nova to boot instead of tempest, so just a download the image and boot it. | 22:24 |
| clarkb | johnsom: right but against what cloud? | 22:26 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Implement node cleanup https://review.openstack.org/433736 | 22:26 |
| johnsom | My local devstack, but we see the gates failing on OSIC, RAX, and infracloud | 22:26 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Re-enable devstack test job https://review.openstack.org/431649 | 22:27 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add lock state to node listing https://review.openstack.org/434397 | 22:27 |
| clarkb | johnsom: right so I would try using very specifically the same devstack setup used by that job. Easy way to get that is via the reproduce.sh script | 22:27 |
| *** armax has joined #openstack-infra | 22:27 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Wait for server deletion before deleting ZK node https://review.openstack.org/435091 | 22:28 |
| johnsom | Yeah, just working through the layers... | 22:28 |
| *** fguillot has quit IRC | 22:28 | |
| clarkb | could be something changed in nova even | 22:28 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Partial dead code deletion https://review.openstack.org/435095 | 22:29 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Use thread name in launcher ID https://review.openstack.org/435125 | 22:29 |
| mtreinish | johnsom: so the issue is most likely caused by your devstack plugin. You're configuring tempest to use this image: http://logs.openstack.org/18/434518/2/check/gate-octavia-v1-dsvm-py3x-scenario-multinode-nv/31ad545/logs/devstacklog.txt.gz#_2017-02-16_19_48_43_588 | 22:30 |
| mtreinish | which is created with dib as part of your devstack plugin | 22:30 |
| mtreinish | my guess is the shift to qcow broke however you're customizing the image you're using for testing | 22:30 |
| mtreinish | I didn't trace through all the code in the devstack plugin to see exactly where the issue is though | 22:31 |
| johnsom | mtreinish The test doesn't get far enough to use that image | 22:31 |
| *** ihrachys has quit IRC | 22:32 | |
| johnsom | It fails booting cirros web servers before we can even boot one of those | 22:32 |
| mtreinish | johnsom: yes it does, it creates a server with that image. Then you try to ssh into it | 22:32 |
| johnsom | No, that image never gets ssh | 22:32 |
| mtreinish | create_server will use the configured image_ref by default, and that's what it's set to in the tempest.conf | 22:32 |
| clarkb | fwiw the failure I was linked to was for ubuntu not cirros | 22:33 |
| clarkb | at least according to the kernel and things | 22:33 |
| johnsom | If you look at the server boot call, the id is different | 22:33 |
| johnsom | Yeah, cirros is using an ubuntu kernel. I checked that local too | 22:33 |
| johnsom | surprising | 22:34 |
| *** adriant has joined #openstack-infra | 22:34 | |
| *** yamamoto has joined #openstack-infra | 22:35 | |
| *** ijw has joined #openstack-infra | 22:36 | |
| *** annegentle has quit IRC | 22:36 | |
| mtreinish | johnsom: the test does not override the image_id: https://github.com/openstack/octavia/blob/master/octavia/tests/tempest/v1/scenario/base.py#L180 which means 07c68e47-f902-4035-b7ff-cd2a2e4cb288 is what is used in the create server call | 22:37 |
| mtreinish | johnsom: show me where you think the image id tempest is using in create server is not the one you're creating in the plugin | 22:37 |
| cloudnull | fungi: thanks! idk what the api spike was from, infra spike was from. i really dont think the root cause was infra api requests. It seems like something else was happening at that time though I'm not 100% sure. mudpuppy will have to continue to investigate. | 22:37 |
| *** pahuang has joined #openstack-infra | 22:38 | |
| johnsom | mtreinish Yeah, I just compared the Ids, it is booting with the wrong image. ???? Do you see where we are telling tempest to use that image? | 22:38 |
| *** rfolco has joined #openstack-infra | 22:39 | |
| *** thorst_afk has quit IRC | 22:39 | |
| *** rfolco has quit IRC | 22:39 | |
| mtreinish | johnsom: it's in the tempest.conf: http://logs.openstack.org/18/434518/2/check/gate-octavia-v1-dsvm-py3x-scenario-multinode-nv/31ad545/logs/tempest_conf.txt.gz | 22:39 |
| *** flepied has quit IRC | 22:39 | |
| mtreinish | unless you explicitly tell the server create call to use a different id tempest will always boot the image it's configured to | 22:39 |
| johnsom | Yeah, now the question is how is that getting set... We don't have image_ref in our code | 22:41 |
| *** burgerk has quit IRC | 22:41 | |
| *** rhallisey has quit IRC | 22:42 | |
| *** matt-borland has quit IRC | 22:42 | |
| *** Sukhdev has joined #openstack-infra | 22:43 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Fix possible race with node request handling https://review.openstack.org/435128 | 22:44 |
| *** beslemon has quit IRC | 22:46 | |
| fungi | default? | 22:46 |
| *** tphummel has quit IRC | 22:47 | |
| clarkb | mtreinish: johnsom regardless of what image is booting the image should be bootable ya? but I guess this helps narrow down source of problem | 22:47 |
| johnsom | It's non-deterministic in devstack: https://github.com/openstack-dev/devstack/blob/master/lib/tempest#L152 It's just the first one in the list | 22:47 |
| johnsom | Maybe OSC is alpha sort now | 22:48 |
| mtreinish | johnsom: no it's booting alphabetically because of the bug I fix in: https://review.openstack.org/#/c/435106/ | 22:48 |
| mtreinish | it's always what broke grenade when I tried to backport it | 22:48 |
| *** acabot has quit IRC | 22:48 | |
| mtreinish | johnsom: clarkb is right though, you're still creating a broken image in the devstack plugin | 22:49 |
| *** gildub has joined #openstack-infra | 22:49 | |
| mtreinish | it's just being booted by mistake earlier | 22:49 |
| *** furlongm has quit IRC | 22:50 | |
| johnsom | Well, that is new too. Maybe we have been running our web servers on our image and not cirros for some time and just didn't know it. That would have changed in the last few months though as DIB was broken for us not too long ago. | 22:51 |
| *** furlongm has joined #openstack-infra | 22:53 | |
| *** Lekensteyn has left #openstack-infra | 22:54 | |
| *** marst has quit IRC | 22:54 | |
| *** ccamacho has left #openstack-infra | 22:58 | |
| *** askb has joined #openstack-infra | 22:58 | |
| *** ihrachys has joined #openstack-infra | 22:58 | |
| *** john-davidge has joined #openstack-infra | 22:59 | |
| *** acabot has joined #openstack-infra | 23:01 | |
| *** jamielennox is now known as jamielennox|away | 23:02 | |
| *** gordc has quit IRC | 23:02 | |
| *** thorst_afk has joined #openstack-infra | 23:03 | |
| *** john-davidge has quit IRC | 23:04 | |
| *** mdrabe has quit IRC | 23:04 | |
| *** rbuzatu has quit IRC | 23:05 | |
| *** dave-mccowan has quit IRC | 23:06 | |
| *** thorst_afk has quit IRC | 23:07 | |
| *** dimtruck is now known as zz_dimtruck | 23:07 | |
| openstackgerrit | Sachin proposed openstack-infra/project-config master: Hi, Sorry, just asking as I don't know openstack review process well. At this point, is there something else I need to do? https://review.openstack.org/435145 | 23:09 |
| mordred | that may be the best commit message I've ever seen | 23:11 |
| fungi | awesome | 23:11 |
| mordred | it's a commit message only followup patch | 23:11 |
| mordred | what is in IRC is the entirety of the content | 23:12 |
| johnsom | clarkb and mtreinish Thanks for helping track this down. I guess we need to wait for that to merge. While I wait I will figure out the better way for this tempest setup to be booting cirros images. | 23:12 |
| fungi | mordred: my best guess is they wanted a review comment and didn't know how to add one | 23:12 |
| mordred | fungi: yah. OR - wanted to ask a question in IRC and that was the best way they could come up with :) | 23:12 |
| fungi | hah, maybe! | 23:13 |
| mordred | if so - they win the prize for best makeshift IRC client ever | 23:13 |
| *** jamielennox|away is now known as jamielennox | 23:14 | |
| johnsom | Our image isn't booting as the cirros flavor is too small... | 23:15 |
| *** marst has joined #openstack-infra | 23:16 | |
| mtreinish | mordred: that's an idea for a fun project, a gerritbot based irc client | 23:16 |
| mtreinish | heh of course it'd be write only :) | 23:16 |
| *** kzaitsev_mb has quit IRC | 23:16 | |
| mtreinish | johnsom: oh, then I guess the image isn't broken :) | 23:17 |
| clarkb | mordred: /me gives a 43 minute warning to db backup awesomeness | 23:17 |
| clarkb | mordred: I am planning to try and use the gerrits right around 0000UTC | 23:17 |
| *** baoli has quit IRC | 23:18 | |
| mordred | mtreinish: you could have gerritbot write comments back to the changes if someone /msg it ... | 23:19 |
| mordred | clarkb: thanks! | 23:19 |
| clarkb | mordred: I will likely leave review comments, maybe someone else can try pushing a change | 23:19 |
| clarkb | since those are the two big things that lock up iirc | 23:19 |
| openstackgerrit | Merged openstack-infra/shade master: Actually normalize nova usage data https://review.openstack.org/434481 | 23:20 |
| mordred | clarkb: I dunno. I'm pretty bad at writing changes to things | 23:20 |
| *** jamesdenton has quit IRC | 23:23 | |
| *** denisra has quit IRC | 23:23 | |
| *** salv-orlando has quit IRC | 23:24 | |
| pabelanger | cloudnull: mudpuppy: fungi: we are still seeing a high rate of failures with osic-cloud1: http://grafana.openstack.org/dashboard/db/nodepool-osic | 23:25 |
| pabelanger | http://logs.openstack.org/20/434920/2/gate/gate-puppet-panko-puppet-lint/eb275d8/console.html#_2017-02-16_22_45_45_441743 | 23:25 |
| pabelanger | too | 23:25 |
| *** tphummel has joined #openstack-infra | 23:27 | |
| *** rlandy is now known as rlandy|afk | 23:29 | |
| mriedem | anyone ever seen this with nested virt? http://logs.openstack.org/50/435050/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/49adcde/logs/subnode-2/libvirt/qemu/instance-00000001.txt.gz | 23:32 |
| mriedem | KVM: entry failed, hardware error 0x0 | 23:32 |
| mriedem | this is during a live migration test, the guest is created, goes resumed -> paused | 23:32 |
| mriedem | and i see that kvm hw error in the guest logs | 23:32 |
| mriedem | https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1329434 | 23:33 |
| openstack | Launchpad bug 1329434 in linux (Ubuntu Trusty) "nested KVM fails on intel hardware - KVM: entry failed, hardware error 0x0" [Medium,Fix released] - Assigned to Chris J Arges (arges) | 23:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Revert "Revert "Temporarily disable osic-cloud1"" https://review.openstack.org/435148 | 23:33 |
| clarkb | mriedem: is it kvm/nested virt or qemu? | 23:33 |
| *** xyang1 has quit IRC | 23:33 | |
| clarkb | mriedem: it should be qemu by default | 23:33 |
| mriedem | http://logs.openstack.org/50/435050/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/49adcde/logs/subnode-2/etc/nova/nova.conf.txt.gz | 23:33 |
| mriedem | virt_type = kvm | 23:33 |
| clarkb | mriedem: if it is in fact kvm then I would switch to qemu ad kvm is known to have all sorts of issues | 23:33 |
| pabelanger | fungi: clarkb: ^ we might want to disable osic-cloud1 again, seeing some jobs failing again because of networking issues | 23:33 |
| mriedem | yeah, wtf | 23:33 |
| clarkb | mriedem: and we don't support it for that reason | 23:33 |
| mriedem | i didn't think we even tried kvm | 23:33 |
| mriedem | in thegate | 23:34 |
| clarkb | mriedem: we don't by default but you can force it one (and some projects do against our advice) | 23:34 |
| mriedem | i thought devstack always picked qemu | 23:34 |
| mriedem | i'll dig though | 23:34 |
| clarkb | mriedem: no devstack will pick kvm if its able | 23:34 |
| mriedem | orly | 23:34 |
| clarkb | devstack-gate attempts to force people onto qemu | 23:35 |
| clarkb | but thats something you can work around | 23:35 |
| clarkb | unless maybe that stopped being the case with the local.conf changes? /me is looking | 23:35 |
| *** hashar has quit IRC | 23:35 | |
| clarkb | mriedem: http://logs.openstack.org/50/435050/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/49adcde/logs/localrc.txt.gz libvirt type there is qemu | 23:36 |
| mriedem | aha | 23:36 |
| mriedem | http://logs.openstack.org/50/435050/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/49adcde/logs/etc/nova/nova.conf.txt.gz | 23:36 |
| clarkb | huh we don't set it on the subnode localrc for some reason thats really weird | 23:36 |
| mriedem | it's set on the primary node | 23:36 |
| mriedem | virt_type = qemu | 23:36 |
| mriedem | but it's kvm on the subnode | 23:36 |
| clarkb | ya | 23:37 |
| * clarkb checks a different multinode job | 23:37 | |
| clarkb | pabelanger: if its still unreliable I would say just go ahead and approve it | 23:38 |
| *** Julien-zte has quit IRC | 23:39 | |
| mriedem | clarkb: we just don't set LIBVIRT_TYPE in the subnode localrc it looks like | 23:40 |
| mriedem | probably just a simple oversight | 23:40 |
| clarkb | mriedem: ya | 23:40 |
| clarkb | mriedem: well it is supposed to run the same localrc creation function | 23:41 |
| clarkb | so I am confused why it doesn't currently | 23:41 |
| mriedem | i blame devstack-tools :) | 23:41 |
| mriedem | working on a patch | 23:42 |
| mudpuppy | pabelanger: Looks like instances start and then networking craps out on you correct? | 23:42 |
| *** ihrachys has quit IRC | 23:42 | |
| clarkb | mriedem: do you see what the issue is? | 23:43 |
| *** wolverin_ has quit IRC | 23:45 | |
| clarkb | mriedem: because I don't see it, we basically call setup_localrc twice, first for primary node where it gets set correctly, then again for subnode where it doesn't | 23:45 |
| mriedem | yeah i think so | 23:45 |
| clarkb | should be using the same env and everything | 23:45 |
| clarkb | cool I await the patching | 23:45 |
| mriedem | well, https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L387 | 23:46 |
| mriedem | should that be >>"$localrc_file" ? | 23:46 |
| mriedem | it is for everything else in there | 23:46 |
| mriedem | i don't know why that would be different between primary and subnode | 23:46 |
| clarkb | mriedem: hahaha I bet its because for primary we call the file localrc | 23:48 |
| clarkb | mriedem: but subnode is something like localrc.subnode | 23:48 |
| clarkb | good catch | 23:48 |
| mriedem | setup_localrc $old_or_new "$sub_localrc" "sub" | 23:48 |
| mriedem | setup_localrc "new" "localrc" "primary" | 23:48 |
| mriedem | so yeah | 23:48 |
| mriedem | derp | 23:48 |
| mriedem | patch coming | 23:48 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/devstack-gate master: Set LIBVIRT_TYPE in the subnode localrc https://review.openstack.org/435154 | 23:50 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: remove test-environments/multinode_major_upgrade.yaml https://review.openstack.org/426375 | 23:51 |
| *** dingyichen has joined #openstack-infra | 23:52 | |
| clarkb | as soon as firefox catches up with me I will review | 23:52 |
| clarkb | actually let me restart firefox | 23:52 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: remove test-environments/multinode_major_upgrade.yaml https://review.openstack.org/426375 | 23:52 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: upgrades: switch 'newton' to $UPGRADE_RELEASE https://review.openstack.org/433209 | 23:52 |
| mriedem | clarkb: thanks for helping put the pieces together, i'm heading home but will be back online later | 23:54 |
| clarkb | mriedem: ok I will likely approve that soonish if no one else reviews it. | 23:54 |
| mriedem | clarkb: maybe throw that into your how to debug shit talk next week :) | 23:54 |
| clarkb | mriedem: indeed | 23:54 |
| mriedem | "does libvirt just randomly not do what you think it should?" | 23:54 |
| mordred | mriedem: daily | 23:55 |
| fungi | sounds like a feature to me | 23:55 |
| mriedem | well have i got some snake oil for you! | 23:55 |
| clarkb | mordred: fungi want to review that change too? | 23:55 |
| fungi | sure | 23:55 |
| clarkb | a wild jogo appeared! | 23:55 |
| fungi | wha? whoah! | 23:56 |
| mriedem | don't look directly at it | 23:56 |
| *** mriedem has quit IRC | 23:56 | |
| *** sshnaidm|off has quit IRC | 23:56 | |
| * jogo goes back into hiding | 23:56 | |
| *** jaypipes has quit IRC | 23:56 | |
| clarkb | mordred: just a few minutes left | 23:56 |
| * mordred hands jogo a pie | 23:57 | |
| clarkb | I just double checked the crontab entry on review.o.o was udpated | 23:57 |
| mordred | clarkb: as did I :) | 23:57 |
| mordred | I have a patch ready to submit | 23:57 |
| *** armax has quit IRC | 23:58 | |
| fungi | one minute to showtime! | 23:58 |
| openstackgerrit | Merged openstack-infra/shade master: Change request_id logging to match nova format https://review.openstack.org/434916 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!