| openstackgerrit | Steve Baker proposed openstack/tripleo-common: Implement ViewManager list_software_deployments https://review.openstack.org/268822 | 00:03 |
|---|---|---|
| openstackgerrit | Steve Baker proposed openstack/tripleo-common: show_software_deployment https://review.openstack.org/270545 | 00:03 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common: list_stack_failures https://review.openstack.org/270546 | 00:03 |
| *** cwolferh has joined #tripleo | 00:06 | |
| *** absubram has quit IRC | 00:07 | |
| *** dsneddon has quit IRC | 00:11 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Remove zero length files https://review.openstack.org/270550 | 00:11 |
| *** dsneddon has joined #tripleo | 00:15 | |
| *** dsneddon has quit IRC | 00:15 | |
| *** dsneddon has joined #tripleo | 00:16 | |
| *** weshay has quit IRC | 00:19 | |
| *** dmacpher has quit IRC | 00:21 | |
| *** rasca has quit IRC | 00:22 | |
| *** rasca has joined #tripleo | 00:23 | |
| *** apetrich_ has joined #tripleo | 00:42 | |
| *** apetrich has quit IRC | 00:45 | |
| *** morazi has quit IRC | 00:48 | |
| *** cwolferh has quit IRC | 01:00 | |
| *** cwolferh has joined #tripleo | 01:00 | |
| *** dmacpher has joined #tripleo | 01:05 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Fix rpmlint script-without-shebang errors https://review.openstack.org/270562 | 01:18 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Remove shebangs from non-execute files https://review.openstack.org/270563 | 01:18 |
| *** dmacpher has quit IRC | 01:29 | |
| *** penick has quit IRC | 01:33 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Remove shebangs from non-execute files https://review.openstack.org/270563 | 01:33 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Remove shebangs from non-execute files https://review.openstack.org/270563 | 01:36 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Fix rpmlint script-without-shebang errors https://review.openstack.org/270562 | 01:36 |
| *** shivrao has quit IRC | 01:40 | |
| *** sthillma has quit IRC | 01:52 | |
| *** tiswanso has joined #tripleo | 02:01 | |
| *** tiswanso has quit IRC | 02:03 | |
| *** tiswanso has joined #tripleo | 02:04 | |
| *** rhallisey has quit IRC | 02:28 | |
| *** cwolferh has quit IRC | 02:32 | |
| *** julim has quit IRC | 02:32 | |
| *** yamahata has quit IRC | 03:00 | |
| *** masco has joined #tripleo | 04:10 | |
| *** regebro is now known as Guest95110 | 04:15 | |
| *** rebrego has joined #tripleo | 04:15 | |
| *** Guest95110 has quit IRC | 04:15 | |
| *** rebrego is now known as regebro | 04:15 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: add support for package-installs to gentoo https://review.openstack.org/270597 | 04:19 |
| *** trozet has joined #tripleo | 04:21 | |
| *** tzumainn has quit IRC | 04:31 | |
| *** rasca has quit IRC | 04:35 | |
| *** rasca has joined #tripleo | 04:36 | |
| *** rasca has quit IRC | 04:41 | |
| *** rasca has joined #tripleo | 04:41 | |
| *** coolsvap|away is now known as coolsvap | 04:49 | |
| EmilienM | could someone look https://review.openstack.org/#/c/270477/ please? | 04:52 |
| EmilienM | run keystone in a wsgi process on the undercloud -> passing CI and already +2 | 04:52 |
| *** stendulker has joined #tripleo | 04:56 | |
| *** david-lyle has quit IRC | 05:02 | |
| *** david-lyle has joined #tripleo | 05:03 | |
| *** Marga_ has quit IRC | 05:07 | |
| *** cwolferh has joined #tripleo | 05:08 | |
| *** Marga_ has joined #tripleo | 05:08 | |
| *** rasca has quit IRC | 05:11 | |
| *** rasca has joined #tripleo | 05:13 | |
| *** shivrao has joined #tripleo | 05:18 | |
| *** rlandy has quit IRC | 05:28 | |
| *** tiswanso has quit IRC | 05:31 | |
| *** yuanying has joined #tripleo | 05:32 | |
| *** yuanying_ has quit IRC | 05:36 | |
| *** xinwu has quit IRC | 05:42 | |
| *** jaosorior has joined #tripleo | 05:47 | |
| *** xinwu has joined #tripleo | 05:52 | |
| *** xinwu has quit IRC | 06:08 | |
| *** rasca has quit IRC | 06:10 | |
| *** rasca has joined #tripleo | 06:11 | |
| *** ayoung has quit IRC | 06:16 | |
| *** anande has joined #tripleo | 06:18 | |
| *** spredzy has quit IRC | 06:18 | |
| *** spredzy has joined #tripleo | 06:18 | |
| *** liverpooler has quit IRC | 06:19 | |
| *** anande has quit IRC | 06:20 | |
| *** derekh has joined #tripleo | 06:31 | |
| openstackgerrit | Merged openstack/tripleo-common: Adds a simple overcloud tenant vm ping test to tripleo.sh https://review.openstack.org/241167 | 06:32 |
| derekh | Anybody want to add a second +2 to the ping test? https://review.openstack.org/#/c/262028/ | 06:33 |
| * derekh has done too much to the patch to merge it himself | 06:33 | |
| *** Marga_ has quit IRC | 06:37 | |
| *** xinwu has joined #tripleo | 06:37 | |
| *** jprovazn has joined #tripleo | 06:38 | |
| *** yamahata has joined #tripleo | 06:41 | |
| *** dmacpher has joined #tripleo | 06:47 | |
| *** rwsu has quit IRC | 06:52 | |
| *** trozet has quit IRC | 07:03 | |
| openstackgerrit | Derek Higgins proposed openstack/tripleo-common: WIP: split overcloud setup out of pingtest https://review.openstack.org/270631 | 07:07 |
| derekh | afazekas: I've propsed a patch to split out the setup stuff from that pingtest, still a WIP but we can discuss the details there https://review.openstack.org/270631 | 07:10 |
| derekh | afazekas: on the train at the moment, will work on it more today | 07:10 |
| *** shivrao has quit IRC | 07:11 | |
| afazekas | derekh, for foobar reason I can't get a working install since last night. What are the common issues with reusing the `root` vm ? Do you know about some `unbreaking` change from the prev days ? | 07:13 |
| afazekas | I should have the `not use tmpfs` for images | 07:13 |
| *** ukalifon has joined #tripleo | 07:13 | |
| afazekas | BTW, can we create swap on the instack vm ? | 07:13 |
| derekh | afazekas: the vm I gave you? you can do what ever you want with it, like this https://review.openstack.org/#/c/270274/ | 07:14 |
| derekh | afazekas: maybe you are hitting yesterdays bug, one sec | 07:14 |
| derekh | afazekas: have you got this https://review.openstack.org/#/c/270095/ | 07:15 |
| afazekas | /tmp/swap ??? | 07:15 |
| afazekas | isn't tmp already tmpfs ? | 07:15 |
| afazekas | you need /swap to be clear | 07:16 |
| afazekas | derekh, also consider swapping the bs with the count | 07:16 |
| derekh | nope, at least I didn't think it was, I'll double check | 07:16 |
| afazekas | derekh: even if it is not tmpfs at the moment, not recommended | 07:17 |
| derekh | afazekas: will do | 07:17 |
| *** alop has quit IRC | 07:19 | |
| afazekas | yee, it is really not a tmpfs :) | 07:19 |
| *** rcernin has joined #tripleo | 07:20 | |
| derekh | afazekas: yup http://paste.openstack.org/show/484510/ , although maybe I should move it incase it ever becomes one again | 07:20 |
| marios | morning | 07:21 |
| marios | derekh: thanks https://review.openstack.org/#/c/241167/ | 07:21 |
| marios | derekh: been trying to get back to that but i keep getting pulled under :/ | 07:21 |
| derekh | marios: no problem, just gotta get the one that wires it up merged now | 07:23 |
| afazekas | derekh, on the `root` vm, might be better to just to change the flavor, and extending it with some swap | 07:23 |
| derekh | afazekas: ahh yes, your talking about the undercloud, wouldn't that be a change to the undercloud image we build? That instance isn't started by nova, the undercloud image is copies to /var/lib/libvirt/... and we call a virsh start | 07:30 |
| derekh | afazekas: yes it would be better, the mkswap patch is just to find out if it help things before spending more time on it. | 07:30 |
| afazekas | derekh: root vm = what is the jenkins slave, for the undercloud and to the other it is image build part can help, also if you have cloud init (we have) if the domain created with preformed swap disk it is supposed to jump on it | 07:33 |
| afazekas | As I remember there is bz request for using swap on the overcloud vms (or just changing the memory over commit ratio to <=1.0 on n-cpu nodes) | 07:34 |
| *** jcoufal has joined #tripleo | 07:35 | |
| derekh | afazekas: "What are the common issues with reusing the `root` vm ?" | 07:39 |
| derekh | afazekas: so reusing that VM in the passed, the problems have been fairly obvious, things like "mkdir " failing because the directory already exists the second time round | 07:40 |
| afazekas | BTW: some magic from devstack, we might use it or something similar http://logs.openstack.org/04/225204/10/check/gate-tempest-dsvm-centos7/16a93fe/logs/dstat-csv.txt.gz | 07:49 |
| afazekas | one older variant also indicated the highest resource user process | 07:52 |
| derekh | nice | 07:53 |
| *** jaosorior has quit IRC | 07:53 | |
| afazekas | it is based on dstat (similar to iostat and vmstat), and easir to decode the output than the sa1 | 07:53 |
| *** liverpooler has joined #tripleo | 07:54 | |
| *** liverpooler has quit IRC | 07:55 | |
| *** jtomasek has joined #tripleo | 07:55 | |
| *** liverpooler has joined #tripleo | 07:55 | |
| *** dshulyak_ has joined #tripleo | 07:56 | |
| *** ayoung has joined #tripleo | 08:01 | |
| *** bvandenh has joined #tripleo | 08:05 | |
| *** jaosorior has joined #tripleo | 08:06 | |
| *** fgimenez has joined #tripleo | 08:06 | |
| *** fgimenez has quit IRC | 08:06 | |
| *** fgimenez has joined #tripleo | 08:06 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Add some swap to the undercloud https://review.openstack.org/270274 | 08:08 |
| *** tosky has joined #tripleo | 08:09 | |
| *** bvandenh has quit IRC | 08:13 | |
| marios | derekh: was that "Error: Could not prefetch mongodb_replset provider 'mongo'" resolved in ci? (thinking if i should recheck reviews yet) | 08:19 |
| derekh | marios: yup we pinned the mondo puppet module, EmilienM is working on the fix | 08:19 |
| derekh | *mongo | 08:20 |
| marios | derekh: thx | 08:20 |
| jaosorior | marios: Well, it seems to me that it's working | 08:20 |
| marios | jaosorior: yeah some reviews that ran yesterday | 08:21 |
| marios | (are failed and am going to recheck) | 08:21 |
| derekh | marios: wait | 08:22 |
| derekh | marios: well go ahead | 08:22 |
| marios | derekh: k, i only rechecked one | 08:22 |
| derekh | marios: just we said at the meeting that we wouldn't merge anything else without the ping test | 08:22 |
| marios | derekh: ah ack sorry missed that thx | 08:23 |
| derekh | so if you want to not have to recheck again you should wait untill this is merged | 08:23 |
| derekh | marios: https://review.openstack.org/#/c/262028/ | 08:23 |
| marios | derekh: right thanks | 08:24 |
| *** bvandenh has joined #tripleo | 08:24 | |
| derekh | marios: I hadn't +2'd bcause I submitted the last patchset, but I guess since both of us are core, one of us can | 08:25 |
| *** aufi has joined #tripleo | 08:25 | |
| marios | derekh: yeah i mean, i didn't do it because i am listed as owner... | 08:27 |
| *** mkovacik has joined #tripleo | 08:27 | |
| *** dmacpher has quit IRC | 08:27 | |
| derekh | marios: ok I'm going to merge it, it was just waiting on ci up to no | 08:27 |
| marios | derekh: but i guess it changed enough for it to be ok | 08:27 |
| derekh | *now | 08:27 |
| derekh | race ya | 08:28 |
| marios | derekh: ok am going to merge sorry was haing a closer look at the diff | 08:29 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Wire up the new --overcloud-pingtest into the tripleo.sh heredoc https://review.openstack.org/262028 | 08:30 |
| derekh | marios: cool | 08:31 |
| *** gfidente has joined #tripleo | 08:32 | |
| *** gfidente has quit IRC | 08:32 | |
| *** gfidente has joined #tripleo | 08:32 | |
| marios | derekh: thanks to you and bnemec for sorting it out | 08:32 |
| derekh | I wouldn't be surprised if we now hit more failed CI tests, but at least they are for reasons we need to fix.... | 08:32 |
| derekh | marios: no prob | 08:32 |
| *** paramite has joined #tripleo | 08:35 | |
| *** shardy has joined #tripleo | 08:39 | |
| openstackgerrit | afazekas proposed openstack-infra/tripleo-ci: [WIP] trying to add one block node https://review.openstack.org/270423 | 08:39 |
| *** mcornea has joined #tripleo | 08:43 | |
| marios | jaosorior: can you check https://review.openstack.org/#/c/213175/12/puppet/hieradata/controller.yaml | 08:46 |
| jaosorior | marios: Will do, I'm currently checking out this one though https://review.openstack.org/#/c/270110/1 | 08:48 |
| *** paramite is now known as paramite|afk | 08:48 | |
| marios | jaosorior: sure just think it is a rebase nit (removes the HTTP_X_FORWARDED_PROTO) | 08:49 |
| jaosorior | marios: Taking a quick look it seems to me like yeah, that config shouldn't have been deleted | 08:50 |
| *** mkovacik has quit IRC | 08:50 | |
| marios | jaosorior: k, just pointing it out cos i know you added that stuff | 08:50 |
| jaosorior | marios: Thanks dude | 08:51 |
| marios | np :) | 08:52 |
| *** derekh has quit IRC | 08:52 | |
| jaosorior | marios: this commit seems alright https://review.openstack.org/#/c/270110/1, but just a quick question. Why add the nova_ipv6 to the structuredconfig instead of using the nova::use_ipv6 that's already there? | 08:53 |
| jaosorior | you could probably still get that using hiera, could you? | 08:53 |
| marios | jaosorior: that is a good question. i saw it was actually setting something in the nova puppet class and wasn't sure it'd be available like 'normal' hiera data. i should check though | 08:54 |
| marios | i think i still have a v6 beaker box... checking now | 08:55 |
| marios | jaosorior: yeah :/ | 08:57 |
| *** bvandenh has quit IRC | 08:57 | |
| *** dtantsur|afk is now known as dtantsur | 08:58 | |
| marios | jaosorior: well it *is* in the normal hieradata... i wonder if we can pick it up like normal though... i mean | 08:58 |
| marios | [root@overcloud-controller-0 ~]# grep 'nova.*ipv6' /etc/puppet/hieradata/controller.yaml | 08:58 |
| marios | nova::use_ipv6: true | 08:58 |
| marios | nova_ipv6: true | 08:58 |
| marios | jaosorior: yeah, should be fine, example | 08:59 |
| jaosorior | marios: yeah, would be nicer if it's possible :D | 08:59 |
| marios | hiera('neutron::enable_metadata_agent',true) | 08:59 |
| jaosorior | aaah, there you go | 08:59 |
| marios | jaosorior: yeah it was one of the many things that were fixed over the weekend, hurredly | 08:59 |
| marios | jaosorior: thanks! | 08:59 |
| jaosorior | marios: Everything else looks good in that CR | 09:00 |
| marios | jaosorior: tx, if you add a comment i can push a revision in a sec | 09:00 |
| marios | jaosorior: i can add a comment, just thought you may want to cos you thought of it | 09:01 |
| *** nico_auv has joined #tripleo | 09:02 | |
| marios | tx | 09:02 |
| *** mbound has joined #tripleo | 09:02 | |
| *** bvandenh has joined #tripleo | 09:10 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Fixup the memcached servers string in nova.conf for v6 https://review.openstack.org/270110 | 09:12 |
| *** paramite|afk is now known as paramite | 09:12 | |
| *** devvesa has joined #tripleo | 09:14 | |
| *** jistr has joined #tripleo | 09:16 | |
| afazekas | http://fpaste.org/313100/33679391/ | 09:19 |
| afazekas | I have tried to run a minimal tempest ^ , 4G memory is simply not enough for an operating controller | 09:19 |
| shardy | Simple patch needs a +A, passing CI: https://review.openstack.org/#/c/267917/ | 09:19 |
| afazekas | Are there any chance to reconfigure the domains with 6G memory at least ? | 09:20 |
| jaosorior | shardy: This one also :D https://review.openstack.org/#/c/268647/ | 09:21 |
| *** derekh has joined #tripleo | 09:23 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Apply --min-tmpfs 5 to agent ramdisk build too https://review.openstack.org/269192 | 09:23 |
| * afazekas FYI: something like this should be enough to run tempest: http://fpaste.org/313103/36840314/ | 09:27 | |
| * afazekas http://fpaste.org/313104/33685331/ | 09:29 | |
| *** bvandenh has quit IRC | 09:30 | |
| shardy | jaosorior: Yeah, need to get that one to pass the HA job tho and/or fix the issues causing that job to fail so often :( | 09:32 |
| shardy | afazekas: Are you planning to run tempest as part of an upstream CI run? | 09:33 |
| *** bvandenh has joined #tripleo | 09:34 | |
| shardy | afazekas: we're hitting various OOM issues, so increaing the defaults may be a good idea, but the other challenge is the time taken to run the tests | 09:35 |
| shardy | e.g we're regularly hitting the infra timeouts on HA/ceph jobs already | 09:35 |
| shardy | derekh can probably advise if we have capacity to increase memory defaults any further | 09:36 |
| jaosorior | shardy: Funky thing is that patch isn't even being used, it's just the definition of a resource in puppet. So I'm pretty puzzled that it was failing, but gonna recheck it. | 09:36 |
| shardy | jaosorior: it's probably failing for an unrelated reason, if you check the undercloud logs you'll probably find something got killed by the OOM killer :( | 09:37 |
| jaosorior | OOM? | 09:38 |
| shardy | https://lwn.net/Articles/317814/ | 09:38 |
| shardy | out of memory killer | 09:38 |
| jaosorior | I see | 09:39 |
| shardy | we're running with not-really-enough RAM and no swap, everywhere | 09:39 |
| afazekas | shardy: yes, i am planing | 09:39 |
| shardy | afazekas: cool, better testing (and help with upstream CI) will be awesome, but I wonder about the runtime | 09:40 |
| shardy | how long does tempest take to run? | 09:40 |
| *** jcoufal has quit IRC | 09:40 | |
| afazekas | shardy: it depends on the performance of the cluster | 09:40 |
| afazekas | tempest it self is parallel by default | 09:40 |
| afazekas | 40m..80m is full run with small clusters | 09:41 |
| shardy | afazekas: Ok, I don't think we'll have capacity for that duration on any of our current tests | 09:41 |
| afazekas | it can be limited to smaller test set , like just the scenarios | 09:41 |
| shardy | because we're hitting timeouts already | 09:42 |
| shardy | Yeah, I think it'll need to be limited to something which can run in less than about 10 minutes, and even then it'll probably only work on the nonha job | 09:42 |
| *** bvandenh has quit IRC | 09:42 | |
| afazekas | I know, I frequently crying about installation speed :) | 09:42 |
| shardy | afazekas: well, it's not just the speed, it's the performance of our CI | 09:42 |
| shardy | anyway, just so long as you're aware before committing too much time to it :) | 09:43 |
| shardy | hopefully we can work out a solution | 09:43 |
| afazekas | shardy: do not think is it significantly faster on bigger machine, one cpu user process, which is cpu bound will not be faster anywhere else | 09:44 |
| jaosorior | Maybe it's time to start separating undercloud CI from overcloud CI? like, use a pre-built undercloud when running tests related to t-h-t for instance | 09:44 |
| afazekas | shardy: the task description was not involved full runs :) | 09:44 |
| shardy | jaosorior: Yeah, there may be optimisations like that, or not building images for every change | 09:45 |
| afazekas | shardy: I'll try to cut to 4..12 test to the voting jobs, and try to do full run on the periodic ones | 09:45 |
| shardy | afazekas: ack, sounds good, thanks! :) | 09:46 |
| jaosorior | shardy: Yeah, that would probably help too | 09:46 |
| shardy | jaosorior: the problem is, what happens when a t-h-t change depends-on a puppet-* change | 09:46 |
| shardy | then you have to build the images, or change CI to use DeployArtefacts and patch in the new modules | 09:46 |
| jaosorior | uhm... yep, there would be a lot more logic required in the C I | 09:47 |
| jaosorior | *CI | 09:47 |
| afazekas | shardy: BTW, who know the hw spec of the BM nodes which hosting the overcloud vms? | 09:49 |
| shardy | afazekas: derekh will know | 09:51 |
| afazekas | Is I remember one gearman worker was sized to much more small VM than the jobs actually uses, so reserving less, but bigger nodes seams possible | 09:52 |
| afazekas | ack | 09:52 |
| derekh | afazekas: shardy on the run time, we're initially hopeing to get tempest into our periodic jobs, so increasing the run time with tempest shouldn't be an issue | 09:53 |
| derekh | i.e. is wont effect the time devs have to wait on feedback | 09:53 |
| shardy | derekh: ack, Ok thanks | 09:54 |
| derekh | afazekas: yes, your correct, each testenv has a load of nodes we don't use, 10 I think | 09:54 |
| derekh | afazekas: but we just did that so that we wouldn't need to rebuild the nodes if we wanted to add anything | 09:55 |
| *** kbyrne has joined #tripleo | 09:55 | |
| derekh | afazekas: and beacuase we were using the devtest_testenv script to build it, it create lots of nodes incase developers wanted to use them | 09:56 |
| derekh | afazekas: there is no way we could actually use all 10 nodes in the testenv, we are already overcomited as it is | 09:56 |
| derekh | afazekas: some number | 09:56 |
| derekh | afazekas: each test env host has 64G RAM , and supports 4 testenvs | 09:57 |
| afazekas | derekh, not just 3 env ? | 09:58 |
| afazekas | yes it is over-committed anyway | 09:58 |
| *** tosky has quit IRC | 09:59 | |
| derekh | afazekas: if a host get 4 HA tests at the same time, that (4G * 5 ) 20G by 4 , 80G commited , we are relying on the fact that some of the jobs on a testenv will be HA and some nonha, also that each job will be at a different s=tage in the test | 09:59 |
| afazekas | derekh: do you have stats about how many env are reserved ? | 09:59 |
| derekh | afazekas: I think it 4 per Host, I'll check now one sec | 09:59 |
| afazekas | lest time I hear 3 :) | 10:00 |
| afazekas | derekh, do they have available DIMM slot? | 10:00 |
| openstackgerrit | Merged openstack/instack-undercloud: Remove Py33 support https://review.openstack.org/266112 | 10:01 |
| *** aufi has quit IRC | 10:01 | |
| afazekas | usually dmdecode can tell | 10:01 |
| derekh | afazekas: 4 test envs per host on rh1 , http://paste.openstack.org/show/484535/ | 10:01 |
| derekh | afazekas: not sure about spare dims checking | 10:01 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates: Make External Load Balancer templates work with IPv6 [DO NOT MERGE] https://review.openstack.org/270700 | 10:02 |
| *** openstackgerrit has quit IRC | 10:02 | |
| *** openstackgerrit has joined #tripleo | 10:02 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Let Puppet update all packages on non-controllers https://review.openstack.org/268291 | 10:03 |
| afazekas | derekh: can you show also a /proc/cpuinfo ? | 10:03 |
| *** rasca_ has joined #tripleo | 10:03 | |
| *** pblaho has quit IRC | 10:03 | |
| *** electrofelix has joined #tripleo | 10:03 | |
| *** aufi has joined #tripleo | 10:04 | |
| *** yamahata has quit IRC | 10:04 | |
| *** rasca has quit IRC | 10:04 | |
| derekh | afazekas: cpuinfo http://paste.fedoraproject.org/313120/70671145/ | 10:04 |
| openstackgerrit | Merged openstack/instack-undercloud: Remove answers file support https://review.openstack.org/269147 | 10:06 |
| *** paramite has quit IRC | 10:07 | |
| *** aufi has quit IRC | 10:08 | |
| *** jaosorior has quit IRC | 10:09 | |
| *** aufi has joined #tripleo | 10:09 | |
| derekh | afazekas: If I'm reading this correctly we have 24 slots, and we're using 8 of them (8 x 8G | 10:09 |
| *** jaosorior has joined #tripleo | 10:10 | |
| *** akrivoka has joined #tripleo | 10:10 | |
| *** mkovacik has joined #tripleo | 10:10 | |
| *** jaosorior has quit IRC | 10:10 | |
| *** jaosorior has joined #tripleo | 10:11 | |
| *** athomas has quit IRC | 10:12 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/tripleo-ci: Remove unused urllib https://review.openstack.org/261182 | 10:12 |
| afazekas | derekh: how many cpu core/thread , is the machines expected remain in service after the warranty expired ? | 10:13 |
| *** dtantsur is now known as dtantsur|brb | 10:15 | |
| afazekas | derekh, the CPU type is not exposed to the guest, it is rumored to using cpu pass-trough can make things faster | 10:17 |
| *** xinwu has quit IRC | 10:19 | |
| *** athomas has joined #tripleo | 10:19 | |
| derekh | afazekas: are you talking about exposing it to the jenkins node, or the testenv nodes (probably both) | 10:19 |
| afazekas | both | 10:20 |
| *** xinwu has joined #tripleo | 10:20 | |
| afazekas | https://wiki.openstack.org/wiki/LibvirtXMLCPUModel 'host-passthrough' the rumors from here | 10:20 |
| derekh | afazekas: to be honest my preference would be to redeploy the whole thing with OVB soon | 10:20 |
| *** xinwu has quit IRC | 10:20 | |
| afazekas | disadvantage: live migration to inferior host will not work :) | 10:20 |
| derekh | afazekas: not sure about plans after warranty, the topic has never come up but its a question worth asking | 10:21 |
| *** paramite has joined #tripleo | 10:21 | |
| derekh | afazekas: I wouldn't worry about live migration, for this rack, ci jobs can be restarted, far simpler | 10:22 |
| afazekas | derekh: I am asking for nr host cpu core, does it worth to invest memory into this machines | 10:22 |
| afazekas | Obviously the controller nodes will use more and more memory when we get more and more service | 10:23 |
| *** aufi has quit IRC | 10:23 | |
| afazekas | The only thing what may mitigate it is to reducing the number of service worker processes | 10:23 |
| *** rasca_ has quit IRC | 10:24 | |
| *** aufi has joined #tripleo | 10:24 | |
| derekh | afazekas: each host has 2 X ( 6 cores, 12 threads), http://ark.intel.com/products/64594/Intel-Xeon-Processor-E5-2620-15M-Cache-2_00-GHz-7_20-GTs-Intel-QPI | 10:25 |
| derekh | afazekas: I should find out how long we intend the machines to stick around before we looking into more memory | 10:25 |
| *** rasca has joined #tripleo | 10:26 | |
| afazekas | derekh: for running mainly install test is it still ok to extend with memory | 10:27 |
| derekh | afazekas: to extend what with memory? I'm not sure if your talking about the testenv hosts, the jenkins nodes, the undercloud, or one of the overcloud nodes. | 10:29 |
| afazekas | derekh: other memory saving thing can be, using less memory with != controller (or undercloud) vms . for example with tempest we usually using this flavor for vms nova flavor-create m1.nano 42 64 0 1 | 10:29 |
| *** StevenK has quit IRC | 10:29 | |
| afazekas | derekh: testenv | 10:29 |
| afazekas | with m1.nano you can spin up a `lot of` vm even on 2G memory compute node | 10:30 |
| derekh | afazekas: yup, sounds like a good flavor to use | 10:31 |
| openstackgerrit | Merged openstack/python-tripleoclient: Sahara integration https://review.openstack.org/221418 | 10:32 |
| derekh | afazekas: back in a bit, got a meeting | 10:32 |
| *** paramite has quit IRC | 10:49 | |
| openstackgerrit | Jaume Devesa proposed openstack/python-tripleoclient: image build: add --builder-extra-args https://review.openstack.org/265827 | 10:52 |
| *** StevenK has joined #tripleo | 10:53 | |
| *** tosky has joined #tripleo | 10:58 | |
| *** fgimenez_ has joined #tripleo | 10:59 | |
| *** fgimenez has quit IRC | 10:59 | |
| *** fgimenez_ is now known as fgimenez | 11:00 | |
| *** paramite has joined #tripleo | 11:01 | |
| *** aufi has quit IRC | 11:01 | |
| *** aufi has joined #tripleo | 11:03 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Rename validate tox env to linters https://review.openstack.org/258173 | 11:07 |
| *** regebro has quit IRC | 11:11 | |
| *** mgould has joined #tripleo | 11:18 | |
| *** shadower has joined #tripleo | 11:19 | |
| gfidente | shardy, I saw your comment on https://review.openstack.org/#/c/235423/37 | 11:24 |
| gfidente | I think it'd be indeed nicer to do it that way | 11:24 |
| *** pblaho has joined #tripleo | 11:25 | |
| shardy | gfidente: cool - I felt bad as it looks nearly ready to land, but I think it may simplify things somewhat | 11:25 |
| shardy | in general, we've been pretty bad at copying everything everywhere for network stuff, so it'd be good to start getting away from that and parameterizing things where possible | 11:26 |
| *** regebro has joined #tripleo | 11:27 | |
| hewbrocca | shardy: code reuse! | 11:31 |
| *** jcoufal has joined #tripleo | 11:34 | |
| *** olap has joined #tripleo | 11:36 | |
| shardy | jistr: Hey did you get to the root-cause of your no valid host errors yesterday? | 11:36 |
| shardy | I've built a fresh environment and getting similar issues | 11:37 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: SSL/Cinder: enable ssl_header_handler filter https://review.openstack.org/268647 | 11:38 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Enable SSL middleware for cinder https://review.openstack.org/268651 | 11:38 |
| jaosorior | Just a rebase ^^ | 11:39 |
| jistr | shardy: hi, yes i did, it was most probably OOM issues (root cause is probably heat-engine eating memory). The first problem i saw in ironic-conductor logs was OOM error, and from that point on various issues appeared. Most probably OS started killing things. I restarted heat-engine to clear up memory, lowered it to 2 workers. Restarted ironic-conductor to re-register it with ironic-api which got me rid of ironic errors, then i was getting | 11:41 |
| jistr | neutron errors (in nova-compute.log) because ovs agent was down (probably killed too), so to make sure i restarted all neutron and nova services too. | 11:41 |
| gfidente | shardy, so I want to update the external-lb thing to make it work first | 11:42 |
| shardy | jistr: ack, OK thanks for the info | 11:42 |
| gfidente | but then I'll try to rewrite things using both the format param AND the str_split | 11:42 |
| gfidente | updating those same patches | 11:42 |
| shardy | gfidente: Ok, sounds great | 11:42 |
| gfidente | tks! | 11:43 |
| shardy | if we need to backport that patch to kilo, it's fine to do the str_split in a separate patch, or I can rebase https://review.openstack.org/#/c/266246/ on to it | 11:43 |
| *** stendulker_ has joined #tripleo | 11:43 | |
| *** stendulker has quit IRC | 11:47 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Make External Load Balancer templates work with IPv6 https://review.openstack.org/270700 | 11:50 |
| *** stendulker_ has quit IRC | 12:00 | |
| *** fgimenez has quit IRC | 12:05 | |
| *** fgimenez has joined #tripleo | 12:05 | |
| *** fgimenez has quit IRC | 12:05 | |
| *** fgimenez has joined #tripleo | 12:05 | |
| *** coolsvap is now known as coolsvap|away | 12:07 | |
| *** dtantsur|brb is now known as dtantsur | 12:13 | |
| gfidente | jistr, marios I noticed the tunnelidranges and vniranges parameters are not wired from top into controller and compute templates anymore | 12:16 |
| gfidente | do you know if that was done on purpose? | 12:16 |
| marios | gfidente: not something i did afaicr... who does git blame? | 12:17 |
| jistr | hmm i don't recall that | 12:17 |
| gfidente | I am not sure because the lines are actually missing so git blame doesn't say much | 12:17 |
| gfidente | I'll wire those back in and add a bunch of people to see if anything comes up | 12:17 |
| gfidente | I am not sure if more are missing | 12:19 |
| gfidente | shardy, I wonder if we shouldn't remove the default for things supposed to come from the parent stack? | 12:19 |
| gfidente | and the description! :) | 12:20 |
| jistr | gfidente: so you mean that there exist top level params but they aren't wired to the node params? | 12:21 |
| gfidente | jistr, yeah | 12:21 |
| jistr | gfidente: i just went through the code and it looks like this wiring never existed. We missed that because if you pass it via parameter_defaults, it works anyway. | 12:22 |
| gfidente | uhm like never never? | 12:22 |
| gfidente | I remember client in 7x was passing those as parameter | 12:22 |
| marios | ah right :/ there is some way by doing sthing like git log -p but can't figure out how to get the commit or change id yet | 12:23 |
| marios | gfidente: ^^ (about the deletion) | 12:23 |
| jistr | never never, at least upstream. The patch which introduces the params to overcloud-without-mergepy.yaml only adds them to parameter list, doesn't use them anywhere else in the file. | 12:23 |
| shardy | gfidente: IMO it depends which templates, e.g when we move to composable roles we'll probably stop passing nearly all parameters from overcloud.py, and instead rely on parameter_defaults for per-service parameters | 12:23 |
| shardy | if we're sure the parent will always pass the value, removing the default is probably fine tho | 12:24 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Wire NeutronTunnelIdRanges and NeutronVniRanges to to node templates https://review.openstack.org/270764 | 12:24 |
| marios | gfidente: yeah like git log -p puppet/controller.yaml should help you find it at least | 12:24 |
| *** thrash|g0ne is now known as thrash | 12:24 | |
| jistr | marios, gfidente: be0d3f3520037c4e2e9fb4f9a235171aaad7e253 | 12:25 |
| marios | https://review.openstack.org/#/c/199509/ | 12:25 |
| marios | (jistr i used that commit id to get the review) | 12:26 |
| marios | jistr: but this adds them... gfidente are they removed? | 12:26 |
| marios | 14:16 < gfidente> jistr, marios I noticed the tunnelidranges and vniranges parameters are not wired from top into controller and compute templates anymore | 12:26 |
| gfidente | marios, nah they're not wired | 12:27 |
| gfidente | and we didn't notice I think | 12:27 |
| jistr | marios: i think what gfidente meant is that they're only in the param list but not wired further here https://review.openstack.org/#/c/199509/3/overcloud-without-mergepy.yaml | 12:27 |
| jistr | (passed to compute/controller) | 12:27 |
| * marios palmface | 12:28 | |
| jistr | if you pass it via parameter_defaults, it doesn't change actual behavior though | 12:28 |
| gfidente | shardy, so I am going to try a mass delete of default in controller and compute yaml | 12:28 |
| jistr | so that's why we probably didn't notice | 12:28 |
| marios | jistr: yeah cos they are wired into the controller/compute itself | 12:28 |
| jistr | yea | 12:28 |
| shardy | gfidente: I personally am -1 on that, for the reason I just mentioned (composable roles) | 12:29 |
| gfidente | but again, the client was using parameters: initially | 12:29 |
| shardy | all those parameters are going to move into per-service templates, where the defaults will be needed | 12:29 |
| shardy | and all parameters for services will be removed from overcloud.yaml | 12:29 |
| shardy | gfidente: however, I'm +1 on removing any defaults where no sane default actually exists :) | 12:30 |
| gfidente | shardy, yeah I get that with composables we'll add back the default | 12:30 |
| shardy | gfidente: Ok, cool | 12:32 |
| shardy | https://review.openstack.org/#/c/237370/7/puppet/roles/pacemaker/glance-api.yaml | 12:32 |
| gfidente | shardy, but my point is to sanitize where it is needed and where it isn't | 12:32 |
| gfidente | and fail if we miss it | 12:32 |
| shardy | gfidente: ack, that makes sense, I'm just hoping it doesn't make the refactoring for composable roles harder | 12:32 |
| shardy | I guess we'll copy the parameter definitions from overcloud.yaml instead, which should be OK | 12:33 |
| * shardy goes to lunch | 12:33 | |
| jistr | gfidente: what about removing the top-level params instead? everybody who used parameter_defaults will have it working still, and everybody who used parameters never had it working anyway | 12:34 |
| gfidente | jistr, that is my point for using always parameter_defaults | 12:35 |
| gfidente | like we're doing now | 12:35 |
| jistr | yea +1. i'll mention it on the review then | 12:35 |
| gfidente | as long as we take one route | 12:36 |
| gfidente | and to be honest, that would also cleanup the templates from lines which are just passing variables along, so I'd like that too | 12:37 |
| jistr | yea | 12:38 |
| jistr | hiera-way | 12:39 |
| gfidente | jistr, comments, I'd like us to act into one of the two directions | 12:46 |
| *** trown|outttypeww is now known as trown | 12:48 | |
| jistr | gfidente: is there something else unwired though? or do you mean even for the params which are wired? | 12:48 |
| gfidente | I mean for all | 12:49 |
| gfidente | a proper cleanup of a 1000lines! | 12:49 |
| gfidente | :P | 12:49 |
| jistr | yea, this probably makes sense to be done while extracting the composable roles | 12:51 |
| jistr | removing top level params was problematic when CLI used parameters instead of parameter_defaults, but i think nowadays we don't have to worry about that | 12:52 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: [WIP] OpenContrail integration https://review.openstack.org/238097 | 12:59 |
| afazekas | derekh, I guess the nested guest support is disabled on testenv, however the n-cpu is configured to use 'kvm' instead of 'qemu' | 13:05 |
| derekh | afazekas: yes, nested kvm is disabled, we (only yesterday), fixed that config to set it to qemu https://review.openstack.org/#/c/262028/6/toci_gate_test.sh | 13:07 |
| afazekas | nova.conf [libvirt].virt_type is expected to be configured by the user, or is it based on auto detection ? | 13:07 |
| afazekas | derekh: if the host at least el 7.1 nested guest can be enabled (unofficial) | 13:08 |
| afazekas | derekh, thx, I'll apply that change | 13:09 |
| *** jtomasek has quit IRC | 13:09 | |
| *** rhallisey has joined #tripleo | 13:10 | |
| derekh | afazekas: last time I saw that tried (not sure what version it was) the host kernels were core dumping a lot, would be worth trying again when doing a redeploy | 13:10 |
| *** akuznetsov has joined #tripleo | 13:11 | |
| *** tzumainn has joined #tripleo | 13:16 | |
| *** jayg|g0n3 is now known as jayg | 13:16 | |
| *** dprince has joined #tripleo | 13:17 | |
| *** weshay_xchat has joined #tripleo | 13:18 | |
| *** weshay_xchat is now known as weshay | 13:18 | |
| *** fgimenez has quit IRC | 13:19 | |
| afazekas | derekh: It can made the 'heat-slow' named job 'heat-fast' :) | 13:19 |
| *** fgimenez has joined #tripleo | 13:19 | |
| *** fgimenez has quit IRC | 13:19 | |
| *** fgimenez has joined #tripleo | 13:19 | |
| derekh | afazekas: ;-) | 13:20 |
| afazekas | If by accident machines get replaced >=haswell cpu also recommended for 'heat-even-faster' ;-) | 13:23 |
| *** lblanchard has joined #tripleo | 13:33 | |
| EmilienM | jistr: can you look https://review.openstack.org/#/c/270477/ please ? | 13:36 |
| dprince | jistr: https://review.openstack.org/#/c/269068/ too | 13:39 |
| *** masco has quit IRC | 13:39 | |
| jistr | alright :) | 13:40 |
| jistr | yea i think UC is fine with httpd as there's no HA arch to be worried about | 13:40 |
| EmilienM | derekh: good news, the mongodb should work | 13:41 |
| EmilienM | derekh: looking at zuul, gate-tripleo-ci-f22-nonha is green when unpinning | 13:41 |
| derekh | EmilienM: great, thanks | 13:42 |
| EmilienM | derekh: so https://review.openstack.org/#/c/270451/ should be ok (after job results) | 13:42 |
| EmilienM | jistr: well, FYI keystone under wsgi is not an option anymore in Mitaka | 13:42 |
| *** julim has joined #tripleo | 13:42 | |
| jistr | hmm though what was the result re yesterday's discussion about not landing anything unless it moves the pingtest forward? (i just approved and un-approved the change :) ) | 13:43 |
| EmilienM | jistr: why don't you merge it? | 13:43 |
| marios | jistr: i think that is all done now derekh ? | 13:43 |
| marios | derekh: the toci change was what we were waiting for? | 13:43 |
| * marios driveby comment back to rebases | 13:43 | |
| marios | jistr: (this landed this morning https://review.openstack.org/#/c/262028/6) | 13:44 |
| * jistr checks recent CI logs for pingtest | 13:48 | |
| openstackgerrit | Merged openstack/instack-undercloud: Clean out os-refresh-config on every run https://review.openstack.org/266404 | 13:48 |
| EmilienM | jistr: that's an undercloud patch | 13:49 |
| EmilienM | I don't get why we don't move forward with that | 13:49 |
| jistr | EmilienM: yea i was just going to say "pingtest *is* in CI, this would probably not cause overcloud to become unpingable all of a sudden" | 13:50 |
| jistr | EmilienM: i'm just doing "measure twice, cut once" (not sure if that's just a czech saying or a more global one :) ) | 13:51 |
| hewbrocca | jistr: I've certainly heard it in English | 13:51 |
| trown | jistr: definitely global, and definitely appreciated | 13:52 |
| shardy | derekh: Hey, I'm trying to build oslo.utils with tripleo.sh --delorean-build, and it's failing because map-project-name only maps from oslo-utils to the rdopkg python-oslo-utils | 13:52 |
| openstackgerrit | Brad P. Crochet proposed openstack/instack-undercloud: Clean out os-refresh-config on every run https://review.openstack.org/270809 | 13:52 |
| shardy | derekh: do you think a patch to map-project-name adding a --upstream option would be OK? | 13:52 |
| shardy | Or, I can special-case it inside tripleo.sh | 13:52 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Set NeutronMetadataProxySharedSecret https://review.openstack.org/268131 | 13:53 |
| shardy | https://github.com/openstack-packages/delorean/blob/master/scripts/map-project-name#L16 | 13:53 |
| shardy | basically I want package["upstream"], not just "name" | 13:53 |
| openstackgerrit | Merged openstack/instack-undercloud: run keystone in a wsgi process https://review.openstack.org/270477 | 13:53 |
| shardy | Sorry, I just realized apevec actually wrote that script, I should probably ask him ;) | 13:54 |
| derekh | jistr: As I understand it we're ok to merge now that the ping test is running, of course you'll need to recheck if your CI results are from before the ping test | 13:54 |
| *** akuznetsov has quit IRC | 13:54 | |
| jistr | derekh: yeah though the nature of the change wasn't overcloud related, as EmilienM said | 13:55 |
| *** egafford has left #tripleo | 13:55 | |
| *** egafford has joined #tripleo | 13:56 | |
| *** weshay has quit IRC | 14:01 | |
| *** shardy has quit IRC | 14:03 | |
| *** shardy has joined #tripleo | 14:05 | |
| marios | [m@m openstack-tripleo-heat-templates]$ git log | grep "Use KeystoneAdminApiVirtualIP for heat keystone domain admin setup" Use KeystoneAdminApiVirtualIP for heat keystone domain admin setup | 14:11 |
| *** trozet has joined #tripleo | 14:13 | |
| openstackgerrit | Ethan Gafford proposed openstack/python-tripleoclient: Trove integration https://review.openstack.org/233241 | 14:17 |
| *** weshay_xchat has joined #tripleo | 14:18 | |
| *** morazi has joined #tripleo | 14:20 | |
| *** tiswanso has joined #tripleo | 14:22 | |
| *** tiswanso has quit IRC | 14:26 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: [WIP] OpenContrail integration https://review.openstack.org/238097 | 14:26 |
| *** rlandy has joined #tripleo | 14:27 | |
| *** fgimenez has quit IRC | 14:27 | |
| *** fgimenez has joined #tripleo | 14:30 | |
| *** fgimenez has quit IRC | 14:30 | |
| *** fgimenez has joined #tripleo | 14:30 | |
| *** jtomasek has joined #tripleo | 14:31 | |
| *** tiswanso has joined #tripleo | 14:32 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-heat-templates: Currently the vnc server on the compute nodes binds on 0.0.0.0. While this works with IPv4 addresses it breaks connectivity with IPv6 addressing. This change allows the vnc server on the compute nodes to listen on the address assigned on the network speci https://review.openstack.org/270831 | 14:36 |
| *** absubram has joined #tripleo | 14:37 | |
| *** egafford has quit IRC | 14:41 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-heat-templates: Allow the vnc server to bind on IPv6 address on computes Currently the vnc server on the compute nodes binds on 0.0.0.0. While this works with IPv4 addresses it breaks connectivity with IPv6 addressing. This change allows the vnc server on the compute nod https://review.openstack.org/270831 | 14:42 |
| *** egafford has joined #tripleo | 14:42 | |
| *** akuznetsov has joined #tripleo | 14:46 | |
| *** egafford has quit IRC | 14:48 | |
| shardy | Hey all, can we land this change, which adds some swap to the undercloud? | 14:50 |
| shardy | https://review.openstack.org/#/c/270274/ | 14:50 |
| shardy | a bunch of jobs have been failing CI because the OOM killer randomly kills heat-engine processes | 14:50 |
| *** liverpooler has quit IRC | 14:50 | |
| shardy | until we work out a more permanent solution, some swapping seems less-bad | 14:51 |
| shardy | devvesa: ^^ FYI | 14:51 |
| therve | shardy, Is that a recent issue? | 14:53 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-heat-templates: Allow the vnc server to bind on IPv6 address on computes https://review.openstack.org/270831 | 14:53 |
| shardy | therve: Yes, we increased the vcpu count for the undercloud, now we're running out of ram because of lots more workers | 14:54 |
| therve | shardy, Ah yeah, I thought that it may be related to that. Reducing the number of workers is a potential solution too. | 14:54 |
| shardy | therve: but, we needed the workers in some cases, e.g there were RPC timeouts w/heat because we were attempting to use a single heat-engine process for huge tripleo stacks | 14:55 |
| shardy | therve: Yeah, there may be some tuning we can do, but the main reason for the CPU count increase was to increase the number of workers ;) | 14:55 |
| shardy | basically, we need more memory | 14:55 |
| afazekas | shardy: we can add swap creation as pre deployment script/hook as temporary workaround | 14:56 |
| *** julim has quit IRC | 14:56 | |
| *** akuznetsov has quit IRC | 14:56 | |
| afazekas | As I remember the templates has this option | 14:56 |
| shardy | afazekas: It's the undercloud that needs it | 14:56 |
| shardy | afazekas: the patch I referenced above adds some swap prior to attempting the overcloud deploy | 14:56 |
| slagle | shardy: it lgtm, i'm happy to land it | 14:57 |
| afazekas | sorry, I already +1 ed it, but we will need more mmeory on the controllers soon (if not now) | 14:57 |
| shardy | afazekas: ack, yeah that's a separate issue - atm we just need to get CI jobs which were working back to a reliable state | 14:58 |
| shardy | slagle: thanks | 14:58 |
| afazekas | shardy: for the monitoring we could add a similar dstat script as devstack uses | 14:58 |
| shardy | afazekas: Yeah, something like that would be helpful | 14:59 |
| afazekas | http://logs.openstack.org/04/225204/10/check/gate-tempest-dsvm-centos7/16a93fe/logs/dstat-csv.txt.gz | 14:59 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add some swap to the undercloud https://review.openstack.org/270274 | 14:59 |
| *** rpothier has joined #tripleo | 15:01 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Generate a unique DeployIdentifier on updates https://review.openstack.org/268126 | 15:02 |
| *** julim has joined #tripleo | 15:02 | |
| afazekas | shardy, what is the right mailing list to speak about triplo-ci infra (rh1) ? | 15:04 |
| shardy | afazekas: openstack-dev with the [TripleO] subject, although I think only derekh and dprince have access to the actual infrastructure servers | 15:09 |
| openstackgerrit | Dougal Matthews proposed openstack/tuskar: Retire the Tuskar codebase. https://review.openstack.org/270851 | 15:11 |
| derekh | shardy: afazekas the people on this list have access, http://git.openstack.org/cgit/openstack/tripleo-incubator/tree/tripleo-cloud/tripleo-cd-admins | 15:11 |
| derekh | shardy: afazekas although in reality only a small subset of people use the access and know where things are | 15:12 |
| shardy | derekh: thanks, looks like that list should be pruned down to three ;) | 15:12 |
| afazekas | derekh, is jog0 still around ? | 15:12 |
| derekh | shardy: yup probably | 15:13 |
| openstackgerrit | Dougal Matthews proposed openstack/tuskar-ui: Retire the Tuskar UI codebase. https://review.openstack.org/270854 | 15:13 |
| *** akuznetsov has joined #tripleo | 15:13 | |
| afazekas | derekh, may be memory is broken, but as I remember nowadays he does something else | 15:14 |
| derekh | afazekas: yup, a bunch of people can be taken from the list, I'll submit a patch now, not sure if the script that syncs access hadn't delete thought | 15:15 |
| derekh | dprince: do you know ? ^ | 15:15 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tuskarclient: Retire the Tuskar Client codebase. https://review.openstack.org/270855 | 15:15 |
| dprince | derekh: I think the script adds, but I seem to remember deleting being manual | 15:15 |
| dprince | derekh: Will and Kambiz can help us w/ that on the bastion | 15:16 |
| dprince | derekh: I would just email them to double check... | 15:16 |
| derekh | dprince: I'm going to submit a patch to remove people from the list that are no longer on tripleo, then ping one of them about it when its merged | 15:16 |
| dprince | derekh: cool, that is the correct way to proceed here I think | 15:17 |
| *** lazy_prince has joined #tripleo | 15:19 | |
| openstackgerrit | Derek Higgins proposed openstack/tripleo-incubator: Remove admin no longer working on tripleo https://review.openstack.org/270858 | 15:19 |
| derekh | dprince: shardy afazekas ^ | 15:19 |
| openstackgerrit | Derek Higgins proposed openstack/tripleo-incubator: Remove admins no longer working on tripleo https://review.openstack.org/270858 | 15:20 |
| *** akuznetsov has quit IRC | 15:23 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: OpenContrail heat templates https://review.openstack.org/238097 | 15:23 |
| *** gchamoul has quit IRC | 15:24 | |
| EmilienM | jistr: can you land https://review.openstack.org/#/c/270451/ please ? | 15:25 |
| EmilienM | thanks | 15:25 |
| *** akuznetsov has joined #tripleo | 15:25 | |
| jistr | EmilienM: landing | 15:25 |
| dprince | derekh: fire when ready | 15:25 |
| openstackgerrit | Merged openstack/tripleo-common: Revert "Pin the puppetlabs mongodb puppet module" https://review.openstack.org/270451 | 15:27 |
| EmilienM | jistr: you on fire! | 15:27 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: OpenContrail heat templates https://review.openstack.org/238097 | 15:28 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-common: tripleo.sh --delorean-build handle oslo.* package builds https://review.openstack.org/270864 | 15:34 |
| *** dprince has quit IRC | 15:35 | |
| *** akuznetsov has quit IRC | 15:35 | |
| *** mbound has quit IRC | 15:37 | |
| *** tiswanso has quit IRC | 15:38 | |
| *** tiswanso has joined #tripleo | 15:39 | |
| *** egafford has joined #tripleo | 15:40 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Don't write CLOUDNAME to the hosts file. https://review.openstack.org/267917 | 15:44 |
| *** masco has joined #tripleo | 15:47 | |
| openstackgerrit | Nicolas Hicher proposed openstack/tripleo-heat-templates: OpenContrail heat templates https://review.openstack.org/238097 | 15:50 |
| *** aufi has quit IRC | 15:57 | |
| *** shardy has quit IRC | 15:57 | |
| *** shardy has joined #tripleo | 15:59 | |
| *** jdob has joined #tripleo | 16:03 | |
| *** gchamoul has joined #tripleo | 16:08 | |
| *** rcernin has quit IRC | 16:09 | |
| *** lazy_prince has quit IRC | 16:10 | |
| *** yamahata has joined #tripleo | 16:10 | |
| *** regebro has quit IRC | 16:10 | |
| *** mbound has joined #tripleo | 16:13 | |
| *** mkovacik has quit IRC | 16:14 | |
| *** regebro has joined #tripleo | 16:14 | |
| *** slagle has quit IRC | 16:14 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/python-tripleoclient: Install python-hardware package on the agent image by default https://review.openstack.org/270885 | 16:15 |
| afazekas | derekh,dprince: I started to log how many testenv is in use, how long sample history can be considered as big enough to say what is the avg usage ? | 16:16 |
| EmilienM | ayoung: the rebase on https://review.openstack.org/213175 failed... | 16:17 |
| EmilienM | ayoung: I'll update the patch | 16:17 |
| ayoung | EmilienM, yeah, there were a slew of lint errors, but I couldn't get lint to run on my home system. tox was just horribly anti-social when I tried. | 16:18 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-common: Update pingtest for newer heatclient https://review.openstack.org/270890 | 16:18 |
| derekh | afazekas: probably a few weeks, but we need to allow for peak times of the day at peak times of the cycle, so average usage doesn't tell us much | 16:18 |
| *** bnemec has quit IRC | 16:18 | |
| *** trozet has quit IRC | 16:19 | |
| *** trozet has joined #tripleo | 16:19 | |
| *** egafford1 has joined #tripleo | 16:19 | |
| *** egafford1 has quit IRC | 16:19 | |
| derekh | afazekas: for example we know most of the time most of the testenvs arn't in use, but we need to be able to deal with peaks in order to ensure a backlog doesn't develop | 16:20 |
| afazekas | derekh: I can weight it, but probbaly we can also decrees the number of env in use we can make faster each env .. | 16:20 |
| *** bnemec has joined #tripleo | 16:20 | |
| *** egafford has quit IRC | 16:22 | |
| *** jdob has quit IRC | 16:25 | |
| *** jdob has joined #tripleo | 16:25 | |
| *** regebro has quit IRC | 16:27 | |
| *** regebro has joined #tripleo | 16:29 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common: Use the new yaml syntax in CI https://review.openstack.org/254886 | 16:30 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common: Build image files from definitions in yaml https://review.openstack.org/235569 | 16:30 |
| *** jdob1 has joined #tripleo | 16:33 | |
| *** marios has quit IRC | 16:34 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud: Update puppet configuration for ironic-inspector https://review.openstack.org/270904 | 16:34 |
| dtantsur | I've no idea about puppet, so this ^^ might be terribly wrong | 16:35 |
| *** jdob has quit IRC | 16:38 | |
| *** olap has quit IRC | 16:39 | |
| *** paramite has quit IRC | 16:42 | |
| *** regebro has quit IRC | 16:43 | |
| *** rcernin has joined #tripleo | 16:49 | |
| trown | dtantsur: it is not terribly wrong... we just want to put those values in the hiera | 16:50 |
| trown | dtantsur: https://review.openstack.org/gitweb?p=openstack/instack-undercloud.git;a=blob;f=elements/puppet-stack-config/puppet-stack-config.yaml.template;h=39bf30448f1a4d6accacbcf2a9cf9b68cf95f916;hb=25436a2991e622a5509fc27f16b33877c7bcd99e | 16:50 |
| *** masco has quit IRC | 16:51 | |
| *** bnemec has quit IRC | 16:52 | |
| *** Marga_ has joined #tripleo | 16:52 | |
| *** marios has joined #tripleo | 16:54 | |
| dtantsur | trown, oh, that's how it is done! thanks. do you know why some values are set in puppet-stack-config.pp ? | 16:54 |
| trown | dtantsur: legacy I think | 16:54 |
| dtantsur | got it, will fix | 16:54 |
| trown | dtantsur: anything new should go in hiera | 16:54 |
| EmilienM | jistr: could we land https://review.openstack.org/#/c/238013/ please? | 16:56 |
| * jistr the lander | 16:58 | |
| jistr | oof CI run Dec 10 | 16:58 |
| *** davidlenwell has quit IRC | 16:58 | |
| jistr | i don't suppose it's going to break but 1.5 months is a lot, i'm going to recheck to make sure | 16:58 |
| EmilienM | jistr: sure | 16:59 |
| EmilienM | jistr: yeah I have a lot of old patches | 16:59 |
| EmilienM | I'm trying to get rid of themm | 16:59 |
| *** davidlenwell has joined #tripleo | 16:59 | |
| *** ChanServ sets mode: +v davidlenwell | 16:59 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud: Update puppet configuration for ironic-inspector https://review.openstack.org/270904 | 17:00 |
| dtantsur | trown, thanks, done ^^ | 17:00 |
| *** bnemec has joined #tripleo | 17:02 | |
| EmilienM | jistr: same for https://review.openstack.org/233236 | 17:03 |
| EmilienM | jistr: and https://review.openstack.org/252051 | 17:04 |
| EmilienM | at least some feedback | 17:04 |
| jistr | the 2nd one is merged | 17:04 |
| EmilienM | oh yeah | 17:04 |
| EmilienM | sorry my gertty was not updated | 17:04 |
| *** dtantsur is now known as dtantsur|afk | 17:06 | |
| *** Marga_ has quit IRC | 17:07 | |
| *** coolsvap|away has quit IRC | 17:07 | |
| *** Marga_ has joined #tripleo | 17:08 | |
| *** fgimenez has quit IRC | 17:09 | |
| *** ukalifon has quit IRC | 17:09 | |
| *** fgimenez has joined #tripleo | 17:12 | |
| *** olap has joined #tripleo | 17:18 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-common: Use git.openstack.org where possible https://review.openstack.org/270927 | 17:21 |
| *** dcain has joined #tripleo | 17:23 | |
| *** jistr has quit IRC | 17:24 | |
| *** fgimenez has quit IRC | 17:24 | |
| *** dprince has joined #tripleo | 17:33 | |
| *** shivrao has joined #tripleo | 17:34 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Allow the vnc server to bind on IPv6 address on computes https://review.openstack.org/270831 | 17:40 |
| *** shadower has quit IRC | 17:41 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/268438 | 17:41 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Allow vncproxy to work with ssl enabled https://review.openstack.org/268356 | 17:42 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Configure keystone public_endpoint https://review.openstack.org/265922 | 17:42 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Don't write CLOUDNAME to the hosts file. https://review.openstack.org/267919 | 17:42 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Enable TLS in loadbalancer if cert path is detected https://review.openstack.org/266469 | 17:42 |
| *** stevebaker has quit IRC | 17:43 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Allow the vnc server to bind on IPv6 address on computes https://review.openstack.org/270831 | 17:44 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/268528 | 17:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements https://review.openstack.org/268549 | 17:48 |
| *** rwsu has joined #tripleo | 17:48 | |
| *** shivrao_ has joined #tripleo | 17:48 | |
| *** ukalifon1 has joined #tripleo | 17:49 | |
| *** davidlenwell has quit IRC | 17:50 | |
| *** shivrao has quit IRC | 17:51 | |
| ayoung | EmilienM, OK, now I am ready to test things again http://adam.younglogic.com/2016/01/resize-disks-centos-7/ | 17:51 |
| *** shivrao_ has quit IRC | 17:53 | |
| *** bvandenh has joined #tripleo | 17:54 | |
| *** davidlenwell has joined #tripleo | 17:55 | |
| *** ChanServ sets mode: +v davidlenwell | 17:55 | |
| *** egafford has joined #tripleo | 17:58 | |
| *** marios has quit IRC | 17:59 | |
| egafford | bnemec, dprince: Any notion of how likely it might be to get reviews and merges on https://review.openstack.org/#/c/233236/, https://review.openstack.org/#/c/233240/, and https://review.openstack.org/#/c/233241/ in the near future? | 18:00 |
| egafford | Trying to determine whether to wait on the Trove patches to hit master, or to go ahead and switch my dev env over to stable/liberty to start backporting Sahara before Trove hits (if it is likely to be a significant while longer.) | 18:00 |
| egafford | As always, any answer is sensible; just looking for information. | 18:00 |
| *** akrivoka has quit IRC | 18:00 | |
| egafford | https://review.openstack.org/#/c/233236/ (puppet elements) should probably be a slam dunk; heat templates and tripleoclient may be a bit dodgier, but they follow form from Sahara's patches very directly. | 18:01 |
| *** Marga_ has quit IRC | 18:03 | |
| *** derekh has quit IRC | 18:03 | |
| dprince | egafford: I will try and look at the trove stuff again shortly | 18:06 |
| *** bvandenh has quit IRC | 18:07 | |
| egafford | dprince: Cool; as noted, you've got to do what you've got to do. I just want to know the most efficient path for me to take in re: dev cycles. I'll hold off on stable/liberty for the moment, then. Thanks! | 18:08 |
| *** jaosorior has quit IRC | 18:09 | |
| *** devvesa has quit IRC | 18:10 | |
| *** jaosorior has joined #tripleo | 18:10 | |
| *** slagle has joined #tripleo | 18:13 | |
| bnemec | It looks like something is breaking the undercloud install right now. I'm looking into it. | 18:17 |
| *** eggmaster has quit IRC | 18:17 | |
| *** trown is now known as trown|lunch | 18:22 | |
| Slower_ | shardy: can you look at https://review.openstack.org/#/c/253217/ ? | 18:23 |
| Slower_ | shardy: should be good to go.. | 18:23 |
| *** Slower_ is now known as Slower | 18:23 | |
| Slower | we've been using it for weeks.. :) | 18:23 |
| *** stevebaker has joined #tripleo | 18:28 | |
| *** athomas has quit IRC | 18:31 | |
| *** tosky_ has joined #tripleo | 18:31 | |
| *** alop has joined #tripleo | 18:32 | |
| *** tosky has quit IRC | 18:32 | |
| *** tosky_ is now known as tosky | 18:32 | |
| *** mgould has quit IRC | 18:32 | |
| *** shivrao has joined #tripleo | 18:33 | |
| shardy | Slower: ack, done | 18:34 |
| *** jaosorior_ has joined #tripleo | 18:36 | |
| openstackgerrit | Merged openstack/tripleo-common: Heat Docker agent built with Dockerfile https://review.openstack.org/253217 | 18:38 |
| *** electrofelix has quit IRC | 18:39 | |
| *** penick has joined #tripleo | 18:39 | |
| *** athomas has joined #tripleo | 18:39 | |
| Slower | shardy: woot! | 18:39 |
| Slower | dprince: what do you think of this one? I like your hack better but.. :) | 18:42 |
| Slower | dprince: https://review.openstack.org/#/c/266066/1 | 18:42 |
| *** sthillma has joined #tripleo | 18:43 | |
| *** ukalifon1 has quit IRC | 18:43 | |
| *** ukalifon has joined #tripleo | 18:43 | |
| Slower | dprince: it does cut about 3mins off the deploy time | 18:44 |
| *** Marga_ has joined #tripleo | 18:44 | |
| *** gfidente is now known as gfidente|afk | 18:45 | |
| dprince | Slower: probably fine for now I think | 18:47 |
| dprince | Slower: I've got a new idea for the "hack" BTW. One that I think everyone will find acceptable | 18:47 |
| dprince | Slower: essentially we can create a custom puppet package "provider" that creates the default directories and users for us | 18:48 |
| dprince | Slower: that is like 90% of the issue, there are still a few exceptions though | 18:48 |
| EmilienM | ayoung: nice | 18:49 |
| *** ukalifon has quit IRC | 18:51 | |
| Slower | dprince: ah good idea! | 18:51 |
| dprince | Slower: haven't had the cycles yet to pursue it, but I'll get back to it | 18:52 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Remove shebangs from non-execute files https://review.openstack.org/270563 | 18:53 |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Explicitly decode to utf-8 https://review.openstack.org/270983 | 19:01 |
| *** tosky has quit IRC | 19:01 | |
| *** xinwu has joined #tripleo | 19:02 | |
| *** Marga_ has quit IRC | 19:11 | |
| *** yamahata has quit IRC | 19:12 | |
| *** Marga_ has joined #tripleo | 19:16 | |
| *** alop has quit IRC | 19:19 | |
| *** alop has joined #tripleo | 19:19 | |
| bnemec | ln: failed to create symbolic link ���/usr/local/bin/dib-python���: File exists | 19:22 |
| bnemec | Aha. Smoking gun. | 19:22 |
| bnemec | We should still merge the utf-8 fix at some point so we can debug this stuff. | 19:23 |
| bnemec | Oh, and now I see why I couldn't reproduce this. I didn't build my undercloud image with DIB, so the link wasn't created already. | 19:24 |
| slagle | easy review if someone could take a look, https://review.openstack.org/#/c/257856/ | 19:25 |
| *** trown|lunch is now known as trown | 19:26 | |
| openstackgerrit | Ben Nemec proposed openstack/diskimage-builder: Force dib-python symlink creation https://review.openstack.org/270994 | 19:27 |
| bnemec | slagle: dprince: ^ will be needed to unblock CI. | 19:28 |
| trown | ah that is why I dont hit it with RDO too... was wondring that | 19:28 |
| *** pradk has quit IRC | 19:29 | |
| dprince | bnemec: fire when ready, thanks | 19:29 |
| bnemec | Interesting. The CI runs on that patch never got to the point of installing the undercloud. | 19:30 |
| bnemec | That should _not_ have merged. | 19:31 |
| *** ukalifon has joined #tripleo | 19:31 | |
| *** eggmaster has joined #tripleo | 19:33 | |
| bnemec | lol, the zuul eta for the nonha job has dropped to about 40 minutes because the jobs have been failing so fast. | 19:35 |
| *** xinwu has quit IRC | 19:35 | |
| bnemec | dprince: I'm thinking I'll just merge that as soon as it gets past the point where the other jobs were failing. It can't really make the CI situation worse. :-) | 19:36 |
| *** gfidente|afk has quit IRC | 19:39 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add update yaml backward compatibe with PublicVirtualIP on ctlplane https://review.openstack.org/257856 | 19:41 |
| *** yamahata has joined #tripleo | 19:41 | |
| dprince | bnemec: ++ | 19:43 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Add update yaml backward compatibe with PublicVirtualIP on ctlplane https://review.openstack.org/271005 | 19:44 |
| bnemec | Away she goes. Now we just have to wait for the infra gate to catch up. | 19:53 |
| *** ayoung has quit IRC | 19:56 | |
| *** jcoufal has quit IRC | 19:59 | |
| *** ukalifon has quit IRC | 20:06 | |
| *** eggmaster has quit IRC | 20:07 | |
| *** xinwu has joined #tripleo | 20:10 | |
| *** ukalifon has joined #tripleo | 20:12 | |
| *** gchamoul has quit IRC | 20:13 | |
| *** gchamoul has joined #tripleo | 20:20 | |
| *** Marga_ has quit IRC | 20:22 | |
| *** Marga_ has joined #tripleo | 20:22 | |
| *** ukalifon has quit IRC | 20:34 | |
| *** ryansb has quit IRC | 20:40 | |
| *** tiswanso has quit IRC | 20:42 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Manage keystone initialization directly in t-h-t manifests https://review.openstack.org/244162 | 20:42 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: endpoint_map: bump nova api from 2.0 to 2.1 https://review.openstack.org/263366 | 20:43 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: puppet: specify server lists for mongodb_conn_validator provider https://review.openstack.org/269871 | 20:44 |
| *** ryansb has joined #tripleo | 20:44 | |
| *** ryansb has joined #tripleo | 20:44 | |
| egafford | Is tripleo.sh the recommended dev env spinup tool for stable/liberty? | 20:48 |
| egafford | Seeing some fairly early errors without modification; figured I'd check. | 20:49 |
| *** mkovacik has joined #tripleo | 20:49 | |
| bnemec | egafford: I would say yes, although I haven't actually tried it myself. | 20:50 |
| * bnemec intends to work on the stable branch at some point, when everything stops breaking | 20:51 | |
| egafford | bnemec: How do you build stable/liberty envs? | 20:51 |
| bnemec | egafford: I don't. | 20:51 |
| bnemec | I've only done backports, and I rely on CI to tell me if I broke something. | 20:51 |
| egafford | Okey doke. | 20:51 |
| *** yamahata has quit IRC | 20:52 | |
| *** lblanchard has quit IRC | 20:54 | |
| egafford | bnemec: So... does openstack/os-cloud-config have a stable/liberty branch? | 21:05 |
| egafford | If not, what? | 21:05 |
| bnemec | egafford: No, that's supposed to be strictly backwards compatible. | 21:05 |
| egafford | bnemec: Okey doke. | 21:06 |
| *** yamahata has joined #tripleo | 21:06 | |
| egafford | <super downstream concern in public>Will the master be pulled into RHOS 8 again prior to release?</sdcip> | 21:06 |
| EmilienM | having this CI error: http://logs.openstack.org/62/244162/15/check-tripleo/gate-tripleo-ci-f22-nonha/03a0ee4/console.html#_2016-01-21_21_04_18_016 | 21:10 |
| openstackgerrit | Ethan Gafford proposed openstack/puppet-tripleo: Sahara integration https://review.openstack.org/271038 | 21:10 |
| EmilienM | is it a known bug? | 21:10 |
| EmilienM | dprince, bnemec ^ | 21:10 |
| bnemec | egafford: I _think_ so, but you probably want to talk to somebody downstream about it to know for sure. | 21:10 |
| bnemec | EmilienM: Yes, a fix is in the gate, but was failing the check job last I looked so it may be a while. :-/ | 21:11 |
| EmilienM | cool | 21:11 |
| *** prometheanfire has joined #tripleo | 21:11 | |
| bnemec | I'm looking for a way to cherry-pick it into CI, but that all go blown away in the move to tripleo-common, so I'm not sure where to put it. | 21:11 |
| prometheanfire | gate sucking? https://review.openstack.org/#/c/270597 | 21:11 |
| bnemec | prometheanfire: Yes | 21:11 |
| dprince | bnemec: can you like the patch again? | 21:11 |
| bnemec | A bad change got merged. | 21:11 |
| prometheanfire | k, that's what I thought | 21:11 |
| bnemec | dprince: https://review.openstack.org/#/c/270994/ | 21:12 |
| bnemec | dprince: Or do you want the broken one? | 21:12 |
| dprince | bnemec: nah, that was fine. Not sure why I failed to see that in my reviewed patches. :) | 21:13 |
| dprince | EmilienM: ^^ that was the issue | 21:13 |
| bnemec | dprince: I'm seeing a lot of failures today in the dib gate too. It may be tricky to get this to merge. :-/ | 21:14 |
| openstackgerrit | Ethan Gafford proposed openstack/tripleo-puppet-elements: Sahara integration https://review.openstack.org/271040 | 21:14 |
| bnemec | Bleepity bleep bleep bleeping bleep. | 21:14 |
| bnemec | My undercloud just died building an image. | 21:15 |
| *** absubram has quit IRC | 21:15 | |
| bnemec | Apparently even 12 GB is no longer enough to build in memory safely. | 21:15 |
| *** dshulyak_ has quit IRC | 21:18 | |
| trown | building images directly on the undercloud is one of the more inefficient things in our workflow | 21:20 |
| openstackgerrit | Ethan Gafford proposed openstack/tripleo-heat-templates: Removing Sahara password default https://review.openstack.org/271042 | 21:20 |
| trown | it is pretty much always going to be faster to build from the host machine | 21:20 |
| bnemec | trown: Yeah, unfortunately it's also about the only way to ensure all the things are in sync right now. | 21:23 |
| bnemec | We used to build images on the host, but unless you start over from scratch and reinstall the host from the latest delorean every time you end up with the images being out of date. | 21:23 |
| trown | bnemec: ya that part seems odd to me... it seems like what is on the host and what is in the image do not need to be coupled | 21:25 |
| trown | for RDO for instance, I am building images with the liberty release rpm installed on the host... even for mitaka delorean images | 21:26 |
| *** dshulyak_ has joined #tripleo | 21:26 | |
| bnemec | trown: Yeah, that's where dprince's deploy from swift artifacts spec comes in. It should decouple the image from the thing being deployed a lot more. | 21:26 |
| *** mbound has quit IRC | 21:28 | |
| trown | bnemec: ya what I am talking about is a bit different... ie if you ensure the image has the up to date correct repos, it shouldnt matter what is on the host | 21:28 |
| trown | the host just needs some semi-recent version of dib and tripleoclient | 21:29 |
| trown | and those are not even really hard reqs, they are only required to try to follow our workflow | 21:30 |
| trown | really what we want from an overcloud image is an image with certain repos and packages installed... that is a pretty trivial problem compared to the pretty complex machinery in our workflow | 21:31 |
| trown | the ramdisk is slightly more complex, but I am not sure even that has to be | 21:31 |
| bnemec | I guess downstream already does kind of work that way. You export a bunch of env vars that tell the included elements how to set up the repos. The only client dep is that the client has to include the appropriate elements. | 21:33 |
| *** jayg is now known as jayg|g0n3 | 21:33 | |
| dprince | trown: In my dev environment I always build on a "host" machine. I'd like to get to the point where the undercloud was treated more like a black box | 21:36 |
| *** julim has quit IRC | 21:36 | |
| dprince | trown: you can actually build your images elsewhere | 21:36 |
| trown | dprince: ya for RDO the images (including an undercloud.qcow2) are getting built in CI | 21:36 |
| dprince | trown: tripleoclient currently does a local ramdisk copy to a web root | 21:36 |
| trown | so whenever a delorean repo is promoted, it pushes new images | 21:37 |
| *** penick_ has joined #tripleo | 21:37 | |
| *** penick has quit IRC | 21:37 | |
| *** penick_ is now known as penick | 21:37 | |
| trown | I have thought of proposing a spec for tripleo, but am not sure it fits in our CI | 21:37 |
| dprince | trown: a spec to build images outside of the workflow? | 21:38 |
| trown | dprince: a spec to build an undercloud.qcow2 with overcloud images built in | 21:38 |
| dprince | trown: I'd actually like to see us go the other way | 21:39 |
| trown | dprince: since we also install most packages twice, and package installs are the thing that takes the longest in both the image building and undercloud install | 21:39 |
| dprince | trown: I would love a pre-built undercloud.qcow (with packages installed). This is actually what we used to do with seed.qcow in devtest.sh | 21:40 |
| dprince | trown: that sort of utility model is quite nice, but I don't think the overcloud images belong inside of it | 21:40 |
| bnemec | It's possible we could pre-populate the dib yum cache based on what is already installed on the build system though. | 21:41 |
| dprince | trown: those could be pre-built elsewhere too, and if you were say only testing a puppet-nova patch you wouldn't have to rebuild them necessarily | 21:41 |
| trown | dprince: ya, it is pretty nice to just virt-customize in a simple fix to a pre-built overcloud image | 21:41 |
| bnemec | Although that wouldn't hugely help CI or me because we both have squid configured to avoid re-downloading packages anyway. | 21:41 |
| trown | takes on the order of a minute, vs 20 minutes to build images | 21:41 |
| dprince | trown: I'm keen on customizing a stock image if it saves us this sort of time, very keen | 21:42 |
| *** jprovazn has quit IRC | 21:42 | |
| dprince | trown: for undercloud we might need to customize | 21:43 |
| *** shardy has quit IRC | 21:43 | |
| dprince | trown: for overcloud though I think simply building the packages, and tarballing the puppet modules and we could just deploy them in CI with the swift artifacts deployment mechanism | 21:43 |
| bnemec | We might want to point CI at a fully updated base cloud image. I know in my image builds I lose a bunch of time updating 3 or 400 packages that are just part of the base image. | 21:44 |
| trown | bnemec: ya I made that optimization for RDO CI | 21:44 |
| bnemec | Maybe leave like the nonha job pointing at the base image just to make sure it works. | 21:44 |
| trown | it shaves a good 5 minutes | 21:44 |
| trown | since the base cloud image needs like 200 package updates | 21:45 |
| *** trozet_ has joined #tripleo | 21:45 | |
| *** trozet has quit IRC | 21:47 | |
| trown | dprince: what do you think we would need to customize on the undercloud? assuming the undercloud is just a cloud image with packages and repos installed | 21:47 |
| *** trozet_ is now known as trozet | 21:47 | |
| trown | ie `openstack undercloud install` has never run | 21:47 |
| dprince | trown: I think cracking the image, or splicing into it would be fine | 21:48 |
| dprince | trown: I've used DIB for this sort of thing before (like with the old seed) | 21:48 |
| dprince | trown: once upon a time I had patches posted to do that. They weren't well received but we could revive those. It really is quite simple w/ DIB | 21:49 |
| dprince | trown: or any virt-customize, libguestfs sort of approach would be fine too | 21:49 |
| trown | dprince: I am pretty partial to libguestfs, but I get that is not the default for tripleo | 21:49 |
| bnemec | Does guestfs work in vms? | 21:50 |
| dprince | bnemec: I think so | 21:50 |
| trown | bnemec: it does | 21:52 |
| trown | I use it from the undercloud all the time | 21:52 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common: Convenience methods in base model class https://review.openstack.org/268821 | 21:57 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common: Implement ViewManager list_software_deployments https://review.openstack.org/268822 | 21:57 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common: Implement ViewManager list_stack_failures https://review.openstack.org/270546 | 21:57 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common: Implement ViewManager show_software_deployment https://review.openstack.org/270545 | 21:57 |
| *** dsneddon is now known as dsneddon_afk | 21:58 | |
| trown | dprince: how could a pre-built undercloud.qcow2 be used in our CI? assuming it had a delorean server on it to build packages under test | 21:58 |
| openstackgerrit | Ben Nemec proposed openstack/python-tripleoclient: Remove ntp element from overcloud images https://review.openstack.org/271048 | 21:59 |
| bnemec | slagle: ^ fixes the non-sticky puppet ntp.conf problem. | 22:00 |
| dprince | trown: download the qcow2, crack it -> install new packages and puppet modules, etc. Then deploy it as normal to the testenvs | 22:00 |
| dprince | trown: I think we'd build the delorean packages on the Jenkins slave right? | 22:00 |
| dprince | trown: this could be done ahead of time.... | 22:01 |
| trown | dprince: hmm, and the qcow2 would be created in a periodic job? | 22:01 |
| dprince | trown: sure, we aren't doing this sort of thing yet, but infra does | 22:01 |
| trown | dprince: ok, I will try to get a spec started for this, I am doing this in RDO right now | 22:01 |
| dprince | trown: like nodepool creates its own pre-cached version of things and copies over an /opt/stack blob for each job with all the refspecs set accordingly | 22:02 |
| stevebaker | trown: hey, are there undercloud images being generated and put somewhere? | 22:02 |
| *** penick has quit IRC | 22:02 | |
| trown | stevebaker: ya https://ci.centos.org/artifacts/rdo/images/ is where they end up | 22:02 |
| dprince | trown: we could just use them from RDO as a test too | 22:02 |
| stevebaker | trown: oh, sweet | 22:03 |
| trown | stevebaker: they are just overwritten, so stable matches delorean current-passed-ci | 22:03 |
| trown | stevebaker: and testing is used during the CI test run itself | 22:03 |
| stevebaker | trown: is there any boot-time config needed? | 22:03 |
| dprince | stevebaker: hey, just noticed your patches above. What would you think if we implemented those via Mistral workflows instead? | 22:04 |
| trown | stevebaker: the undercloud.qcow2 is just packages and repos, so you still have to create an isntackenv.json and ssh keys | 22:04 |
| dprince | stevebaker: the Mistral API would let us expose the output to python-tripleoclient and a UI just as well | 22:04 |
| trown | stevebaker: I have some ansible I have been working on for both the image creation and consumption | 22:05 |
| stevebaker | trown: but the undercloud services are running? | 22:05 |
| trown | https://github.com/redhat-openstack/tripleo-quickstart | 22:05 |
| trown | stevebaker: nope, you have to run the undercloud install, but it is super fast because the packages are all installed | 22:05 |
| stevebaker | trown: ok, cool | 22:06 |
| bnemec | How is CI so broken?! | 22:06 |
| bnemec | There hasn't been a dib patch that passed tripleo-ci in two days because of package build failures. | 22:06 |
| bnemec | https://bugs.launchpad.net/tripleo/+bug/1536807 | 22:08 |
| openstack | Launchpad bug 1536807 in tripleo "TOCI failing to build package on dib patches" [Critical,Triaged] | 22:08 |
| stevebaker | dprince: these methods just do get and list calls, then process and return the results. I'm not sure that would fit into a workflow API. I'm not against mistral handling these in the future but I would *really* like these commands to get into liberty to help users debug and manage their overclouds | 22:08 |
| stevebaker | dprince: there will be some corresponding tripleoclient changes which call these tripleo-common methods | 22:09 |
| *** jaosorior_ has quit IRC | 22:09 | |
| dprince | stevebaker: right, looking at the initial spec I see you got some feedback on this already | 22:09 |
| dprince | stevebaker: I probably would've just landed them in python-tripleoclient first | 22:10 |
| stevebaker | dprince: yeah, I don't mind. These methods would be useful to a UI too | 22:10 |
| dprince | stevebaker: I realize these are simple things, but Mistral already has native support for interacting with openstack API's quite nicely | 22:10 |
| stevebaker | dprince: but isn't the idea of a workflow that it performs a change on the target? these are just observing commands | 22:11 |
| dprince | stevebaker: there are discussions underway to use Mistral in place of the TripleO API. | 22:11 |
| dprince | stevebaker: a workflow takes inputs, and provides outputs | 22:12 |
| dprince | stevebaker: This is my prototype work to make python-tripleoclient call a Mistral workflow to do the deployment for us: https://github.com/dprince/python-tripleoclient/blob/mistral/tripleoclient/v1/overcloud_deploy.py#L275 | 22:12 |
| prometheanfire | bnemec: let me know when ci is working? | 22:13 |
| dprince | stevebaker: I've got some example custom actions here: https://github.com/dprince/tripleo-common/tree/mistral/tripleo_common/actions, but many of the stock "OpenStack" actions are supported natively by Mistral | 22:13 |
| *** pradk has joined #tripleo | 22:13 | |
| stevebaker | dprince: I would bet you beer that mistral wouldn't be able to implement this sort of logic without some custom python in the mix https://review.openstack.org/#/c/268822/5/tripleo_common/core/view.py | 22:14 |
| dprince | stevebaker: probably more than you wanted to chat about right now... but the main reason I suggest these to you is that the UI may want to run your code too, and in that case I would propose perhaps just having both tripleoclient and the UI use the same codepath (API) | 22:14 |
| dprince | stevebaker: custom python is totally fine for Mistral. Stevedore's entrypoints load the custom actions quite nicely | 22:15 |
| dprince | stevebaker: but I'll take your bet ;) | 22:15 |
| dprince | stevebaker: because I think I win either way | 22:15 |
| bnemec | prometheanfire: Yeah, there's a whole chain of fail going on right now. There are multiple things broken in dib. :-( | 22:16 |
| prometheanfire | ya, have a patch in neutron in the same position | 22:16 |
| prometheanfire | gate is probably over 36 hours now | 22:16 |
| stevebaker | dprince: ok, so those stevedore things would still be contributed by tripleo-common, so we're still future-proofing ourselves | 22:17 |
| dprince | stevebaker: yep, no patches to Mistral required https://github.com/dprince/tripleo-common/blob/mistral/setup.cfg#L48 | 22:17 |
| dprince | stevebaker: we can get a "stable" API by simply versioning our workflows | 22:18 |
| dprince | stevebaker: tripleo.v1.<workflow_name> | 22:18 |
| *** penick has joined #tripleo | 22:18 | |
| * bnemec needs to finish writing his email about this whole Mistral thing | 22:19 | |
| bnemec | Let's just say I have concerns. | 22:19 |
| egafford | Wait, are we integrating Mistral as a strict dependency of TripleO? | 22:22 |
| stevebaker | Dan has gone to dinner | 22:22 |
| *** dshulyak_ has quit IRC | 22:22 | |
| egafford | (Fair.) | 22:23 |
| *** rpothier has quit IRC | 22:28 | |
| *** olap has quit IRC | 22:30 | |
| *** weshay_xchat has quit IRC | 22:40 | |
| *** trown is now known as trown|outttypeww | 22:43 | |
| *** ayoung has joined #tripleo | 22:58 | |
| *** egafford has quit IRC | 22:58 | |
| NobodyCam | good morning / afternoon / evening / night TripleO folks | 23:00 |
| NobodyCam | any one encountered this error today? | 23:00 |
| NobodyCam | Could not fetch URL https://pypi.python.org/simple/pip/: There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581) - skipping | 23:00 |
| bnemec | NobodyCam: Yes. It's killing all of the dib functests right now. https://review.openstack.org/#/c/270010/ appears to fix it. | 23:01 |
| bnemec | Note that there are two other problems in dib that also need to be resolved before anything should be merging there too. | 23:02 |
| bnemec | https://bugs.launchpad.net/tripleo/+bug/1536807 | 23:02 |
| openstack | Launchpad bug 1536807 in tripleo "TOCI failing to build package on dib patches" [Critical,Triaged] | 23:02 |
| bnemec | And another one I haven't actually opened a bug for but is killing all of the TOCI jobs in other projects too. | 23:02 |
| NobodyCam | oh pinning to 7.1.2 not 8.0.1 | 23:03 |
| NobodyCam | bnemec: :( ugh | 23:03 |
| NobodyCam | bnemec: Thank you for the pointer(help) :) | 23:05 |
| openstackgerrit | Merged openstack/diskimage-builder: Use pip 7 for ironic https://review.openstack.org/270010 | 23:05 |
| NobodyCam | lol wow what timing :) | 23:06 |
| bnemec | \o/ | 23:06 |
| bnemec | Now to find out if it actually fixes the problem. :-) | 23:06 |
| bnemec | Although it's the only patch that has passed the functest CI in two days, so I feel pretty safe saying it will. | 23:06 |
| NobodyCam | :) | 23:07 |
| *** xinwu is now known as xinwu_ | 23:07 | |
| openstackgerrit | Ben Nemec proposed openstack/python-tripleoclient: Remove ntp element from overcloud images https://review.openstack.org/271078 | 23:07 |
| *** dcain has quit IRC | 23:13 | |
| *** trozet has quit IRC | 23:31 | |
| *** mcornea has quit IRC | 23:39 | |
| *** dprince has quit IRC | 23:40 | |
| *** rcernin has quit IRC | 23:47 | |
| *** isq_ has quit IRC | 23:48 | |
| *** isq has joined #tripleo | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!