| *** tosky has quit IRC | 00:00 | |
| *** d0ugal__ has joined #oooq | 00:28 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 00:42 |
|---|---|---|
| *** d0ugal__ has quit IRC | 00:58 | |
| *** d0ugal__ has joined #oooq | 01:06 | |
| *** d0ugal__ has quit IRC | 01:39 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 02:42 |
| *** hamzy has quit IRC | 02:54 | |
| *** hamzy has joined #oooq | 02:54 | |
| *** d0ugal__ has joined #oooq | 03:06 | |
| *** d0ugal__ has quit IRC | 03:21 | |
| *** d0ugal__ has joined #oooq | 03:23 | |
| *** d0ugal__ has quit IRC | 03:29 | |
| *** d0ugal__ has joined #oooq | 03:36 | |
| *** d0ugal__ has quit IRC | 03:47 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 04:42 |
| EmilienM | weshay and CI team, I went ahead and approved https://review.openstack.org/565349 to clear up the gate | 04:56 |
| *** links has joined #oooq | 05:03 | |
| *** saneax-_-|AFK is now known as saneax | 06:38 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 06:42 |
| *** gkadam has joined #oooq | 07:45 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 08:42 |
| *** jbadiapa has quit IRC | 09:04 | |
| *** panda|ruck|off is now known as panda|ruck | 09:12 | |
| *** panda|ruck is now known as panda|ruckrover | 09:12 | |
| *** jbadiapa has joined #oooq | 09:17 | |
| *** ykarel|away has joined #oooq | 09:32 | |
| *** d0ugal has joined #oooq | 10:00 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 10:42 |
| *** ykarel|away has quit IRC | 10:51 | |
| *** apetrich has quit IRC | 11:15 | |
| *** atoth has joined #oooq | 11:18 | |
| *** jbadiapa has quit IRC | 11:26 | |
| panda|ruckrover | sshnaidm: http://logs.openstack.org/93/563293/6/gate/openstack-tox-linters/bda385d/job-output.txt.gz#_2018-05-01_09_44_21_146447 | 11:40 |
| panda|ruckrover | sshnaidm: the new rule is blocking the gate | 11:40 |
| sshnaidm | panda|ruckrover, need to merge this: https://review.openstack.org/#/c/564574/ | 11:41 |
| panda|ruckrover | I think we need not to test linters on tripleo-upgrades fi we are trying to merge a quickstart patch | 11:45 |
| panda|ruckrover | half of the world is PTO today, not sure when we'll be able to merge that patch in tripleo-upgrade | 11:46 |
| panda|ruckrover | also, this https://review.openstack.org/561911 is already adding most if not all the pipefails | 11:47 |
| sshnaidm | panda|ruckrover, we can merge it if you +2 it | 12:08 |
| sshnaidm | panda|ruckrover, it'll be better then starting reverts now | 12:08 |
| sshnaidm | panda|ruckrover, testing upgrades didn't start today, need to check why it happens.. | 12:09 |
| *** lucas-hungry is now known as lucasagomes | 12:11 | |
| *** tosky has joined #oooq | 12:13 | |
| sshnaidm | well, seems like it's because we include tripleo-upgrades role in our playbook http://logs.openstack.org/93/563293/6/gate/openstack-tox-linters/bda385d/job-output.txt.gz#_2018-05-01_09_44_19_782546 | 12:16 |
| sshnaidm | so ansible-lint will test it | 12:16 |
| panda|ruckrover | 2018-05-01 09:44:19.782546 | ubuntu-xenial | Examining ./multinode-undercloud-upgrade.yml of type playbook | 12:17 |
| panda|ruckrover | ansible-list follow the playbook, and descends into roles | 12:17 |
| sshnaidm | panda|ruckrover, yes, so it will check it always | 12:19 |
| weshay | %gatestatus | 12:19 |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 12:19 |
| sshnaidm | weshay, panda|ruckrover can you vote on https://review.openstack.org/#/c/564574/ ? | 12:20 |
| weshay | panda|ruckrover, do happen to know if we have a bug on stable/ocata: openstack-tox-pep8, yet? | 12:20 |
| panda|ruckrover | weshay: we have | 12:21 |
| panda|ruckrover | weshay: https://bugs.launchpad.net/tripleo/+bug/1768076 | 12:21 |
| openstack | Launchpad bug 1768076 in tripleo "GATE-CHECK change fails pep8-linters" [Medium,Triaged] - Assigned to Matt Young (halcyondude) | 12:21 |
| weshay | k.. thanks will look for it | 12:21 |
| weshay | thanks | 12:21 |
| weshay | assigned to matt young? | 12:22 |
| panda|ruckrover | reassigned to me now | 12:22 |
| weshay | OH | 12:22 |
| weshay | it's only failing on our change lolz | 12:22 |
| panda|ruckrover | yes | 12:22 |
| weshay | panda|ruckrover, literally doing exactly the opposite of what we wanted.. lolz | 12:23 |
| weshay | warning us when things are broken and ignoring false positives.. | 12:23 |
| weshay | that's funny | 12:23 |
| weshay | ok.. thanks panda|ruckrover | 12:23 |
| *** rlandy has joined #oooq | 12:26 | |
| weshay | sshnaidm, speaking of linters.. have you seen | 12:28 |
| weshay | | linters installed: ----------------------------------------,Error when trying to get requirement for VCS system Command "git config --get-regexp remote\..*\.url" failed with error code 1 in /home/zuul/src/git.openstack.org/openstack/tripleo-quickstart-extras, falling back to uneditable format,Could not determine repository location of /home/zuul/src/git.openstack.org/openstack/tripleo-quickstart-extras,alabaster==0.7.10,ansible==2.5.2,ansib | 12:28 |
| weshay | le- | 12:28 |
| weshay | http://logs.openstack.org/56/564156/2/gate/openstack-tox-linters/0671600/job-output.txt.gz | 12:28 |
| weshay | we have two gate failures on the linter in tqe | 12:28 |
| weshay | http://logs.openstack.org/93/563293/6/gate/openstack-tox-linters/bda385d/job-output.txt.gz | 12:29 |
| sshnaidm | weshay, panda|ruckrover again, please vote on https://review.openstack.org/#/c/564574/ to unblock gates and fix linters | 12:30 |
| weshay | sshnaidm, if that is fixing the gate failures sshnaidm you need a bug | 12:30 |
| weshay | http://logs.openstack.org/93/563293/6/gate/openstack-tox-linters/bda385d/job-output.txt.gz | 12:31 |
| sshnaidm | weshay, I'm not sure, it was on review much before | 12:31 |
| weshay | seems to be on tqe not tu | 12:31 |
| weshay | k | 12:31 |
| weshay | sshnaidm, that patch looks great.. just waiting on ci to return | 12:31 |
| sshnaidm | weshay, because we include tripleo-upgrade role in our playbook, ansible-lint will test it also in quickstart-extras patches | 12:31 |
| sshnaidm | weshay, so we need to keep linters and code style same in oooq-extras and tripleo-upgrade | 12:32 |
| sshnaidm | weshay, it's all we have in CI there | 12:32 |
| sshnaidm | weshay, nothing to wait for :) | 12:33 |
| panda|ruckrover | is there any way not to test tripleo-upgrades in tqe gates ? | 12:33 |
| panda|ruckrover | can we ignore the multinode playbook | 12:33 |
| sshnaidm | panda|ruckrover, yeah, to exclude upgrade playbook | 12:33 |
| sshnaidm | all of them | 12:33 |
| sshnaidm | but why not? | 12:33 |
| sshnaidm | if they will have the same linter rules, it won't happen | 12:34 |
| weshay | oh .. lol | 12:34 |
| weshay | fucking upgrade repo | 12:34 |
| sshnaidm | and pipefail is rule that worth to have | 12:34 |
| sshnaidm | weshay, and I should have port linters to all branches btw.. | 12:34 |
| weshay | barf | 12:35 |
| sshnaidm | queens: https://review.openstack.org/#/c/565065/ | 12:36 |
| panda|ruckrover | hm. can't find the card on the board with the discussion on the linters to add. | 12:37 |
| rlandy | sshnaidm: pls see response to your comment on https://review.openstack.org/#/c/565280/ - still want to keep that -1? | 12:39 |
| rlandy | sshnaidm: also wrt https://review.openstack.org/#/c/564865/ - are you defining the volume pool to be something custom? | 12:41 |
| panda|ruckrover | is the linter patch associated to a bug of some sort ? | 12:41 |
| sshnaidm | rlandy, ok.. but dependent patch seems like not ready yet? because if when we merge it, it will stop to work | 12:41 |
| rlandy | sshnaidm: we would try to merge all three patches together | 12:42 |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 12:42 |
| sshnaidm | rlandy, well, with your patch it works only if using user stack | 12:43 |
| rlandy | I am to w-1 it | 12:43 |
| rlandy | sshnaidm: which patch are we talking about? there are two | 12:43 |
| rlandy | https://review.openstack.org/#/c/564865/? | 12:43 |
| *** trown|outtypewww is now known as trown | 12:43 | |
| sshnaidm | https://review.openstack.org/#/c/564865/ | 12:43 |
| rlandy | ok | 12:43 |
| sshnaidm | rlandy, if it's acceptable for the beginning, I'm fine.. | 12:44 |
| sshnaidm | rlandy, but need to mention it anywhere | 12:44 |
| rlandy | so now I know what the problem is ... although it should have done this before | 12:44 |
| rlandy | sshnaidm: no - I just need to know what to fix | 12:44 |
| rlandy | iow - did the libvirt-pool stuff work w/o stack user before the patch? | 12:45 |
| sshnaidm | rlandy, yep | 12:45 |
| sshnaidm | rlandy, with your patch I have: http://paste.openstack.org/show/720178/ | 12:45 |
| weshay | panda|ruckrover, I'm going to try and fix the slaves in rdo2 re: the dlrnapi | 12:46 |
| sshnaidm | rlandy, you include vars instead of using playbook vars and this lower their precedence afaik, so seems like something overrides one of vars anywhere | 12:46 |
| weshay | panda|ruckrover, you can rove.. I'll be ur ruck | 12:46 |
| *** weshay is now known as weshay|ruck | 12:46 | |
| sshnaidm | rlandy, but what and where - I have no idea.. it's difficult to trace errors in this mess of roles in libvirt | 12:47 |
| rlandy | sshnaidm: ok - will fix it | 12:47 |
| sshnaidm | rlandy, it's odd that virt module tries to access pool with uid=1000 - it's uid of "stack" user, while I' | 12:47 |
| sshnaidm | m running it as centos user | 12:47 |
| rlandy | we ca set user | 12:48 |
| rlandy | sshnaidm: you made a similar comment for the reproducer and we changed it to use ... | 12:48 |
| sshnaidm | rlandy, I tried to hardcode ssh_user, undercloud_user, non_root_user, nothing helps.. maybe I missed some_other_user :) | 12:48 |
| rlandy | -e non_root_user=$USER \ | 12:49 |
| rlandy | -e non_root_group=$USER \ | 12:49 |
| sshnaidm | rlandy, yep, it works in current state | 12:49 |
| sshnaidm | rlandy, but doesn't work with your patch | 12:49 |
| panda|ruckrover | so, to recap, we merged a linter that wasn't discussed anywhere, it broke the gates, and to fix it we self-pushed a change to upgrades without waiting for guys from upgrades, and that conflicted with a previous patch that did pretty much the same I've been working for the last weeks that already had +2 and I now have to recheck | 12:49 |
| *** apetrich has joined #oooq | 12:50 | |
| rlandy | sshnaidm: livbirt_volume_path is different | 12:50 |
| rlandy | libvirt_volume_path: /opt/vm_images - was defined as before | 12:51 |
| rlandy | and I removed that | 12:51 |
| rlandy | to reference the default | 12:51 |
| sshnaidm | panda|ruckrover, linter patch was submitted and available for discussion for everywhere, I've added everybody as reviewers there. You could block it if you think it need a discussion | 12:52 |
| sshnaidm | rlandy, I'll try it now with /opt/vm_images | 12:53 |
| rlandy | sshnaidm: I don't like that as a default = but that is what is was previously | 12:53 |
| sshnaidm | rlandy, well, path seems fine, but in default installations we could have a few place in / for heavy images | 12:54 |
| *** panda|ruckrover is now known as panda|rover | 12:55 | |
| rlandy | sshnaidm: does it work with /opt/vm_images? | 12:55 |
| sshnaidm | rlandy, hmm.. testing now.. I set -e livbirt_volume_path=/opt/imgs but it used $WORKSPACE/pool anyway | 12:57 |
| rlandy | maybe include_vars overrides | 12:58 |
| rlandy | but its' not defined there :( | 12:58 |
| sshnaidm | rlandy, no, I use -e, it should override everything | 12:58 |
| sshnaidm | maybe something wrong with my setup, checking.. | 12:58 |
| panda|rover | I don't think the patch is the best place to discuss design. Indeed I don't see any discussion in the patch either. | 12:59 |
| sshnaidm | panda|rover, I'm not sure every patch needs a design session, there is no any new feature, it's obvious linter rule | 13:00 |
| myoung | o/ | 13:05 |
| rlandy | myoung: some issues with the patches | 13:06 |
| rlandy | just waiting for sshnaidm to test if it's linbirt_volume_path or not | 13:06 |
| rlandy | then I'll fix y first patch | 13:06 |
| myoung | rlandy, trown: last night I hit this testing fs10 https://trello.com/c/hI3xxjuQ/685-validate-fs-10-w-instruction-breadcrumb-trail#comment-5ae7d48a9b350fb5199dabd6 | 13:07 |
| myoung | rlandy, trown: guessing I'm missing a step (or we need to doc and/or check for) around networking. I started with a vanilla f27 | 13:07 |
| panda|rover | this change broke the gate. We're behind with reviews for the existing cards and bugs. Adding undiscussed patches for review is adding unanticipated load for the cores | 13:08 |
| rlandy | myoung: can you check resolv.comf on that node | 13:09 |
| rlandy | resolv.conf | 13:09 |
| rlandy | what is there? | 13:09 |
| sshnaidm | rlandy, ok, tested in /opt/imgs, and it worked - but only because images are created with "stack" user | 13:10 |
| sshnaidm | rlandy, and /opt/imgs can be accessed by any user | 13:11 |
| rlandy | sshnaidm: ok - so that puts us back to the situation we had before the patch? | 13:11 |
| rlandy | or it's still worse? | 13:11 |
| sshnaidm | rlandy, let me try to delete the user stack.. | 13:11 |
| rlandy | I think that should match what we had before | 13:12 |
| rlandy | sshnaidm: trown: myoung: so we can 1) put back the /opt/vm_images in the defaults, 2) set the user explicitly like we do in the reproducer or both | 13:19 |
| trown | rlandy: well.. I will override to /opt/vm_images no matter what we chose, so that works for me | 13:20 |
| trown | but it is a bit annoying for a default install with small root partition | 13:20 |
| rlandy | trown: I am just trying to use default that avoids problems like sshnaidm is hitting | 13:20 |
| rlandy | which /opt did | 13:21 |
| rlandy | although the root default partition is too small and that's another issue | 13:21 |
| myoung | as part of the breadcrumb trail I'm keeping a list of things we might consider for future sprints and/or incremental improvements | 13:22 |
| rlandy | I deleted the stack user on my test box | 13:22 |
| * myoung adds these | 13:22 | |
| sshnaidm | rlandy, trown well, seems like I tested the only case with won't work - when you have user "stack" on host but runs reproducer as other user. All the rest should work.. | 13:23 |
| sshnaidm | maybe need to get back to QA.. | 13:23 |
| rlandy | trown: what do you think of defining the user in the defaults? | 13:23 |
| rlandy | sshnaidm: your case is not unusual | 13:23 |
| rlandy | trown: putting non_root_user=$USER non_root_group=$USER type thing in the defaults? | 13:24 |
| rlandy | we set it for the reproducer | 13:24 |
| sshnaidm | rlandy, ok, let's leave the default then.. | 13:24 |
| rlandy | sshnaidm: the /opt/ default? | 13:24 |
| sshnaidm | rlandy, no, the workspace pool | 13:25 |
| sshnaidm | this is the price we pay for reusing libvirt role.. | 13:26 |
| trown | rlandy: im fine with putting those in the reproducer script | 13:26 |
| rlandy | sshnaidm:so you use the reproducer script to test or run the playbook on its own? | 13:28 |
| sshnaidm | rlandy, reproducer script of course | 13:28 |
| myoung | sshnaidm: I (still) think it's worth paying the price, what we're doing it harder but when done will mesh with our existing codebase | 13:28 |
| rlandy | well - that has to work off the bat | 13:28 |
| myoung | rlandy: resolve.conf just has 8.8.8.8 | 13:28 |
| rlandy | myoung: that should have outside access | 13:28 |
| sshnaidm | myoung, it will mess, not mesh | 13:28 |
| sshnaidm | rlandy, yeah, another problem I have it's 8.8.8.8 | 13:29 |
| myoung | rlandy: aye, wonder if I'm hitting google | 13:29 |
| sshnaidm | rlandy, why do we override it? it hardly works in rh network | 13:29 |
| myoung | and getting rate limited | 13:29 |
| rlandy | sshnaidm: repos didn't resolve with straight internal | 13:29 |
| rlandy | originally I had an append clause there | 13:30 |
| rlandy | so both internal and external were available | 13:30 |
| rlandy | that was voted out for s supersede clause which replaces | 13:30 |
| sshnaidm | rlandy, hmm.. I have only 8.8.8.8 in resolv.conf, and it barely works | 13:30 |
| rlandy | sshnaidm: try this ... | 13:31 |
| sshnaidm | rlandy, which repos didn't resolve with internal dns? | 13:31 |
| rlandy | https://github.com/openstack/tripleo-quickstart/blob/master/roles/libvirt/setup/overcloud/tasks/fake_nodepool.yml#L168 and L209 | 13:32 |
| rlandy | make that append rather than supersede | 13:32 |
| sshnaidm | rlandy, when i set internal dns - everything works like a charm | 13:33 |
| rlandy | myoung: ^^ can you try that | 13:33 |
| sshnaidm | rlandy, I'm not sure what was problem, but seems odd that internal dns didn't resolve.. | 13:34 |
| rlandy | sshnaidm: you can set a custom_nameserver= to override the 8.8.8.8 default | 13:34 |
| sshnaidm | rlandy, but if 8.8.8.8 doesn't work, why do we need it? | 13:34 |
| rlandy | sshnaidm: iirc it was the centos repos that were not resolving for those of us working over vpn | 13:35 |
| rlandy | trown" ^^ correct? | 13:35 |
| rlandy | 8.8.8.8 was working better for our testing | 13:35 |
| trown | ya centos repos dont resolve with redhat internal dns | 13:35 |
| rlandy | I am open to switching the default again | 13:35 |
| rlandy | let's just agree on it | 13:35 |
| trown | and make sure it works :P | 13:36 |
| rlandy | works?? minor detail | 13:36 |
| sshnaidm | trown, rlandy maybe some temporary problem in your net? | 13:36 |
| sshnaidm | can you check it with dig? | 13:36 |
| rlandy | I really think we can get by this with append | 13:36 |
| rlandy | you will get both | 13:36 |
| sshnaidm | rlandy, yeah, also good | 13:36 |
| rlandy | can you test that, pls? | 13:36 |
| rlandy | that is what I wanted to do originally | 13:37 |
| *** apetrich has quit IRC | 13:38 | |
| myoung | rlandy: aye trying internal dns now | 13:38 |
| myoung | (sry delay had phone call) | 13:38 |
| trown | sshnaidm: it is not temporary... I hit it consistently | 13:39 |
| rlandy | thank you | 13:39 |
| trown | sshnaidm: and ya nslookup failed on the repo url | 13:39 |
| sshnaidm | trown, which url do you check? | 13:40 |
| sshnaidm | rlandy, just tried - I get my virthost IP as first DNS and then 8.8.8.8 | 13:40 |
| * myoung is a huge fan of an approach that works both inside/outside...by "append" do you mean specifying multiple dns (e.g. have 8.8.8.8 and if specified via param append it to the list?) | 13:40 | |
| sshnaidm | rlandy, my virthost IP works fine as DNS | 13:40 |
| trown | sshnaidm: mirrorlist.centos.org | 13:40 |
| rlandy | sshnaidm: perfect - thank you | 13:40 |
| rlandy | I will put in a patch to put back append | 13:41 |
| myoung | hrm...would it be crazy to look at virthost /etc/resolv.conf, then use *that* for dns servers? | 13:41 |
| *** apetrich has joined #oooq | 13:41 | |
| myoung | (in the subnodes) | 13:41 |
| sshnaidm | trown, strange, resolves in a moment for me in internal net | 13:41 |
| rlandy | sshnaidm: to digress to the previous issue ... https://review.openstack.org/#/c/565282/1/roles/create-reproducer-script/templates/reproducer-quickstart.sh.j2 adds the option to define libvirt_volume_path so hopefully people can figure out a good option for their case and define it | 13:41 |
| myoung | ^^ i like this...our audience (developers) i would guess have all manner of storage/drives on their boxes...i do... | 13:42 |
| rlandy | myoung: to answer your question above, yes - append gives you both options on dns | 13:43 |
| sshnaidm | rlandy, seems great | 13:43 |
| * rlandy is sorry I have no backbone :( | 13:43 | |
| myoung | (figured...i was out when this all went down) | 13:43 |
| rlandy | ok - putting up patch | 13:44 |
| rlandy | sshnaidm: myoung: thank you for your testing - I appreciate your feedback | 13:45 |
| myoung | rlandy: ok holding off, will test with your patch | 13:47 |
| myoung | CI squad: (gentle reminder) I've added this to the #tripleo meeting agenda "CI Community Meeting starts immediately upon this meeting closing @ https://bluejeans.com/7050859455. All are welcome to ask/discuss $ciThings. Add to agenda (or just join): https://etherpad.openstack.org/p/tripleo-ci-squad-meeting" | 13:49 |
| *** tosky has quit IRC | 13:51 | |
| *** links has quit IRC | 13:54 | |
| rlandy | myoung: https://review.openstack.org/565508 Change libvirt-nodepool to use both internal and external DNS | 13:55 |
| rlandy | myoung: it will clash with your rename patch :( | 13:55 |
| rlandy | maybe we can combine/rebase | 13:55 |
| myoung | rlandy: I'll just rebase it...no worries | 13:55 |
| rlandy | let me know what is easier for you | 13:55 |
| * myoung looks | 13:55 | |
| rlandy | make it dep on your patch is fine | 13:55 |
| myoung | in hindsight would have made sense to make a isolated patch earlier last week that just does the s/fake/libvirt/g change and slam it in :) | 13:56 |
| *** apetrich has quit IRC | 13:57 | |
| myoung | rlandy: ahh i see...we could roll it into my patch i guess...prob easier | 13:57 |
| myoung | 2 line change | 13:57 |
| rlandy | myoung: that is fine | 13:57 |
| rlandy | sshnaidm: ^^ fyi | 13:57 |
| myoung | want me to make it in https://review.openstack.org/#/c/565280/ | 13:57 |
| myoung | k doing that now | 13:57 |
| rlandy | myoung: we need to figure out the merging timetable for all these patches that depend one on the other | 13:58 |
| myoung | rlandy: ack, and done (https://review.openstack.org/#/c/565280/3..4/roles/libvirt/setup/overcloud/tasks/libvirt_nodepool.yml) | 14:02 |
| * rlandy looks | 14:02 | |
| rlandy | cool | 14:03 |
| rlandy | abandoning my patch | 14:03 |
| myoung | one less patch in the chain | 14:03 |
| myoung | rlandy, trown, sshnaidm: imho we should land https://review.openstack.org/#/c/564865 asap to get things rolling | 14:04 |
| myoung | we can iterate in subsequent patches, we've already got a laundry list building for a future sprint as well. | 14:05 |
| myoung | sshnaidm: capturing your comment re: stack user on the host case on that list as well | 14:05 |
| sshnaidm | myoung, where is the list? | 14:06 |
| myoung | sshnaidm: it organically started last night when testing fs10, I'll move it to an etherpad or card. https://trello.com/c/hI3xxjuQ/685-validate-fs-10-w-instruction-breadcrumb-trail#comment-5ae7d48a9b350fb5199dabd6 now | 14:07 |
| weshay|ruck | chandankumar, panda|rover I was out last week.. there is an action item in #tripleo mtg re: https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset021-master/ | 14:07 |
| weshay|ruck | <mwhahaha> ci team, panda|ruck, quiquell|rover to investigate why fs021 is running on master if it's not properly containerized | 14:07 |
| weshay|ruck | %gatestatus | 14:08 |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 14:08 |
| weshay|ruck | hrm.. hubbot should be reporting on fs21 I think | 14:08 |
| sshnaidm | rlandy, so 565508 will move to myoung patch? | 14:08 |
| sshnaidm | rlandy, oh I see | 14:09 |
| * myoung moves the ideas list to a card so all can add/edit as we test | 14:09 | |
| rlandy | yep - otherwise name clash | 14:10 |
| sshnaidm | myoung, and where is the list itself there? | 14:11 |
| myoung | https://trello.com/c/2y2BVXgO/726-ideas-brainstorm-resulting-from-testing-in-sprint-12 | 14:11 |
| myoung | ^^ just made that | 14:11 |
| *** trown is now known as trown|brb | 14:11 | |
| myoung | sshnaidm: it was at the bottom of my comments in fs10 testing card...i was just keeping notes as I tested (pretending I knew nothing about reproducer) and doc'd stuff I hit on vanilla f27 | 14:12 |
| *** jbadiapa has joined #oooq | 14:13 | |
| *** trown|brb is now known as trown | 14:14 | |
| sshnaidm | rlandy, not sure you tried, but did beaker reprovision work for you lately? | 14:23 |
| rlandy | sshnaidm: on your rdu box? I have a ticket out for that | 14:27 |
| rlandy | myoung and rfolco reported their boxes could not pxe boot either | 14:28 |
| sshnaidm | rlandy, so there is a problem? | 14:28 |
| rlandy | jpainter is on the ticket | 14:28 |
| sshnaidm | I see | 14:28 |
| sshnaidm | rlandy, thanks | 14:28 |
| rlandy | no dhcp offers | 14:28 |
| myoung | rlandy: I had issues with beaker+pxe when attempting to use centos from beaker. f27 and RHEL seem to work without a hitch | 14:28 |
| myoung | rfolco: ^^ | 14:29 |
| myoung | rlandy: could you please add me to the watchlist for your IT ticket? | 14:29 |
| sshnaidm | rlandy, hmm.. actually I have dhcp offer. But one host just ignores provisioning, both from site and command line, and second is stuck on boot.. | 14:29 |
| rlandy | sshnaidm: yep - forwarding you the ticket info | 14:30 |
| rlandy | sshnaidm: ok - you are on the ticket and I forwarded the link to you | 14:32 |
| rlandy | fell free to add to that ticket | 14:32 |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 14:42 |
| *** gkadam has quit IRC | 14:43 | |
| weshay|ruck | panda|rover, fyi.. upgrading a slave from f23 -> f25 | 14:51 |
| panda|rover | weshay|ruck: on ci.centos ? | 14:52 |
| weshay|ruck | panda|rover, rdo2 | 14:52 |
| panda|rover | weshay|ruck: for the setuptools ? | 14:53 |
| weshay|ruck | panda|rover, the jobs are passing but most fail on the dlrn_api call | 14:53 |
| weshay|ruck | ya | 14:53 |
| *** jbadiapa has quit IRC | 15:04 | |
| *** apetrich has joined #oooq | 15:11 | |
| weshay|ruck | panda|rover, looks like that fixed it | 15:12 |
| weshay|ruck | panda|rover, myoung we need to schedule some downtime in rdo2 to upgrade the slaves. After I get a master promotion in rdo2 I think it can be anytime | 15:18 |
| *** gkadam has joined #oooq | 15:26 | |
| *** rfolco has quit IRC | 15:29 | |
| myoung | rlandy, trown, hitting a new error, when attempting to start the VM's getting access issues to the qcows | 15:32 |
| myoung | libvirtError: Cannot access storage file '/root/s12/validate-fs010/pool/subnode-0.qcow2' (as uid:107, gid:107): Permission denied | 15:32 |
| trown | why are they in /root/? | 15:32 |
| myoung | I was running at root :) | 15:33 |
| trown | dont do that :P | 15:33 |
| myoung | aye lol | 15:33 |
| myoung | (emu can't access) | 15:33 |
| trown | no way $HOME will work as /root | 15:33 |
| myoung | qemu | 15:33 |
| rlandy | same issue as sshnaidm hit | 15:35 |
| rlandy | maybe we should go back to old default | 15:36 |
| trown | meh... or just dont run the script as root | 15:36 |
| rlandy | unless I missed something it's become: true | 15:39 |
| trown | become: true is different than running as root user though | 15:40 |
| trown | /home/$USER is fine ... /root/ will not be allowed world readable | 15:40 |
| *** hamzy has quit IRC | 15:46 | |
| *** ccamacho has quit IRC | 15:49 | |
| myoung | I added this to the brainstorm/idea list | 15:53 |
| rlandy | myoung: are you running from the reproducer? | 15:58 |
| rlandy | this is a problem | 15:58 |
| rlandy | everyone is hitting it | 15:59 |
| rlandy | I think the old default is better | 15:59 |
| myoung | rlandy: I'm running it now...i created a user (somedev), added to libvirt group (and wheel), and running now | 15:59 |
| myoung | i re-hit having to add own key to authorized_users too...I'm about to get to the same place it failed before | 16:00 |
| rlandy | we need to think about this | 16:00 |
| myoung | test velocity a little reduced (in meetings)...will have a data point in a min (watching it run now) | 16:00 |
| *** tosky has joined #oooq | 16:02 | |
| myoung | rlandy: Error: Cannot access storage file '/home/somedev/s12/validate-fs010/pool/subnode-0.qcow2' (as uid:107, gid:107): Permission denied | 16:04 |
| myoung | 107 --> qemu | 16:04 |
| rlandy | myoung: what is your root partition like? | 16:04 |
| myoung | mean like partition layout? | 16:05 |
| myoung | [somedev@rdo-ci-fx2-02-s6 validate-fs010]$ df -h | 16:05 |
| myoung | Filesystem Size Used Avail Use% Mounted on | 16:05 |
| myoung | devtmpfs 32G 0 32G 0% /dev | 16:05 |
| myoung | tmpfs 32G 0 32G 0% /dev/shm | 16:05 |
| myoung | tmpfs 32G 1.9M 32G 1% /run | 16:05 |
| myoung | tmpfs 32G 0 32G 0% /sys/fs/cgroup | 16:05 |
| myoung | /dev/mapper/fedora-root 49G 8.8G 38G 19% / | 16:05 |
| myoung | tmpfs 32G 5.3G 27G 17% /tmp | 16:05 |
| myoung | /dev/sda1 976M 114M 796M 13% /boot | 16:05 |
| myoung | /dev/mapper/fedora-home 285G 2.9G 268G 2% /home | 16:05 |
| myoung | tmpfs 6.3G 0 6.3G 0% /run/user/0 | 16:05 |
| myoung | tmpfs 6.3G 0 6.3G 0% /run/user/1002 | 16:05 |
| myoung | fedora defaults are to dump all the space --> /home | 16:05 |
| myoung | (beaker made this box) | 16:06 |
| rlandy | default | 16:06 |
| rlandy | myoung: I made a dir /home/temp/images | 16:15 |
| rlandy | set permissions to be wide open | 16:15 |
| rlandy | and I run from there | 16:15 |
| rlandy | you may get by with this root space | 16:15 |
| rlandy | I did not | 16:15 |
| rlandy | myoung: can you try with the /opt/vm_images settings | 16:16 |
| rlandy | and see if that works for you? | 16:16 |
| rlandy | if that works on your default install, I will revert to that setting | 16:16 |
| myoung | sure I'll do that now | 16:16 |
| rlandy | thank you | 16:18 |
| weshay|ruck | panda|rover, you still around? | 16:31 |
| panda|rover | weshay|ruck: yes | 16:32 |
| weshay|ruck | panda|rover, what's the update on the rhos-12 gate? | 16:32 |
| panda|rover | weshay|ruck: last time I heard from quiquell|off he was testing it in downstream jenkins and it needed to set the build_test_packages variable to true because he wasn't fetching any change | 16:34 |
| panda|rover | weshay|ruck: but he was able to clear the resources downstream to test it properly | 16:35 |
| *** lucasagomes is now known as lucas-afk | 16:36 | |
| weshay|ruck | panda|rover, quiquell|off k.. we'll need a summary of the rhos-12 gate issue to properly hand that off to the next ruck/rover | 16:36 |
| weshay|ruck | panda|rover, fyi.. I ff the master stable branches on code.eng | 16:36 |
| *** tosky has quit IRC | 16:37 | |
| weshay|ruck | panda|rover, where are the docs for this card? https://trello.com/c/yhzulA3V/698-validate-undercloud-single-node-only-on-the-current-deployment | 16:38 |
| myoung | rlandy, trown: it's working now from /opt, but it still feels like a red herring, I manually nuked the pool as well, could this be more sticky pool issue folks are hitting? | 16:39 |
| weshay|ruck | myoung, rlandy trown docs for the reproducer are where? | 16:39 |
| myoung | it's running now, when passing /opt path for the volume path...but on a default install yields | 16:39 |
| myoung | [somedev@rdo-ci-fx2-02-s6 vm_images]$ sudo virsh vol-list --pool oooq_pool | 16:39 |
| myoung | Name Path | 16:39 |
| myoung | ------------------------------------------------------------------------------ | 16:39 |
| myoung | subnode-0.qcow2 /opt/vm_images/subnode-0.qcow2 | 16:39 |
| myoung | subnode-1.qcow2 /opt/vm_images/subnode-1.qcow2 | 16:39 |
| myoung | [somedev@rdo-ci-fx2-02-s6 vm_images]$ sudo virsh pool-info oooq_pool | 16:39 |
| myoung | Name: oooq_pool | 16:39 |
| myoung | UUID: 57f06ce9-dca7-463d-80f1-c37862a07fcb | 16:39 |
| myoung | State: running | 16:39 |
| myoung | Persistent: yes | 16:39 |
| myoung | Autostart: yes | 16:39 |
| myoung | Capacity: 48.97 GiB | 16:39 |
| myoung | Allocation: 10.41 GiB | 16:40 |
| myoung | Available: 38.56 GiB | 16:40 |
| myoung | [somedev@rdo-ci-fx2-02-s6 vm_images]$ ll | 16:40 |
| myoung | total 19008264 | 16:40 |
| myoung | -rw-------. 1 qemu qemu 9710600192 May 1 12:38 subnode-0.qcow2 | 16:40 |
| myoung | -rw-------. 1 qemu qemu 9754378240 May 1 12:38 subnode-1.qcow2 | 16:40 |
| myoung | (eww that formatted badly) - the volume pool is created in /opt, so just using storage in the tiny partition | 16:40 |
| myoung | (testing my theory) | 16:40 |
| myoung | rlandy: also passing internal dns via custom_nameserver prop to test the s/supercede/append | 16:41 |
| rlandy | weshay|ruck: https://github.com/openstack/tripleo-docs/blob/master/doc/source/contributor/reproduce-ci.rst | 16:42 |
| rlandy | need an update :) | 16:42 |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 16:42 |
| *** trown is now known as trown|lunch | 16:43 | |
| rlandy | myoung: yep - that is the problme | 16:43 |
| rlandy | problem | 16:43 |
| weshay|ruck | rlandy, jeez.. the libvirt one :)) | 16:43 |
| weshay|ruck | the one you are working on now | 16:43 |
| rlandy | weshay|ruck: ohhh ... | 16:43 |
| rlandy | sorry - I'm a bit drugged up on cold/flu meds | 16:44 |
| weshay|ruck | as is the rest of the team apparently :) | 16:44 |
| weshay|ruck | thanks for responding | 16:44 |
| rlandy | https://review.openstack.org/#/c/565282/2/roles/create-reproducer-script/templates/reproducer-quickstart.sh.j2 | 16:44 |
| rlandy | is where we are working now | 16:44 |
| weshay|ruck | thanks :) | 16:44 |
| rlandy | there are a set of three patches under test | 16:44 |
| rlandy | you will need them all | 16:44 |
| weshay|ruck | rlandy, and this is executed on the virthost ya? | 16:44 |
| rlandy | yes - we stuck with that | 16:45 |
| weshay|ruck | k | 16:45 |
| rlandy | weshay|ruck: you will need https://review.openstack.org/#/c/565280/ | 16:45 |
| rlandy | and | 16:45 |
| rlandy | https://review.openstack.org/#/c/564865 | 16:45 |
| myoung | weshay|ruck: here's the beginnings of a show-n-tell doc, i'm using it now for test notes | 16:45 |
| myoung | https://trello.com/c/hI3xxjuQ/685-validate-fs-10-w-instruction-breadcrumb-trail#comment-5ae7d48a9b350fb5199dabd6 | 16:45 |
| rlandy | weshay|ruck: the issue under discussion is the libvirt_volume_path | 16:45 |
| rlandy | used to default to /opt/vm_images | 16:46 |
| weshay|ruck | myoung, that card is VERY hard to parse fyi | 16:46 |
| rlandy | now back to original i workspace | 16:46 |
| *** panda|rover is now known as panda|rover|off | 16:47 | |
| weshay|ruck | rlandy, thanks | 16:47 |
| rlandy | weshay|ruck: try it out - and weigh in - what's you preference | 16:48 |
| myoung | weshay|ruck: I guess I can take it down and keep notes privatly...was trying to capture all the steps and output | 16:48 |
| weshay|ruck | rlandy, thanks just getting caught up | 16:48 |
| myoung | rlandy: i think the problem is that the volumes get created as owned by root, but qemu is trying to access them | 16:48 |
| rlandy | pretty much | 16:49 |
| rlandy | which is why opt worked | 16:49 |
| myoung | i don't get why it's working form /opt, even when creating $HOME/pool directly (and chown --> qemu), they still land as owned by root | 16:49 |
| * myoung tries something | 16:50 | |
| myoung | i guess the disconnect for me is why do they end up owned by qemu in the /opt case, and root in the $home case? | 16:51 |
| rlandy | myoung: we could do what I have been doing by default - create a /home/temp dir | 16:54 |
| rlandy | but this current default is definitely not working | 16:54 |
| myoung | confirmed | 16:54 |
| rlandy | myoung: is your run with /opt still going? | 16:55 |
| myoung | no i've been trying to get to root cause. i can kick another | 16:55 |
| myoung | will try /home/vm_images | 16:56 |
| rlandy | myoung: sorry - what was the failure with /opt/vm_images? | 16:56 |
| *** hamzy has joined #oooq | 16:56 | |
| myoung | rlandy: it works | 16:57 |
| myoung | rlandy: the only issue is on a typical fedora / beaker install, the root partition where /opt lives is generally quite small, while /home has the bulk of avail storage | 16:57 |
| rlandy | myoung: ok - so I think we should put that back as the default in the vars file | 16:57 |
| myoung | yup | 16:57 |
| rlandy | I know | 16:57 |
| myoung | agreed | 16:57 |
| myoung | it works and it's tuesday :) | 16:57 |
| rlandy | people have the option to define another place | 16:58 |
| weshay|ruck | rlandy, run as root? | 16:58 |
| rlandy | myoung: could yet get all the way through overcloud deploy? | 16:58 |
| rlandy | mine failed in overcloud deploy | 16:58 |
| myoung | weshay|ruck: heh...i banged into that as well. need to run as non-root | 16:58 |
| rlandy | weshay|ruck: before you start, make a change ... | 16:58 |
| weshay|ruck | rlandy, oh w/ deps on the patches | 16:59 |
| weshay|ruck | k | 16:59 |
| weshay|ruck | rlandy, do you have one I can use? | 16:59 |
| rlandy | set roles/libvirt/setup/overcloud/tasks/vars/fake_nodepool_vars.yml ( or whatever it is renamed to - libvirt_volume_path=/opt/vm_images) | 16:59 |
| weshay|ruck | we don't all need our own DNM test patch | 16:59 |
| rlandy | weshay|ruck: a reproducer with the changes? yes - will send you | 17:01 |
| myoung | weshay|ruck: I've been using this https://curldu.mp/952b3959af2a7a145a5bc5684c062fdeef0880b4 | 17:01 |
| rlandy | myoung: so you want to add the libvirt_volume_path change to your patch? | 17:01 |
| rlandy | mine was merged | 17:01 |
| rlandy | myoung: https://review.openstack.org/#/c/565280/4/roles/libvirt/setup/overcloud/tasks/vars/libvirt_nodepool_vars.yml | 17:02 |
| *** hamzy has quit IRC | 17:03 | |
| weshay|ruck | this should work http://logs.openstack.org/82/565282/2/check/tripleo-ci-centos-7-undercloud-upgrades/ae7a59a/logs/reproducer-quickstart.sh | 17:03 |
| myoung | sure...can update | 17:03 |
| rlandy | weshay|ruck: export LIBVIRT_VOLUME_PATH=${LIBVIRT_VOLUME_PATH:="/home/zuul/pool"} is the problematic piece | 17:04 |
| rlandy | you can try it | 17:04 |
| weshay|ruck | ok.. | 17:04 |
| weshay|ruck | thanks | 17:04 |
| rlandy | but looking to change that default | 17:05 |
| myoung | rlandy: https://review.openstack.org/#/c/565280/4..5/roles/libvirt/setup/overcloud/tasks/vars/libvirt_nodepool_vars.yml | 17:06 |
| rlandy | weshay|ruck: oh - you will need to get the zuul changes into your local tq/tqe | 17:06 |
| rlandy | I have another patch to address that | 17:06 |
| rlandy | was voted out of sprint | 17:06 |
| weshay|ruck | ah ya | 17:06 |
| weshay|ruck | k | 17:06 |
| rlandy | myoung: cool - just get rid of git clone https://github.com/openstack/tripleo-quickstart | 17:06 |
| rlandy | git clone https://github.com/openstack/tripleo-quickstart-extras | 17:06 |
| weshay|ruck | looks like we need update the check for tenant creds | 17:07 |
| rlandy | myoung: and I will update the reproducer | 17:07 |
| rlandy | it is updated | 17:07 |
| rlandy | weshay|ruck, ^^ | 17:07 |
| rlandy | myoung: sorry - get rid of the comment on that review | 17:08 |
| weshay|ruck | you sure.. don't see any change to prevent | 17:08 |
| weshay|ruck | bash reproducer-quickstart.sh -w /var/tmp/test_repro/ -v -l | 17:08 |
| weshay|ruck | Tenant credentials are not sourced. | 17:08 |
| rlandy | weshay|ruck: if [[ (! -v OS_TENANT_NAME) && ("$LIBVIRT" != "1") ]]; then | 17:08 |
| rlandy | you need to run with -l | 17:08 |
| rlandy | to get a libvirt reproducer | 17:09 |
| rlandy | it's multinode by default | 17:09 |
| myoung | rlandy: k | 17:09 |
| weshay|ruck | ah rlandy we have a bug.. I ran -v -l | 17:09 |
| weshay|ruck | if you run just -l it works | 17:09 |
| rlandy | weshay|ruck: -v true -l | 17:09 |
| myoung | rlandy: https://review.openstack.org/#/c/565280/4..6/roles/libvirt/setup/overcloud/tasks/vars/libvirt_nodepool_vars.yml | 17:10 |
| weshay|ruck | hrm.. k.. may I fix that? | 17:10 |
| weshay|ruck | SEC | 17:10 |
| weshay|ruck | sec | 17:10 |
| weshay|ruck | sorry | 17:10 |
| weshay|ruck | oh.. I'm thinking python args | 17:10 |
| weshay|ruck | hrm. | 17:10 |
| rlandy | myoung: awesomeness - thanks | 17:10 |
| rlandy | weshay|ruck: it's actually a terrible user interface (-v true) - we should rethink that in another sprint | 17:11 |
| myoung | weshay|ruck, rlandy: aye first few times I ran reproducer I kept dropping the "true" from '-v true' | 17:11 |
| weshay|ruck | rlandy, ya.. /me pokes at that for a minute | 17:11 |
| rlandy | i know :( | 17:11 |
| weshay|ruck | rlandy, no worries | 17:12 |
| rlandy | lunch quickly - back in a few | 17:12 |
| myoung | rlandy: same, re-running with PS6 | 17:14 |
| rlandy | cool | 17:15 |
| myoung | weshay|ruck: if you're running inside RH might want to tack on -e custom_nameserver="10.11.5.19" to the 'ansible-playbook libvirt-nodepool.yml ' invocation from reproducer, 8.8.8.8 has been an issue again | 17:17 |
| *** sshnaidm is now known as sshnaidm|afk | 17:17 | |
| myoung | (for me) | 17:17 |
| *** gkadam has quit IRC | 17:17 | |
| rlandy | ok - back | 17:32 |
| rlandy | myoung: a problem even with append? | 17:34 |
| rlandy | myoung: my deploy failed on docker pull | 17:37 |
| myoung | kicking off the actual test now, reproducer just finished (and I rapidly ate some lunch) | 17:39 |
| myoung | rlandy: passing /home/groovy_pool_name btw works like a charm. | 17:39 |
| rlandy | ('Connection aborted.', error(13, 'Permission denied')) | 17:41 |
| rlandy | with docker | 17:41 |
| rlandy | myoung: ^^ let me know if you see that | 17:41 |
| rlandy | could be unrelated | 17:41 |
| myoung | toci is running now...will be a few. how far in did it fail for you? | 17:42 |
| rlandy | prep-containers | 17:42 |
| myoung | (aside) this is totally awesome. being able to repro toci in libvirt locally. | 17:44 |
| myoung | it's making me happy lol | 17:44 |
| myoung | rlandy: re: the append...seems to have worked just fine... | 17:46 |
| myoung | [zuul@subnode-0 ~]$ cat /etc/resolv.conf | 17:46 |
| myoung | ; generated by /usr/sbin/dhclient-script | 17:46 |
| myoung | nameserver 192.168.122.1 | 17:46 |
| myoung | nameserver 10.11.5.19 | 17:46 |
| rlandy | okie dokie | 17:46 |
| myoung | for a future sprint i captured an item to change that to an array vs. singular input (for custom_nameserver) | 17:47 |
| *** hamzy has joined #oooq | 17:57 | |
| *** saneax is now known as saneax-_-|AFK | 18:02 | |
| *** myoung is now known as myoung|biab | 18:06 | |
| * myoung|biab is a taxi for 50m | 18:06 | |
| *** agopi has joined #oooq | 18:10 | |
| *** trown|lunch is now known as trown | 18:12 | |
| chandankumar | weshay|ruck: hello | 18:20 |
| weshay|ruck | chandankumar, howdy | 18:21 |
| chandankumar | weshay|ruck: for doc generation from playbook, i was trying with this patch https://review.openstack.org/#/c/565435/ | 18:21 |
| chandankumar | but it is not working | 18:21 |
| weshay|ruck | chandankumar, ya.. ci team needs to fix that | 18:21 |
| weshay|ruck | chandankumar, probably need a quick lp | 18:21 |
| chandankumar | we need to specify the tag which we is used in tqe.yaml na? | 18:22 |
| weshay|ruck | chandankumar, let's chat about the same job in that patch.. which job were you looking at? | 18:26 |
| *** agopi has quit IRC | 18:28 | |
| rlandy | weshay|ruck: wrt https://review.openstack.org/#/c/565282/2/roles/create-reproducer-script/templates/reproducer-quickstart.sh.j2@87 | 18:35 |
| rlandy | the problem is that there are two defaults | 18:35 |
| rlandy | venv is true for ovb and false for multinode | 18:35 |
| weshay|ruck | rlandy, by default libvirt=0 | 18:35 |
| rlandy | right | 18:36 |
| rlandy | but this is a unified reproducer | 18:36 |
| rlandy | so it's false by default | 18:36 |
| weshay|ruck | rlandy, you'll have to show me how that plays in | 18:36 |
| weshay|ruck | rlandy, ok... that has nothing to do w/ the flag though | 18:36 |
| weshay|ruck | we're just changing it so you dont need to also say "true" | 18:37 |
| rlandy | weshay|ruck: we decided this - create_virtualenv flag should be able to switch from true to false and from false to true | 18:37 |
| weshay|ruck | so we can eliminate -v true.. for -v | 18:37 |
| rlandy | how do you make the ovb case false then? | 18:38 |
| rlandy | {% if 'ovb' in toci_jobtype %} | 18:38 |
| rlandy | : ${CREATE_VIRTUALENV:=true} | 18:38 |
| rlandy | {% else %} | 18:38 |
| rlandy | : ${CREATE_VIRTUALENV:=false} | 18:38 |
| rlandy | {% endif %} | 18:38 |
| rlandy | always be true for ovb then | 18:38 |
| weshay|ruck | rlandy, what line is that? | 18:39 |
| rlandy | top of the file | 18:39 |
| weshay|ruck | rlandy, is it a requirement for ovb? | 18:40 |
| rlandy | it was | 18:40 |
| weshay|ruck | and now? | 18:40 |
| rlandy | to be able to let users switch | 18:40 |
| rlandy | afaik, it has not changed | 18:40 |
| rlandy | so I am happy to add the checks | 18:40 |
| rlandy | but changing the way the options work | 18:40 |
| rlandy | that I think the team should agree on | 18:40 |
| weshay|ruck | rlandy, if one picks ovb, why not set CREATE_VIRTUALENV=true there | 18:40 |
| rlandy | somebody objected | 18:41 |
| rlandy | sshnaidm|afk maybe wanted to use installed | 18:41 |
| weshay|ruck | ya.. line 129 | 18:41 |
| weshay|ruck | rlandy, ok.. then just get rid of the defaults | 18:41 |
| weshay|ruck | and let the user pick | 18:41 |
| rlandy | if you pass -w w/o an option it's the same problem | 18:42 |
| weshay|ruck | just make sure it's doc'd | 18:42 |
| chandankumar | weshay|ruck: https://bugs.launchpad.net/tripleo/+bug/1768326 | 18:42 |
| openstack | Launchpad bug 1768326 in tripleo "TQE collect-logs ansible role does not generates documentation for the tags defined in featureset" [Undecided,New] | 18:42 |
| weshay|ruck | chandankumar, thanks | 18:42 |
| rlandy | weshay|ruck: I will add the checks in this review and change the options in another. | 18:42 |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 18:43 |
| weshay|ruck | rlandy, ya.. that's a solid plan :) | 18:43 |
| rlandy | panda|rover|off will not be happy of we delay sprint goals on this | 18:43 |
| *** agopi has joined #oooq | 18:43 | |
| weshay|ruck | rlandy, I can work with panda|rover|off :) | 18:43 |
| rlandy | brave man | 18:44 |
| trown | haha | 18:47 |
| panda|rover|off | ... | 18:49 |
| weshay|ruck | panda|rover|off, :) | 18:52 |
| trown | myoung|biab: where is there anything misnamed "noodpool" being fixed in https://review.openstack.org/565280 | 18:54 |
| trown | oh it is in extras, and rlandy is responsible for the "noodpool" https://github.com/openstack/tripleo-quickstart-extras/commit/5338db0ffaca130e5f8efa088ce5337859966d8b | 18:59 |
| rlandy | yep - myoung|biab has all the renames set | 19:00 |
| * trown has not been near a noodpool in over a decade | 19:00 | |
| * trown will show himself out | 19:00 | |
| rlandy | lol | 19:00 |
| *** myoung|biab is now known as myoung | 19:16 | |
| myoung | trown: ack, we re-did the patches when rlandy and I both changed the same things. a previous iteration (now abandoned, https://review.openstack.org/#/c/564382) renamed libvirt-fake-noodpool to libvirt-nodepool. when rlandy and I reworked things yesterday as part of merging the rename ended up in https://review.openstack.org/#/c/564865, the parent of https://review.openstack.org/#/c/565280. The commit message still ref'd the older patch. | 19:21 |
| myoung | I'll fix it, thanks. | 19:21 |
| myoung | the rename is covered here https://review.openstack.org/#/c/565282/2/roles/create-reproducer-script/templates/reproducer-quickstart.sh.j2 for the repro script | 19:23 |
| myoung | (for testing I've just hand edited repro script) | 19:23 |
| *** agopi has quit IRC | 19:39 | |
| *** agopi has joined #oooq | 19:39 | |
| myoung | trown: fixed up https://review.openstack.org/#/c/565280, is gtg? | 19:56 |
| myoung | rlandy, trown, the last little (trivial) morsel for the POC nicer card: https://review.openstack.org/565576 Rename get-undercloud-ip.sh --> get-domain-ip.sh | 20:11 |
| myoung | rlandy: my fs10 test deployed successfully, is running tempest now | 20:17 |
| myoung | rlandy: I didn't hit the docker error you did | 20:17 |
| myoung | (err...skipping tempest rather, anyhow...worked on latest patches) | 20:20 |
| rlandy | weshay|ruck: ugh - the problem that solution does not solve is that if you don't pass an argument to a parameter that expects one, it takes the next option as the parameter | 20:20 |
| weshay|ruck | rlandy, want to go through it on blue? | 20:20 |
| rlandy | working through some options | 20:21 |
| weshay|ruck | I think we're making this too hard | 20:21 |
| rlandy | I have included the test you suggested | 20:21 |
| rlandy | it works if that is the last/only option | 20:21 |
| weshay|ruck | ya.. if virtual_env requires and arg | 20:21 |
| * rlandy thinks a bit more - will ping you | 20:21 | |
| weshay|ruck | I noticed that as well.. | 20:21 |
| weshay|ruck | rlandy, imho.. I think it should just be the users option to use or to not use the python-virtenv | 20:27 |
| weshay|ruck | rlandy, we should "recommend" it for ovb | 20:27 |
| rlandy | weshay|ruck: we just need this type of check on each parameter taking an argument if [[ "$2" == "-"* ]]; then echo "mistake"; exit 1; fi | 20:28 |
| weshay|ruck | ok.. sounds gtm | 20:28 |
| weshay|ruck | something to return an error to the use is what is important | 20:29 |
| weshay|ruck | w/ a message | 20:29 |
| rlandy | just thinking about how to do the efficiently | 20:29 |
| rlandy | really I will change the behaviour in another patch | 20:29 |
| rlandy | because everyone tripe over that | 20:29 |
| weshay|ruck | that's fine | 20:29 |
| rlandy | but for today | 20:29 |
| *** myoung is now known as myoung|biab | 20:36 | |
| trown | i really dont think that is part of the sprint goals though... or even part of the libvirt reproducer script patch | 20:38 |
| trown | rlandy: weshay|ruck I think https://review.openstack.org/#/c/565282/ is good to go for what it is trying to do | 20:39 |
| trown | this business with variable validation is tangential to that patch | 20:39 |
| weshay|ruck | trown, ya.. there is going to be a patch on top of this | 20:39 |
| weshay|ruck | re: when to work on it.. if it's the first thing I hit, or Alex hits, or others when they use the script and if it's something we can fix fairly easily... imho we should fix it | 20:41 |
| trown | define fairly easily when sprint ends tomorrow and there are unmerged patches still | 20:42 |
| * trown just wants to get work for sprint merged | 20:42 | |
| weshay|ruck | ya.. I'll remove my -1 that's fine | 20:42 |
| weshay|ruck | we're only delivering this to ourselves to be used in the next sprint | 20:42 |
| weshay|ruck | and we said that up front | 20:42 |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 20:43 |
| weshay|ruck | I want to pretty careful when it's appropriate and when it's not appropriate to feel a termendous amount of pressure | 20:43 |
| weshay|ruck | I don't think this is one of those times | 20:44 |
| trown | ya I have started using it already to look into some of the upgrade jobs... what we have works quite well really | 20:44 |
| weshay|ruck | that is the most important thing | 20:44 |
| trown | snapshotting after the bridge between subnodes is created will take some more work... it doesnt survive a reboot currently | 20:45 |
| rlandy | weshay|ruck: trown: myoung|biab: https://review.openstack.org/#/c/565282/ updated review | 20:45 |
| trown | but the PLAYBOOKS variable seems to work in env_vars_to_src.sh, so it can be used to stop at some point that requires investigation | 20:46 |
| rlandy | with the new /opt/vm_images defeult | 20:46 |
| rlandy | default | 20:46 |
| *** agopi has quit IRC | 20:46 | |
| rlandy | the opt checking stuff is in another patch | 20:46 |
| rlandy | doing last test on that | 20:47 |
| weshay|ruck | trown, I'd rather not keep tripping over that issue w/ the cli | 20:47 |
| trown | weshay|ruck: sorry what issue? I think the reproducer script in general will not be used much on our team with libvirt | 20:48 |
| trown | weshay|ruck: I almost always want to use my snapshots | 20:48 |
| weshay|ruck | trown, so ya.. I wanted to jump back in today.. folks pointed me at the reproducer scripot | 20:48 |
| weshay|ruck | script | 20:48 |
| weshay|ruck | so... | 20:48 |
| weshay|ruck | you tell me folks.. | 20:49 |
| trown | https://etherpad.openstack.org/p/libvirt-setup-fake-nodepool-poc | 20:49 |
| trown | that is where I would have pointed you :P | 20:49 |
| weshay|ruck | ya.. so the fact we have multiple people saying different things indicates something | 20:49 |
| trown | once you have a snapshot after nodepool-setup... there is really not much reason to ever run the previous steps again | 20:50 |
| * weshay|ruck tries the etherpdad | 20:50 | |
| weshay|ruck | etherpad | 20:50 |
| weshay|ruck | trown, w/ regards to the virt_env.. I'm wondering why that can't just be a user's choice.. yes/no | 20:52 |
| weshay|ruck | rlandy, ^ | 20:52 |
| weshay|ruck | having two different defaults for the same variable is not ideal | 20:53 |
| rlandy | weshay|ruck: I think since we are doing this in a new patch, we should just forget that two diff options | 20:53 |
| rlandy | we will default one way | 20:53 |
| rlandy | and forget the rest | 20:53 |
| weshay|ruck | that sounds better.. cool | 20:53 |
| trown | weshay|ruck: ya, that seems legit, just didnt want to block that patch on it | 20:53 |
| trown | since it is not really even related to that patch | 20:54 |
| weshay|ruck | well.. I sent rlandy something that I thought fixed it privately | 20:54 |
| rlandy | I will put in patch for discussion when we can schedule it | 20:54 |
| weshay|ruck | I wasn't trying to slow you down w/o helping | 20:54 |
| rlandy | it fixes it if and only if tha is the last/only option | 20:54 |
| *** hamzy has quit IRC | 20:55 | |
| *** agopi has joined #oooq | 21:03 | |
| *** trown is now known as trown|outtypewww | 21:06 | |
| *** agopi has quit IRC | 21:10 | |
| weshay|ruck | panda|rover|off, quiquell|off FYI.. there was some issue w/ the jenkins for rdo2 where the roles for tqe were not getting added to the role path | 21:25 |
| weshay|ruck | panda|rover|off, quiquell|off I have it working now.. but I think the issue was caused by jenkins not properly cleaning up the workspace | 21:26 |
| weshay|ruck | was really odd | 21:26 |
| weshay|ruck | panda|rover|off, quiquell|off I have rm -Rf /home/rhos-ci/jenkins/workspace/* and things are working again | 21:27 |
| weshay|ruck | so.. upgraded fedora 23 -> 25 and nuked the workspace to get rdo2 back on it's feet | 21:27 |
| weshay|ruck | yay jenkins | 21:27 |
| weshay|ruck | ah crap | 21:27 |
| weshay|ruck | it's not working | 21:27 |
| weshay|ruck | lolz | 21:27 |
| panda|rover|off | yay jenkins | 21:28 |
| weshay|ruck | 17:13:48 Requirement already satisfied: tripleo-quickstart-extras from file:///home/rhos-ci/jenkins/workspace/tripleo-quickstart-master-rdo_trunk-baremetal-dell_fc430_envB-single_nic_vlans/tripleo-quickstart-extras/#egg=tripleo-quickstart-extras in /home/rhos-ci/jenkins/workspace/tripleo-quickstart-master-rdo_trunk-baremetal-dell_fc430_envB-single_nic_vlans/lib/python2.7/site-packages (from -r quickstart-extras-requirements.txt (line | 21:47 |
| weshay|ruck | 1)) (2.1.1.dev743) | 21:47 |
| myoung|biab | rlandy: looking | 21:54 |
| *** myoung|biab is now known as myoung | 21:54 | |
| myoung | rlandy, weshay|ruck, trown|outtypewww: regarding args and such, I would love to have a discussion after sprint, I have opinions as well heh. I'll put em in an RFE (after tomorrow) | 22:01 |
| *** tcw has quit IRC | 22:03 | |
| myoung | ^^ TL;DR - as we are wrapping ansible-playbook in a script, would be cool to have an optional param to specify a .yml with params...so things I want can just be kept in my own file, that's fed to reproducer (so VM's land in the right partition, keys, whatever else) can just be managed that way...vs. having to duplicate the params in bash on a per-parameter basis | 22:03 |
| myoung | (that was longer than i had intended) | 22:03 |
| *** tcw has joined #oooq | 22:04 | |
| rlandy | myoung: sure | 22:05 |
| rlandy | there is a patch out there | 22:05 |
| myoung | trown|outtypewww, rlandy: snapshot create/restore works for me. I found a nit to fix, but don't think we need it for this sprint DoD | 22:13 |
| rlandy | nice | 22:14 |
| rlandy | prep-containers still fails for me | 22:15 |
| panda|rover|off | is there a status somewhere for the validate cards ? | 22:24 |
| myoung | panda|rover|off: i put my status in the fs10 card, WFM | 22:25 |
| panda|rover|off | what's not working ? | 22:26 |
| myoung | re: snapshot QE, doc'd in card as well. | 22:26 |
| panda|rover|off | myoung: what's not working ? I've seen turmoil in the channel for the testing | 22:35 |
| rlandy | myoung:any idea about the prep-containers error? | 22:36 |
| myoung | panda|rover|off: Re snapshots I think we're good to go. I just doc'd a few things I think moving foward we should address, but what we've got now satisfies DoD and for s12 IMHO done | 22:37 |
| myoung | re validate testing, i did fs10 and didn't hit an error with containers | 22:37 |
| * myoung looks at cards for details and attempts to repro fail... | 22:37 | |
| myoung | rlandy: i probably missed it in the battle spam but can you point me to the reproducer you're running? I can see if it repros for me | 22:38 |
| rlandy | myoung: prep-contianers keeps failing for me | 22:39 |
| myoung | rlandy: which fs | 22:41 |
| myoung | rlandy: can you 'curl -T your_reproducer.sh curldu.mp" and I'll run it on my virthost to see parity ? | 22:42 |
| rlandy | trying the latest reproducer | 22:42 |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 22:43 |
| myoung | rlandy, panda|rover|off, attempting to replicate the failure, keeping notes here: https://trello.com/c/yhzulA3V/698-validate-undercloud-single-node-only-on-the-current-deployment#comment-5ae8f0139959f4509752708a | 23:05 |
| * myoung makes dinner and will check back in a bit | 23:06 | |
| rlandy | cool | 23:07 |
| rlandy | testing fs003 | 23:07 |
| rlandy | this fs is doing better | 23:20 |
| rlandy | will check 10/37 afterwards | 23:20 |
| rlandy | fs003 looks good | 23:48 |
| rlandy | myoung: ^^ | 23:48 |
| myoung | rlandy: one thing I've noticed with pulling down the change | 23:48 |
| rlandy | trying fs010 | 23:48 |
| myoung | it didn't seem to chain in the dependant change to TQ | 23:48 |
| myoung | but dunno if was supposed to | 23:49 |
| rlandy | yep - I had to pull in that change as well | 23:52 |
| rlandy | I have a patch to do that automatiaclly - was voted out of sorint | 23:52 |
| rlandy | testing fs010 now | 23:52 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!