Thursday, 2018-05-03

*** dougbtv__ has joined #oooq00:13
*** dougbtv_ has quit IRC00:16
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56429100:45
*** sai_ has quit IRC01:03
*** sai_ has joined #oooq01:06
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/56044502:45
*** saneax-_-|AFK is now known as saneax02:49
*** honza has quit IRC03:06
*** hamzy has joined #oooq03:34
*** udesale has joined #oooq03:36
*** tcw has quit IRC04:08
*** tcw has joined #oooq04:09
*** tcw has quit IRC04:11
*** tcw has joined #oooq04:13
*** jaganathan has joined #oooq04:16
*** jaganathan has joined #oooq04:17
*** tcw has quit IRC04:17
*** tcw has joined #oooq04:20
*** ykarel has joined #oooq04:27
*** tcw has quit IRC04:27
*** jaganathan has quit IRC04:29
*** weshay has quit IRC04:29
*** lhinds has quit IRC04:29
*** jaganathan has joined #oooq04:30
*** tcw has joined #oooq04:30
*** rasca has quit IRC04:30
*** rnoriega_ has quit IRC04:31
*** jschluet has quit IRC04:31
*** lhinds has joined #oooq04:31
*** faceman has quit IRC04:31
*** rook has quit IRC04:31
*** pgadiya has joined #oooq04:32
*** rnoriega has joined #oooq04:33
*** faceman has joined #oooq04:33
*** tcw has quit IRC04:33
*** pgadiya has quit IRC04:34
*** rook has joined #oooq04:34
*** rook is now known as Guest9498304:34
*** jschlueter has joined #oooq04:34
*** tcw has joined #oooq04:34
*** weshay has joined #oooq04:35
*** rasca has joined #oooq04:36
*** tcw has quit IRC04:39
*** tcw has joined #oooq04:40
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/56044504:45
*** links has joined #oooq04:46
*** tcw has quit IRC04:48
*** skramaja has joined #oooq04:55
EmilienMsshnaidm|rover, please look https://review.openstack.org/#/c/553633/05:16
*** marios has joined #oooq05:19
*** saneax is now known as saneax-_-|AFK05:31
*** jfrancoa has joined #oooq05:36
*** tcw has joined #oooq05:43
*** quiquell|off is now known as quiquell|rover05:45
*** quiquell|rover is now known as quiquell05:46
sshnaidm|roverEmilienM, does it upgrade master to master?05:51
*** saneax-_-|AFK is now known as saneax05:54
*** jtomasek has quit IRC05:59
*** kopecmartin has joined #oooq06:10
*** jtomasek has joined #oooq06:11
*** anande has joined #oooq06:15
*** tcw has quit IRC06:17
*** tcw has joined #oooq06:18
*** jtomasek has quit IRC06:18
*** jtomasek has joined #oooq06:19
*** tcw has quit IRC06:21
quiquellsshnaidm|rover: Anything pending from my previous rovering ?06:23
*** tcw has joined #oooq06:24
sshnaidm|roverquiquell, do you have something to pass?06:39
quiquellHave some bugs assing but not in a very advance state06:40
quiquellDo I have to reassign them ?06:40
quiquellAlso the promoter, is working sequancial now, in a tmux session using a script06:41
quiquellDon't know what to do with the CIX cards06:42
quiquellLet's wait for myoung I think06:42
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/56044506:45
sshnaidm|roverquiquell, can you please send me the list of bugs?06:55
*** ccamacho has joined #oooq06:58
*** tcw has quit IRC07:02
*** tcw has joined #oooq07:03
quiquellsshnaidm|rover: ack07:03
quiquellsshnaidm|rover: Do you have a ruck/rover etherpad already =07:03
quiquellI can update it with the latest status07:03
sshnaidm|roverquiquell, https://etherpad.openstack.org/p/rover-03051807:06
sshnaidm|rovermyoung|ruck, ^^07:06
*** sshnaidm|rover has quit IRC07:08
*** tcw has quit IRC07:12
*** quiquell is now known as quiquell|afk07:12
*** tcw has joined #oooq07:13
*** jbadiapa has joined #oooq07:16
*** tesseract has joined #oooq07:18
*** tcw has quit IRC07:21
*** tcw has joined #oooq07:24
*** florianf has joined #oooq07:28
*** tosky has joined #oooq07:29
*** quiquell|afk is now known as quiquell07:29
*** ykarel is now known as ykarel|lunch07:38
*** gkadam has joined #oooq07:39
*** apetrich_ has joined #oooq07:50
*** amoralej|off is now known as amoralej07:50
*** sshnaidm|rover has joined #oooq07:55
*** pliu_ has joined #oooq08:06
quiquellsshnaidm|rover: etherpad updated08:13
quiquellI am preparing a review with stuff about the promoter08:13
sshnaidm|roverquiquell, thanks08:14
sshnaidm|roverquiquell, is there a bug about promoter?08:15
quiquellsshnaidm|rover: Kind of, after running too much promotions in parallel it was failing08:15
*** lucas-afk is now known as lucasagomes08:17
sshnaidm|roverquiquell, is the root cause known?08:21
quiquellsshnaidm|rover: Everything point to not using docker's overlay208:22
quiquellbut maybe we don't need parallel promotions08:22
quiquellI am just going to put some stuff in a review to make it sequancial, kind of WIP08:22
*** ykarel|lunch is now known as ykarel08:25
quiquellsshnaidm|rover: WIP added to the etherpad08:38
*** tcw has quit IRC08:40
*** tcw has joined #oooq08:41
*** links has quit IRC08:43
*** links has joined #oooq08:44
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/56044508:45
sshnaidm|roverquiquell, infinite loop of sequential promotions for all branches seems like a good solution..08:47
quiquellsshnaidm|rover: We have been runing the script for a week or so, and the timing is good enough08:48
sshnaidm|roverquiquell, do you run it right now in promtoer?08:49
quiquellsshnaidm|rover: yes, inside a tmux session08:50
quiquellssh promoter and tmux a08:50
sshnaidm|roverquiquell, tmux session instead of service you mean?08:50
quiquellsshnaidm|rover: Yep, the review is to convert it into a systemd service08:50
sshnaidm|roverquiquell, ok08:51
quiquellsshnaidm|rover: Now it's kind of debug mode inside a tmux08:51
quiquellsshnaidm|rover: For stuff like sova this is the righ place ? https://github.com/rdo-infra/ci-config/tree/master/ci-scripts08:55
sshnaidm|roverquiquell, what do you mean?08:55
*** tcw has quit IRC08:55
quiquellsshnaidm|rover: Hummm, na just a brain fart.08:57
*** jrist has quit IRC08:57
*** jrist has joined #oooq09:02
*** tcw has joined #oooq09:03
*** apetrich_ has quit IRC09:11
*** tcw has quit IRC09:11
*** zoli is now known as zoli|lunch09:30
*** marios has quit IRC09:44
*** marios has joined #oooq09:45
*** yolanda_ has joined #oooq09:49
*** yolanda has quit IRC09:51
*** yolanda has joined #oooq09:56
*** yolanda_ has quit IRC09:59
*** yolanda has quit IRC10:02
*** anande_ has joined #oooq10:07
*** anande_ has quit IRC10:09
*** yolanda has joined #oooq10:15
*** yolanda has quit IRC10:25
*** anande has quit IRC10:31
*** yolanda has joined #oooq10:39
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56429110:45
*** dtantsur|afk is now known as dtantsur10:50
*** apetrich_ has joined #oooq11:00
*** apetrich has quit IRC11:05
*** lucasagomes is now known as lucas-hungry11:10
*** apetrich_ has quit IRC11:10
*** zoli|lunch is now known as zoli11:13
*** moguimar has quit IRC11:30
*** atoth has joined #oooq11:30
*** panda|off is now known as panda|lunch11:33
*** moguimar has joined #oooq11:35
*** tcw has joined #oooq11:51
*** amoralej is now known as amoralej|off11:53
*** amoralej|off is now known as amoralej|lunch11:54
*** quiquell is now known as quiquell|lunch11:56
*** sshnaidm|rover is now known as sshnaidm|rover|a12:08
*** sshnaidm|rover|a is now known as sshnaidm|rover12:09
weshayis review.openstack.org back?12:15
quiquell|lunchweshay: Hi, feels like it12:16
*** quiquell|lunch is now known as quiquell12:16
weshayquiquell|lunch, hrm.. can't hit it12:16
*** lucas-hungry is now known as lucasagomes12:16
weshaypanda|lunch, so w/ regards to merging in increments.. I think we said we have a python tool, with unit tests before we begin ansible integration right?12:17
*** panda|lunch is now known as panda12:17
pandaweshay: yes, but we may need to reassess. The reason we wanted a python script is because we may need this outside of quickstart. I remember anyway seeing one of two parts that need the release file *BEFORE* quickstart launches. We may be able to integrate them in quickstart anyway. Trying to find the exact parts that needed this12:19
*** apetrich has joined #oooq12:19
weshaytrown, re: the reproducer12:20
weshay## Run the libvirt-fake-nodepool playbook12:20
weshayansible-playbook -v tripleo-quickstart/playbooks/libvirt-noodpool.yml -i $WORKSPACE/virthost -e local_working_dir=$PWD/.quickstart -e working_dir=$PWD/.quickstart12:20
weshayis the playbook libvirt-fake-nodepool or libvirt-nodepool12:20
trownweshay: https://github.com/openstack/tripleo-quickstart/blob/master/playbooks/libvirt-nodepool.yml it got changed to libvirt-nodepool12:21
trownweshay: updated etherpad12:21
weshaytrown, k.. and the hosts file required?12:22
weshaytrown, I think that was in your initial doc12:23
weshaybut I don't see it now12:23
arxcruzweshay: hi boss, do you have 5 minutes? want to show you something12:23
weshayarxcruz, in a few.. on a recruiting call atm12:23
arxcruzweshay: k12:24
weshaypanda, are you still leaning towards using python to do this though?12:24
trownweshay: it is in that command line12:27
trownweshay:  -i $WORKSPACE/virthost12:27
trownweshay: which is created with an echo on line 3012:27
pandaweshay: yes, if we want to use ansible, we can alsways make it a module.12:28
weshaytrown, ha.. sorry.. too early12:31
weshaythanks12:31
*** rlandy has joined #oooq12:34
weshayman.. I can not get through to review.openstack :(12:36
weshayI can ping it12:37
trownweshay: do you have ipv6 enabled...12:37
weshayhrm.. will check12:37
trownweshay: I had lots of issues with ipv6 on my external interfaces connecting to openstack.org urls12:38
trownhad to disable it everywhere12:38
weshayya.. looks like it's set to automatic, and I have an address12:39
weshayrestarting network.. brb12:40
chandankumarweshay: myoung|ruck arxcruz kopecmartin feel free to dump your questions and ideas on this card for next sprint https://trello.com/c/efqE5XMr/734-sprint-13-refactor-python-tempestconf12:41
toskychandankumar: the part about tempest-deployer-input is not directly related to the refactor of tempestconf12:44
myoung|ruckweshay: at this point should just be able to run the actual reproducer script, now that patches have landed12:45
myoung|ruckquiquell: o/  welcome back12:45
myoung|ruckall: sprint 13 planning session in 15 min !12:45
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56429112:45
toskychandankumar: it's an important point, but I wouldn't put it as DoD for that card, or it's not going to end for the end of the sprint12:45
quiquellmyoung|ruck: Thanks12:45
chandankumartosky: I have just putten the throughts there, tomorrow we will review the card and will add or subtract the things from that card, feel free to put the ideas there12:46
myoung|ruckquiquell, panda, sshnaidm|rover: after sprint planning would like to touch base (I'll send email).  TL;DR it would be wildly helpful if previous ruck/rover could draft an email summarizing the previous 3 weeks of ruck/rover work.  The etherpad is great, but a higher level summary, pulling out the interesting and/or notable bits (to the community, and to our team).  Also identify any burning/hot issues (e.g. promoter).  We met yesterday,12:48
myoung|ruckbut having something durable would be great.  I'll review the etherpad...perhaps that's already got what's needed.  Over the next few sprints I woudl like to see what works well for handoffs (and community communication) and codify it into our process @tripleo-spec/policy12:48
myoung|ruckalso, I think I speak for the team here, but to panda/quiquell (and always weshay), thanks for your hard work as ruck/rover over the duration of sprint 12.  Rock on.  \nn/12:49
chandankumartosky: for DoD, we generally put what we can achieve by end of the sprint based on discussion12:49
quiquellmyoung|ruck: I have do some handover of the previous etherpad to the new one12:49
pandamyoung|ruck: I sais I'll give you my blood if you need it. All my blood is yours. Set up a meeting and I'll trasfer all my red cells packet by packet.12:50
myoung|rucklolz12:50
myoung|ruckno blood needed (ick, eww)12:50
pandathat's all I have left to offer12:50
*** yolanda has quit IRC12:57
arxcruzmyoung|ruck: is the meeting now ?12:58
myoung|ruckSprint 13 planning starts in 2 mins ;)12:58
myoung|ruckhttps://bluejeans.com/705085945512:59
weshayarxcruz, nice work..13:03
weshaynot great timing :)13:03
arxcruzweshay: ¯\_(ツ)_/¯13:03
weshayarxcruz, really when you are comfortable that would be an excellent email to rh-openstack-dev for everyone to use13:03
myoung|rucksshnaidm|rover: sprint 13 planning is now13:04
toskychandankumar: I can't commit on that board, so please consider my comment13:05
toskys/commit/comment/13:05
chandankumaradding to the board13:06
chandankumartosky: you have an invite13:06
toskychandankumar: oh, thanks13:06
*** honza has joined #oooq13:06
*** honza is now known as Guest9522413:07
*** zoli is now known as zoli|afk13:07
*** Guest95224 is now known as honza_13:07
*** zoli|afk is now known as zoli13:07
*** yolanda has joined #oooq13:11
*** pliu_ has quit IRC13:14
*** Goneri has joined #oooq13:14
*** amoralej|lunch is now known as amoralej13:26
*** ykarel is now known as ykarel|away13:27
*** agopi has quit IRC13:31
*** ykarel|away has quit IRC13:32
EmilienMsshnaidm|rover: no, queens to rocky13:34
EmilienMsshnaidm|rover: if you look at the logs, you can see it13:34
sshnaidm|roverrlandy, please read my comments re https://review.openstack.org/#/c/56583913:36
sshnaidm|roverrlandy, I can use my role for libvirt reproducer still, but I'd like to try the new one too..13:36
sshnaidm|roverrlandy, I hope explained it better in the patch13:36
sshnaidm|roverEmilienM, can you point exactly where it's in the logs? I'm trying to find...13:39
EmilienMsshnaidm|rover: http://logs.openstack.org/33/553633/67/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/8d4b41b/logs/undercloud/var/log/yum.log.txt.gz13:41
EmilienMsshnaidm|rover: you can see package are upgrades to rocky13:41
EmilienMsshnaidm|rover: if you check the tags: https://releases.openstack.org/rocky/13:41
EmilienMsshnaidm|rover: and http://logs.openstack.org/33/553633/67/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/8d4b41b/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz13:42
EmilienMTASK [upgrade all packages] ****************************************************13:42
EmilienMchanged: [centos-7-ovh-gra1-0003845456]13:42
sshnaidm|roverEmilienM, yeah, found it here too: http://logs.openstack.org/33/553633/67/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/8d4b41b/job-output.txt.gz#_2018-05-02_18_00_26_09379913:46
sshnaidm|roverEmilienM, is any deep dive about uc planned? :)13:47
rlandysshnaidm|rover: I am removing my -1 - as I understand you are blocked and there is a workaround for those who nee another dns (although both trown and I did, but we worked on a similar setup). Let's let trown and myoung|ruck  vote on the patch.13:48
sshnaidm|roverrlandy, what is your issue with DNS? why do you need this?13:49
myoung|ruckrlandy, sshnaidm|roverI would like to see 2 things (I can comment in review after sprint planning is done)13:49
rlandysshnaidm|rover: trying to listen to the sprint meeting atm ... will send dns info afterwards13:50
myoung|ruck1. default of 8.8.8.8 since outside red hat that's generally workable.  make additional custom dns setting(s) an "append" vs. "replace"13:50
sshnaidm|roverrlandy, ok13:50
myoung|ruck2. make the input for customn_dns be of type "array" vs. "singleton" . in ansible it's easy to append to an array, then jinja out.  For example for my machines internally, I need to add 3 additional DNS ideally...as lab things happen (packet load spikes, etc) so we need primary, secondary, etc)13:51
EmilienMsshnaidm|rover: yes, I should plan it13:53
EmilienMI will plan it13:53
EmilienMeverytime I postpone :(13:53
*** skramaja has quit IRC13:55
*** agopi has joined #oooq14:02
*** yolanda has quit IRC14:19
*** honza_ is now known as honza14:21
*** apetrich has quit IRC14:23
*** sshnaidm|rover is now known as sshnaidm|afk14:26
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56429114:45
*** yolanda has joined #oooq14:53
*** links has quit IRC15:06
*** ccamacho has quit IRC15:24
*** ccamacho1 has joined #oooq15:24
*** kopecmartin has quit IRC15:29
*** apetrich has joined #oooq15:33
*** ccamacho1 has quit IRC15:35
weshaytrown, when you have a sec.. you used something like  sysctl -w net.ipv6.conf.all.disable_ipv6=1 ?15:36
weshaysysctl -w net.ipv6.conf.default.disable_ipv6=115:36
trownya I think I used exactly that15:37
trownweshay: you can also delete the interface using `ip addr delete <IPv6 address> dev <interface name>` to test it actually helps you15:38
trownweshay: that will come back though15:38
weshayah k15:39
weshaythanks15:39
rlandyweshay: I am having the same problem with access15:56
*** hamzy has quit IRC15:56
rlandyalso disabled ipv615:56
*** panda is now known as panda|off15:57
myoung|ruckarxcruz: I would love to see the work you've done around trello automation - would certainly make life easier.15:59
myoung|ruckarxcruz: can we schedule a time with myself, panda|off. chandankumar, and anyone else interested to see / review / learn about what you've created?15:59
weshayrlandy, it's working for me now..16:02
weshayFYI.. doc review easy one https://review.openstack.org/#/c/565875/16:02
*** quiquell is now known as quiquell|off16:02
panda|offarxcruz: myoung|ruck yeah, trello cli!16:04
*** yolanda has quit IRC16:06
arxcruzmyoung|ruck: sure whenever you want16:08
arxcruzor have time16:08
chandankumarmyoung|ruck: sure16:08
weshaychandankumar, do you have an epic card started for sprint 13, I saw the email thank you :)16:11
chandankumarweshay: https://trello.com/c/efqE5XMr/734-sprint-13-refactor-python-tempestconf16:13
weshaychandankumar++16:13
hubbotweshay: chandankumar's karma is now 216:13
chandankumarweshay: need your mercy sir to get it merged on these evil corps https://review.openstack.org/565657 https://review.openstack.org/56580016:15
weshaychandankumar, so for https://review.openstack.org/#/c/565800/ that is specific to running tempest against the undercloud16:20
weshayya?16:20
chandankumarweshay: yup16:21
weshaychandankumar, do you have any dnm patches to test the older branches?16:21
chandankumarweshay: https://review.openstack.org/#/c/560351/16:22
chandankumarfor pike16:22
chandankumarweshay: i need to realive the dnm patches16:22
weshaychandankumar, export OS_AUTH_URL="$OS_AUTH_URL/v$OS_IDENTITY_API_VERSION"16:24
weshaylooks good.. would be better if it was strings and not vars.. but what ever :)16:25
weshaychandankumar, workflowed both16:26
*** tesseract has quit IRC16:28
weshaychandankumar, arxcruz https://review.openstack.org/56610916:30
*** marios has quit IRC16:33
weshayTASK [libvirt/setup/overcloud : get a list of MACs to use] **********************************************************************************************************************************************************16:38
weshayfatal: [127.0.0.2]: FAILED! => {"msg": "[{u'bridge': u'brovc', u'name': u'overcloud'}, {u'bridge': u'brext', u'forward_mode': u'nat', u'name': u'external', u'netmask': u\"{{ external_network_cidr|ipaddr('netmask') }}\", u'nat_port_range': [1024, 65535], u'address': u'{{ external_network_cidr|nthhost(1) }}', u'dhcp_range': [u'{{ external_network_cidr|nthhost(10) }}', u'{{ external_network_cidr|nthhost(50) }}']}]: The ipaddr filter requires16:38
weshaypython-netaddr be installed on the ansible controller"}16:38
weshayto retry, use: --limit @/var/tmp/test_repro/tripleo-quickstart/playbooks/libvirt-nodepool.retry16:38
weshay16:38
weshayrlandy, trown should we add a yum ensure python-netaddr ?16:39
myoung|ruckweshay: that's one of the items on the todo list16:40
myoung|rucki hit it as well (on a clean f27)16:40
weshayk.. I'll do it now16:40
* myoung|ruck steps out for lunch16:40
myoung|rucki (think?) there's a few others as  well, it's buried in an etherpad/card, will bring it back.  you'll also need a few others that TQ puts down by default16:41
weshaymyoung|ruck, probably should be using centos, not fedora :)16:41
weshayfor the  virthost16:41
myoung|ruckpart of DoD was to ensure that fedora works16:41
myoung|ruckb/c developers us it16:41
myoung|ruckuse it16:41
myoung|ruckalso internal beaker is borked for centos atm...they are fixing it.  can add to watchlist for ticket.  need to dash, back in 1 hr16:42
weshaymyoung|ruck, if the DOD was that fedora works, that is a bad DOD16:43
weshaypanda|off, ^16:43
weshayfedora is called out as not supported for a reason16:44
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56429116:45
weshayhttps://review.openstack.org/#/c/566114/16:48
*** jfrancoa has quit IRC16:50
*** dougbtv_ has joined #oooq16:52
*** yolanda has joined #oooq16:53
*** dougbtv__ has quit IRC16:54
*** tesseract has joined #oooq16:57
*** dougbtv__ has joined #oooq16:57
*** dougbtv_ has quit IRC17:00
*** yolanda_ has joined #oooq17:01
*** gkadam has quit IRC17:02
*** dtantsur is now known as dtantsur|afk17:02
*** hamzy has joined #oooq17:03
*** yolanda has quit IRC17:04
*** saneax is now known as saneax-_-|AFK17:05
*** yolanda has joined #oooq17:05
*** lucasagomes is now known as lucas-afk17:08
*** yolanda_ has quit IRC17:08
*** trown is now known as trown|lunch17:17
*** tesseract has quit IRC17:25
*** florianf has quit IRC17:41
*** udesale has quit IRC17:50
weshaytrown|lunch, rlandy is this right/wrong? https://review.openstack.org/56612717:51
*** Guest94983 is now known as rook17:54
*** zoli is now known as zoli|goe17:57
*** zoli|goe is now known as zoli|gone17:57
*** zoli|gone is now known as zoli17:58
myoung|ruckweshay: re fedora, confirmed that at sprint planning in epic https://trello.com/c/JEGLSVh6/51-reproduce-ci-jobs-with-libvirt we had wanted to ensure that fedora worked. afaik there was a caveat that centos was MOST important, but the practical reality is that folks do use fedora for development work for $reasons, and ideally a libvirt reproducer aimed at that folks would work just fine18:00
weshayhrm..18:02
weshaywho wrote that part?18:03
weshayI think I have to fundementally disagree w/ the fedora part being a "goal" or "DOD" when it's best effort only, no support18:03
weshaypanda|off, ^18:03
weshaymyoung|ruck, do you disagree?18:05
weshayshame on me for not seeing that18:06
weshaymyoung|ruck, panda|off https://github.com/openstack/tripleo-quickstart/blob/master/roles/provision/support_check/tasks/main.yml18:07
*** dougbtv_ has joined #oooq18:12
myoung|ruckweshay: I think if the goal is to provide a tool to reproduce CI for a community, and we as part of planning stipulate that a chunk of that community might use Fedora, it's not crazy to at least understand the impact of not supporting it *for the reproducer* - not for TQ as a whole.  Again this sprint's goal was not to support *anything* specifically.  Understanding the implications was in scope...18:13
weshaymmm18:14
myoung|ruckagain...I'm not proposing that TQ should support fedora...18:14
myoung|ruckor that we were not successful if it didn't work...but we did want the data point...and if (all things being equal) a technical descision needed to be made mid sprint...we would opt to not "on purpose" break fedora...18:14
weshaymyoung|ruck, panda|off I would challenge both of you that it was a sprint goal or priority for the sprint simply based that it's called out as not supported18:15
*** dougbtv__ has quit IRC18:15
rlandyweshay: wrt https://review.openstack.org/#/c/566127, you should not need that18:20
rlandyall tasks that require root are become: true18:20
*** yolanda has quit IRC18:20
myoung|ruckweshay: I think knowing is better than being blindsided...and this is something we should talk about as part of planning for the next sprint that we're working on the reproducer.  Today, the reproducer hasn't been announced yet as something we support  When it is, I seriously doubt that our team would take on supporting Fedora formally.  As i recall in sprint planning ...we were evaluating a POC, and looking with open eyes at what designs18:20
myoung|ruckwould make sense.  I think in that context understanding if Fedora (or RHEL for that matter) was a "you can't get there" for how we were approaching the design.18:20
weshayrlandy, ya..18:21
weshayrlandy, doesn't work that way :)18:21
weshayrlandy, fails from the etherpad notes, and reproducer18:21
myoung|ruckweshay: ...understanding impact... was a good thing to understand18:21
rlandyweshay: can you paste?18:21
weshaymyoung|ruck, let me explain this to you18:21
weshaybluejeans18:22
myoung|ruckk inc18:22
weshayrlandy, ya /me gets18:22
myoung|ruckweshay: joining my channel18:23
rlandy yum ensure python-netaddr - not opposed to that18:23
*** trown|lunch is now known as trown18:29
*** yolanda has joined #oooq18:35
*** amoralej is now known as amoralej|off18:42
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56429118:45
weshaytrown, rlandy http://paste.openstack.org/show/720315/18:46
* weshay gets error from my other run18:46
weshaytrown, rlandy http://paste.openstack.org/show/720316/18:48
trownweshay: just looks like you cant `ssh $USER@127.0.0.2`18:51
weshaytrown, that sounds right18:51
weshaytrown, I was following the usual18:51
weshayssh root@VIRTHOST18:51
weshayjust a habit we shouldn't care about?18:52
trownweshay: so `cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys` and you should be good18:52
weshayk18:52
trownya I think we want to run as the user and sudo18:53
weshayroger that18:53
* rlandy looks18:53
weshayabandoned the patch18:53
rlandywe had some instructions somewhere on how to run on the virthost in general18:56
rlandytrown: any thoughts on the war going on over this patch: https://review.openstack.org/#/c/565839/?418:57
weshayrlandy, ya.. I was apparently just defaulting my behavior to what we do w/ quickstart.sh18:57
weshayfirst thing in the doc there is root@virthost18:57
weshayso18:57
weshayold habits18:57
rlandyweshay: you won't be the only one18:57
weshayya18:57
rlandyI kind if really want some readme or user doc18:58
rlandyI was thinking of adding a readme to the role18:58
weshayrlandy let's add it to the repro doc we have in ci18:58
weshaythe stuff we went through before18:58
weshaywhich reminds me https://review.openstack.org/#/c/566109/18:58
rlandyweshay: sorry which stuff we went through before?18:58
jristweshay: do we need this anymore?18:59
jristhttps://review.openstack.org/#/c/518043/18:59
weshayrlandy, http://logs.openstack.org/33/553633/67/check/tripleo-ci-centos-7-containers-multinode/8517f8d/logs/README-reproducer-quickstart.html19:00
trownrlandy: hmm I dont really understand why we are changing the default from 8.8.8.8 to do nothing, with no way encoded in the reproducer script to override the default of doing nothing19:01
trownrlandy: aka, I think your -1 is correct... I will put comment in the patch19:01
rlandytrown: I could not get to the bottom of why append is not working19:02
rlandysupersede, I understand19:02
rlandybut why restarting unbound - which I tested should kill the first dns server, I don;t know19:02
rlandyweshay: ok - will put up a patch for some doc on the libvirt stuff19:03
weshayrock on19:03
* jrist winks and nudges weshay 19:10
weshayjrist, ah.. crud19:11
*** atoth has quit IRC19:17
*** atoth has joined #oooq19:17
weshayrlandy, what is the login for the subnodes19:19
weshay?19:19
rlandyzuul@19:19
rlandylook at your hosts file19:19
rlandyin workspace19:19
rlandycat hosts file19:19
rlandyyou will see the two nodes and the access info19:20
rlandyweshay: ^^19:20
weshayah see it19:20
weshaythanks19:20
rlandymaking a note to dic that19:22
rlandydoc19:22
rlandyoh my gosh - this doc needs a real update19:32
myoung|ruckweshay: you're hitting stuff in https://trello.com/c/2y2BVXgO/726-ideas-brainstorm-resulting-from-testing-in-sprint-12 :)  good to see that some of the things on that list are good19:32
myoung|ruck(re running as root, key issue, etc)19:33
myoung|ruckweshay, trown, rlandy: what do you think of https://review.openstack.org/#/c/566149?  would remove the needs to have your own key in authorized_users...19:38
*** hubbot has quit IRC19:47
*** hubbot has joined #oooq19:47
sshnaidm|afktrown, re https://review.openstack.org/#/c/565839/ - I think there is misunderstanding there20:02
sshnaidm|afktrown, custom namserver is for special issue when you dns doesn't work and it's problem of user - to have working dns20:03
sshnaidm|afktrown, I can't understand why we need to change defaults of DNS in the host because of that20:03
sshnaidm|afktrown, afaiu we shouldn't take care about any users problem in their hosts20:04
agopimyoung|ruck, weshay please review and merge https://code.engineering.redhat.com/gerrit/#/c/137708/ whenever you find time :) thanks20:04
sshnaidm|afktrown, it's fixing problem in wrong place20:04
agopiweshay++20:05
hubbotagopi: weshay's karma is now 120:05
myoung|ruckhah!  weshay was faster20:06
agopiikr myoung|ruck. must have a bot to automagically merge any commit i post here.20:10
myoung|ruckagopi: it's not magic, weshay is just EVERYWHERE20:10
myoung|ruckagopi: how are thinks in perf ci land generally of late?20:11
myoung|rucks/thinks/things20:11
agopimyoung|ruck, haha20:11
myoung|ruck(ulimit / container perf issues aside)20:11
agopibetter than a month back, but still worse than when justin took off.20:11
trownsshnaidm|afk: my argument is not against changing the default, but changing the default with no ability to change to something else20:13
sshnaidm|afktrown, well, adding additional arguments for script will make it less readable and maintainable, not working dns is very special case and I don't think it happens a lot. If user has non-working dns, he has much more bigger problem than reproducer20:14
sshnaidm|afktrown, otherwise we won't stop adding there additional arguments which resolve some specific host problems for everybody..20:15
trownsshnaidm|afk: that was not put there because we thought it might happen to someone... it flat out didnt work for me without modifying dns20:15
trownsame with rlandy20:15
myoung|rucksame...20:15
sshnaidm|afktrown, does it work now?20:16
trownso this is not hypothetical case20:16
myoung|ruckRH internal anything is borked for 8.8.8.820:16
sshnaidm|afktrown, if it was hypotetical we wouldn't add this option at all, but not resolving mirrorlist is very specific and temporary problem20:16
sshnaidm|afktrown, and changing it to DNS that doesn't work in RH network is not really good solution20:17
sshnaidm|afktrown, I think it's fine to suppose that user has no problem with DNS by default20:17
trownsshnaidm|afk: I still dont understand why append breaks you20:17
sshnaidm|afktrown, please read my comment to rlandy20:18
trownbut sure lets just merge your use case to the detriment of all others20:18
trownnot big deal20:18
trownclearly you feel strongly enough to be on late in your evening about it...20:18
sshnaidm|afktrown, I'm just from concert, and saw comments, couldn't wait :(20:20
*** agopi is now known as agopi|out20:23
sshnaidm|afktrown, rlandy please see this too: https://review.openstack.org/#/c/566063/ it fixes mistake in my previous patch20:23
weshayhrm.. how does the format of the lines fix that?20:26
rlandyweshay: for doc notes ... https://review.openstack.org/#/c/56615520:26
weshayrlandy, very nice20:27
*** agopi|out has quit IRC20:27
weshayrlandy, /me wonders how many people are going to think they can run this from their laptop20:28
rlandyweshay: heaven help them20:28
weshaywondering if we should make that LOUDER20:28
* rlandy turns up the volume20:28
weshayreally.. it's not a terrible thing20:28
weshaybut certainly not the intent20:28
rlandywill make that clearer20:29
weshaysshnaidm|afk, reminder to look at the rdo2 issues tomorrow :)20:29
sshnaidm|afkweshay, tomorrow if Fri :P20:30
weshaydoh20:30
weshaysshnaidm|afk, go drink I mean20:30
sshnaidm|afk\o/20:30
weshaysshnaidm|afk, I hate it when you are ruck/rover20:30
weshayfak20:30
*** weshay is now known as weshay|rover20:30
weshay|roverok.. then20:31
sshnaidm|afkweshay|rover, I'll cover you in Sun, so that's fine20:31
weshay|rovertrown, I'm not ready to merge this.. but initial comments would be welcomed https://review.openstack.org/#/c/565630/20:31
rlandyare we still unresolved on dns server append??20:32
* rlandy hates to not allow for everyone's use case20:32
weshay|roverrlandy, how does this change the code?20:32
weshay|roverhttps://review.openstack.org/#/c/566063/20:32
weshay|rovernot poking, but srsly don't get it20:33
myoung|ruckback in a little bit20:33
* myoung|ruck burns some calories20:33
rlandylooks better20:33
rlandyNODEPOOL_PROVIDER=rdo-cloud-tripleoNODEPOOL_CLOUD=fake-cloud20:36
rlandy^^ not properly defined on subnode020:36
weshay|roveroh.. it's just format? doesn't change the function?20:36
rlandyshould be20:36
rlandyNODEPOOL_PROVIDER=rdo-cloud-tripleo20:36
rlandyNODEPOOL_CLOUD=fake-cloud20:37
rlandylook at your subnode-020:37
rlandyso the job takes those values into account20:37
rlandyprobably not effective here20:37
rlandybut the change is correct20:37
weshay|roverrlandy, k.. thanks20:39
weshay|roverrlandy, trown myoung|ruck sshnaidm|afk the libvirt repro is pretty awesome.. thank you!20:40
weshay|roveroh and panda|off :)20:40
trownweshay|rover: have you tried snapshots yet?20:40
rlandymostly trown's work20:40
trownnot true20:40
weshay|rovertrown, no.. working on an undercloud cont upgrade issue for someone20:40
trownweshay|rover: k, snapshots are pretty great... if we can get snapshots working in between playbooks... life would be easy20:42
weshay|rovertrown, k.. good to know20:43
weshay|roverwill try20:43
weshay|rovertrown, any thoughts on https://review.openstack.org/#/c/565630/ ?20:43
weshay|roverfairly small patch20:43
trownya seems reasonable... will have to try it20:43
weshay|roverI'd rather not reinstall requirments, just quickstart-extra-requirments20:43
sshnaidm|afktrown, weshay|rover rlandy btw, also good for reproducing: https://review.openstack.org/#/c/565759/20:43
weshay|roveror just not [0]20:44
sshnaidm|afkallows you to stop after a specific playbook20:44
weshay|rovertrown, k.. thakns20:44
trownsshnaidm|afk: oh ya I have been trying that out manually... it sometimes works20:44
sshnaidm|afktrown, doesn't it stop?20:44
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56429120:45
trownsshnaidm|afk: it always works in that way... I more meant in terms of being able to run again with different playbooks on the same env20:45
rlandyweshay|rover: https://review.openstack.org/#/c/566155/2/roles/create-reproducer-script/templates/README-reproducer-quickstart.html.j2  loud enough?20:46
weshay|rover<p>Note that the script sets the virthost to be 127.0.0.2 so the reproducer20:46
weshay|rover<b>must be run from the virt host itself</b>. Attempting to use a20:46
weshay|roverworking machine/laptop as the Ansible host and targeting a test box20:46
weshay|roverindirectly is not recommended.20:46
weshay|roverI like it20:46
sshnaidm|afktrown, yeah, this is where snapshots help20:47
weshay|roverrlandy, I would guess we could use this sprint to work out kinks, and announce it at the end of this sprint20:47
trownsshnaidm|afk: ya snapshots are a bit weird too though... because we have to reboot... which destroys vxlan tunnel20:47
trownsshnaidm|afk: i have been messing with it... but a bit of work to do correctly20:47
sshnaidm|afktrown, hm.. I think it only paused for me, but seems I used internal snapshots20:48
sshnaidm|afktrown, in this case it just pause vm and then continue, without reboot20:49
weshay|roverwhen is the last time you saw master consistent --> rdo2 at 0 days20:50
trownsshnaidm|afk: ya I think that might be possible with external snapshots too... but you would have to store memory state in another file... not sure though20:50
sshnaidm|afktrown, if you use backing image, then yes imho20:50
sshnaidm|afknot sure too though..20:50
rlandyweshay|rover: right - b/c essentially we will be using the libvirt reproducer to test upgrades.20:53
weshay|roverhttps://review.openstack.org/#/c/566088/20:53
weshay|roverrlandy, yup20:53
*** hamzy has quit IRC20:53
rlandywhat happened to the 'create workflow/test scenarios' card?20:55
rlandypanda|off: ^^?20:55
*** agopi|out has joined #oooq21:03
weshay|roverhrybacki, you around?21:03
weshay|roverif I see an error like21:04
weshay|rover2018-05-03 16:58:57 | /home/jenkins/workspace/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/.quickstart/usr/local/share/ansible/roles/collect-logs/docs/source/index.rst:21: WARNING: toctree contains reference to document u'overcloud-validate' that doesn't have a title: no link will be generated21:04
weshay|roverthat just means it's missing the title21:04
weshay|roversomething with21:04
weshay|rovertitle21:04
weshay|rover=====21:04
weshay|roverhrybacki, right?21:04
hrybackiweshay|rover: (on phone client so bear with me)21:05
hrybackidocument makes me wonder if its something else21:05
hrybackinvm I see21:05
*** trown is now known as trown|outtypewww21:06
weshay|roverhrybacki, https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/36b4ba5/docs/sphinx_build.txt.gz21:07
weshay|roverit's missing the tempest run21:07
hrybackiweshay|rover: I'd need to pull things up on a bigger screen to be of use here weshay|rover =/ been too long since I looked at this stuff. Need this tonight? I could peak at it first thing when I get in tomorrow morning21:10
weshay|roverhttps://review.openstack.org/#/c/566162/21:10
weshay|roverhrybacki, no no.. this is no rush21:11
weshay|roverhrybacki, we can see how that change impacts it21:11
hrybackiOkay -- will peak tomorrow morning weshay|rover :)21:11
weshay|roverthat's fine, but again no rush..21:11
weshay|roverthanks21:11
weshay|roverchandankumar, fyi ^21:11
*** Goneri has quit IRC21:14
*** jtomasek has quit IRC21:30
myoung|ruckweshay|rover, snapshots are indeed pretty cool.  there's a few low hanging fruit items as well there https://trello.com/c/3aSQKPt6/69-add-snapshotting-capabilites-to-the-workflow#comment-5ae8ce37b2bd1af5775bd35b21:45
myoung|ruckrlandy,  weshay|rover, trown|outtypewww, panda|off, quiquell|off: I've just created https://trello.com/c/bst4Pgel/739-create-human-readable-scenario-definitions-for-upgrades-updates-sprint-13 and added to in progress column21:52
rlandymyoung|ruck: thank you21:53
*** brault has quit IRC21:53
rlandymyoung|ruck: wrt https://review.openstack.org/#/c/566149/22:01
rlandynot really sure about that22:01
rlandyI actually thought about it22:02
rlandybut then you are using 127.0.0.1 in theory, iiuc22:02
myoung|ruckit causes ansible to not use SSH at all afaik, and just run the tasks locally22:05
myoung|ruckseems to run in testing, but I wasn't sure about what else 'local' connection plugin implies22:07
myoung|ruckor if there was a reason to  use 127.0.0.2 vs. just using local.  I'm guessing yall discussed the week I was out during design, just put it up as an idea22:07
*** rlandy is now known as rlandy|bbl22:30
hubbotFAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56429122:45
*** hubbot has quit IRC23:07
*** hubbot has joined #oooq23:08
*** hubbot has quit IRC23:19
*** dmellado has quit IRC23:19
*** dmellado has joined #oooq23:51
*** hubbot has joined #oooq23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!