Tuesday, 2017-08-08

*** thorst has joined #openstack-ansible00:00
*** schwicht has joined #openstack-ansible00:00
*** thorst has quit IRC00:02
*** thorst has joined #openstack-ansible00:05
*** pmannidi has joined #openstack-ansible00:15
*** thorst has quit IRC00:23
*** thorst has joined #openstack-ansible00:23
*** yifei has joined #openstack-ansible00:24
*** schwicht has quit IRC00:25
*** lucasxu has joined #openstack-ansible00:27
*** thorst has quit IRC00:27
*** thorst has joined #openstack-ansible00:38
*** hw_wutianwei has joined #openstack-ansible00:50
*** markvoelker has joined #openstack-ansible00:55
*** schwicht has joined #openstack-ansible00:55
*** lucasxu has quit IRC00:57
*** markvoelker_ has joined #openstack-ansible00:57
*** charcol has joined #openstack-ansible00:57
*** tobberydberg has joined #openstack-ansible00:59
*** markvoelker has quit IRC01:01
*** gouthamr has quit IRC01:02
*** tobberydberg has quit IRC01:04
*** lucasxu has joined #openstack-ansible01:06
*** tobberydberg has joined #openstack-ansible01:08
*** tobberydberg has quit IRC01:14
*** schwicht has quit IRC01:14
logan-cloudnull let me rebase my munin patch on your fix and see what the cpu looks like. just got my logs working correctly with gzip this afternoon so http://osa-ci.protiumit.com/491005/9/128/logs/munin/localdomain/localhost.localdomain/cpu.html shows the cinder blowup nicely now...01:19
logan-nvm i can't rebase on your fix until we get https://review.openstack.org/#/c/491508/1 merged01:21
*** thorst has quit IRC01:24
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Update worker threads to better optimize gate memory  https://review.openstack.org/49164101:34
cloudnulllogan-: ^01:34
cloudnullthat saves us ~4GiB01:34
*** gkadam has joined #openstack-ansible01:35
cloudnullI think i can find more saving elsewhere.01:35
cloudnullbut that's a start01:35
*** woodard has quit IRC01:42
*** gkadam has quit IRC01:43
*** woodard has joined #openstack-ansible01:48
*** coolj_ is now known as coolj02:09
*** charcol has quit IRC02:09
*** charcol has joined #openstack-ansible02:09
logan-great 4gb wil help a lot02:11
*** tobberydberg has joined #openstack-ansible02:14
*** afranc has quit IRC02:17
*** jowisz has quit IRC02:18
*** NewJorg_ has quit IRC02:18
*** tnarg has quit IRC02:18
*** tobberydberg has quit IRC02:19
*** jowisz has joined #openstack-ansible02:19
*** NewJorg has joined #openstack-ansible02:19
*** afranc has joined #openstack-ansible02:22
*** schwicht has joined #openstack-ansible02:42
openstackgerritLogan V proposed openstack/openstack-ansible master: [DNM] Testing Limestone 3rd party CI  https://review.openstack.org/48840903:10
*** charcol has quit IRC03:16
cloudnulllook, no swap http://logs.openstack.org/41/491641/1/check/gate-openstack-ansible-openstack-ansible-ceph-ubuntu-xenial/a2f9be0/logs/openstack/instance-info/host_system_info_03-07-15.log.txt.gz03:33
logan-:)03:34
logan-i stacked munin + cinder hack + memory optimization into https://review.openstack.org/#/c/488409/ so we can see how the gate timings look with both fixes together03:34
cloudnullsweet !03:35
*** gkadam has joined #openstack-ansible03:36
*** lucasxu has quit IRC03:43
*** Dinesh_Bhor has joined #openstack-ansible03:45
*** udesale has joined #openstack-ansible03:45
*** esberglu has quit IRC03:59
*** poopcat has quit IRC04:01
cloudnulllogan-: looking at your pr i think that we'll need to do the combined merge, to lift the gate pressure.04:21
cloudnullassuming it all works that is :)04:22
*** hybridpollo has quit IRC04:24
*** thorst has joined #openstack-ansible04:25
*** thorst has quit IRC04:30
*** esberglu has joined #openstack-ansible04:49
*** esberglu has quit IRC04:53
cloudnulllogan-: it passed.05:17
cloudnullidk if you're around but i think we might need to take the DNM off, close those other patches and propose this as a gate unblocker.05:18
cloudnullit passed on both rax and infracloud05:19
cloudnullwhich is a pretty good spread05:19
*** woodard has quit IRC05:20
cloudnulllooks like the munin things are not working quite right though05:20
cloudnullthat said, there's very little swap05:21
cloudnullso that's a good start05:21
*** thorst has joined #openstack-ansible05:58
*** markus_z has joined #openstack-ansible05:58
*** udesale__ has joined #openstack-ansible06:02
*** thorst has quit IRC06:03
*** udesale has quit IRC06:03
*** tobberydberg has joined #openstack-ansible06:06
*** Oku_OS-away is now known as Oku_OS06:08
*** drifterza has joined #openstack-ansible06:55
*** mbuil has joined #openstack-ansible06:58
*** pcaruana has joined #openstack-ansible07:00
*** markvoelker_ has quit IRC07:01
*** holmsten has joined #openstack-ansible07:03
*** markvoelker has joined #openstack-ansible07:07
*** markvoelker has quit IRC07:08
*** markvoelker has joined #openstack-ansible07:08
*** gtrxcb has quit IRC07:11
*** jvidal has joined #openstack-ansible07:16
openstackgerritJuan Vidal Allende proposed openstack/openstack-ansible master: Add support for OpenDaylight deployment in neutron  https://review.openstack.org/48013107:21
*** udesale has joined #openstack-ansible07:23
*** udesale__ has quit IRC07:24
*** armaan has joined #openstack-ansible07:24
*** mpranjic has quit IRC07:24
hw_wutianweihi, I use openstack-ansible deploy openstack with ceph, I want to know could i use different network cidr in public and cluster network?07:25
*** udesale__ has joined #openstack-ansible07:26
*** udesale has quit IRC07:29
openstackgerritManuel Buil proposed openstack/openstack-ansible-os_tacker master: Fill the tests directory  https://review.openstack.org/49044007:29
*** jvidal has quit IRC07:37
*** pmannidi has quit IRC07:37
*** jvidal has joined #openstack-ansible07:37
*** cpuga has joined #openstack-ansible07:43
*** thorst has joined #openstack-ansible07:59
*** jvidal has quit IRC08:04
*** thorst has quit IRC08:04
*** shardy has joined #openstack-ansible08:06
*** jafeha has joined #openstack-ansible08:08
*** pbandark has joined #openstack-ansible08:09
*** jvidal has joined #openstack-ansible08:19
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Apply pip_install after container creation  https://review.openstack.org/48520308:23
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder master: [WIP] Remove pip_install & galera_client meta deps  https://review.openstack.org/49171408:25
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_designate master: [WIP] Remove pip_install & galera_client meta deps  https://review.openstack.org/49171608:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: [WIP] Remove pip_install & galera_client meta deps  https://review.openstack.org/49171708:27
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_heat master: [WIP] Remove pip_install & galera_client meta deps  https://review.openstack.org/49171808:27
*** esberglu has joined #openstack-ansible08:28
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_horizon master: [WIP] Remove pip_install & galera_client meta deps  https://review.openstack.org/49171908:28
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone master: [WIP] Remove pip_install & galera_client meta deps  https://review.openstack.org/49172008:30
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_neutron master: [WIP] Remove pip_install & galera_client meta deps  https://review.openstack.org/49172108:31
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: [WIP] Remove pip_install & galera_client meta deps  https://review.openstack.org/49172208:32
evrardjpodyssey4me: thanks for the pip_install removal :)08:32
*** esberglu has quit IRC08:32
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_swift master: [WIP] Remove pip_install & galera_client meta deps  https://review.openstack.org/49172308:32
odyssey4meevrardjp just doing a set of patches for the standard AIO to test it out08:33
evrardjpDepends on https://review.openstack.org/#/c/485203/ ?08:33
evrardjpor standalone?08:33
odyssey4methat will depend on all the others - busy modifying now08:33
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Apply pip_install & galera_client after container creation  https://review.openstack.org/48520308:38
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [WIP] Apply pip_install & galera_client after container creation  https://review.openstack.org/48520308:40
*** cpuga has quit IRC08:44
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_neutron master: [WIP] Remove pip_install & galera_client meta deps  https://review.openstack.org/49172108:45
*** electrofelix has joined #openstack-ansible08:46
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: [WIP] Remove pip_install & galera_client meta deps  https://review.openstack.org/49172208:46
mbuilguys, is there any issue with the gates? My patch was passing them yesterday evening and now I get a time out from Zuul: https://hastebin.com/qoyabubesi.md08:56
jvidalmbuil: I'm having some issues as well09:00
odyssey4meyup, it's very dependent on the nodepool provider - some of them are very, very slow09:02
jvidalodyssey4me: I am getting errors in tempest, it's like it is failing to pull some dependencies, and when it goes to run the tests it fails importing09:04
odyssey4meoh, that's a different matter09:04
jvidalodyssey4me: have you seen anything similar?09:05
odyssey4mepersonally, no09:05
jvidalI was able to manually install those dependencies yesterday, then run tempest09:05
jvidalbut I have no idea where to check the dependency installation09:05
openstackgerritAndy McCrae proposed openstack/openstack-ansible-tests master: Set wsgi_processes to 2 for testing  https://review.openstack.org/49172709:06
odyssey4meandymccr you may wish to peek at https://review.openstack.org/49164109:06
*** armaan has quit IRC09:40
*** armaan has joined #openstack-ansible09:41
*** jafeha has quit IRC09:42
*** shardy has quit IRC09:43
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [WIP] Apply pip_install & galera_client after container creation  https://review.openstack.org/48520309:44
*** yifei has quit IRC09:45
*** pmannidi has joined #openstack-ansible09:48
*** udesale__ has quit IRC09:49
*** shardy has joined #openstack-ansible09:56
*** mk___ has joined #openstack-ansible09:56
*** thorst has joined #openstack-ansible10:00
*** fxpester has quit IRC10:02
*** ivveh has quit IRC10:02
*** fxpester has joined #openstack-ansible10:03
*** thorst has quit IRC10:05
*** stuartgr has joined #openstack-ansible10:09
*** tnarg has joined #openstack-ansible10:17
*** markvoelker has quit IRC10:17
*** mk___ has quit IRC10:17
*** ivveh has joined #openstack-ansible10:17
*** esberglu has joined #openstack-ansible10:18
*** odyssey4me has quit IRC10:18
*** odyssey4me has joined #openstack-ansible10:19
*** esberglu has quit IRC10:21
*** thorst has joined #openstack-ansible10:42
*** armaan has quit IRC10:48
*** armaan has joined #openstack-ansible10:49
*** drifterza has quit IRC10:52
*** udesale has joined #openstack-ansible10:53
*** armaan has quit IRC10:53
*** thorst has quit IRC10:54
*** thorst has joined #openstack-ansible10:54
*** thorst has quit IRC10:59
*** sxc731 has joined #openstack-ansible11:07
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [WIP] Apply pip_install & galera_client after container creation  https://review.openstack.org/48520311:10
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [WIP] Apply pip_install & galera_client after container creation  https://review.openstack.org/48520311:14
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts master: [WIP] Remove pip_install & galera_client meta deps  https://review.openstack.org/49175611:16
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_client master: [WIP] Remove pip_install & galera_client meta deps  https://review.openstack.org/49175711:18
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [WIP] Install pip/galera_client once per host/container  https://review.openstack.org/48520311:19
*** yifei has joined #openstack-ansible11:22
openstackgerritAndy McCrae proposed openstack/openstack-ansible-os_sahara master: Implement uWSGI for sahara-api  https://review.openstack.org/49142111:22
*** drifterza has joined #openstack-ansible11:25
*** schwicht has quit IRC11:26
*** armaan has joined #openstack-ansible11:37
*** yifei has quit IRC11:39
*** smatzek has joined #openstack-ansible11:46
*** smatzek_ has joined #openstack-ansible11:48
*** smatzek has quit IRC11:48
*** thorst has joined #openstack-ansible11:51
*** sxc731 has quit IRC11:56
odyssey4melogan- if you're around, this will help the role tests get moving as well: https://review.openstack.org/49172711:58
*** jafeha has joined #openstack-ansible11:59
*** sxc731 has joined #openstack-ansible12:01
*** esberglu has joined #openstack-ansible12:04
*** esberglu has quit IRC12:09
lvdombrkrhello guys, router both interface external and internal are down i check nutron_l3_agent logs and found error: ERROR neutron.agent.linux.iptables_manager12:10
lvdombrkrwhat it meabs?12:10
*** hw_wutianwei has quit IRC12:11
*** hw_wutianwei has joined #openstack-ansible12:12
jafehahi everybody. i'm currently trying to deploy an additional service with osa (in this case barbican). the deployment fails because our repo hosts did not fetch the sources for this service on initial deployment and it did not by simply creating ht containers + installing the service. can someone tell me how to solve this the right way?12:13
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [WIP] Install pip/galera_client once per host/container  https://review.openstack.org/48520312:14
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_sahara master: Updated from global requirements  https://review.openstack.org/48806812:19
*** yifei has joined #openstack-ansible12:21
jafehanevermind, i found the repo-build.yml :)12:24
*** schwicht has joined #openstack-ansible12:25
*** schwicht has quit IRC12:26
*** mpranjic has joined #openstack-ansible12:28
andymccrhwoarang: ping - or are you still away? :D12:31
*** schwicht has joined #openstack-ansible12:35
*** hw_wutianwei has quit IRC12:40
*** yifei has quit IRC12:45
*** gkadam has quit IRC12:45
*** woodard has joined #openstack-ansible12:45
*** woodard has quit IRC12:45
*** woodard has joined #openstack-ansible12:46
*** sxc731 has quit IRC12:48
*** yifei has joined #openstack-ansible12:49
*** schwicht has quit IRC12:55
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [WIP] Install pip/galera_client once per host/container  https://review.openstack.org/48520312:55
openstackgerritMarc Gariépy proposed openstack/openstack-ansible-repo_build stable/ocata: Fix from OpenStack Ansible Tests  https://review.openstack.org/49179012:56
openstackgerritMarc Gariépy proposed openstack/openstack-ansible-repo_build stable/ocata: Add protection against no local facts being defined  https://review.openstack.org/49155612:57
*** esberglu has joined #openstack-ansible12:58
*** sauloaislan has quit IRC12:59
odyssey4memhayden I've been mucking with your pip install patch - hope you don't mind :)13:02
mhaydengo for it13:02
*** lucasxu has joined #openstack-ansible13:02
*** sc68cal_ has quit IRC13:03
*** clayton has quit IRC13:03
*** sc68cal has joined #openstack-ansible13:05
*** clayton has joined #openstack-ansible13:05
openstackgerritStuart Grace proposed openstack/openstack-ansible master: Improve proxy vars in user_variables.yml  https://review.openstack.org/49179113:06
*** markvoelker has joined #openstack-ansible13:12
*** schwicht has joined #openstack-ansible13:13
mgariepyodyssey4me, my repo server finally built correctly13:18
openstackgerritMajor Hayden proposed openstack/openstack-ansible-lxc_hosts master: [WIP] Try wget for downloading LXC image  https://review.openstack.org/49179813:19
mgariepyI had to nuke repo/pools repo/os-release and repo/links for it to stop using 16.04 whl on 14.0413:19
*** sxc731 has joined #openstack-ansible13:22
openstackgerritMajor Hayden proposed openstack/openstack-ansible-lxc_hosts master: [WIP] Try wget for downloading LXC image  https://review.openstack.org/49179813:22
*** rodolof has joined #openstack-ansible13:25
openstackgerritMerged openstack/openstack-ansible-tests master: Set wsgi_processes to 2 for testing  https://review.openstack.org/49172713:28
*** cshen has joined #openstack-ansible13:29
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Use common pip cache for host & containers  https://review.openstack.org/49180313:29
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_sahara master: Updated from OpenStack Ansible Tests  https://review.openstack.org/48847613:30
mhaydenoof, looks like we might have a bug in openstack-ansible-tests13:33
mhayden/home/jenkins/workspace/gate-openstack-ansible-lxc_hosts-ansible-func-centos-7/tests/common/test-log-collect.sh: line 61: ${$WORKING_DIR}/logs/dmesg.log: bad substitution13:33
* mhayden scurries off to look13:33
*** pbandark has quit IRC13:34
mhaydenfixing13:34
*** pbandark has joined #openstack-ansible13:34
openstackgerritMajor Hayden proposed openstack/openstack-ansible-tests master: Fix bash sub typo  https://review.openstack.org/49180713:36
mhaydencould we get eyes on that one ASAP please? ^^ :)13:36
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [WIP] Install pip/galera_client once per host/container  https://review.openstack.org/48520313:38
mhaydenthanks for working on that one, odyssey4me13:38
mhaydennot sure how that bash typo made it into openstack-ansible-tests13:38
mhaydeni figured the gate would have bombed13:38
odyssey4memhayden youcandowhateveryouwant || true13:39
mhaydentrolololol13:39
odyssey4mekeep an eye on this one: https://review.openstack.org/49180313:39
mhaydenah, so the us.images.linuxcontainers.org mirror is throwing 400's on occasion :/13:39
odyssey4meoh, there's that13:39
mhaydenodyssey4me: oh wow, that's a lot less code than my idea13:39
odyssey4mebut there's a UK one too13:39
odyssey4meand aria can multi-part download from multiple sources13:39
mhaydenvery nice13:39
odyssey4mehttps://aria2.github.io/13:40
logan-aria2 is nice13:40
logan-very fast w/ the threaded downloads13:41
odyssey4meyup, we use it for python staging: https://github.com/rcbops/rpc-openstack/blob/52576d7f48a08f101eb4da52f61ba7a3c5fb40ad/rpcd/playbooks/stage-python-artifacts.yml#L191-L20213:41
odyssey4mein this case we can do without parts of that as it's only a single file13:41
mhaydenso aria2c is about a half second slower than wget in this case13:41
logan-if the files are large play around with -x13:42
odyssey4mewith a single source - how about with two? ;)13:42
logan-like -x4 etc13:42
logan-it will concurrently download chunks of the file across multiple threads13:42
odyssey4mewget?13:43
*** smatzek_ has quit IRC13:43
logan-aria213:43
logan-wget can't do that I don't thikn13:43
odyssey4menope13:43
odyssey4melftp can IIRC, but it's more complicated13:43
odyssey4meok, lemme leave that test patch to you - I'll look into how we get a base image cached for infra13:43
*** exoduswtf has joined #openstack-ansible13:46
mhaydenso wget seems to be quite reliable so far13:47
mhaydenthe image download takes 2-2.5 sec from my colo and ~ 12s at home on a 300mbit connection13:47
odyssey4methe lxc tooling used wget13:47
odyssey4mewhat're we currently using?13:47
mhaydenmachinectl13:47
mhaydenmy WIP uses wget with read timeouts13:47
odyssey4meis there some special reason why we're using machinectl?13:48
odyssey4meand what is it using underneath?13:48
mhaydeni'd have to dig in systemd's innards to see what it's using13:48
mhaydenso aria -x4 is marginally faster than wget at home for me13:49
mhaydenbut that means an extra package install in the gate13:49
logan-yeah I think the gain is probably too minimal for container caches13:50
logan-they're not big enough13:50
andymccrif its gonna take 40 mins its worth it :P13:51
odyssey4meokie dokey - although I wonder if the multiple source download may help with some nodepool providers13:51
odyssey4meespecially the EU ones13:51
odyssey4mefyi mhayden you broken idempotence for ubuntu with https://github.com/openstack/openstack-ansible/commit/bbbd367db74a3b772596af5d9a2f2cadb0a15855 for the 'Ensure python is installed' patch13:53
mhaydenoops13:54
odyssey4methe result of 2 was only available if the full script was run13:54
mhaydenoh, i wasn't aware that was the reason :/13:54
mhaydeni was head scratching quite a bit on that one13:54
odyssey4methere needs to be at least a check for python being there, then install if it isn't and return results13:55
odyssey4meor remove the changed_when and failed_when stuff13:55
mhaydeni can try to put that back in later today if needed13:55
mhaydenbut on centos, it's not possible to have a system installed w/o python13:55
odyssey4meyeah - saw that - good validation13:56
openstackgerritMajor Hayden proposed openstack/openstack-ansible-pip_install master: Combine RDO/EPEL tasks into one  https://review.openstack.org/49181613:57
openstackgerritMajor Hayden proposed openstack/openstack-ansible-pip_install master: Use 'present' state for pkgs  https://review.openstack.org/49181713:57
mhaydenandymccr: there's the combined yum task i was talking about ^^13:58
openstackgerritStuart Grace proposed openstack/openstack-ansible master: Improve proxy vars in user_variables.yml  https://review.openstack.org/49179113:59
andymccrmhayden: sweet13:59
*** gouthamr has joined #openstack-ansible14:00
mhaydenthe package state for the external repo key doesn't make sense there14:01
mhaydenso i omitted it14:01
mhaydenit would make sense if the deployer provided their own (which is handled in other tasks14:01
*** smatzek has joined #openstack-ansible14:01
*** cshen has quit IRC14:02
*** marst has joined #openstack-ansible14:05
*** cshen has joined #openstack-ansible14:05
*** smatzek has quit IRC14:05
*** smatzek has joined #openstack-ansible14:06
mudpuppyo snap14:11
mudpuppyhttps://antsle.com/product/14:11
odyssey4memhayden how big is that image in MB/GB/whatever14:13
mhayden86M for ubuntu14:13
*** pmannidi has quit IRC14:14
odyssey4mecentos?14:14
lvdombrkrwho can help with external network configuration?14:14
mhaydenodyssey4me: pretty close to the same last i looked14:14
mhayden< 100MB14:14
mhaydenmudpuppy: other than the software, that's something you could build on your own though14:15
mhaydenneat idea14:15
mudpuppyyeah14:15
*** cshen has quit IRC14:18
*** sxc731 has quit IRC14:19
*** drifterza has quit IRC14:22
openstackgerritMajor Hayden proposed openstack/openstack-ansible-os_swift master: [WIP] Enable dnf on host/containers  https://review.openstack.org/49182514:28
mhaydenandymccr: another patch to get dnf rolling in the integrated gate -> https://review.openstack.org/49158414:29
lvdombrkrandymccr: hi,if i want add external-bridge in my openstack_user_conf, afteri do that wich playboks a need to re-run?14:30
andymccrmhayden: if i wouldnt get apprehended by security at the airport i would bring you some biltong!14:30
mhaydentrolololol14:30
andymccrlvdombrkr: what do you mean by external bridge?14:30
odyssey4memhayden andymccr ok, FYI I have the info about creating a reverse proxy for images.linuxcontainers.org - I'm on a conf call and will look into it afterwards and push some patches up for infra to poke at14:31
mhaydencoo14:31
andymccrodyssey4me: biltong for you too!14:31
lvdombrkrandymccr: i want to add additional provider network14:31
evrardjpdear cloudnull, DimGR, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, mhayden, scarlisle, luckyinva, ntt, javeriak, spotz, vdo, jmccrory, alextricity25, jasondotstar, admin0, michaelgugino, ametts, v1k0d3n, severion, bgmccollum, darrenc, JR14:32
evrardjpobinson__, asettle, colinmcnamara, thorst, adreznec, eil397, qwang,nishpatwa_, cathrichardson, drifterza, hwoarang, cshen:14:32
evrardjpthe osa bug triage will start in 1h30’. Please have a look at the bug list before starting: https://etherpad.openstack.org/p/osa-bugtriage14:32
evrardjpThank you14:32
*** cpuga has joined #openstack-ansible14:33
andymccrhmm lvdombrkr - you may be able to just run the lxc-containers-create.yml play14:34
andymccrlvdombrkr: yeah just run that lxc-containers-create.yml play and you should be good to go14:34
*** cpuga_ has joined #openstack-ansible14:35
lvdombrkrandymccr: thank you, i will try14:35
openstackgerritMajor Hayden proposed openstack/openstack-ansible-tests master: Use 'present' pkg state for pip_install  https://review.openstack.org/49183114:38
odyssey4memhayden ^ perhaps we should do that for all the role package_state vars14:38
*** cpuga has quit IRC14:39
mhaydennot a bad idea14:39
mhaydenlemme tinker on this lxc business and then give that a go14:39
*** exoduswtf has quit IRC14:40
*** sxc731 has joined #openstack-ansible14:43
openstackgerritAndy McCrae proposed openstack/openstack-ansible-os_tempest master: Add required pip packages for Neutron tests  https://review.openstack.org/49183314:44
andymccrmhayden: ^ should fix the tempest tests for neutron.14:44
mhaydenyay14:44
openstackgerritAndy McCrae proposed openstack/openstack-ansible-os_neutron master: Add dnf support  https://review.openstack.org/49014414:46
andymccr^ did a dependson14:46
*** smatzek has quit IRC14:46
mhaydenyeah, i think i still need aria :/14:47
mhaydenthat server is flaky14:47
andymccrmhayden: so far ive only seen that image download timeout/fail on infracloud so perhaps that is a line we should look at14:48
mhaydenit just timed out on a test vm14:48
mhayden:/14:48
*** admin0 has joined #openstack-ansible14:48
admin0hello \o14:49
andymccrmhayden: hahahahaha14:49
andymccri stand corrected14:49
* admin0 finally fixes his znc14:49
mhaydenodyssey4me: what's the aria package called on ubuntu?14:49
logan-aria2 i think14:49
admin0aria2 works great :)14:50
*** cshen has joined #openstack-ansible14:50
odyssey4memhayden aria214:50
mhaydengracias14:50
mhaydenis the executable called aria2 or aria2c on ubuntu?14:51
odyssey4mearia2c14:51
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Change the cinder playbook to run API commands first  https://review.openstack.org/49155014:51
cloudnull^ odyssey4me jmccrory logan- this now goes the way of removing the hups all together.14:52
cloudnullI can't quite put my finger on why backup is freaking out now14:53
cloudnull# /now/also14:53
cloudnullbut with gate times as long as they are now, I think we can investigate how to best roll the cinder restarts / reloads when the integrated gate is again stable14:54
odyssey4mewell, there's another question to ask here - why is that restart even being initiated, it shouldn't need to be - the local fact logic to enable/disable it is obviously wrong14:54
cloudnull^ this also could be14:54
mhaydenwe should use gentoo and use -funroll-loops14:55
logan-is it possible to trigger it outside of deployment runs. ie rebooting containers or restarting the host and allowing the containers to restart in random order14:55
mhaydenwould give us a big boost14:55
cloudnull:|14:55
* mhayden waits for it ...14:55
logan-i think you said it was possible cloudnull14:55
logan-w/ restarting in a specific order14:56
cloudnulllogan-:  I can trigger it by redeploying it14:56
cloudnulli've not figured out the restart order.14:56
cloudnullto trigger the issue, stop the volume/backup services.14:56
cloudnulldelete the api container14:56
cloudnullcreate the api container14:56
cloudnullrun os-cinder-install14:57
logan-gotcha14:57
*** tobberyd_ has joined #openstack-ansible14:57
cloudnullat the end of the run cinder-scheduler/backup will be freaking out14:57
cloudnullthose processes run at 100% cpu14:57
odyssey4mecan you register a bug for it with those steps, it'll help to track the various efforts and give us a link to share with the cinder crew14:57
cloudnullwhich is likely a major reason why our gate times are suffering14:58
cloudnullodyssey4me: sure.14:58
*** tobberydberg has quit IRC15:00
*** dmsimard is now known as dmsimard|afk15:00
*** tobberyd_ has quit IRC15:04
openstackgerritMajor Hayden proposed openstack/openstack-ansible-tests master: Use 'present' pkg state for tests repo  https://review.openstack.org/49183115:05
*** sxc731 has quit IRC15:06
logan-wow http://logs.openstack.org/09/488409/7/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-xenial/c4bd4d3/ and http://logs.openstack.org/09/488409/7/check/gate-openstack-ansible-openstack-ansible-ceph-ubuntu-xenial/60c485f/ include both of cloudnull's speedups (cinder and memory opt) and didnt even get past repo_build in 2 hours timeout :/15:08
lvdombrkrandymccr: i added my new network into linuxbridge_agent.ini as physical_interface_mappings but after that i got error in linux_bridge logs: Interface brex3 for physical network physnet1 does not exist. Agent terminated!15:10
lvdombrkri recreated containers as you advised15:10
andymccrlvdombrkr: it shouldnt have recreated the containers it should've just changed the configuration - but the linuxbridge_agent is managed by the neutron playbook so you'd have to run that again too15:11
*** jvidal has quit IRC15:11
*** yifei has quit IRC15:11
logan-oh i see why, it only took an hour to fetch the lxc base image in both of those haha15:13
logan-wonder what kind of throughput this infracloud vanilla and chocolate is on15:14
lvdombrkrandymccr:  neutron.yml ?15:15
*** Oku_OS is now known as Oku_OS-away15:16
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-ops master: Implement logging for leapfrog  https://review.openstack.org/49183715:20
cloudnullodyssey4me: https://bugs.launchpad.net/openstack-ansible/+bug/170934615:21
openstackLaunchpad bug 1709346 in openstack-ansible "cinder scheduler/backup using 100% CPU" [Undecided,New]15:21
*** smatzek has joined #openstack-ansible15:22
*** holmsten has quit IRC15:22
*** jvidal has joined #openstack-ansible15:23
*** marst_ has joined #openstack-ansible15:26
*** marst has quit IRC15:27
*** armaan has quit IRC15:29
*** armaan has joined #openstack-ansible15:29
*** sxc731 has joined #openstack-ansible15:31
*** tobberydberg has joined #openstack-ansible15:38
openstackgerritMajor Hayden proposed openstack/openstack-ansible-lxc_hosts master: Download LXC image with async via aria2  https://review.openstack.org/49179815:40
*** cshen has quit IRC15:40
mhaydenodyssey4me: ^^15:41
*** tobberyd_ has joined #openstack-ansible15:41
*** tobberydberg has quit IRC15:42
odyssey4memhayden hmm, that's looking good15:43
openstackgerritAndy McCrae proposed openstack/openstack-ansible-os_neutron master: Updated from OpenStack Ansible Tests  https://review.openstack.org/49054315:43
mhaydenodyssey4me: tests look okay so far15:43
mhaydenran it thrice and the image had downloaded by the time the tasks checked to see if it was done15:43
openstackgerritMerged openstack/openstack-ansible-os_sahara master: Updated from global requirements  https://review.openstack.org/48806815:43
* mhayden crosses fingers15:43
lvdombrkrandymccr: unfortinetly same error15:44
andymccrlvdombrkr: does the linterface exist?15:44
openstackgerritAndy McCrae proposed openstack/openstack-ansible-os_neutron master: Add dnf support  https://review.openstack.org/49014415:45
*** tobberyd_ has quit IRC15:45
lvdombrkrandymcc: yes its exist15:46
*** markus_z has quit IRC15:46
andymccrwhere is that error, on the container or on the host? linuxbridge agent runs on the container too15:47
lvdombrkrandymccr: on the container15:48
andymccrand does brex3 exist on the container?15:49
*** jamesdenton has quit IRC15:49
*** jamesdenton has joined #openstack-ansible15:50
*** cshen has joined #openstack-ansible15:53
*** hamzy has quit IRC15:55
*** armaan has quit IRC15:57
*** armaan has joined #openstack-ansible15:57
mhaydenandymccr: my 4k monitor looks like this today -> http://i.imgur.com/FsVfHR4.png15:58
spotzeeeep this isn't slack!!:)15:59
mhaydenspotz: if it was slack i would be posting more gifs15:59
lvdombrkrandymccr: on container there is no any bridges, if i try brctl show15:59
spotzheheh15:59
*** smatzek has quit IRC15:59
*** smatzek has joined #openstack-ansible16:00
evrardjpBug triage cloudnull, DimGR, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, mhayden, scarlisle, luckyinva, ntt, javeriak, spotz, vdo, jmccrory, alextricity25, jasondotstar, admin0, michaelgugino, ametts, v1k0d3n, severion, bgmccollum, darre16:01
evrardjpnc, JRobinson__, asettle, colinmcnamara, thorst, adreznec, eil397, qwang,nishpatwa_, cathrichardson, drifterza, hwoarang, cshen16:01
evrardjpHere is our bug list for today https://etherpad.openstack.org/p/osa-bugtriage16:01
evrardjp#startmeeting openstack_ansible_meeting16:01
openstackMeeting started Tue Aug  8 16:01:26 2017 UTC and is due to finish in 60 minutes.  The chair is evrardjp. Information about MeetBot at http://wiki.debian.org/MeetBot.16:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:01
openstackThe meeting name has been set to 'openstack_ansible_meeting'16:01
andymccrlvdombrkr: there wouldn't be thats why the mapping should point to the interface that maps into the container - e.g. brex3 maps to eth12 on the container so it creates that link there.16:01
prometheanfirehi16:01
mhaydenyo16:01
andymccro/16:01
evrardjpwoot16:02
evrardjpWe have quorum to start the meeting.16:02
evrardjp#topic this week's bugs16:02
asettleo/16:02
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/170929116:02
openstackLaunchpad bug 1709291 in openstack-ansible "rsyslog_server: /var/log/log-storage is not synchronized" [Undecided,New]16:02
openstackgerritMelissa Kam proposed openstack/openstack-ansible-os_tempest stable/ocata: Support installing plugins from packages  https://review.openstack.org/49184216:02
evrardjpI think this is worth investigating, because it looks we are missing logs.16:03
evrardjpHas anyone an instance available to verify this?16:03
prometheanfirenot sure I'm actually here, at airport16:03
lvdombrkrandymccr: its not clear for me...16:03
prometheanfireand wifi only works for 30 min, so if we want to talk about the newton utf8 config template thing...16:04
cloudnullo/16:04
asettleprometheanfire: in the middle of bug triage16:05
evrardjpthanks asettle :)16:05
prometheanfireasettle: which is why I bring it up :P16:05
asettleprometheanfire: got a link? :)16:05
asettleThe next bug is a docs bug, which I've just looked at16:06
prometheanfirelet me find it16:06
prometheanfiresure, https://bugs.launchpad.net/bugs/170546916:07
openstackLaunchpad bug 1705469 in openstack-ansible "newton config-template change breaks previously working configuration" [High,Confirmed] - Assigned to Matthew Thode (prometheanfire)16:07
evrardjpconfirmed medium16:07
evrardjpso16:07
evrardjpnext one16:07
*** tobberydberg has joined #openstack-ansible16:07
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/170906816:07
openstackLaunchpad bug 1709068 in openstack-ansible "Docs: Available overrides out-of-date" [Undecided,New]16:07
evrardjpYou have an opinion asettle?16:07
asettlelow, confirmed16:07
spotzWe can add the suggested line easily enough I think, not sure about the proposal bot though16:07
asettleProposal bot is too far, agree with spotz16:08
evrardjpprometheanfire: we can have your bug triaged now16:08
asettlespotz: do you want to take that?16:08
evrardjpasettle:  ok with your triage, let me update the bug16:08
odyssey4memy suggesiton for the docs bug is to remove the list of overrides and just have a paragraph informing people how ot find it16:08
asettleevrardjp: already done16:08
asettle:)16:08
prometheanfirecool16:08
evrardjpnext16:08
odyssey4meproposal bot is not something infra wants more of16:08
spotzbah, grab the bug then asettle:)16:08
evrardjpodyssey4me: good to know.16:08
evrardjpso prometheanfire16:09
evrardjp#link https://bugs.launchpad.net/bugs/170546916:09
openstackLaunchpad bug 1705469 in openstack-ansible "newton config-template change breaks previously working configuration" [High,Confirmed] - Assigned to Matthew Thode (prometheanfire)16:09
prometheanfireright16:09
evrardjpthis is already triaged16:09
prometheanfireoh? guess it is, sorry, been out for the last week16:09
odyssey4meprometheanfire in our early discussions when this was discovered, and I'm pretty sure in the bug log too, it was requested that tests be added to confirm now and forever that the config template is doing the right things16:09
evrardjpyes.16:10
odyssey4methis still hasn't been done, and another issue was raised by hughsaunders regarding other values16:10
odyssey4meso the issue is not solved, nor are there tests16:10
evrardjpso the bug is still opened.16:10
prometheanfireok, will work on it when I get back then16:11
odyssey4meso either we revert the last two patches to get it all back to the previous state where all was well, except unicode things - or we move forward with a complete fix16:11
*** tobberydberg has quit IRC16:11
evrardjpFor context of last week's bug triage: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2017/openstack_ansible_meeting.2017-08-01-16.00.log.html#l-14416:11
odyssey4methe unicode things are fixed in ocata onwards, so I do kinda feel like trying to change newton now is perhaps a little too far to step16:12
prometheanfireI'm inclined to back out, but I'm fairly sure it breaks on 16.0416:12
prometheanfireodyssey4me: ya, agreed16:12
odyssey4meso my suggestion is to back out of https://github.com/openstack/openstack-ansible-plugins/commit/782460644902f391cadb83e61c882956c0e569ee and https://github.com/openstack/openstack-ansible-plugins/commit/64353d8ff0d0c2d803aba1f2af607e7ffb13c66216:12
odyssey4methen if there is a problem, a complete fix can be added16:13
prometheanfireyep16:13
evrardjpI am fine with the revert if the revert doesn't break more things.16:13
prometheanfireI'll put that in the bug and submit reviews16:13
prometheanfireevrardjp: it's more of a situation where you do one step forward, one back16:13
prometheanfirefix one thing, break something else16:14
evrardjpyeah. Fine for the revert.16:14
evrardjpand for the inclusion of tests in upper branches.16:14
odyssey4methanks prometheanfire16:14
evrardjpthanks!16:14
evrardjpnext16:14
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/170836016:14
openstackLaunchpad bug 1708360 in openstack-ansible "Openstack-Ansible Install Fails with OVS" [Undecided,New]16:14
evrardjpovs...16:14
evrardjpdo we have resources to check that?16:15
evrardjpsomeone here?16:15
evrardjpelse I'd leave the bug as is. Let me mark on the bug we don't have enough resources to triage this bug for now.16:16
andymccryeah id love if somebody was interested and had time to fix up our ovs bits16:17
andymccrand add reasonable gates too16:17
evrardjpI think we let enough time for ppl to step up on this bug, if nobody answers, it will have to wait.16:18
andymccrthis is true16:18
admin0we did the install with ovs16:18
admin0it fails16:18
admin0but we had to manually work around it16:18
*** strattao has joined #openstack-ansible16:19
evrardjpexactly the same bug admin0 ?16:19
evrardjpDid you document it for others?16:19
evrardjpOr a patch?16:19
evrardjpElse we don't have any code we can show to this bug reporter...16:19
admin0solution was to not use neutron nodes in controllers16:20
admin0but move to hypervisors/compute direct16:20
admin0i will update that16:20
evrardjpnext16:20
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/170796716:20
openstackLaunchpad bug 1707967 in openstack-ansible "ANSIBLE_ROLE_FETCH_MODE = 'git-clone' should not source openstack-ansible.rc" [Undecided,New]16:20
evrardjpFair enough16:21
evrardjpConfirmed16:21
evrardjpWhat's the level though?16:21
evrardjpIt can be fairly disruptive.16:21
evrardjpif loading plugins for old versions when bootstrapping new version :)16:21
evrardjphigh?16:21
strattaoI'm trying to do a quick AIO install (stable/newton) but it keeps failing on the galera-install.yml trying to do the pip_install... Any thoughts on what I can do to troubleshoot?16:22
odyssey4meyeah, but also low hanging fruit16:22
evrardjpyeah.16:22
odyssey4meI'm hoping that someone will pick that up16:22
evrardjpnew contributors? ^^^^^^^^^^^^^^^^^^^^^^^^^^^16:22
evrardjpeasy to pick !16:22
evrardjpI'd like to have the slack @here thing.16:23
evrardjpor a buzz warning.16:23
evrardjpanyway16:23
evrardjpnext16:23
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/170796316:23
openstackLaunchpad bug 1707963 in openstack-ansible "Keystone role needs an LDAP scenario test" [Wishlist,New]16:23
evrardjpis there anyone wanting this very badly? :p16:23
mhaydenimages.linuxcontainers.org is tossing out 400's randomly16:24
mhayden:/16:24
admin0most are i tink16:24
admin0think16:24
admin0because very comany that does private cloud, they have an existing ldap16:24
admin0if there can be a test/ldap  .. its good16:24
evrardjpthat doesn't mean they integrate ldap.16:24
evrardjp:p16:24
evrardjpbut let's mark it confirmed and wishlist16:24
strattaosorry - forgot it was meeting time :/16:24
evrardjpif those companies want it, they will contribute or make it higher prio.16:24
*** strattao has quit IRC16:25
odyssey4memhayden if you change the patch to cater for a list, then put both the us and uk sources in the list? ;)16:25
evrardjpstrattao: don't worry.16:25
openstackgerritMerged openstack/openstack-ansible-openstack_hosts master: Rename package install task  https://review.openstack.org/49160716:25
mhaydenodyssey4me: considering that16:25
evrardjpok confirmed wishlist.16:25
evrardjpnext16:25
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/170714216:25
openstackLaunchpad bug 1707142 in openstack-ansible "pip_install: "The following packages cannot be authenticated"" [Undecided,New]16:25
*** chyka has joined #openstack-ansible16:26
*** cshen has quit IRC16:27
*** ggillies_ has quit IRC16:27
*** pcaruana has quit IRC16:27
evrardjpThe problem is the reproducibility of this bug.16:28
odyssey4mewell, it looks weird - but I haven't had a chance to try to reproduce16:29
evrardjpIf there is someone with that image, shoot.16:29
odyssey4mesomeone has to feedback16:29
evrardjpyeah.16:29
admin0start a virtualbox vagrant VM with box "geerlingguy/ubuntu1604 — somone forgot to pull in right pacakges in the box i guess16:29
*** ggillies has joined #openstack-ansible16:30
admin0i guess the ca-certificates package is missing16:30
evrardjpthat could be it, but then they have a dependency problem with their package if ca-certificates isn't included.16:30
evrardjp:p16:30
evrardjpnot our fault in that case.16:30
evrardjpAnyway.16:30
evrardjpnext16:31
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/170644616:31
openstackLaunchpad bug 1706446 in openstack-ansible "Keystone bootstrap failed when using LDAP" [Undecided,New]16:31
*** nollide has joined #openstack-ansible16:31
admin0\o/ LDAP finally :D16:31
odyssey4meneeds ldap testing to verify16:31
odyssey4meif someone can put together a repeatable ldap test, we'll be able to verify16:31
*** cshen has joined #openstack-ansible16:31
evrardjplooking at last week's conversation, nothing needs updating.16:32
*** sliver has joined #openstack-ansible16:32
evrardjpnext16:32
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/170542016:32
openstackLaunchpad bug 1705420 in openstack-ansible "os-neutron-install.yml fails during LBaaS v2 deployment" [Undecided,New]16:32
evrardjpwhat do you think of this one?16:32
evrardjpis there any neutron db migration expert out there?16:33
evrardjpok let's continue our path then...16:34
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/170537416:34
openstackLaunchpad bug 1705374 in openstack-ansible "realpath() of /etc/uwsgi/nova-placement-uwsgi.ini failed" [Undecided,New]16:34
evrardjpuwsgi! andymccr?16:34
evrardjp:D16:34
andymccrhmm16:34
andymccrnot sure how that could happen16:34
andymccrthe first task in the uwsgi thing is literally to setup /etc/uwsgi dir :P16:34
andymccrhmm16:36
evrardjpweird "could not find the requested service nova-compute"16:36
*** markvoelker has quit IRC16:36
evrardjpit looks like a hurricane went through the deploy16:36
andymccrhmm16:36
sliverquick q, is there a way to force repo nodes to rebuild/resync everything recursively they should contain? I tried adding a 16.04 node as a repo server and it won't rsync the data across to the other repo nodes16:36
andymccrwe could move uwsgi bits before the service setup that may be part of it16:36
*** chyka has quit IRC16:37
evrardjpIt sounds ok, but I am thinking at the impact for upgrades.16:38
andymccrthat error still seems weird16:40
andymccrit wont have an upgrade impact but assign it to me16:40
andymccrim basically fixing it up as part of the wsgi bits anyway16:40
evrardjpok thanks.16:40
evrardjpnext16:41
evrardjp... is pending an answer16:41
evrardjpnext16:41
evrardjphttps://bugs.launchpad.net/openstack-ansible/+bug/170361616:41
openstackLaunchpad bug 1703616 in openstack-ansible "Error " This is not a recognized Fernet token" when installing os-trove" [Undecided,New]16:41
evrardjpdo we have someone to help triage this trove bug?16:41
evrardjpif not let's move on16:42
evrardjp516:42
evrardjp416:42
evrardjp316:42
evrardjp216:42
evrardjp116:42
evrardjpnext.16:42
evrardjphttps://bugs.launchpad.net/openstack-ansible/+bug/169887116:42
openstackLaunchpad bug 1698871 in openstack-ansible "[master] [os_gnocchi]Gnocchi role tests failing on installing pip packages" [Undecided,New] - Assigned to Miguel Alejandro Cantu (miguel-cantu)16:42
evrardjpI think we should remove the assignee16:42
evrardjpHas someone worked on gnocchi recently ?16:43
spotzstevelle was our expert:(16:43
evrardjpalex was taking over the work, but...16:43
*** dmsimard|afk is now known as dmsimard16:43
evrardjpok let's leave it that way16:43
evrardjpnext16:44
evrardjpand last16:44
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/169778216:44
openstackLaunchpad bug 1697782 in openstack-ansible "Mounting of ceph-backed cinder volumes is broken after Ocata upgrade" [Undecided,New] - Assigned to Logan V (loganv)16:44
evrardjplogan-: do you have an update there?16:44
evrardjpI think you were trying to verify this on your env right?16:44
openstackgerritMajor Hayden proposed openstack/openstack-ansible-lxc_hosts master: Download LXC image with async via aria2  https://review.openstack.org/49179816:45
*** shardy has quit IRC16:46
evrardjpok let's close it up for today16:47
evrardjpthanks everyone!16:47
evrardjpNo additional bug someone wants to raise?16:47
evrardjp(I often forgot to ask that...sorry!)16:47
evrardjpEnd meeting in 1016:48
evrardjp516:48
evrardjp416:48
evrardjp316:48
evrardjp216:48
evrardjp116:48
evrardjp#endmeeting16:48
openstackMeeting ended Tue Aug  8 16:48:24 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:48
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2017/openstack_ansible_meeting.2017-08-08-16.01.html16:48
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2017/openstack_ansible_meeting.2017-08-08-16.01.txt16:48
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2017/openstack_ansible_meeting.2017-08-08-16.01.log.html16:48
*** chyka has joined #openstack-ansible16:49
*** chyka has joined #openstack-ansible16:49
mhaydenandymccr: openstack-ansible-tests finished in 30 minutes with package_state present16:52
mhaydenon centos16:52
*** smatzek has quit IRC16:53
andymccrmhayden: ship it!16:53
andymccrmhayden: we were discussing today - setting that to present but moving to latest if we do upgrades16:53
andymccrso default position is "present" but we set the var to latest in upgrade tasks16:53
mhaydenandymccr: https://review.openstack.org/49183116:54
mhaydenfeel free to review16:54
andymccrmhayden: oh wait16:54
andymccrwe'll need to change the upgrade jobs to all use "latest" in their individual override vars16:55
mhaydenorly?16:55
mhaydenoof16:55
andymccralso perhaps the best approach is to actually ajdust the vars to be present in role repo - but set upgrades to latest16:55
andymccrodyssey4me: evrardjp thoughts?16:55
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_client master: Allow MariaDB repo overrides without extra-vars  https://review.openstack.org/49185816:55
mhaydeni thought about tickling individual repo test vars, but ithink odyssey4me suggested doing it globally16:56
andymccrmhayden: no i more mean change the default in repo to be present rather than latest, and only override for upgrades to --> latest16:57
*** chyka has quit IRC16:58
andymccrso that'd only be cinder, glance, swift, neutron, nova, keystone - that would need the overrides for upgrades vars. and we may need some consideration for the upgrades themselves in integrated build.16:58
evrardjplet's do it!17:03
*** kylek3h has joined #openstack-ansible17:04
*** cpuga_ has quit IRC17:04
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_server master: Allow MariaDB repo overrides without extra-vars  https://review.openstack.org/49186217:06
*** udesale has quit IRC17:07
*** sxc731 has quit IRC17:13
*** mbuil has quit IRC17:13
*** strattao has joined #openstack-ansible17:15
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-rabbitmq_server master: Allow RabbitMQ/Erlang repo overrides without extra-vars  https://review.openstack.org/49186317:16
strattaosorry for the interruption earlier. So, anyone got thoughts as to what I can do to troubleshoot the galera-install pip_install failures?17:16
*** smatzek has joined #openstack-ansible17:17
odyssey4mestrattao typically that's a container connectivity issue, or a load balancer issue17:18
odyssey4metroubleshoot the connection through each step17:18
*** cpuga has joined #openstack-ansible17:19
openstackgerritMajor Hayden proposed openstack/openstack-ansible-lxc_hosts master: Download LXC image with async via aria2  https://review.openstack.org/49179817:20
mhaydenandymccr: should i change the package state: present in tests/<repo>-overrides.yml or defaults/main.yml?17:23
*** hamzy has joined #openstack-ansible17:24
mhaydenor odyssey4me ^^17:24
*** electrofelix has quit IRC17:24
mhaydenweird, it looks like the script that runs before bindep in project-config isn't running any longer17:26
openstackgerritStuart Grace proposed openstack/openstack-ansible-ops master: Add setup of OpenStack with flavors, images, etc  https://review.openstack.org/49186717:30
mhaydenlooks like we're running an older version of what appears in project-config17:31
*** rodolof has quit IRC17:31
*** rodolof has joined #openstack-ansible17:31
openstackgerritMajor Hayden proposed openstack/openstack-ansible-lxc_hosts master: Download LXC image with async via aria2  https://review.openstack.org/49179817:31
*** tobberydberg has joined #openstack-ansible17:33
*** chhavi has quit IRC17:33
*** markvoelker has joined #openstack-ansible17:37
openstackgerritMajor Hayden proposed openstack/openstack-ansible-pip_install master: Combine RDO/EPEL tasks into one  https://review.openstack.org/49181617:37
*** tobberydberg has quit IRC17:37
*** cshen has quit IRC17:42
*** markvoelker has quit IRC17:44
mhaydenandymccr: okay, the LXC aria2 stuff is working on CentOS http://logs.openstack.org/98/491798/6/check/gate-openstack-ansible-lxc_hosts-ansible-func-centos-7/ce5a8e7/console.html#_2017-08-08_17_43_51_65790617:45
mhaydendistro package install was horribly slow17:46
mhaydenbut the role finished in < 5 minutes17:46
mhaydentotal time consumed by downloading the image is < 1 second according to ARA17:48
mhaydenit was fully downloaded before the async_status task ran17:48
* mhayden tips his hat to dmsimard again for ARA17:48
spotzwoot!17:50
dmsimardwoot17:54
*** tobberydberg has joined #openstack-ansible18:00
openstackgerritStuart Grace proposed openstack/openstack-ansible-ops master: Pass deployment_environment_variables to OSA  https://review.openstack.org/49187518:01
openstackgerritMatt Thompson proposed openstack/openstack-ansible master: recheck  https://review.openstack.org/49187618:02
*** cshen has joined #openstack-ansible18:03
*** tobberydberg has quit IRC18:04
*** sxc731 has joined #openstack-ansible18:10
*** stuartgr has left #openstack-ansible18:11
*** poopcat has joined #openstack-ansible18:11
strattaoodyssey4me: so, you're right - from within the galera container, I get a connection refused trying to hit the url in the pip.conf18:16
*** albertcard1 has joined #openstack-ansible18:16
strattaonot sure what I need to change though to get a response on the port (8181)18:16
*** poopcat has quit IRC18:17
*** jvidal has quit IRC18:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ceph_client master: Update UCA to Pike  https://review.openstack.org/49188018:27
odyssey4mestrattao check whether you lb is working18:27
*** armaan has quit IRC18:27
*** armaan has joined #openstack-ansible18:28
*** cshen has quit IRC18:28
*** cshen has joined #openstack-ansible18:30
strattaoodyssey4me you are right... haproxy is dead and won't start... grrr18:30
*** nollide has quit IRC18:31
*** tobberydberg has joined #openstack-ansible18:32
*** nollide has joined #openstack-ansible18:35
*** albertcard1 is now known as poopcat18:36
*** vishwanathj has quit IRC18:36
*** vishwanathj has joined #openstack-ansible18:37
*** tobberydberg has quit IRC18:37
andymccrdmsimard: yeah i wish i could get ara to run if it times out :P18:40
*** markvoelker has joined #openstack-ansible18:40
*** tobberydberg has joined #openstack-ansible18:45
admin0first time, failed at this step:  TASK [resolvconf : Gather variables for each operating system]  .steup hosts ..  fatal: [v11]: FAILED! => {"failed": true, "msg": "No file was found when using with_first_found. Use the 'skip: true' option to allow this task to be skipped if no files are found"}18:47
admin0logan-:  here  ?18:48
*** markvoelker has quit IRC18:48
dmsimardandymccr: that's not my fault :D18:49
admin0kind of stuck :  Set resolver rotate when physical_host is not an unbound host" has been omitted from the job because the conditional "['resolvconf_enabled | bool', 'resolver_list is not defined']" was evaluated as "False"18:49
*** tobberydberg has quit IRC18:49
dmsimardandymccr: I may have mentioned this before but to collect logs if the job times out, the log collection needs to be part of a publisher, not the job itself18:49
dmsimardandymccr: publishers run after the job completes (or times out)18:50
andymccrdmsimard: ahh yeah i think we discussed, is there a perf impact on that?18:52
dmsimardandymccr: not really, you just need to separate the log collection logic out of your job and into a publisher instead18:53
dmsimardandymccr: there's an example of such publisher here: https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/infra.yaml#L29918:53
andymccroh nice18:54
dmsimardandymccr: so in that publisher, you'd do like mkdir $WORKSPACE/logs, send everything there or something and then run ara18:54
openstackgerritMerged openstack/openstack-ansible-os_sahara master: Updated from OpenStack Ansible Tests  https://review.openstack.org/48847618:55
dmsimardNote that I'm not entirely sure "postbuildscript" actually works? It's the only instance used in project-config and I know that it's a jenkins plugin .. not sure if there is a zuul-launcher equivalent18:55
*** cshen has quit IRC18:56
dmsimardmost of the publishers I see are scp publishers18:56
* dmsimard digs18:56
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Add dnf support to ceph playbook  https://review.openstack.org/49158418:57
dmsimardlooks like there's scp, ftp, afs.. not sure if it's possible to run an arbitrary shell builder18:58
dmsimardyeah -infra is saying it's better to wait for zuul v3 apparently19:01
* SamYaple points out infra has been saying that for over a year19:01
openstackgerritMerged openstack/openstack-ansible-repo_build stable/ocata: Fix from OpenStack Ansible Tests  https://review.openstack.org/49179019:02
dmsimardSamYaple: but it's really really shipping now :)19:03
dmsimardSamYaple: flipping zuul v3 switch around ptg time19:03
SamYapledmsimard: ive been hurt to often...19:04
dmsimardlol19:04
admin0can anyone check this and  help me: https://gist.github.com/a1git/bf73a5217e680175afe28a52a8138b1319:11
admin0looks like i am mising something . tiny ..19:11
admin0every playbook i run always gets stuck at the  Gather variables for each operating system19:12
admin0step19:12
palendaeI tend to see that if I'm running Ocata or newer on 14.04. Only 16.04 is supported on O+19:14
admin0all are xenial ..19:14
odyssey4meor your paths are wrong - the initiating directory sometimes19:14
odyssey4meie run the playbook from /opt/openstack-ansible/playbooks19:15
admin0it is being run from /opt/openstack-ansible/playbooks19:15
admin0i will bootstrap again19:16
admin0hmm. no help ..    all servers are Ubuntu 16.04.3 LTS ..19:21
admin0setup-hosts being run from inside /opt/openstack-ansible/playbooks/19:21
admin0aha ..    Task "Gathering Facts" has been omitted from the job because the conditional "['resolvconf_enabled | bool']" was evaluated as "False"19:23
admin0what omits this ?19:23
admin0grep -ri resolvconf_enabled /etc/openstack_deploy/*  is null ..19:24
admin0do we now have to declare unbound_hosts ?19:27
admin0i had to create /etc/openstack_deploy/conf.d/unbound.yml   , give my infra hosts and it worked !!! … ??19:31
*** tobberydberg has joined #openstack-ansible19:38
*** markvoelker has joined #openstack-ansible19:44
*** markvoelker has quit IRC19:51
*** rromans has quit IRC19:52
openstackgerritMerged openstack/openstack-ansible-os_tempest master: Add required pip packages for Neutron tests  https://review.openstack.org/49183319:58
*** sxc731 has quit IRC19:59
cloudnulladmin0: did you make it go?20:03
*** jamesdenton has quit IRC20:07
*** goldenfri has joined #openstack-ansible20:07
openstackgerritMerged openstack/openstack-ansible-galera_client master: Allow MariaDB repo overrides without extra-vars  https://review.openstack.org/49185820:08
*** jamesdenton has joined #openstack-ansible20:10
*** hybridpollo has joined #openstack-ansible20:10
mhaydenodyssey4me / andymccr: when we talked about package state present earlier for roles, should that be in defaults/main.yml or tests/<repo>-overrides.yml?20:11
openstackgerritMajor Hayden proposed openstack/openstack-ansible-pip_install master: Set package state to 'present'  https://review.openstack.org/49190420:12
openstackgerritMajor Hayden proposed openstack/openstack-ansible-galera_client master: Set package state to 'present'  https://review.openstack.org/49190520:15
mhaydentrying it with overrides for the moment20:15
*** lucasxu has quit IRC20:17
openstackgerritMajor Hayden proposed openstack/openstack-ansible-galera_server master: Set package state to 'present'  https://review.openstack.org/49190820:18
*** rromans has joined #openstack-ansible20:20
openstackgerritMajor Hayden proposed openstack/openstack-ansible-pip_install master: Use dnf in gate jobs  https://review.openstack.org/49190920:24
openstackgerritMajor Hayden proposed openstack/openstack-ansible-galera_client master: Use dnf in gate jobs  https://review.openstack.org/49191020:26
openstackgerritMajor Hayden proposed openstack/openstack-ansible-galera_server master: Set package state to 'present'  https://review.openstack.org/49190820:28
openstackgerritMajor Hayden proposed openstack/openstack-ansible-galera_server master: Use dnf in gate jobs  https://review.openstack.org/49191220:28
openstackgerritMajor Hayden proposed openstack/openstack-ansible-galera_client master: Use dnf in gate jobs  https://review.openstack.org/49191020:28
openstackgerritMajor Hayden proposed openstack/openstack-ansible-galera_client master: Set package state to 'present'  https://review.openstack.org/49190520:28
openstackgerritMajor Hayden proposed openstack/openstack-ansible-pip_install master: Use dnf in gate jobs  https://review.openstack.org/49190920:29
openstackgerritMajor Hayden proposed openstack/openstack-ansible-pip_install master: Set package state to 'present'  https://review.openstack.org/49190420:29
*** tobberydberg has quit IRC20:30
*** tobberydberg has joined #openstack-ansible20:30
openstackgerritMerged openstack/openstack-ansible-pip_install master: Rename package install task  https://review.openstack.org/49158520:33
admin0cloudnull:  yeah ..  i had to declare the unbound hoss20:34
admin0hosts*20:34
admin0right now its at failed to update apt cache ..  Install distro packages — stuck there :D20:34
admin0some big changes went in in the last 10 days i guess20:34
admin0i do this every other day .. never stuck like this before :)20:35
*** tobberydberg has quit IRC20:35
*** marst_ has quit IRC20:44
*** marst has joined #openstack-ansible20:55
*** schwicht has quit IRC21:01
*** strattao has quit IRC21:06
openstackgerritMerged openstack/openstack-ansible-os_tempest stable/ocata: Add ceph_client role  https://review.openstack.org/49108521:07
cloudnulladmin0: maybe something to do with the apt-proxy?21:07
*** cpuga has quit IRC21:20
*** cpuga_ has joined #openstack-ansible21:21
*** schwicht has joined #openstack-ansible21:21
openstackgerritMajor Hayden proposed openstack/openstack-ansible-galera_client master: Use dnf in gate jobs  https://review.openstack.org/49191021:22
openstackgerritMajor Hayden proposed openstack/openstack-ansible-galera_client master: Set package state to 'present'  https://review.openstack.org/49190521:22
openstackgerritMajor Hayden proposed openstack/openstack-ansible-galera_client master: Install MariaDB-shared  https://review.openstack.org/49193021:22
*** pbandark has quit IRC21:23
*** cpuga_ has quit IRC21:28
*** cpuga has joined #openstack-ansible21:28
admin0cloudnull: .. something to do with DNS21:30
admin0it setups dns on containers21:31
admin0chicken and egg21:31
admin0how do I disable it ?21:31
admin0so it created nameservers as 172.29.237.33  .. but that container is not even created21:32
*** cpuga has quit IRC21:33
admin0setup hosts fails on installing packages .. so containers are never created ..  setup unbound fails because the containers are not there ..   removing unbound does not even allow me to proceed21:34
admin0why is there now a need to have  unbound_hosts to proceed ?21:35
*** markvoelker has joined #openstack-ansible21:37
*** thorst has quit IRC21:37
cloudnullsorry was not watching21:39
cloudnulladmin0: its not required.21:39
cloudnullyou should not need it for a basic deployment21:40
cloudnullif the unbound hosts are not set the entire playbook should skip21:40
cloudnulllogan-: ^21:40
admin0it did not :(21:42
*** markvoelker has quit IRC21:44
admin0dunno how to proceed21:51
*** schwicht has quit IRC21:57
*** smatzek has quit IRC22:03
*** esberglu has quit IRC22:03
*** markvoelker has joined #openstack-ansible22:04
*** esberglu has joined #openstack-ansible22:04
*** markvoelker_ has joined #openstack-ansible22:05
*** esberglu has quit IRC22:08
*** markvoelker has quit IRC22:09
jmccroryadmin0 cloudnull looks like facts are only gathered once per play. it'll be skipped if resolvconf_enabled is False and fail out.22:09
jmccrory`ansible all -m setup` should get you through it, but that should probably get fixed some other way22:10
admin0how shoudl i proceed ?22:10
admin0o22:11
admin0so delete ansible facts, set resolvconf_enabled : False in vairables and retry ?22:11
jmccroryrun ansible all -m setup to force gathering facts for all hosts/containers22:11
jmccroryif you added unbound hosts already and don't want to use unbound, you'll have to clean that out of your inventory manually22:12
admin0the problem is, if i do not say unbound hosts — anywhere, it stops at TASK [resolvconf : Gather variables for each operating system22:13
admin0as I pasted earlier22:13
jmccroryright...so gather facts with the setup command22:13
admin0so setup-hosts, if its stuck, manually do -m setup to reset that, then again rin setup hosts, but wtih —start-at-task .. not not have it stuck in the same spot again ..22:15
*** esberglu has joined #openstack-ansible22:16
admin0hmm.. control c, run -m setup, it resets the nameservers back to the default .. but then running setup-hosts again overrides it .  i did rerun it with openstack-ansible setup-hosts.yml --start-at-task="openstack_hosts : Install distro packages" … but that just gives a list of  PLAY PLAY and nothing else22:18
jmccroryi don't think i'm following you. maybe file a bug to better track this22:18
admin0ok .. from start ..  if unbound is false or not even declared anwhere, setup-hosts stops at  the “gathering facts step” ..  so to overcome that, i declare ubbound hosts and point to the controlles like all others .. and run setup-hosts again ..     it runs, upto a point where it says installing distro packages .. which fails because the nameservers are already set to some unbound containers that are not created yet …. now i press control C here,22:21
admin0run the -m setup to re-do the facts .. which brings the namesrevers back to the original .. now if I run setup-hosts again from start, it will redo the nameserers and stop at this same point .. so quesiton is, how do I proceed .. should I do a —step22:21
*** Test878 has joined #openstack-ansible22:24
jmccrorytry removing the unbound hosts and containers from your inventory first, there's a script to help with that https://docs.openstack.org/openstack-ansible/latest/contributor/inventory.html#inspecting-and-managing-the-inventory22:25
*** schwicht has joined #openstack-ansible22:29
admin0./scripts/inventory-manage.py =>     from osa_toolkit import manage  ImportError: No module named osa_toolkit22:32
admin0looks like bad luck :D22:32
*** markvoelker has joined #openstack-ansible22:36
*** markvoel_ has joined #openstack-ansible22:37
*** thorst has joined #openstack-ansible22:38
*** Test878 has quit IRC22:38
admin0i removed everything .. unbound_setup is set to false. ran setup hosts again .. boom:  fails at Gather variables for each operating system22:40
*** markvoelker_ has quit IRC22:40
*** markvoelker has quit IRC22:41
*** thorst has quit IRC22:43
*** nollide has left #openstack-ansible22:47
cooljadmin0: jmccrory was saying that if you run "ansible all -m setup" now it should create the fact cache for all hosts and skip the hanging task (which is trying to gather facts) when you run the plays again22:48
admin0resolvconf_enabled : False  does not even pass the setup-hosts step  ..  somehow the setup-hosts only contine only when unbound_hosts are given22:48
admin0you mean from another terminal while its hanging on unable to resolve ?22:48
cooljno like ctrl-c and run it then run the plays again. in theory it will skip the task that is hanging...in theory :D22:49
admin0it did not .. becuase it changes the resolve.conf back22:49
admin0to the nameservers ips that do not exist yet22:49
admin0and chaning nameservers could have been done in setup-infra or at the end .. not at the point when the containers are not even created yet22:50
admin0also when resolvconf_enabled : False .. the playbook should not stop, should it ..   while it does not scroll past 1 screen .. dies22:53
cooljidk, i've worked around issues with fact gathering timeout being too short in the past with the setup module trick, but sounds like there's more going on here. opening a bug is probably best22:56
*** markvoelker has joined #openstack-ansible23:01
admin0how do I skip this whole step of overrirding the dns ?23:02
admin0my setup and config is here: https://gist.github.com/a1git/0da0018db9f9fba9e976e821833554bd23:03
*** markvoel_ has quit IRC23:03
*** charcol has joined #openstack-ansible23:04
*** schwicht has quit IRC23:05
admin0its 1AM for me .. i have to be off now23:05
*** marst has quit IRC23:06
*** rodolof has quit IRC23:09
admin0filed a bug report23:12
admin0https://bugs.launchpad.net/openstack-ansible/+bug/170949723:12
openstackLaunchpad bug 1709497 in openstack-ansible "new openstack installation fails " [Undecided,New]23:12
*** markvoelker has quit IRC23:18
*** markvoelker has joined #openstack-ansible23:18
*** markvoelker has quit IRC23:23

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!