Monday, 2018-12-03

*** wolverineav has quit IRC00:04
*** eharney has quit IRC00:07
rdogerritSagi Shnaidman created config master: Change OVB experimental job  https://review.rdoproject.org/r/1763600:23
rdogerritMerged config master: Change OVB experimental job  https://review.rdoproject.org/r/1763600:27
rdogerritSagi Shnaidman proposed rdo-jobs master: WIP: Add OVB role to rdo-jobs repo for experiments  https://review.rdoproject.org/r/1740400:29
*** wolverineav has joined #rdo00:34
*** openstackgerrit has joined #rdo02:58
*** ChanServ sets mode: +v openstackgerrit02:58
openstackgerrityatin proposed openstack/packstack master: Replace deprecated 'auth_uri' with 'www_authenticate_uri'  https://review.openstack.org/62146702:58
*** wolverineav has quit IRC03:04
*** wolverineav has joined #rdo03:04
openstackgerrityatin proposed openstack/packstack master: Replace deprecated 'auth_uri' with 'www_authenticate_uri'  https://review.openstack.org/62146703:21
*** wolverineav has quit IRC03:28
*** ramishra has joined #rdo03:31
*** hamzy__ is now known as hamzy03:36
*** wolverineav has joined #rdo03:55
openstackgerrityatin proposed openstack/packstack master: Replace deprecated 'auth_uri' with 'www_authenticate_uri'  https://review.openstack.org/62146704:03
rdogerritrdo-trunk created openstack/paunch-distgit rpm-master: python-paunch: failed to build 08ac829  https://review.rdoproject.org/r/1763705:01
*** agopi has joined #rdo05:05
*** Son_Goku has quit IRC05:21
*** wolverineav has quit IRC05:24
*** zaneb has quit IRC05:27
*** faizy98 has joined #rdo05:31
*** faizy98 has quit IRC05:32
*** faizy98 has joined #rdo05:32
*** wolverineav has joined #rdo05:43
*** gkadam has joined #rdo05:45
*** wolverineav has quit IRC05:48
*** faizy_ has joined #rdo05:50
*** faizy98 has quit IRC05:52
*** faizy_ has quit IRC05:59
rdogerritYatin Karel proposed rdoinfo master: Bump rdoinfo stein-uc tags to latest upper-constraints  https://review.rdoproject.org/r/1756606:02
rdogerritYatin Karel proposed rdoinfo master: Promote CBS tags update for rocky-release  https://review.rdoproject.org/r/1759506:03
rdogerritYatin Karel proposed rdoinfo master: Promote CBS tags update for queens-release  https://review.rdoproject.org/r/1759406:03
*** zul has quit IRC06:04
*** ykarel has joined #rdo06:08
*** apetrich has quit IRC06:25
rdogerritYatin Karel created rdo-jobs master: [DNM] Testing upstream change  https://review.rdoproject.org/r/1763806:28
*** chasmo77 has joined #rdo06:30
*** wolverineav has joined #rdo06:34
rdogerritYatin Karel proposed rdoinfo master: deps: bump Ansible to 2.6.6 for Queens, Stein and Rocky  https://review.rdoproject.org/r/1496006:34
*** apetrich has joined #rdo06:40
*** radeks has joined #rdo06:46
*** radeks_ has joined #rdo06:47
*** radeks has quit IRC06:50
*** kopecmartin|off is now known as kopecmartin06:53
*** wolverineav has quit IRC06:55
*** wolverineav has joined #rdo06:56
*** rcernin has quit IRC06:57
*** wolverineav has quit IRC07:01
*** jfrancoa has joined #rdo07:09
*** quiquell|off is now known as quiquell07:10
*** dpawlik has joined #rdo07:15
*** ccamposr has joined #rdo07:16
*** avivgt has joined #rdo07:17
*** avivgt has quit IRC07:17
*** avivgt has joined #rdo07:18
*** ksambor_ has joined #rdo07:18
*** ksambor_ is now known as ksambor07:18
*** dpawlik has quit IRC07:20
*** dpawlik_ has joined #rdo07:20
*** pcaruana has joined #rdo07:25
*** wolverineav has joined #rdo07:27
*** wolverineav has quit IRC07:28
*** wolverineav has joined #rdo07:28
*** cylopez has joined #rdo07:37
*** quiquell is now known as quiquell|brb07:40
*** wolverineav has quit IRC07:43
*** agurenko has joined #rdo07:52
*** rpittau has joined #rdo07:59
rdogerritMerged rdoinfo master: Bump rdoinfo stein-uc tags to latest upper-constraints  https://review.rdoproject.org/r/1756608:02
*** hseipp has joined #rdo08:05
*** slaweq has joined #rdo08:06
*** jlibosva has joined #rdo08:09
*** shardy has joined #rdo08:11
*** zaneb has joined #rdo08:12
*** jtomasek has joined #rdo08:15
*** jtomasek has quit IRC08:15
*** jtomasek has joined #rdo08:16
*** quiquell|brb is now known as quiquell08:18
*** ktibi has joined #rdo08:23
*** amoralej|off is now known as amoralej08:23
rdogerritYatin Karel proposed rdo-jobs master: [DNM] Testing upstream change  https://review.rdoproject.org/r/1763808:23
amoralejgood morning08:23
ykarelo/08:23
ykarelamoralej, jpena|off can u help in clearing https://review.openstack.org/#/c/621467, multinode packstack jobs failing on tempest, looks firewall related08:26
ykarelbut i can't get why FIREWALL_NEUTRON_TUNNEL_RULES_213.32.78.8 is not in hiera08:26
amoralejlet me check08:26
ykarelokk, /me going for lunch08:26
*** ykarel is now known as ykarel|lunch08:26
*** paramite has joined #rdo08:27
*** jpena|off is now known as jpena08:28
*** ttsiouts has joined #rdo08:29
*** holser_ has joined #rdo08:35
ykarel|lunchamoralej, might be related to puppet update?08:37
ykarel|lunchit was updated on 29th, and last pass i see on 29th: http://zuul.openstack.org/builds?job_name=packstack-multinode-scenario002-tempest08:37
amoralejwe ran packstack scenario in gate to update puppet08:39
ykarel|lunchpackstack multinode?08:41
amoralejoh, right08:43
amoralejnot multinode08:43
ykarel|lunchi can see similar error in update puppet patch: https://centos.logs.rdoproject.org/weirdo-generic-packstack-scenario002/3672/weirdo-project/logs/latest/manifests/192.168.1.38_compute.pp.log.txt.gz08:43
ykarel|lunchamoralej, ^^08:43
ykarel|lunchand in last passing job i can't see ^^ hiera issue, so can say it's puppet related08:45
*** ccamacho has joined #rdo08:45
ykarel|lunchfrom 29th last passing multinode:- http://logs.openstack.org/39/620539/3/gate/packstack-multinode-scenario002-tempest/94da6aa/logs/latest/manifests/104.130.201.54_compute.pp.log.txt.gz08:45
amoralejykarel|lunch, but i see the rules in hiera08:45
amoralejhttp://logs.openstack.org/67/621467/3/check/packstack-multinode-scenario002-tempest/a0f9a09/logs/latest/hieradata/defaults.yaml.txt.gz08:45
*** ttsiouts has quit IRC08:46
jpenaamoralej: it looks like the firewall rule is not created, see http://logs.openstack.org/67/621467/3/check/packstack-multinode-scenario002-tempest/a0f9a09/logs/compute/diag/iptables_-vnL.txt.gz08:46
*** ttsiouts has joined #rdo08:46
amoralejyeah, i'm checking08:47
jpenafor reference:  https://review.rdoproject.org/r/1763708:47
jpenanot that one08:47
jpenahttps://review.openstack.org/#/c/619605/08:47
jpenaI can see the rules being applied there08:47
amoralejyeah, i see the lookup in http://logs.openstack.org/67/621467/3/check/packstack-multinode-scenario002-tempest/a0f9a09/logs/latest/manifests/213.32.78.8_compute.pp.log.txt.gz08:48
*** tosky has joined #rdo08:48
amoralejbut it can't find it08:48
amoralejmmm08:48
amoralej          No such key: "FIREWALL_NEUTRON_TUNNEL_RULES_213"08:48
amoralejit doesn't work with points?08:48
amoralej'.'08:48
ykarel|lunchmaybe that's the issue with new puppet08:48
jpenadid we update the puppet version?08:49
jpenathat seems to be the issue. We have some other rules (for qemu live migration, for instance) with dots, and they're not applied either08:50
amoralejjpena, yes, we updated to 5.5.608:50
jpenaok, so that's it08:50
*** ttsiouts has quit IRC08:51
amoralejlet me look for all hiera info having a '.'08:51
amoralejFIREWALL_NEUTRON_TUNNEL_RULES_213.32.74.15808:52
amoralejFIREWALL_NEUTRON_TUNNEL_RULES_213.32.78.808:52
amoralejFIREWALL_NOVA_QEMU_MIG_RULES_213.32.78.808:52
amoralejwe may need to update those ^08:53
jpenathere are more, amqp for example08:53
jpenaeverything where the host IP is part of the rule name08:53
*** ttsiouts has joined #rdo08:53
*** holser_ has quit IRC08:53
amoralejmmm08:54
amoralejmaybe they are not in this example08:54
*** holser_ has joined #rdo08:54
amoralejbut i didn't find any '.' in the hiera key08:54
amoralejfor amqp08:54
*** holser_ has joined #rdo08:55
jpenayou're right, it's just i nthe comment08:55
openstackgerritJavier Peña proposed openstack/packstack master: Fix Hiera keys with a dot  https://review.openstack.org/62151109:01
amoralejjpena, ykarel|lunch https://pastebin.com/3VDtv4He09:02
amoralejconfirmed09:02
*** jpich has joined #rdo09:03
amoralejpuppet does not work with keys which include dots09:03
ykarel|lunchamoralej, yup i checked the same09:03
ykarel|lunchjpena, can u run openstack-tempest build in Fedora builder, oslo-policy just fixed09:05
ykarel|lunchto get consistent there09:05
ykarel|lunchhttps://trunk.rdoproject.org/fedora/status_report.html09:05
jpenaykarel|lunch: yes, I'll check09:06
ykarel|lunchokk Thanks09:06
*** chem has joined #rdo09:10
*** moguimar has joined #rdo09:15
*** aludwar has quit IRC09:16
*** aludwar has joined #rdo09:18
*** agurenko has quit IRC09:20
*** aufi has joined #rdo09:22
quiquellnumber80: Good morning mate09:32
*** gfidente has joined #rdo09:38
*** ykarel|lunch is now known as ykarel09:41
*** sshnaidm|off is now known as sshnaidm09:43
ykareljpena, Thanks Fedora trunk is consistent now09:44
jpenanice09:44
tristanCwe would like to restart review.rdoproject.org/zuul, is this an issue if we do it now (and re-enqueue all the queues) ?09:44
ykareljpena, is current POST_FAILURES issue known09:47
ykarel"msg": "Source /home/zuul/.ansible/tmp/ansible-tmp-1543824029.898324-74232862558372/source not found"09:47
jpenaykarel: when did that happen? We are in the middle of an SF upgrade, so it might be related09:48
ykareljpena, in current openstack-periodic run09:48
ykarelmany Jobs either timed out or post failure09:50
ykarelalso seein some RETRY_LIMIT09:52
tristanCykarel: is it ok if we restart zuul now?09:52
ykareltristanC, okk for me09:52
ykarelso u will enqueu third party jobs again?09:53
tristanCykarel: yes, thanks09:53
ykareltristanC, ack09:53
jpenaykarel: there's something strange going on, definitely. For example, https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master/28279b2/job-output.txt.gz timed out while installing from pip09:55
ykarelhmm jobs failing with random errors09:56
jpenalet's wait until the upgrade is done, then see09:56
ykarelack09:56
tristanCzuul almost finished re-loading the configuration,queues will be restored soon09:57
ykareljpena, btw your hiera fix patch has jobs failing09:57
jpenaok, I'll chekc it09:57
*** derekh has joined #rdo09:57
ykareljpena, i think we need to combine with my patch10:04
ykarelseeing http://logs.openstack.org/11/621511/1/check/packstack-integration-scenario002-tempest/344011c/logs/latest/manifests/104.130.127.194_controller.pp.log.txt.gz10:05
jpenaykarel: ok then. Feel free to squash my patch into yours10:05
ykareljpena, okk doing10:06
rdogerritJavier Peña created config master: Support "True" and "true" when checking in dlrnapi_report.sh  https://review.rdoproject.org/r/1763910:08
jpenaykarel: it looks like ^^ was also an issue for periodic jobs10:09
*** dciabrin has joined #rdo10:10
tristanCupgrade is now completed, please let us know if something looks broken10:10
openstackgerrityatin proposed openstack/packstack master: Fix Hiera keys with a dot  https://review.openstack.org/62146710:10
rdogerritMerged rdoinfo master: Promote CBS tags update for rocky-release  https://review.rdoproject.org/r/1759510:12
ykareljpena, yes looks like, but what changed zuul_success to True?10:12
ykarelthe upgrade?10:12
rdogerritMerged rdoinfo master: Promote CBS tags update for queens-release  https://review.rdoproject.org/r/1759410:12
jpenaI suspect so10:20
jpenatristanC: ^^10:20
jpenazuul_success | bool is now True instead of true as it used to be10:21
ykarelamoralej, ur comment: Depends-On for upstream changes in dependency updates does not work in rdoinfo. amoralej will it not work even with:- https://review.rdoproject.org/r/#/c/17638/?10:31
ykareli pushed a DNM ^^10:32
*** shardy has quit IRC10:35
*** shardy has joined #rdo10:43
rdogerritGiulio Fidente created rdo-jobs master: Add Rocky to tripleo-ceph-integration job  https://review.rdoproject.org/r/1764010:43
rdogerritGiulio Fidente created config master: Add Rocky to ceph-ansible job  https://review.rdoproject.org/r/1764110:44
amoralejykarel, with rdo-jobs it will work10:47
*** panda|pto is now known as panda10:47
ykarelamoralej, ack so let's see how it goes10:47
ykareljob just failed, and got into queued state10:47
ykarelseen 2018-12-03 10:43:54.087656 | primary | MODULE FAILURE in10:48
ykarelin live console10:48
amoralejjruzicka, number80 could you add openstack-sig group to https://src.fedoraproject.org/rpms/python-heatclient ?10:48
number80amoralej: I'll do10:49
number80any reason?10:49
number80oh I can't10:50
number80jruzicka: ^10:50
number80ykarel: FYI, I received your email, I'll answer asap10:52
amoralejnumber80, i have a patch to remove python210:52
ykarelnumber80, ack10:52
* number80 is on a phone call and I am old10:52
amoralejand i think the agreement was to add all openstack packages to openstack-sig, isn't it?10:52
number80(and I need my full attention to give proper answer :o) )10:52
number80amoralej: yes, if you have patch and need to push it asap, I can push it for you10:53
number80But only pagure admins can add people10:53
number80which are ryansb, jruzicka and apevec for heatclient10:53
amoralejno, no problem10:53
amoraleji can wait10:53
number80:)10:53
amoralejhttps://src.fedoraproject.org/rpms/python-designateclient is also missing openstack-sig group10:53
amoralejvkmc, ^ could you add openstack-sig group as admin too?10:54
*** shardy has quit IRC10:55
jruzickaamoralej, number80 admin granted on openstack-sig10:57
*** rpittau has quit IRC10:57
*** rpittau_ has joined #rdo10:57
amoralejthanks jruzicka10:57
jruzickasure thing ;)10:57
jruzickathanks amoralej for taking care of the packages ;)10:58
amoralejno prob10:59
rdogerritMerged rdo-jobs master: Add Rocky to tripleo-ceph-integration job  https://review.rdoproject.org/r/1764011:01
*** rpittau_ has quit IRC11:03
*** rpittau_ has joined #rdo11:03
number80jruzicka: you rock :)11:04
* jruzicka pressed the button real good :D11:05
*** rpittau_ has quit IRC11:09
*** rpittau__ has joined #rdo11:09
rdogerritJavier Peña created rdo-infra/rdo-infra-playbooks master: [WIP] Add sf.io backup to the backup role  https://review.rdoproject.org/r/1764211:10
rdogerritDaniel Alvarez proposed rdoinfo master: Bump OVS package to 2.9.3 for Queens  https://review.rdoproject.org/r/1383911:11
*** morazi has quit IRC11:11
*** rpittau__ has quit IRC11:14
*** rpittau_ has joined #rdo11:14
*** sshnaidm has quit IRC11:16
*** ttsiouts has quit IRC11:16
*** ktibi has quit IRC11:16
*** sshnaidm has joined #rdo11:16
*** ttsiouts has joined #rdo11:17
rdogerritMerged config master: Add Rocky to ceph-ansible job  https://review.rdoproject.org/r/1764111:17
*** Son_Goku has joined #rdo11:18
*** sshnaidm has quit IRC11:18
*** rfolco has joined #rdo11:18
*** sshnaidm has joined #rdo11:19
*** ttsiouts has quit IRC11:20
*** ttsiouts has joined #rdo11:20
*** quiquell is now known as quiquell|brb11:21
*** Son_Goku has quit IRC11:21
*** owalsh_ has quit IRC11:24
*** owalsh has joined #rdo11:24
*** jpich has quit IRC11:25
*** ksambor has quit IRC11:25
*** jpich has joined #rdo11:26
*** hamzy_ has joined #rdo11:42
*** hamzy has quit IRC11:42
*** ttsiouts_ has joined #rdo11:43
*** quiquell|brb is now known as quiquell11:45
*** ttsiouts has quit IRC11:45
*** rpittau_ has quit IRC11:48
ykareljpena, tristanC seeing 2018-12-03 11:40:03.943960 | primary | MODULE FAILURE11:49
ykareland RETRY_LIMIT11:49
jpenaykarel: in that example, can you find the zuul executor name? It's at the beginning of the log11:50
jpenafbo: ^^11:50
ykareljpena, https://logs.rdoproject.org/39/13839/11/check/legacy-rdoinfo-tripleo-master-testing-centos-7-multinode-1ctlr-featureset016/1777609/job-output.txt.gz11:50
ykarel Executor: ze05.softwarefactory-project.io11:51
jpena2018-12-03 11:40:03,788 DEBUG zuul.AnsibleJob: [build: 1777609b46f446dcb3dd849206e76447] Ansible output: b"An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: [Errno 2] No such file or directory: '/home/zuul/workspace'"11:51
jpenao_O11:52
tristanCjpena: that is odd indeed :) did the image change or something?11:55
*** fbo has joined #rdo11:55
jpenatristanC: dunno, but it's possible. I've seen a strange error in a fedora image too, where zuul-cloner did not have the right permissions11:56
tristanCjpena: since the upgrade?11:56
jpenayep11:56
tristanCansible got upgraded from 2.5.5 to 2.5.1111:57
jpenawe have https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-rdo-base/pre.yaml#L72-L95 in the tripleo jobs11:58
jpenaI suspect something has changed in zuul_legacy_vars11:58
jpenaeither that, or ansible_user_dir11:58
jpenatristanC: did that change in the zuul update?11:59
tristanCjpena: that sounds like a good lead...11:59
tristanCzuul got upgraded from 3.2.0 to 3.3.111:59
tristanCwe can work on a downgrade playbook to rollback if the issue can't be fixed easily12:00
*** rpittau has joined #rdo12:00
tristanClet me check git logs12:00
jpenaok, I think the issue is in zuul-cloner too12:01
jpenahttps://logs.rdoproject.org/39/13839/11/check/legacy-rdoinfo-tripleo-master-testing-centos-7-multinode-1ctlr-featureset016/1777609/ara-report/result/cfe500cb-091c-48c0-a0dc-6a87c200cae3/12:01
jpenawhat that task does is to run zuul-cloner12:01
jpenathat seems to cause everything else12:02
*** ykarel is now known as ykarel|afk12:03
*** owalsh has quit IRC12:03
*** ksambor has joined #rdo12:04
*** ttsiouts_ has quit IRC12:05
*** vnogin has joined #rdo12:05
*** ttsiouts has joined #rdo12:05
tristanCthe zuul_legacy_vars filter doesn't seems to have changed12:08
jpenaI'd bet for zuul-cloner12:09
jpenafor some reason, the prepare-workspace role expects it to be there (even before we run fetch-zuul-cloner-fork), but it's not12:09
*** shardy has joined #rdo12:09
*** ttsiouts has quit IRC12:10
*** ykarel|afk is now known as ykarel12:11
rdogerritGrzegorz Grasza created openstack/novajoin-distgit rpm-master: Change novajoin-ipa-setup and novajoin-install directory  https://review.rdoproject.org/r/1764312:15
*** ktibi has joined #rdo12:15
tristanCjpena: it seems like some jobs are working fine, what is affected?12:16
jpenahttps://logs.rdoproject.org/42/17642/1/check/legacy-rdo-infra-playbooks-integration-sensu-client-fedora/b89194f/ara-report/ is an example12:17
jpenaanother one is https://logs.rdoproject.org/39/13839/11/check/legacy-rdoinfo-tripleo-master-testing-centos-7-multinode-1ctlr-featureset016/1777609/ara-report/12:17
*** dprince has joined #rdo12:18
*** Son_Goku has joined #rdo12:19
*** ykarel is now known as ykarel|afk12:19
*** owalsh has joined #rdo12:21
ssbarnea|roverjpena: one question if you do not mind, i tried to find the error mentioned on http://paste.openstack.org/raw/736569/ on kibana, but I unable to find it there. any idea why?12:24
ssbarnea|roverhttps://review.rdoproject.org/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-7d,mode:quick,to:now))&_a=(columns:!(_source),index:'logstash-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'message:%22failed%20to%20reach%20SAVING%20state%22')),sort:!('@timestamp',desc))12:24
jpenassbarnea|rover: I see that error is in the tempest.log.gz file, maybe not everything is uploaded to logstash12:29
jpenayou should check the role to see what gets uploaded12:30
*** ykarel|afk has quit IRC12:30
rdogerritTristanCacqueray created rdo-jobs master: fetch-zuul-cloner: ensure env permission  https://review.rdoproject.org/r/1764412:31
*** ttsiouts has joined #rdo12:34
*** jpena is now known as jpena|lunch12:34
rdogerritMerged rdo-jobs master: fetch-zuul-cloner: ensure env permission  https://review.rdoproject.org/r/1764412:36
*** vnogin has quit IRC12:39
*** dhill_ has joined #rdo12:40
tristanCjpena|lunch: fbo: it's getting late here. If the issues can't be fixed soon, then here is a playbook to downgrade zuul version back to before the upgrade: https://review.rdoproject.org/paste/show/169/12:42
rdogerritFabien Boucher created rdo-jobs master: Fix missing use of item in the with_items  https://review.rdoproject.org/r/1764512:48
*** ramishra has quit IRC12:48
*** artom has joined #rdo12:49
*** morazi has joined #rdo12:51
*** rlandy has joined #rdo12:54
vkmcamoralej, hi, wrt https://bugzilla.redhat.com/show_bug.cgi?id=163454912:57
openstackbugzilla.redhat.com bug 1634549 in python-designateclient "python-designateclient: Remove (sub)packages from Fedora 30+: python2-designateclient, python2-designateclient-tests" [Unspecified,Modified] - Assigned to vimartin12:57
vkmcamoralej, what are we doing in other clients?12:57
vkmcI can do it12:57
vkmcit's quite confusing the different processes we are following for rdo and fedora, not entirely sure what is missing there12:57
amoralejvkmc, i'll send a PR12:58
vkmcI want to learn12:58
vkmcbut sure12:58
amoralejso that you can see12:58
amoralejlet me show some example12:58
rdogerritFélix Enrique Llorente Pastora proposed rdo-infra/ci-config master: Add ci reproducer using zuul quickstart  https://review.rdoproject.org/r/1755213:00
amoralejvkmc, https://src.fedoraproject.org/rpms/python-os-client-config/c/9bf267c927de12f7f3c043667b2145bea0511779?branch=master13:02
amoralejthis is what we are doing right now13:02
amoralejafter stein GA13:02
amoralejwe will use existing approach in RDO13:03
amoralejwhich is different13:03
vkmcall right, checking13:03
vkmcthanks amoralej13:03
rdogerritFélix Enrique Llorente Pastora proposed rdo-infra/ci-config master: Add ci reproducer using zuul quickstart  https://review.rdoproject.org/r/1755213:04
*** ramishra has joined #rdo13:04
*** rh-jelabarre has joined #rdo13:06
amoralejvkmc, https://src.fedoraproject.org/rpms/python-designateclient/pull-request/213:09
amoralejfeel free if you prefer to push your own patch13:09
*** holser_ has quit IRC13:13
*** dneary has joined #rdo13:13
*** fkluknav has quit IRC13:16
*** fkluknav has joined #rdo13:17
*** ykarel|afk has joined #rdo13:18
*** NuxRo has quit IRC13:18
*** NuxRo has joined #rdo13:18
*** ykarel|afk is now known as ykarel13:19
ykareljpena|lunch, still failing http://logs.openstack.org/67/621467/4/check/packstack-multinode-scenario002-tempest/7e9a062/logs/latest/manifests/198.72.124.144_compute.pp.log.txt.gz13:19
*** mcornea has joined #rdo13:21
amoralejykarel, hiera is fine but puppet apply is still looking up name with dots13:21
*** amoralej is now known as amoralej|lunch13:22
ykarelamoralej|lunch, hmm in defaults.yml it's populated correctly:- FIREWALL_NEUTRON_TUNNEL_RULES_198_72_124_13913:22
ykarellooking if i can find the link13:22
amoralej|lunchyeah, but [0;36mDebug: Lookup of 'FIREWALL_NEUTRON_TUNNEL_RULES_198.72.124.144'13:23
amoralej|lunchhttp://logs.openstack.org/67/621467/4/check/packstack-multinode-scenario002-tempest/7e9a062/logs/latest/manifests/198.72.124.144_compute.pp.log.txt.gz13:23
*** agopi has quit IRC13:24
*** jpena|lunch is now known as jpena13:26
ykarelyes it' $my_ip = choose_my_ip(hiera('HOST_LIST'))13:28
ykarel    $neutron_tunnel_rule_name = "FIREWALL_NEUTRON_TUNNEL_RULES_${my_ip}"13:28
ykareljpena, ^^ similar also needs to be taken care13:28
*** dneary has quit IRC13:29
*** agopi has joined #rdo13:30
*** dneary has joined #rdo13:30
*** udesale has joined #rdo13:30
ssbarnea|roverjpena: regarding tempest log no being indexed I don't know if we have similar code as upstream but I remember clarb telling me that only files with .txt extensions were uploaded or something like this. not sure if it helps.13:32
ssbarnea|roverstill, the problem is important because it prevents us from searching for common issues. (obviously less important than a rdo downtime)13:33
rdogerritJavier Peña created rdo-jobs master: Fix fetch-zuul-cloner-fork  https://review.rdoproject.org/r/1764613:35
*** zul has joined #rdo13:35
*** imcleod has joined #rdo13:40
*** holser_ has joined #rdo13:40
*** holser_ has quit IRC13:44
*** holser_ has joined #rdo13:44
rdogerritMerged rdo-jobs master: Fix fetch-zuul-cloner-fork  https://review.rdoproject.org/r/1764613:45
jpenaykarel: I've rechecked the rdoinfo commit, it looks like it is passing now13:47
jpenaI wonder if the issue we saw was related to the zuul executors being restarted13:47
*** jaosorior has joined #rdo13:47
ykareljpena, which patch u rechecked13:48
jpenaykarel: https://review.rdoproject.org/r/1383913:48
ykareljpena, i was seeing issue in ansible one13:48
ykarelbut yes jobs are same13:49
jpenaykarel: which one?13:49
ykarelso this also ok13:49
jpenalet's give it a try, too13:49
ykareljpena, https://review.rdoproject.org/r/#/c/14960/13:49
rdogerritJavier Peña proposed rdoinfo master: deps: bump Ansible to 2.6.6 for Queens, Stein and Rocky  https://review.rdoproject.org/r/1496013:49
ykarelack let's see13:49
rdogerritJavier Peña created rdo-jobs master: Second fix to fetch-zuul-cloner-fork  https://review.rdoproject.org/r/1764713:53
*** rpittau_ has joined #rdo13:56
ykareljpena, should we give try to regsubst for substituting . with _ in FIREWALL_NOVA_QEMU_MIG_RULES and FIREWALL_NEUTRON_TUNNEL_RULES13:56
rdogerritMerged rdo-jobs master: Second fix to fetch-zuul-cloner-fork  https://review.rdoproject.org/r/1764713:56
ykarelas hiera is overridden but puppet lookup is failing13:56
*** kushalAgrawal has joined #rdo13:58
*** rpittau has quit IRC13:59
jpenaykarel: yes, let's go for it13:59
jpenaare you going to do the change, or do you prefer me to do it?14:00
ykareljpena, okk pushing patch14:00
ykareljpena, if u would like go for it14:00
jpenaykarel: it's ok, I'm looking after some other stuff :)14:00
*** amoralej|lunch is now known as amoralej14:00
ykareljpena, ack /me will take care14:00
*** quiquell is now known as quiquell|lunch14:01
*** jcoufal has joined #rdo14:02
*** kgiusti has joined #rdo14:03
*** jcoufal has quit IRC14:07
*** rpittau_ is now known as rpittau14:07
rdogerritFabien Boucher proposed rdo-infra/rdo-infra-playbooks master: Add sf.io and koji.sf.io backup to the backup role  https://review.rdoproject.org/r/1764214:09
openstackgerrityatin proposed openstack/packstack master: Fix Hiera keys with a dot  https://review.openstack.org/62146714:10
ykareljpena, ^^14:10
*** jcoufal has joined #rdo14:11
rdogerritSagi Shnaidman proposed rdo-jobs master: WIP: Add OVB role to rdo-jobs repo for experiments  https://review.rdoproject.org/r/1740414:14
*** jcoufal has quit IRC14:17
*** jcoufal has joined #rdo14:19
*** dprince has quit IRC14:20
rdogerritSagi Shnaidman proposed rdo-jobs master: WIP: Add OVB role to rdo-jobs repo for experiments  https://review.rdoproject.org/r/1740414:21
*** dprince has joined #rdo14:22
rdogerritTim Rozet proposed rdoinfo master: Fixes puppet-opendaylight versions  https://review.rdoproject.org/r/1759214:25
rdogerritNicolas HICHER proposed rdo-infra/rdo-infra-playbooks master: afs hosts: add mirror.ca-ymq-1.vexxhost-rdo host  https://review.rdoproject.org/r/1762714:25
*** jmlowe has quit IRC14:25
*** jmlowe has joined #rdo14:26
rdogerritJavier Peña created rdo-jobs master: Complete fix for venv permissions in fetch-zuul-cloner-fork  https://review.rdoproject.org/r/1764814:26
*** trozet has joined #rdo14:29
*** leanderthal has joined #rdo14:29
rdogerritJavier Peña proposed rdo-jobs master: Complete fix for venv permissions in fetch-zuul-cloner-fork  https://review.rdoproject.org/r/1764814:30
trozetamoralej: could you please review https://review.rdoproject.org/r/#/c/17592/ again when you have a minute?14:30
*** ktibi has quit IRC14:37
*** quiquell|lunch is now known as quiquell14:40
*** ktibi has joined #rdo14:41
jpenassbarnea|rover: going back to the issue, the definition of which files are submitted to logstash in review.rdo is in https://github.com/rdo-infra/rdo-jobs/blob/master/roles/submit-logstash-jobs/defaults/main.yaml14:41
jpenawe have limited storage capacity, so we may want to pick the files we want to upload. Currently, only job-output.txt is uploaded14:42
*** jpich has quit IRC14:42
ssbarnea|roverjpena: thanks, i will make a CR to add tempest which is likely small, so it will match upstream config and allow us to search errors.14:43
jpenassbarnea|rover: ack14:43
ssbarnea|roverjpena: i also found that kibana can be configured to query multiple clusters which means that we should be able to use one kibana to query both elasticsearch clusters.14:44
ykareljpena, /bin/sh: line 44: /usr/zuul-env/bin/zuul-cloner: Permission denied is also be fixed, right14:45
ykarelseen in https://review.rdoproject.org/r/#/c/17085/2, just rechecked to see14:46
*** ktibi has quit IRC14:48
*** jpich has joined #rdo14:50
jpenaykarel: that's happening in the fedora jobs, it's related to the umask I think14:51
jpenahttps://review.rdoproject.org/r/17648 is the last of a series of fixes14:51
ykarelokk so not fixed yet14:52
rdogerritMerged rdo-jobs master: Complete fix for venv permissions in fetch-zuul-cloner-fork  https://review.rdoproject.org/r/1764814:54
ykareljpena, recheck failed with different error14:55
ykarelImportError: No module named yaml14:55
ykarelhttps://logs.rdoproject.org/85/17085/2/check/legacy-DLRN-rpmbuild-fedora/eded215/job-output.txt.gz#_2018-12-03_14_46_54_43269014:55
jpenaykarel: yep14:55
rdogerritAlex Schultz proposed openstack/ansible-pacemaker-distgit rpm-master: Python3 ansible-pacemaker  https://review.rdoproject.org/r/1763014:56
rdogerritSorin Sbarnea created rdo-jobs master: Added logstash match pattern for tempest log  https://review.rdoproject.org/r/1764914:56
rdogerritMerged rdoinfo master: Fixes puppet-opendaylight versions  https://review.rdoproject.org/r/1759214:56
jpenaykarel: it should work now, I've rechecked15:00
ykareljpena, Thanks will keep an eye15:00
*** beekneemech is now known as bnemec15:01
*** baha has joined #rdo15:03
rdogerritSagi Shnaidman proposed rdo-jobs master: WIP: Add OVB role to rdo-jobs repo for experiments  https://review.rdoproject.org/r/1740415:10
*** dneary has quit IRC15:20
*** dneary has joined #rdo15:20
rdogerritYatin Karel created rdo-infra/fedora-stable-config master: Include ceph packages  https://review.rdoproject.org/r/1765015:25
ykarelamoralej, ^^15:25
amoralejack15:25
*** Son_Goku has quit IRC15:26
*** ksambor has quit IRC15:28
rdogerritYatin Karel created rdo-infra/fedora-stable-config master: Add requires for oswin-tempest-plugin  https://review.rdoproject.org/r/1765115:28
ykarelamoralej, also need python3-winrm for oswin-tempest-plugin ^^15:29
amoralejups15:29
amoralejwe only added python2?15:29
ykarelamoralej, for what?15:30
amoralejwinrm?15:30
amoralejdidn't we add it last week?15:30
ykarelamoralej, that was for stein-testing15:30
amoralejoh, right15:30
ykarelin centos stein-testing rocky-testing wre missing15:30
amoralejETOOMANYTHINGSINMYHEAD15:30
ykarel:)15:30
*** moguimar has quit IRC15:30
ykarelamoralej, jpena can u also review -distgit patches for py3: https://review.rdoproject.org/r/#/q/topic:py3-enable15:30
*** dpawlik_ has quit IRC15:31
ykarelno hurry, whenever u have some bandwidth :)15:31
ykarelupstream zuul queue is big, jobs not started for about 1.5 hr15:33
*** dtrainor has joined #rdo15:34
*** Vorrtex has joined #rdo15:40
sfbenderMerged distroinfo master: Fix filter python3 compatibility  https://softwarefactory-project.io/r/1445515:41
*** Vorrtex has quit IRC15:43
*** Vorrtex has joined #rdo15:43
*** ykarel has quit IRC15:53
*** ykarel has joined #rdo15:53
*** rdogerrit has quit IRC15:54
*** lennyb_ has quit IRC15:55
*** morazi has quit IRC15:56
*** morazi has joined #rdo15:56
*** lennyb has joined #rdo15:57
*** quiquell is now known as quiquell|off15:58
*** janki has joined #rdo15:58
*** rdogerrit has joined #rdo16:02
rdogerritFélix Enrique Llorente Pastora proposed rdo-infra/ci-config master: Add ci reproducer using zuul quickstart  https://review.rdoproject.org/r/1755216:02
*** ttsiouts has quit IRC16:03
*** ttsiouts has joined #rdo16:03
rdogerritJavier Peña proposed config master: Make ansible-role-rdo-kolla-build use the rdo-infra acls  https://review.rdoproject.org/r/1417116:03
rdogerritFélix Enrique Llorente Pastora proposed rdo-infra/ci-config master: Add ci reproducer using zuul quickstart  https://review.rdoproject.org/r/1755216:04
ykareljpena, also seeing issues in fedora stable jobs:  cp: failed to access '/home/fedora-master/data/repos/stable-base/latest': Permission denied16:04
ykarelhttps://logs.rdoproject.org/51/17651/1/check/legacy-fedora-stable-repoclosure/e287472/job-output.txt.gz16:04
*** jmlowe has quit IRC16:04
jpenaykarel: yes, we're having similar umask issues16:04
ykareljpena, ack so u are already looking into it16:05
ykareland other is http://mirror.centos.org/centos/7/configmanagement/x86_64/yum4/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found16:05
*** Son_Goku has joined #rdo16:06
ykarellooks like yum4 is removed from there16:06
ykareli only see ansible26 http://mirror.centos.org/centos/7/configmanagement/x86_64/16:06
ykarelokk looks like centos 7.6 released16:07
jpenawe'll need to fix https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/legacy/fedora-stable-repoclosure/run.yaml#L91-L9616:07
*** ttsiouts has quit IRC16:08
rdogerritFabien Boucher proposed rdo-infra/rdo-infra-playbooks master: Add sf.io and koji.sf.io backup to the backup role  https://review.rdoproject.org/r/1764216:09
*** dpawlik has joined #rdo16:11
*** dpawlik has quit IRC16:16
*** dtantsur is now known as dtantsur|afk16:17
*** dpawlik has joined #rdo16:20
rdogerritFabien Boucher proposed rdo-infra/rdo-infra-playbooks master: Add sf.io and koji.sf.io backup to the backup role  https://review.rdoproject.org/r/1764216:25
*** janki has quit IRC16:26
rdogerritFabien Boucher proposed rdo-infra/rdo-infra-playbooks master: Add sf.io and koji.sf.io backup to the backup role  https://review.rdoproject.org/r/1764216:27
rdogerritFabien Boucher proposed rdo-infra/rdo-infra-playbooks master: Add sf.io and koji.sf.io backup to the backup role  https://review.rdoproject.org/r/1764216:29
*** leanderthal is now known as leanderthal|afk16:30
*** kushalAgrawal has quit IRC16:33
rdogerritFabien Boucher proposed rdo-infra/rdo-infra-playbooks master: Add sf.io and koji.sf.io backup to the backup role  https://review.rdoproject.org/r/1764216:35
rdogerritrdo-trunk created openstack/cinder-distgit rocky-rdo: openstack-cinder-13.0.2-1  https://review.rdoproject.org/r/1765216:37
*** trown is now known as trown|lunch16:41
*** dpawlik has quit IRC16:42
rdogerritMerged config master: Support "True" and "true" when checking in dlrnapi_report.sh  https://review.rdoproject.org/r/1763916:45
*** ccamposr has quit IRC16:46
*** Son_Goku has quit IRC16:48
rdogerritrdo-trunk created openstack/horizon-distgit rocky-rdo: python-django-horizon-14.0.2-1  https://review.rdoproject.org/r/1765316:50
*** dpawlik has joined #rdo16:52
rdogerritUser mjturek created rdo-infra/ci-config master: Add scm for ci-config to tripleo-upstream jobs  https://review.rdoproject.org/r/1765416:54
*** social has quit IRC16:55
*** cylopez has quit IRC17:01
*** udesale has quit IRC17:02
rdogerritJavier Peña created rdo-jobs master: Ensure OUTPUT_DIR is accessible for fedora-stable jobs  https://review.rdoproject.org/r/1765517:04
rdogerritJavier Peña proposed rdo-infra/fedora-stable-config master: Add requires for oswin-tempest-plugin  https://review.rdoproject.org/r/1765117:05
*** paramite has quit IRC17:09
*** baha has quit IRC17:13
*** aufi has quit IRC17:13
*** frippe75 has joined #rdo17:16
*** dpawlik has quit IRC17:19
frippe75amoralej: Just to report back since you put in a lot of effort. I somehow got ironic installed in the process. This was causing the compute/hypervisor to not get pulled in in the allinone install. Thought an allinone with ironic meant the controller services would get installed only.17:19
amoralejok, then you were enabling ironic?17:20
frippe75yes17:22
frippe75I meant to enable sahara/magnum.. But I must have enabled ironic in the same process.17:22
rdogerritJavier Peña proposed rdo-jobs master: Ensure OUTPUT_DIR is accessible for fedora-stable jobs, fix yum4  https://review.rdoproject.org/r/1765517:22
*** jlibosva has quit IRC17:23
openstackgerritAlfredo Moralejo proposed openstack/packstack master: Use libvirt_inject_partition default value  https://review.openstack.org/62164817:24
frippe75But now I'm failing to understand what CONFIG_IRONIC_INSTALL=y  means. In an allinone I would expect ironic-api and conductor to be installed but not "client side ironic"17:24
*** masonjoyers has joined #rdo17:24
frippe75But maybe leave this out since i'm not planning on using ironic at this stage.17:24
*** jpich has quit IRC17:25
*** rpittau has quit IRC17:26
openstackgerritAlfredo Moralejo proposed openstack/packstack stable/rocky: Revert "Temporary switch to ovs in scenario 001, fix gate"  https://review.openstack.org/62165117:26
*** hseipp has quit IRC17:26
openstackgerritAlfredo Moralejo proposed openstack/packstack stable/queens: Revert "Temporary switch to ovs in scenario 001, fix gate"  https://review.openstack.org/62165217:28
*** Son_Goku has joined #rdo17:31
*** baha has joined #rdo17:32
*** dpawlik has joined #rdo17:35
*** dpawlik has quit IRC17:39
*** kopecmartin is now known as kopecmartin|off17:40
rdogerritJavier Peña proposed rdo-jobs master: Ensure OUTPUT_DIR is accessible for fedora-stable jobs, fix yum4  https://review.rdoproject.org/r/1765517:40
*** Son_Goku has quit IRC17:42
frippe75Question about magnum error deployed using packstack. Cannot list clusters or stats is reported in Horizon. magnum-api reveals keystone trustee issues. Traced it to trustee_domain_id = admin_osc.keystone().trustee_domain_id17:42
frippe75fails here due to trustee_domain_id not being part of the dict returned from keystone()17:43
frippe75Also found a thread about auth_uri being depricated in favor of www_authenticate_uri. Tried to make those changes but still fails.17:45
*** shardy has quit IRC17:46
frippe75none of the test scenarious seems to cover magnum, correct?17:46
jpenafrippe75: that looks similar to https://bugs.launchpad.net/packstack/+bug/1801913, which was recently fixed. We don't have any tempest tests for magnum in the gate, so there could be some more issues17:48
openstackLaunchpad bug 1801913 in Packstack "magnum does not work in Rocky" [High,Fix released] - Assigned to Javier Peña (jpena-c)17:48
openstackgerritJavier Peña proposed openstack/packstack stable/rocky: Replace deprecated puppet-magnum parameters  https://review.openstack.org/62166217:49
frippe75jpena: Ok, think I found that one and tried to manually "patch" it. only being a single line. But it still does not work. Getting the same error restarting magnum-api and conductor.17:53
*** Son_Goku has joined #rdo17:56
*** vnogin has joined #rdo17:56
frippe75jpena: the install guide recommends creating the  magnum_domain_admin user but packstack opted for magnum_admin which I guess is fine since trustee_domain_admin_name is set to magnum_admin.17:56
jpenafrippe75: did you add /v3 to the www_authenticate_uri parameter in magnum.conf?17:57
frippe75jpena: tried both ways since there seems to be a conflict around that setting.17:58
frippe75with and without17:58
jpenaat least with that, magnum cluster-list worked for me17:58
*** derekh has quit IRC18:02
frippe75jpena: does not for me with /v3 added..18:03
jpenafrippe75: you restarted both conductor and api after the change, didn't you?18:03
frippe75sure18:03
frippe75jpena: checked time on both. they are restarted18:04
frippe75jpena: there is an option to set trustee_domain_id explicitly (as i interpret it). Maybe list the magnum domain id and set it... not sure18:10
*** jpena is now known as jpena|off18:17
*** apetrich has quit IRC18:18
*** dsneddon has joined #rdo18:19
*** ttsiouts has joined #rdo18:21
openstackgerritMerged openstack/packstack master: Fix Hiera keys with a dot  https://review.openstack.org/62146718:24
*** amoralej is now known as amoralej|off18:24
*** vnogin has quit IRC18:25
*** udesale has joined #rdo18:26
*** vnogin has joined #rdo18:26
*** ttsiouts has quit IRC18:28
*** avivgta has joined #rdo18:28
*** avivgt has quit IRC18:30
*** apetrich has joined #rdo18:30
*** ykarel is now known as ykarel|away18:36
*** jmlowe has joined #rdo18:36
*** avivgta is now known as avivgt18:38
*** jmlowe has quit IRC18:39
*** holser_ has quit IRC18:40
sfbenderMerged DLRN master: Make mock_base_packages a config option for Koji driver  https://softwarefactory-project.io/r/1426918:43
*** Caterpillar has quit IRC18:45
*** dprince has quit IRC18:47
*** ykarel|away has quit IRC18:47
*** wolverineav has joined #rdo18:57
*** trown|lunch is now known as trown|outtypewww18:59
*** trozet has quit IRC19:00
*** wolverineav has quit IRC19:01
*** wolverineav has joined #rdo19:01
*** Son_Goku has quit IRC19:02
*** jmlowe has joined #rdo19:07
*** dprince has joined #rdo19:09
*** dneary has quit IRC19:14
*** dneary has joined #rdo19:16
*** shardy has joined #rdo19:17
*** morazi has quit IRC19:17
rdogerritrdo-trunk created openstack/designate-dashboard-distgit queens-rdo: openstack-designate-ui-6.0.1-1  https://review.rdoproject.org/r/1765619:23
*** wolverineav has quit IRC19:27
*** wolverineav has joined #rdo19:28
*** wolverineav has quit IRC19:32
*** wolverineav has joined #rdo19:32
*** gfidente is now known as gfidente|afk19:43
*** shardy has quit IRC19:48
*** wolverineav has quit IRC19:57
*** wolverineav has joined #rdo19:58
*** vnogin has quit IRC20:02
*** wolverineav has quit IRC20:03
*** jmlowe has quit IRC20:23
rdogerritUser mjturek created rdo-infra/ci-config master: Create job for building tripleo images with cico resources  https://review.rdoproject.org/r/1765720:24
*** udesale has quit IRC20:30
*** wolverineav has joined #rdo20:32
*** jmlowe has joined #rdo20:40
*** jmlowe has quit IRC20:41
*** holser_ has joined #rdo20:44
*** morazi has joined #rdo20:56
*** apetrich has quit IRC20:58
*** udesale has joined #rdo20:58
sfbenderMerged distroinfo master: tests: use the zuul-jobs tox  https://softwarefactory-project.io/r/1445821:05
*** apetrich has joined #rdo21:18
*** kgiusti has left #rdo21:22
*** udesale has quit IRC21:23
*** dprince has quit IRC21:32
*** dsneddon has quit IRC21:32
*** jfrancoa has quit IRC21:37
*** jcoufal has quit IRC21:37
*** holser_ has quit IRC21:39
*** amoralej|off is now known as amoralej21:46
*** amoralej is now known as amoralej|off21:47
*** wolverineav has quit IRC21:56
*** Vorrtex has quit IRC22:06
*** radeks_ has quit IRC22:09
*** jaosorior has quit IRC22:13
*** baha has quit IRC22:15
*** dsneddon has joined #rdo22:16
*** jaosorior has joined #rdo22:17
*** rcernin has joined #rdo22:18
*** pcaruana has quit IRC22:18
*** udesale has joined #rdo22:23
*** udesale has quit IRC22:25
*** wolverineav has joined #rdo22:25
*** udesale has joined #rdo22:26
rdogerritMerged openstack/os-collect-config-distgit pike-rdo: Restart service on SIGPIPE  https://review.rdoproject.org/r/1756122:26
rdogerritBen Nemec created openstack/os-collect-config-distgit ocata-rdo: Restart service on SIGPIPE  https://review.rdoproject.org/r/1765822:27
*** ramishra has quit IRC22:28
*** bdperkin has quit IRC22:37
*** udesale has quit IRC22:46
*** irclogbot_3 has quit IRC22:47
rdogerritMerged openstack/os-collect-config-distgit ocata-rdo: Restart service on SIGPIPE  https://review.rdoproject.org/r/1765822:47
*** jaosorior has quit IRC22:53
*** rh-jelabarre has quit IRC22:54
*** rh-jelabarre has joined #rdo22:57
*** rh-jelabarre has quit IRC23:02
*** irclogbot_3 has joined #rdo23:09
*** jtomasek has quit IRC23:10
sfbenderMerged distroinfo master: tests: make integration environment python version aware  https://softwarefactory-project.io/r/1446023:47
*** gkadam has quit IRC23:48
sfbenderMerged distroinfo master: tests: use common nodesets and remove os version number  https://softwarefactory-project.io/r/1446423:49

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!