Friday, 2020-03-20

*** jmasud has quit IRC00:53
*** dsneddon_ has joined #oooq00:53
*** tosky has quit IRC00:54
*** jmasud has joined #oooq00:56
*** dsneddon_ has quit IRC01:00
*** rlandy has quit IRC01:41
*** dsneddon_ has joined #oooq01:51
*** dsneddon_ is now known as dxstest01:51
*** dxstest has quit IRC01:51
*** rlandy has joined #oooq02:03
rlandybeagles: hi ... was just writing you an email ...02:04
rlandyhttps://review.rdoproject.org/r/#/c/25325/ tetsproject job passed with https://review.opendev.org/#/c/713960/ depends-on change so the network component promoted.02:04
rlandythis may or may not have been intended02:04
*** rlandy has quit IRC02:08
*** skramaja has joined #oooq03:55
*** raukadah is now known as chandankumar04:24
*** ykarel|away is now known as ykarel04:27
*** ratailor has joined #oooq04:28
*** udesale has joined #oooq05:38
*** marios has joined #oooq06:09
*** ykarel is now known as ykarel|afk06:21
*** pojadhav is now known as pojadhav|lunch07:21
*** dpawlik has joined #oooq07:21
*** marios has quit IRC07:25
*** marios has joined #oooq07:25
*** bogdando has joined #oooq07:46
*** pojadhav|lunch is now known as pojadhav08:04
*** ccamacho has quit IRC08:08
*** jpena|off is now known as jpena08:12
zbrhttps://review.opendev.org/#/c/713963/08:16
*** jbadiapa has joined #oooq08:16
*** ccamacho has joined #oooq08:41
*** ccamacho has quit IRC08:45
*** jfrancoa has joined #oooq08:46
*** ratailor has quit IRC08:48
*** dtantsur|afk is now known as dtantsur08:50
*** ratailor has joined #oooq08:50
*** tosky has joined #oooq08:51
*** holser has joined #oooq08:53
*** dtantsur is now known as dtantsur|learnin08:58
pandazbr:  https://review.rdoproject.org/r/2602109:10
zbrdone09:11
*** apetrich has joined #oooq09:13
pandazbr: what about now (It was really meant for that job in the first place, but it's too early for my brain)09:14
*** ykarel|afk is now known as ykarel09:14
mariosykarel: o/ hey can you help with getting hold on a node? running some component tempest jobs there https://review.rdoproject.org/r/#/c/26020/ ... do we have to do it before i start the run ?09:15
marioschandankumar: ^ fyi09:15
ykarelmarios, yes can do09:15
ykarelbefore job fails hold should work09:16
ykarelmarios, which job?09:16
ykarelperiodic-tripleo-ci-centos-8-standalone-full-tempest-tempest-master or periodic-tripleo-ci-centos-8-standalone-full-tempest-compute-master?09:17
mariosykarel: well not sure yet either or both ;)09:17
mariosykarel: thansk very much! just pick one if you have to09:17
ykarelok will do tempest one09:17
mariosykarel: thans09:17
mariosthanks09:17
ykarelmarios, your public key09:20
mariosykarel: https://github.com/marios.keys09:20
chandankumarmarios, ok09:21
ykarelmarios, ssh zuul@38.145.34.6709:21
ykarelif jobs fails, node gets hold09:21
ykarelu can try ssh now09:21
zbrpanda: this would clearly not break but i still have some doubts regarding how we scale it, lets talk about it in few minutes.09:22
mariosykarel: thanks yes works09:22
zbri need to visit my garage, supermicro got offline over the night and ipmi seems to misbehave09:23
pandazbr: aren't you on you PTO today ?09:25
mariostripleo-ci* needs reviews please when you have time thanks https://review.opendev.org/#/c/713659/09:36
zbryep...09:38
mariosif you aren't using your pto zbr can i have it please?09:38
* marios hands over keyboard09:39
marioskthx09:39
zbrpanda: i am back, if you want to meet, let me know.09:41
zbrbasically i am inclined to believe that we do not need/want more than one tox env for each molecule job, and that is would be easier if we just reuse existing one with different parameters09:42
chandankumarkopecmartin, soniya29, arxcruz|rover, weshay|ruck skiplist proposal drafted https://hackmd.io/twHmV2f2SGCMvfHjLMoGoA?view09:43
zbrbecause multiple-envs does not scale well, think about tripleo-ansible repo, which has 78+ scenarios, and job for each of them.09:43
soniya29chandankumar, ack09:45
chandankumarsoniya29, kopecmartin arxcruz|rover today's tempest meeting agenda https://hackmd.io/fIOKlEBHQfeTZjZmrUaEYQ?view#2020-03-20 please add yours09:52
pandazbr: well, I think you suggestes to have a different tox env for the delegates tests09:53
pandazbr: I'm open to different solutions, right now, the delegates job is not running the scenario I need09:54
zbrprobably, but this was long time ago, lets talk about it, better to do it right now when is still a little to fix.09:54
zbri am willing to spend 30min of my PTO day to get it right.09:55
pandazbr: ok09:55
zbrmeet.google.com/aaq-biua-whi09:56
*** bhagyashris_ has joined #oooq10:03
*** dtantsur|learnin is now known as dtantsur10:03
*** bhagyashris_ has quit IRC10:03
*** holser has quit IRC10:24
*** holser has joined #oooq10:28
chandankumarcgoncalves, Hello10:43
chandankumarcgoncalves, scenario 10 jsut failed10:43
chandankumarcgoncalves, https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-master/d04ee1c/logs/undercloud/var/log/containers/neutron/server.log.txt.gz10:44
chandankumarcgoncalves, 2020-03-20 10:05:35.160 27 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 576, in check_for_ha_chassis_group_address10:44
chandankumar2020-03-20 10:05:35.160 27 ERROR futurist.periodics     high_prio_ch = max(default_ch_grp.ha_chassis, key=lambda x: x.priority)10:44
chandankumar2020-03-20 10:05:35.160 27 ERROR futurist.periodics ValueError: max() arg is an empty sequence10:44
cgoncalveschandankumar, hey. looking10:46
cgoncalveschandankumar, logs suggest it is not octavia related but OVN10:46
chandankumarcgoncalves, it failed at https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-master/d04ee1c/logs/undercloud/home/zuul/tempest.log.txt.gz10:47
chandankumarcgoncalves, config_tempest.services.base.ServiceError: Request on service 'load-balancer' with url 'http://192.168.24.1:9876' failed with code 50010:47
chandankumar2020-03-20 10:14:32 | 2020-03-20 10:14:32.371 128480 ERROR tempest [00m10:47
cgoncalveshttps://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-master/d04ee1c/logs/undercloud/var/log/containers/octavia/octavia.log.txt.gz10:47
*** ysandeep is now known as ysandeep|brb10:47
cgoncalves2020-03-20 10:18:01.362 28 ERROR octavia.api.drivers.driver_factory [-] Unable to load provider driver ovn due to: No 'octavia.api.drivers' driver found, looking for 'ovn': stevedore.exception.NoMatches: No 'octavia.api.drivers' driver found, looking for 'ovn'10:47
cgoncalvesOVN...10:47
chandankumarcgoncalves, ok, let me file a bug thanks!10:48
cgoncalvesthis will hopefully fix: https://review.opendev.org/#/c/713960/10:48
chandankumarcgoncalves, ok so it might happen that, we promoted the component without that fix, let me do a test project run10:49
chandankumarcgoncalves, fired here https://review.rdoproject.org/r/26027 let's see how it goes10:53
*** arxcruz|rover is now known as arxcruz|off10:56
*** ysandeep|brb is now known as ysandeep10:58
cgoncalvesOctavia is a project that depends on many core OpenStack services and does advance stuff with that the projects don't even test (e.g. allowed address pairs, GARPs, etc). also, Octavia relies on those services at deployment. failure rate of octavia tripleo jobs will be higher than others, I foresee. folks need to be considerate of that and not point fingers at octavia ;)10:59
cgoncalvesjust for example, in the last 24 hours the job failed due to OVN and Glance issues11:01
*** chem has quit IRC11:02
*** chem has joined #oooq11:03
pandazbr:  https://review.rdoproject.org/r/2602211:08
zbrtox -re molecule --notest11:10
kopecmartinzbr: hi, i need  a quick molecule help .. do you know why 'tox -emolecule' skips infrared tests and runs only sova ones? line 38: http://paste.openstack.org/show/790934/11:36
zbrdefault behavior is to skip any delegated tests, as safety measures.11:37
kopecmartinzbr: hm, i didn't realise sova and infrared tests are different , ok, thanks11:38
zbralso this is one of the reasons why i recommend using docker driver instead, but if i remember there was an open change for addressing this issue.11:39
zbrand coverting it to use docker11:39
zbrcontainer one is... container, not risking to mess your system w/ testing.11:39
kopecmartinzbr: makes sense, i'm trying to rebase this https://review.opendev.org/#/c/705552/ and want to reproduce the error locally11:40
chandankumarweshay|ruck, needs votes for new repo creation https://review.opendev.org/713809 and https://review.opendev.org/71403011:43
pandazbr: tox jobs actually revoke sudo privilege to zuul user ....11:43
pandazbr: this is deal breaking for all molecule delegated jobs .... unless there's an option to not revoke11:44
zbrnope, that is by design11:44
*** ccamacho has joined #oooq11:45
*** ccamacho has quit IRC11:45
zbrbut also an interesting question to ask on #zuul, maybe is possible11:45
*** ccamacho has joined #oooq11:45
zbrpanda: how did we use delegated so far?11:45
pandazbr: you saw it, the jobs parent off base11:47
pandazbr: as far as I see, there are no variables in the role11:47
zbrok time to ask on #zuul11:47
pandazbr: if you call revoke-sudo, it disables sudo privilege11:48
pandaand playbooks/tox/run.yml have it hard coded11:48
pandazbr: who's gonna ask ?11:50
zbri already asked11:50
pandaghostbusters!11:50
*** dpawlik has quit IRC11:50
pandazbr: https://review.rdoproject.org/r/2603012:00
zbrdid you get confirmation that it worked?12:01
pandazbr: not yet12:03
*** ysandeep is now known as ysandeep|afk12:04
*** ccamacho has quit IRC12:10
zbrping me when so, to merge it.12:12
*** dpawlik has joined #oooq12:21
weshay|ruckchandankumar, soniya29 kopecmartin /me in mtg :) things returning to normal12:29
chandankumarweshay|ruck, my home network is not working pretty well12:35
*** skramaja has quit IRC12:42
*** skramaja has joined #oooq12:43
chandankumarweshay|ruck, needs votes for new repo creation https://review.opendev.org/713809 and https://review.opendev.org/71403012:45
weshay|ruckhttps://review.opendev.org/#/c/706218/12:46
weshay|ruckchandankumar, did my connection die?12:48
chandankumarweshay|ruck, mine died12:48
kopecmartinmine too12:49
weshay|ruckchandankumar, soniya29 kopecmartin sorry.. connection was lost.. we're good :)12:50
weshay|ruckthanks for joining12:50
marioshttps://bugs.launchpad.net/tripleo/+bug/186794513:05
openstackLaunchpad bug 1867945 in tripleo "periodic centos8 standalone-full-tempest-tempest-master timeout" [Critical,Triaged]13:05
weshay|ruckmarios, https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-master/d04ee1c/logs/undercloud/home/zuul/tempest.log.txt.gz13:11
chandankumarmarios, https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-master/d04ee1c/logs/undercloud/home/zuul/tempest.log.txt.gz13:12
chandankumarhttps://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-master/d04ee1c/logs/undercloud/var/log/containers/octavia/octavia.log.txt.gz13:12
weshay|ruck2020-03-20 10:14:32 | 2020-03-20 10:14:32.371 128480 ERROR tempest config_tempest.services.base.ServiceError: Request on service 'load-balancer' with url 'http://192.168.24.1:9876' failed with code 50013:12
*** jpena is now known as jpena|lunch13:23
chandankumarweshay|ruck, marios https://review.rdoproject.org/r/#/c/25919/13:24
weshay|ruckchandankumar, https://review.rdoproject.org/r/#/c/25780/13:27
chandankumarmarios, Here is my keys https://github.com/raukadah.keys13:31
*** udesale_ has joined #oooq13:35
*** TrevorV has joined #oooq13:37
marioschandankumar: ack sorry was getting coffee adding now sec13:37
kopecmartinchandankumar: do you know where's the source code for this bug filter in cockpit? http://dashboard-ci.tripleo.org/d/cockpit/cockpit?orgId=1&fullscreen&panelId=34513:37
chandankumarkopecmartin, give me a min13:38
*** udesale has quit IRC13:38
chandankumarkopecmartin, https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/mariadb/launchpad_bugs_mariadb.py and https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/mariadb/bugzilla_bugs_mariadb.py13:39
marioschandankumar: try it when you get chance (see pvt)13:39
kopecmartinchandankumar: thanks!13:39
-openstackstatus- NOTICE: Gerrit (review.opendev.org) will be down for maintenance starting at 14:00 (in less than 20 mins), probably until 15:00 UTC13:41
*** amoralej is now known as amoralej|lunch13:47
*** ratailor has quit IRC13:49
*** derekh has joined #oooq13:51
-openstackstatus- NOTICE: The Gerrit service on review.opendev.org is offline for maintenance until 15:00 UTC http://lists.opendev.org/pipermail/service-announce/2020-March/000001.html14:04
*** ChanServ changes topic to "The Gerrit service on review.opendev.org is offline for maintenance until 15:00 UTC http://lists.opendev.org/pipermail/service-announce/2020-March/000001.html"14:04
*** jpena|lunch is now known as jpena14:05
chandankumarmarios, I am not sure ssh timeout in the current failed is related to current ovn issue14:10
chandankumar*failed test14:10
marioschandankumar: yeah exactly that was my point it fails in 'yet another way'14:10
marioschandankumar: i didn't say it was related to ovn issue14:10
chandankumarmarios, ok, let's release the node then, we can work splitting the job and see what happens14:12
marioschandankumar: fails that test tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps in the held node14:13
marioschandankumar: added in https://bugs.launchpad.net/tripleo/+bug/1867945/comments/314:13
openstackLaunchpad bug 1867945 in tripleo "periodic centos8 standalone-full-tempest-tempest-master timeout" [Critical,Triaged]14:13
*** matbu has quit IRC14:13
chandankumartempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_port_security_disable_security_group14:13
chandankumarmarios, yes, thanks14:13
marioschandankumar: ack ykarel thanks very much for help please feel free to release when you can (node held @ https://review.rdoproject.org/r/#/c/26020/ )14:14
chandankumarykarel, please delete the holded node14:14
mariosthank you ykarel !14:14
ykarelmarios, chandankumar ok releasing14:15
ykarelmarios, deleted14:16
mariosykarel: thx#14:17
*** matbu has joined #oooq14:24
*** udesale_ has quit IRC14:33
marioswoo gerrit is back \o/14:35
*** TrevorV has quit IRC14:42
*** TrevorV has joined #oooq14:45
*** amoralej|lunch is now known as amoralej14:45
*** skramaja has quit IRC14:58
*** ChanServ changes topic to "Docs: https://docs.openstack.org/tripleo-quickstart/latest/ || 17th march promoter critical bugfix https://review.rdoproject.org/r/25938"15:01
-openstackstatus- NOTICE: Gerrit maintenance is concluded at this time and requested renames have been performed15:01
*** apetrich has quit IRC15:04
chandankumarmarios, weshay|ruck arxcruz|off full tempest split https://review.rdoproject.org/r/#/c/26035/15:07
chandankumararxcruz|off, please take over this patch https://review.rdoproject.org/r/#/c/26035/ on monday and finish it15:08
marioschandankumar: ack will check15:08
weshay|ruckchandankumar, rock on king kong15:08
*** apetrich has joined #oooq15:22
marioschandankumar: wait... so we removed all the 'us_os_tempest' from tripleo-ci/zuul.d/* ?15:30
marioschandankumar: cos its the default now or something like that (use_os_tempest)15:31
chandankumarmarios, nope, I have not15:31
marioschandankumar: k sorry bad grep i see it now15:31
*** ratailor has joined #oooq15:36
*** ccamacho has joined #oooq15:46
*** ykarel is now known as ykarel|afk16:06
*** ratailor has quit IRC16:15
*** openstack has joined #oooq16:55
*** ChanServ sets mode: +o openstack16:55
*** bogdando has quit IRC16:56
*** marios has quit IRC17:01
*** dtantsur is now known as dtantsur|afk17:02
*** ykarel is now known as ykarel|away17:12
*** jpena is now known as jpena|off17:44
*** jfrancoa has quit IRC18:10
*** ccamacho has quit IRC18:31
*** amoralej is now known as amoralej|off18:36
*** jmasud has quit IRC18:37
*** jmasud has joined #oooq18:38
*** rlandy has joined #oooq18:46
rlandyweshay|ruck: hi!18:46
weshay|ruck0/18:47
rlandyweshay|ruck: all ok with the network stuff?18:47
weshay|rucksounded like it18:47
rlandyiiuc, it will fail integration w/o the depends-on patch18:47
weshay|ruckhaven't heard from beagles about it18:47
rlandythat's what I was trying to say18:47
weshay|ruckya.. we saw that and it did18:47
rlandyokie dokie18:47
weshay|ruckchandan and I talked about it this morning18:47
weshay|ruckand marios18:48
weshay|ruckthe patch just needs to land... I and I need to get the cockpit to show testproject jobs18:48
rlandyweshay|ruck: yeah - will have to think about using pipeline jobs to testproject changes18:48
weshay|ruckrlandy, I'm querying the by job name.. criteria is job name18:49
beaglesyeah that patch is just hanging out in the gate18:49
beagleswe had to recheck it this a.m.18:49
weshay|ruckI'll have to see what queues we're pulling in the monitoring18:49
rlandybeagles: thanks - should be fine18:49
weshay|ruckbeagles, last night and this morning it was a little choppy18:49
beaglesack18:50
weshay|ruckseems to have settled18:50
rlandyyou're right though - hard to track when we get a testproject promote18:50
weshay|ruckfreaks me out :)18:50
weshay|ruckbut I'll fix it18:50
rlandyweshay|ruck: in other news ... baremetal nics and networking ...18:50
rlandyso - I remember we chose to disable network manager before in favor of network service18:51
weshay|ruckya.. so centos8 no longer support linux bridges?18:51
rlandybut that seems to be obsoleted  - almost18:51
rlandyI put it back18:51
weshay|ruckya.. ok.. and nmcli is usable now18:51
weshay|ruckk18:51
rlandybut when I read around, seems like the wrong thing18:51
rlandyand brctl is gone, gone18:52
weshay|ruckrlandy, I need to be brought into context a little bit more18:52
weshay|ruckrlandy, maybe monday18:52
rlandyweshay|ruck: sure - whenever18:52
weshay|ruckso are you in St Louis18:52
rlandyI am18:52
rlandygot through LGA in 3 mins18:53
weshay|rucklolz18:53
rlandyvery weird18:53
weshay|ruckno zombies I hope18:53
rlandyI'm in quarantine though - all visitors from NY are18:53
weshay|ruckoh jebus18:53
weshay|ruckhow long18:53
rlandy14 days18:54
weshay|ruckWHAT18:54
weshay|ruckat home?18:54
rlandythere were 4 people on plane18:54
weshay|ruckomg18:54
rlandyin my mother's basement18:54
weshay|ruckwow18:54
weshay|ruckso if you want to help me understand the issue.. I can chat18:54
weshay|ruckor just chat lolz18:54
rlandyyou have lunch plans, no?18:54
weshay|ruckneed a metal file?18:54
weshay|ruckrlandy, no man.. my family has the coodies18:55
rlandyok - we can chat then18:55
weshay|ruckhttps://meet.google.com/yor-kswv-xkn?authuser=118:56
rlandygetting baremetal undercloud18:56
weshay|rucknet.ifnames=0 biosdevname=019:10
weshay|ruckgrub2-mkconfig -o /boot/grub2/grub.cfg19:11
weshay|ruckrlandy, https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-disabling_consistent_network_device_naming19:14
*** kopecmartin is now known as kopecmartin|afk19:55
*** holser has quit IRC20:42
*** holser has joined #oooq20:44
*** kopecmartin|afk has quit IRC21:18
*** kopecmartin has joined #oooq21:19
*** TrevorV has quit IRC21:27
*** rlandy has quit IRC21:48
*** jjoyce has quit IRC22:01
*** fuzzball81 has joined #oooq22:01
*** irclogbot_0 has quit IRC23:27
*** irclogbot_2 has joined #oooq23:27
*** gcerami_ has joined #oooq23:35
*** panda has quit IRC23:37
*** gcerami_ is now known as panda23:37
*** holser has quit IRC23:41
*** tosky has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!