Tuesday, 2020-07-21

« Monday, 2020-07-20 Index Wednesday, 2020-07-22 »
*** openstack has joined #oooq07:30
*** ChanServ sets mode: +o openstack07:30
*** bogdando has joined #oooq07:33
*** holser has joined #oooq07:34
*** tosky has joined #oooq07:37
*** jpena|off is now known as jpena07:50
*** dtantsur|afk is now known as dtantsur07:55
*** derekh has joined #oooq08:25
*** skramaja has joined #oooq08:48
arxcruzchandankumar: namaste09:03
arxcruzchandankumar: letś merge https://review.opendev.org/#/c/741905/09:04
chandankumararxcruz: namaste09:04
chandankumararxcruz: testproject done?09:04
arxcruzchandankumar: nope, but it seems to be working, i'll do a check experimental and rdo experimental09:05
arxcruzdo you know exactly the syntax ?09:05
*** jfrancoa has joined #oooq09:06
chandankumararxcruz: check experimental09:07
chandankumararxcruz: since fs10 multinode containers are working fine09:07
chandankumarso there will be no issue09:08
chandankumararxcruz: please add the taiga link to the review, rest is good to go09:08
chandankumararxcruz: please have a look on this https://bugzilla.redhat.com/show_bug.cgi?id=1859086 when free, thanks!09:08
openstackbugzilla.redhat.com bug 1859086 in openstack-tempest "Broken tempest - Request (main): 502 PUT ends with Unhandled error: tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received" [Unspecified,New] - Assigned to chkumar09:08
arxcruzchandankumar: check experimental is upstream, on rdo side, do you know the syntax ?09:09
arxcruznot sure if is check-rdo-experimental or check-experimental09:09
chandankumararxcruz: https://github.com/rdo-infra/review.rdoproject.org-config/blob/000293ed7d62632c26dfcec8bdf6dd796ca06c94/zuul.d/pipelines.yaml#L19109:10
arxcruzokay09:10
arxcruzdone, let's see :)09:10
*** soniya29 has joined #oooq09:18
*** soniya29 has quit IRC09:20
*** pojadhav is now known as pojadhav|lunch09:20
*** soniya29 has joined #oooq09:22
*** holser has quit IRC09:26
*** holser has joined #oooq09:27
*** Tengu has quit IRC09:39
*** Tengu has joined #oooq09:41
*** pojadhav|lunch is now known as pojadhav09:50
mariosstable/train cherrypick needs reviews when u next have time please https://review.opendev.org/741631 thanks10:02
*** ratailor_ has quit IRC10:10
*** ratailor_ has joined #oooq10:10
marioschandankumar: arxcruz: ** need reviews here please when you next have time https://review.opendev.org/#/c/739217/ https://review.opendev.org/#/c/739218/10:30
chandankumaron it10:30
chandankumarmarios: https://review.opendev.org/#/c/739217/ left some comment10:37
marioschandankumar: thanks very much for reviewing I have replied inline we can't do that10:42
marioschandankumar: (revisit when you next can doesn't have to be right now thanks for your time)10:42
mariosthanks for review arxcruz10:52
arxcruz:* <310:52
*** udesale_ has joined #oooq10:56
*** udesale has quit IRC10:58
*** pojadhav is now known as pojadhav|brb11:29
chandankumarzbr|rover: Hello, do we have patch up for https://review.rdoproject.org/zuul/builds?job_name=mol-promoter job?11:32
weshay|ruckzbr|rover, is there an open bug on https://logserver.rdoproject.org/30/28630/7/check/periodic-tripleo-ci-centos-7-scenario010-standalone-train-0/75cf833/logs/undercloud/var/log/tempest/stestr_results.html.gz ?11:32
chandankumarit is broken currently11:32
weshay|ruckit's not the same issue as https://bugs.launchpad.net/tripleo/+bug/188560211:32
openstackLaunchpad bug 1885602 in tripleo "Octavia component: failing tempest, Details: {'faultcode': 'Client', 'faultstring': 'Policy does not allow this request to be performed.'," [Critical,Triaged] - Assigned to Brent Eagles (beagles)11:32
chandankumarweshay|ruck: nope, different one, Details: (LoadBalancerScenarioTest:test_load_balancer_ipv4_CRUD) show_loadbalancer provisioning_status failed to update to ACTIVE within the required time 900. Current status of show_loadbalancer: PENDING_CREATE11:33
weshay|ruckaye11:33
weshay|ruckchandankumar, so.. neutron-server gets skipped if neutron_driver=ovn11:33
weshay|ruckre: https://review.rdoproject.org/r/#/c/28624/11:33
chandankumarlet me check the .j2 file11:34
zbr|roverchandankumar: no, looking now. i also think we need to enable emails when this fails on periodic11:34
chandankumarweshay|ruck: something wrong on our rendering side11:35
chandankumarweshay|ruck: need to dig locally11:35
zbr|roverhow where we able to break it? afaik, mol-promoter is now in gate.11:36
chandankumarzbr|rover: last successful merge is this https://review.rdoproject.org/r/#/c/28617/11:37
zbr|roversomething else broke it11:37
chandankumarright now broke11:37
chandankumar    TASK [Ensure docker manifest command is enabled by default]11:37
chandankumarhttps://logserver.rdoproject.org/24/28624/4/check/mol-promoter/8621b49/tox/reports.html11:37
zbr|roverlikely from outside ci-config repo11:38
weshay|ruckzbr|rover, can you please attend the program call tomorrow11:39
zbr|roversure11:40
chandankumarzbr|rover: let's get a bug up, then we can deal11:40
*** jpena is now known as jpena|lunch11:40
weshay|ruckcool.. thanks11:40
chandankumarweshay|ruck: does the data gathered from yaml and j2 file for containers were different?11:40
chandankumarin case of overcloud_containers.yaml file?11:40
zbr|roveri will raise it and assign to me, starting on it. but if someone finds the source it would be even better11:41
*** ratailor_ has quit IRC11:41
arxcruzweshay|ruck: marios https://review.opendev.org/#/c/741905/11:41
arxcruzthis will fix also the ronelle issue on non zuul jobs11:41
*** sshnaidm|afk is now known as sshnaidm11:41
mariosarxcruz: ack adding to reviews list11:42
arxcruzsudo marios do it now!11:42
mariosarxcruz: show me some candy11:42
arxcruzmarios: https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Fimg0.etsystatic.com%2F020%2F0%2F9066975%2Fil_570xN.553914598_pts0.jpg&f=1&nofb=111:43
*** jbadiapa has joined #oooq11:43
marios:D arxcruz11:43
zbr|roverraised as https://bugs.launchpad.net/tripleo/+bug/188838711:44
openstackLaunchpad bug 1888387 in tripleo "mol-promoter: broken on Ensure docker manifest command is enabled by default" [Undecided,New] - Assigned to Sorin Sbarnea (ssbarnea)11:44
arxcruzmarios: btw, this tv show is amazing https://www.youtube.com/watch?v=3sxg1xXmd0I11:46
mariosarxcruz: :D ack will check it out this evening11:48
arxcruzand it's based on a real person, living in Leipzig11:48
arxcruzhttps://www.telegraph.co.uk/news/worldnews/europe/germany/11739946/Man-caught-selling-600kg-of-drugs-online-in-one-of-Germanys-biggest-narcotics-raids.html11:48
*** soniya29 has quit IRC11:50
zbr|roverchandankumar: this error is bit mindblowing as I do not see any case where assert is expected to give this error11:51
*** ykarel_ has joined #oooq11:53
*** ykarel has quit IRC11:56
zbr|roveri will have a big laugh if this is related to last night ansible new release11:57
chandankumarzbr|rover: let's see11:58
* chandankumar steps out for a bit11:58
zbr|rovereasy to findout as we print ansible version at top of the report11:58
zbr|roveryes, it is 2.9.1111:59
zbr|roverthe reality is that documentation states that that receives a *list*11:59
zbr|roverthey do mention anything about a string11:59
zbr|roverand we all know a string is a list in python.... but not the kind of list we would want12:00
*** rlandy has joined #oooq12:00
*** rlandy is now known as rlandy|ruck12:01
rlandy|ruckzbr|rover: weshay|ruck: hi - how goes it today?12:02
zbr|roverrlandy|ruck: ansible released last night, so you can guess ;)12:03
rlandy|ruckzbr|rover: :( ... what do you need me to pick up?12:05
* rlandy|ruck just looking at master promotion12:06
weshay|ruckrlandy|ruck, it goes... https://logserver.rdoproject.org/30/28630/7/check/periodic-tripleo-ci-centos-7-scenario010-standalone-train-0/75cf833/logs/undercloud/var/log/tempest/stestr_results.html.gz12:07
*** rfolco has joined #oooq12:08
weshay|ruckis that being worked on by https://review.opendev.org/#/c/714639/12:08
weshay|ruckhttps://bugs.launchpad.net/tripleo/+bug/188766612:08
openstackLaunchpad bug 1887666 in tripleo "tripleo-ci-centos-8-scenario010-ovn-provider-standalone is failing on tempest tests with error Details: {'faultcode': 'Client', 'faultstring': "Provider 'ovn' is not enabled.", 'debuginfo': None}" [High,Triaged]12:08
weshay|ruckor diff issue12:08
weshay|ruckah.. diff issue I think12:08
rlandy|ruckdiff issue12:08
rlandy|rucknever even looked at ovn12:09
weshay|ruckrlandy|ruck, so .. standalone-scenario010 train is running really well in train12:09
rlandy|rucknever really been pasing12:09
*** amoralej is now known as amoralej|lunch12:09
rlandy|ruckpassing12:09
weshay|ruckhttps://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario010-standalone12:09
weshay|ruckcan't skip this one12:09
rlandy|ruckgate looks unhappy12:09
rlandy|ruckweshay|ruck: did we break promotions on master?12:10
weshay|ruckhrm.. don't think so12:10
rlandy|ruckno log12:10
rlandy|ruckweshay|ruck: going to restart service so we get some log on master12:12
rlandy|ruckdon;t know what's missing atm12:12
*** svyas has joined #oooq12:14
*** pojadhav|brb is now known as pojadhav12:17
*** Goneri has joined #oooq12:17
zbr|roveri am unable to reproduce the error locally....12:22
rlandy|ruckweshay|ruck: ah ... master had a repeated requirement12:25
rlandy|rucknow we have a log12:26
zbr|roververy weird someone changed PyYAML -- different sha, it was supposed to be impossible12:27
zbr|rover Created wheel for pyyaml: filename=PyYAML-5.3.1-cp27-cp27mu-linux_x86_64.whl size=387412 sha256=129f3621b1b71f170f6dec7d6f0b063fb93092cbc30ade1ab54c5eeed9a0fb7512:28
zbr|roverCreated wheel for pyyaml: filename=PyYAML-5.3.1-cp27-cp27mu-linux_x86_64.whl size=387425 sha256=9ac364b1114b721c03784ae2d2b37507c78ecfced19f35714d38fc5b505fdb1612:28
chandankumarrlandy|ruck: \o/12:34
chandankumarrlandy|ruck: weshay|ruck http://paste.openstack.org/show/796151/ check this same script, neutron-server is there12:34
rlandy|ruckchandankumar: iiuc, depends on release12:35
chandankumarrlandy|ruck: which release?12:35
chandankumarrlandy|ruck: train?12:36
rlandy|ruckwhere we logged the missing containers12:36
rlandy|ruckyeah12:36
rlandy|ruckgetting12:36
chandankumarlet me do a for loop12:36
chandankumarwe will find out easily12:36
rlandy|ruckhttps://opendev.org/openstack/tripleo-common/src/branch/stable/train/container-images/overcloud_containers.yaml.j2#L53112:36
rlandy|ruckchandankumar: ^^12:36
rlandy|rucksee the diff12:36
rlandy|ruckin train ( and maybe others)12:36
rlandy|ruckif you get ovn, you don;t get odl rtc.12:37
chandankumargive me some time12:37
chandankumari will find it12:37
rlandy|ruckchandankumar: why don;t we just always use the rendered yaml file?12:37
* rlandy|ruck hasn;t checked ussuri yet12:37
rlandy|ruckbut I gather it has the same issue12:38
chandankumarrlandy|ruck: let me check with EmilienM12:40
EmilienMchandankumar: yes?12:40
chandankumarEmilienM: which one is the source of truth for containers for stable release https://opendev.org/openstack/tripleo-common/src/branch/master/container-images/overcloud_containers.yaml.j2 or https://opendev.org/openstack/tripleo-common/src/branch/master/container-images/overcloud_containers.yaml ?12:41
EmilienMhttps://opendev.org/openstack/tripleo-common/src/branch/master/container-images/overcloud_containers.yaml.j2 makes the link between THT services and their container images12:42
EmilienMovercloud_containers.yaml is the list of images in a release and where to pull them by default12:42
chandankumarEmilienM: ok, got it12:43
*** jpena|lunch is now known as jpena12:43
chandankumarI think on promoter side, we can use overcloud_containers.yaml to push containers to docker.io instead of parsing .j2 file12:44
chandankumarrlandy|ruck: weshay|ruck ^^12:44
weshay|ruckya.. we did it last night :)12:44
rlandy|ruckchandankumar: ack - definitely12:45
chandankumarlet me remove the code of j2 parsing and use only yaml stuff12:45
rlandy|ruckchandankumar: we just need to add the override to ussuri12:45
rlandy|ruckchandankumar: you can leave the code12:45
chandankumarEmilienM: thanks :-)12:45
rlandy|ruckdoesn't matter12:46
EmilienMsure12:46
chandankumarrlandy|ruck: let me keep the code base clean12:46
weshay|ruckso crisp and so clean clean12:46
rlandy|ruckchandankumar: will we break something else in doing so?12:46
chandankumarit will be easier for us to maintain otherwise it will again bring confusion12:46
rlandy|ruckchandankumar: ok - we need to update ussuri then12:47
chandankumarrlandy|ruck: will test it for sure12:47
chandankumarlet me get the info at one place first, then start cleaning up12:47
rlandy|ruckhmmm ...12:48
weshay|ruckzbr|rover, rlandy|ruck is a ruck / rover sync in order 10min?12:48
rlandy|ruckwe have a real introspection issue12:49
rlandy|ruckweshay|ruck: ack12:49
weshay|ruckzbr|rover, you avail?12:55
*** rascasoft has joined #oooq12:55
rlandy|rucksync?13:00
mariosneed reviews here please when you next have time https://review.opendev.org/#/c/739217/ https://review.opendev.org/#/c/739218/13:02
weshay|ruckrlandy|ruck, waiting on zbr|rover13:03
weshay|ruckrlandy|ruck, zbr|rover fyi .. https://bugs.launchpad.net/tripleo/+bug/188839713:03
openstackLaunchpad bug 1888397 in tripleo "train standalone-scenario10, loadbalancer fails to get to active state" [Critical,Triaged]13:03
*** ykarel_ is now known as ykarel13:03
rlandy|ruckweshay|ruck: did they change every file they own in that patch??13:04
weshay|ruckheh13:04
rlandy|ruckreally narrows things down13:05
rlandy|ruckweshay|ruck: digging into the introspection failure atm13:05
rlandy|ruckmaster on rdocloud as well13:05
rlandy|ruckstopping master13:05
rlandy|ruckwill look at loadbalancer next13:06
weshay|ruckrlandy|ruck, maybe it's a nick issue13:06
rlandy|rucknick issue?13:07
weshay|ruckzbr|rover,13:07
weshay|ruckhe has two nicks at times.. but zbr is not active atm13:07
rlandy|ruckoh - yeah13:07
rlandy|ruckbut today he only has one active nick13:07
weshay|ruckrfolco, can you please try to remember to make sure I join the next TC call please13:08
rfolcoweshay|ruck, next tuesday, 2813:08
rlandy|ruckweshay|ruck: hmm zbr is not on internal - maybe he gt disconnected13:08
rlandy|ruckgot13:08
weshay|ruckrfolco++ thanks13:09
rlandy|ruckweshay|ruck: should we meet and zbr|rover will join when he is connected again?13:11
weshay|rucksec.. let me ping him on hangouts13:12
weshay|ruckanyone need reviews?13:20
*** holser has quit IRC13:20
*** holser_ has joined #oooq13:20
mariosneed reviews here please when you next have time https://review.opendev.org/#/c/739217/ https://review.opendev.org/#/c/739218/13:20
mariosweshay|ruck: ^13:20
*** amoralej|lunch is now known as amoralej13:21
*** TrevorV has joined #oooq13:23
zbr|roverthat was caused by 2.9.11 release, pinning down avoids it13:26
chandankumarzbr|rover: weshay|ruck https://review.rdoproject.org/zuul/builds?pipeline=openstack-regular&project=ceph%2Fceph-ansible&job_name=tripleo-ceph-ansible-integration-centos-8-scenario001-standalone-ussuri&job_name=tripleo-ceph-ansible-integration-centos-8-scenario001-standalone&branch=stable-4.013:30
chandankumarmore breakage i think13:30
chandankumarhttps://logserver.rdoproject.org/openstack-regular/github.com/ceph/ceph-ansible/stable-4.0/tripleo-ceph-ansible-integration-centos-8-scenario001-standalone-ussuri/4d03c59/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz13:30
mariosrfolco: you can't hear us13:30
mariosrfolco: we can hear you13:30
chandankumarthird party ceph-ansible master and ussuri job also got busted13:30
rlandy|ruckzbr|rover: weshay|ruck: sync after community call?13:31
weshay|ruckmarios, man.. https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bab/739457/8/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-upgrades-ussuri/babae60/logs/undercloud/home/zuul/overcloud_upgrade_run_Controller.log13:31
mariosweshay|ruck: o/13:31
rfolcoping community call13:31
rfolcoarxcruz, rfolco, zbr, sshnaidm, rlandy, marios, ysandeep, bhagyashris, soniya29, pojadhav, akahat, weshay, chandankumar13:31
weshay|ruckCallback Exception:13:31
weshay|ruck2020-07-21 11:21:16 |   File "/usr/lib/python3.6/site-packages/ansible/executor/task_queue_manager.py", line 327, in send_callback13:31
weshay|ruck2020-07-21 11:21:16 |     method(*new_args, **kwargs)13:31
weshay|ruck2020-07-21 11:21:16 |    File "/usr/lib/python3.6/site-packages/ansible/plugins/callback/profile_tasks.py", line 161, in v2_playbook_on_task_start13:31
weshay|ruck2020-07-21 11:21:16 |     self._record_task(task)13:31
weshay|ruck2020-07-21 11:21:16 |    File "/usr/share/ansible/plugins/callback/tripleo_profile_tasks.py", line 76, in _record_task13:31
weshay|ruck2020-07-21 11:21:16 |     self._output_previous_timings(self.current)13:31
*** holser_ has quit IRC13:32
sshnaidmweshay|ruck, I think there was a patch disabling callbacks somewhere13:32
*** svyas is now known as svyas|afk13:33
*** holser has joined #oooq13:33
zbr|rover https://review.rdoproject.org/r/2864113:38
rfolcozbr|rover, did I break ansible become ?13:48
rfolcozbr|rover, sorry if it was me13:48
*** rascasoft has quit IRC13:51
weshay|ruckchandankumar, unrelated https://bugs.launchpad.net/tripleo/+bug/188649713:54
openstackLaunchpad bug 1886497 in tripleo "ceph/train: chown failed: failed to look up user tripleo-admin" [High,Fix released] - Assigned to Giulio Fidente (gfidente)13:54
chandankumarrlandy|ruck: zbr|rover weshay|ruck fyi https://bugs.launchpad.net/tripleo/+bug/188840413:55
openstackLaunchpad bug 1888404 in tripleo " Unexpected templating type error occurred while generating Generate ReaR config file during standalone deploy" [High,Triaged]13:55
weshay|ruckzbr|rover, rlandy|ruck let's get back on13:55
weshay|rucksorry13:55
rlandy|ruckweshay|ruck: zbr|rover: do we still need to sync?13:55
weshay|ruckjust to make sure we sync13:55
rlandy|ruckok13:56
weshay|rucksame call13:56
chandankumarzbr|rover: weshay|ruck https://bugs.launchpad.net/tripleo/+bug/1888404 is also caused by ansible-2.9.1114:03
openstackLaunchpad bug 1888404 in tripleo " Unexpected templating type error occurred while generating Generate ReaR config file during standalone deploy" [High,Triaged]14:03
*** rascasoft has joined #oooq14:04
zbr|roverchandankumar: where is ansible upstream bug for it?14:07
chandankumarzbr|rover: https://github.com/ansible/ansible/issues/64398 is already a bug for that14:09
chandankumarweshay|ruck: do we want to cix ceph-ansibel failure?14:10
pojadhavweshay|ruck, chandankumar, rlandy|ruck , marios : please have a look https://review.opendev.org/#/c/741644/ , https://review.opendev.org/#/c/741891/ and https://review.rdoproject.org/r/#/c/28284/14:16
weshay|ruckzbr|rover, https://review.rdoproject.org/r/#/c/28624/4/ci-scripts/dlrnapi_promoter/repo_client.py14:16
mariospojadhav: will do14:17
EmilienMchandankumar: https://bugs.launchpad.net/tripleo/+bug/188840414:35
openstackLaunchpad bug 1888404 in tripleo " Unexpected templating type error occurred while generating Generate ReaR config file during standalone deploy" [High,Triaged]14:35
EmilienMwhy are we hitting a bug in REAR stuff14:35
EmilienMare we running the role?14:35
EmilienMwhy ReaR is deploying in tripleo-ceph-ansible-integration-centos-8-scenario001-standalone?14:36
chandankumarEmilienM: it is standalone sc1 job as upstream have  https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/ceph-ansible.yaml#L19014:38
*** rascasoft has quit IRC14:38
chandankumarEmilienM: need to check why it is coming there14:39
chandankumarEmilienM: jbadiapa is looking into that14:40
weshay|ruckmarios, any reason to not enable ussuri upgrade w/ those errors?14:48
mariosweshay|ruck: what do you mean by 'enable' ?14:48
mariosweshay|ruck: you mean https://review.opendev.org/742182 ?14:48
weshay|ruckmarios, https://review.opendev.org/73921814:49
weshay|ruckI was looking at the ussuri test job in https://review.opendev.org/#/c/739457/14:49
mariosweshay|ruck: we do enable it .... we have ussuri undercloud upgrade job14:50
weshay|ruckI think the errors were in https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bab/739457/8/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-upgrades-ussuri/babae60/logs/undercloud/home/zuul/overcloud_upgrade_run_Controller.log14:50
mariosweshay|ruck: its just not "split" into two jobs yet i.e. -master and -ussuri14:50
weshay|ruckya.. not contesting that part14:50
mariosweshay|ruck: that one is the overcloud upgrade 17:50 < weshay|ruck> I think the errors were in14:50
marioshttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bab/739457/8/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-upgrades-ussuri/babae60/logs/undercloud/home/zuul/overcloud_upgrade_run_Controller.log14:50
weshay|ruckya.. just callback exceptions14:51
mariosweshay|ruck: ok i blocked https://review.opendev.org/742182 for now cos i want to dig a bit more (making it voting)14:51
mariosweshay|ruck: but anyway we need to merge th two parent reviews first14:51
mariosweshay|ruck:  https://review.opendev.org/#/c/739217/ & 21814:51
mariosweshay|ruck: then we start the process of 'fix/sanity check/make voting asap'14:52
weshay|ruckmarios++ ok.. that sounds like a plan14:52
mariosweshay|ruck: we have a couple nasty blockers especially ussuri standalone upgrade and train minor update .. the ussuri one might be difficult (switched ha by default in train )14:52
weshay|ruckya.. I recall that...14:53
weshay|ruckbut not well enough to be helpful14:54
weshay|ruckre: standalone and ha settings14:54
mariosweshay|ruck: ack np just fyi for now14:55
*** rascasoft has joined #oooq14:55
*** skramaja has quit IRC15:17
*** ykarel is now known as ykarel|away15:20
*** ykarel|away has quit IRC15:26
rlandy|rucksshnaidm: is there a list of rpms in the bmc image?15:29
rlandy|ruckdeps that it built with15:29
sshnaidmrlandy|ruck, I don't think so, I can extract15:30
rlandy|rucksshnaidm: actually it's the ipxe-boot I need15:35
sshnaidmrlandy|ruck, I doubt there is installed anything15:37
*** yolanda has quit IRC15:42
*** yolanda has joined #oooq15:44
rlandy|ruckarxcruz, https://review.opendev.org/#/c/741977/ is merged15:44
rlandy|ruckwe still see the error in ci.centos15:44
rlandy|ruckI assume that change need to go up through the tempest line?15:44
arxcruzrlandy|ruck: yes it requires https://review.opendev.org/#/c/741905/ as well15:45
rlandy|ruckah ok15:45
*** bogdando has quit IRC15:50
*** udesale_ has quit IRC15:59
*** marios has quit IRC16:07
rlandy|ruckamoralej: hi ... https://review.rdoproject.org/r/#/c/28604/ errors associated with https://bugs.launchpad.net/tripleo/+bug/1887856 but in the nodes that boot in introspection we see 'No module name importlib.metadata'16:08
openstackLaunchpad bug 1887856 in tripleo "centos-8 master tripleo component tests are failing with "ModuleNotFoundError: No module named 'blazarclient'"" [Critical,In progress] - Assigned to Ronelle Landy (rlandy)16:08
amoralejrlandy|ruck, i think the fix for that was pushed at least yesterday16:08
amoralejmmm16:08
amoralejrlandy|ruck, can i see the logs?16:09
rlandy|ruckamoralej: yeah - wondering of we need to add something to get the fix into image build16:09
rlandy|rucklol ... it's a screenshot but I will send the output logs16:09
rlandy|ruckamoralej: https://logserver.rdoproject.org/83/21683/41/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/bc40b18/16:10
weshay|ruckrlandy|ruck, so... is that package in the image build for ipa?16:10
* weshay|ruck looks16:10
rlandy|ruckweshay|ruck: image build include delorean*16:10
rlandy|ruckwhich should have picked up delorean16:11
amoralejrlandy|ruck, what's the job building the ipa image in that run?16:11
rlandy|ruckidk if it picks up https://review.rdoproject.org/r/#/c/28604/16:11
rlandy|ruckamoralej: images were built in ...16:11
rlandy|ruckhttps://review.rdoproject.org/zuul/build/bc4c8b8dcf1e433693485b0bbac0d1ef16:12
rlandy|ruckand16:12
rlandy|ruckhttps://review.rdoproject.org/zuul/build/4e68752c99b24931ba9bd9b6e37db7a516:12
weshay|ruck2020-07-21 08:25:17.531 | > delorean-python-stevedore-bc04ca91e07b067bcd19d 163 kB/s | 126 kB     00:0016:12
weshay|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-ironic-python-agent-master/bc4c8b8/build.log16:12
rlandy|ruckso it picked up the new stevedore16:12
weshay|ruck2020-07-21 08:27:28.659 | ---> Package python3-stevedore.noarch 3.1.0-0.20200717071436.bc04ca9.el8 will be installed16:12
rlandy|ruckbut idk that it picked up  https://review.rdoproject.org/r/#/c/28604/ update16:12
rlandy|ruckthat is what I am asking16:12
chandankumarrlandy|ruck: weshay|ruck fyi https://tree.taiga.io/project/tripleo-ci-board/task/189516:13
rlandy|ruckchandankumar: thanks16:13
rlandy|ruck    cloud8-openstack-victoria-testing: python-importlib-metadata-1.7.0-1.el816:13
rlandy|ruck- project: python-importlib-resources16:13
rlandy|ruck  buildsys-tags:16:13
rlandy|ruck    cloud8-openstack-victoria-testing: python-importlib-resources-3.0.0-1.el816:13
rlandy|ruck- project: python-influxdb16:13
rlandy|ruckidk how to tell if we have ^^16:14
rlandy|ruckI did not see any trace in the image build logs16:14
amoralejmm16:15
amoralejis it missing https://review.rdoproject.org/r/#/c/28585 ?16:15
amoralejthat was merged four days ago16:15
weshay|ruckya.. Requires:       python3-importlib-metadata16:17
weshay|ruck is missing16:17
amoralejmm16:18
amoralejlast stevedore build is python3-stevedore-3.1.0-0.20200717103921.bc04ca9.el8.noarch.rpm16:18
weshay|ruckI see it here https://trunk.rdoproject.org/centos8-master/component/clients/bc/04/bc04ca91e07b067bcd19d5d9ba4cdc61b0d7bd2f_f6c2a2cd/rpmbuild.log16:18
amoralejwhich is newer16:18
rlandy|ruckwe have that build16:18
amoralejthat the one in the log16:19
*** chandankumar is now known as raukadah16:19
rlandy|ruckmaybe the image build is missing including that?16:19
weshay|ruckso we need, DEBUG:   Verifying        : python3-stevedore-3.1.0-0.20200717103921.bc04ca9.el8   3/316:19
amoraleji'm checking the repos in the job16:19
weshay|ruck?16:19
weshay|ruckvs.. 3.1.0-0.20200717071436.bc04ca9.16:19
amoralejmm16:19
weshay|ruckamoralej, https://trunk.rdoproject.org/centos8-master/component/clients/consistent/16:21
weshay|ruckhttps://trunk.rdoproject.org/centos8-master/component/clients/consistent/python-stevedore-3.1.0-0.20200717103921.bc04ca9.el8.src.rpm16:21
amoralejyeah16:21
amoralejthat's good16:21
amoralejbut in the job it's using =https://trunk.rdoproject.org/centos8/component/clients/bc/04/bc04ca91e07b067bcd19d5d9ba4cdc61b0d7bd2f_f6c2a2cd16:21
weshay|ruckamoralej, rlandy|ruck we just need to promote clients16:21
rlandy|ruckconsistent wouldn't matter - tripleo-ci-testing would16:22
amoralejoh, right16:22
weshay|ruckrlandy|ruck, it's not a dep... it's clients16:22
rlandy|ruckhttps://trunk.rdoproject.org/centos8-master/tripleo-ci-testing/delorean.repo16:22
amoralejthat's component integration16:22
amoralejnot component promotion16:22
amoralejthat's the point16:22
rlandy|ruck[delorean-component-clients]16:22
rlandy|ruckname=delorean-python-stevedore-bc04ca91e07b067bcd19d5d9ba4cdc61b0d7bd2f16:22
rlandy|ruckbaseurl=https://trunk.rdoproject.org/centos8/component/clients/bc/04/bc04ca91e07b067bcd19d5d9ba4cdc61b0d7bd2f_3c8dd65616:22
rlandy|ruckenabled=116:22
rlandy|ruckin tripleo-ci-testing16:22
amoralejah16:22
rlandy|ruckwhy promote again?16:22
amoralejin the component promotions it uses ipa image from last integrated promotion, right?16:22
amoralejthat's why it was promoted16:22
amoralejrlandy|ruck, we need to get a more recent python3-stevedore16:23
weshay|ruckrlandy|ruck, because we don't have the proper version of stevedore16:23
weshay|ruckright16:23
amoralejwhich is not in the last client promotion16:23
weshay|ruckwhat he said16:23
rlandy|ruckok - too many stevedore patches flying around :)16:23
weshay|ruckamoralej, k.. to get the latest stevedore16:24
weshay|ruckwe need blazar_client https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-clients-master/d1870de/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz16:24
weshay|ruckwhich is...16:24
amoralejmmm16:24
weshay|rucka true dep16:24
weshay|ruckI think16:24
weshay|rucknot in dlrn16:25
amoralejthere was a patch somewhere for that16:25
amoralejykarel mentioned16:25
amoralejlemme check16:25
amoralejwasn't it heat?16:25
rascasoftwow rlandy|ruck weshay|ruck there are so many rucks in this world :D16:25
weshay|ruckrascasoft!!!16:25
rlandy|ruckrascasoft: hey there!!16:26
rlandy|ruckhow are you?16:26
weshay|ruckrascasoft, we're scoring on goal .. killin it16:26
rascasoftreally good thanks, always lurking you folks16:26
amoralejweshay|ruck, https://bugs.launchpad.net/tripleo/+bug/1887856/comments/616:26
openstackLaunchpad bug 1887856 in tripleo "centos-8 master tripleo component tests are failing with "ModuleNotFoundError: No module named 'blazarclient'"" [Critical,In progress] - Assigned to Ronelle Landy (rlandy)16:26
amoralejlatest heat and stevedore-3.1.0 was tested with https://logserver.rdoproject.org/29/28529/2/experimental/distgit-tripleo-master-testing-centos-8-standalone-scenario001/0785157/logs/undercloud/var/log/extra/package-list-installed.txt.gz, and it passed both heat-launcher and heat-engine container phase.16:27
weshay|ruckamoralej, don't see blazar in that list16:28
amoralejmmm, it seems we need to update heat and stevedore together16:28
amoralejweshay|ruck, the point is that it's actually not needed16:28
weshay|ruckk16:28
amoralejwith the right versions of heat and stevedore16:28
amoralejthe problem is that heat and stevedore are in different components16:28
weshay|ruckso perhaps we move stevedore from clients to tripleo16:28
amoralejthere is no way to promote two components together, right?16:29
weshay|ruckamoralej, we can... however if they need to be together16:29
weshay|ruckthen...16:29
rlandy|ruckthey only come together in the integration line16:29
weshay|ruckamoralej, both tripleo and clients component is failing16:29
weshay|ruckhttp://dashboard-ci.tripleo.org/d/FE8Hf29Wz/component-pipeline?orgId=1&from=now-7d&to=now16:29
weshay|ruckamoralej, /me thinks we update the component.. and kick off tripleo16:30
weshay|ruckwe'll be good to go16:30
amoralejyeah, they need to be updated toguether16:30
rlandy|ruckweshay|ruck: update the component but removing criteria to promote?16:30
rlandy|ruckby16:30
amoralejupstream gates doesn't use component promotions for jobs, right? only integration promoted16:31
weshay|ruckrlandy|ruck, no16:31
weshay|ruckjust move stevedore into tripleo16:31
weshay|ruckand we rekick the tripleo line16:31
amoralejweshay|ruck, yes, that would work, but i'm not sure if moving a package to a different component is so easy16:32
amoralejjpena, ^16:32
*** rascasoft has quit IRC16:32
weshay|ruckamoralej, jpena https://review.rdoproject.org/r/2864516:34
weshay|ruckamoralej, what is your concern?16:35
amoralejweshay|ruck, stevedore is used by many packages and clients16:35
amoraleji think it makes sense to keep in clients16:35
amoralejalso, i was discussing what's needed16:36
weshay|ruckamoralej, they all will have access16:36
amoralejafter moving16:36
amoralejwe need to force build of packages in both components16:36
jpenamoving stevedore to the tripleo component does not make a lot of sense from a logical grouping point of view16:36
amoralejwhich is not a problem16:36
amoralejweshay|ruck, if we force clients promotions manually somehow16:36
weshay|ruckjpena, if it's this tied to heat...16:36
weshay|ruckamoralej, f.. that16:36
amoralejweshay|ruck, it's tied to all services, i'd say16:37
jpenaI mean, I see the current issue, but moving packages around will create the same issue in the future with other packages (e.g. most/all clients depend on stevedore)16:37
weshay|ruckall the jobs are red.. I have no idea what the quality is16:37
jpenaso, if we have a catch-22 right now (clients and tripleo components cannot promote because they both depend on a fix to the other)16:37
weshay|ruckaye16:37
amoralejyep16:37
rlandy|ruckweshay|ruck: we could build a line with both components16:38
jpenawhy don't we break it by manually promoting one of the two components?16:38
rlandy|ruckie: two files that update16:38
rlandy|ruckand test16:38
weshay|ruckjpena, amoralej it's an option16:38
jpenaor what rlandy|ruck proposes: a (maybe temporary) job that would promote both at the same time16:38
weshay|ruckno16:38
weshay|ruckone at a time16:38
weshay|ruckrlandy|ruck, ok.. let's pick on https://meet.google.com/efi-eofx-xns?authuser=116:40
* weshay|ruck thinks promoting tripleo is a smarter play here16:40
* rlandy|ruck thinks testing is the smarter play here :)16:40
rlandy|ruckjoining16:40
weshay|ruckya.. let's see what we can do there as well16:41
amoralejweshay|ruck, note that the only difference between the last component-promoted clients and consistent is that build16:42
amoralejthe one we need16:42
amoralejno more packages updates16:42
amoralejso, manually promoting that repo should be pretty safe16:42
weshay|ruckamoralej, jpena we're going to establish a pattern for testing two components together in testproject.. should be pretty quick16:44
weshay|ruckamoralej, jpena thanks for the hand.. we have it from here16:44
amoralejcool, it'd be great to use this case to create that pattern16:44
weshay|ruck++16:44
amoralejwill happen again for sure16:44
weshay|ruckyup16:44
rlandy|ruckyeah - testing that pattern now16:54
rlandy|ruckamoralej: ^^ thanks16:54
amoralejuh, that was fast rlandy++16:55
weshay|ruckrlandy|ruck, https://hackmd.io/5uYRmLaOTI2raTbHWsaiSQ#Speculatively-test-two-unpromoted-components-together16:56
*** derekh has quit IRC17:01
*** amoralej is now known as amoralej|off17:05
*** jpena is now known as jpena|off17:05
*** yolanda has quit IRC17:10
*** yolanda has joined #oooq17:13
*** dtantsur is now known as dtantsur|afk17:15
*** jbadiapa has quit IRC17:36
rlandy|ruckweshay|ruck: k - back ... do you want to do the manual promotion of both one/both components ( when the job completes) or should I prep a patch to change criteria?17:40
weshay|ruckrlandy|ruck, let's just promote clients17:46
weshay|ruckand I'll retrigger tripleo17:46
weshay|ruckrlandy|ruck, I'll do it now... on the rdo promoter17:47
weshay|ruckin tmux17:47
rlandy|ruckk - ... getting tmux17:47
weshay|ruckhttps://meet.google.com/tkx-pcrw-iwh17:49
weshay|ruckrlandy|ruck, ^17:50
rlandy|ruckweshay|ruck: https://github.com/openstack/tripleo-common/blob/stable/ussuri/container-images/overcloud_containers.yaml.j218:04
weshay|ruckrlandy|ruck, gate failures are infra related18:33
rlandy|ruckweshay|ruck: infra fixable or infra "we will just have to live with this"?18:36
weshay|ruckrlandy|ruck, not fixable18:37
rlandy|ruckokie dokie18:37
rlandy|ruckweshay|ruck: OMG .. fs001 on master us deploying overcloud20:27
rlandy|ruckis20:28
rlandy|ruckno rerun20:28
*** TrevorV has quit IRC20:31
arxcruzweshay|ruck: lol when I saw the abandoned I was like: whaaaaaaat?20:47
*** jbadiapa has joined #oooq21:17
*** Goneri has quit IRC21:50
rlandy|ruckweshay|ruck: around?22:30
weshay|ruckrlandy|ruck, :)22:30
weshay|ruckhey.. had to move some bricks22:30
rlandy|ruckweshay|ruck: you see master run?22:30
weshay|ruckya22:31
rlandy|ruckso 30 failed22:31
weshay|ruckso .. 1 failed job in tempest22:31
rlandy|ruck001 is being reun22:31
weshay|ruck35 passed22:31
rlandy|ruck20!!!22:31
weshay|ruckah.. was just looking for that22:31
weshay|ruck20?22:31
weshay|ruck20 passed22:31
rlandy|ruckperiodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master22:31
weshay|ruckgett out of here22:31
rlandy|ruckI know22:31
rlandy|ruckwe should promote components more often22:31
weshay|ruckwow22:31
rlandy|ruckwhole new world22:32
weshay|rucklolz22:32
rlandy|ruckweshay|ruck: I am going to rerun 3022:32
weshay|ruckso.. if 20 passed we can waive 00122:32
rlandy|ruckbut in this case, I kind of think we can promote22:32
weshay|ruckya.. let's look at that22:32
* rlandy|ruck looks at 30 22:32
rlandy|rucksec22:32
weshay|ruckhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset030-master%0922:33
rlandy|ruck2020-07-21 20:36:00.676139 | primary | TASK [ensure the deployment result has been read into memory] ******************22:33
rlandy|ruck2020-07-21 20:36:00.676148 | primary | Tuesday 21 July 2020  20:36:00 +0000 (0:00:09.063)       0:33:37.303 **********22:33
rlandy|ruck2020-07-21 22:23:34.464342 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]22:33
rlandy|ruck2020-07-21 22:23:34.465149 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-periodic-base/post.yaml@master]22:33
weshay|rucktimed out in post?22:33
rlandy|rucksandeep saw this on internal22:33
weshay|rucktimed out right after deployment success22:33
rlandy|ruckyeah ... I think it's been there for a while22:34
weshay|ruckthat's scenario00722:34
weshay|ruckmultinode22:34
weshay|ruckrlandy|ruck, it only runs against tqe22:35
weshay|ruckhttps://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario007-multinode-oooq-container22:35
weshay|ruckso.. no big deal if it breaks22:35
rlandy|ruckweshay|ruck: just reran it22:36
weshay|ruckprobably most concerned w/ https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/2e0b656/logs/undercloud/var/log/tempest/stestr_results.html.gz22:37
weshay|ruckDetails: {'code': 404, 'message': 'No more floating IPs in pool public.'}22:37
weshay|ruckmeh22:37
weshay|rucktempest conf setting22:37
rlandy|ruckperiodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-mastertestprojectmastercheck21683,4256082020-07-21T20:58:23SUCCESS22:37
weshay|ruckDetails: {'type': 'IpAddressGenerationFailure', 'message': 'No more IP addresses available on network d790c45f-cc27-455d-b333-0179401e3ddd.', 'detail': ''}22:37
rlandy|ruckweshay|ruck: ^^ I reran it22:37
weshay|ruckoh nice22:38
weshay|ruckok22:38
rlandy|ruckas soon as it failed22:38
rlandy|ruckpassed above22:38
weshay|ruckso maybe we ping arxcruz to tune that22:38
rlandy|ruckweshay|ruck: want to change criteria so it will promote or wait on 30 to run?22:39
rlandy|ruckwe can switch 1 and 2022:39
rlandy|ruckwe can give it a few hours22:39
weshay|ruckhttps://trunk.rdoproject.org/api-centos8-master-uc/api/civotes_agg_detail.html?ref_hash=1ca4999bbce13cb4137cf00d0deab7fb22:39
weshay|ruckrlandy|ruck, if 20 passed.. that's good enough for 122:40
rlandy|ruckok22:40
weshay|rucksame job.. diff tempest setting22:40
weshay|ruckrlandy|ruck, ok.. so you want to revisit in the morning?22:41
weshay|ruckand try to promote?22:41
rlandy|ruckweshay|ruck: either way22:41
rlandy|ruckscenario010 still broken22:41
weshay|ruckya.. but it's nv atm22:41
rlandy|ruckDetails: (LoadBalancerScenarioTest:test_load_balancer_ipv4_CRUD) show_loadbalancer provisioning_status failed to update to ACTIVE within the required time 900. Current status of show_loadbalancer: PENDING_CREATE22:42
rlandy|ruckweshay|ruck: up to you22:42
weshay|ruckrlandy|ruck, http://dashboard-ci.tripleo.org/d/D6klBpGGz/non-voting-jobs?orgId=1&from=now-2d&to=now22:42
weshay|ruckrlandy|ruck, so is 00722:42
weshay|ruckmultinode22:42
weshay|ruckrlandy|ruck, I think we can promote22:43
rlandy|ruckok22:43
rlandy|rucktmux?22:43
weshay|rucksure22:43
weshay|ruck007-multinode is not in criteria22:44
weshay|rucklolz22:45
rlandy|ruckso just 03022:45
weshay|ruckmah ode22:45
rlandy|ruckit's ok22:46
rlandy|rucknot running22:46
weshay|ruckrlandy|ruck, ack to restart?22:46
rlandy|ruckweshay|ruck: go4it22:46
weshay|ruckrlandy|ruck, woot.. hit the right hash22:47
weshay|ruck:)22:47
rlandy|ruckand away we go22:48
weshay|ruckrlandy|ruck, ok dokie smokey22:48
weshay|ruckrlandy|ruck, fyi.. this is a handy page when things calm down http://dashboard-ci.tripleo.org/d/D6klBpGGz/non-voting-jobs?orgId=1&from=now-2d&to=now22:48
weshay|ruckto return things to voting22:48
rlandy|ruckweshay|ruck: k, going running - back in a bit to check22:48
weshay|ruck0/22:48
*** rlandy|ruck is now known as rlandy|ruck|bbl22:48
*** Goneri has joined #oooq22:59
*** Goneri has quit IRC23:36
*** Tengu has quit IRC23:45
*** Tengu has joined #oooq23:46
*** tosky has quit IRC23:47
« Monday, 2020-07-20 Index Wednesday, 2020-07-22 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!