Friday, 2021-06-11

*** opendevtest <opendevtest!~limnoria@104.239.144.232> has joined #oooq01:15
*** opendevmeet <opendevmeet!~limnoria@104.239.144.232> has joined #oooq01:23
*** opendevmeet is now known as Guest165301:23
*** opendevmeet` <opendevmeet`!~limnoria@104.239.144.232> has joined #oooq01:34
*** opendevmeet` <opendevmeet`!~limnoria@104.239.144.232> has joined #oooq01:39
*** opendevmeet` <opendevmeet`!~limnoria@104.239.144.232> has joined #oooq01:43
*** opendevmeet <opendevmeet!~limnoria@104.239.144.232> has joined #oooq02:00
*** apetrich_ <apetrich_!~apetrich@212.86.48.46> has joined #oooq02:06
*** apetrich <apetrich!~apetrich@79.140.122.89> has quit IRC (Ping timeout: 480 seconds)02:13
*** ysandeep|out is now known as ysandeep02:34
ysandeeprlandy: hi, thanks for leaving notes on email. 02:35
ysandeephttps://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/795838 lgtm, thanks!02:49
*** opendevmeet <opendevmeet!~limnoria@104.239.144.232> has joined #oooq03:04
*** pojadhav is now known as pojadhav|pto04:03
*** ysandeep <ysandeep!~sandy@202.173.126.121> has quit IRC (Ping timeout: 480 seconds)04:07
*** rlandy <rlandy!~rlandy@pool-108-6-195-195.nycmny.fios.verizon.net> has quit IRC (Quit: Leaving)04:16
*** ykarel <ykarel!~ykarel@2405:201:5c10:d062:7dc:c662:5028:45c8> has joined #oooq04:22
*** ysandeep <ysandeep!~sandy@202.173.126.121> has joined #oooq04:50
*** marios <marios!~marios@62-171-24.netrun.cytanet.com.cy> has joined #oooq05:37
*** apetrich_ <apetrich_!~apetrich@212.86.48.46> has quit IRC (Quit: Leaving)05:38
*** apetrich <apetrich!~apetrich@212.86.48.46> has joined #oooq05:38
*** marios is now known as marios|ruck05:41
*** jpodivin <jpodivin!~jpodivin@118.108.broadband14.iol.cz> has joined #oooq06:06
*** soniya29 <soniya29!~soniya@203.194.104.164> has joined #oooq06:26
*** soniya <soniya!~soniya@203.194.104.164> has joined #oooq06:37
*** amoralej <amoralej!~amoralej@153.red-80-26-161.dynamicip.rima-tde.net> has joined #oooq06:43
*** soniya <soniya!~soniya@203.194.104.164> has quit IRC ()06:47
*** soniya <soniya!~soniya@203.194.104.164> has joined #oooq06:47
*** soniya29 <soniya29!~soniya@203.194.104.164> has quit IRC (Quit: Leaving)06:52
ysandeepmarios|ruck, anbanerj|rover fyi.. added another improvement in ruck/rover script it can now check components, will post soon on gitlab after cleaning some linter issues -  http://paste.openstack.org/show/806559/07:05
*** soniya <soniya!~soniya@203.194.104.164> has quit IRC ()07:08
*** soniya <soniya!~soniya@203.194.104.164> has joined #oooq07:08
marios|ruckysandeep: thanks07:10
marios|ruckysandeep: dealing with this release thing https://bugs.launchpad.net/tripleo/+bug/1931223 will check in a bit07:10
ysandeepack o/ i just read your comment, make sense to me to tag all the master repos to avoid confusion.07:12
*** tosky <tosky!~luigi@dynamic-adsl-78-13-253-141.clienti.tiscali.it> has joined #oooq07:14
*** soniya <soniya!~soniya@203.194.104.164> has quit IRC (Quit: Leaving)07:14
*** soniya <soniya!~soniya@203.194.104.164> has joined #oooq07:15
*** soniya <soniya!~soniya@203.194.104.164> has quit IRC ()07:16
*** soniya <soniya!~soniya@203.194.104.164> has joined #oooq07:17
*** soniya <soniya!~soniya@203.194.104.164> has quit IRC (Remote host closed the connection)07:17
*** soniya29 <soniya29!~soniya@203.194.104.164> has joined #oooq07:17
anbanerj|roverMorning marios|ruck O/ , I'm checking the cix board07:20
marios|ruckanbanerj|rover: thank you good morning. upstream doesn't look to be on fire yet queue is reasonable but we should keep an eye on the fails (only 2 today so far )07:21
marios|ruckanbanerj|rover: so ack yeah we should try and make progress on cix and I will dig into the promotions after i finish this07:22
anbanerj|rovermarios|ruck, ok, lemme check rrcockpit. If there are repeated failures in check/gate I'll add a bug and maybe put it on health dashboard07:23
*** jpena|off is now known as jpena07:33
*** yamamoto <yamamoto!~yamamoto@2404:2d00:5400:61:3069:c574:ebba:477b> has quit IRC (Remote host closed the connection)07:37
*** yamamoto <yamamoto!~yamamoto@202.12.244.32> has joined #oooq07:41
*** ysandeep is now known as ysandeep|lunch08:06
*** opendevstatus is now known as Guest168408:09
*** anbanerj|rover <anbanerj|rover!~anbanerj@2a02:908:1878:3ce0::28d3> has quit IRC (Remote host closed the connection)08:10
*** anbanerj|rover <anbanerj|rover!~anbanerj@2a02:908:1878:3ce0:81aa:ae31:12c5:936e> has joined #oooq08:10
*** derekh <derekh!~derekh@2a01:b340:80:6d6e:6423:15e0:3703:2> has joined #oooq08:10
*** tosky <tosky!~luigi@dynamic-adsl-78-13-253-141.clienti.tiscali.it> has quit IRC (Ping timeout: 480 seconds)08:13
*** sshnaidm is now known as sshnaidm|afk08:24
*** tosky <tosky!~luigi@dynamic-adsl-78-13-253-141.clienti.tiscali.it> has joined #oooq08:24
*** ykarel is now known as ykarel|lunch08:31
*** opendevmeet <opendevmeet!~limnoria@104.239.144.232> has joined #oooq08:39
*** opendevmeet is now known as Guest168508:40
*** opendevstatus_ <opendevstatus_!~opendevst@104.130.70.91> has joined #oooq08:45
*** opendevstatus_ is now known as opendevstatus__08:47
*** opendevstatus__ is now known as opendevstatus___08:47
*** opendevstatus___ is now known as opendevstatus____08:47
*** opendevstatus____ is now known as opendevstatus_____08:48
*** opendevstatus_____ is now known as opendevstatus______08:48
*** opendevstatus______ <opendevstatus______!~opendevst@104.130.70.91> has quit IRC (Ping timeout: 480 seconds)08:58
*** Guest1684 <Guest1684!~opendevst@eavesdrop01.openstack.org> has quit IRC (Remote host closed the connection)08:58
*** ysandeep|lunch <ysandeep|lunch!~sandy@202.173.126.121> has quit IRC (Ping timeout: 480 seconds)08:59
*** opendevstatus_ <opendevstatus_!~opendevst@158.69.72.85> has joined #oooq09:13
*** opendevstatus_ is now known as opendevstatus__09:16
*** opendevstatus__ is now known as opendevstatus___09:16
*** opendevstatus___ is now known as opendevstatus____09:16
*** opendevstatus____ is now known as opendevstatus_____09:16
*** opendevstatus_____ is now known as opendevstatus______09:16
*** opendevstatus______ <opendevstatus______!~opendevst@158.69.72.85> has quit IRC (Ping timeout: 480 seconds)09:25
anbanerj|roverHey kopecmartin, we are trying to send a list of failed tempest tests to logstash so that we can run queries on it. When you get some time could you pl check if this can be merged https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/79526409:26
anbanerj|rovermarios|ruck, tripleo-ci-centos-7-containers-multinode-stein and tripleo-ci-centos-7-containers-multinode-queens are failing consistently. Checking if there is a bug already. 09:31
*** opendevstatus <opendevstatus!~opendevst@104.239.144.232> has joined #oooq09:31
*** opendevstatus <opendevstatus!~opendevst@104.239.144.232> has quit IRC (Remote host closed the connection)09:31
*** opendevstatus <opendevstatus!~opendevst@104.239.144.232> has joined #oooq09:31
marios|ruckanbanerj|rover: thank you bug #  ?09:31
marios|ruckanbanerj|rover: ah sorry i thought you said there *was* a bug ack you are checking thanks09:32
anbanerj|roverno, checking if there is one. Else I'll create09:32
*** ysandeep|lunch <ysandeep|lunch!~sandy@202.173.126.121> has joined #oooq09:41
*** ysandeep|lunch is now known as ysandeep09:47
*** opendevmeet <opendevmeet!~limnoria@104.239.144.232> has joined #oooq09:55
*** opendevmeet is now known as Guest168809:56
*** Guest1689 <Guest1689!~limnoria@104.239.144.232> has joined #oooq09:59
anbanerj|rovermarios|ruck, https://bugs.launchpad.net/tripleo/+bug/1931694 for tripleo-ci-centos-7-containers-multinode-stein09:59
*** Guest1690 <Guest1690!~limnoria@104.239.144.232> has joined #oooq10:02
*** opendevmeet <opendevmeet!~limnoria@104.239.144.232> has joined #oooq11:13
*** Guest1712 <Guest1712!~opendevst@104.239.144.232> has quit IRC (Remote host closed the connection)11:17
*** Guest1713 <Guest1713!~opendevst@149.202.169.13> has quit IRC (Ping timeout: 480 seconds)11:18
*** opendevstatus <opendevstatus!~opendevst@104.239.144.232> has joined #oooq11:18
*** soniya29 <soniya29!~soniya@203.194.104.164> has quit IRC (Remote host closed the connection)11:18
*** soniya29 <soniya29!~soniya@203.194.104.164> has joined #oooq11:19
kopecmartinanbanerj|rover: i replied on https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/79526411:21
*** jpena is now known as jpena|lunch11:27
*** yamamoto <yamamoto!~yamamoto@202.12.244.32> has joined #oooq11:40
*** opendevstatus_ <opendevstatus_!~opendevst@104.130.219.52> has joined #oooq11:40
*** opendevstatus_ is now known as opendevstatus__11:43
*** opendevstatus__ is now known as opendevstatus___11:43
*** opendevstatus___ is now known as opendevstatus____11:43
*** opendevstatus____ is now known as opendevstatus_____11:43
*** opendevstatus_____ is now known as opendevstatus______11:43
*** rlandy <rlandy!~rlandy@pool-108-6-195-195.nycmny.fios.verizon.net> has joined #oooq11:44
*** opendevstatus______ <opendevstatus______!~opendevst@104.130.219.52> has quit IRC (Ping timeout: 480 seconds)11:51
*** yamamoto <yamamoto!~yamamoto@202.12.244.32> has quit IRC (Ping timeout: 480 seconds)11:57
*** ykarel is now known as ykarel|afk12:07
*** whayutin <whayutin!~weshay|ru@c-73-229-75-146.hsd1.co.comcast.net> has joined #oooq12:09
whayutinanbanerj|rover, marios|ruck 0/ need anything?12:11
marios|ruckwhayutin: not currently - replied on the gchat to your pings with some info let me know if need sthing12:11
whayutincool.. /me looks.. I had never seen a review like that.. never heard of it12:12
*** opendevstatus_ <opendevstatus_!~opendevst@104.130.219.164> has joined #oooq12:12
marios|ruckwhayutin: yeah that one ate my time and gave me a heart attack this morning (was worried about conflicts) anyway info in the bug about that one with replies from me harald ykarel12:12
marios|ruckwhayutin: also added a note just now about master (new thread)12:13
marios|ruckwhayutin: tl;dr we have a good candidate though we may have to waive fs39 details at https://bugs.launchpad.net/tripleo/+bug/1931545/comments/312:13
*** opendevstatus_ is now known as opendevstatus__12:14
*** opendevstatus__ is now known as opendevstatus___12:15
*** opendevstatus___ is now known as opendevstatus____12:15
*** opendevstatus____ is now known as opendevstatus_____12:15
*** opendevstatus_____ is now known as opendevstatus______12:15
whayutinanbanerj|rover, do you need help debugging https://trello.com/c/mU1qeAkG/1996-cixlp1881087tripleociproa-tripleo-ci-centos-8-scenario010-standalone-timing-out-frequently12:15
*** opendevstatus <opendevstatus!~opendevst@104.239.144.232> has quit IRC (Remote host closed the connection)12:18
*** opendevstatus <opendevstatus!~opendevst@104.239.144.232> has joined #oooq12:18
whayutinmarios|ruck, what's the agg hash?12:22
*** opendevstatus <opendevstatus!~opendevst@104.239.144.232> has quit IRC (Remote host closed the connection)12:22
marios|ruckwhayutin:         * 2021-06-11 10:42:28,336 2256214 WARNING  promoter Candidate hash 'aggregate: 882777ccfc662a70d1c3ada62992ffb4, commit: 35f34ffdabe5869a612c2d266bdb01222fba428e, distro: b430c44187b5eee1c712aa5f2076c085a2094c14, extended: None, component: validation, timestamp: 1623370428': missing jobs {'periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master', 12:22
marios|ruck'periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master'}       12:23
marios|ruck        * http://10.0.148.74/promoter_logs/centos8_master.log12:23
marios|ruckwhayutin: ^^12:23
* whayutin looks https://trunk.rdoproject.org/api-centos8-master-uc/api/civotes_agg_detail.html?ref_hash=882777ccfc662a70d1c3ada62992ffb412:23
*** yamamoto <yamamoto!~yamamoto@202.12.244.32> has joined #oooq12:23
*** opendevstatus______ <opendevstatus______!~opendevst@104.130.219.164> has quit IRC (Ping timeout: 480 seconds)12:24
marios|ruckwhayutin: yes as i said i linked all of that in https://bugs.launchpad.net/tripleo/+bug/1931545/comments/312:24
whayutinmarios|ruck++ bhagyashris akahat please promote https://trunk.rdoproject.org/api-centos8-master-uc/api/civotes_agg_detail.html?ref_hash=882777ccfc662a70d1c3ada62992ffb412:24
bhagyashriswhayutin, ack12:25
whayutinbhagyashris, thanks12:25
marios|ruckwhayutin: ack but what does that mean "please promote" we don't know yet if we need to waive fs3912:25
marios|ruckwhayutin: do you mean manually promote that hash?12:25
whayutinbhagyashris, you only need to take out 03912:25
marios|ruckwhayutin: waiting for report from https://review.rdoproject.org/r/c/testproject/+/34057 12:26
marios|ruckwhayutin: i expect fs2 OK , but maybe fs39 need waive TBD (ipa job)12:26
*** opendevstatus <opendevstatus!~opendevst@104.239.144.232> has joined #oooq12:26
whayutinfs02 had already passed 12:26
marios|ruckthank you bhagyashris 12:26
marios|ruckwhayutin: did it? checking12:26
marios|ruckwhayutin: no that was yesterday you are looking at12:27
marios|ruckwhayutin: see current run @ https://review.rdoproject.org/zuul/status#34057 12:27
whayutinoh.. wait.. sorry12:27
whayutinbhagyashris, hold sec.. /me checks image server12:27
marios|ruckwhayutin: my fault i re-used that patch ;)12:27
marios|ruckwhayutin: i was running it yesterday for fs2 wallaby 12:27
bhagyashriswhayutin, ack12:27
whayutinbhagyashris, we're good http://images.rdoproject.org/centos8/master/rdo_trunk/882777ccfc662a70d1c3ada62992ffb4/12:28
ysandeepwhayutin, rlandy fyi.. we have a new request for octavia component in 16.2 , package is now at octavia current and will be picked in next component line run (In 2 hours)12:28
whayutinbhagyashris, images are there... you can proceed12:28
bhagyashrisyeah12:28
marios|ruckwhayutin: fs2 running now/still at https://review.rdoproject.org/zuul/stream/38ac03e9bcff4fe2b12dd1ce16f11ac3?logfile=console.log12:28
bhagyashrisok12:28
whayutinmarios|ruck, cool.. thank you /me watches12:28
marios|ruckwhayutin: yeah so we can be fairly certain that fs2 is OK but for the record it dind't report yet for master 12:28
whayutinyes.. I know.. but the images are avail12:29
*** ysandeep is now known as ysandeep|mtg12:29
marios|ruckwhayutin: ack12:29
whayutinthat's all that is 100% required.. the rest of that job is gravy12:29
*** jpena|lunch is now known as jpena12:30
rlandyysandeep|mtg: requests need to go on whayutin's doc12:31
whayutinysandeep|mtg, we'll need 2-3 people for design mtgs for version lock imho.. please include me as an extra12:31
ysandeep|mtgrlandy: delivery have added those requirements on weshay's doc12:32
rlandyok12:32
rlandythen we will work from there - thanks12:32
anbanerj|roverwhayutin, tripleo-ci-centos-8-scenario010-standalone is now hitting 'ClientManager' object has no attribute 'sdk_connection'12:32
anbanerj|roverit has gone from time out to fail now12:32
anbanerj|roverthis is the bug now https://bugs.launchpad.net/tripleo/+bug/1930811 12:33
whayutinanbanerj|rover, I don't see any errors in that bug12:33
ysandeep|mtgwhayutin, ack o/ 12:34
whayutinanbanerj|rover, needs more detail12:34
whayutinysandeep|mtg, howdy12:34
ysandeep|mtgwhayutin: i got good progress on script, we can now track component promotions: http://paste.openstack.org/show/806559/12:35
whayutinya.. /me looking at the code now12:35
*** opendevstatus_ <opendevstatus_!~opendevst@213.32.72.249> has joined #oooq12:41
anbanerj|roverwhayutin, periodic-tripleo-ci-centos-8-scenario010-standalone-train this one is timing out at TASK Configure octavia on overcloud12:42
anbanerj|roverhttps://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-train/0116bb3/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz12:42
anbanerj|rovertripleo-ci-centos-8-scenario010-standalone (non-voting) from upstream is failing with 'ClientManager' object has no attribute 'sdk_connection'12:43
*** opendevstatus_ is now known as opendevstatus__12:44
*** opendevstatus__ is now known as opendevstatus___12:44
*** opendevstatus___ is now known as opendevstatus____12:44
*** opendevstatus____ is now known as opendevstatus_____12:44
*** opendevstatus_____ is now known as opendevstatus______12:44
whayutinanbanerj|rover, marios|ruck we merged 64 patches in the last 24hrs... I think that's the most I've ever seen :)12:45
marios|ruckwhayutin: thankfully the computers did much of that heavy lifting ;)12:45
whayutinhttp://pastebin.test.redhat.com/97108612:45
whayutinaye..12:45
whayutinjust an indication for the most part we're hitting on all cylinders12:46
*** amoralej is now known as amoralej|lunch12:46
bhagyashriswhayutin, hi wes i am not sure but promoter is unable to promote master saying 021-06-11 12:29:09,429 2275967 WARNING  promoter Candidate hash 'aggregate: 882777ccfc662a70d1c3ada62992ffb4, commit: 35f34ffdabe5869a612c2d266bdb01222fba428e, distro: b430c44187b5eee1c712aa5f2076c085a2094c14, extended: None, component: validation, timestamp: 1623370428': missing jobs {'periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master'}12:46
marios|ruckwhayutin: been watching the queue today it has remained reasonable < 10 hrs12:46
bhagyashrismissing job fs00212:46
marios|ruckbhagyashris: whayutin: ack it didn't report yet12:46
whayutinbhagyashris, ya.. you can remove that one too.. 12:46
marios|ruckits what i was saying 12:46
whayutinwe have the image12:46
whayutinbut put 002/039 back when ur done12:46
bhagyashrisack12:47
whayutinysandeep|mtg, want to chat12:49
ysandeep|mtgwhayutin, http://paste.openstack.org/show/806559/12:50
bhagyashrisfyi http://10.0.148.74/promoter_logs/container-push/20210611-124006.log12:52
*** opendevstatus______ <opendevstatus______!~opendevst@213.32.72.249> has quit IRC (Ping timeout: 480 seconds)12:53
bhagyashris\0/ lucky ppc containers are also available on our source registry now i can test my change https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/33815 12:54
rlandyundercloud deploy takes forever these days 12:55
*** ykarel|afk is now known as ykarel12:57
*** yamamoto <yamamoto!~yamamoto@202.12.244.32> has quit IRC (Remote host closed the connection)13:04
*** yamamoto <yamamoto!~yamamoto@2404:2d00:5400:61:1d74:f60b:a382:1c0a> has joined #oooq13:08
*** opendevstatus_ <opendevstatus_!~opendevst@104.130.26.53> has joined #oooq13:12
*** opendevstatus_ is now known as opendevstatus__13:15
*** opendevstatus__ is now known as opendevstatus___13:15
*** opendevstatus___ is now known as opendevstatus____13:15
*** opendevstatus____ is now known as opendevstatus_____13:15
*** opendevstatus_____ is now known as opendevstatus______13:15
rlandywhayutin; let me know when you have a moment to discuss multinode options13:17
whayutinysandeep|mtg, https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf_py3/telegraf.d/last_promotions.conf13:20
whayutinysandeep|mtg, http://git.app.eng.bos.redhat.com/git/openstack/rrcockpit.git/tree/13:23
whayutinysandeep|mtg, http://git.app.eng.bos.redhat.com/git/openstack/rrcockpit.git/tree/files/telegraf_py3/telegraf.d/internal_last_promotions.conf13:23
*** opendevstatus______ <opendevstatus______!~opendevst@104.130.26.53> has quit IRC (Ping timeout: 480 seconds)13:24
*** amoralej|lunch is now known as amoralej13:28
anbanerj|roverwhayutin, https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/796005 and https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/796007 are two ways to add complex queries/with multiple params right now. 13:30
anbanerj|roverIf it fails tox I'll fix, but does the syntax make sense?13:30
anbanerj|roverI'll add to readme later13:31
marios|ruckanbanerj|rover: hopefully just a one-off but fyi 13:31
marios|ruckhttps://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/795742/1#message-665d3cc26c4e8085058c2c0f04c14368eea1f2d713:31
*** yamamoto <yamamoto!~yamamoto@2404:2d00:5400:61:1d74:f60b:a382:1c0a> has quit IRC (Remote host closed the connection)13:32
marios|ruckanbanerj|rover: doesn't looks like one off though     The user requested (constraint) ruamel-yaml===0.17.7     openstack-tempest-skiplist 0.0.1.dev275 depends on ruamel.yaml==0.16.12 https://zuul.opendev.org/t/openstack/build/d7a2b9fe2a0140b0965134a47a4cb45a 13:32
marios|ruck:/13:32
anbanerj|rovermarios|ruck, lemme check13:32
marios|ruckanbanerj|rover: but not consistent .. https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-py36 13:32
marios|ruckanbanerj|rover: thanks13:33
*** ysandeep|mtg is now known as ysandeep13:37
whayutinanbanerj|rover, this looks good.. imho let's merge and see what we get13:43
anbanerj|roverwhayutin, ok, once we see the graph I'll post a revert patch13:43
*** ysandeep is now known as ysandeep|out13:45
whayutinanbanerj|rover, meh.. let's leave it for it a bit as an example13:45
whayutinonce we post some new queries based on current issues that taake advantage of build_name and build_status we can remove it13:46
anbanerj|roverack13:46
anbanerj|rovermarios|ruck, looks like openstack-tox-* is failing on only that patch. I added 2 tripleo ci health patches. If it fails there I'll raise a bug13:49
marios|ruckanbanerj|rover: ack thanks for checking hyeah it looked like it was one-off but the trace suggested otherwise (package issues aren't typically one-off)13:54
marios|ruckanbanerj|rover: let's see 13:54
marios|ruckanbanerj|rover: thank you :)13:54
* marios|ruck steps away few mins screen break 13:55
anbanerj|rovermarios|ruck, looking for patches merged between yesterday and today on openstack-tox-* The last check job on your patch used 0.16.12, today it wants 17 for some reason13:56
anbanerj|roverwhayutin, where can I find openstack-tox-* base jobs or which patches merged since yesterday related to openstack-tox base jobs?14:01
*** yamamoto <yamamoto!~yamamoto@2404:2d00:5400:61:4931:20d7:c863:8a54> has joined #oooq14:04
*** soniya29 <soniya29!~soniya@203.194.104.164> has quit IRC (Remote host closed the connection)14:06
*** soniya29 <soniya29!~soniya@203.194.104.164> has joined #oooq14:06
anbanerj|rovermarios|ruck, did https://review.opendev.org/c/openstack/requirements/+/795270 break your tox jobs?14:07
marios|ruckanbanerj|rover: checking (but why only on mine in particular)14:08
anbanerj|roveryours or any tempest skiplist patches since yesterday should fail because of https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/requirements.txt#L3 and https://review.opendev.org/c/openstack/requirements/+/795270 I think14:10
marios|ruckanbanerj|rover: yeah its possible that is it the error is pointing at that version14:11
marios|ruck        *     The user requested (constraint) ruamel-yaml===0.17.714:11
marios|ruck        * https://zuul.opendev.org/t/openstack/build/2e9e4bc4a02c4b31892193ccb447032c14:11
marios|ruckanbanerj|rover: ^^ thanks 14:11
anbanerj|rovershould I update  https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/requirements.txt#L3 to 17?14:12
*** yamamoto <yamamoto!~yamamoto@2404:2d00:5400:61:4931:20d7:c863:8a54> has quit IRC (Ping timeout: 480 seconds)14:21
marios|ruckanbanerj|rover: sure if you like do it in https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/795742 update that or if you post a new patch we can rebase /795742 onto it14:22
marios|ruckanbanerj|rover++14:23
rlandywhayutin: ysandeep|out: 16.2 promotion is running now14:30
rlandyI assume we don;t have the BM patches yet?14:30
rlandypromoted components is 06/0814:30
rlandyhttps://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/79583814:31
rlandywe'd need to rerun fs001 jobs with ^^14:31
rlandynot yet merged14:31
whayutinrlandy, which agg hash?14:32
rlandyce8f8eb1e91aa2d6fea44bfaf15bf1c9 is running in tripleo-ci-testing14:33
marios|ruckrlandy: meant to say earlier and forgot thank you for dealin with the bm component promotion issue 14:33
rlandymarios|ruck: np - I broke it, I need to fix it14:33
*** ysandeep|out <ysandeep|out!~sandy@202.173.126.121> has quit IRC (Ping timeout: 480 seconds)14:38
whayutinrlandy, ya.. no news on bm, but checking something on octavia14:41
rlandyk - thanks14:41
whayutinrlandy, we're good on octavia14:44
rlandyhappiness is14:44
whayutinanbanerj|rover, merged your two patches.. 30-40  min wait for it to show up?14:46
*** soniya29 <soniya29!~soniya@203.194.104.164> has quit IRC ()15:02
marios|ruckykarel: o/ we have the same discussion with them every time but they are just doing a good job (they are right in theory but we do it differently with the major bumps in tripleo)15:17
marios|ruckykarel: https://review.opendev.org/c/openstack/releases/+/795924/1#message-c3afa86b8939505035a483b853e5edfd1844619c 15:17
*** pojadhav|pto is now known as pojadhav15:19
anbanerj|roverwhayutin, somehow the graphs didnt show up. /me Checking.15:21
anbanerj|rovermarios|ruck, updated https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/795742. Hope it passes now15:25
marios|ruckanbanerj|rover: looking15:26
marios|ruckanbanerj|rover: ack thank you 15:26
marios|ruckwhayutin: anbanerj|rover: i am starting shutdown sequence need something before I hit the friday button?15:27
anbanerj|rovermarios|ruck, all good from my side. Happy weekend!15:28
marios|ruckanbanerj|rover: you too and don't worry the fun continues on monday !15:28
* marios|ruck cries15:28
anbanerj|roverlol, yeah15:28
anbanerj|roverwhayutin, The graph titled "Bug 1929461 - yum repo mirrors are down 5/24/2021" is our graph, because I copy pasted a wrong bug URL in my query. Lemme update15:30
whayutinanbanerj|rover, k15:32
whayutinrlandy, have time if you want to chat15:32
rlandywhayutin: spoke with Alex and Tristan15:33
rlandywhayutin: trying out  https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/247154 Stop NM from updating dns15:33
rlandynot the best place to do this15:33
rlandybut a workable one15:33
rlandythis should be taken care of by either glean or tripleo-bootstrap but it's not15:34
rlandypossibly covering up in CI15:34
whayutinya.. why?15:34
rlandyhave a test run going now15:34
whayutinya15:34
whayutinlet's test w/ it..15:34
rlandywhayutin: see #tripleo15:34
rlandylooks like nobody cares too much to fix this15:35
rlandypossibly another artifact of virt-customize vs diskimage-builder15:35
rlandythis could hit the customer if and only if they run multinode15:36
rlandyso ...15:36
rlandymaybe it's ok to fix in it in CI15:36
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/3051e2839ab4455a80565823e0878b07?logfile=console.log15:36
rlandytest run there15:36
rlandywhayutin: in other news ...15:36
rlandyhttps://review.opendev.org/q/topic:%22custom-registry%22+(status:open%20OR%20status:merged)15:37
rlandytesting those to deal with multinode on 1715:37
whayutin[07:32:37] <mwhahaha> should probably update the way we do it in tripleo-bootstrap to be consistent with the way glean does it15:37
*** marios|ruck is now known as marios|out15:37
rlandywe probably want some bug and tracking on this15:37
rlandycurrent run will confirm if changing NM in pre will actually do the trick15:38
whayutinaye15:38
*** jpodivin <jpodivin!~jpodivin@118.108.broadband14.iol.cz> has quit IRC (Quit: Leaving)15:43
ykarelmarios|out, ack, Thanks15:44
*** ykarel is now known as ykarel|away15:44
rlandylooks like we are having issues with OVB downstream - checking15:48
*** ykarel|away <ykarel|away!~ykarel@2405:201:5c10:d062:7dc:c662:5028:45c8> has quit IRC (Ping timeout: 480 seconds)15:56
rlandyneed to pick something up - brb15:58
*** rlandy is now known as rlandy|brb15:58
*** ysandeep <ysandeep!~sandy@202.173.126.240> has joined #oooq16:04
*** ysandeep <ysandeep!~sandy@202.173.126.240> has quit IRC ()16:04
*** marios|out <marios|out!~marios@62-171-24.netrun.cytanet.com.cy> has quit IRC (Ping timeout: 480 seconds)16:11
*** yamamoto <yamamoto!~yamamoto@202.12.244.32> has joined #oooq16:12
anbanerj|roverwhayutin, updated https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/796007 with both the patches. 16:14
anbanerj|roverwhayutin, rlandy|brb anything else I should check before I leave?16:16
*** jpena is now known as jpena|off16:17
*** amoralej is now known as amoralej|off16:23
*** amoralej|off <amoralej|off!~amoralej@153.red-80-26-161.dynamicip.rima-tde.net> has quit IRC (Quit: Leaving)16:24
*** rlandy|brb is now known as rlandy16:29
whayutinrlandy, https://review.opendev.org/c/openstack/tripleo-heat-templates/+/79604616:29
rlandywhayutin: ack16:30
rlandynice16:30
rlandywhayutin: much easier override16:30
rlandytrying that16:30
rlandywhayutin: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/3051e2839ab4455a80565823e0878b07?logfile=console.log16:32
rlandylooks like it's working16:32
rlandyso ...16:32
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/3051e2839ab4455a80565823e0878b07?logfile=console.log16:32
rlandycan hold that and just use it for get patches through16:32
*** yamamoto <yamamoto!~yamamoto@202.12.244.32> has quit IRC (Ping timeout: 480 seconds)16:32
rlandybut I think it's our solution16:32
rlandyidk that they are going to fix this in 16.216:33
rlandywhayutin: at least we have something for two days of debug16:33
whayutinhas.. never seen tripleo-ci-centos-8-content-providerlost16:33
whayutinhas.. never seen tripleo-ci-centos-8-content-provider lost16:34
whayutinin the gate16:34
whayutineveryone say a prayer16:34
* rlandy grabs tehilim book16:34
rlandywhayutin: here you go ... https://rmbhcharities.com/lost-object/?campaign=2044821528&content=358148855260&keyword=jewish%20prayer%20for%20lost%20items&utm_term=jewish%20prayer%20for%20lost%20items&utm_campaign=Lost+And+Found&utm_source=adwords&utm_medium=ppc&hsa_acc=5746379462&hsa_cam=2044821528&hsa_grp=72699258419&hsa_ad=358148855260&hsa_src=g&hsa_tgt=kwd-331014156148&hsa_kw=jewish%20prayer%20for%20lost%20items&hsa_mt=b&hsa_net=a16:36
rlandydwords&hsa_ver=3&gclid=EAIaIQobChMIgeyPwIGQ8QIVrP_jBx06mgAtEAAYASAAEgKfzfD_BwE16:36
rlandyTefillah for Finding Lost Item - Rabbi Meir Baal Haness16:36
whayutingot any of dem tiny urls16:36
whayutinah16:36
whayutinlolz16:36
whayutinnice16:36
whayutinwas that on s/codesearch/prayersearch/g16:36
rlandytop hit when you google prayer for lost item16:37
rlandyI mean you don;t need that campaign16:37
rlandyjust the Rabbi Meir Baal Haness piece16:37
rlandymy sister says it works ?!16:37
whayutinthanks bhagyashris 16:45
rlandywhayutin: looks like OVB is struggling donwstream16:45
whayutinI'll watch it from here16:45
rlandygoing to recheck those jobs after the line is done16:45
whayutinrlandy, clean tenant?16:46
rlandyisk - not when I last checked16:46
rlandyall nodes were new16:46
rlandymaybe overloade16:46
rlandyor no nodes available16:46
whayutinrlandy, aye.. if bm passes we can waive16:46
rlandygoing to try again this afternoon16:46
rlandywhayutin: ack - multinode is passing with the patch16:46
whayutinrlandy, NICE.. so let's file a bug.. and look at bootstrap16:47
whayutinrlandy, we can adjust it and patch https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_bootstrap16:48
whayutinwe can probably waive it too as we know what it is.. and what it impacts16:48
whayutinas long as it's bugged16:48
rlandywhayutin: filing bug16:48
rlandywe can always rerun the job with the patch is needed16:49
rlandyor accept the patch while we debug16:49
rlandywrt patching tripleo-bootstrap16:49
rlandyidk why yet it is not being applied16:49
rlandyI need to debug that more16:49
rlandyI just know what the problem is16:49
whayutinmaybe here:16:49
whayutinhttps://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_bootstrap/tasks/main.yml#L6616:49
whayutinturn of nm dns right before legacy network16:50
rlandyfirstly we should look at the train one16:50
whayutinoh ya16:50
whayutinsorry16:50
rlandyand compare16:50
rlandymaybe there is an error just in that one16:50
rlandybut upstream works fine16:50
rlandyhttps://opendev.org/openstack/tripleo-ansible/src/branch/stable/train/tripleo_ansible/roles/tripleo-bootstrap/tasks/main.yml#L7116:51
rlandywhayutin: ^^ that one16:51
whayutinhttps://opendev.org/openstack/tripleo-ansible/src/branch/stable/train/tripleo_ansible/roles/tripleo-bootstrap/tasks/main.yml#L5616:51
whayutinoh ya16:51
whayutin++16:51
whayutinrlandy, maybe just change the when16:52
whayutinif the file exists16:52
* rlandy opens bug first - to which component?16:52
whayutinrlandy, tripleo16:52
rlandytripleo-ansible16:52
whayutinya16:52
rlandyhttp://git.app.eng.bos.redhat.com/git/tripleo-ansible.git/tree/tripleo_ansible/roles/tripleo-bootstrap/tasks/main.yml?h=rhos-16.2-trunk-patches16:54
rlandythere in 16.2 trunk patches16:55
rlandyso we don't have a missing bakcport16:55
*** derekh <derekh!~derekh@2a01:b340:80:6d6e:6423:15e0:3703:2> has quit IRC (Quit: Leaving)17:00
rlandywhayutin: want this https://bugzilla.redhat.com/show_bug.cgi?id=1971030 CIX'ed?17:09
rlandyhmmm also struggling for nodes in rdo 17:15
whayutinrlandy, yes.. it's blocking promotins17:34
whayutinpromotions17:34
rlandybreaking the world17:34
rlandyCIX'ing17:34
whayutinanbanerj|rover, shall we merge?17:45
whayutinhttps://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/79600717:45
-opendevstatus- NOTICE: Zuul is being restarted for server reboots17:46
whayutinha17:47
rlandyugh17:51
rlandytrying to get component image build patch though17:52
rlandyfor baremetal17:52
whayutinrlandy, we don't need bm17:52
rlandywhayutin: yeah - but I want this fixed17:52
whayutink ++17:52
whayutinppc manifests17:54
rlandyrdo stuck17:57
rlandychecking out17:57
rlandyjobs haven't started in over hour17:57
whayutin882777ccfc662a70d1c3ada62992ffb4_ppc64le17:58
whayutindocker pull tripleomaster/openstack-base:882777ccfc662a70d1c3ada62992ffb4_ppc64le17:58
whayutinLast pushedan hour agobyrdotripleomirror17:58
whayutinDIGEST17:58
whayutinOS/ARCH17:58
whayutinCOMPRESSED SIZE 17:58
whayutin72cb9f492eae17:58
whayutinlinux/ppc64le17:58
rlandynice17:58
rlandylooks like infa knows about rdo nodes18:00
rlandyso ok18:00
*** ykarel|away <ykarel|away!~ykarel@2405:201:5c10:d062:7dc:c662:5028:45c8> has joined #oooq18:05
*** ykarel|away <ykarel|away!~ykarel@2405:201:5c10:d062:7dc:c662:5028:45c8> has quit IRC (Ping timeout: 480 seconds)18:14
*** dviroel <dviroel!uid349012@id-349012.stonehaven.irccloud.com> has quit IRC (Quit: Connection closed for inactivity)18:16
rlandylunch - back in a few18:21
whayutinrlandy, k.. check ur email from me.. and ping me.. we should double check master and see if we can manually promote safely18:24
whayutinbrb.. helping neighbor18:27
*** pojadhav <pojadhav!~poojajadh@103.149.196.157> has quit IRC (Remote host closed the connection)18:31
*** pojadhav <pojadhav!~poojajadh@103.149.196.157> has joined #oooq18:31
rlandywhayutin: reading18:34
whayutinrlandy, k.. back18:41
rlandywhayutin: let's chat18:41
whayutink18:41
rlandynot sure what none hash is18:41
rlandyoh18:42
whayutinmeet.google.com/kyc-nddy-pnz18:42
whayutinupstream18:42
whayutinrlandy, ^18:42
whayutinrlandy, https://trunk.rdoproject.org/centos8-master/tripleo-ci-testing/88/27/882777ccfc662a70d1c3ada62992ffb4/19:02
whayutinrlandy, https://github.com/rdo-infra/ci-config/commit/6e9a9bed5f463c38b96b707a40668f6b326b7e48#diff-2af95aca33cf215e1e0959fa8eebec98cf87f70b9ef5b23c5f3f1d0284538b7419:29
*** dviroel <dviroel!uid349012@id-349012.stonehaven.irccloud.com> has joined #oooq20:28
rlandywhayutin: failed again ... but rdo is down20:28
rlandyand probably dlrn as well20:29
*** jbadiapa <jbadiapa!~jbadiapa@1.red-83-46-246.dynamicip.rima-tde.net> has quit IRC (Quit: Leaving)21:54
rlandywhayutin: hi - can we promote 16.2?21:57
rlandycommon multinode is still failing21:58
rlandythink it needs updated network21:58
whayutinrlandy, heh22:02
rlandywhayutin; hi there22:02
whayutinsorry that took so much longer22:03
whayutinugh22:03
rlandyno problem22:03
rlandywell actually we have problems22:03
whayutindidn't work eh22:03
rlandydlrn is down22:03
rlandyrdo is down22:03
whayutinah.. dlrn is?22:03
whayutinit's on aws22:03
rlandydownstream dlrn is also wacked22:03
whayutinmeh.. the heck w/ it then22:03
whayutingo have a weekend22:03
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-rhos-16.2-component-tripleo-promote-consistent-to-component-ci-testing/49db30e/job-output.txt22:04
rlandywhayutin: ^^22:04
rlandy<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>22:04
whayutin2021-06-11 16:34:04.799505 | primary | dlrnapi_client.rest.ApiException: (500)22:04
whayutinawesome22:04
rlandywhayutin: that's internal22:04
whayutinya22:04
rlandybut wrt rdo dlrn maybe that is also out to lunch22:04
rlandymaybe that is why can't find the hash combination22:04
rlandysame error22:04
rlandywith the change22:05
rlandyjason said we have until wed to promote22:05
whayutinfun22:06
rlandyoy vey infra22:07
rlandywhayutin: I'm around sunday if you want to try again then22:07
rlandywill have to try the common and network component together to get multinode to promote22:08
rlandyyou trying dlrn? 22:09
whayutindepends... I have some time now.. but kids..22:10
whayutinya.. was trying curling to it22:11
rlandywhayutin: k - let's give it a rest today .. email me if you want to give these promotions a try on sunday22:11
whayutinit's up22:12
whayutinbut ya22:12
whayutinit's not worth it22:12
rlandyI have some time22:12
rlandyuntil 7pm my time22:12
*** tosky <tosky!~luigi@dynamic-adsl-78-13-253-141.clienti.tiscali.it> has quit IRC ()22:13
rlandybut our change made no diff22:13
rlandyidk if it just can't source any hashes22:13
rlandycan't determine22:13
whayutinare you in the file?22:16
rlandyno22:17
*** pojadhav <pojadhav!~poojajadh@103.149.196.157> has quit IRC (Remote host closed the connection)22:27
*** pojadhav <pojadhav!~poojajadh@103.149.196.157> has joined #oooq22:28
rlandygetting anywhere>22:28
*** yamamoto <yamamoto!~yamamoto@2404:2d00:5400:61:c050:7b36:3e56:a104> has joined #oooq22:30
rlandywhayutin: I don't want to speak too soon, but ...22:37
rlandyhttps://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-on-multinode-ipa22:37
rlandymore stable since last change merged on 06/1022:37
* rlandy out22:39
*** rlandy <rlandy!~rlandy@pool-108-6-195-195.nycmny.fios.verizon.net> has quit IRC (Quit: Leaving)22:40
*** yamamoto <yamamoto!~yamamoto@2404:2d00:5400:61:c050:7b36:3e56:a104> has quit IRC (Remote host closed the connection)22:41
*** yamamoto <yamamoto!~yamamoto@2404:2d00:5400:61:c050:7b36:3e56:a104> has joined #oooq22:41
*** yamamoto <yamamoto!~yamamoto@2404:2d00:5400:61:c050:7b36:3e56:a104> has quit IRC (Remote host closed the connection)23:14
*** yamamoto <yamamoto!~yamamoto@202.12.244.32> has joined #oooq23:17

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!