Tuesday, 2022-08-16

*** ysandeep|holiday is now known as ysandeep05:28
*** ysandeep is now known as ysandeep|ruck05:29
ysandeep|ruckgood morning oooci o/05:30
ysandeep|ruckchandankumar: Thank you! so much for covering ruck/rover yesterday :)05:31
chandankumarysandeep|ruck: marios good morning o/05:38
chandankumarysandeep|ruck: happy to help :-)05:38
chandankumarysandeep|ruck: let me know when free, we can sync05:38
ysandeep|ruckchandankumar, we can sync now if you are available05:40
chandankumarysandeep|ruck: yes sure05:40
ysandeep|ruckchandankumar, meet.google.com/qsx-vqkj-pxv05:41
marioso/ ysandeep|ruck 05:42
marioschandankumar: \o05:42
ysandeep|ruck\o/05:43
marios:)05:44
chandankumarysandeep|ruck: marios https://review.opendev.org/c/openstack/tripleo-quickstart/+/852914 please have a look05:56
*** evallesp is now known as evallesp|afk06:18
*** amoralej|off is now known as amoralej06:18
Tenguysandeep|ruck: heya! any news about the OC with rebuilt image? fyi, Takashi pointed another repo for the "nftables" package addition, leading to: https://review.opendev.org/c/openstack/tripleo-puppet-elements/+/85322406:21
mariosTengu: what happened with the tripleo-common image-yaml change? 06:22
ysandeep|ruckTengu, infra was in bad shape over the weekend.. will try today06:22
Tengumarios: dropped, replaced by the other one I just pointed06:22
mariosah i see tpe was better fix06:22
Tenguysandeep|ruck: heh - no problem. Maybe update the Depends-On then06:22
Tengumarios: yup :)06:23
Tenguback from extended weekend, already all on fire06:23
Tengu:]06:23
marioschandankumar: done06:23
dpawlik4chandankumar: hey, I saw that you ping me06:30
dpawlik4is it ok now?06:30
chandankumardpawlik: tristan fixed it yesterday, all good now, thank you :-)06:31
dpawlikcool06:33
*** evallesp|afk is now known as evallesp06:50
Tengumarios: zuul's all green for the tpe patch adding nftables. I checked the log, and we indeed see the package being installed.06:53
mariosTengu: ack will revisit 07:06
ysandeep|ruckykarel++ thank you!07:09
ykarelyw07:11
Tenguthanks marios :)07:15
Tenguysandeep|ruck: we should soon get actual "official" OC images with nftables.07:15
Tengufor real.07:15
Tengumarios: guess we'll need some kind of promotion in order to get the acutal OC image with nftables?07:22
ysandeep|ruckTengu, sry was in a mtg to sort downstream infra situation07:23
ysandeep|ruckTengu, fyi..07:23
ysandeep|ruck~~~07:23
ysandeep|ruck      featureset_override:07:23
ysandeep|ruck        to_build: true07:23
ysandeep|ruck~~~07:23
ysandeep|ruck^^ we can add this in testproject and it will build image in job itself07:24
ysandeep|ruckwe don't have to wait for testing07:24
Tenguysandeep|ruck: ah, cool! though I'm pretty sure I'll have to do some local testing for weird package investigations ;). we'll see.07:24
* ysandeep|ruck updates the testproject07:24
Tenguysandeep|ruck: had to restart my firefox last week following updates, care to re-link the testproject?07:24
mariosTengu: available for gate jobs yeah 07:25
Tengumarios: 'k. well, we'll see when we get it.07:25
mariosTengu: otherwise yeah as ysandeep|ruck pointed you can ask for it to build and use for rdo/testing 07:25
mariosTengu: but for gates yeah you'll need a promotion with the new change 07:26
Tengumarios: what's the status of last week blocker(s) btw? things are under control?07:26
Tenguor may I help?07:26
ysandeep|ruckTengu, https://review.rdoproject.org/r/c/testproject/+/31954 this was the testproject  (last testing we did was for upgrade but we enabled nftables only for UC)- let me update it for a ovb job 07:27
mariosTengu: so ysandeep|ruck is the new ruck. yesterday we had a new blocker on rdo which cost us 2x days of promotions but was fixed midday 07:27
Tengumarios: 'k07:27
mariosTengu: i believe gate is OK now but promotions still not green (though ruck rover are constantly chasing and keeping it kind of green with the promotion frequency)07:27
Tenguysandeep|ruck: thanks :)07:27
Tengumarios: sounds under control, more or less, then :)07:28
mariosTengu: well as much as it ever is ;)07:28
* marios prepares sacrifices for zuul07:28
Tenguysandeep|ruck: wait - the upgrade was good? or was it the run where we still end with iptables?07:28
Tengumeh. ok. still good old iptables. pfrrrtt.07:29
Tenguysandeep|ruck: hmm, depends-on isn't good: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85196207:30
Tenguwe want https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85280807:30
Tengulemme edit it.07:30
ysandeep|ruckyes, I am updating atm to avoid midair conflict07:31
Tenguoh, ok. I give you the hand07:31
ysandeep|ruckTengu, upgrade ran here in your patch already: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/852808/4#message-88df3a7c77c9685ac7d3ee68e027525032220cda 07:33
Tenguoh, right.07:33
Tenguand the failure was.... lemme investigate.07:33
Tenguah. networkmanager.07:33
Tengumirror issue then,07:33
Tenguunrelated.07:34
Tenguwe'll see with the current run.07:34
*** jpena|off is now known as jpena07:34
Tenguhmmmmmmmmmm I think I'll need to edit the tripleo_bootstrap to inject nftables.07:34
Tenguthough, for #reasons, it seems to be installed by default on the UC right now...07:34
ysandeep|ruckTengu, This ssh error: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c81/852808/2/check/tripleo-ci-centos-9-scenario007-multinode-oooq-container/c815403/logs/undercloud/home/zuul/overcloud_deploy.log , what's the fix for this one?07:38
ysandeep|ruckI don't think we use overcloud images for this job.07:38
Tenguysandeep|ruck: seems to still be using iptables: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c81/852808/2/check/tripleo-ci-centos-9-scenario007-multinode-oooq-container/c815403/logs/subnode-1/var/log/extra/network.txt07:40
Tenguysandeep|ruck: and it should be accepted:   -A openstack-INPUT -p tcp -m tcp --dport 22 -j ACCEPT07:41
Tengudon't think it's firewall related at this point07:41
ysandeep|ruckLet me chat if tht have correct driver07:41
Tengusure07:42
ysandeep|ruckcheck*07:42
ysandeep|ruckTengu, https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c81/852808/2/check/tripleo-ci-centos-9-scenario007-multinode-oooq-container/c815403/logs/undercloud/home/zuul/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/tripleo-firewall/tripleo-firewall-baremetal-ansible.yaml07:43
ysandeep|ruck~~~07:43
ysandeep|ruck  FirewallEngine:07:43
ysandeep|ruck    default: 'nftables'07:43
ysandeep|ruck~~~07:43
ysandeep|ruckTengu, do you have few mins for gmeet? meet.google.com/krk-kmnz-wem07:44
Tenguysandeep|ruck: sure, sorry, was downstairs.07:49
ysandeep|ruckTengu, https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-scenario007-multinode-oooq-container&skip=007:58
*** ysandeep|ruck is now known as ysandeep|ruck|lunch08:05
* pojadhav stepping out for few hours08:16
*** ysandeep|ruck|lunch is now known as ysandeep|ruck08:28
akahatmarios, o/ 08:50
akahatmarios, I've added compute component job: https://review.rdoproject.org/r/c/rdo-jobs/+/44415 08:51
akahatmarios, are we going to add it for other components?08:52
mariosakahat: hi looking08:53
marioscool i'll check it out (can you add testproject info there so i can  see it running08:54
mariosakahat: yeah maybe tripleo and common but not sure lets get compute one in first and we can discuss 08:55
akahatmarios, ok. I've added Testproject links in the comments.08:56
Tenguysandeep|ruck: OC nodes are being provisioned.... let's see.09:10
*** ysandeep|ruck is now known as ysandeep|ruck|afk10:46
Tenguysandeep|ruck|afk: SUCCESS! https://logserver.rdoproject.org/54/31954/66/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/f79e0ef/logs/overcloud-controller-0/var/log/extra/nftables.txt.gz11:03
TenguOC with nftables is apparently a thing, and we're all green for the multinode job you had in the testproject!11:03
Tenguwoohooo11:03
Tengumarios: -^^   good news!11:04
mariosTengu: nice 11:09
mariosTengu: so is the 40 mins thing gone then? 11:09
mariosTengu: will check noting for now 11:09
Tengumarios: I can check, I have the log directory in the browser, gimme a minute.11:09
Tengumarios: we're apparently under the MINUTE to run the tripleo_firewall role.11:10
mariosTengu: nice11:11
Tengu2022-08-16 09:22:26 | 2022-08-16 09:22:26.136684 | fa163eb5-24b1-a7bb-735a-000000002288 |       TASK | Run firewall role11:11
Tengu2022-08-16 09:22:37 | 2022-08-16 09:22:37.378843 | fa163eb5-24b1-a7bb-735a-0000000025ee |     TIMING | tripleo_nftables : Reload custom nftables ruleset WITH jumps | overcloud-controller-2 | 0:02:42.491790 | 0.37s11:11
Tengulast apparition.11:11
Tenguguess we're even better than before the 40 minutes thing appeared.11:12
Tengusooo. yeah. problem solved.11:13
Tengu:]11:13
*** ysandeep|ruck|afk is now known as ysandeep|ruck11:17
ysandeep|ruckafaik.. 40 mins.. I think that was noticed in sc001 /me not sure if we were seeing in ovb jobs too.11:31
* ysandeep|ruck will run sc001 with depends-on as well to confirm11:32
Tenguysandeep|ruck: ah, yeah, would be good to get an sc001 with that then. Still, I'm pretty confident we'll be under the minute as well.11:35
Tenguysandeep|ruck: don't forget to get the OC image built as well :)11:35
ysandeep|ruckTengu, OC images will be built automatically in the next scheduled run 11:36
ysandeep|ruckonce tripleo component promotes.. so by tomorrow11:36
*** dviroel_ is now known as dviroel11:38
Tenguysandeep|ruck: yep - in the meanwhile, testproject will need to rebuild with HEAD content.11:41
ysandeep|ruckTengu, ack o/ I am trying to do a master promotion before tomorrow program mtg, as soon as I get a master promotion - I will trigger the line manully so that we rebuild with HEAD content.11:43
Tenguysandeep|ruck: lemme know if I can help :)11:43
ysandeep|ruckthanks!11:43
ysandeep|ruckmarios, chandankumar fyi.. we will be meeting Dev Productivity team tomorrow - for prow in downstream discussion - Do you want to be part of that discussion? I can invite you to that mtg as well.11:45
chandankumarysandeep|ruck: yes sure11:45
ysandeep|ruckchandankumar, marios I have added you guys.. If you have questions for this team - please add here: https://hackmd.io/WDRaF2v6TumwurAD9DXPuw#QuestionsDoubts-for-DPTP-Teamforum-testplatform-team-on-17th-Aug 11:47
ysandeep|ruckdviroel, we need to modify(i.e 15 mins late)/cancel - Operator testing kickoff mtg11:48
ysandeep|ruckdviroel, errr.. I don't think we can modify that mtg, its owned by rlandy11:50
dviroelysandeep|ruck: i see, we will need to notify everybody that we will skip this time.11:52
ysandeep|ruckdviroel, yeah I don't see a point in mtg twice tomorrow.. so lets skip "Operator testing kickoff"11:53
dviroelysandeep|ruck: ack11:54
mariosthanks ysandeep|ruck 11:57
Tenguysandeep|ruck: btw - I'm pretty confident an upgrade from iptables to nftables will be just invisible due to the active cleaning of existing rules.12:11
ysandeep|ruckTengu, nice!12:12
Tenguoh, and once we get nftables as default, I'll be able to remove the useless "drop" jump in the default rules :).12:12
Tenguthanks to the chain policy.12:12
Tenguand then, no more ordering issues.12:12
*** amoralej is now known as amoralej|lunch13:00
*** jgilaber_ is now known as jgilaber13:30
bhagyashrisarxcruz, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, anbanerj, dviroel, rcastillo, dasm, jm1, marios ysandeep|ruck rcastillo|rover 13:30
bhagyashrisTripleo ci community meeting 13:30
bhagyashrishttps://meet.google.com/igc-nxwj-gws?authuser=013:30
bhagyashrishttps://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both#2022-08-16-Community-Call @ line 2913:31
* pojadhav facing fluctuation in network...13:31
rcastillo|rovero/13:48
rcastillo|roverysandeep|ruck: rr sync?13:48
pojadhavthank you bhagyashris for driving community call !!13:49
ysandeep|ruckhey rcastillo|rover o/ give me few mins.. need to push a patch first.. will ping you in few13:49
rcastillo|roverysandeep|ruck: ack13:49
*** amoralej|lunch is now known as amoralej13:54
*** dasm|off is now known as dasm13:55
dasmo/13:55
frenzyfridayhey arxcruz, rcastillo|rover aoc call14:01
ysandeep|ruckrcastillo|rover, if you want to join aoc mtg, please go ahead.. I am around for ~1 more hour.. so we can sync later.14:14
* ysandeep|ruck trying to bring downstream back in shape atm14:14
chandankumarysandeep|ruck++ dpawlik++ thank you for fixing the downstream node failures :-)14:29
ysandeep|ruckdasm, I am seeing some error - missing jobs in rr script https://paste.openstack.org/show/bSQ09lbMZyqaBUMD493Q/ , Could you please take a look when you get a chance.14:29
dasmysandeep|ruck: checking14:33
dasmysandeep|ruck: it's because of long times it was getting to pull from your side of the world.14:34
dasmysandeep|ruck: it should be renamed differently.14:35
dasmysandeep|ruck: because we're not querying zuul for jobs details, it returns empty dict, hence no details for those jobs.14:35
dasmit's innocuous14:36
dasmysandeep|ruck: does it make sense?14:36
ysandeep|ruckdasm, may be we can reduce the log to debug instead of error in this case?14:36
dasmysandeep|ruck: error and warning both land on the console. i might need to change that to "info" then.14:37
ysandeep|ruckack, sounds good to change to info14:37
dasmysandeep|ruck: https://paste.opendev.org/show/bqcQ8sxWNCRfn3FRdLqA/14:38
ysandeep|ruck10s wow.. i wish these server are closer to me14:40
dasmysandeep|ruck: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4453414:40
* ysandeep|ruck thinking about creating a psi instance and runnning rr script from there.14:40
dasmysandeep|ruck: yeah, that's why i've been pushing all changes to rr. Those speed gains!14:41
ysandeep|ruckdasm, thanks +wed14:41
dasm++14:42
ysandeep|ruckchandankumar, marios rcastillo|rover wohoo. ovb passed the earlier failing step.. 14:42
ysandeep|ruck~~~14:42
ysandeep|ruck2022-08-16 14:42:13.004231 | TASK [ovb-manage : Find out UUID of instance with metadata URL]14:42
ysandeep|ruck2022-08-16 10:42:14.892154 | primary | 691422fe-9061-463d-84a1-41c30b9072e214:42
ysandeep|ruck2022-08-16 14:42:16.031186 | primary | ok: Runtime: 0:00:02.34279914:42
ysandeep|ruck~~~14:42
chandankumarysandeep|ruck: awesome, great work man \o/14:42
dasmysandeep|ruck: psi instance might be a good idea. it might be even shared for all of us14:43
dasmysandeep|ruck: actually, cockpit is already there, so as well we might use it too.14:43
* ysandeep|ruck can work on ovb-manage role with proper testing now.14:43
dasmysandeep|ruck: we would need to update dockerfile to incorporate other templates, and that would be all14:43
ysandeep|ruckdasm, we used to have a staging cockpit - i think we can use that instead of prod cockpit.14:44
ysandeep|ruckrcastillo|rover, let me know when you want to sync14:44
rcastillo|roverysandeep|ruck: I'm ready14:45
ysandeep|ruckrcastillo|rover, meet.google.com/kaj-xxku-uew14:47
mariosakahat: please see comment @ https://review.opendev.org/c/openstack/tripleo-quickstart/+/852733/1/config/release/tripleo-ci/CentOS-8/promotion-testing-hash-wallaby.yml once you update i can test with it (and switch containers to use same in another place)14:59
dasmysandeep|ruck: fyi https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4453515:01
*** amoralej is now known as amoralej|off15:01
ysandeep|ruckdasm, ack.. I can spend time to review that after my ruck/rover15:05
dasmk, it's one-liner15:05
ysandeep|ruckdone15:09
dasm\( ゚ヮ゚)/15:09
*** ysandeep|ruck is now known as ysandeep|dinner15:11
*** marios is now known as marios|out15:29
chandankumarsee ya!15:49
dasmchandankumar: o/15:49
*** ysandeep|dinner is now known as ysandeep15:54
ysandeeprcastillo|rover, ovb job is progressing well: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/b7cfcad165e64846a003cc92eb4ecf5a?logfile=console.log , passed stack creation point.. /me triggering 17/9 line15:55
ysandeeprcastillo|rover, done: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status#openstack-periodic-integration-rhos-17-rhel915:58
*** ysandeep is now known as ysandeep|out16:00
ysandeep|outrcastillo|rover, fyi.. rerunning master failing component jobs here: https://review.rdoproject.org/r/c/testproject/+/28446 16:21
ysandeep|outand wallaby failed components here: https://review.rdoproject.org/r/c/testproject/+/4265716:23
rcastillo|roverysandeep|out: ack16:24
ysandeep|outrcastillo|rover, only one job left for master promotion: https://review.rdoproject.org/r/c/testproject/+/38348/33#message-fcc96f0c6d01e2a73f566c415d147e5571904f67 :D16:28
rcastillo|roverysandeep|out++ nice16:28
ysandeep|outrcastillo|rover, random tempest https://logserver.rdoproject.org/48/38348/33/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/e6a12a9/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz 16:29
ysandeep|outif it don't pass in rerun - I will skip in my morning and promote.16:29
rcastillo|roversure, I'll keep trying it 16:30
ysandeep|outthanks, i have rechecked it for now16:30
*** jpena is now known as jpena|off16:33
ysandeep|outrcastillo|rover, Don't wake up early for program mtg :), I will join it.. 16:38
ysandeep|outrcastillo|rover, see you tomorrow o/ 16:39
rcastillo|roverysandeep|out: I appreciate that, 5am for me...16:39
rcastillo|roverysandeep|out: have a good night, talk to you tomorrow16:39
* dasm is going for a walk. bbl17:04
dviroelrcastillo|rover: hey o/17:06
rcastillo|roverdviroel: hi17:07
dviroelrcastillo|rover: some failures in gate, is this new?17:07
dviroelFailed to connect to the host via ssh: kex_exchange_identification: Connection closed by remote host\r\nConnection closed by 127.0.0.2 17:07
dviroelin different tasks17:07
dviroelseems to be infra related.17:08
rcastillo|roverhaven't seen before17:08
dviroelrcastillo|rover: ack - better to keep an eye if will be consistent17:08
rcastillo|roverdviroel: ack17:08
dviroelrcastillo|rover: somebody scanning public ips17:14
dviroelAug 16 15:52:28 undercloud.localdomain sshd[112513]: Invalid user minecraft from 34.86.209.138 port 3730617:14
dviroelAug 16 15:52:28 undercloud.localdomain sshd[112505]: Invalid user admin from 34.86.209.138 port 3736017:14
dviroelAug 16 15:52:28 undercloud.localdomain sshd[112508]: Invalid user pi from 34.86.209.138 port 3738017:14
dviroelAug 16 15:52:28 undercloud.localdomain sshd[112509]: Invalid user default from 34.86.209.138 port 3730017:14
dviroelAug 16 15:52:28 undercloud.localdomain sshd[112512]: Invalid user guest from 34.86.209.138 port 3726817:14
rcastillo|roverD:17:14
dasmback18:06
rcastillo|rovercoffee brb18:24
rcastillo|roverdviroel: so we think the ssh issue is some itermittent infra thing for now?18:38
dviroelrcastillo|rover: started to chat with infra on #opendev18:38
rcastillo|roveryeah I saw18:38
dviroelrcastillo|rover: we have lots of jobs failing in the same way, we should file a bug18:59
rcastillo|roverdviroel: ack, on it19:00
dviroelcentos9-scenario003 just failed too19:02
dviroelhttps://zuul.opendev.org/t/openstack/build/f3aa91645999406d8e9221a61fdfa6ca/log/job-output.txt#484219:02
rcastillo|roverdviroel: https://bugs.launchpad.net/tripleo/+bug/198670819:21
dviroelrcastillo|rover: i think that we can add promotion-blocker, and raise to cix19:24
rcastillo|roverdviroel: ok, I just wasn't sure if it was promotion blocker, since it's only in opendev19:25
dviroelthe promotion blocker tag is needed to create a CIX19:25
dviroelI think that is important to create the cix19:25
dviroelif this issue disapear, we just need to close bug and card tomorrow19:26
rcastillo|roverok done19:26
dviroelnice, thanks19:26
rcastillo|roverseems that all of these jobs are getting scanned19:27
rcastillo|rovermaybe not so unrelated?19:28
rcastillo|roversshd[65800]: error: beginning MaxStartups throttling19:28
dviroelrelated to the number of attempts to ssh from external 19:29
dviroelbut shouldn't affect active sessions19:30
dviroelmissed review time :|19:33
rcastillo|roveroh yeah oops19:33
dviroelrcastillo|rover: we need to add to hackmd too, so ysandeep can look when starting the day19:49
rcastillo|roveryeah I was about to do that19:49
dviroel++19:49
* dviroel brb20:02
*** dviroel is now known as dviroel|brb20:02
rcastillo|roversome stuff is getting through gate20:19
dasmysandeep|out: rcastillo|rover when you're able could you check https://review.rdoproject.org/r/c/rdo-jobs/+/44153 ? It depends on rr approvael20:26
rcastillo|roverdasm: looks fine to me from the tp. I'll let ysandeep|out w+ it in his morning20:36
*** dviroel|brb is now known as dviroel21:15
dasmrcastillo|rover: ack, thx21:38
* dviroel afk21:40
*** dviroel is now known as dviroel|afk21:40
rcastillo|roverbbl to check results22:04
*** dasm is now known as dasm|off22:26
* dasm|off => offline22:26

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!