Monday, 2019-08-19

*** d0ugal has joined #oooq00:06
*** dsneddon has joined #oooq00:34
*** dsneddon has quit IRC00:39
*** dsneddon has joined #oooq01:07
*** dsneddon has quit IRC01:12
*** dsneddon has joined #oooq01:40
*** dsneddon has quit IRC01:44
*** bhagyashris has joined #oooq01:48
*** dsneddon has joined #oooq02:13
*** dsneddon has quit IRC02:18
*** redrobot has quit IRC02:23
*** Guest90568 has joined #oooq02:29
*** Guest90568 is now known as redrobot02:32
*** dsneddon has joined #oooq02:51
*** dsneddon has quit IRC02:56
*** surpatil has joined #oooq03:10
*** dsneddon has joined #oooq03:27
*** dsneddon has quit IRC03:31
*** ykarel has joined #oooq03:51
*** dsneddon has joined #oooq04:04
*** dsneddon has quit IRC04:09
*** saneax has quit IRC04:09
*** ratailor has joined #oooq04:14
*** udesale has joined #oooq04:30
*** dsneddon has joined #oooq04:41
*** dsneddon has quit IRC04:46
*** ykarel has quit IRC05:09
*** bhagyashris has quit IRC05:09
*** skramaja has joined #oooq05:09
*** ykarel has joined #oooq05:10
*** suraj_ has joined #oooq05:12
*** surpatil has quit IRC05:12
*** ykarel is now known as ykarel|afk05:14
*** suraj_ is now known as surpatil05:14
*** surpatil has joined #oooq05:15
*** surpatil has joined #oooq05:17
*** kopecmartin|off is now known as kopecmartin05:18
*** surpatil has joined #oooq05:18
*** dsneddon has joined #oooq05:20
*** raukadah is now known as chkumar|ruck05:23
*** dsneddon has quit IRC05:25
*** dsneddon has joined #oooq05:45
*** ykarel|afk is now known as ykarel05:49
*** jaosorior has joined #oooq05:51
*** marios has joined #oooq05:52
*** jbadiapa has joined #oooq06:05
*** skramaja_ has joined #oooq06:07
*** skramaja has quit IRC06:07
*** marios has quit IRC06:18
*** matbu has joined #oooq06:19
*** ykarel is now known as ykarel|afk06:21
*** marios has joined #oooq06:25
*** saneax has joined #oooq06:27
*** ykarel|afk is now known as ykarel06:38
*** dsneddon has quit IRC06:48
*** ksambor has quit IRC06:50
*** ksambor has joined #oooq06:51
*** jtomasek has joined #oooq06:58
*** amoralej|off is now known as amoralej07:05
*** udesale has quit IRC07:05
*** udesale has joined #oooq07:06
*** jaosorior has quit IRC07:14
*** ykarel is now known as ykarel|afk07:16
*** apetrich has joined #oooq07:17
*** jtomasek has quit IRC07:21
*** dsneddon has joined #oooq07:21
*** ykarel|afk has quit IRC07:23
*** dsneddon has quit IRC07:26
*** yolanda has quit IRC07:42
*** yolanda__ has joined #oooq07:43
*** udesale has quit IRC07:43
*** jpena|off is now known as jpena07:44
*** udesale has joined #oooq07:44
*** ykarel|afk has joined #oooq07:46
*** ykarel|afk is now known as ykarel07:46
*** dsneddon has joined #oooq07:59
*** dsneddon has quit IRC08:04
*** jaosorior has joined #oooq08:05
*** chem has joined #oooq08:06
*** matbu has quit IRC08:12
*** skramaja has joined #oooq08:12
*** skramaja_ has quit IRC08:12
*** ykarel is now known as ykarel|afk08:23
chkumar|rucksshnaidm: Hello08:26
chkumar|rucksshnaidm: on this patch https://review.opendev.org/#/c/677063/ all the multi node jobs got failed with following error Data could not be sent to remote host "10.4.70.109". Make sure this host can be reached over ssh: ssh: connect to host 10.4.70.109 port 22: Network is unreachable08:27
chkumar|ruckis it seen?08:27
*** yolanda__ is now known as yolanda08:28
*** derekh has joined #oooq08:34
*** dsneddon has joined #oooq08:35
sshnaidmchkumar|ruck, if it's limestone cloud, I wrote it in last mail08:35
sshnaidmchkumar|ruck, but I didn't submit a patch for it08:37
sshnaidmchkumar|ruck, s/patch/bug/08:37
sshnaidmit should be discussed with infra08:38
chkumar|rucksshnaidm: yes, all are limestone related08:39
*** dsneddon has quit IRC08:40
sshnaidmgreat.. ovh cloud fails because of unathorized errors, limestone fails in all jobs.. where should we run jobs then? :D08:41
sshnaidmseems like rdo cloud is island of stability now08:43
chkumar|rucksshnaidm: we have issue with rrcockpit also08:44
chkumar|rucksshnaidm: here http://rhos-release.virt.bos.redhat.com:3030/rhosp stein promotion is 2d ago but in cockpit it is 11 days08:45
chkumar|ruckI am not sure why it is showing like this08:47
*** ykarel|afk is now known as ykarel08:51
*** matbu has joined #oooq09:02
*** surpatil has quit IRC09:04
*** dsneddon has joined #oooq09:10
*** dsneddon has quit IRC09:14
*** bhagyashris has joined #oooq09:17
*** surpatil has joined #oooq09:30
sshnaidmchkumar|ruck, seems like something changed in dlrn api09:35
*** surpatil has quit IRC09:39
chkumar|rucksshnaidm: nope https://softwarefactory-project.io/r/#/q/project:dlrnapi_client09:40
*** surpatil has joined #oooq09:40
chkumar|ruckchanges are in progress09:40
chkumar|ruckhttps://tree.taiga.io/project/morucci-software-factory/us/264809:40
chkumar|rucksshnaidm: do we have specific error?09:41
sshnaidmchkumar|ruck, yeah, weird.. script doesn't return repo_hash09:41
*** pierreprinetti has joined #oooq09:43
*** dsneddon has joined #oooq09:46
*** pierreprinetti has quit IRC09:48
*** pierreprinetti has joined #oooq09:48
*** dsneddon has quit IRC09:50
pandamarios: all test jobs in https://tree.taiga.io/project/tripleo-ci-board/task/1232?kanban-status=1447276 are failing, but I think they are passing the critical point of your card, right ?09:50
mariospanda: yeah, i was goign to add a depends-on to sagi patch in a sec to see how far it can get09:52
mariospanda: but for your question yes commented in https://review.rdoproject.org/r/#/c/21787/6/zuul.d/standalone-jobs.yaml@330209:53
pandamarios: ok, call me when you're satisfied and we can merge this.09:54
pandamarios: 1-800-panda09:54
*** apetrich has quit IRC09:56
mariospanda: thanks for looking. zbr volunteered to baby sit that when i'm out09:56
pandamarios: when are you out ?09:57
mariospanda: (see pvt) & will email later09:58
sshnaidmCI is mostly down, limestone cloud fails all centos jobs, ovh cloud fails with unauthorized errors09:59
sshnaidmmarios, panda ^^10:00
sshnaidmonly thing that works is rdo cloud :D10:00
sshnaidmnever believed I'd say that10:00
chkumar|rucksshnaidm: if it possible to hold one of the limestone centos node and debug there?10:02
pandawe are having only two providers ?10:02
chkumar|ruck*is it10:02
mariospanda: please also check https://review.rdoproject.org/r/#/c/21832/ when you have time for the layout10:04
mariossshnaidm: thanks10:05
sshnaidmchkumar|ruck, no idea, better ask on #openstack-infra10:06
pandamarios: each extra review is going to cost you a day of PTO.10:14
mariospanda: i am considering entering witness protection10:15
*** dsneddon has joined #oooq10:17
pandamarios: you can't, you lack the wit.10:17
mariosdamn foiled again !10:18
pandamarios: it's a nice way to get used to aluminium.10:21
*** dsneddon has quit IRC10:22
mariospanda: does wonders for your hair too and helps to keep indoor lighting requirements to a minimum as long as you sit near a window10:23
zbrmarios: can you explain me why you added a variable that gets assigned "false" string?10:29
zbrtrying to figureout what made you dothat10:29
marioszbr: maybe but it depends on if you can point me to where you are referring to :D10:30
marioszbr: or even better comment there and point to it here10:30
marioszbr: and i'll update if there is somehting missing10:30
zbrahh. is was not your change. it was sshnaidm https://review.opendev.org/#/c/676497/510:30
marioszbr: k10:30
sshnaidmzbr, what is the problem?10:32
zbrsshnaidm: ^ can you help on that? if possible would like to avoid use of strings for boolean vars, it's a road to pain. if no special reason I can fix it.10:32
zbrboth python and yaml support booleans.10:33
sshnaidmzbr, booleans don't work well in heat template + jinja + yaml + godknowswhatelsethere10:34
sshnaidmzbr, but you can try10:34
*** skramaja has quit IRC10:39
*** bhagyashris has quit IRC10:50
*** pierreprinetti has quit IRC10:53
*** dsneddon has joined #oooq10:55
*** dsneddon has quit IRC11:00
* marios food biab11:12
*** marios has quit IRC11:12
*** udesale has quit IRC11:15
*** tesseract has joined #oooq11:15
chkumar|rucksshnaidm: as per donnnyd on #openstack-infra, there is no issue with centos image side11:16
sshnaidmchkumar|ruck, cool, then problem is in different place of limestone :)11:17
*** pierreprinetti has joined #oooq11:20
*** dsneddon has joined #oooq11:33
chkumar|rucksshnaidm: do we have a filed bug for rhel8 tempest issue?11:36
*** apetrich has joined #oooq11:36
sshnaidmchkumar|ruck, I don't think so11:37
sshnaidmchkumar|ruck, will submit11:37
*** dsneddon has quit IRC11:38
*** jpena is now known as jpena|lunch11:39
sshnaidmchkumar|ruck, https://bugs.launchpad.net/tripleo/+bug/184064511:40
openstackLaunchpad bug 1840645 in tripleo "Tempest network scenario test fails on rhel8 host because "No such file or directory: 'ping': 'ping'"" [High,Triaged]11:40
sshnaidmchkumar|ruck, ok, got it.. " OK for 11 days " it means the alarm is OK for 11 days, not "last promotion was 11 days before"11:45
sshnaidmchkumar|ruck, http://dashboard-ci.tripleo.org/d/YRJtmtNWk/cockpit?panelId=263&fullscreen&edit&tab=alert - for last 11 days no alarms for stein promotions11:46
chkumar|rucksshnaidm: ok got it, thanks :-)11:47
*** rlandy has joined #oooq11:47
*** rlandy is now known as rlandy|rover11:48
rlandy|roverchkumar|ruck: hi - are we switching today?11:49
chkumar|ruckrlandy|rover: yes11:49
rlandy|roverk11:49
*** chkumar|ruck is now known as chkumar|rover11:49
*** rlandy|rover is now known as rlandy|ruck11:49
rlandy|ruckchkumar|rover: https://bugs.launchpad.net/tripleo/+bug/1840451 - is this something we fix or a dfg?11:51
openstackLaunchpad bug 1840451 in tripleo "Master promotions are failing tempest test (fs020) tempest.api.volume.admin.test_volume_quotas.VolumeQuotasAdminTestJSON" [Critical,Triaged] - Assigned to chandan kumar (chkumar246)11:51
chkumar|roverrlandy|ruck: we need to bug storage dfg11:52
rlandy|ruckI am editing the card on escalation board11:52
rlandy|ruckk11:52
chkumar|roverrlandy|ruck: I have left all my finding on the bug itself11:52
rlandy|ruckI see that  - thanks11:52
rlandy|ruckdo you know who is our storage contact?11:52
chkumar|rovermight be eric harney11:52
rlandy|ruckk11:53
rlandy|ruckchkumar|rover: what's the story with https://trello.com/c/ZvV5ul81/1029-cixlp1836046tripleociproa-tempestscenariotestnetworkbasicopstestnetworkbasicops-failing-on-queens - queens is promoting now11:56
chkumar|roverrlandy|ruck: I am trying with config_drive there in my patch11:57
rlandy|ruckchkumar|rover: the card is still open but queens has been promoting successfully11:58
chkumar|roverrlandy|ruck: it comes every twice in aprox 5-6 runs11:58
rlandy|ruckha - and master promoted http://rhos-release.virt.bos.redhat.com:3030/rhosp11:58
* rlandy|ruck is confused11:58
*** weshay_pto is now known as weshay12:00
weshay0/12:00
*** marios has joined #oooq12:05
*** ykarel is now known as ykarel|afk12:05
*** dsneddon has joined #oooq12:06
chkumar|roversshnaidm: https://review.opendev.org/#/c/677191/ might fix the tempest issue12:07
chkumar|roversshnaidm: does kolla patches can be tested in ovb jobs?12:09
sshnaidmchkumar|rover, no, we don't build containers in ovb job12:09
sshnaidmchkumar|rover, actually we use kolla in containers-build jobs only12:09
chkumar|roversshnaidm: do adding the same in tripleo-common kolla override will also not help?12:10
sshnaidmchkumar|rover, what do you mean?12:11
*** dsneddon has quit IRC12:11
chkumar|roversshnaidm: I think i got the answer adding the tempest_packages_append here https://github.com/openstack/tripleo-common/blob/master/container-images/tripleo_kolla_template_overrides.j2 is also not going to help12:12
chkumar|roverI need to find another way of testing it12:12
*** jaosorior has quit IRC12:14
sshnaidmchkumar|rover, maybe we can do like that: 1) build the tempest container with your modifications, 2) upload it somewhere on dockerhub, 3) in tripleo-common replace path with name_prefix of tempest container to yours, 4) depends on this patch and run ovb job12:16
*** amoralej is now known as amoralej|lunch12:16
chkumar|roversshnaidm: sounds like a good plan12:17
sshnaidmchkumar|rover, but in general we need to develop such option - to build a container and test it in a job12:17
sshnaidmfor future12:17
*** apetrich has quit IRC12:20
ykarel|afksshnaidm, running both jobs in check pipeline with depends-on should also work12:25
ykarel|afkovb depend on container push12:25
ykarel|afkand patch depend on kolla12:26
sshnaidmykarel|afk, ovb job takes containers from dockerhub, it doesn't build them12:26
ykarel|afksshnaidm, periodic one don't take from trunk registry?12:26
ykarel|afki think they do, let me check again12:26
sshnaidmykarel|afk, yes, periodic does take them from trunk12:27
sshnaidmykarel|afk, but we talk about check job12:27
ykarel|afksshnaidm, ohhk12:27
ykarel|afksshnaidm, so you facing issue only in check?12:27
sshnaidmif building containers job could hold these containers and pass it anyhow to ovb job, it would work12:27
sshnaidmykarel|afk, we need to check container modification in ovb job, that's the problem12:28
*** apetrich has joined #oooq12:28
*** ykarel|afk is now known as ykarel12:28
rlandy|rucksshnaidm: after scrum, can we sync for re: sova changes?12:28
*** rfolco has joined #oooq12:28
sshnaidmykarel|afk, we need option to build a specific container like we build a specific rpm..12:29
sshnaidmrlandy|ruck, sure12:29
ykarelsshnaidm, okk got it, currently we either build or update all containers12:29
sshnaidmyep12:29
*** jpena|lunch is now known as jpena12:31
chkumar|roversshnaidm: can we do a force push for this patch with the help of openstack-infra root admins?12:32
chkumar|rovercheck jobs are broken12:32
chkumar|roversshnaidm: https://review.opendev.org/#/c/677063/12:32
chkumar|roversshnaidm: the issue while doing gathering facts on subnode 2 is known https://github.com/ansible/ansible/issues/5754212:36
sshnaidmchkumar|rover, not sure I understand12:37
*** jaosorior has joined #oooq12:38
chkumar|roversshnaidm: about this issue Data could not be sent to remote host "10.4.70.30". Make sure this host can be reached over ssh: ssh: connect to host 10.4.70.30 port 22: Network is unreachable12:38
chkumar|roversshnaidm: https://object-storage-ca-ymq-1.vexxhost.net/v1/86bbbcfa8ad043109d2d7af530225c72/logs_82/677182/1/check/tripleo-ci-centos-7-containers-multinode-stein/0b4a29b/job-output.txt12:38
sshnaidmchkumar|rover, yeah?12:39
chkumar|roversshnaidm: as per this warning [WARNING]: Unhandled error in Python interpreter discovery for host subnode-2 -> it appears it is related to this ansible issue https://github.com/ansible/ansible/issues/5754212:40
weshaychkumar|rover congrats!!!12:42
weshaychkumar|rover rlandy|ruck anything I can do for you?12:42
chkumar|roverweshay: thanks :-)12:42
*** dsneddon has joined #oooq12:42
sshnaidmchkumar|rover, hmm.. idk12:42
chkumar|roverbrb for 20 mins!12:43
rlandy|ruckweshay: there a bunch of stuff going on12:46
rlandy|ruckweshay: we can sync after team scrum12:46
weshayk12:46
*** ykarel is now known as ykarel|away12:47
*** dsneddon has quit IRC12:47
rlandy|rucklog moves, rrcockpit updates, gate failure lots of fun12:47
sshnaidmykarel|away, chkumar|rover congrats to you both :)12:48
rlandy|ruckoh wow - congrats chkumar|rover and ykarel|away - well deserved12:50
weshayykarel|away wooooot!!! congrats12:51
*** ykarel|away has quit IRC12:52
chkumar|roversshnaidm: rlandy|ruck thanks :-)12:57
weshaychkumar|rover rlandy|ruck arxcruz I see the periodic reproducer jobs are all red, in post_failure..  anyone know why?12:57
rlandy|ruckweshay: yep - saw that - will check12:58
rlandy|ruckweshay: do we plan to add redhat8-master jobs to http://rhos-release.virt.bos.redhat.com:3030/rhosp?12:58
rlandy|ruckwe are not tracking that progress anywhere public12:58
weshayrlandy|ruck it would just be lumped in w/ master12:59
chkumar|roverweshay: and do we plan to have RDO phase 1 jobs also on rhel-8?12:59
chkumar|rover*have plan12:59
weshayso if standalone rhel 8 is stable.. we could add that to criteria12:59
*** pierreprinetti has quit IRC12:59
rlandy|ruckno redhat nodes12:59
rlandy|ruckon ci.centos12:59
weshaythat's fine12:59
rlandy|ruckwe would have ti add those jobs somewhere else12:59
weshaythose jobs are almost worthless12:59
*** pierreprinetti has joined #oooq12:59
rlandy|ruckphase 1?13:00
weshayya13:00
rlandy|ruckin general?13:00
rlandy|ruckk - so we should add that to our sync discussions13:00
*** dsneddon has joined #oooq13:00
rlandy|ruckalong with other fun13:00
*** pierreprinetti has quit IRC13:00
rlandy|ruckno rest for the ruck/rover this sprint13:00
*** ratailor has quit IRC13:00
rfolcoping scrum zbr13:01
rfolcosshnaidm, ping scrum13:02
*** pierreprinetti has joined #oooq13:06
rlandy|ruckhmmm .. redhat8 has not promoted in a while13:07
chkumar|roverrlandy|ruck: /me has seen check jobs green last week13:07
rlandy|ruckwhy were the build jobs skipped13:08
rlandy|rucklast promoted on August 1613:08
rlandy|ruck2019-08-19 12:34:29.975352 | TASK [prepare-node : Dump hostvars on the host to pass it to inner ansible]13:08
rlandy|ruck2019-08-19 12:34:30.038849 | primary | ERROR13:08
rlandy|ruck2019-08-19 12:34:30.039207 | primary | {13:08
rlandy|ruck2019-08-19 12:34:30.039321 | primary |   "msg": "The task includes an option with an undefined variable. The error was: 'featureset' is undefined\n\nThe error appears to have been in '/var/opt/rh/rh-python35/lib/zuul/builds/dddf778627ee4c51af594ed4516b80b9/untrusted/project_3/opendev.org/openstack/tripleo-ci/roles/prepare-node/tasks/main.yaml': line 40, column 3, but may\nbe elsewhere in the file depending on the exact13:08
rlandy|rucksyntax problem.\n\nThe offending line appears to be:\n\n\n- name: Dump hostvars on the host to pass it to inner ansible\n  ^ here\n"13:08
rlandy|ruck2019-08-19 12:34:30.039387 | primary | }13:08
chkumar|roverrlandy|ruck: it is a known issue13:09
chkumar|roverrlandy|ruck: https://review.opendev.org/#/c/67706313:09
chkumar|rovergot stuck in gate issue13:09
chkumar|roverthanks to sshnaidm for fixing on weekend13:09
chkumar|roverrlandy|ruck: line https://etherpad.openstack.org/p/ruckroversprint14 30 to 3313:10
rlandy|ruckit broke over the weekend13:11
*** aakarsh has quit IRC13:18
*** amoralej|lunch is now known as amoralej13:25
weshayhttps://mojo.redhat.com/groups/omega-prime/overview13:26
chkumar|roverarxcruz: congrats :-)13:30
arxcruzchkumar|rover:thanks :)13:36
*** surpatil has quit IRC13:36
*** ykarel|away has joined #oooq13:43
*** ykarel|away is now known as ykarel13:45
*** aakarsh has joined #oooq13:54
weshaychkumar|rover a lot of these user stories are in parked afaict13:55
chkumar|roverweshay: /me is looking at the board, I will ask the team to update the status or will do that,13:58
chkumar|roverrlandy|ruck: is it possible to put this in fs020 CinderLVMLoopDeviceSize itself?13:59
rlandy|ruckchkumar|rover: the master bug?14:00
chkumar|roverrlandy|ruck: yes14:00
chkumar|roverrlandy|ruck: https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/cinder/cinder-volume-container-puppet.yaml#L9514:01
chkumar|roverrlandy|ruck: https://bugs.launchpad.net/tripleo/+bug/184045114:01
openstackLaunchpad bug 1840451 in tripleo "Master promotions are failing tempest test (fs020) tempest.api.volume.admin.test_volume_quotas.VolumeQuotasAdminTestJSON" [Critical,Triaged] - Assigned to chandan kumar (chkumar246)14:01
rlandy|ruckchkumar|rover: if that is passed to the deploy, yes14:01
rlandy|ruckwe have a number of files that pass parameters14:01
rlandy|ruckif that is passed directly to tempest, not sure14:02
rlandy|ruckchkumar|rover: getting files14:02
*** dsneddon has quit IRC14:03
chkumar|roverrlandy|ruck: we need to increase CinderLVMLoopDeviceSize value in deployment itself during overcloud deploy14:03
rlandy|ruckchkumar|rover: ack ... sorry having concurrent conversation on sf-ops re: reproducer failures14:04
rlandy|ruckwe have these options14:04
ykarelchkumar|rover, rlandy|ruck just noticed in stein full tempest job, all tests are not running14:05
rlandy|ruckhttps://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/1ea6dd7/logs/undercloud/home/zuul/overcloud-deploy.sh14:05
ykarelchkumar|rover, rlandy|ruck tempest plugins are not installed in tempest container14:05
rlandy|ruckwill look in a bit14:06
chkumar|roverykarel: we install openstack-tempest-all there in contauner14:06
ykarelchkumar|rover, nope it's not installed in stein14:06
rlandy|ruckchkumar|rover: k - I'll try add that option14:06
ykarelso no plugins14:06
ykarelchkumar|rover, https://review.opendev.org/#/c/651601/5/docker/tempest/Dockerfile.j214:07
ykarelneed to fix it in stein ^^14:07
chkumar|roverykarel: ok fix that14:07
ykarelchkumar|rover, /me propose or u will take care?14:08
chkumar|roverykarel: taking care14:09
weshaymarios want to sync?14:09
arxcruzi have a doctor appointment brb14:09
ykarelchkumar|rover, ack14:09
weshaymarios I don't hvave much, so your call14:09
mariosweshay: ack pvt14:09
weshaymarios have anything you want to discuss?14:10
mariosweshay: not particularly will summarize my stuff and sold most of it already14:10
zbrrlandy|ruck: are you aware about an error The task includes an option with an undefined variable. The error was: 'featureset' is undefined ?14:12
sshnaidmzbr, arxcruz are you sure ansible-version is inherited via different zuul systems?14:12
sshnaidmI'd check that on #zuul14:12
rfolcopanda, weshay: please give me a time to discuss staging env today or tomorrow.14:12
chkumar|roverzbr: https://review.opendev.org/#/c/677063/14:12
zbrsshnaidm: almost sure is just a variable, so it will be inhereted. also very easy to discover. we only need to define it in a job.14:13
rfolcoarxcruz, congrats man14:13
*** pierreprinetti has quit IRC14:13
rlandy|ruckzbr: yes14:14
pandarfolco: all the time in the world14:14
*** pierreprinetti has joined #oooq14:14
sshnaidmzbr, arxcruz I think it's worth to switch one rhel8 job to 2.8 in zuul, to remove hostvars and see if it helps, before doing any global changes14:14
* panda raises both his hands to the sky14:14
rlandy|ruckzbr: if you mean https://github.com/openstack/tripleo-ci/commit/84018c747d5a1a1da7ad4fcbc31c2c7ff3b5963e change14:14
sshnaidmzbr, I believe it is, but just to be sure..14:14
pandarfolco: here take my life energy14:15
zbrrlandy|ruck: not really, is about this error https://45da25d3022f8ec0f64e-bc6452672c10420faa7f3759f0d39f73.ssl.cf1.rackcdn.com/677059/1/check/tripleo-buildimage-overcloud-full-centos-7/b577b12/job-output.txt -- which has nothing to do with the change being tested.14:15
*** sshnaidm is now known as sshnaidm|bbl14:16
rlandy|ruckhttps://review.opendev.org/#/c/67706314:16
rlandy|ruckzbr: ^^14:16
zbrdamn, containers already failed.14:16
rlandy|ruckthat is the fix14:17
chkumar|roverykarel: rlandy|ruck https://review.opendev.org/#/c/677212/14:17
rfolcoweshay, need to discuss staging env, the earlier the better... you seem to be free 40 min from now. Is it a good time ?14:17
rfolcopanda, ^14:17
rlandy|ruckchkumar|rover: thanks14:17
* panda gives rfolco his bleeding heart14:18
ykarelchkumar|rover, ack14:18
ykarel+114:18
pandarfolco: just extracted, take it, it's yours to use.14:18
rlandy|ruckugh - everything on fire today14:18
chkumar|roverweshay: rlandy|ruck sshnaidm|bbl https://review.opendev.org/#/c/677063/ can we get this patch force merged with the help of openstack-infra roots14:19
ykarelchkumar|rover, one comment14:19
chkumar|roverit will help to bring back the ci green except limestone 401 issues14:19
ykarelwrong link14:20
rlandy|ruckas long as the errors are not legit14:20
zbrwhy we do have this double compression and broken file type on undercloud-install archive? https://6484f2cfee4024eb280c-d3a3c82ce3aa1074eb73e263d71495ca.ssl.cf5.rackcdn.com/677063/8/check/tripleo-ci-centos-7-undercloud-containers/e860eac/logs/undercloud/home/zuul/14:21
rfolcopanda, weshay: staging env call starts in 38 min14:22
rfolcoinvitation sent14:22
rfolcozbr, added you and others as optional to attend ^14:22
pandarfolco: I'm sorry, I can't14:25
rfolcopanda, :(14:26
rfolcopanda, you hurt my feelings14:26
pandarfolco: I have judo lessons in 15 minutes14:26
rfolcopanda, you gave my heart14:26
rfolcoyour hear14:26
rfolcot14:26
pandarfolco: it was a pig heart14:26
rfolcoyou broke my heart14:26
weshayrfolco ya.. add me to a call for sure14:26
rfolcoweshay, panda can't moving to tomorrow14:27
pandarfolco: weshay just joking14:27
pandarfolco: weshay I'm here14:27
rfolcodamnn14:27
pandarfolco: stop taking me seriously14:27
rfolcopanda, you broke my heart and laugh on my face14:27
rfolcopanda, what kind of human being you are ?14:27
zbrrlandy|ruck: it is normal to get "authentication required" when pulling from dockerhub? https://6484f2cfee4024eb280c-d3a3c82ce3aa1074eb73e263d71495ca.ssl.cf5.rackcdn.com/677063/8/check/tripleo-ci-centos-7-undercloud-containers/e860eac/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz14:28
rlandy|ruckzbr: https://bugs.launchpad.net/tripleo/+bug/183953214:30
openstackLaunchpad bug 1839532 in tripleo "tripleo gate jobs are failing to pull containers when running on ovh provider with "UNAUTHORIZED" error" [Critical,Triaged]14:30
rlandy|ruckgoing to bring that up with weshay14:30
rlandy|ruckwe have no fix and it's a major roblem14:30
rlandy|ruckproblem14:30
weshayrlandy|ruck cix it14:30
rlandy|ruckjust fixing two other bugs atm14:30
zbryeah, i see it. thanks for the link.14:30
rlandy|ruckin a few14:30
*** yolanda has quit IRC14:31
*** dtrainor has joined #oooq14:31
*** yolanda has joined #oooq14:36
*** dsneddon has joined #oooq14:39
*** dsneddon has quit IRC14:43
*** saneax has quit IRC14:49
weshayrlandy|ruck chkumar|rover I have 10 min if you guys need something14:50
rlandy|ruckin the middle of bug fixing - will ping later14:50
*** surpatil has joined #oooq14:52
*** ykarel has quit IRC14:57
rlandy|ruckchkumar|rover: let's see what this does ... https://review.opendev.org/677227 Add overcloud deploy parameter to set CinderLVMLoopDeviceSize15:00
rlandy|ruckhttps://review.rdoproject.org/r/#/c/21851/ rocky should promote15:02
rlandy|ruckqueens is in trouble15:02
*** jaosorior has quit IRC15:04
chkumar|roverrlandy|ruck: what is the new queens issue? except tempest fs0115:13
rlandy|ruckchkumar|rover: https://review.rdoproject.org/r/#/c/21862/ rerun15:14
rlandy|ruckall failed15:14
rlandy|ruckwill look at it after fixing reproducer failures15:14
*** dsneddon has joined #oooq15:14
chkumar|roverchecking15:15
chkumar|roverrlandy|ruck: it is tempest15:15
chkumar|roverfor queens15:15
chkumar|roverlet me try with my fix https://review.opendev.org/#/c/676439/15:17
*** saneax has joined #oooq15:17
chkumar|roverrlandy|ruck: https://review.rdoproject.org/r/#/c/21862/ updated here, we will know in another 3 hrs15:18
rlandy|ruckchkumar|rover: k - thanks - will watch15:18
*** dsneddon has quit IRC15:19
chkumar|roverrlandy|ruck: https://bugs.launchpad.net/tripleo/+bug/1840616 needs attension15:20
openstackLaunchpad bug 1840616 in tripleo "Master/stein check and promotion OVB jobs are randomly giving ansible time out while overcloud deploy" [Critical,Triaged]15:20
chkumar|roveransibel timed out during overcloud deploy15:20
chkumar|roverI have seen last week and today also15:20
rlandy|ruckchkumar|rover: we can increase the deploy timeout15:21
chkumar|roverrlandy|ruck: it is related to docker pull at some places15:21
rlandy|ruck2019-08-14 03:57:14 | >f+++++++++ fOvercloud configuration failed.15:21
chkumar|roverrlandy|ruck: weshay I am logging out, please get this merged https://review.opendev.org/#/c/677063/ if possible, I will take over from tomorrow15:24
chkumar|rovermorning15:24
chkumar|roversee ya15:24
rlandy|ruckchkumar|rover: ack - thanks15:24
*** chkumar|rover is now known as raukadah15:24
rlandy|ruckweshay: let me know when you have time to sync15:29
weshayk.. need a few15:31
mariospanda: feauxmotion15:36
mariosplease that ^15:36
rfolcomarios, need a 5 min handoff if you can15:37
mariosrfolco: ack15:38
mariosrfolco: as soon as we finish  on this call?15:38
rfolcoyep15:38
pandamarios: faux15:41
pandamarios: also foomotion could be good15:42
rlandy|ruck2019-08-19 14:23:18,259 773 INFO     promoter Promoting the container images for dlrn hash aef1c3ef1ed640e269fbb45e067934e7cdbad697 on rocky to current-tripleo15:42
rlandy|ruckrocky promoting - one step15:42
*** dsneddon has joined #oooq15:50
*** dsneddon has quit IRC15:55
* marios almost out15:56
weshayrlandy|ruck k.. ready whenever you are15:57
rlandy|ruckweshay: k - joing your bj15:58
*** marios is now known as marios|out16:03
*** Vorrtex has joined #oooq16:08
*** jpena is now known as jpena|off16:11
weshayrlandy|ruck https://wiki.centos.org/About/Building_816:20
weshayraukadah fyi ^.. they are getting close.. no action required atm16:20
weshayjust fyi16:20
raukadahweshay: /me will wait for the announcements.16:21
*** dsneddon has joined #oooq16:23
*** surpatil has quit IRC16:25
*** dsneddon has quit IRC16:27
rlandy|rucklunch - brb\16:28
weshayrlandy|ruck sorry.. my computer overheated16:31
weshayshutdown16:31
*** marios|out has quit IRC16:33
*** tesseract has quit IRC16:35
*** dsneddon has joined #oooq16:39
rlandy|ruckweshay: np - finished my tales of woe16:41
weshay:)16:41
zbrsshnaidm|bbl: rlandy|ruck can we do https://review.opendev.org/#/c/676474/3 ?16:43
*** derekh has quit IRC17:03
rlandy|ruckback looking17:04
rlandy|ruckweshay, https://review.rdoproject.org/r/21856 Remove Fedora28 stein/master and add RedHat 8 master (rrcockpit updates)17:05
rlandy|ruckpls see if that works for you17:05
weshayk.. thanks17:05
*** amoralej is now known as amoralej|off17:08
zbrweshay: brilliant design: i change one job and I see >20 running, none of them based on the one I touched: https://review.opendev.org/#/c/677256/17:11
rlandy|ruckhttps://review.opendev.org/#/c/677063 - still failing - may be legit17:11
weshaytripleo-build-containers-centos-7-buildahFAILURE in 1h 13m 44s17:17
weshaythat one?17:17
weshayhttps://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_63/677063/8/gate/tripleo-build-containers-centos-7-buildah/e459824/logs/containers-build-errors.log.txt.gz17:19
weshay SystemError: The following jobs were incomplete: set([<Future at 0x7f927d9fdb10 state=finished returned NoneType>])17:22
weshay2019-08-19 03:43:50 | The following jobs were incomplete: set([<Future at 0x7f927d9fdb10 state=finished returned NoneType>])17:22
weshay2019-08-19 03:43:50 | ova-serialproxy/nova-serialproxy-build.log -t 127.0.0.1:8787/tripleomaster/centos-binary-nova-serialproxy:6cd9e55d69e90ea73219abced4e9a3f2372b204e_10e135ca /tmp/kolla-SRgmFH/docker/nova/nova-serialproxy17:22
weshay2019-08-19 03:43:50 | Building nova-api image with: sudo buildah bud --tls-verify=False --logfile /tmp/kolla-SRgmFH/docker/nova/nova-api/nova-api-build.log -t 127.0.0.1:8787/tripleomaster/centos-binary-nova-api:6cd9e55d69e90ea73219abced4e9a3f2372b204e_10e135ca /tmp/kolla-SRgmFH/docker/nova/nova-api17:22
*** kopecmartin is now known as kopecmartin|off17:33
*** pierreprinetti has quit IRC18:15
*** Vorrtex has quit IRC18:15
rlandy|ruckweshay: what column does this card move to? https://trello.com/c/rPZGwJmU/1081-cixlp1839532tripleociproa-tripleo-gate-jobs-are-failing-to-pull-containers-when-running-on-ovh-provider-with-unauthorized-error18:29
rlandy|ruckweshay: slow - pls vote on https://review.rdoproject.org/r/#/c/21875/ - ci results are looking promising - will merge when ci completes18:30
rlandy|ruckthanks18:32
rlandy|rucksaw it move18:32
weshaynp18:34
*** aakarsh has quit IRC18:35
rlandy|ruckzbr: thanks for merging https://review.rdoproject.org/r/#/c/21856/ - did break the rrcockpit :)18:36
rlandy|ruckweshay: ^^ no worries to test18:36
rlandy|ruckalready merged18:36
weshaycool.. the prod cockpit looks good.. nice job rlandy|ruck zbr18:37
*** aakarsh has joined #oooq18:39
*** saneax has quit IRC18:47
*** pierreprinetti has joined #oooq18:55
rlandy|ruckweshay: zbr:  what do you think of increasing deploy timeout for https://bugs.launchpad.net/tripleo/+bug/1840616?19:15
openstackLaunchpad bug 1840616 in tripleo "Master/stein check and promotion OVB jobs are randomly giving ansible time out while overcloud deploy" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy)19:15
weshayrlandy|ruck I don't think we can make a case for a timeout bug w/o trying it locally19:16
rlandy|ruckweshay: yeah - I know it's a workaround19:17
rlandy|ruckfailure is sporadic19:17
weshayrlandy|ruck I would add alert so it's on our radar19:18
weshaybut that's it.. wouldnt cix it yet19:18
rlandy|ruckci and alert?19:18
rlandy|ruckwill look a little more into it19:19
*** sshnaidm|bbl is now known as sshnaidm19:19
*** dsneddon has quit IRC19:20
*** dtrainor has quit IRC19:21
sshnaidmzbr, please don't change anything in rhel containers patches, I'm testing the current version19:21
*** dtrainor has joined #oooq19:23
rlandy|ruckhttps://review.opendev.org/#/c/677063 will kill us19:23
rlandy|ruckcant get it through zuul :(19:23
*** dsneddon has joined #oooq19:26
rlandy|ruckzbr: pls vote on https://review.rdoproject.org/r/#/c/21875/ so I don't just self merge19:29
rlandy|ruckanybody: ^^19:31
*** pierreprinetti has quit IRC19:33
*** saneax has joined #oooq19:36
rlandy|ruckweshay: rfolco: ^^ anyone else vote here? required change to get rid of periodic post failure19:43
weshayk..s ec19:44
weshaysec19:44
weshayrlandy|ruck done19:44
weshaythanks19:44
rlandy|ruckthank you19:46
rfolcowes hit the button first19:46
rlandy|ruckone less failure hopefully19:47
rlandy|ruckrfolco: know who has +2 on openstack/kolla- https://review.opendev.org/#/c/677212/19:48
*** sanjayu_ has joined #oooq19:53
*** saneax has quit IRC19:56
*** sanjayu_ has quit IRC19:57
jjoycerlandy|ruck: Do you happen to have a link handy to the docker register setup playbook?20:00
rlandy|ruckjjoyce: on the undercloud?20:01
rfolcorlandy|ruck, I don't know20:01
rlandy|ruckor a regular public registry?20:01
jjoyceJust a regular registry20:02
*** pierreprinetti has joined #oooq20:02
rlandy|ruckjjoyce: we have a couple pieces of it depending on deployment ... https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/container-prep , https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/undercloud-deploy/templates/containers-prepare-parameter.yaml.j220:06
rlandy|ruckthen there is if you are working standalone20:06
rlandy|ruckhttps://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-containers/tasks/main.yml-> overcloud20:07
rlandy|ruckI am not sure I am answering the questions20:07
rlandy|ruckmostly this is done in the product itself20:08
rlandy|ruckwe just pass parameters20:08
jjoyceI was looking for something that just installs all the pieces needed for a local registry20:08
jjoyceI know Wes said that their were some playbooks that did that.20:09
*** pierreprinetti has quit IRC20:09
rlandy|ruckweshay: we do?20:10
weshayjjoyce rlandy|ruck ya.. the container check job installs a registry.. /me gets20:11
rlandy|ruckoh that20:11
* rlandy|ruck was looking in deployment20:11
weshayjjoyce rlandy|ruck https://github.com/openstack/ansible-role-container-registry20:12
jjoyceweshay: Perfect! Just what I was looking for.20:14
weshayjjoyce I've been looking for you for all my life too!!!20:15
weshayI found my soul mate20:15
jjoyceweshay: <320:15
jjoyce:-)20:15
weshay:)20:15
weshayrlandy|ruck https://review.rdoproject.org/r/#/c/21884/20:39
rlandy|ruckweshay: oh - bless you - finally20:55
rlandy|ruckugh - stein - one stupid tempest test - so close20:59
*** aakarsh has quit IRC21:10
*** sshnaidm is now known as sshnaidm|afk21:24
rlandy|ruckugh - we have t get this patch through or disable image builds22:01
rlandy|rucktripleo-buildimage-overcloud-full-centos-722:03
*** aakarsh has joined #oooq22:35
rlandy|ruckhttps://review.opendev.org/#/c/677063/ keeps failing https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_63/677063/8/check/tripleo-ci-centos-7-containers-multinode/dc1abe2/22:43
rlandy|ruckweshay: hello22:43
rlandy|ruckhttps://review.opendev.org/#/c/677063/ keeps failing some check/gate22:43
rlandy|ruckthis is supposed to clear up the image build failure22:44
rlandy|ruckalso this error is failing other jobs on gates22:44
rlandy|ruckwe have to option to get the patch pushed through by infra cores22:44
rlandy|ruckthoughts?22:44
*** rlandy|ruck is now known as rlandy|ruck|bbl22:52
*** d0ugal has quit IRC23:25
*** d0ugal has joined #oooq23:41

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!