| rlandy | akahat|rover: pls check promoter c7 is not promoting | 00:45 |
|---|---|---|
| *** rlandy is now known as rlandy|out | 00:50 | |
| ysandeep | >> <rlandy> ysandeep: akahat|rover: periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-16.2 keeps failing to push containers - ack will check. | 05:59 |
| akahat|rover | ysandeep, yeah.. saw that.. first checking with promoter. | 06:01 |
| marios | o/ morning | 06:04 |
| marios | akahat|rover: let me know if i can help with something ? | 06:04 |
| marios | akahat|rover: something wrong with upstream? i see a few retry_limit @ https://zuul.openstack.org/status#tripleo | 06:06 |
| akahat|rover | marios, yeah.. sure i'll let you know. about retry_limits.. not sure. if we saw more failures then may be we need to check with ops. | 06:16 |
| *** amoralej|off is now known as amoralej | 07:10 | |
| akahat|rover | marios, ysandeep Q: When new hash is pointing to tripleo-ci-testing, when it's container going to build? | 07:21 |
| ysandeep | In container build job in integration line run | 07:22 |
| marios | akahat|rover: o/ | 07:22 |
| marios | akahat|rover: what ysandeep said.. there is a continer-build-push or something like that job in th eintegration lines | 07:23 |
| ysandeep | akahat|rover, hash promote from promoted-component -> tripleo-ci-testing at the start of pipeline run in promote job , and then container build job trigger | 07:23 |
| akahat|rover | consider hash is pointing to the tripleo-ci-testing and integration lines are yet to run. But jobs which are running using this hash are getting failed | 07:23 |
| marios | akahat|rover: periodic-tripleo-ci-build-containers-centos-9-push-master | 07:24 |
| akahat|rover | check: openstack-component-common line | 07:24 |
| akahat|rover | https://review.rdoproject.org/zuul/status | 07:24 |
| ysandeep | looking | 07:24 |
| ysandeep | marios, component jobs use current-tripleo hash | 07:24 |
| ysandeep | marios, sry unping | 07:24 |
| ysandeep | akahat|rover, component jobs uses current-tripleo hash | 07:24 |
| ysandeep | containers.. | 07:25 |
| ysandeep | akahat|rover, https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-common-master/87d8adc/logs/undercloud/home/zuul/containers-prepare-parameters.yaml.txt.gz | 07:26 |
| akahat|rover | ysandeep, if that the case then, why containers are no there in registry | 07:26 |
| ysandeep | tag: 7c3595fcdce0ec20189de8d5b99dec16 ---> its current-tripleo https://trunk.rdoproject.org/centos9-master/current-tripleo/delorean.repo.md5 | 07:26 |
| marios | akahat|rover: 2022-04-05 02:41:56.358337 | fa163e3c-8869-dccc-9443-000000001504 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: https://trunk.registry.rdoproject.org/v2/tripleomastercentos9/openstack-keystone/manifests/7c3595fcdce0ec20189de8d5b99dec16", "msg": "Error running container image prepare: Not found image: | 07:26 |
| marios | https://trunk.registry.rdoproject.org/v2/tripleomastercentos9/openstack-keystone/manifests/7c3595fcdce0ec20189de8d5b99dec16", "params": {}, "success": false} | 07:26 |
| marios | akahat|rover: strange it cant find the container | 07:26 |
| akahat|rover | https://sf.hosted.upshift.rdu2.redhat.com/logs/9c/9c27aacfcba453c8e739a2d36929375d9dd6a958/openstack-periodic-integration-rhos-16.2/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-16.2/0bffa7c/logs/build.log | 07:27 |
| akahat|rover | https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario001-standalone-common-master/96d683a/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 07:27 |
| ysandeep | something pruned that? when was last master promotion.. | 07:28 |
| marios | ysandeep: akahat|rover: https://trunk.registry.rdoproject.org:8443/oapi/v1/namespaces/tripleomastercentos9/imagestreamtags/ i see it there | 07:28 |
| marios | "name": "openstack-keystone:7c3595fcdce0ec20189de8d5b99dec16_x86_64", | 07:28 |
| akahat|rover | https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario002-standalone-common-master/9e88903/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 07:28 |
| akahat|rover | marios, yeah.. but those jobs are not able to pull the containers. | 07:29 |
| marios | akahat|rover: yes the whole line seems https://review.rdoproject.org/zuul/status#openstack-component-common | 07:30 |
| marios | was it a recent promotion? /me checks | 07:30 |
| marios | hmm no couple days now? | 07:31 |
| akahat|rover | marios, yeah.. that whole line is affected | 07:31 |
| ysandeep | wait.. openstack-keystone:7c3595fcdce0ec20189de8d5b99dec16 doesn't exist | 07:31 |
| ysandeep | "name": "openstack-keystone:7c3595fcdce0ec20189de8d5b99dec16_x86_64", exists but its with x86_64 | 07:31 |
| ysandeep | https://trunk.registry.rdoproject.org:8443/oapi/v1/namespaces/tripleomastercentos9/imagestreamtags/ | 07:31 |
| marios | ysandeep: where i found it in the registry | 07:32 |
| marios | also has that one ysandeep | 07:32 |
| marios | "name": "openstack-keystone:7e7b7f6b4f791337ddc3c116902bd1d8", | 07:32 |
| marios | "namespace": "tripleomastercentos9", | 07:32 |
| marios | "creationTimestamp": "2022-04-01T17:13:11Z" | 07:32 |
| marios | strange why it cant pull | 07:32 |
| marios | not even that it cant find it | 07:32 |
| ysandeep | marios, ahh, correct I got my regex wrong in grep | 07:32 |
| marios | oh thats a different hash wait | 07:32 |
| marios | :D | 07:32 |
| marios | 7e7b7f 7c35 sec let me check | 07:33 |
| marios | ysandeep: so it is not there the openstack-keystone:7c3595fcdce0ec20189de8d5b99dec16 | 07:33 |
| marios | how/why ... ? | 07:33 |
| marios | only the "name": "openstack-keystone:7c3595fcdce0ec20189de8d5b99dec16_x86_64", | 07:33 |
| marios | and why only this line now after a few days hits that | 07:34 |
| ysandeep | marios, let's check in container build job.. if it build both hashes | 07:34 |
| ysandeep | if it build both hashes, then something pruned the containers | 07:35 |
| * ysandeep checks | 07:35 | |
| marios | ysandeep: akahat|rover: so you should rerun container build push | 07:35 |
| marios | akahat|rover: just rerun the job and pass this hash | 07:35 |
| marios | and then it should be good | 07:35 |
| marios | ? | 07:35 |
| marios | ? | 07:35 |
| marios | profit | 07:35 |
| akahat|rover | marios, okay. i'll rerun job | 07:36 |
| marios | akahat|rover: does it make sense? rerun periodic-tripleo-ci-build-containers-centos-9-push-master with hash 7c3595fcdce0ec20189de8d5b99dec16 | 07:36 |
| akahat|rover | meanwhile I open lp: https://bugs.launchpad.net/tripleo/+bug/1967833 | 07:36 |
| marios | akahat|rover: k | 07:36 |
| *** jpena|off is now known as jpena | 07:37 | |
| ysandeep | marios, akahat|rover job created containers with both tag: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-centos-9-push-master/d9ed429/logs/containers-built.log | 07:42 |
| ysandeep | something pruned that then. | 07:42 |
| akahat|rover | i'm re running job here: https://review.rdoproject.org/r/c/testproject/+/41465 | 07:43 |
| akahat|rover | let's see. | 07:43 |
| * akahat|rover lunch | 07:43 | |
| marios | ysandeep: ack thanks for checking | 07:47 |
| ysandeep | no recent changes in https://github.com/rdo-infra/rdo-infra-playbooks/tree/master/roles/rdo-infra/registry-image-pruning | 07:48 |
| ysandeep | akahat|rover: I think we should run by this issue with someone from infra team may be dpawlik to check how/why those container tag were pruned. | 07:49 |
| * dpawlik reading | 07:58 | |
| dpawlik | so the promotion 7c3595fcdce0ec20189de8d5b99dec16 was removed? | 07:59 |
| dpawlik | I don't see in pruner logs that it remove such images | 08:03 |
| marios | dpawlik: don't think it would be the pruner... it is a relatively new hash. may be something else we don't know yet | 08:05 |
| *** ysandeep is now known as ysandeep|lunch | 08:28 | |
| *** ysandeep|lunch is now known as ysandeep | 09:00 | |
| *** pojadhav is now known as pojadhav|afk | 09:50 | |
| *** pojadhav|afk is now known as pojadhav | 10:12 | |
| marios | ysandeep: chandankumar: please review when you have time https://review.opendev.org/c/openstack/tripleo-ci/+/834861/ & https://review.rdoproject.org/r/c/rdo-jobs/+/40842 | 10:22 |
| ysandeep | marios, ack | 10:22 |
| marios | thanks | 10:23 |
| ysandeep | hah, we don't have -option naming for sc010 | 10:25 |
| *** rlandy|out is now known as rlandy | 10:27 | |
| marios | ysandeep: yeah it has its own base job ... :/ | 10:27 |
| marios | :) | 10:27 |
| rlandy | ysandeep: thanks for logging ticket | 10:31 |
| rlandy | akahat|rover: chandankumar: ysandeep: ley | 10:31 |
| rlandy | ;et | 10:31 |
| rlandy | slet's sync | 10:31 |
| ysandeep | rlandy, akahat|rover ++ logged the ticket, I just gave him an example | 10:31 |
| rlandy | https://meet.google.com/pfy-vzus-yhw?pli=1&authuser=0 | 10:31 |
| rlandy | akahat|rover++ | 10:32 |
| rlandy | akahat|rover: ^^ pls join | 10:32 |
| ysandeep | rlandy, joining in a min, searching my earphones | 10:32 |
| rlandy | chandankumar: ^^ | 10:32 |
| rlandy | we need to bring in the reinforcements here | 10:32 |
| rlandy | akahat|rover: have you checked the promoter? | 10:33 |
| rlandy | train c7 still trying to promote | 10:33 |
| ysandeep | chandankumar, do you know if we have something like this in internal: https://trunk.registry.rdoproject.org:8443/oapi/v1/namespaces/tripleomastercentos9/imagestreamtags/ | 10:46 |
| ysandeep | where we can see all the tags. | 10:46 |
| chandankumar | ysandeep, I have not seen such url there | 10:50 |
| ysandeep | chandankumar: ack, thanks | 10:50 |
| rlandy | ysandeep: chandankumar: akahat|rover: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/23894 | 10:56 |
| ysandeep | rlandy, https://review.opendev.org/c/openstack/tripleo-ci/+/836422 | 11:00 |
| akahat|rover | chandankumar, ysandeep rlandy https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41506 | 11:05 |
| rlandy | akahat|rover: python3 roles/rrcockpit/files/telegraf_py3/ruck_rover.py --release train --distro centos-8 --aggregate_hash tripleo-ci-testing/c9/b9/c9b936e84e19bd33f9f0cefeac8f8461 | 11:10 |
| chandankumar | ysandeep, want this temp exclude fix https://review.opendev.org/c/openstack/tripleo-ci/+/836422 merged? | 11:22 |
| ysandeep | chandankumar: leave it till container registry issue resolves. | 11:22 |
| chandankumar | ysandeep, ack | 11:22 |
| rlandy | ysandeep: ack - let's merge tomorrow | 11:23 |
| rlandy | we can't push as it is | 11:23 |
| ysandeep | rlandy: can we merge https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/39335 , so that I have time to confirm everything works on our dashboard before I leave today. | 11:26 |
| rlandy | ysandeep: yep - go ahead | 11:36 |
| rlandy | so you can revert if need be | 11:36 |
| ysandeep | thanks! | 11:36 |
| rlandy | akahat|rover: pls join #rhos-delivery | 11:44 |
| rlandy | we are discussing the registry there | 11:44 |
| * bhagyashris brb | 11:54 | |
| chandankumar | akahat|rover, c8s wallaby tripleo network component testproject https://review.rdoproject.org/r/c/testproject/+/40642 | 11:56 |
| akahat|rover | chandankumar, ack | 11:57 |
| rlandy | akahat|rover: put in testproject for fs035 master | 11:58 |
| akahat|rover | rlandy, okay | 11:59 |
| *** beagles_ptg is now known as beagles | 12:01 | |
| rlandy | chandankumar: akahat|rover: dasm|off: 16.2 components are clear ... 17 components need some help | 12:03 |
| rlandy | network | 12:03 |
| rlandy | possibly glance | 12:04 |
| chandankumar | looking at 17 component network line | 12:04 |
| rlandy | cleared a few on those yesterday | 12:04 |
| rlandy | chandankumar: rhel-9 and rhel-8 pls | 12:04 |
| rlandy | ^^ you can pass that info to dasm|off when he is in | 12:04 |
| chandankumar | sure! | 12:06 |
| rlandy | chandankumar: these were the ones I did not get to yesterday: | 12:06 |
| rlandy | * periodic-tripleo-rhel-8-rhos-17-component-clients-promote-to-promoted-components | 12:07 |
| rlandy | * periodic-tripleo-rhel-8-rhos-17-component-network-promote-to-promoted-components | 12:07 |
| rlandy | * periodic-tripleo-rhel-8-rhos-17-component-tripleo-promote-to-promoted-components | 12:07 |
| chandankumar | ok going over rhel8 rhos17 | 12:08 |
| rlandy | chandankumar: periodic-tripleo-rhel-9-rhos-17-component-network-promote-to-promoted-components was success at promoting so that one looks ok | 12:08 |
| *** amoralej is now known as amoralej|lunch | 12:15 | |
| frenzy_friday | chandankumar, rlandy, marios pls add to your review lists : https://review.rdoproject.org/r/c/config/+/41514 (c8 build containers job for quay) | 12:26 |
| reviewbot | I have added your review the to Review list | 12:26 |
| rlandy | frenzy_friday: ack - will check review list Hackmd - pls put on there | 12:27 |
| marios | frenzy_friday: sure will do | 12:31 |
| marios | rcastillo: o/ fyi your patch came up on openstack-discuss in case you missed it ;) http://lists.openstack.org/pipermail/openstack-discuss/2022-April/027990.html | 12:32 |
| marios | rlandy: fyi http://lists.openstack.org/pipermail/openstack-discuss/2022-April/028026.html | 12:33 |
| rlandy | marios+++++++++++ | 12:34 |
| rlandy | yeah - get that thing gone!! | 12:34 |
| * akahat|rover back in few mins | 12:35 | |
| marios | rlandy: ;) | 12:36 |
| rlandy | akahat|rover: pls join #openstack-pcci on internal | 12:39 |
| rlandy | dasm|off: ^^ you too | 12:40 |
| rlandy | pining attila re: failed jenkins jobs | 12:40 |
| chandankumar | rlandy, akahat|rover dasm|off fyi 834861: Rename the centos-8-scenarioxxx-standalone-options jobs to generic | https://review.opendev.org/c/openstack/tripleo-ci/+/834861 approving it, not to break the world | 13:09 |
| rlandy | chandankumar: ' not to break the world'?? | 13:09 |
| rlandy | it is already broken? | 13:09 |
| chandankumar | rlandy, sorry, *hope it does not break the CI | 13:10 |
| rlandy | chandankumar: now is probably not the time | 13:10 |
| rlandy | can it hold? | 13:10 |
| chandankumar | rlandy, sure, will merge tomorrow morning | 13:10 |
| rlandy | chandankumar: ack - thanks | 13:10 |
| *** amoralej|lunch is now known as amoralej | 13:13 | |
| rcastillo | marios: I saw that indeed thanks for bringing it up :) | 13:25 |
| marios | rcastillo: my favorite bit was where "the team discussed it" ;) | 13:30 |
| marios | :D | 13:30 |
| marios | (I wasn't involved in any conversations i am guessing you had them; with yourself also counts ) | 13:30 |
| marios | rcastillo: thanks for working on that | 13:31 |
| rcastillo | marios: there were maybe two messages about it on here so that counts | 13:32 |
| marios | works for me ;) | 13:33 |
| marios | rcastillo:++ | 13:33 |
| rlandy | akahat|rover: https://review.rdoproject.org/r/c/testproject/+/41469 | 13:35 |
| rlandy | periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-victoria needs a rerun - no consistent | 13:35 |
| rlandy | periodic-tripleo-ci-centos-8-scenario001-standalone-victoria | 13:35 |
| rlandy | possibly consistent | 13:36 |
| rlandy | pls bug | 13:36 |
| rlandy | add skiplist | 13:36 |
| rlandy | run again | 13:36 |
| rlandy | dasm|off: ^^ | 13:36 |
| rlandy | marios: for ussuri, do we need one last promotion or nobody cares? | 13:47 |
| rlandy | it's 8 days out | 13:47 |
| marios | rlandy: nobody cares i think | 13:49 |
| marios | rlandy: we've already called it now | 13:49 |
| marios | rlandy: imo | 13:49 |
| rlandy | marios: perfect | 13:50 |
| rlandy | thank you for taking care of all that | 13:50 |
| rlandy | marios rocking out | 13:50 |
| marios | rlandy: ack ;) not that much... we should have more people that know how to do the releases stuff maybe i can show in some community call | 13:50 |
| rlandy | marios: absolutely | 13:52 |
| rlandy | marios: minimally I should know this workflow | 13:52 |
| dviroel | marios: +1 for sharing this knowledge with the team | 13:56 |
| marios | ack sure dviroel | 13:58 |
| *** dasm|off is now known as dasm | 13:59 | |
| *** dasm is now known as dasm|ruck | 14:00 | |
| dasm|ruck | o/ | 14:00 |
| rlandy | dasm|ruck: will sync with you after PTG | 14:08 |
| rlandy | there are a bunch of testprojects in fight | 14:08 |
| rlandy | internal is out for an hour or so | 14:08 |
| dasm|ruck | ack | 14:08 |
| rlandy | still following broken registry push | 14:08 |
| dasm|ruck | do we know what's wrong? | 14:08 |
| rlandy | dasm|ruck: infra or we have too many containers | 14:09 |
| rlandy | ysandeep, and dpawlik are working on a prune script for donwstream | 14:10 |
| dasm|ruck | ack | 14:11 |
| dasm|ruck | do we have any stats -- what it means "too many"? | 14:11 |
| ysandeep | rlandy, dpawlik is working on sf upgrade currently, /me trying to change script for downstream case before he comes back. | 14:11 |
| dasm|ruck | ysandeep: sf upgrade? are we expecting more downtime/issues? | 14:12 |
| * ysandeep attending PTG as well | 14:12 | |
| rlandy | ysandeep: ack we'll all parallel processing :) | 14:13 |
| rlandy | also infra probably needs to fix this | 14:13 |
| ysandeep | dasm|ruck, downstream zuul already out | 14:14 |
| ysandeep | The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later. | 14:14 |
| dasm|ruck | ysandeep: ack :/ | 14:14 |
| akahat|rover | dasm|ruck, rlandy ysandeep marios we are hitting this lp again: https://bugs.launchpad.net/tripleo/+bug/1967833 | 14:41 |
| akahat|rover | This time network line: https://review.rdoproject.org/zuul/status#openstack-component-network | 14:41 |
| akahat|rover | only x86_64 containers are there in the registry. | 14:42 |
| ysandeep | akahat|rover: rerunning container build job didn't push containers without x86_64 ? | 14:42 |
| rlandy | akahat|rover; master is heading up 5 days old | 14:42 |
| rlandy | possible got pruned | 14:42 |
| rlandy | akahat|rover: can you try arxcruz's script | 14:43 |
| rlandy | manual run | 14:43 |
| rlandy | and reru those jobs | 14:43 |
| akahat|rover | ysandeep, here it says it build : https://logserver.rdoproject.org/65/41465/2/check/periodic-tripleo-ci-build-containers-centos-9-push-master/ac282a1/logs/containers-built.log | 14:43 |
| akahat|rover | but we didn't fond it there. | 14:43 |
| rlandy | akahat|rover: that is why we need to promote master today | 14:43 |
| rlandy | dasm|ruck: ^^ | 14:43 |
| rlandy | we are two jobs out | 14:43 |
| ysandeep | rlandy, something else is wrong there | 14:43 |
| rlandy | we may need to skip and promote | 14:43 |
| ysandeep | ~~~ | 14:43 |
| ysandeep | <dpawlik> so the promotion 7c3595fcdce0ec20189de8d5b99dec16 was removed? | 14:43 |
| ysandeep | <dpawlik> I don't see in pruner logs that it remove such images | 14:43 |
| ysandeep | <marios> dpawlik: don't think it would be the pruner... it is a relatively new hash. may be something else we don't know yet | 14:43 |
| ysandeep | ~~~ | 14:43 |
| rlandy | we see that after 5 days | 14:44 |
| akahat|rover | rlandy, waiting for those two master jobs | 14:44 |
| marios | ysandeep: did the new job run not help? | 14:44 |
| marios | ysandeep: sorry reading back | 14:44 |
| ysandeep | marios, yes that's what akahat|rover is mentioning: https://logserver.rdoproject.org/65/41465/2/check/periodic-tripleo-ci-build-containers-centos-9-push-master/ac282a1/logs/containers-built.log | 14:44 |
| ysandeep | he see job build those container but he don't find those on rdo registry | 14:45 |
| akahat|rover | we need this job: periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master https://review.rdoproject.org/zuul/status#41437 | 14:45 |
| ysandeep | akahat|rover, marios wait | 14:45 |
| ysandeep | it build with different hash: fde3f54bf41b89f0a02fac7d90513e07 | 14:46 |
| ysandeep | akahat|rover, could you share your testproject want to check what hash you passed? | 14:46 |
| akahat|rover | ysandeep, yeah has was different: https://review.rdoproject.org/r/c/testproject/+/41465/2/.zuul.yaml#8 | 14:47 |
| akahat|rover | looks like we need to run it again. | 14:47 |
| rlandy | akahat|rover: only fs001 | 14:47 |
| rlandy | I think | 14:47 |
| akahat|rover | rlandy, yeah fs01 | 14:47 |
| ysandeep | akahat|rover, don't use force_periodic: https://review.rdoproject.org/r/c/testproject/+/41465/2/.zuul.yaml | 14:47 |
| akahat|rover | ysandeep, okay | 14:47 |
| ysandeep | akahat|rover, I generally do something like this instead: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/208803/12/zuul.d/tripleo-build-containers.yaml | 14:48 |
| dasm|ruck | so, we don't have a week of SLA. It's much less, probably we're looking at 3 days max? | 14:49 |
| ysandeep | akahat|rover, so if you override buildcontainers_override_repos & promote_source in vars that should work.. | 14:49 |
| ysandeep | akahat|rover, do you mind if I update your testpatch. | 14:50 |
| akahat|rover | ysandeep, yeah. sure | 14:50 |
| ysandeep | akahat|rover, let see how this goes: https://review.rdoproject.org/r/c/testproject/+/41465/4/.zuul.yaml | 14:51 |
| akahat|rover | ysandeep, words better than numbers :P | 14:52 |
| rlandy | akahat|rover: dasm|ruck: fs001 is in rerun - if it fails, let's temp get that out of criteria and promote | 14:53 |
| akahat|rover | rlandy, ack | 14:53 |
| ysandeep | akahat|rover, your previous patch looks good, but looks like container build job doesn't work well with dlrn_hash_tag. | 14:55 |
| dasm|ruck | rlandy: ack | 14:55 |
| *** pojadhav is now known as pojadhav|afk | 14:56 | |
| chandankumar | ysandeep, akahat|rover you might want to rename the job and remove periodic from that to work for current-tripleo due to this https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/tripleo-build-jobs-repos/tasks/load-vars.yaml#L13 | 14:56 |
| chandankumar | in case of container build | 14:57 |
| rlandy | dasm|ruck: pls prep a patch to tem get fs001 out of master promotion | 15:00 |
| dasm|ruck | ack | 15:00 |
| rlandy | akahat|rover: we will have the same problem if we don;t promote wallaby c8 soon | 15:03 |
| rlandy | you may need the a similar jobs there | 15:03 |
| ysandeep | chandankumar, lets see how this goes: https://review.rdoproject.org/r/c/testproject/+/41465/5/.zuul.yaml | 15:03 |
| chandankumar | ysandeep, ack, thanks! | 15:04 |
| ysandeep | rlandy, do you have few mins to sync about containers cleanup. | 15:07 |
| rlandy | ysandeep:yep - in 5 | 15:07 |
| marios | chandankumar: rlandy: its almost my end of day but as i said feel free to revert that patch if you need to... otherwise we can continue to dig tomorrow depending on how the testing goes now (alfredo is postign testproject) | 15:11 |
| dasm|ruck | akahat|rover: rlandy https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41557 | 15:12 |
| chandankumar | marios, sure | 15:12 |
| rlandy | marios: ack - if the testproject works, can just merge it | 15:13 |
| * akahat|rover back in half hour | 15:13 | |
| chandankumar | rlandy, logging out now. testproject links on hackmd | 15:13 |
| dasm|ruck | chandankumar: thanks | 15:14 |
| rlandy | ysandeep: https://meet.google.com/ovj-hgvp-ukg?pli=1&authuser=0 | 15:15 |
| *** marios is now known as marios|out | 15:47 | |
| rlandy | arxcruz: hey - when will your script run again? | 15:50 |
| rlandy | we need master and wallaby c8 now | 15:50 |
| arxcruz | rlandy need to merge https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41439 | 15:51 |
| arxcruz | rlandy i'll run manually now | 15:52 |
| rlandy | arxcruz: pls do | 15:52 |
| rlandy | w+'ed | 15:52 |
| arxcruz | rlandy running | 15:52 |
| rlandy | arxcruz++ | 15:52 |
| rlandy | thank you | 15:52 |
| dasm|ruck | https://review.rdoproject.org/r/c/testproject/+/41278 | 15:55 |
| dasm|ruck | rlandy: akahat|rover ^ | 15:55 |
| akahat|rover | rlandy, http://promoter.rdoproject.org/promoter_logs/centos9_master.log | 15:56 |
| *** pojadhav|afk is now known as pojadhav | 15:58 | |
| * ysandeep out for the day | 16:03 | |
| *** ysandeep is now known as ysandeep|out | 16:03 | |
| akahat|rover | dasm|ruck, rlandy http://promoter.rdoproject.org/promoter_logs/centos9_master.log | 16:07 |
| dasm|ruck | rlandy: akahat|rover https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41558 | 16:08 |
| rlandy | dasm|ruck: https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/rrcockpit | 16:09 |
| rlandy | lunch - brb | 16:15 |
| * dviroel back from lunch | 16:28 | |
| *** jpena is now known as jpena|off | 16:31 | |
| arxcruz | rlandy akahat|rover everything copied | 16:44 |
| dasm|ruck | 2022-04-05 16:41:23,423 2510338 ERROR promoter Error while trying to promote tripleo-ci-testing to current-tripleo | 16:52 |
| dasm|ruck | 2022-04-05 16:41:23,424 2510338 WARNING promoter Candidate label 'tripleo-ci-testing': NO candidate hash promoted to current-tripleo | 16:52 |
| dasm|ruck | http://promoter.rdoproject.org/promoter_logs/centos9_master.log | 16:52 |
| dasm|ruck | akahat|rover: ^ | 16:53 |
| rlandy | dasm|ruck: looks like it's trying again | 17:00 |
| dasm|ruck | ack | 17:01 |
| *** dasm|ruck is now known as dasm|ruck|mtg | 17:01 | |
| akahat|rover | Promotion is failing.. it is same issue which we were facing for c7_train. Sometimes system is failed to push container to registry. that's why it is failing. | 17:29 |
| akahat|rover | it's rerunning again.. let's see. | 17:30 |
| rlandy | dasm|ruck|mtg: pls rekick your jobs | 17:31 |
| rlandy | and testprojects | 17:31 |
| *** dasm|ruck|mtg is now known as dasm|ruck | 17:57 | |
| rlandy | dasm|ruck: waiting for master to successfully promote | 18:05 |
| rlandy | then we need to revert that criteria and we can try merge your patch | 18:05 |
| rlandy | dasm|ruck: looking for akahat|rover's victotia rerun | 18:06 |
| dasm|ruck | ack | 18:07 |
| rlandy | dasm|ruck: rekicked https://review.rdoproject.org/r/c/testproject/+/41469 | 18:09 |
| rlandy | scenario001 may be legit failure | 18:09 |
| rlandy | if so, we need a bug and a skiplist entry | 18:09 |
| dasm|ruck | latest test run failed due to mismatch in python deps: https://logserver.rdoproject.org/69/41469/2/check/periodic-tripleo-ci-centos-8-scenario001-standalone-victoria/3a02c5d/job-output.txt | 18:12 |
| dasm|ruck | > subunit2sql-0.8.0-py2.py3-none-any.whl (59 kB)\nINFO: pip is looking at multiple versions of python-subunit to determine which version is compatible with other requirements | 18:12 |
| dasm|ruck | checking if that's something new | 18:13 |
| rlandy | dasm|ruck: pls see https://bugs.launchpad.net/tripleo/+bug/1965540 | 18:18 |
| rlandy | looks like skiplist revert was merged | 18:18 |
| rlandy | can we close this out? | 18:18 |
| dasm|ruck | checking | 18:18 |
| dasm|ruck | i believe so. closing | 18:19 |
| rlandy | thanks | 18:20 |
| dasm|ruck | i see a lot of successes there | 18:21 |
| dasm|ruck | rlandy: it's already as > status: In Progress → Fix Released | 18:23 |
| dasm|ruck | I don't think I can "Close" it, can I? | 18:24 |
| rlandy | dasm|ruck: sorry - closing our related Bugzilla | 18:24 |
| dasm|ruck | ah | 18:24 |
| rlandy | it's fine - the launchpad is correct | 18:24 |
| rlandy | 2022-04-05 18:15:25.304740 | primary | ERROR: Cannot install -r requirements.txt (line 18) because these package versions have conflicting dependencies. | 18:25 |
| rlandy | 2022-04-05 18:15:25.305043 | primary | | 18:25 |
| rlandy | 2022-04-05 18:15:25.305079 | primary | The conflict is caused by: | 18:25 |
| rlandy | 2022-04-05 18:15:25.305123 | primary | oslo-config 8.5.1 depends on stevedore>=1.20.0 | 18:25 |
| rlandy | 2022-04-05 18:15:25.305141 | primary | The user requested (constraint) stevedore===3.3.1 | 18:25 |
| rlandy | here we go again | 18:25 |
| dasm|ruck | where do you see that? | 18:28 |
| rlandy | wallaby c8 rerun | 18:29 |
| rlandy | fs001 | 18:29 |
| dasm|ruck | i see similar for victoria: https://logserver.rdoproject.org/69/41469/2/check/periodic-tripleo-ci-centos-8-scenario001-standalone-victoria/3a02c5d/job-output.txt | 18:30 |
| dasm|ruck | but | 18:30 |
| dasm|ruck | > The conflict is caused by:\n subunit2sql 1.5.0 depends on oslo.db!=1.12.0 and <2.0.0\n | 18:30 |
| dasm|ruck | heh | 18:30 |
| dasm|ruck | rlandy: wrt https://bugs.launchpad.net/tripleo/+bug/1965540 i can't find bugzilla associated with that. did we have any? cix card doesn't have any pointers | 18:32 |
| dasm|ruck | checking rr hackmd | 18:32 |
| *** rlandy is now known as rlandy|mtg | 18:33 | |
| *** rlandy|mtg is now known as rlandy | 19:02 | |
| rlandy | dasm|ruck: pls join review time | 19:03 |
| rlandy | I closed out the BZ | 19:03 |
| dasm|ruck | rlandy: https://logserver.rdoproject.org/78/41278/8/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby/b564518/job-output.txt | 19:13 |
| dasm|ruck | rlandy: https://opendev.org/openstack/requirements/src/branch/stable/wallaby/upper-constraints.txt | 19:15 |
| rlandy | dasm|ruck: you mentioned update 4 days ago | 19:51 |
| rlandy | jm1: ^^ | 19:51 |
| dasm|ruck | rlandy: it might be red herring | 19:53 |
| rlandy | https://opendev.org/openstack/requirements/commit/8b04528e19dbc0c9ebc2b0674eebc3f2bcc86a4a | 19:53 |
| rlandy | last update | 19:53 |
| dasm|ruck | rlandy: job started failing about 5h ago | 19:54 |
| dasm|ruck | * C8 Wallaby https://bugs.launchpad.net/tripleo/+bug/1967943 | 19:54 |
| dasm|ruck | * C9 Master|Wallaby: https://bugs.launchpad.net/tripleo/+bug/1967945 | 19:54 |
| jm1 | rlandy: not sure what i should see? | 19:57 |
| dasm|ruck | jm1: i created 2 bugs. we're having issues with versions mismatch at gates. | 19:59 |
| dasm|ruck | Not sure if that's anyhow connected to you | 19:59 |
| jm1 | dasm|ruck: how can it be connected to me? | 20:00 |
| dasm|ruck | jm1: you're playing with openstack sdk. Just trying piece elements to find what could go wrong | 20:04 |
| dasm|ruck | > openstacksdk===0.55.1 | 20:04 |
| dasm|ruck | that's one of latest changes | 20:04 |
| dasm|ruck | dviroel: wrt https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41458 i had to rebase. can you +W it again? | 20:06 |
| dviroel | dasm|ruck: done | 20:08 |
| dasm|ruck | dviroel++ thank you | 20:12 |
| jm1 | dasm|ruck: i am using openstacksdk but did sent a patch only once a while ago. looks like these jobs are failing often. have no idea yet what could have caused this | 20:12 |
| dasm|ruck | jm1: no worries. we're gonna figure that out | 20:13 |
| jm1 | dasm|ruck: it is probably unrelated to openstacksdk 0.55.1 because pip install worked with this sdk at least once before: https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby/b9f9699/job-output.txt | 20:19 |
| dasm|ruck | ack | 20:22 |
| dasm|ruck | jm1: just trying to eliminate all unknown variables | 20:23 |
| rlandy | dasm|ruck; one more ... | 20:23 |
| rlandy | https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-train/45d2ac6/job-output.txt | 20:23 |
| rlandy | killing train | 20:24 |
| rlandy | send that one to arxcruz | 20:24 |
| * rlandy needs to get on meeting | 20:24 | |
| dasm|ruck | ack | 20:24 |
| rlandy | jm1: sorry - this one is for you https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_aad/834938/1/gate/tripleo-ci-centos-9-content-provider/aad014f/job-output.txt | 20:25 |
| rlandy | 2022-04-05 15:23:09.659126 | primary | The conflict is caused by: | 20:25 |
| rlandy | 2022-04-05 15:23:09.659138 | primary | os-client-config 2.1.0 depends on openstacksdk>=0.13.0 | 20:25 |
| rlandy | 2022-04-05 15:23:09.659149 | primary | The user requested (constraint) openstacksdk===0.55.1 | 20:25 |
| rlandy | https://review.opendev.org/c/openstack/tripleo-heat-templates/+/834938/ | 20:25 |
| rlandy | https://e6807958cf1af5738f4c-b8d0e37c8407321b7ba0b13922acea87.ssl.cf5.rackcdn.com/834938/1/gate/tripleo-ci-centos-8-content-provider/75d9c4e/job-output.txt | 20:26 |
| *** rlandy is now known as rlandy|mtg | 20:26 | |
| dasm|ruck | it sounds like one thing which went wrong. I think it might be connected to one and the same thing, just with different outcomes | 20:27 |
| jm1 | dasm|ruck: yep, looks like pip's dependency resolution has issues. probably not because of pip itself but because either requirements.txt or quickstart-extras-requirements.txt has changed | 20:30 |
| dasm|ruck | yes | 20:31 |
| rcastillo | https://github.com/pypa/pip/issues/11009 | 20:31 |
| rcastillo | related? | 20:31 |
| jm1 | dasm|ruck: actually both files have not changed for at least 5 month, so it must be something else | 20:31 |
| dasm|ruck | rcastillo++ | 20:32 |
| dasm|ruck | that actually might be it. | 20:32 |
| dasm|ruck | checking pip version | 20:32 |
| dasm|ruck | we started seeing the issue today, around 1500 UTC | 20:32 |
| * jm1 too late for me, off for today 👋 | 20:34 | |
| dasm|ruck | jm1: o/ | 20:34 |
| *** dviroel is now known as dviroel|out | 20:36 | |
| dasm|ruck | pip 22.0.4 has been released on Mar 6 | 20:38 |
| dasm|ruck | so, it's been a while | 20:38 |
| dasm|ruck | rcastillo: https://lists.openstack.org/pipermail/openstack-discuss/2022-April/028037.html cc rlandy|mtg | 20:42 |
| dasm|ruck | i'm gonna check that in a moment | 20:42 |
| rcastillo | yup, just saw | 20:43 |
| dasm|ruck | that could make sense | 20:43 |
| dasm|ruck | brb | 20:43 |
| dasm|ruck | back | 21:11 |
| dasm|ruck | https://status.python.org/incidents/mxgkk3xxr9v7?u=v8pzlr5n28h8 | 21:17 |
| dasm|ruck | it might be the issue we were seeing | 21:17 |
| dasm|ruck | rechecking some thingr | 21:17 |
| dasm|ruck | *thinds | 21:17 |
| dasm|ruck | *things | 21:17 |
| dasm|ruck | (i can't type) | 21:17 |
| dasm|ruck | akahat|rover: hey. we had some blip with pypi. I'm rerunning jobs to see if that's intermittent. Please see https://hackmd.io/M1UXHF7iTbmJFkK4bQET9g?both#Apr-05-2022 for more details. | 21:27 |
| dasm|ruck | i'm still seeing issues with packages. | 21:42 |
| dasm|ruck | akahat|rover: i asked on #vexxhost i didn't get a follow up yet. | 21:42 |
| dasm|ruck | probably you'd need to check that tomorrow | 21:43 |
| rlandy|mtg | dasm|ruck: ok - I'm going to try promote 16.2 again | 22:20 |
| dasm|ruck | rlandy|mtg: that's hit or miss. i rerun jobs and they failed again | 22:21 |
| dasm|ruck | on vexx at least | 22:21 |
| dasm|ruck | i didn't try downstream | 22:21 |
| dasm|ruck | *reran | 22:21 |
| *** dasm|ruck is now known as dasm|off | 22:43 | |
| *** rlandy|mtg is now known as rlandy | 23:16 | |
| rlandy | dasm|off: ysandeep|out: akahat|rover: ticket was answered, registry push stilled failed for me - ols try again in your morning | 23:54 |
| *** rlandy is now known as rlandy|out | 23:55 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!