| *** ahosam has quit IRC | 00:03 | |
| *** jamesmcarthur has joined #openstack-infra | 00:06 | |
| *** d34dh0r53 has quit IRC | 00:14 | |
| *** cloudnull has quit IRC | 00:14 | |
| *** rh-jelabarre has quit IRC | 00:29 | |
| *** rfolco has quit IRC | 00:32 | |
| *** gyee has quit IRC | 00:58 | |
| *** jamesmcarthur has quit IRC | 01:36 | |
| *** jamesmcarthur has joined #openstack-infra | 01:38 | |
| *** jamesmcarthur has quit IRC | 01:43 | |
| *** jamesmcarthur has joined #openstack-infra | 01:45 | |
| *** zhurong has joined #openstack-infra | 01:52 | |
| *** zhurong has joined #openstack-infra | 01:54 | |
| *** spsurya has quit IRC | 02:03 | |
| *** jamesmcarthur has quit IRC | 02:19 | |
| *** jamesmcarthur has joined #openstack-infra | 02:28 | |
| *** lbragstad has quit IRC | 02:32 | |
| *** lbragstad has joined #openstack-infra | 02:33 | |
| *** cloudnull has joined #openstack-infra | 02:54 | |
| *** d34dh0r53 has joined #openstack-infra | 02:55 | |
| *** rlandy|bbl has quit IRC | 03:29 | |
| *** mattw4 has quit IRC | 03:31 | |
| *** jamesmcarthur has quit IRC | 03:42 | |
| *** jamesmcarthur has joined #openstack-infra | 04:07 | |
| *** Lucas_Gray has joined #openstack-infra | 04:16 | |
| *** ykarel|away has joined #openstack-infra | 04:31 | |
| *** jamesmcarthur has quit IRC | 04:45 | |
| *** jamesmcarthur has joined #openstack-infra | 04:46 | |
| *** jamesmcarthur has quit IRC | 04:51 | |
| *** goldyfruit has joined #openstack-infra | 04:54 | |
| *** ramishra has joined #openstack-infra | 04:57 | |
| *** ociuhandu has joined #openstack-infra | 05:06 | |
| *** ociuhandu has quit IRC | 05:11 | |
| *** jamesmcarthur has joined #openstack-infra | 05:17 | |
| *** jamesmcarthur has quit IRC | 05:29 | |
| *** Lucas_Gray has quit IRC | 05:29 | |
| *** ociuhandu has joined #openstack-infra | 05:30 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-infra | 05:33 | |
| *** ociuhandu has quit IRC | 05:35 | |
| *** jamesmcarthur has joined #openstack-infra | 05:40 | |
| *** surpatil has joined #openstack-infra | 05:46 | |
| *** jamesmcarthur has quit IRC | 06:14 | |
| *** ykarel|away is now known as ykarel | 06:25 | |
| *** SurajPatil has joined #openstack-infra | 06:59 | |
| *** surpatil has quit IRC | 07:02 | |
| *** ahosam has joined #openstack-infra | 07:09 | |
| *** surpatil has joined #openstack-infra | 07:11 | |
| *** SurajPatil has quit IRC | 07:13 | |
| *** pgaxatte has joined #openstack-infra | 07:13 | |
| *** pcrews has quit IRC | 07:14 | |
| *** jamesmcarthur has joined #openstack-infra | 07:20 | |
| *** jamesmcarthur has quit IRC | 07:27 | |
| *** dpawlik has joined #openstack-infra | 07:42 | |
| *** psachin has joined #openstack-infra | 08:00 | |
| *** jtomasek has joined #openstack-infra | 08:04 | |
| *** ykarel is now known as ykarel|lunch | 08:05 | |
| *** tesseract has joined #openstack-infra | 08:14 | |
| *** ociuhandu has joined #openstack-infra | 08:22 | |
| *** jamesmcarthur has joined #openstack-infra | 08:23 | |
| *** ociuhandu has quit IRC | 08:27 | |
| *** jamesmcarthur has quit IRC | 08:28 | |
| *** tosky has joined #openstack-infra | 08:34 | |
| *** xek_ has joined #openstack-infra | 08:34 | |
| *** Lucas_Gray has joined #openstack-infra | 08:34 | |
| *** pcaruana has joined #openstack-infra | 08:35 | |
| *** xek has quit IRC | 08:37 | |
| *** rpittau|afk is now known as rpittau | 08:41 | |
| *** Lucas_Gray has quit IRC | 08:48 | |
| *** Lucas_Gray has joined #openstack-infra | 08:58 | |
| *** pkopec has joined #openstack-infra | 09:03 | |
| *** ykarel|lunch is now known as ykarel | 09:05 | |
| *** lucasagomes has joined #openstack-infra | 09:13 | |
| *** psachin has quit IRC | 09:18 | |
| *** Lucas_Gray has quit IRC | 09:26 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report retried builds in a build set via mqtt. https://review.opendev.org/632727 | 09:27 |
|---|---|---|
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report retried builds via sql reporter. https://review.opendev.org/633501 | 09:27 |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Store information about gate resets in MQTT and SQL reporter https://review.opendev.org/696670 | 09:27 |
| *** ociuhandu has joined #openstack-infra | 09:32 | |
| *** Lucas_Gray has joined #openstack-infra | 09:44 | |
| *** SurajPatil has joined #openstack-infra | 09:59 | |
| *** derekh has joined #openstack-infra | 10:00 | |
| *** dtantsur|afk is now known as dtantsur | 10:02 | |
| *** surpatil has quit IRC | 10:02 | |
| *** ociuhandu has quit IRC | 10:06 | |
| *** ociuhandu has joined #openstack-infra | 10:07 | |
| *** ykarel is now known as ykarel|afk | 10:16 | |
| *** surpatil has joined #openstack-infra | 10:32 | |
| *** roman_g has quit IRC | 10:33 | |
| *** SurajPatil has quit IRC | 10:34 | |
| *** ociuhandu has quit IRC | 10:38 | |
| *** hrw has joined #openstack-infra | 10:39 | |
| hrw | morning | 10:39 |
| hrw | fungi, corvus: I added you to https://review.opendev.org/#/c/701011 patch. It adds job to kolla which depends on two other ones. Not sure did I made it right | 10:40 |
| *** ykarel|afk is now known as ykarel | 10:42 | |
| *** SurajPatil has joined #openstack-infra | 10:59 | |
| *** surpatil has quit IRC | 11:02 | |
| AJaeger | hrw: there's no dependency in that patch on other jobs | 11:02 |
| AJaeger | oh, wait - there it is, far bottom. | 11:02 |
| AJaeger | I would have done it differently, let me check... | 11:02 |
| hrw | AJaeger: it's my first time playing with job dependencies | 11:03 |
| hrw | open to all comments | 11:03 |
| AJaeger | indeed, that syntax is fine. | 11:05 |
| AJaeger | The multiarch job is not run at all, you need to add it to the project stanza | 11:05 |
| hrw | ok | 11:05 |
| hrw | done | 11:06 |
| hrw | have some changes to do and then will send update | 11:07 |
| AJaeger | hrw: also, you have a file-matcher in the kolla-base job - but don't use one in the multi-arch. See https://zuul-ci.org/docs/zuul/user/config.html#attr-job.dependencies.soft | 11:09 |
| *** surpatil has joined #openstack-infra | 11:09 | |
| AJaeger | Why not run this *temporarily* in the post pipeline instead of periodic? That way you can easily iterate on it.. | 11:10 |
| *** SurajPatil has quit IRC | 11:12 | |
| hrw | AJaeger: have to write proper playbook first and get rid of hardcoded values from shell script. when will play with moving publish jobs to check/check-arm64 to see how it goes | 11:12 |
| *** Lucas_Gray has quit IRC | 11:13 | |
| *** ociuhandu has joined #openstack-infra | 11:17 | |
| AJaeger | you might want to move to check and disable the publishing part... | 11:19 |
| hrw | yep | 11:23 |
| *** Lucas_Gray has joined #openstack-infra | 11:26 | |
| *** Lucas_Gray has quit IRC | 11:31 | |
| *** Lucas_Gray has joined #openstack-infra | 11:32 | |
| *** ociuhandu has quit IRC | 11:33 | |
| *** Lucas_Gray has quit IRC | 11:34 | |
| *** Lucas_Gray has joined #openstack-infra | 11:34 | |
| *** ociuhandu has joined #openstack-infra | 11:37 | |
| *** ociuhandu has quit IRC | 11:38 | |
| *** ociuhandu has joined #openstack-infra | 11:40 | |
| *** ociuhandu has quit IRC | 11:40 | |
| *** ociuhandu has joined #openstack-infra | 11:42 | |
| hrw | new version pushed, waiting for our cores to check does it make sense and then going to check does it work | 11:44 |
| *** ociuhandu has quit IRC | 11:48 | |
| openstackgerrit | Merged openstack/devstack-gate master: Add more logging to test_matrix.py https://review.opendev.org/700238 | 11:57 |
| openstackgerrit | Merged openstack/devstack-gate master: Disable c-bak and etcd3 in grenade jobs https://review.opendev.org/700233 | 11:57 |
| *** Lucas_Gray has quit IRC | 11:58 | |
| *** Lucas_Gray has joined #openstack-infra | 12:11 | |
| *** SurajPatil has joined #openstack-infra | 12:12 | |
| *** surpatil has quit IRC | 12:14 | |
| *** Lucas_Gray has quit IRC | 12:20 | |
| *** slaweq has joined #openstack-infra | 12:27 | |
| *** ociuhandu has joined #openstack-infra | 12:28 | |
| *** roman_g has joined #openstack-infra | 12:35 | |
| roman_g | re: > if only all of those companies would have spent as much effort improving openstack as they did on snark and on internal duplication projects | 12:38 |
| roman_g | mordred: totally agree | 12:38 |
| *** slaweq has quit IRC | 12:42 | |
| *** slaweq has joined #openstack-infra | 12:51 | |
| *** ociuhandu has quit IRC | 12:52 | |
| *** lseki has joined #openstack-infra | 12:58 | |
| *** surpatil has joined #openstack-infra | 13:06 | |
| *** slaweq has quit IRC | 13:06 | |
| *** rlandy has joined #openstack-infra | 13:08 | |
| *** SurajPatil has quit IRC | 13:08 | |
| *** ykarel is now known as ykarel|away | 13:16 | |
| *** ociuhandu has joined #openstack-infra | 13:17 | |
| *** ociuhandu has quit IRC | 13:21 | |
| *** ociuhandu has joined #openstack-infra | 13:23 | |
| *** rh-jelabarre has joined #openstack-infra | 13:26 | |
| *** slaweq has joined #openstack-infra | 13:26 | |
| *** ykarel|away has quit IRC | 13:29 | |
| *** pfallenop has quit IRC | 13:30 | |
| *** rh-jelabarre has quit IRC | 13:31 | |
| *** rh-jelabarre has joined #openstack-infra | 13:32 | |
| *** slaweq has quit IRC | 13:32 | |
| *** rfolco has joined #openstack-infra | 13:46 | |
| AJaeger | hrw: the Zuul part of your change is fine, I don't know whether all the variables in the playbook will have proper values. | 13:48 |
| *** surpatil has quit IRC | 13:56 | |
| *** ykarel has joined #openstack-infra | 14:07 | |
| *** ociuhandu has quit IRC | 14:13 | |
| *** ociuhandu has joined #openstack-infra | 14:14 | |
| hrw | AJaeger: that's for my team to check ;D | 14:21 |
| hrw | AJaeger: thanks for checking zuul part | 14:21 |
| AJaeger | ;) | 14:26 |
| *** weshay|ruck has quit IRC | 14:35 | |
| *** weshay has joined #openstack-infra | 14:42 | |
| *** weshay is now known as weshay|ruck | 14:42 | |
| openstackgerrit | Merged openstack/devstack-gate master: Add a feature support matrix test for grenade subnode https://review.opendev.org/700241 | 14:55 |
| *** slaweq has joined #openstack-infra | 14:57 | |
| *** adam_g has quit IRC | 14:57 | |
| *** adam_g has joined #openstack-infra | 14:58 | |
| *** adam_g has quit IRC | 15:09 | |
| *** adam_g has joined #openstack-infra | 15:11 | |
| *** slaweq has quit IRC | 15:29 | |
| *** dpawlik is now known as dpawlik|afk | 15:32 | |
| *** eernst has joined #openstack-infra | 15:32 | |
| *** jamesmcarthur has joined #openstack-infra | 15:34 | |
| *** bnemec is now known as beekneemech | 15:47 | |
| roman_g | clarkb: sent an e-mail to you regarding Airship clusters | 15:48 |
| roman_g | Hi :) | 15:48 |
| *** jamesmcarthur has quit IRC | 15:51 | |
| clarkb | roman_g: it would be helpful to move that discussion to the infra list (I asked chris for that in my previous email) | 15:59 |
| *** cmurphy is now known as cmorpheus | 16:00 | |
| roman_g | clarkb: Sure. Let me fwd it. Thank you. | 16:00 |
| *** adam_g has quit IRC | 16:02 | |
| *** ramishra has quit IRC | 16:08 | |
| *** toabctl has quit IRC | 16:08 | |
| *** toabctl has joined #openstack-infra | 16:08 | |
| *** ahosam has quit IRC | 16:09 | |
| *** jamesmcarthur has joined #openstack-infra | 16:14 | |
| *** ociuhandu has quit IRC | 16:19 | |
| *** mattw4 has joined #openstack-infra | 16:22 | |
| *** jamesmcarthur_ has joined #openstack-infra | 16:23 | |
| *** jamesmcarthur has quit IRC | 16:27 | |
| *** pgaxatte has quit IRC | 16:30 | |
| *** jamesmcarthur has joined #openstack-infra | 16:31 | |
| *** jamesmcarthur_ has quit IRC | 16:34 | |
| *** eernst has quit IRC | 16:36 | |
| *** takamatsu has joined #openstack-infra | 16:37 | |
| *** eernst has joined #openstack-infra | 16:37 | |
| *** dpawlik|afk has quit IRC | 16:37 | |
| *** lucasagomes has quit IRC | 16:40 | |
| openstackgerrit | sean mooney proposed openstack/project-config master: Remove networking-ovs-dpdk jobs https://review.opendev.org/701046 | 16:46 |
| *** sean-k-mooney has joined #openstack-infra | 16:47 | |
| sean-k-mooney | o/ anyone around today or are people back next week? | 16:48 |
| *** jamesmcarthur has quit IRC | 16:49 | |
| AJaeger | sean-k-mooney: some are showing up briefly, so don't count on us. What's up? | 16:54 |
| *** ociuhandu has joined #openstack-infra | 16:55 | |
| *** ociuhandu has quit IRC | 16:56 | |
| *** ociuhandu has joined #openstack-infra | 16:58 | |
| *** dtantsur is now known as dtantsur|afk | 16:58 | |
| *** ociuhandu has quit IRC | 17:01 | |
| *** ociuhandu has joined #openstack-infra | 17:01 | |
| *** adam_g has joined #openstack-infra | 17:02 | |
| sean-k-mooney | AJaeger: noting major, im trying to convert networking-ovs-dpdk to only run python3 test but it still has tests defiend in project-config | 17:02 |
| sean-k-mooney | AJaeger: https://review.opendev.org/#/c/701046/ so i added this to remove them | 17:02 |
| *** gyee has joined #openstack-infra | 17:03 | |
| sean-k-mooney | AJaeger: a i see you started reviewing my patches :) | 17:04 |
| *** jamesmcarthur has joined #openstack-infra | 17:04 | |
| *** ykarel has quit IRC | 17:05 | |
| *** ykarel has joined #openstack-infra | 17:06 | |
| *** ykarel is now known as ykarel|away | 17:06 | |
| *** ociuhandu has quit IRC | 17:12 | |
| *** ociuhandu has joined #openstack-infra | 17:14 | |
| *** pkopec has quit IRC | 17:15 | |
| *** jamesmcarthur has quit IRC | 17:15 | |
| *** iurygregory has quit IRC | 17:18 | |
| *** rpittau is now known as rpittau|afk | 17:18 | |
| *** ociuhandu has quit IRC | 17:23 | |
| *** ociuhandu has joined #openstack-infra | 17:24 | |
| *** ociuhandu has quit IRC | 17:25 | |
| *** ociuhandu has joined #openstack-infra | 17:25 | |
| openstackgerrit | sean mooney proposed openstack/openstack-zuul-jobs master: remove networking-ovs-dpdk legacy job https://review.opendev.org/701051 | 17:26 |
| *** ociuhandu has quit IRC | 17:31 | |
| *** ociuhandu has joined #openstack-infra | 17:32 | |
| *** evrardjp has quit IRC | 17:33 | |
| *** evrardjp has joined #openstack-infra | 17:33 | |
| *** ociuhandu has quit IRC | 17:36 | |
| mordred | sean-k-mooney: p-c patch lgtm | 17:42 |
| *** jamesmcarthur has joined #openstack-infra | 17:43 | |
| *** ykarel|away has quit IRC | 17:45 | |
| KeithMnemonic | is anyone aware of issues building any of the *-dashboard projects on stable/train | 17:47 |
| KeithMnemonic | this for example https://review.opendev.org/#/c/698908/ and stable/train fails on a fresh clone | 17:48 |
| clarkb | KeithMnemonic: I sent email about nodejs version and host OS mismatches causing failures but that is all I am awareof | 17:48 |
| clarkb | and that was for rocky I think | 17:48 |
| fungi | pip._vendor.pkg_resources.ContextualVersionConflict: (python-keystoneclient 3.21.0 (/home/zuul/src/opendev.org/openstack/designate-dashboard/.tox/pep8/lib/python3.6/site-packages), Requirement.parse('python-keystoneclient>=3.22.0'), {'horizon'}) | 17:49 |
| KeithMnemonic | yup | 17:49 |
| *** hwoarang has quit IRC | 17:49 | |
| KeithMnemonic | and the cherry-pick i have up should have no bearing ob that. and even a fresh clone and 'tox -e py36' generates that | 17:50 |
| *** hwoarang has joined #openstack-infra | 17:50 | |
| fungi | https://zuul.opendev.org/t/openstack/build/178e5890d70b42c8aeace5a448c2ab3a/log/tox/pep8-1.log#443 | 17:51 |
| fungi | ERROR: horizon 17.1.0 has requirement python-keystoneclient>=3.22.0, but you'll have python-keystoneclient 3.21.0 which is incompatible. | 17:51 |
| fungi | https://opendev.org/openstack/requirements/src/branch/stable/train/upper-constraints.txt#L608 | 17:53 |
| fungi | python-keystoneclient===3.21.0 | 17:53 |
| KeithMnemonic | but what is pulling in horizon 17. it should be 16 https://github.com/openstack/releases/blob/master/deliverables/train/horizon.yaml | 17:54 |
| fungi | https://opendev.org/openstack/horizon/src/branch/stable/train/requirements.txt#L36 | 17:54 |
| fungi | python-keystoneclient>=3.15.0 # Apache-2.0 | 17:54 |
| fungi | however... | 17:54 |
| fungi | https://opendev.org/openstack/horizon/src/branch/master/requirements.txt#L37 | 17:54 |
| fungi | python-keystoneclient>=3.22.0 # Apache-2.0 | 17:54 |
| sean-k-mooney | mordred: cool thanks for looking. | 17:55 |
| fungi | KeithMnemonic: https://zuul.opendev.org/t/openstack/build/178e5890d70b42c8aeace5a448c2ab3a/log/tox/pep8-1.log#194-195 | 17:55 |
| fungi | it's installing master horizon | 17:55 |
| KeithMnemonic | but it should not be, correct? | 17:56 |
| fungi | er, no wait, double-checking that it's using a package | 17:56 |
| *** jamesmcarthur has quit IRC | 17:57 | |
| fungi | https://opendev.org/openstack/horizon/src/tag/17.1.0/requirements.txt#L36 | 17:57 |
| fungi | but anyway, the issue is the same | 17:59 |
| fungi | https://opendev.org/openstack/designate-dashboard/src/branch/stable/train/requirements.txt#L9 | 17:59 |
| fungi | designate-dashboard in stable/train claims it will work with the latest release of horizon | 17:59 |
| fungi | and it's treating it like a library | 18:00 |
| *** derekh has quit IRC | 18:00 | |
| fungi | however it's not generally considered a library project and isn't tracked in the upper-constraints.txt file in openstack/requirements, so its version won't be further constrained | 18:00 |
| fungi | https://pypi.org/project/horizon/#history shows that a new 17.1.0 release of horizon was tagged and published to pypi on 2019-12-16 | 18:02 |
| fungi | i'm cloning now to inspect the tag message so i can backtrack that to a release request | 18:03 |
| fungi | however i suspect this is something the release and requirements teams are going to need to hash out, if dashboard projects are treating horizon like a library and listing it in their requirements files | 18:05 |
| KeithMnemonic | thanks for helping fungi: | 18:05 |
| fungi | KeithMnemonic: yep, confirmed, horizon has been making intermediate releases out of master, after train: https://opendev.org/openstack/releases/src/branch/master/deliverables/ussuri/horizon.yaml | 18:08 |
| fungi | as their release model (cycle-with-intermediary) suggests | 18:08 |
| *** jamesmcarthur has joined #openstack-infra | 18:09 | |
| fungi | so if dashboard projects are going to put horizon in their requirements files, they'll need some way to limit it to the appropriate versions for their respective branches | 18:09 |
| fungi | for example, the designate-dashboard requirements.txt could add a <17 to the allowed horizon versions | 18:10 |
| fungi | that way it will still get 16.x.x point releases but won't try to use 17.x.x from ussuri | 18:11 |
| fungi | but if there's a bunch of them, it may make more sense to get horizon added to upper-constraints.txt in openstack/requirements | 18:11 |
| fungi | i have a feeling this is the first cycle horizon started making intermediary releases out of master, which would explain why it took folks by surprise | 18:13 |
| *** jamesmcarthur has quit IRC | 18:13 | |
| fungi | aha, yep, they used the cycle-with-rc model in train: https://opendev.org/openstack/releases/src/branch/master/deliverables/train/horizon.yaml#L3 | 18:13 |
| *** jamesmcarthur has joined #openstack-infra | 18:14 | |
| sean-k-mooney | fungi speaking of releases you can only list/release project in the releases repo if they are list in the governace repos projects.yaml file right? so i can ues the releases repos to release my unaffiliated x/ repos | 18:14 |
| fungi | KeithMnemonic: so anyway, as i suspected this is a conversation for #openstack-requirements and/or #openstack-release (as well as #openstack-horizon) | 18:14 |
| fungi | sean-k-mooney: correct | 18:14 |
| sean-k-mooney | im not planning on doing a release by the way just checking | 18:14 |
| sean-k-mooney | cool | 18:14 |
| fungi | sean-k-mooney: it's the release team for official openstack deliverables | 18:15 |
| sean-k-mooney | ya that makes sense | 18:15 |
| mordred | fungi: speaking of sean-k-mooney - if you have a sec, https://review.opendev.org/#/c/701046 could use a +A | 18:15 |
| fungi | sean-k-mooney: much of the automation is available to all, and the tools the openstack release team uses to review release requests is also all in git should you want to reuse any of it yourself | 18:15 |
| fungi | it's mainly the expertise of the release team reviewers you're not going to have available to a project which isn't part of openstack | 18:16 |
| sean-k-mooney | fungi: honestly the repos that fall into that catagory are basically devstack plugins so i i just branch/tag them as needed | 18:16 |
| fungi | yep, we try to make good generic release automation available anyway, the point of the openstack release team and their review tools is more to coordinate release activity across related projects and push back on poor choices | 18:18 |
| sean-k-mooney | fungi: yep i was wondering because my pypi account was tied to my old intel email and i cant reset the passward so i need to get in contact with the pypi team to recover it | 18:18 |
| *** jamesmcarthur has quit IRC | 18:20 | |
| sean-k-mooney | the alternitive to that woudl be to release via the release repo since the openstack account has permissions on networkig-ovs-dpdk but it does not have owner rights only i had that | 18:20 |
| *** jamesmcarthur has joined #openstack-infra | 18:20 | |
| fungi | well, the "openstackci" account name is slightly misleading, we use that for publishing to any pypi projects. these days we might instead have chosen to call it "opendev" | 18:21 |
| sean-k-mooney | ya well it predates opendev by a few years | 18:22 |
| fungi | by roughly 7-8 years, i'll wager | 18:24 |
| sean-k-mooney | am has it been that long? im pretty sure the release repo and automation is not quite that old. i still rememebr pushing gpg signed tags and manually building and signing the pypi packages | 18:26 |
| sean-k-mooney | its been a good 5 or 6 anyway | 18:26 |
| openstackgerrit | Merged openstack/project-config master: Remove networking-ovs-dpdk jobs https://review.opendev.org/701046 | 18:27 |
| sean-k-mooney | :) | 18:27 |
| sean-k-mooney | ty | 18:27 |
| mordred | sean-k-mooney: ^^ shold be able to recheck those others now | 18:27 |
| sean-k-mooney | yep i have notice a few issues that ill fix up first but that should drop the py27 jobs at least | 18:28 |
| fungi | i already rechecked 701051 | 18:29 |
| sean-k-mooney | well that one should be fine | 18:30 |
| fungi | looks like it got enqueued so i've gone ahead and aproved it | 18:30 |
| sean-k-mooney | https://review.opendev.org/#/c/656580/ still has some stuff from when it was written for fedora instead of ubunut | 18:30 |
| * mordred likes removing legacy jobs | 18:31 | |
| *** jamesmcarthur has quit IRC | 18:31 | |
| sean-k-mooney | ill be removing the last legacy job form os-vif next week | 18:31 |
| sean-k-mooney | that is a comunity goal right get rid fo the legacy jobs | 18:31 |
| mordred | \o/ | 18:31 |
| mordred | yup | 18:31 |
| mordred | it's just an especially exciting one | 18:31 |
| * mordred hands sean-k-mooney a cupcake | 18:32 | |
| sean-k-mooney | :) its actully almost simpler to do it now as part of also moving to python3 only | 18:33 |
| mordred | yah | 18:33 |
| sean-k-mooney | actully have we offically passed the p27 eol or is that later in january | 18:33 |
| *** jamesmcarthur has joined #openstack-infra | 18:33 | |
| mordred | it has passed - we are officially EOL now | 18:34 |
| tosky | sean-k-mooney: https://pythonclock.org/ | 18:34 |
| sean-k-mooney | ya it was jan 1st | 18:34 |
| sean-k-mooney | damb i should have been celebrating that at new years too | 18:34 |
| mordred | ikr? | 18:34 |
| sean-k-mooney | ok time for dinner. enjoy the weekend o/ | 18:38 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: remove networking-ovs-dpdk legacy job https://review.opendev.org/701051 | 18:42 |
| *** Lucas_Gray has joined #openstack-infra | 18:48 | |
| *** jamesmcarthur has quit IRC | 18:48 | |
| *** jamesmcarthur has joined #openstack-infra | 18:48 | |
| KeithMnemonic | fungi: do i need to post in release/ requirements or were you going to? | 18:50 |
| *** jamesmcarthur has quit IRC | 18:52 | |
| KeithMnemonic | fungi: lol it is there already https://review.opendev.org/#/c/700379/2 | 18:56 |
| KeithMnemonic | so i guess i need one for train, stein as well? | 18:57 |
| fungi | KeithMnemonic: https://review.opendev.org/700376 | 18:59 |
| fungi | looks like consensus is that doing it directly in stable branches is preferred, and that it shouldn't be added in master | 19:00 |
| fungi | similar to the proposed changes referenced for neutron | 19:00 |
| fungi | KeithMnemonic: so maybe try to rally some momentum for amotoki's changes there, now that holidays are behind us | 19:01 |
| KeithMnemonic | yup , doing that now in horizon | 19:01 |
| fungi | well, 700376 is teh stable/train addition for horizon | 19:02 |
| fungi | and 700377 is one he pushed up for stable/stein | 19:03 |
| fungi | https://review.opendev.org/#/q/I421ee2d498ea5100d9b52e356b46f2d594638b5b shows there's also 700378 to do it for stable/rocky, so i think it's covered. just needs stable requirements reviewers to weigh in | 19:04 |
| KeithMnemonic | yup, thanks | 19:08 |
| *** paladox has quit IRC | 19:10 | |
| *** adriant has quit IRC | 19:22 | |
| *** adriant has joined #openstack-infra | 19:25 | |
| *** dayou has quit IRC | 19:26 | |
| *** dayou has joined #openstack-infra | 19:27 | |
| *** paladox has joined #openstack-infra | 19:34 | |
| *** ahosam has joined #openstack-infra | 19:34 | |
| *** paladox has quit IRC | 19:35 | |
| *** paladox has joined #openstack-infra | 19:35 | |
| *** jamesmcarthur has joined #openstack-infra | 19:46 | |
| *** dave-mccowan has quit IRC | 19:52 | |
| *** dciabrin__ has joined #openstack-infra | 19:57 | |
| *** dciabrin_ has quit IRC | 20:00 | |
| *** Lucas_Gray has quit IRC | 20:00 | |
| *** dave-mccowan has joined #openstack-infra | 20:12 | |
| *** jamesmcarthur has quit IRC | 20:31 | |
| *** tosky has quit IRC | 20:48 | |
| *** takamatsu has quit IRC | 20:51 | |
| *** rfolco has quit IRC | 21:17 | |
| *** rlandy has quit IRC | 21:20 | |
| *** sshnaidm has joined #openstack-infra | 21:28 | |
| *** ociuhandu has joined #openstack-infra | 22:02 | |
| *** ociuhandu has quit IRC | 22:06 | |
| TheJulia | Hey infra folks, there is a bifrost post-release job that likely needs to be killed. Out there for nearly 22 hour :\ | 22:24 |
| fungi | oh, fun... looking | 22:26 |
| fungi | neat, it hasn't actually scheduled any jobs in release-post for that id | 22:30 |
| fungi | granted, that pipeline is pretty much only used by the openstack/releases repo i think? | 22:30 |
| fungi | so this ref must be stuck in the ephemeral state where zuul is deciding whether to run jobs for it | 22:31 |
| fungi | it doesn't actually hurt anything being there, but i'd like to go poking in the logs and see if i can find a bug in zuul or something | 22:32 |
| TheJulia | fungi: okay, I just noticed it and was like "oh... well... great!" | 22:36 |
| * TheJulia goes back to waiting for a test job to work locally to fix bifrost's ci | 22:36 | |
| fungi | yeah, an internal implementation detail of zuul is that all refs appear in all pipelines briefly while it determines whether they need builds created there | 22:41 |
| fungi | usually it's so quick you won't notice | 22:41 |
| fungi | but something has caused this one to get "stuck" there | 22:41 |
| * TheJulia wonders if it is a zombie ref | 22:45 | |
| fungi | not in the sense i would consider the term, it's the git sha for the current master branch tip | 22:54 |
| *** tesseract has quit IRC | 22:54 | |
| *** tosky has joined #openstack-infra | 22:54 | |
| fungi | 2020-01-03 00:39:18,213 DEBUG zuul.Pipeline.openstack.release-post: [e: 2d7976896aba4656aef3f31e326a22a3] Event <GerritTriggerEvent ref-updated opendev.org/openstack/bifrost> for change <Branch 0x7ff40a76b668 openstack/bifrost refs/heads/master updated 15e7b0f89fd0c71e4547b8bbfaa7c82fe650f247..3f74295a0d23203ec871ba5051d6ff52a4f85731> matched <GerritEventFilter types: ref-updated refs: | 22:57 |
| fungi | ^refs/heads/.*$ ignore_deletes: True> in pipeline <IndependentPipelineManager release-post> | 22:57 |
| fungi | that's where it started considering it | 22:58 |
| *** pcaruana has quit IRC | 22:59 | |
| mordred | fungi: "fun" | 23:03 |
| fungi | looks like it may have never completed the merge request associated with it | 23:05 |
| fungi | at least that's the last non-stasis log entry i find for it in that pipeline | 23:07 |
| fungi | 2020-01-03 00:39:18,255 DEBUG zuul.Pipeline.openstack.release-post: [e: 2d7976896aba4656aef3f31e326a22a3] Scheduling merge for item <QueueItem 0x7ff4c99d66d8 for <Branch 0x7ff40a76b668 openstack/bifrost refs/heads/master updated 15e7b0f89fd0c71e4547b8bbfaa7c82fe650f247..3f74295a0d23203ec871ba5051d6ff52a4f85731> in release-post> (files: ['zuul.yaml', '.zuul.yaml'], dirs: ['zuul.d', '.zuul.d']) | 23:08 |
| fungi | i never see a corresponding "...Freezing job graph for <QueueItem 0x7ff2a3b2beb8 ..." which would normally appear just a couple seconds later | 23:10 |
| fungi | i'll check the mergers | 23:12 |
| fungi | whee, right off the bat... zm02 is unresponsive to ssh | 23:13 |
| fungi | 22/tcp is listening and gives an openssh banner | 23:14 |
| *** factor has quit IRC | 23:14 | |
| *** factor has joined #openstack-infra | 23:15 | |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1547&rra_id=all | 23:15 |
| fungi | place your bets! | 23:16 |
| fungi | last snmp response was at 00:55z | 23:16 |
| fungi | oof, yeah, cpu was spiking up to 100% for the last few minutes there, when it's normally almost zero | 23:18 |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1541&rra_id=all | 23:18 |
| fungi | cpu utilization seems to have started between 00:30 and 00:35z | 23:19 |
| fungi | and reached 100% around the time that merge request was made, so i'll wager it was the one which accepted the request, only moments before it went out to lunch (for seemingly unrelated reasons) | 23:20 |
| mordred | fungi: that definitely seems like a good candidate culprit | 23:20 |
| mordred | now that we have a good theory on that merge request - why did zm02 go our to lunch I wonder? | 23:21 |
| fungi | yes, that's where our harrowed tail takes a turn for the sinister | 23:21 |
| mordred | fungi: also - if zm02 goes away goes away, does the scheduler retry the gearman merge job? | 23:21 |
| fungi | or so i'm guessing, i haven't read that far yet ;) | 23:21 |
| fungi | well, seeing if i can squeeze blood from this zm02 stone before i give up and issue a nova reboot | 23:22 |
| mordred | (like, I think gearman tracks tcp connections and has some way of knowing a worker actually went awol?) | 23:22 |
| mordred | fungi: ++ | 23:22 |
| fungi | i have a hunch we're not going to find out the answer to that particular mystery. ssh repeatedly ends (after a lengthy hang) with "client_loop: send disconnect: Broken pipe" but i'll see if the oob console has anything useful | 23:23 |
| fungi | if we're lucky, there might be something in the syslog after a reboot | 23:24 |
| fungi | mordred: oh, while you're around, there was a rackspace trouble ticket a few hours ago letting us know they rebooted the host where the "mttest" nova instance resides. i'm going to assume that's yours, and you don't need any followup on the ticket | 23:25 |
| mordred | fungi: yes to both. in fact, I could almost certainly delete that. I do not remember why I had booted it | 23:26 |
| mordred | I blame elves | 23:26 |
| fungi | console has a bunch of kmesg lines like "task kworker/... blocked for more than 120 seconds" | 23:28 |
| fungi | nothing useful other than to confirm that the instance is well hosed | 23:28 |
| fungi | #status log issued server reboot via api for zm02, hung and unresponsive since ~00:40z on 2019-01-03 | 23:30 |
| openstackstatus | fungi: finished logging | 23:30 |
| fungi | i closed the rackspace ticket about mttest too | 23:31 |
| fungi | according to the nova api, that instance has been in "reboot" state for nearly 5 minutes already. i wonder if there's a problem with the underlying host | 23:35 |
| fungi | i followed up with a reboot --hard and it seems to have gotten the idea | 23:35 |
| fungi | login prompt showing at the console again now | 23:36 |
| fungi | i can ssh into it now as well | 23:36 |
| fungi | last syslog entry was Jan 3 00:36:41 | 23:37 |
| fungi | as suspected, no smoking gun in the syslog either | 23:38 |
| fungi | a puppet apply completed at 00:17:09 and then there was another root session for ansible at 00:33:25 | 23:39 |
| fungi | looks like ansible was updating ssh keys and the usual | 23:40 |
| fungi | nothing out of the ordinary, and certainly nothing i would expect to suddenly eat all the cpu | 23:41 |
| clarkb | live migrations seem to have created similar odd behaviors in the past | 23:43 |
| mordred | yah | 23:44 |
| fungi | i concur. i'll consider this rca as thorough as is warranted given the usual unknowns of the cloud | 23:46 |
| fungi | it may be worth observing that it seems like if a merger accepts a merge request and then hangs and disappears, and is later rejoined to the system after a reboot, you can end up with a ref stuck in a pipeline indefintiely | 23:47 |
| fungi | er, indefinitely | 23:47 |
| fungi | TheJulia: thanks for your observation! it led us to discover a hung zuul-merger server | 23:47 |
| TheJulia | \o; | 23:51 |
| TheJulia | err \o/ | 23:51 |
| clarkb | fungi: I think we can make gearman timeout and fail I seem to recall tobiash adding support for that | 23:51 |
| fungi | i'm guessing \o; is a villager with a torch and a pitchfork | 23:52 |
| fungi | (or a keyboard with the / and ; keys adjacent) | 23:53 |
| TheJulia | I like the villager one, but I do typo a bit much | 23:54 |
| fungi | i treat emoticon typos like a rorschach-style inkblot | 23:56 |
| *** gyee has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!