| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Just test rally baby https://review.opendev.org/c/openstack/kolla-ansible/+/913728 | 00:44 |
|---|---|---|
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Swift test https://review.opendev.org/c/openstack/kolla-ansible/+/913942 | 01:35 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Just test rally baby https://review.opendev.org/c/openstack/kolla-ansible/+/913728 | 02:05 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Swift test https://review.opendev.org/c/openstack/kolla-ansible/+/913942 | 02:05 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Swift test https://review.opendev.org/c/openstack/kolla-ansible/+/913942 | 03:09 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Swift test https://review.opendev.org/c/openstack/kolla-ansible/+/913942 | 03:21 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Swift test https://review.opendev.org/c/openstack/kolla-ansible/+/913942 | 04:02 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Just test rally baby https://review.opendev.org/c/openstack/kolla-ansible/+/913728 | 04:30 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Swift test https://review.opendev.org/c/openstack/kolla-ansible/+/913942 | 04:30 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Swift test https://review.opendev.org/c/openstack/kolla-ansible/+/913942 | 05:20 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/2023.2 failed: CI: Increase galera node timeouts https://review.opendev.org/c/openstack/kolla-ansible/+/913810 | 07:40 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Just test rally baby https://review.opendev.org/c/openstack/kolla-ansible/+/913728 | 09:26 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Swift test https://review.opendev.org/c/openstack/kolla-ansible/+/913942 | 09:26 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Refactor of kolla_container_facts https://review.opendev.org/c/openstack/kolla-ansible/+/911417 | 09:32 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Move actions to kolla_container_facts https://review.opendev.org/c/openstack/kolla-ansible/+/911505 | 09:33 |
| SvenKieske | kevko: what is wrong with parted that we need to loop there?oO https://review.opendev.org/c/openstack/kolla-ansible/+/913942/8/tests/setup_disks.sh | 09:33 |
| kevko | SvenKieske: i don't know ..i was trying to wait ...because there was a fail that it's used ... | 09:46 |
| kevko | SvenKieske: /dev/* is used ..try to restart ..blablabla | 09:46 |
| kevko | SvenKieske: but this is very veird ... /tmp/zuul-logs.QmYhf3/primary/logs/kolla/all-WARNING.txt:2024-03-22 05:55:41.558 7 WARNING nova.compute.manager [None req-a9925dbc-4810-4354-b54c-269e484e048b - - - - - -] [instance: fda729ca-e859-45e0-b02d-72c657b0469e] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current | 09:46 |
| kevko | task_state: None, original DB power_state: 1, current VM power_state: 4 | 09:46 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Merge of container_facts modules https://review.opendev.org/c/openstack/kolla-ansible/+/912460 | 09:47 |
| kevko | SvenKieske: there is something weird in upgrade process | 09:49 |
| kevko | SvenKieske: https://paste.openstack.org/show/b7yV9dJXuLYl6wUaJrAt/ | 09:49 |
| SvenKieske | I'm also currently trying to figure out if our CI is green or if there is something newly broken again :D | 09:50 |
| SvenKieske | your ubuntu-ovn job also fails, interesting | 09:50 |
| kevko | SvenKieske: there is tempest directory ...tempests are passing ... | 09:51 |
| SvenKieske | + openstack server create --wait --image cirros --flavor m1.tiny --key-name mykey --network demo-net lb_member --wait | 09:51 |
| kevko | (most of the time) | 09:51 |
| SvenKieske | Request to http://192.0.2.10:8774/v2.1/servers/7951da63-e0f6-4198-80a4-85e8b7caa2e9 timed out | 09:51 |
| SvenKieske | https://zuul.opendev.org/t/openstack/build/affe567db343421c8bcf6d32fa649499/log/primary/logs/ansible/test-ovn | 09:52 |
| kevko | now im looking into https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f92/913728/24/check/kolla-ansible-debian-upgrade/f9279f1/primary/logs/ansible/test-core-openstack-upgrade <<< | 09:54 |
| kevko | kolla-upgrade-test instance is shutted down after upgrade .... i reaaaaallly don't know why | 09:55 |
| kevko | even if I've already slept ...i can't get it :D | 09:55 |
| SvenKieske | looking at the IRC history you couldn't have slept that much? :D | 09:59 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.2: CI: Increase galera node timeouts https://review.opendev.org/c/openstack/kolla-ansible/+/913810 | 10:02 |
| kevko | SvenKieske: Nope, I'm not used to going to sleep until I solve something, I fell asleep involuntarily :D | 10:03 |
| kevko | SvenKieske: can u give some advice ? hint ? | 10:03 |
| SvenKieske | not really, are there any CI related automatisms that shut down VMs? was there some maintenance ongoing (I don't think so, that would be announced). Maybe the host itself had a problem. I never have seen this: | 10:06 |
| SvenKieske | Instance shutdown by itself. Calling the stop API | 10:06 |
| kevko | aaaa | 10:06 |
| kevko | this is another job ...debian classic | 10:06 |
| kevko | https://7becaa41bf65a7eaafce-2bb61388e3d961ebec04f637f3a6e452.ssl.cf5.rackcdn.com/913728/24/check/kolla-ansible-debian/a0739a6/primary/logs/tempest/reports/tempest-smoke.1.html | 10:06 |
| kevko | tempest.lib.exceptions.IdentityError: Got identity error | 10:07 |
| kevko | Details: Unexpected status code 503 | 10:07 |
| SvenKieske | why print a nice error message containing the error code when you also can just produce a stacktrace? :D | 10:07 |
| SvenKieske | 503 so there's clearly something wrong with the webserver or the backend app | 10:08 |
| kevko | SvenKieske: something with neutron maybe ? .. ican see some errors .. | 10:09 |
| kevko | SvenKieske: there are also bunch of keystone not found errors ....trust ...domain ... role ...user .... | 10:09 |
| kevko | SvenKieske: wondering ..if some of them can't cause problems | 10:10 |
| SvenKieske | there is afaik still my patch missing..wait | 10:12 |
| SvenKieske | mhm but that was only on 2023.2 branch: https://review.opendev.org/c/openstack/kolla-ansible/+/912452 | 10:13 |
| SvenKieske | Fl1nt had some other keystone related patches | 10:13 |
| SvenKieske | Fl1nt: sorry btw, I still had not the time to look into your patches! | 10:14 |
| frickler | for stopped instances, check whether qemu got oom killed? | 10:14 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Just test rally baby https://review.opendev.org/c/openstack/kolla-ansible/+/913728 | 10:14 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Swift test https://review.opendev.org/c/openstack/kolla-ansible/+/913942 | 10:14 |
| SvenKieske | frickler: I was about to suggest the same, but afaik that doesn't log "instance shutdown itself"? or does it? | 10:15 |
| frickler | I think it could | 10:15 |
| SvenKieske | I have seen many oom kills, admittedly I never logged at those logs then, because from an ops persepctive one of the first things I check is dmesg/kernel and then you don't need to look any further. | 10:16 |
| SvenKieske | looked* | 10:16 |
| frickler | because from nova perspective this cannot be distinguished from doing poweroff from within the instance | 10:16 |
| kevko | frickler: please, If you can check -> https://zuul.opendev.org/t/openstack/build/f9279f19ca5249a7918946131827ba1c <- | 10:17 |
| SvenKieske | well, strictly speaking it could. because qemu should get ENOMEM and could bubble up that to nova, no? :) | 10:17 |
| SvenKieske | well, in overcommit scenarios (so the usual scenario) I guess ENOMEM might not be reliable | 10:18 |
| frickler | bingo: Mar 22 05:55:26 primary kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/machine/qemu-12-instance-0000000a.libvirt-qemu,task=qemu-system-x86,pid=113588,uid=42436 | 10:18 |
| frickler | that's in syslog.txt from kevko's upgrade job above | 10:18 |
| frickler | but it's not qemu that triggered it, so really difficult to handle: Mar 22 05:55:26 primary kernel: python invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 | 10:20 |
| SvenKieske | mhm | 10:20 |
| frickler | anyway, the solution is run less stuff in test jobs or only in multinode jobs | 10:20 |
| frickler | oh, wait, is that for the tempest test? does it run with --serial? else I would not be surprised by this happening | 10:21 |
| SvenKieske | did anybody talk btw about the recent python CVEs? I believe there was a memory corruption/leak also in there, maybe someone already triggerd it? :D | 10:21 |
| frickler | I haven't seen those yet, do you have a link? | 10:22 |
| SvenKieske | ok, tempest tests running in parallel might be the culprit. | 10:22 |
| SvenKieske | they are on oss-sec, since 2 days I think, wait.. | 10:22 |
| SvenKieske | https://seclists.org/oss-sec/2024/q1/240 | 10:22 |
| SvenKieske | 2 days ago | 10:22 |
| SvenKieske | with regular updates everybody should be secure. everybody does regular updates, right? lol (sorry I'm cynical when it comes to infosec) | 10:23 |
| SvenKieske | I also did no analysis how grave these are, I usually read them when lying in bed :D | 10:24 |
| SvenKieske | bundled libexpat, zipfile and a crash in socket-if_indextoname and the ssl stuff look most promising for exploits I guess. the windows only stuff should be boring | 10:26 |
| SvenKieske | dunno about the codecs, their names are cryptic and I don't know them by name. | 10:28 |
| frickler | at least ubuntu doesn't have new pkgs yet, also no reference to those cves, hmm | 10:29 |
| SvenKieske | is python community maintained on ubuntu? I wouldn't be surprised. | 10:32 |
| SvenKieske | no, it i maintained by "Ubuntu Core Developers" https://packages.ubuntu.com/jammy/python3 | 10:34 |
| SvenKieske | I just asked over in #ubuntu-security on libera chat if anybody cares :) they have really not even a public launchpad bug | 10:37 |
| SvenKieske | I bet they have some private stuff about it somewhere. | 10:37 |
| SvenKieske | this is another datapoint to not use distro packages @kevko ;) seems they are not up to speed when it comes to security updates. | 10:40 |
| SvenKieske | and I write that as someone who actually likes the distro dev model itself. :( | 10:40 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: common: Stop using transitional labels https://review.opendev.org/c/openstack/kolla-ansible/+/913960 | 10:46 |
| SvenKieske | mhm, found a bug, nice :D | 10:52 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe master: Fix the glob for the custom RabbitMQ configuration https://review.opendev.org/c/openstack/kayobe/+/909113 | 10:53 |
| SvenKieske | it's a bug in a shellscript ;) but no CI would've catched that I guess :D | 10:53 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible master: CI: fix check-failure.sh sudo missing https://review.opendev.org/c/openstack/kolla-ansible/+/913961 | 10:54 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible master: Add ovn-exporter https://review.opendev.org/c/openstack/kolla-ansible/+/855498 | 10:58 |
| kevko | frickler: no it's not tempest ..it's regular bash instance ...tempests ran all OK | 10:59 |
| SvenKieske | well scratch that, CI would've found this if we would have tests for our tests :D but no linter would've catched it, was what I meant. | 10:59 |
| mnasiadka | master complainers those Kolla developers | 11:13 |
| mnasiadka | :) | 11:13 |
| opendevreview | Matt Crees proposed openstack/kayobe master: CI: rework RMQ steps for queue migrations https://review.opendev.org/c/openstack/kayobe/+/913878 | 11:15 |
| opendevreview | Matt Crees proposed openstack/kayobe master: Bump KA Ansible versions to match new defaults https://review.opendev.org/c/openstack/kayobe/+/913571 | 11:15 |
| opendevreview | Matt Crees proposed openstack/kayobe master: Bump up Ansible supported versions to 8.x/9.x https://review.opendev.org/c/openstack/kayobe/+/910513 | 11:15 |
| opendevreview | Matt Crees proposed openstack/kayobe master: Use new collections in Kayobe https://review.opendev.org/c/openstack/kayobe/+/910742 | 11:15 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Swift test https://review.opendev.org/c/openstack/kolla-ansible/+/913942 | 11:24 |
| kevko | btw, why we are stopping this in upgrade jobs ? | 11:39 |
| kevko | kolla-ansible -i /etc/kolla/inventory -vvv stop --tags heat,keystone,neutron,nova --yes-i-really-really-mean-it | 11:39 |
| kevko | If i will stop neutron and nova on production upgrade ..i am fired :D | 11:39 |
| SvenKieske | follow the git? :D | 11:41 |
| SvenKieske | I always look at git blame and then find an unrelated commit ;) | 11:41 |
| kevko | SvenKieske: I do that too, but today I need more to calm down and distract myself rather than to get even more stressed out :D | 11:43 |
| SvenKieske | xD I can relate to that | 11:43 |
| kevko | :D | 11:44 |
| kevko | https://github.com/openstack/kolla-ansible/blob/80253410234cc74aab25af0f9bb90f1cbb764295/tests/upgrade.sh#L17-L19 it would probably be good to add a condition that checks if they are turned on or not... and based on that, at least write an echo... so I don't have to look for it and it's in the test log :( | 11:46 |
| SvenKieske | sounds like a good idea | 11:47 |
| kevko | SvenKieske I know you're not a fan of bash, neither am I, however... if you've seen any of my bash scripts in kolla or kolla-ansible... then you know that I do it... and it's fine to know what script is doing ...especially if it fail ... :) | 11:49 |
| kevko | did we have btw swift scenartio broken for a longer time right ? | 11:50 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: common: Stop using transitional labels https://review.opendev.org/c/openstack/kolla-ansible/+/913960 | 11:54 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/2023.2 failed: fix creation of keystone service role on upgrade https://review.opendev.org/c/openstack/kolla-ansible/+/912452 | 11:55 |
| kevko | anyone for approve ? https://review.opendev.org/c/openstack/kolla-ansible/+/913653 | 12:15 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: common: Stop using transitional labels https://review.opendev.org/c/openstack/kolla-ansible/+/913960 | 12:57 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Swift test https://review.opendev.org/c/openstack/kolla-ansible/+/913942 | 12:59 |
| opendevreview | Merged openstack/kolla-ansible master: Add conditionals for IPv6 sysctl settings https://review.opendev.org/c/openstack/kolla-ansible/+/905831 | 13:26 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/2023.2 failed: fix creation of keystone service role on upgrade https://review.opendev.org/c/openstack/kolla-ansible/+/912452 | 13:31 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Bump cirros images to 0.5.3 https://review.opendev.org/c/openstack/kayobe/+/913976 | 13:59 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: CI: Bump cirros images to 0.5.3 https://review.opendev.org/c/openstack/kayobe/+/913976 | 14:00 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Swift test https://review.opendev.org/c/openstack/kolla-ansible/+/913942 | 14:02 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Swift test https://review.opendev.org/c/openstack/kolla-ansible/+/913942 | 14:23 |
| kevko | SvenKieske: do we need this also in master ? | 14:23 |
| kevko | https://review.opendev.org/c/openstack/kolla-ansible/+/912452 | 14:23 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Just test rally baby https://review.opendev.org/c/openstack/kolla-ansible/+/913728 | 14:29 |
| SvenKieske | kevko: see the linked bug report: https://bugs.launchpad.net/kolla-ansible/+bug/2056761 "when upgrading from older releases the service role introduced by [1] in kolla-ansible might not be there." | 14:33 |
| SvenKieske | so that should only be an issue for old releases, so it depends how many releases you jump into the future during upgrade | 14:34 |
| SvenKieske | I guess the short answer is: no :) | 14:35 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/2023.2: Add conditionals for IPv6 sysctl settings https://review.opendev.org/c/openstack/kolla-ansible/+/913818 | 14:38 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/2023.1: Add conditionals for IPv6 sysctl settings https://review.opendev.org/c/openstack/kolla-ansible/+/913819 | 14:39 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/zed: Add conditionals for IPv6 sysctl settings https://review.opendev.org/c/openstack/kolla-ansible/+/913820 | 14:40 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Swift test https://review.opendev.org/c/openstack/kolla-ansible/+/913942 | 14:41 |
| opendevreview | Merged openstack/kolla-ansible master: CI: fix check-failure.sh sudo missing https://review.opendev.org/c/openstack/kolla-ansible/+/913961 | 15:12 |
| opendevreview | Olivier Delhomme proposed openstack/kolla-ansible master: kolla-ansible now uses default inventory file https://review.opendev.org/c/openstack/kolla-ansible/+/913993 | 15:31 |
| kevko | tempest looks better and better :) | 15:58 |
| kevko | okay, why I am failing here ? https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6ac/913728/27/check/kolla-ansible-rocky9-multinode-ipv6/6acfa58/primary/logs/tempest/reports/index.html | 15:59 |
| kevko | >> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6ac/913728/27/check/kolla-ansible-rocky9-multinode-ipv6/6acfa58/ | 15:59 |
| frickler | Bad floatingip request: Network 9fc7cc0e-c40c-44e5-b9a3-72f028b3a28c does not contain any IPv4 subnet. hmm | 16:00 |
| kevko | frickler: interisting no ? :) | 16:07 |
| kevko | frickler: let me rework little bit the script as i have one idea ... | 16:07 |
| kevko | frickler: btw, did we have swift scenario working ? :D | 16:07 |
| frickler | kevko: I have not idea, I have no interest in swift, ceph it is for me | 16:08 |
| kevko | frickler: for me also ... i don't have interest also ..but if i am writing testing ..i really need to fix swift :D ..to test if it is really working :D | 16:08 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Just test rally baby https://review.opendev.org/c/openstack/kolla-ansible/+/913728 | 16:20 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Swift test https://review.opendev.org/c/openstack/kolla-ansible/+/913942 | 16:22 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Just test rally baby https://review.opendev.org/c/openstack/kolla-ansible/+/913728 | 16:23 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.2: fix creation of keystone service role on upgrade https://review.opendev.org/c/openstack/kolla-ansible/+/912452 | 16:28 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Just test rally baby https://review.opendev.org/c/openstack/kolla-ansible/+/913728 | 16:33 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Swift test https://review.opendev.org/c/openstack/kolla-ansible/+/913942 | 16:44 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Just test rally baby https://review.opendev.org/c/openstack/kolla-ansible/+/913728 | 16:44 |
| opendevreview | Olivier Delhomme proposed openstack/kolla-ansible master: kolla-ansible now uses default inventory file https://review.opendev.org/c/openstack/kolla-ansible/+/913993 | 16:51 |
| kevko | Hmm, pulls are not working :( mirror.bhs1.ovh.opendev.org:4447/openstack.kolla/fluentd:master-debian-bookworm <<< | 17:14 |
| kevko | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_657/913728/31/check/kolla-ansible-debian/657d19d/primary/logs/ansible/pull | 17:15 |
| frickler | that some issue with quay.io, then. wait and retry I guess | 17:20 |
| kevko | it looks like I win a friday to do something else ;) :D | 17:23 |
| kevko | status.redhat.com | 17:24 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!