| *** hjensas is now known as hjensas|afk | 00:04 | |
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: Remove rally deployment https://review.opendev.org/c/openstack/kolla-ansible/+/794359 | 01:19 |
|---|---|---|
| opendevreview | jinyuanliu proposed openstack/kolla master: venus: add log management system https://review.opendev.org/c/openstack/kolla/+/793795 | 02:16 |
| opendevreview | jinyuanliu proposed openstack/kolla-ansible master: ADD venus for kolla-ansible https://review.opendev.org/c/openstack/kolla-ansible/+/793897 | 03:07 |
| opendevreview | jinyuanliu proposed openstack/kolla-ansible master: ADD venus for kolla-ansible https://review.opendev.org/c/openstack/kolla-ansible/+/793897 | 05:50 |
| opendevreview | Piotr Parczewski proposed openstack/kolla-ansible master: Reduce container metrics cardinality https://review.opendev.org/c/openstack/kolla-ansible/+/783887 | 06:11 |
| mnasiadka | morning | 06:35 |
| yoctozepto | morning | 06:35 |
| yoctozepto | what a sync | 06:36 |
| mnasiadka | yeah | 06:40 |
| *** rpittau|afk is now known as rpittau | 07:15 | |
| opendevreview | jinyuanliu proposed openstack/kolla-ansible master: ADD venus for kolla-ansible https://review.opendev.org/c/openstack/kolla-ansible/+/793897 | 07:34 |
| opendevreview | jinyuanliu proposed openstack/kolla-ansible master: ADD venus for kolla-ansible https://review.opendev.org/c/openstack/kolla-ansible/+/793897 | 07:34 |
| hrw | morning | 08:46 |
| mgoddard | morning | 09:03 |
| jinyuanliu | morning | 09:06 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe master: TLS certificates management sync with Kolla-Ansible https://review.opendev.org/c/openstack/kayobe/+/793697 | 09:19 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Update previous_release to Wallaby https://review.opendev.org/c/openstack/kolla-ansible/+/796422 | 09:27 |
| *** jpic is now known as Guest2380 | 10:33 | |
| *** jpic_ is now known as jpic | 10:33 | |
| ozzzo | following the instructions here to update a document: https://docs.opendev.org/opendev/infra-manual/latest/gettingstarted.html#getting-started | 13:52 |
| ozzzo | Does my commit message need a special format? I'm getting this when I try to commit: | 13:52 |
| ozzzo | git: 'interpret-trailers' is not a git command. See 'git --help'. | 13:53 |
| ozzzo | cannot insert change-id line in .git/COMMIT_EDITMSG | 13:53 |
| ozzzo | or do I need to install a newer version of git? | 13:54 |
| ozzzo | oic my git VM is running centos 7; trying centos 8 | 14:03 |
| opendevreview | Albert Braden proposed openstack/kolla-ansible master: Added upgrade note for separate nova and cinder keys. https://review.opendev.org/c/openstack/kolla-ansible/+/796681 | 14:12 |
| yoctozepto | mgoddard: might be good to advertise kolla http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023104.html | 14:27 |
| mgoddard | yoctozepto: user survey should include kolla | 14:30 |
| yoctozepto | yup, but the more mentions, the better | 14:31 |
| priteau | especially with OSA already mentioning themselves ;) | 14:44 |
| opendevreview | Verification of a change to openstack/kolla failed: [CI] Support building source images with in-review changes https://review.opendev.org/c/openstack/kolla/+/793568 | 14:52 |
| opendevreview | Verification of a change to openstack/kolla-ansible failed: [CI] Support building source images with in-review changes https://review.opendev.org/c/openstack/kolla-ansible/+/793564 | 14:53 |
| mgoddard | mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt | 14:55 |
| mgoddard | ^ meeting in 5 | 14:55 |
| yoctozepto | "oslo_db.exception.DBError: (pymysql.err.InternalError) (1054, \"Unknown column '`neutron`.`networks_1`.`admin_state_up`' in 'CHECK'\")", | 14:56 |
| yoctozepto | there is some race in neutron upgrade scripts or something | 14:56 |
| yoctozepto | as this cripples various upgrade jobs | 14:56 |
| mgoddard | yoctozepto: definitely only running on one host? | 14:59 |
| mgoddard | #startmeeting kolla | 15:00 |
| opendevmeet | Meeting started Wed Jun 16 15:00:15 2021 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 15:00 |
| mgoddard | #topic rollcall | 15:00 |
| mgoddard | 8< 8< 8< | 15:01 |
| parallax | o/ | 15:03 |
| priteau | \o | 15:03 |
| yoctozepto | \o/ | 15:03 |
| hrw | ]o[ | 15:03 |
| mgoddard | #topic agenda | 15:04 |
| yoctozepto | (mgoddard: yup, but it could be threaded) | 15:04 |
| mgoddard | * Roll-call | 15:04 |
| mgoddard | * Agenda | 15:04 |
| mgoddard | * Announcements | 15:04 |
| mgoddard | * Review action items from the last meeting | 15:04 |
| mgoddard | * CI status | 15:04 |
| mgoddard | * Wallaby release planning | 15:04 |
| mgoddard | ** libvirt/OVMF bug | 15:04 |
| mgoddard | * Xena cycle planning | 15:04 |
| mgoddard | ** master branch life cycle https://review.opendev.org/c/openstack/kolla/+/795320 | 15:04 |
| mgoddard | ** Testing log coverage (and more?) https://review.opendev.org/c/openstack/kolla-ansible/+/795512 | 15:04 |
| mgoddard | * Open discussion | 15:04 |
| mgoddard | #topic announcements | 15:05 |
| mgoddard | None from me | 15:05 |
| yoctozepto | none | 15:06 |
| mgoddard | #topic | 15:06 |
| hrw | nope | 15:06 |
| mgoddard | #topic Review action items from the last meeting | 15:06 |
| mgoddard | mgoddard email openstack-discuss about quay.io credentials | 15:06 |
| mgoddard | mnasiadka perform R-17 release tasks | 15:06 |
| mgoddard | mgoddard: nope | 15:06 |
| mgoddard | mnasiadka: started | 15:06 |
| mgoddard | I'm dropping that one. It's on a personal TODO list, just low priority | 15:07 |
| yoctozepto | ok | 15:07 |
| mgoddard | #topic CI status | 15:07 |
| yoctozepto | centos8 broke masakari-monitors in train | 15:07 |
| yoctozepto | otherwise pretty green | 15:07 |
| mgoddard | kolla: [train] libvirt-python dep broke masakari-monitors build on c8 | 15:07 |
| yoctozepto | with the random failure on neutron upgrades | 15:07 |
| yoctozepto | re masakari-monitors and c8 | 15:08 |
| yoctozepto | we have a bug report | 15:08 |
| yoctozepto | that it fails up to victoria | 15:08 |
| yoctozepto | fails to build* | 15:08 |
| yoctozepto | but CI confirms only for train | 15:08 |
| yoctozepto | I did not have time to dive in | 15:08 |
| yoctozepto | but priteau did | 15:09 |
| priteau | I said I would try a build in victoria but lost access to my env so haven't completed it | 15:09 |
| yoctozepto | https://review.opendev.org/c/openstack/kolla/+/796387 | 15:09 |
| yoctozepto | no problem | 15:09 |
| yoctozepto | I wanted to mention the workaround ^^ | 15:09 |
| mgoddard | doesn't seem to work | 15:10 |
| yoctozepto | ? | 15:11 |
| priteau | Are you saying this because of Zuul -1? The centos jobs passed | 15:11 |
| yoctozepto | it built | 15:11 |
| mgoddard | not debian | 15:11 |
| yoctozepto | debian's patch is broken on centos8 not passing | 15:11 |
| yoctozepto | https://review.opendev.org/c/openstack/kolla/+/795874 | 15:11 |
| yoctozepto | and so is final bump | 15:11 |
| yoctozepto | and so is our em release | 15:11 |
| mgoddard | so should we combine those patches and merge | 15:12 |
| mgoddard | ? | 15:12 |
| yoctozepto | debian is non-voting | 15:12 |
| mgoddard | the workaround doesn't seem any worse than the existing code | 15:13 |
| yoctozepto | I guess we could go with the workaround | 15:13 |
| yoctozepto | and bury train | 15:13 |
| yoctozepto | then worry about ussuri+ | 15:13 |
| mgoddard | we might get pressure to tag our EM soon | 15:13 |
| yoctozepto | indeed | 15:13 |
| mgoddard | don't want to get caught with our pants down | 15:14 |
| yoctozepto | I'm rechecking priteau's workaround | 15:14 |
| mgoddard | ok | 15:14 |
| yoctozepto | priteau to lift w-1 | 15:14 |
| yoctozepto | and /me rebasing other patches | 15:14 |
| mgoddard | k | 15:14 |
| priteau | I'll recheck too because of the POST_FAILURE | 15:14 |
| opendevreview | Radosław Piliszek proposed openstack/kolla stable/train: Enable Debian updates repo https://review.opendev.org/c/openstack/kolla/+/795874 | 15:15 |
| priteau | You already did | 15:15 |
| opendevreview | Radosław Piliszek proposed openstack/kolla stable/train: Bump final OpenStack versions for Train https://review.opendev.org/c/openstack/kolla/+/795258 | 15:15 |
| yoctozepto | fingers crossed | 15:15 |
| yoctozepto | priteau: yeah | 15:15 |
| yoctozepto | priteau: just need w0 from you | 15:15 |
| priteau | done | 15:15 |
| mgoddard | ok | 15:15 |
| yoctozepto | thx | 15:15 |
| yoctozepto | mgoddard: cr+2w+1? | 15:15 |
| mgoddard | the train is ready to go | 15:15 |
| yoctozepto | all the patches | 15:15 |
| mgoddard | I'll wait for zuul | 15:16 |
| mgoddard | #topic Wallaby release planning / libvirt/OVMF bug | 15:16 |
| yoctozepto | mgoddard ok | 15:16 |
| mgoddard | How are we looking with this one? | 15:16 |
| yoctozepto | priteau also knows | 15:16 |
| priteau | So, we appear to have a fix because c8 AV imported libvirt 7.0, although I would think it should *not* work | 15:17 |
| priteau | But anyway, c8s is due to fix their AV repo package today | 15:17 |
| priteau | Which will give us libvirt 7.4 | 15:17 |
| priteau | So I am refreshing http://mirror.centos.org/centos/8-stream/extras/x86_64/os/Packages/ multiple times, but still no updated package | 15:18 |
| priteau | hughesjr said by wednesday cob us time | 15:18 |
| mgoddard | So we continue to wait | 15:18 |
| priteau | unfortunately yes | 15:19 |
| mgoddard | hopefully tomorrow | 15:19 |
| mgoddard | #topic Xena cycle planning / master branch life cycle | 15:19 |
| priteau | If someone wants to proactively test libvirt 7.4, we could just update the repo file after the package has been installed | 15:19 |
| mgoddard | #link https://review.opendev.org/c/openstack/kolla/+/795320 | 15:19 |
| mgoddard | Not much to say on this one | 15:20 |
| mgoddard | Except please review | 15:20 |
| mgoddard | #topic Testing log coverage (and more?) | 15:20 |
| mgoddard | #link https://review.opendev.org/c/openstack/kolla-ansible/+/795512 | 15:20 |
| mgoddard | This is a patch that I've been iterating on recently, that could improve our log coverage | 15:21 |
| mgoddard | We currently have no real test coverage of our logging | 15:22 |
| yoctozepto | yeah, we need something in that direction | 15:22 |
| mgoddard | This should help | 15:22 |
| mgoddard | Basic idea is, we have an output plugin in fluentd image that writes a tab separated values file containing various labels for each log message | 15:23 |
| mgoddard | Then in kolla-ansible we have a new test that ingests that TSV file and verifies that everything is as expected | 15:23 |
| mgoddard | The test is implemented using py.test / testinfra | 15:24 |
| mgoddard | #link https://review.opendev.org/c/openstack/kolla-ansible/+/795512/13/tests/test-fluentd.py | 15:24 |
| yoctozepto | this obviously tests only whether we log at least one message | 15:25 |
| mgoddard | I think we could really tighten up our testing by introducing this | 15:25 |
| mgoddard | yoctozepto: correct, currently | 15:25 |
| yoctozepto | yeah | 15:25 |
| yoctozepto | I think someone worked on improving the fluentd integration | 15:25 |
| yoctozepto | re parsing | 15:25 |
| yoctozepto | so that we don't lose messages or get weird entries | 15:25 |
| mgoddard | right | 15:25 |
| mgoddard | we could do more in the output plugin to validate each fluentd event, currently it is quite minimal | 15:26 |
| mgoddard | e.g. does it have at least Hostname, programname, log_level set | 15:26 |
| mgoddard | I wanted to get the moving parts in place then it can be iterated on | 15:27 |
| yoctozepto | great, I am all in | 15:27 |
| yoctozepto | will review for sure | 15:27 |
| mgoddard | excellent | 15:27 |
| mgoddard | I think the existing tests/check-* scripts could be added to this py.test | 15:28 |
| mgoddard | then we get results in one place | 15:28 |
| mgoddard | and zuul and expose them as artifacts | 15:28 |
| mgoddard | we're using this approach in kayobe already: https://opendev.org/openstack/kayobe/src/branch/master/playbooks/kayobe-overcloud-host-configure-base/tests/test_overcloud_host_configure.py | 15:29 |
| yoctozepto | ++ | 15:29 |
| mgoddard | Any other comments/suggestions? | 15:31 |
| parallax | Looking good :) | 15:32 |
| mgoddard | cool | 15:33 |
| mgoddard | #topic Open discussion | 15:33 |
| mgoddard | Does anyone have anything else to discuss today? | 15:33 |
| headphoneJames | I'm reviving https://review.opendev.org/c/openstack/kolla-ansible/+/692179 | 15:34 |
| hrw | nope | 15:34 |
| mgoddard | that's good headphoneJames | 15:35 |
| mgoddard | are you still planning to email openstack-discuss? | 15:35 |
| headphoneJames | Yes. Would like to get this one at least working first | 15:36 |
| mgoddard | ok | 15:36 |
| opendevreview | Mark Goddard proposed openstack/kolla master: docs: Update release management for Xena process https://review.opendev.org/c/openstack/kolla/+/795320 | 15:39 |
| mgoddard | Let's wrap up there then | 15:41 |
| mgoddard | Thanks al | 15:41 |
| opendevreview | Mark Goddard proposed openstack/kolla master: docs: Update release management for Xena process https://review.opendev.org/c/openstack/kolla/+/795320 | 15:41 |
| mgoddard | *all | 15:41 |
| mgoddard | #endmeeting | 15:41 |
| opendevmeet | Meeting ended Wed Jun 16 15:41:43 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:41 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-06-16-15.00.html | 15:41 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-06-16-15.00.txt | 15:41 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-06-16-15.00.log.html | 15:41 |
| yoctozepto | thanks mgoddard | 15:44 |
| headphoneJames | thanks | 15:44 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: WIP: Log fluentd events in TSV format https://review.opendev.org/c/openstack/kolla-ansible/+/795512 | 15:54 |
| *** rpittau is now known as rpittau|afk | 16:11 | |
| imtiazc | mgoddard: What is the process for adding a topic to the PTG meetings? | 16:31 |
| mgoddard | imtiazc: do you mean the weekly IRC meeting or 6-monthly PTG? | 16:31 |
| imtiazc | I mean the weekly. | 16:32 |
| mgoddard | imtiazc: anyone can update the wiki page: https://wiki.openstack.org/wiki/Meetings/Kolla | 16:32 |
| ozzzo | if VMs on a new cluster don't have network access, what's the best way to allow console login to diagnose the issue? | 16:39 |
| ozzzo | is it possible to set a root password on a VM without having access to it first? | 16:39 |
| mgoddard | ozzzo: make or modify an image with a password | 16:39 |
| imtiazc | mgoddard: Thanks! I just realized that I do have edit access. Question: does Wallaby already support upgrading from Victoria? | 16:39 |
| mgoddard | imtiazc: Wallaby isn't released yet, but we are testing upgrades from Victoria | 16:40 |
| mgoddard | imtiazc: which distro do you use? | 16:40 |
| ozzzo | ok ty mgoddard | 16:40 |
| imtiazc | mgoddard: we are on CentOS 8 planning to move to stream | 16:40 |
| mgoddard | imtiazc: ok. Wallaby only supports stream. We may backport stream support to Victoria, but it is still WIP | 16:41 |
| mgoddard | imtiazc: help always appreciated :) | 16:41 |
| imtiazc | mgoddard: I haven't tried to upgrade yet but does kolla-ansible give an option to perform partial upgrades as in upgrade one component at a time and also upgrade controllers and computes separately? | 16:42 |
| imtiazc | mgoddard: We would love to help. I am encouraging our entire dev team to contribute | 16:43 |
| mgoddard | excellent | 16:43 |
| mgoddard | we're in particular need of reviews these days | 16:43 |
| imtiazc | ack | 16:44 |
| mgoddard | you can use --limit and --tags to restrict upgrades to hosts and/or services | 16:44 |
| mgoddard | but it needs care, you can easily skip mandatory tasks | 16:45 |
| imtiazc | Great! thanks for that. | 16:50 |
| yoctozepto | mgoddard: time to approve the train stack | 16:51 |
| mgoddard | k | 16:51 |
| yoctozepto | hmm, the neutron upgrade failure seems to bug ubuntu and at least train->ussuri and victoria->wallaby | 16:58 |
| yoctozepto | that's interesting considering this is the same error message in both cases | 16:59 |
| yoctozepto | i.e. it's a common migration script that runs wild | 16:59 |
| mgoddard | something in the ussuri migration | 16:59 |
| yoctozepto | " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/ussuri/expand/86274d77933e_change_mtu_to_not_null.py\", line 55, in upgrade", | 17:01 |
| yoctozepto | https://review.opendev.org/c/openstack/neutron/+/679399 | 17:02 |
| yoctozepto | though why did it start failing now | 17:02 |
| yoctozepto | better mariadb, better deps? | 17:03 |
| yoctozepto | "better" | 17:03 |
| yoctozepto | deps did not change, neutron either | 17:04 |
| yoctozepto | I mean, w.r.t anything migration-related | 17:04 |
| yoctozepto | duh | 17:05 |
| yoctozepto | *insert a deprecate joke here* | 17:07 |
| mgoddard | deprecate yoctozepto | 17:11 |
| *** gfidente is now known as gfidente|afk | 17:11 | |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: WIP: Log fluentd events in TSV format https://review.opendev.org/c/openstack/kolla-ansible/+/795512 | 17:12 |
| yoctozepto | upgrade mgoddard for he must be malfunctioning! | 17:14 |
| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Support for keystone scoped authorization https://review.opendev.org/c/openstack/kolla-ansible/+/692179 | 17:47 |
| *** ricolin_ is now known as ricolin | 17:49 | |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: WIP: CI: Test log aggregation https://review.opendev.org/c/openstack/kolla-ansible/+/795512 | 18:50 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: CI: Use testinfra for post-deploy checks https://review.opendev.org/c/openstack/kolla-ansible/+/796722 | 18:50 |
| *** hrw is now known as Guest2427 | 19:10 | |
| opendevreview | Merged openstack/kayobe stable/ussuri: CI: Revert to Docker Hub images for upgrade jobs https://review.opendev.org/c/openstack/kayobe/+/795959 | 19:28 |
| opendevreview | Merged openstack/kayobe stable/train: Revert "CI: switch to quay.io for container images" https://review.opendev.org/c/openstack/kayobe/+/795745 | 19:28 |
| *** hrww is now known as hrw | 20:11 | |
| *** hrw is now known as Guest2432 | 20:12 | |
| opendevreview | Verification of a change to openstack/kolla failed: Fix build of masakari-monitors image https://review.opendev.org/c/openstack/kolla/+/796387 | 20:17 |
| opendevreview | Verification of a change to openstack/kolla failed: Enable Debian updates repo https://review.opendev.org/c/openstack/kolla/+/795874 | 20:17 |
| opendevreview | Verification of a change to openstack/kolla failed: Bump final OpenStack versions for Train https://review.opendev.org/c/openstack/kolla/+/795258 | 20:17 |
| *** Guest2432 is now known as hrw | 21:12 | |
| *** hrw is now known as Guest2441 | 21:13 | |
| *** Guest2441 is now known as hrw | 21:23 | |
| opendevreview | Merged openstack/kolla stable/ussuri: [CI] Support building source images with in-review changes https://review.opendev.org/c/openstack/kolla/+/793568 | 21:28 |
| opendevreview | Merged openstack/kolla stable/train: Fix build of masakari-monitors image https://review.opendev.org/c/openstack/kolla/+/796387 | 23:24 |
| opendevreview | Merged openstack/kolla stable/train: Enable Debian updates repo https://review.opendev.org/c/openstack/kolla/+/795874 | 23:44 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!