openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for self-signed SSL certificates https://review.opendev.org/664517 | 00:16 |
---|---|---|
*** dave-mccowan has quit IRC | 00:47 | |
*** dave-mccowan has joined #openstack-kolla | 00:52 | |
openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for self-signed SSL certificates https://review.opendev.org/664517 | 02:35 |
*** skramaja has joined #openstack-kolla | 02:49 | |
*** BjoernT has joined #openstack-kolla | 02:51 | |
*** BjoernT_ has joined #openstack-kolla | 02:54 | |
*** BjoernT has quit IRC | 02:56 | |
*** k_mouza has joined #openstack-kolla | 03:46 | |
*** k_mouza has quit IRC | 03:50 | |
*** jdwidari has quit IRC | 04:10 | |
*** signed8bit has quit IRC | 04:17 | |
*** signed8bit has joined #openstack-kolla | 04:29 | |
*** pcaruana has joined #openstack-kolla | 04:42 | |
*** Luzi has joined #openstack-kolla | 04:50 | |
*** signed8bit has quit IRC | 05:03 | |
openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for self-signed SSL certificates https://review.opendev.org/664517 | 05:13 |
*** signed8bit has joined #openstack-kolla | 05:18 | |
*** janki has joined #openstack-kolla | 05:21 | |
*** radeks has joined #openstack-kolla | 05:28 | |
*** BjoernT_ has quit IRC | 05:45 | |
*** cah_link has joined #openstack-kolla | 06:08 | |
openstackgerrit | Qitao proposed openstack/kolla-ansible master: Update many invaild links for README https://review.opendev.org/683031 | 06:11 |
yoctozepto | morning | 06:16 |
*** cah_link has quit IRC | 06:24 | |
*** cah_link has joined #openstack-kolla | 06:27 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 06:33 |
mnasiadka | morning | 06:33 |
*** dougsz has joined #openstack-kolla | 06:37 | |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: Fix for haproxy precheck failing on CentOS running non-root https://review.opendev.org/682957 | 06:38 |
yoctozepto | mnasiadka: only randoms now on https://review.opendev.org/670690 :D | 06:39 |
mnasiadka | hmm, download.docker.com | 06:41 |
mnasiadka | another mirror candidate? | 06:41 |
mnasiadka | https://zuul.opendev.org/t/openstack/build/4c4d324012f4409f848f5537f91d7b84/log/primary/logs/ansible/bootstrap-servers#595 | 06:41 |
*** dougsz has quit IRC | 06:42 | |
mnasiadka | deb-docker is there, I guess we would need rpm-docker now :) | 06:42 |
yoctozepto | mnasiadka: yeah, we mirror dockerhub, yet not docker rpms | 06:44 |
yoctozepto | life | 06:44 |
yoctozepto | where do you see deb-docker mirror here? | 06:45 |
yoctozepto | even if we mirror it, we certainly don't use it :-) | 06:45 |
*** cah_link has quit IRC | 06:46 | |
mnasiadka | yoctozepto: I found it on the mirror itself, trying to find related system-config | 06:47 |
mnasiadka | ah, it's still in puppet | 06:48 |
hrw | moanring | 06:48 |
openstackgerrit | Qitao proposed openstack/kolla master: Update invalid link for README https://review.opendev.org/683036 | 06:49 |
mnasiadka | https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/reprepro/debian-docker-updates | 06:49 |
yoctozepto | mnasiadka: ah yeah, I see | 06:50 |
mnasiadka | yeah, we would need another reprepro config and relevant bits in puppet manifests | 06:50 |
yoctozepto | mnasiadka: since when reprepro works for yum repos | 06:51 |
yoctozepto | :D | 06:51 |
mnasiadka | ah, maybe not - let me check how other centos repos are mirrored | 06:51 |
mnasiadka | (too early) :) | 06:51 |
yoctozepto | rsync | 06:51 |
yoctozepto | but smaller probably differently | 06:51 |
yoctozepto | all via rsync | 06:53 |
yoctozepto | doubt docker com is available via rsync though | 06:54 |
yoctozepto | ;p | 06:54 |
*** gfidente has joined #openstack-kolla | 06:54 | |
yoctozepto | but I found something else | 06:54 |
yoctozepto | http://mirror.dfw.rax.opendev.org:8080/docker/ | 06:54 |
yoctozepto | reverseproxy | 06:54 |
yoctozepto | with cache | 06:54 |
yoctozepto | still better than nothing | 06:55 |
*** dougsz has joined #openstack-kolla | 06:57 | |
mnasiadka | sure, let's try that | 06:58 |
mnasiadka | at least if it fails we can blame infra :D | 06:59 |
mnasiadka | now we can blame the internet... | 06:59 |
mnasiadka | I'm more interested why a change is waiting an hour in Zuul's queue | 07:00 |
*** dougsz has quit IRC | 07:00 | |
yoctozepto | mnasiadka: it has started happening recently, other teams noticed as well | 07:02 |
mnasiadka | probably it's all those periodic jobs :) | 07:02 |
*** rpittau|afk is now known as rpittau | 07:05 | |
hrw | folks: https://review.opendev.org/#/c/682673/ https://review.opendev.org/#/c/682068/ as simple something to start a day? | 07:08 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla master: CI: Use docker mirror proxy https://review.opendev.org/683038 | 07:10 |
yoctozepto | hrw: https://review.opendev.org/683038 <- debian docker in docker | 07:10 |
hrw | yoctozepto: docker/dind you mean? | 07:13 |
yoctozepto | yup | 07:14 |
hrw | git rm -rf docker/dind;git commit;git review? | 07:14 |
yoctozepto | fancy doing repo url rewrite? | 07:14 |
yoctozepto | heh xD | 07:14 |
yoctozepto | does anyone use dind anyway lol | 07:14 |
hrw | yoctozepto: I am cleaning tasks today, packing tomorrow | 07:14 |
yoctozepto | it's a dead project | 07:15 |
hrw | drop | 07:15 |
yoctozepto | big backpack | 07:15 |
yoctozepto | then drop | 07:15 |
yoctozepto | and done | 07:15 |
yoctozepto | especially since you add debian support | 07:15 |
yoctozepto | just drop for debian | 07:15 |
yoctozepto | and we are good | 07:15 |
hrw | yoctozepto: I added Debian support to far too many images ;D | 07:15 |
yoctozepto | hrw: hopefully not kube* stuff | 07:16 |
yoctozepto | ;D | 07:16 |
yoctozepto | mgoddard, mnasiadka, hrw: deprecate dind? | 07:16 |
hrw | yoctozepto: kube*? I cleaned that stuff | 07:16 |
yoctozepto | dind's not even deployed by k-a | 07:17 |
yoctozepto | hrw: https://review.opendev.org/677422 | 07:18 |
yoctozepto | seems the U cycle comes with the long-awaited dust cleaner :D | 07:19 |
hrw | ;D | 07:20 |
*** cah_link has joined #openstack-kolla | 07:22 | |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: ppc64le: add UNBUILDABLE_IMAGES entries https://review.opendev.org/682068 | 07:22 |
hrw | comments included | 07:22 |
mnasiadka | yoctozepto: In images department I think we need to first deprecate a lot of them, and then remove in U | 07:31 |
yoctozepto | mnasiadka: what I'm saying indeed | 07:32 |
yoctozepto | already done with kubernetes | 07:32 |
yoctozepto | lemme replicated with dind | 07:32 |
yoctozepto | replicate* | 07:32 |
mnasiadka | and then find other unused :) | 07:34 |
mnasiadka | interesting, all Zuul running jobs are tripleo owned | 07:35 |
mnasiadka | the rest is mainly queued | 07:35 |
yoctozepto | mnasiadka: hmm, I think there was a relevant message on the ml | 07:36 |
*** cah_link has quit IRC | 07:36 | |
yoctozepto | anyways, dind results: http://codesearch.openstack.org/?q=%5Cbdind%5Cb&i=nope&files=&repos= | 07:36 |
yoctozepto | others not using ours | 07:36 |
mnasiadka | yoctozepto: yeah, I remember - but they are sabotaging everything :) | 07:36 |
*** luksky has joined #openstack-kolla | 07:37 | |
openstackgerrit | Merged openstack/kolla stable/rocky: base: fix OracleLinux build https://review.opendev.org/682927 | 07:38 |
yoctozepto | mnasiadka: so that only rhel8 is ready on time? :P | 07:39 |
yoctozepto | sure, sounds legit | 07:39 |
openstackgerrit | Merged openstack/kolla stable/queens: base: fix OracleLinux build https://review.opendev.org/682928 | 07:45 |
*** ivve has joined #openstack-kolla | 07:45 | |
openstackgerrit | Radosław Piliszek proposed openstack/kolla master: Release notes: deprecate dind https://review.opendev.org/683044 | 07:46 |
yoctozepto | periodics seem deadly today | 07:48 |
mnasiadka | well if tripleo jobs run for 5hr 20mins | 07:48 |
mnasiadka | so go figure | 07:48 |
*** zbr is now known as zbr|ruck | 07:57 | |
*** hjensas|afk is now known as hjensas | 08:06 | |
*** lennyb has quit IRC | 08:21 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 08:22 |
mnasiadka | yoctozepto: Repository 'docker-ce-stable': Error parsing config: Error parsing "gpgkey = '/centos/gpg'": URL must be http, ftp, file or https not "" | 08:28 |
mnasiadka | :) | 08:28 |
yoctozepto | mnasiadka: just seen | 08:28 |
yoctozepto | interpolation gone bad ;-( | 08:28 |
openstackgerrit | Scott Solkhon proposed openstack/kolla-ansible master: Adding Prometheus blackbox exporter https://review.opendev.org/681436 | 08:28 |
*** zzzeek has quit IRC | 08:30 | |
*** k_mouza has joined #openstack-kolla | 08:31 | |
*** zzzeek has joined #openstack-kolla | 08:31 | |
openstackgerrit | Radosław Piliszek proposed openstack/kolla master: CI: Use docker mirror proxy https://review.opendev.org/683038 | 08:32 |
*** lennyb has joined #openstack-kolla | 08:34 | |
openstackgerrit | Merged openstack/kolla master: Fix GitPython entry in requirements.txt https://review.opendev.org/682646 | 08:44 |
openstackgerrit | Merged openstack/kolla master: Build pdf doc https://review.opendev.org/682610 | 08:44 |
hrw | uf. another project code merged, ci updated, next week can close jira card | 08:45 |
openstackgerrit | Merged openstack/kolla master: build.py: fix pep8 warnings https://review.opendev.org/682673 | 08:46 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/pike: base: fix OracleLinux build https://review.opendev.org/683064 | 08:55 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Fix for haproxy precheck failing on CentOS running non-root https://review.opendev.org/683068 | 09:03 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: Fix for haproxy precheck failing on CentOS running non-root https://review.opendev.org/682957 | 09:03 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: Fix for haproxy precheck failing on CentOS running non-root https://review.opendev.org/682957 | 09:04 |
*** gfidente has quit IRC | 09:07 | |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Use VXLAN overlay network https://review.opendev.org/670690 | 09:08 |
*** wuchunyang has joined #openstack-kolla | 09:08 | |
*** wuchunyang has quit IRC | 09:10 | |
openstackgerrit | Merged openstack/kolla-ansible stable/stein: [designate] Fix admin api configuration https://review.opendev.org/682847 | 09:10 |
openstackgerrit | Merged openstack/kolla master: Update invalid link for README https://review.opendev.org/683036 | 09:10 |
*** gfidente has joined #openstack-kolla | 09:10 | |
yoctozepto | gate failure | 09:14 |
yoctozepto | INFO:kolla.common.utils.base:Executing: /tmp/apt-key-gpghome.SHhRlU6wvT/gpg.1.sh --no-tty --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 391A9AA2147192839E9DB0315EDB1B62EC4926EA | 09:14 |
yoctozepto | INFO:kolla.common.utils.base:[91mgpg: keyserver receive failed: Connection timed out | 09:14 |
mnasiadka | uhh | 09:14 |
yoctozepto | we need some intern to clean this stuff | 09:14 |
mnasiadka | that we won't just across :) | 09:15 |
mnasiadka | jump across | 09:15 |
mnasiadka | yoctozepto: apply for outreachy :) | 09:15 |
yoctozepto | mnasiadka: sure, I match perfectly | 09:15 |
mnasiadka | yoctozepto: I mean as a project, but still we need a mentor :) | 09:16 |
yoctozepto | mnasiadka: ah, yeah | 09:18 |
mnasiadka | yoctozepto: not that you want to be a mentee :D | 09:18 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: [designate] Fix admin api configuration https://review.opendev.org/683073 | 09:23 |
*** k_mouza has quit IRC | 09:26 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 09:28 |
*** k_mouza has joined #openstack-kolla | 09:39 | |
*** k_mouza has quit IRC | 09:41 | |
*** k_mouza has joined #openstack-kolla | 09:44 | |
mgoddard | morning | 09:46 |
hrw | hi Mark | 09:52 |
*** janki has quit IRC | 09:57 | |
openstackgerrit | Merged openstack/kolla-ansible master: Update many invaild links for README https://review.opendev.org/683031 | 10:08 |
*** egonzalez has joined #openstack-kolla | 10:08 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Add support for libvirt+tls https://review.opendev.org/650448 | 10:43 |
yoctozepto | zuul feels slow today | 10:51 |
mgoddard | it is | 10:53 |
mgoddard | feature freeze, ironically is the most active time :) | 10:53 |
*** pcaruana has quit IRC | 10:54 | |
yoctozepto | mgoddard: as mnasiadka pointed out this is felt even more due to restoration of tripleo gating - they recheck and gate like crazy today :D | 10:54 |
*** rpittau is now known as rpittau|afk | 10:56 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Add support for libvirt+tls https://review.opendev.org/650448 | 10:57 |
andrein | mgoddard: ping | 11:04 |
mgoddard | andrein: pong | 11:05 |
andrein | hello :) | 11:05 |
mgoddard | hi | 11:06 |
andrein | i've been wrestling with my vlan over bond issue for a while now, I *almost* got it working, but I must be missing something | 11:06 |
andrein | I figured out how cloud-init is supposed to set up networking using /mnt/config/openstack/latest/network_data.json | 11:08 |
mgoddard | if you get it working I'd be interested to know how | 11:09 |
andrein | I edited my network_data.json as described in http://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/metadata-service-network-info.html | 11:09 |
mgoddard | there is code in nova to build that file which supports bonds | 11:09 |
mgoddard | ah, you got it | 11:09 |
mgoddard | I'm heading out for lunch now | 11:09 |
andrein | ah, ok, ping me when you're back | 11:10 |
*** jdwidari has joined #openstack-kolla | 11:12 | |
egonzalez | andrein, what is/was the issue? | 11:13 |
andrein | hi egonzalez | 11:13 |
andrein | egonzalez: this is my bifrost server inventory https://www.irccloud.com/pastebin/xkDHZvxd/ | 11:14 |
andrein | I'm trying to set up vlan 2119 over a bond between two physical NICs | 11:14 |
*** pcaruana has joined #openstack-kolla | 11:17 | |
*** kplant has joined #openstack-kolla | 11:17 | |
andrein | on the instance, the network_data.json looks good, but on boot the interfaces aren't configured | 11:17 |
*** k_mouza has quit IRC | 11:17 | |
andrein | when I run `cloud-init devel net-convert -p network_data.json -k network_data.json -d / -D centos -O sysconfig` the interface configuration is generated properly | 11:18 |
andrein | and after a `systemctl start network` everything works as expected | 11:18 |
andrein | any idea why cloud-init isn't doing the right thing when the server boots? | 11:18 |
andrein | this is the cloud-init output right after boot. I'm trying to set up the bonds over enp130s0f{0,1} https://www.irccloud.com/pastebin/t7JXO4jX/ | 11:20 |
egonzalez | cloud init have network-config disabled? anything else after tarted Initial cloud-init job task on boot? | 11:23 |
yoctozepto | argh, kolla CI broken, again | 11:26 |
yoctozepto | sensu! https://615effe7fe88720ee720-41ed0c4c4ac97898f47335a8b01b94a8.ssl.cf2.rackcdn.com/683038/2/check/kolla-build-centos-source/33c1fcc/kolla/build/000_FAILED_sensu-client.txt.gz | 11:26 |
*** gfidente has quit IRC | 11:28 | |
andrein | egonzalez: looks like it can find the datasource for network config, so I guess it's not disabled: | 11:29 |
andrein | 2019-09-19 05:56:27,824 - handlers.py[DEBUG]: finish: init-network/search-None: SUCCESS: found network data from DataSourceNone | 11:29 |
andrein | 2019-09-19 05:56:27,825 - stages.py[INFO]: Loaded datasource DataSourceNone - DataSourceNone | 11:29 |
yoctozepto | mgoddard, mnasiadka, hrw: any sensu heroes? ^^^^ | 11:30 |
mnasiadka | yoctozepto, how, mgoddard: how many monitoring suites do we support? | 11:33 |
mnasiadka | Monasca, sensu, prometheus, collects and whatever there is | 11:34 |
mnasiadka | can we also support Nagios and zabbix in our Docker images collection? :) | 11:34 |
kplant | telegraf ^ | 11:34 |
mnasiadka | kplant: thanks for reminding - so let's say the Influx stack :) | 11:34 |
yoctozepto | we are well monitored | 11:34 |
yoctozepto | but broken now :< | 11:35 |
yoctozepto | mnasiadka: what do you propose | 11:35 |
yoctozepto | both short and longterm please | 11:35 |
hrw | INFO:kolla.common.utils.sensu-client:whois requires Ruby version >= 2.4. | 11:35 |
hrw | simple | 11:35 |
mnasiadka | Since there's not a lot of us, deprecate at least half of the solutions we don't maintain and have no clue if work :) | 11:35 |
mnasiadka | So let's start with Sensu :) | 11:36 |
mnasiadka | If it comes to me, Prometheus and Monasca make sense, the rest is just weight we carry - unless there's somebody that wants to support that. | 11:36 |
hrw | yoctozepto: I may fix it in ~1h | 11:38 |
mnasiadka | and none of these monitoring suites have a CI :) | 11:38 |
kplant | mnasiadka: that makes sense this they solve different problems | 11:38 |
kplant | s/this/since | 11:38 |
kplant | sorry.. stroke | 11:38 |
hrw | yoctozepto: whois needs to be == 4.1.0 | 11:38 |
mnasiadka | kplant: we don't need to build the images, sensualising delivers their own - why don't we ask users to use them? | 11:39 |
mnasiadka | geez, word fixer in Mac enabled by itself | 11:39 |
hrw | yoctozepto: as 5.0.0 got released today and wants ruby 2.4 | 11:39 |
mnasiadka | sensu delivers their own :) | 11:39 |
kplant | lol | 11:39 |
kplant | sensualising | 11:39 |
hrw | more and more gems will require ruby 2.4 | 11:39 |
mnasiadka | which exists only on Debian/Ubuntu? | 11:39 |
hrw | which is not present in centos | 11:40 |
hrw | 7 | 11:40 |
egonzalez | sensu was used by tripleo iirc | 11:40 |
kplant | it was, i think they're moving to an rsyslog approach | 11:40 |
kplant | ripping out sensu and fluentd/collectd? it's been a while | 11:40 |
*** gfidente has joined #openstack-kolla | 11:43 | |
yoctozepto | mnasiadka: that's short term | 11:43 |
yoctozepto | what about long term | 11:43 |
yoctozepto | maybe we should build images in two (or more) tiers | 11:43 |
yoctozepto | sensu breaking whole CI is just laughable | 11:44 |
mnasiadka | yoctozepto: well, maybe we should start thinking about consuming 3rd party docker images for some services like non-OpenStack projects | 11:44 |
yoctozepto | mnasiadka: also acceptable | 11:44 |
mnasiadka | like in ceph-ansible case | 11:44 |
mnasiadka | let's add this to next meeting agenda | 11:50 |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: sensu-client: install older whois gem https://review.opendev.org/683117 | 11:51 |
hrw | this should fix CI/centos | 11:51 |
mnasiadka | done | 11:52 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla master: sensu-client: install older whois gem https://review.opendev.org/683117 | 11:52 |
mnasiadka | we love Ruby, don't we? :) | 11:53 |
yoctozepto | hrw: quick edit | 11:53 |
yoctozepto | mnasiadka: thx | 11:53 |
hrw | mnasiadka: I do not care about Ruby. | 11:53 |
yoctozepto | I have nothing against ruby other than I don't know the ecosystem well | 11:53 |
hrw | mnasiadka: for me it is just yet another build failure | 11:53 |
hrw | yoctozepto: s/pip/gem/ and go ;D | 11:54 |
yoctozepto | hrw: but you knew ruby 2.4 migration is happening | 11:54 |
yoctozepto | ;p | 11:54 |
hrw | yoctozepto: already got hit by it in other of our images ;D | 11:55 |
yoctozepto | hrw: that explains it | 11:55 |
hrw | centos8 has ruby 2.5 | 12:01 |
* hrw food | 12:01 | |
*** verdurin has joined #openstack-kolla | 12:04 | |
yoctozepto | hrw: will probably be too new | 12:08 |
mgoddard | I only went for lunch and now CI is busted? | 12:09 |
yoctozepto | mgoddard: yeah, please read our little discussion | 12:09 |
mgoddard | I did | 12:09 |
yoctozepto | wdyt re tiered image building | 12:10 |
yoctozepto | we could align this with tables | 12:10 |
yoctozepto | from the support matrix | 12:10 |
mgoddard | could do, it's come up before as an idea | 12:11 |
*** goldyfruit_ has quit IRC | 12:11 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 12:21 |
*** k_mouza has joined #openstack-kolla | 12:27 | |
*** k_mouza has quit IRC | 12:31 | |
*** eandersson has quit IRC | 12:31 | |
*** priteau has joined #openstack-kolla | 12:36 | |
*** cah_link has joined #openstack-kolla | 12:48 | |
*** vmixor has quit IRC | 12:50 | |
*** vmixor has joined #openstack-kolla | 12:50 | |
*** cah_link has quit IRC | 12:52 | |
andrein | mgoddard: did you get a chance to review the configs i posted above? | 12:53 |
*** zzzeek has quit IRC | 12:56 | |
*** cah_link has joined #openstack-kolla | 12:57 | |
*** zzzeek has joined #openstack-kolla | 12:58 | |
*** vmixor has quit IRC | 12:58 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Check for VT in nova prechecks when using KVM https://review.opendev.org/683147 | 13:03 |
mgoddard | andrein: I did. I'm not sure what to say though. Have you been checking the cloud-init logs to see what it does on boot? | 13:05 |
*** hjensas has quit IRC | 13:13 | |
andrein | mgoddard: https://seashells.io/p/pS8KbbuB do you notice anything weird here? | 13:14 |
mgoddard | andrein: IIRC "Applying network configuration from fallback" means it has not parsed your config correctly | 13:16 |
mgoddard | doesn't really say why | 13:16 |
*** goldyfruit_ has joined #openstack-kolla | 13:16 | |
hrw | fsck. no gem on ubuntu | 13:18 |
*** cah_link has quit IRC | 13:20 | |
hrw | sensu does not depend on ruby | 13:21 |
*** cah_link has joined #openstack-kolla | 13:27 | |
*** pcaruana has quit IRC | 13:28 | |
openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: Fixes Octavia https://review.opendev.org/683159 | 13:31 |
hrw | have fix | 13:33 |
*** wuchunyang has joined #openstack-kolla | 13:34 | |
openstackgerrit | Dincer Celik proposed openstack/kolla master: [horizon] Enable octavia dashboard https://review.opendev.org/683160 | 13:37 |
*** vmixor has joined #openstack-kolla | 13:38 | |
*** devep has joined #openstack-kolla | 13:38 | |
openstackgerrit | Merged openstack/kolla-ansible stable/rocky: [designate] Fix admin api configuration https://review.opendev.org/683073 | 13:42 |
*** gfidente has quit IRC | 13:45 | |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: sensu-client: install older whois gem https://review.opendev.org/683117 | 13:45 |
hrw | this ver works on centos and ubuntu | 13:45 |
hrw | uses gem only on rpm distros and change nothing otherwise | 13:46 |
*** mixor has joined #openstack-kolla | 13:52 | |
*** Luzi has quit IRC | 13:53 | |
*** vmixor has quit IRC | 13:53 | |
*** pcaruana has joined #openstack-kolla | 13:54 | |
*** JamesBenson has joined #openstack-kolla | 13:55 | |
*** zhangfei has joined #openstack-kolla | 13:56 | |
*** BjoernT has joined #openstack-kolla | 13:59 | |
*** BjoernT_ has joined #openstack-kolla | 14:02 | |
*** BjoernT has quit IRC | 14:04 | |
*** hjensas has joined #openstack-kolla | 14:07 | |
*** ivve has quit IRC | 14:09 | |
*** cah_link has quit IRC | 14:11 | |
*** goldyfruit___ has joined #openstack-kolla | 14:15 | |
*** goldyfruit_ has quit IRC | 14:18 | |
*** gfidente has joined #openstack-kolla | 14:23 | |
*** wuchunyang has quit IRC | 14:23 | |
*** wuchunyang has joined #openstack-kolla | 14:24 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Add support for libvirt+tls https://review.opendev.org/650448 | 14:32 |
*** pcaruana has quit IRC | 14:33 | |
*** wuchunyang has quit IRC | 14:36 | |
shubjero | I'm attempting to do a multinode kolla-ansible deployment but it keeps failing at certain mariadb stages. I noticed that the mariadb containers are restarting constantly on all 3 control nodes. I'm trying to deploy rocky on Ubuntu 18.04. It's not entirely clear why the mariadb containers keep cycling. Any suggestions? | 14:38 |
*** wuchunyang has joined #openstack-kolla | 14:40 | |
kplant | docker logs -f mariadb | 14:41 |
kplant | or: tail -f /var/log/kolla/mariadb/mariadb.log | 14:41 |
*** wuchunyang has quit IRC | 14:45 | |
*** iclon has quit IRC | 14:45 | |
shubjero | kplant: well its hard to decipher why considering all 3 mariadb containers keep cycling. I've stopped mariadb on control2 & 3.. going to see what 1 complains about | 14:48 |
shubjero | kplant: https://paste.ubuntu.com/p/bTmsZQsc33/ | 14:49 |
kplant | looks like a network issue | 14:50 |
kplant | have you confirmed reachability between the three nodes across 172.25.10.x? | 14:51 |
shubjero | kplant: yeah, not sure why though.. its just 3 instances on the same network.. | 14:51 |
shubjero | yep | 14:51 |
kplant | iptables? | 14:54 |
openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for self-signed SSL certificates https://review.opendev.org/664517 | 14:55 |
shubjero | kplant: nothing in iptables other than what was put there by docker/kolla-ansible | 14:57 |
shubjero | kplant: i can telnet from each control node to the mariadb port's on all 3 | 14:58 |
openstackgerrit | Mark Goddard proposed openstack/kayobe stable/rocky: Add alternative tenks deploy and teardown entrypoints. https://review.opendev.org/683176 | 14:59 |
*** iclon has joined #openstack-kolla | 14:59 | |
*** skramaja has quit IRC | 15:03 | |
*** wuchunyang has joined #openstack-kolla | 15:06 | |
*** egonzalez has quit IRC | 15:09 | |
*** wuchunyang has quit IRC | 15:12 | |
*** hjensas has quit IRC | 15:24 | |
*** k_mouza has joined #openstack-kolla | 15:24 | |
*** k_mouza has quit IRC | 15:28 | |
*** k_mouza has joined #openstack-kolla | 15:29 | |
zbr|ruck | can someone look at https://bugs.launchpad.net/tripleo/+bug/1844697 -- apparently there is no retry on push and this causes failures | 15:32 |
openstack | Launchpad bug 1844697 in tripleo "[koilla] periodic: container build job can fail during push" [Undecided,New] | 15:32 |
*** gfidente has quit IRC | 15:35 | |
mgoddard | hi zbr|ruck, I think we do have a retry mechanism | 15:36 |
mgoddard | zbr|ruck: kolla-build --retries | 15:37 |
zbr|ruck | hmm... now i need to find how this is called and how do retries apply because clearly i did not see it retrying, even once | 15:39 |
zbr|ruck | my impression is that retries apply only to building but not to pushing | 15:39 |
mgoddard | logs look like it is not retrying | 15:39 |
*** zhangfei has quit IRC | 15:39 | |
zbr|ruck | mgoddard: at least it should easy to fix, reusing the same parameter for pushing too. | 15:40 |
mgoddard | zbr|ruck: looks like it should work for pushing, but doesn't | 15:42 |
mgoddard | 2019-09-19 01:13:00 | INFO:kolla.common.utils:Attempt number: 2 to run task: PushTask(glance-api) | 15:42 |
mgoddard | 2019-09-19 01:13:00 | INFO:kolla.common.utils.glance-api:Trying to push the image | 15:42 |
zbr|ruck | in fact I see retry attempts up to 4 but they all happen during 2s interval, but I do not see additional errors on them. | 15:44 |
mgoddard | zbr|ruck: which branch is this? | 15:44 |
zbr|ruck | master | 15:44 |
mgoddard | given the lack of an error, I think the push must have succeeded, but something is wrong with the state that prevents it from setting self.success = True | 15:46 |
zbr|ruck | in fact, nope, read it wrong: it did wait for 40mins! | 15:46 |
zbr|ruck | i updated the msg to include the retries in it (the log is quite long) | 15:47 |
zbr|ruck | so yes, there are retries, I can see a 40s (not min) delay between 2nd and 3rd attempt. | 15:48 |
zbr|ruck | anyway, i would expect to see 3-4 full timeouts, not only one. | 15:48 |
mgoddard | so sounds to me like first one errored, second one finished successfully | 15:50 |
zbr|ruck | probably but the status remained failed, and the proof is the other msgs. still my first move would be to include exception in "Unknown error when pushing". | 15:51 |
mgoddard | zbr|ruck: something like this? http://paste.openstack.org/show/777708/ | 15:51 |
zbr|ruck | not displaying raised exception makes harder to debug the behavior. any reason for censoring real exception? | 15:51 |
mgoddard | no reason | 15:52 |
mgoddard | hold up, we do include the exception | 15:53 |
mgoddard | log.exception() | 15:53 |
mgoddard | ReadTimeoutError | 15:53 |
mgoddard | timeout interacting with docker via unix socket | 15:53 |
zbr|ruck | why we see only one? | 15:53 |
zbr|ruck | you can easilly see that there are 3 other retries after: Attempt number: 2/3/4 to run task: PushTask(glance-api) | 15:54 |
zbr|ruck | and no error between them, is like the status remained failed. | 15:54 |
zbr|ruck | also "Pushed successfully" is never printed for glande-api | 15:56 |
zbr|ruck | this makes me believe, it was not pushed. | 15:56 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Fix Octavia to use keystone_admin_project variable https://review.opendev.org/683159 | 15:58 |
*** mgoddard has quit IRC | 15:58 | |
*** mgoddard has joined #openstack-kolla | 15:59 | |
mgoddard | zbr|ruck: I think the image was pushed, but the state never got reset from connection_error, so the finally block always thinks it has failed | 16:00 |
*** openstackgerrit has quit IRC | 16:06 | |
zbr|ruck | mgoddard: indeed, it seems that we need to reset image.status before calling push on it, right? | 16:10 |
mgoddard | zbr|ruck: just trying out a fix | 16:10 |
zbr|ruck | mgoddard: thanks! just add me as reviewer. | 16:11 |
andrein | mgoddard: I think I've got bonding + vlans working | 16:16 |
mgoddard | andrein: nice work | 16:16 |
andrein | but... took some manual input | 16:16 |
andrein | so, looks like the problem is in /etc/cloud/cloud.cfg.d/91-dib-cloud-init-datasources.cfg | 16:17 |
andrein | my image has `datasource_list: [ None, None ]` | 16:17 |
andrein | looks like dib put it there, and I'm not sure why | 16:17 |
andrein | that effectively disables the configdrive support in cloud-init (though I'm not sure how on earth the VLAN without bonding worked previously) | 16:18 |
andrein | are you familiar with how bifrost builds the deployment image? | 16:19 |
*** openstackgerrit has joined #openstack-kolla | 16:22 | |
openstackgerrit | Mark Goddard proposed openstack/kolla master: Fix retries when pushing images https://review.opendev.org/683200 | 16:22 |
mgoddard | andrein: aha, try dib_init_element: cloud-init | 16:24 |
*** k_mouza has quit IRC | 16:25 | |
andrein | mgoddard: since I'm rebuilding this, if I want IPMI SOL over ttyS1 instead of ttyS0, how would I do that? | 16:26 |
shubjero | Doing a multinode install (getting further!) but coming across a problem with the 'running placement bootstrap container' task. 404 Client Error: Not Found (\"manifest for kolla/ubuntu-source-placement-api:rocky not found: manifest unknown: manifest unknown\")\\n'"} | 16:28 |
shubjero | https://hub.docker.com/r/kolla/ubuntu-source-placement-api/tags only has master and stein tags. Is the rocky image supposed to be missing? | 16:30 |
*** k_mouza has joined #openstack-kolla | 16:34 | |
*** itlinux has joined #openstack-kolla | 16:35 | |
yoctozepto | shubjero: placement split happened in stein | 16:36 |
yoctozepto | use rocky kolla-ansible for rocky images | 16:36 |
mgoddard | andrein: try extra_kernel_options | 16:37 |
yoctozepto | shubjero: or just use stein | 16:38 |
yoctozepto | why use rocky for new deployments :O | 16:38 |
shubjero | yoctozepto: oh, maybe i downloaded the wrong kolla-ansible branch, crap. Anyways I am using rocky because our prod environment is on rocky and we want to somehow migrate to kolla-ansible on rocky first.. | 16:39 |
yoctozepto | shubjero: roger that | 16:39 |
shubjero | yoctozepto: context: our prod cloud is old school ubuntu packages | 16:39 |
yoctozepto | ah | 16:39 |
shubjero | just looking to modernize.. so deploying multinode kolla in the lab is just trying to get our feet wet | 16:40 |
yoctozepto | step in the right direction | 16:40 |
*** dciabrin has quit IRC | 16:41 | |
*** igordc has joined #openstack-kolla | 16:44 | |
*** jdwidari has quit IRC | 16:51 | |
*** devep has quit IRC | 16:58 | |
*** k_mouza has quit IRC | 16:59 | |
*** pcaruana has joined #openstack-kolla | 17:01 | |
*** hjensas has joined #openstack-kolla | 17:04 | |
shubjero | yoctozepto: If I do 'pip install kolla-ansible' as per ( https://docs.openstack.org/project-deploy-guide/kolla-ansible/rocky/quickstart.html ), is this getting me the correct version of kolla-ansible to deploy rocky? | 17:04 |
yoctozepto | shubjero: good point, we should fix the docs - pip always installs the latest obviously :-) | 17:07 |
*** zbr|ruck is now known as zbr | 17:07 | |
yoctozepto | so it's stein now | 17:07 |
yoctozepto | and will soon be train ;D | 17:07 |
yoctozepto | I seriously recommend you use the version from git, branch stable/rocky | 17:08 |
yoctozepto | it gets latest fixes (unlike release) | 17:08 |
yoctozepto | and for sure will be for rocky ;-) | 17:08 |
*** pcaruana has quit IRC | 17:14 | |
yoctozepto | mgoddard: have you considered whether one of those old changes is actually interesting? | 17:15 |
yoctozepto | shubjero: reported for you: https://bugs.launchpad.net/kolla-ansible/+bug/1844710 | 17:18 |
openstack | Launchpad bug 1844710 in kolla-ansible "deploy docs are misleading wrt version of kolla-ansible being installed" [High,Triaged] | 17:18 |
shubjero | yoctozepto: nice, thanks | 17:19 |
*** gmann_afk is now known as gmann | 17:21 | |
andrein | when I see 'tc@box:~$', does it mean my IPA ramdisk is busted? | 17:23 |
andrein | oh, it rebooted, i think I broke the IPA output somehow | 17:24 |
*** pcaruana has joined #openstack-kolla | 17:25 | |
andrein | if I remove the deployment_image, does it also rebuild the ipa image? | 17:25 |
yoctozepto | andrein: for ironic and bifrost help it might be wise to also hit the #openstack-ironic channel | 17:31 |
yoctozepto | I cannot help you and mgoddard seems afk | 17:32 |
andrein | I’m just heading home atm, will hit the ironic channel in an hour or so | 17:33 |
openstackgerrit | Merged openstack/kolla master: sensu-client: install older whois gem https://review.opendev.org/683117 | 17:40 |
yoctozepto | CI unbroken ^ | 17:43 |
mnasiadka | And let it stay that way at least until tomorrow :) | 17:44 |
yoctozepto | hrw: I see you are quick to recheck, just like me | 17:45 |
hrw | yoctozepto: one day I will write userscript to add 'recheck' button next to 'Reply' one | 17:49 |
* hrw off | 17:49 | |
*** pcaruana has quit IRC | 17:55 | |
*** priteau has quit IRC | 18:07 | |
yoctozepto | hrw: hah | 18:07 |
*** xelaot has joined #openstack-kolla | 18:16 | |
*** igordc has quit IRC | 18:20 | |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Add bifrost Ubuntu job https://review.opendev.org/682292 | 18:27 |
*** priteau has joined #openstack-kolla | 18:27 | |
openstackgerrit | Merged openstack/kolla stable/pike: base: fix OracleLinux build https://review.opendev.org/683064 | 18:29 |
*** priteau has quit IRC | 18:29 | |
yoctozepto | yay, pike unbroken, finally | 18:31 |
*** radeks has quit IRC | 18:51 | |
*** xelaot has quit IRC | 18:58 | |
yoctozepto | https://imgur.com/a/OFz2rpX | 19:00 |
yoctozepto | ;D | 19:00 |
openstackgerrit | Merged openstack/kolla-ansible stable/stein: Fix for haproxy precheck failing on CentOS running non-root https://review.opendev.org/682957 | 19:08 |
openstackgerrit | Merged openstack/kolla-ansible master: Fix for haproxy precheck failing on CentOS running non-root https://review.opendev.org/683068 | 19:08 |
openstackgerrit | Merged openstack/kolla-ansible stable/stein: Fix prometheus-alertmanager cluster bug https://review.opendev.org/682888 | 19:08 |
openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for self-signed SSL certificates https://review.opendev.org/664517 | 19:21 |
*** dougsz has joined #openstack-kolla | 19:40 | |
*** mixor has quit IRC | 19:47 | |
*** pcaruana has joined #openstack-kolla | 19:51 | |
*** kplant has quit IRC | 19:53 | |
*** dougsz has quit IRC | 20:02 | |
*** mixor has joined #openstack-kolla | 20:22 | |
*** mixor has quit IRC | 20:31 | |
*** pcaruana has quit IRC | 20:31 | |
andrein | mgoddard: It's ALIVE | 20:31 |
*** igordc has joined #openstack-kolla | 20:33 | |
* andrein does the victory dance | 20:34 | |
*** scottsol has joined #openstack-kolla | 20:40 | |
openstackgerrit | Merged openstack/kolla master: ppc64le: add UNBUILDABLE_IMAGES entries https://review.opendev.org/682068 | 21:04 |
openstackgerrit | Merged openstack/kolla master: Add kayobe to deliverables.yaml https://review.opendev.org/674510 | 21:16 |
openstackgerrit | Merged openstack/kolla-ansible master: Update "openstack_release" variable to static brach name https://review.opendev.org/681633 | 21:21 |
openstackgerrit | Merged openstack/kolla master: Add package systemd-sysv to bifrost-deploy image https://review.opendev.org/679936 | 21:21 |
*** JamesBenson has quit IRC | 21:29 | |
*** benj_ has quit IRC | 21:31 | |
*** benj has joined #openstack-kolla | 21:31 | |
*** JamesBenson has joined #openstack-kolla | 21:31 | |
*** benj is now known as Guest29634 | 21:32 | |
mloza | hello, I enabled mistral in stable/stein env and it deploys fine but getting error when I access it in the dashboard. Here is the log http://paste.openstack.org/show/777953/ | 21:35 |
mloza | I get the same error in the CLI | 21:35 |
mloza | The mistral endpoint is missing in the API Access | 21:36 |
*** JamesBenson has quit IRC | 21:36 | |
*** scottsol has quit IRC | 21:36 | |
*** mixor has joined #openstack-kolla | 21:37 | |
*** luksky has quit IRC | 21:41 | |
*** scottsol has joined #openstack-kolla | 21:42 | |
openstackgerrit | Merged openstack/kolla-ansible master: Fix Octavia to use keystone_admin_project variable https://review.opendev.org/683159 | 21:43 |
*** scottsol has quit IRC | 21:58 | |
*** BjoernT_ has quit IRC | 22:00 | |
openstackgerrit | Viktor Michalek proposed openstack/kolla master: Fix images based on Debian Buster Stein packages https://review.opendev.org/680447 | 22:02 |
openstackgerrit | Merged openstack/kolla master: Add volume-related packages to zun-compute image https://review.opendev.org/682437 | 22:09 |
openstackgerrit | Viktor Michalek proposed openstack/kolla master: Fix images based on Debian Buster Stein packages https://review.opendev.org/680447 | 22:11 |
*** mixor has quit IRC | 22:19 | |
*** igordc has quit IRC | 22:44 | |
*** tonythomas has quit IRC | 22:45 | |
*** signed8b_ has joined #openstack-kolla | 22:48 | |
*** signed8bit has quit IRC | 22:50 | |
*** Tengu has quit IRC | 22:51 | |
*** goldyfruit___ has quit IRC | 23:00 | |
*** igordc has joined #openstack-kolla | 23:01 | |
*** Tengu has joined #openstack-kolla | 23:03 | |
*** wuchunyang has joined #openstack-kolla | 23:47 | |
*** JamesBenson has joined #openstack-kolla | 23:47 | |
*** JamesBenson has quit IRC | 23:51 | |
*** signed8b_ has quit IRC | 23:56 | |
*** signed8bit has joined #openstack-kolla | 23:56 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!