Thursday, 2019-09-19

« Wednesday, 2019-09-18 Index Friday, 2019-09-20 »
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for self-signed SSL certificates  https://review.opendev.org/66451700:16
*** dave-mccowan has quit IRC00:47
*** dave-mccowan has joined #openstack-kolla00:52
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for self-signed SSL certificates  https://review.opendev.org/66451702:35
*** skramaja has joined #openstack-kolla02:49
*** BjoernT has joined #openstack-kolla02:51
*** BjoernT_ has joined #openstack-kolla02:54
*** BjoernT has quit IRC02:56
*** k_mouza has joined #openstack-kolla03:46
*** k_mouza has quit IRC03:50
*** jdwidari has quit IRC04:10
*** signed8bit has quit IRC04:17
*** signed8bit has joined #openstack-kolla04:29
*** pcaruana has joined #openstack-kolla04:42
*** Luzi has joined #openstack-kolla04:50
*** signed8bit has quit IRC05:03
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for self-signed SSL certificates  https://review.opendev.org/66451705:13
*** signed8bit has joined #openstack-kolla05:18
*** janki has joined #openstack-kolla05:21
*** radeks has joined #openstack-kolla05:28
*** BjoernT_ has quit IRC05:45
*** cah_link has joined #openstack-kolla06:08
openstackgerritQitao proposed openstack/kolla-ansible master: Update many invaild links for README  https://review.opendev.org/68303106:11
yoctozeptomorning06:16
*** cah_link has quit IRC06:24
*** cah_link has joined #openstack-kolla06:27
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637606:33
mnasiadkamorning06:33
*** dougsz has joined #openstack-kolla06:37
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Fix for haproxy precheck failing on CentOS running non-root  https://review.opendev.org/68295706:38
yoctozeptomnasiadka: only randoms now on https://review.opendev.org/670690 :D06:39
mnasiadkahmm, download.docker.com06:41
mnasiadkaanother mirror candidate?06:41
mnasiadkahttps://zuul.opendev.org/t/openstack/build/4c4d324012f4409f848f5537f91d7b84/log/primary/logs/ansible/bootstrap-servers#59506:41
*** dougsz has quit IRC06:42
mnasiadkadeb-docker is there, I guess we would need rpm-docker now :)06:42
yoctozeptomnasiadka: yeah, we mirror dockerhub, yet not docker rpms06:44
yoctozeptolife06:44
yoctozeptowhere do you see deb-docker mirror here?06:45
yoctozeptoeven if we mirror it, we certainly don't use it :-)06:45
*** cah_link has quit IRC06:46
mnasiadkayoctozepto: I found it on the mirror itself, trying to find related system-config06:47
mnasiadkaah, it's still in puppet06:48
hrwmoanring06:48
openstackgerritQitao proposed openstack/kolla master: Update invalid link for README  https://review.opendev.org/68303606:49
mnasiadkahttps://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/reprepro/debian-docker-updates06:49
yoctozeptomnasiadka: ah yeah, I see06:50
mnasiadkayeah, we would need another reprepro config and relevant bits in puppet manifests06:50
yoctozeptomnasiadka: since when reprepro works for yum repos06:51
yoctozepto:D06:51
mnasiadkaah, maybe not - let me check how other centos repos are mirrored06:51
mnasiadka(too early) :)06:51
yoctozeptorsync06:51
yoctozeptobut smaller probably differently06:51
yoctozeptoall via rsync06:53
yoctozeptodoubt docker com is available via rsync though06:54
yoctozepto;p06:54
*** gfidente has joined #openstack-kolla06:54
yoctozeptobut I found something else06:54
yoctozeptohttp://mirror.dfw.rax.opendev.org:8080/docker/06:54
yoctozeptoreverseproxy06:54
yoctozeptowith cache06:54
yoctozeptostill better than nothing06:55
*** dougsz has joined #openstack-kolla06:57
mnasiadkasure, let's try that06:58
mnasiadkaat least if it fails we can blame infra :D06:59
mnasiadkanow we can blame the internet...06:59
mnasiadkaI'm more interested why a change is waiting an hour in Zuul's queue07:00
*** dougsz has quit IRC07:00
yoctozeptomnasiadka: it has started happening recently, other teams noticed as well07:02
mnasiadkaprobably it's all those periodic jobs :)07:02
*** rpittau|afk is now known as rpittau07:05
hrwfolks: https://review.opendev.org/#/c/682673/  https://review.opendev.org/#/c/682068/ as simple something to start a day?07:08
openstackgerritRadosław Piliszek proposed openstack/kolla master: CI: Use docker mirror proxy  https://review.opendev.org/68303807:10
yoctozeptohrw: https://review.opendev.org/683038 <- debian docker in docker07:10
hrwyoctozepto: docker/dind you mean?07:13
yoctozeptoyup07:14
hrwgit rm -rf docker/dind;git commit;git review?07:14
yoctozeptofancy doing repo url rewrite?07:14
yoctozeptoheh xD07:14
yoctozeptodoes anyone use dind anyway lol07:14
hrwyoctozepto: I am cleaning tasks today, packing tomorrow07:14
yoctozeptoit's a dead project07:15
hrwdrop07:15
yoctozeptobig backpack07:15
yoctozeptothen drop07:15
yoctozeptoand done07:15
yoctozeptoespecially since you add debian support07:15
yoctozeptojust drop for debian07:15
yoctozeptoand we are good07:15
hrwyoctozepto: I added Debian support to far too many images ;D07:15
yoctozeptohrw: hopefully not kube* stuff07:16
yoctozepto;D07:16
yoctozeptomgoddard, mnasiadka, hrw: deprecate dind?07:16
hrwyoctozepto: kube*? I cleaned that stuff07:16
yoctozeptodind's not even deployed by k-a07:17
yoctozeptohrw: https://review.opendev.org/67742207:18
yoctozeptoseems the U cycle comes with the long-awaited dust cleaner :D07:19
hrw;D07:20
*** cah_link has joined #openstack-kolla07:22
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: ppc64le: add UNBUILDABLE_IMAGES entries  https://review.opendev.org/68206807:22
hrwcomments included07:22
mnasiadkayoctozepto: In images department I think we need to first deprecate a lot of them, and then remove in U07:31
yoctozeptomnasiadka: what I'm saying indeed07:32
yoctozeptoalready done with kubernetes07:32
yoctozeptolemme replicated with dind07:32
yoctozeptoreplicate*07:32
mnasiadkaand then find other unused :)07:34
mnasiadkainteresting, all Zuul running jobs are tripleo owned07:35
mnasiadkathe rest is mainly queued07:35
yoctozeptomnasiadka: hmm, I think there was a relevant message on the ml07:36
*** cah_link has quit IRC07:36
yoctozeptoanyways, dind results: http://codesearch.openstack.org/?q=%5Cbdind%5Cb&i=nope&files=&repos=07:36
yoctozeptoothers not using ours07:36
mnasiadkayoctozepto: yeah, I remember - but they are sabotaging everything :)07:36
*** luksky has joined #openstack-kolla07:37
openstackgerritMerged openstack/kolla stable/rocky: base: fix OracleLinux build  https://review.opendev.org/68292707:38
yoctozeptomnasiadka: so that only rhel8 is ready on time? :P07:39
yoctozeptosure, sounds legit07:39
openstackgerritMerged openstack/kolla stable/queens: base: fix OracleLinux build  https://review.opendev.org/68292807:45
*** ivve has joined #openstack-kolla07:45
openstackgerritRadosław Piliszek proposed openstack/kolla master: Release notes: deprecate dind  https://review.opendev.org/68304407:46
yoctozeptoperiodics seem deadly today07:48
mnasiadkawell if tripleo jobs run for 5hr 20mins07:48
mnasiadkaso go figure07:48
*** zbr is now known as zbr|ruck07:57
*** hjensas|afk is now known as hjensas08:06
*** lennyb has quit IRC08:21
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637608:22
mnasiadkayoctozepto: Repository 'docker-ce-stable': Error parsing config: Error parsing "gpgkey = '/centos/gpg'": URL must be http, ftp, file or https not ""08:28
mnasiadka:)08:28
yoctozeptomnasiadka: just seen08:28
yoctozeptointerpolation gone bad ;-(08:28
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Adding Prometheus blackbox exporter  https://review.opendev.org/68143608:28
*** zzzeek has quit IRC08:30
*** k_mouza has joined #openstack-kolla08:31
*** zzzeek has joined #openstack-kolla08:31
openstackgerritRadosław Piliszek proposed openstack/kolla master: CI: Use docker mirror proxy  https://review.opendev.org/68303808:32
*** lennyb has joined #openstack-kolla08:34
openstackgerritMerged openstack/kolla master: Fix GitPython entry in requirements.txt  https://review.opendev.org/68264608:44
openstackgerritMerged openstack/kolla master: Build pdf doc  https://review.opendev.org/68261008:44
hrwuf. another project code merged, ci updated, next week can close jira card08:45
openstackgerritMerged openstack/kolla master: build.py: fix pep8 warnings  https://review.opendev.org/68267308:46
openstackgerritRadosław Piliszek proposed openstack/kolla stable/pike: base: fix OracleLinux build  https://review.opendev.org/68306408:55
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Fix for haproxy precheck failing on CentOS running non-root  https://review.opendev.org/68306809:03
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Fix for haproxy precheck failing on CentOS running non-root  https://review.opendev.org/68295709:03
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Fix for haproxy precheck failing on CentOS running non-root  https://review.opendev.org/68295709:04
*** gfidente has quit IRC09:07
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Use VXLAN overlay network  https://review.opendev.org/67069009:08
*** wuchunyang has joined #openstack-kolla09:08
*** wuchunyang has quit IRC09:10
openstackgerritMerged openstack/kolla-ansible stable/stein: [designate] Fix admin api configuration  https://review.opendev.org/68284709:10
openstackgerritMerged openstack/kolla master: Update invalid link for README  https://review.opendev.org/68303609:10
*** gfidente has joined #openstack-kolla09:10
yoctozeptogate failure09:14
yoctozeptoINFO:kolla.common.utils.base:Executing: /tmp/apt-key-gpghome.SHhRlU6wvT/gpg.1.sh --no-tty --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 391A9AA2147192839E9DB0315EDB1B62EC4926EA09:14
yoctozeptoINFO:kolla.common.utils.base:[91mgpg: keyserver receive failed: Connection timed out09:14
mnasiadkauhh09:14
yoctozeptowe need some intern to clean this stuff09:14
mnasiadkathat we won't just across :)09:15
mnasiadkajump across09:15
mnasiadkayoctozepto: apply for outreachy :)09:15
yoctozeptomnasiadka: sure, I match perfectly09:15
mnasiadkayoctozepto: I mean as a project, but still we need a mentor :)09:16
yoctozeptomnasiadka: ah, yeah09:18
mnasiadkayoctozepto: not that you want to be a mentee :D09:18
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/rocky: [designate] Fix admin api configuration  https://review.opendev.org/68307309:23
*** k_mouza has quit IRC09:26
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637609:28
*** k_mouza has joined #openstack-kolla09:39
*** k_mouza has quit IRC09:41
*** k_mouza has joined #openstack-kolla09:44
mgoddardmorning09:46
hrwhi Mark09:52
*** janki has quit IRC09:57
openstackgerritMerged openstack/kolla-ansible master: Update many invaild links for README  https://review.opendev.org/68303110:08
*** egonzalez has joined #openstack-kolla10:08
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Add support for libvirt+tls  https://review.opendev.org/65044810:43
yoctozeptozuul feels slow today10:51
mgoddardit is10:53
mgoddardfeature freeze, ironically is the most active time :)10:53
*** pcaruana has quit IRC10:54
yoctozeptomgoddard: as mnasiadka pointed out this is felt even more due to restoration of tripleo gating - they recheck and gate like crazy today :D10:54
*** rpittau is now known as rpittau|afk10:56
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Add support for libvirt+tls  https://review.opendev.org/65044810:57
andreinmgoddard: ping11:04
mgoddardandrein: pong11:05
andreinhello  :)11:05
mgoddardhi11:06
andreini've been wrestling with my vlan over bond issue for a while now, I *almost* got it working, but I must be missing something11:06
andreinI figured out how cloud-init is supposed to set up networking using /mnt/config/openstack/latest/network_data.json11:08
mgoddardif you get it working I'd be interested to know how11:09
andreinI edited my network_data.json as described in http://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/metadata-service-network-info.html11:09
mgoddardthere is code in nova to build that file which supports bonds11:09
mgoddardah, you got it11:09
mgoddardI'm heading out for lunch now11:09
andreinah, ok, ping me when you're back11:10
*** jdwidari has joined #openstack-kolla11:12
egonzalezandrein, what is/was the issue?11:13
andreinhi egonzalez11:13
andreinegonzalez: this is my bifrost server inventory https://www.irccloud.com/pastebin/xkDHZvxd/11:14
andreinI'm trying to set up vlan 2119 over a bond between two physical NICs11:14
*** pcaruana has joined #openstack-kolla11:17
*** kplant has joined #openstack-kolla11:17
andreinon the instance, the network_data.json looks good, but on boot the interfaces aren't configured11:17
*** k_mouza has quit IRC11:17
andreinwhen I run `cloud-init devel net-convert -p network_data.json -k network_data.json -d / -D centos -O sysconfig` the interface configuration is generated properly11:18
andreinand after a `systemctl start network` everything works as expected11:18
andreinany idea why cloud-init isn't doing the right thing when the server boots?11:18
andreinthis is the cloud-init output right after boot. I'm trying to set up the bonds over enp130s0f{0,1} https://www.irccloud.com/pastebin/t7JXO4jX/11:20
egonzalezcloud init have network-config disabled? anything else after tarted Initial cloud-init job task on boot?11:23
yoctozeptoargh, kolla CI broken, again11:26
yoctozeptosensu! https://615effe7fe88720ee720-41ed0c4c4ac97898f47335a8b01b94a8.ssl.cf2.rackcdn.com/683038/2/check/kolla-build-centos-source/33c1fcc/kolla/build/000_FAILED_sensu-client.txt.gz11:26
*** gfidente has quit IRC11:28
andreinegonzalez: looks like it can find the datasource for network config, so I guess it's not disabled:11:29
andrein2019-09-19 05:56:27,824 - handlers.py[DEBUG]: finish: init-network/search-None: SUCCESS: found network data from DataSourceNone11:29
andrein2019-09-19 05:56:27,825 - stages.py[INFO]: Loaded datasource DataSourceNone - DataSourceNone11:29
yoctozeptomgoddard, mnasiadka, hrw: any sensu heroes? ^^^^11:30
mnasiadkayoctozepto, how, mgoddard: how many monitoring suites do we support?11:33
mnasiadkaMonasca, sensu, prometheus, collects and whatever there is11:34
mnasiadkacan we also support Nagios and zabbix in our Docker images collection? :)11:34
kplanttelegraf ^11:34
mnasiadkakplant: thanks for reminding - so let's say the Influx stack :)11:34
yoctozeptowe are well monitored11:34
yoctozeptobut broken now :<11:35
yoctozeptomnasiadka: what do you propose11:35
yoctozeptoboth short and longterm please11:35
hrwINFO:kolla.common.utils.sensu-client:whois requires Ruby version >= 2.4.11:35
hrwsimple11:35
mnasiadkaSince there's not a lot of us, deprecate at least half of the solutions we don't maintain and have no clue if work :)11:35
mnasiadkaSo let's start with Sensu :)11:36
mnasiadkaIf it comes to me, Prometheus and Monasca make sense, the rest is just weight we carry - unless there's somebody that wants to support that.11:36
hrwyoctozepto: I may fix it in ~1h11:38
mnasiadkaand none of these monitoring suites have a CI :)11:38
kplantmnasiadka: that makes sense this they solve different problems11:38
kplants/this/since11:38
kplantsorry.. stroke11:38
hrwyoctozepto: whois needs to be == 4.1.011:38
mnasiadkakplant: we don't need to build the images, sensualising delivers their own - why don't we ask users to use them?11:39
mnasiadkageez, word fixer in Mac enabled by itself11:39
hrwyoctozepto: as 5.0.0 got released today and wants ruby 2.411:39
mnasiadkasensu delivers their own :)11:39
kplantlol11:39
kplantsensualising11:39
hrwmore and more gems will require ruby 2.411:39
mnasiadkawhich exists only on Debian/Ubuntu?11:39
hrwwhich is not present in centos11:40
hrw711:40
egonzalezsensu was used by tripleo iirc11:40
kplantit was, i think they're moving to an rsyslog approach11:40
kplantripping out sensu and fluentd/collectd? it's been a while11:40
*** gfidente has joined #openstack-kolla11:43
yoctozeptomnasiadka: that's short term11:43
yoctozeptowhat about long term11:43
yoctozeptomaybe we should build images in two (or more) tiers11:43
yoctozeptosensu breaking whole CI is just laughable11:44
mnasiadkayoctozepto: well, maybe we should start thinking about consuming 3rd party docker images for some services like non-OpenStack projects11:44
yoctozeptomnasiadka: also acceptable11:44
mnasiadkalike in ceph-ansible case11:44
mnasiadkalet's add this to next meeting agenda11:50
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: sensu-client: install older whois gem  https://review.opendev.org/68311711:51
hrwthis should fix CI/centos11:51
mnasiadkadone11:52
openstackgerritRadosław Piliszek proposed openstack/kolla master: sensu-client: install older whois gem  https://review.opendev.org/68311711:52
mnasiadkawe love Ruby, don't we? :)11:53
yoctozeptohrw: quick edit11:53
yoctozeptomnasiadka: thx11:53
hrwmnasiadka: I do not care about Ruby.11:53
yoctozeptoI have nothing against ruby other than I don't know the ecosystem well11:53
hrwmnasiadka: for me it is just yet another build failure11:53
hrwyoctozepto: s/pip/gem/ and go ;D11:54
yoctozeptohrw: but you knew ruby 2.4 migration is happening11:54
yoctozepto;p11:54
hrwyoctozepto: already got hit by it in other of our images ;D11:55
yoctozeptohrw: that explains it11:55
hrwcentos8 has ruby 2.512:01
* hrw food12:01
*** verdurin has joined #openstack-kolla12:04
yoctozeptohrw: will probably be too new12:08
mgoddardI only went for lunch and now CI is busted?12:09
yoctozeptomgoddard: yeah, please read our little discussion12:09
mgoddardI did12:09
yoctozeptowdyt re tiered image building12:10
yoctozeptowe could align this with tables12:10
yoctozeptofrom the support matrix12:10
mgoddardcould do, it's come up before as an idea12:11
*** goldyfruit_ has quit IRC12:11
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637612:21
*** k_mouza has joined #openstack-kolla12:27
*** k_mouza has quit IRC12:31
*** eandersson has quit IRC12:31
*** priteau has joined #openstack-kolla12:36
*** cah_link has joined #openstack-kolla12:48
*** vmixor has quit IRC12:50
*** vmixor has joined #openstack-kolla12:50
*** cah_link has quit IRC12:52
andreinmgoddard: did you get a chance to review the configs i posted above?12:53
*** zzzeek has quit IRC12:56
*** cah_link has joined #openstack-kolla12:57
*** zzzeek has joined #openstack-kolla12:58
*** vmixor has quit IRC12:58
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Check for VT in nova prechecks when using KVM  https://review.opendev.org/68314713:03
mgoddardandrein: I did. I'm not sure what to say though. Have you been checking the cloud-init logs to see what it does on boot?13:05
*** hjensas has quit IRC13:13
andreinmgoddard: https://seashells.io/p/pS8KbbuB do you notice anything weird here?13:14
mgoddardandrein: IIRC "Applying network configuration from fallback" means it has not parsed your config correctly13:16
mgoddarddoesn't really say why13:16
*** goldyfruit_ has joined #openstack-kolla13:16
hrwfsck. no gem on ubuntu13:18
*** cah_link has quit IRC13:20
hrwsensu does not depend on ruby13:21
*** cah_link has joined #openstack-kolla13:27
*** pcaruana has quit IRC13:28
openstackgerritDincer Celik proposed openstack/kolla-ansible master: Fixes Octavia  https://review.opendev.org/68315913:31
hrwhave fix13:33
*** wuchunyang has joined #openstack-kolla13:34
openstackgerritDincer Celik proposed openstack/kolla master: [horizon] Enable octavia dashboard  https://review.opendev.org/68316013:37
*** vmixor has joined #openstack-kolla13:38
*** devep has joined #openstack-kolla13:38
openstackgerritMerged openstack/kolla-ansible stable/rocky: [designate] Fix admin api configuration  https://review.opendev.org/68307313:42
*** gfidente has quit IRC13:45
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: sensu-client: install older whois gem  https://review.opendev.org/68311713:45
hrwthis ver works on centos and ubuntu13:45
hrwuses gem only on rpm distros and change nothing otherwise13:46
*** mixor has joined #openstack-kolla13:52
*** Luzi has quit IRC13:53
*** vmixor has quit IRC13:53
*** pcaruana has joined #openstack-kolla13:54
*** JamesBenson has joined #openstack-kolla13:55
*** zhangfei has joined #openstack-kolla13:56
*** BjoernT has joined #openstack-kolla13:59
*** BjoernT_ has joined #openstack-kolla14:02
*** BjoernT has quit IRC14:04
*** hjensas has joined #openstack-kolla14:07
*** ivve has quit IRC14:09
*** cah_link has quit IRC14:11
*** goldyfruit___ has joined #openstack-kolla14:15
*** goldyfruit_ has quit IRC14:18
*** gfidente has joined #openstack-kolla14:23
*** wuchunyang has quit IRC14:23
*** wuchunyang has joined #openstack-kolla14:24
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Add support for libvirt+tls  https://review.opendev.org/65044814:32
*** pcaruana has quit IRC14:33
*** wuchunyang has quit IRC14:36
shubjeroI'm attempting to do a multinode kolla-ansible deployment but it keeps failing at certain mariadb stages. I noticed that the mariadb containers are restarting constantly on all 3 control nodes. I'm trying to deploy rocky on Ubuntu 18.04. It's not entirely clear why the mariadb containers keep cycling. Any suggestions?14:38
*** wuchunyang has joined #openstack-kolla14:40
kplantdocker logs -f mariadb14:41
kplantor: tail -f /var/log/kolla/mariadb/mariadb.log14:41
*** wuchunyang has quit IRC14:45
*** iclon has quit IRC14:45
shubjerokplant: well its hard to decipher why considering all 3 mariadb containers keep cycling. I've stopped mariadb on control2 & 3.. going to see what 1 complains about14:48
shubjerokplant: https://paste.ubuntu.com/p/bTmsZQsc33/14:49
kplantlooks like a network issue14:50
kplanthave you confirmed reachability between the three nodes across 172.25.10.x?14:51
shubjerokplant: yeah, not sure why though.. its just 3 instances on the same network..14:51
shubjeroyep14:51
kplantiptables?14:54
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for self-signed SSL certificates  https://review.opendev.org/66451714:55
shubjerokplant: nothing in iptables other than what was put there by docker/kolla-ansible14:57
shubjerokplant: i can telnet from each control node to the mariadb port's on all 314:58
openstackgerritMark Goddard proposed openstack/kayobe stable/rocky: Add alternative tenks deploy and teardown entrypoints.  https://review.opendev.org/68317614:59
*** iclon has joined #openstack-kolla14:59
*** skramaja has quit IRC15:03
*** wuchunyang has joined #openstack-kolla15:06
*** egonzalez has quit IRC15:09
*** wuchunyang has quit IRC15:12
*** hjensas has quit IRC15:24
*** k_mouza has joined #openstack-kolla15:24
*** k_mouza has quit IRC15:28
*** k_mouza has joined #openstack-kolla15:29
zbr|ruckcan someone look at https://bugs.launchpad.net/tripleo/+bug/1844697 -- apparently there is no retry on push and this causes failures15:32
openstackLaunchpad bug 1844697 in tripleo "[koilla] periodic: container build job can fail during push" [Undecided,New]15:32
*** gfidente has quit IRC15:35
mgoddardhi zbr|ruck, I think we do have a retry mechanism15:36
mgoddardzbr|ruck: kolla-build --retries15:37
zbr|ruckhmm... now i need to find how this is called and how do retries apply because clearly i did not see it retrying, even once15:39
zbr|ruckmy impression is that retries apply only to building but not to pushing15:39
mgoddardlogs look like it is not retrying15:39
*** zhangfei has quit IRC15:39
zbr|ruckmgoddard: at least it should easy to fix, reusing the same parameter for pushing too.15:40
mgoddardzbr|ruck: looks like it should work for pushing, but doesn't15:42
mgoddard2019-09-19 01:13:00 | INFO:kolla.common.utils:Attempt number: 2 to run task: PushTask(glance-api)15:42
mgoddard2019-09-19 01:13:00 | INFO:kolla.common.utils.glance-api:Trying to push the image15:42
zbr|ruckin fact I see retry attempts up to 4 but they all happen during 2s interval, but I do not see additional errors on them.15:44
mgoddardzbr|ruck: which branch is this?15:44
zbr|ruckmaster15:44
mgoddardgiven the lack of an error, I think the push must have succeeded, but something is wrong with the state that prevents it from setting self.success = True15:46
zbr|ruckin fact, nope,  read it wrong: it did wait for 40mins!15:46
zbr|rucki updated the msg to include the retries in it (the log is quite long)15:47
zbr|ruckso yes, there are retries, I can see a 40s (not min) delay between 2nd and 3rd attempt.15:48
zbr|ruckanyway, i would expect to see 3-4 full timeouts, not only one.15:48
mgoddardso sounds to me like first one errored, second one finished successfully15:50
zbr|ruckprobably but the status remained failed, and the proof is the other msgs.  still my first move would be to include exception in "Unknown error when pushing".15:51
mgoddardzbr|ruck: something like this? http://paste.openstack.org/show/777708/15:51
zbr|rucknot displaying raised exception makes harder to debug the behavior. any reason for censoring real exception?15:51
mgoddardno reason15:52
mgoddardhold up, we do include the exception15:53
mgoddardlog.exception()15:53
mgoddardReadTimeoutError15:53
mgoddardtimeout interacting with docker via unix socket15:53
zbr|ruckwhy we see only one?15:53
zbr|ruckyou can easilly see that there are 3 other retries after: Attempt number: 2/3/4 to run task: PushTask(glance-api)15:54
zbr|ruckand no error between them, is like the status remained failed.15:54
zbr|ruckalso "Pushed successfully" is never printed for glande-api15:56
zbr|ruckthis makes me believe, it was not pushed.15:56
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Fix Octavia to use keystone_admin_project variable  https://review.opendev.org/68315915:58
*** mgoddard has quit IRC15:58
*** mgoddard has joined #openstack-kolla15:59
mgoddardzbr|ruck: I think the image was pushed, but the state never got reset from connection_error, so the finally block always thinks it has failed16:00
*** openstackgerrit has quit IRC16:06
zbr|ruckmgoddard: indeed, it seems that we need to reset image.status before calling push on it, right?16:10
mgoddardzbr|ruck: just trying out a fix16:10
zbr|ruckmgoddard: thanks! just add me as reviewer.16:11
andreinmgoddard: I think I've got bonding + vlans working16:16
mgoddardandrein: nice work16:16
andreinbut... took some manual input16:16
andreinso, looks like the problem is in /etc/cloud/cloud.cfg.d/91-dib-cloud-init-datasources.cfg16:17
andreinmy image has `datasource_list: [  None, None ]`16:17
andreinlooks like dib put it there, and I'm not sure why16:17
andreinthat effectively disables the configdrive support in cloud-init (though I'm not sure how on earth the VLAN without bonding worked previously)16:18
andreinare you familiar with how bifrost builds the deployment image?16:19
*** openstackgerrit has joined #openstack-kolla16:22
openstackgerritMark Goddard proposed openstack/kolla master: Fix retries when pushing images  https://review.opendev.org/68320016:22
mgoddardandrein: aha, try dib_init_element: cloud-init16:24
*** k_mouza has quit IRC16:25
andreinmgoddard: since I'm rebuilding this, if I want IPMI SOL over ttyS1 instead of ttyS0, how would I do that?16:26
shubjeroDoing a multinode install (getting further!) but coming across a problem with the 'running placement bootstrap container' task. 404 Client Error: Not Found (\"manifest for kolla/ubuntu-source-placement-api:rocky not found: manifest unknown: manifest unknown\")\\n'"}16:28
shubjerohttps://hub.docker.com/r/kolla/ubuntu-source-placement-api/tags only has master and stein tags. Is the rocky image supposed to be missing?16:30
*** k_mouza has joined #openstack-kolla16:34
*** itlinux has joined #openstack-kolla16:35
yoctozeptoshubjero: placement split happened in stein16:36
yoctozeptouse rocky kolla-ansible for rocky images16:36
mgoddardandrein: try extra_kernel_options16:37
yoctozeptoshubjero: or just use stein16:38
yoctozeptowhy use rocky for new deployments :O16:38
shubjeroyoctozepto: oh, maybe i downloaded the wrong kolla-ansible branch, crap. Anyways I am using rocky because our prod environment is on rocky and we want to somehow migrate to kolla-ansible on rocky first..16:39
yoctozeptoshubjero: roger that16:39
shubjeroyoctozepto: context: our prod cloud is old school ubuntu packages16:39
yoctozeptoah16:39
shubjerojust looking to modernize.. so deploying multinode kolla in the lab is just trying to get our feet wet16:40
yoctozeptostep in the right direction16:40
*** dciabrin has quit IRC16:41
*** igordc has joined #openstack-kolla16:44
*** jdwidari has quit IRC16:51
*** devep has quit IRC16:58
*** k_mouza has quit IRC16:59
*** pcaruana has joined #openstack-kolla17:01
*** hjensas has joined #openstack-kolla17:04
shubjeroyoctozepto: If I do 'pip install kolla-ansible' as per ( https://docs.openstack.org/project-deploy-guide/kolla-ansible/rocky/quickstart.html ), is this getting me the correct version of kolla-ansible to deploy rocky?17:04
yoctozeptoshubjero: good point, we should fix the docs - pip always installs the latest obviously :-)17:07
*** zbr|ruck is now known as zbr17:07
yoctozeptoso it's stein now17:07
yoctozeptoand will soon be train ;D17:07
yoctozeptoI seriously recommend you use the version from git, branch stable/rocky17:08
yoctozeptoit gets latest fixes (unlike release)17:08
yoctozeptoand for sure will be for rocky ;-)17:08
*** pcaruana has quit IRC17:14
yoctozeptomgoddard: have you considered whether one of those old changes is actually interesting?17:15
yoctozeptoshubjero: reported for you: https://bugs.launchpad.net/kolla-ansible/+bug/184471017:18
openstackLaunchpad bug 1844710 in kolla-ansible "deploy docs are misleading wrt version of kolla-ansible being installed" [High,Triaged]17:18
shubjeroyoctozepto: nice, thanks17:19
*** gmann_afk is now known as gmann17:21
andreinwhen I see 'tc@box:~$', does it mean my IPA ramdisk is busted?17:23
andreinoh, it rebooted, i think I broke the IPA output somehow17:24
*** pcaruana has joined #openstack-kolla17:25
andreinif I remove the deployment_image, does it also rebuild the ipa image?17:25
yoctozeptoandrein: for ironic and bifrost help it might be wise to also hit the #openstack-ironic channel17:31
yoctozeptoI cannot help you and mgoddard seems afk17:32
andreinI’m just heading home atm, will hit the ironic channel in an hour or so17:33
openstackgerritMerged openstack/kolla master: sensu-client: install older whois gem  https://review.opendev.org/68311717:40
yoctozeptoCI unbroken ^17:43
mnasiadkaAnd let it stay that way at least until tomorrow :)17:44
yoctozeptohrw: I see you are quick to recheck, just like me17:45
hrwyoctozepto: one day I will write userscript to add 'recheck' button next to 'Reply' one17:49
* hrw off17:49
*** pcaruana has quit IRC17:55
*** priteau has quit IRC18:07
yoctozeptohrw: hah18:07
*** xelaot has joined #openstack-kolla18:16
*** igordc has quit IRC18:20
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Add bifrost Ubuntu job  https://review.opendev.org/68229218:27
*** priteau has joined #openstack-kolla18:27
openstackgerritMerged openstack/kolla stable/pike: base: fix OracleLinux build  https://review.opendev.org/68306418:29
*** priteau has quit IRC18:29
yoctozeptoyay, pike unbroken, finally18:31
*** radeks has quit IRC18:51
*** xelaot has quit IRC18:58
yoctozeptohttps://imgur.com/a/OFz2rpX19:00
yoctozepto;D19:00
openstackgerritMerged openstack/kolla-ansible stable/stein: Fix for haproxy precheck failing on CentOS running non-root  https://review.opendev.org/68295719:08
openstackgerritMerged openstack/kolla-ansible master: Fix for haproxy precheck failing on CentOS running non-root  https://review.opendev.org/68306819:08
openstackgerritMerged openstack/kolla-ansible stable/stein: Fix prometheus-alertmanager cluster bug  https://review.opendev.org/68288819:08
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for self-signed SSL certificates  https://review.opendev.org/66451719:21
*** dougsz has joined #openstack-kolla19:40
*** mixor has quit IRC19:47
*** pcaruana has joined #openstack-kolla19:51
*** kplant has quit IRC19:53
*** dougsz has quit IRC20:02
*** mixor has joined #openstack-kolla20:22
*** mixor has quit IRC20:31
*** pcaruana has quit IRC20:31
andreinmgoddard: It's ALIVE20:31
*** igordc has joined #openstack-kolla20:33
* andrein does the victory dance20:34
*** scottsol has joined #openstack-kolla20:40
openstackgerritMerged openstack/kolla master: ppc64le: add UNBUILDABLE_IMAGES entries  https://review.opendev.org/68206821:04
openstackgerritMerged openstack/kolla master: Add kayobe to deliverables.yaml  https://review.opendev.org/67451021:16
openstackgerritMerged openstack/kolla-ansible master: Update "openstack_release" variable to static brach name  https://review.opendev.org/68163321:21
openstackgerritMerged openstack/kolla master: Add package systemd-sysv to bifrost-deploy image  https://review.opendev.org/67993621:21
*** JamesBenson has quit IRC21:29
*** benj_ has quit IRC21:31
*** benj has joined #openstack-kolla21:31
*** JamesBenson has joined #openstack-kolla21:31
*** benj is now known as Guest2963421:32
mlozahello, I enabled mistral in stable/stein env and it deploys fine but getting error when I access it in the dashboard. Here is the log http://paste.openstack.org/show/777953/21:35
mlozaI get the same error in the CLI21:35
mlozaThe mistral endpoint is missing in the API Access21:36
*** JamesBenson has quit IRC21:36
*** scottsol has quit IRC21:36
*** mixor has joined #openstack-kolla21:37
*** luksky has quit IRC21:41
*** scottsol has joined #openstack-kolla21:42
openstackgerritMerged openstack/kolla-ansible master: Fix Octavia to use keystone_admin_project variable  https://review.opendev.org/68315921:43
*** scottsol has quit IRC21:58
*** BjoernT_ has quit IRC22:00
openstackgerritViktor Michalek proposed openstack/kolla master: Fix images based on Debian Buster Stein packages  https://review.opendev.org/68044722:02
openstackgerritMerged openstack/kolla master: Add volume-related packages to zun-compute image  https://review.opendev.org/68243722:09
openstackgerritViktor Michalek proposed openstack/kolla master: Fix images based on Debian Buster Stein packages  https://review.opendev.org/68044722:11
*** mixor has quit IRC22:19
*** igordc has quit IRC22:44
*** tonythomas has quit IRC22:45
*** signed8b_ has joined #openstack-kolla22:48
*** signed8bit has quit IRC22:50
*** Tengu has quit IRC22:51
*** goldyfruit___ has quit IRC23:00
*** igordc has joined #openstack-kolla23:01
*** Tengu has joined #openstack-kolla23:03
*** wuchunyang has joined #openstack-kolla23:47
*** JamesBenson has joined #openstack-kolla23:47
*** JamesBenson has quit IRC23:51
*** signed8b_ has quit IRC23:56
*** signed8bit has joined #openstack-kolla23:56
« Wednesday, 2019-09-18 Index Friday, 2019-09-20 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!