| *** Sravan has quit IRC | 00:06 | |
| *** lemko has quit IRC | 00:35 | |
| *** zhanglong has joined #openstack-kolla | 00:39 | |
| *** gregoryo has joined #openstack-kolla | 00:52 | |
| *** zhanglong has quit IRC | 01:16 | |
| *** hongbin has joined #openstack-kolla | 01:32 | |
| *** Sravan has joined #openstack-kolla | 01:50 | |
| *** Sravan has quit IRC | 01:55 | |
| openstackgerrit | pangliye proposed openstack/kolla-ansible master: add kapacitor https://review.opendev.org/665841 | 01:56 |
|---|---|---|
| *** goldyfruit has quit IRC | 01:57 | |
| *** zhanglong has joined #openstack-kolla | 02:08 | |
| openstackgerrit | inspurericzhang proposed openstack/kolla-ansible master: Modify influxdb maximum data receipt by http https://review.opendev.org/665845 | 02:16 |
| openstackgerrit | pangliye proposed openstack/kolla master: add kapacitor https://review.opendev.org/665846 | 02:29 |
| openstackgerrit | pangliye proposed openstack/kolla-ansible master: add kapacitor https://review.opendev.org/665841 | 02:31 |
| *** mgoddard has quit IRC | 02:48 | |
| *** whoami-rajat has joined #openstack-kolla | 03:01 | |
| *** mgoddard has joined #openstack-kolla | 03:01 | |
| *** schwicht has quit IRC | 03:11 | |
| *** hongbin has quit IRC | 03:47 | |
| *** Sravan has joined #openstack-kolla | 03:49 | |
| *** threestrands has joined #openstack-kolla | 04:05 | |
| *** Sravan has quit IRC | 04:06 | |
| *** Sravan has joined #openstack-kolla | 04:25 | |
| *** Sravan has quit IRC | 04:29 | |
| *** JangwonLee has quit IRC | 04:32 | |
| *** JangwonLee has joined #openstack-kolla | 04:32 | |
| *** janki has joined #openstack-kolla | 04:35 | |
| *** pcaruana has joined #openstack-kolla | 04:56 | |
| *** zzzeek has quit IRC | 05:07 | |
| *** zzzeek has joined #openstack-kolla | 05:08 | |
| *** ivve has quit IRC | 05:29 | |
| *** rpittau|afk has quit IRC | 05:36 | |
| *** hogepodge has quit IRC | 05:36 | |
| *** rpittau|afk has joined #openstack-kolla | 05:37 | |
| *** hogepodge has joined #openstack-kolla | 05:38 | |
| *** Luzi has joined #openstack-kolla | 05:44 | |
| *** skramaja has joined #openstack-kolla | 05:55 | |
| *** cah_link has joined #openstack-kolla | 06:07 | |
| *** janki has quit IRC | 06:15 | |
| *** zhanglong has quit IRC | 06:20 | |
| *** cah_link has quit IRC | 06:26 | |
| *** rpittau|afk is now known as rpittau | 06:28 | |
| *** cah_link has joined #openstack-kolla | 06:29 | |
| *** igordc has quit IRC | 06:32 | |
| gregoryo | Wheeee. Redeployed from scratch, and I timed this one - 45 minutes to deploy a whole OpenStack to three nodes, from bare operating system (: | 06:32 |
| *** ivve has joined #openstack-kolla | 06:40 | |
| *** zhanglong has joined #openstack-kolla | 06:42 | |
| *** lemko has joined #openstack-kolla | 06:49 | |
| *** dpawlik has joined #openstack-kolla | 06:51 | |
| *** zhanglong has quit IRC | 06:52 | |
| yoctozepto | gregoryo: but you surely mean the whole base, not the whole whole | 07:01 |
| gregoryo | Yeah I suppose so, just core stuff - keystone, nova, neutron, glance etc. | 07:02 |
| gregoryo | Although chatting with a colleague has got me interested in Qinling now! Apparently they're using AWS Lambda so we might be able to offer up something similar. So many optional features (: | 07:04 |
| yoctozepto | gregoryo: yeah, OpenStack is feature-full | 07:06 |
| *** luksky has joined #openstack-kolla | 07:08 | |
| *** vishwanathj has joined #openstack-kolla | 07:13 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla master: Add a variable to configure repos disabled in base CentOS image https://review.opendev.org/665657 | 07:14 |
| *** vishwanathj has quit IRC | 07:15 | |
| hrw | lo | 07:32 |
| yoctozepto | o/ | 07:35 |
| *** owalsh has quit IRC | 07:35 | |
| *** owalsh has joined #openstack-kolla | 07:35 | |
| *** threestrands has quit IRC | 07:36 | |
| *** nathharp has joined #openstack-kolla | 07:37 | |
| *** nathharp has quit IRC | 07:45 | |
| *** nathharp has joined #openstack-kolla | 07:46 | |
| *** nathharp has quit IRC | 07:46 | |
| *** Wasaac has joined #openstack-kolla | 07:48 | |
| *** nathharp has joined #openstack-kolla | 07:48 | |
| *** ktibi has joined #openstack-kolla | 08:01 | |
| *** spsurya has joined #openstack-kolla | 08:01 | |
| *** dougsz has joined #openstack-kolla | 08:01 | |
| *** gregoryo has quit IRC | 08:02 | |
| *** yan0s has joined #openstack-kolla | 08:05 | |
| hrw | and another project patched... | 08:07 |
| hrw | hm. 8 patches in my queue | 08:07 |
| yoctozepto | queue.empty() | 08:09 |
| *** Wasaac has quit IRC | 08:09 | |
| *** Wasaac has joined #openstack-kolla | 08:09 | |
| hrw | yoctozepto: you suggest abandoning? | 08:12 |
| yoctozepto | hrw: no, I thought you have this method implemented better and it makes you actually do the work | 08:13 |
| hrw | yoctozepto: my review queue is usually too long | 08:14 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Add sha256 in ceph fetch keys https://review.opendev.org/655623 | 08:21 |
| yoctozepto | https://bugs.launchpad.net/kolla-ansible/+bug/1832989 | 08:21 |
| openstack | Launchpad bug 1832989 in kolla-ansible "CI: random error in: ceph : Enabling all new Luminous-only functionality (EPERM: not supported)" [Undecided,In progress] - Assigned to Radosław Piliszek (yoctozepto) | 08:21 |
| yoctozepto | so we have different ceph versions supported per distro | 08:22 |
| yoctozepto | and outdated upgrade procedure | 08:22 |
| hrw | yoctozepto: yes, we do | 08:22 |
| yoctozepto | hrw: and that's the plan? :P | 08:23 |
| hrw | yoctozepto: "git rm -r docker/ceph" is a plan. with migration to ceph-ansible | 08:25 |
| yoctozepto | +1 | 08:25 |
| hrw | yoctozepto: debian has luminous, ubuntu has luminous/mimic, centos has nautilus | 08:26 |
| mgoddard | rm_work: oh right, hadn't realised you were at Verizon. Look forward to working with you again :) | 08:26 |
| hrw | yoctozepto: sorry. ubuntu has L/M/N now | 08:26 |
| hrw | http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/c/ceph/ | 08:26 |
| yoctozepto | odd, then we should be getting nautilus, we only have: | 08:27 |
| yoctozepto | Package: ceph* *cephfs* librbd* *rados* python-rbd librgw* | 08:28 |
| yoctozepto | Pin: version 13.* | 08:28 |
| yoctozepto | Pin-Priority: -1 | 08:28 |
| hrw | I know. I wrote that part ;D | 08:28 |
| yoctozepto | so 14 should come | 08:28 |
| hrw | we did not switched UCA to train probably | 08:29 |
| rm_work | mgoddard: :) do you think you'll be at Shanghai> | 08:30 |
| rm_work | ? | 08:30 |
| mgoddard | rm_work: I'm not planning on going currently - I'm expecting a baby in 6 weeks | 08:31 |
| rm_work | Yep, just shuffled around a bit, and will be glad to get our new deployments onto K-A again :D | 08:31 |
| rm_work | ahh! congrats :) | 08:31 |
| mgoddard | thanks :) | 08:31 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: ubuntu: use UCA/train https://review.opendev.org/665908 | 08:31 |
| mgoddard | rm_work: are you going? | 08:31 |
| rm_work | I think so | 08:32 |
| hrw | yoctozepto: ^^ | 08:32 |
| yoctozepto | hrw: or they left it in proposed | 08:32 |
| mgoddard | rm_work: Great. Will be looking out for people to represent kolla | 08:32 |
| yoctozepto | mgoddard: congrats! | 08:32 |
| mgoddard | rm_work: although you're Octavia PTL these days right? Perhaps you'll be busy :) | 08:32 |
| rm_work | yeah, that's part of it T_T | 08:33 |
| mgoddard | yoctozepto: thanks! | 08:33 |
| hrw | yoctozepto: looks like | 08:33 |
| yoctozepto | hrw: sooner we abandon ceph, the better | 08:33 |
| yoctozepto | it's messy to support | 08:33 |
| mnasiadka | morning | 08:33 |
| hrw | yoctozepto: U/V cycle iirc | 08:33 |
| hrw | mnasiadka: elo | 08:33 |
| mgoddard | morning mnasiadka | 08:33 |
| mnasiadka | oh hi mgoddard | 08:34 |
| yoctozepto | mnasiadka: morning | 08:34 |
| mgoddard | yoctozepto: we can never fully remove ceph - we will always need to integrate with external ceph | 08:34 |
| rm_work | staying really focused on Octavia, and Jim was able and willing to take up all the stuff i dropped when i moved and had my priorities jumble :D I'm really glad he's been able to tackle it | 08:34 |
| yoctozepto | mgoddard: external is fine | 08:34 |
| mnasiadka | mgoddard: congrats ;) | 08:34 |
| yoctozepto | using it myself :D | 08:34 |
| mgoddard | thanks mnasiadka :) | 08:34 |
| mgoddard | rm_work: our octavia integration is currently not great - quite a few undocumented steps required to get it running. Just putting that out there :p | 08:35 |
| hrw | yoctozepto: anyway we will have to deal with 'will ceph-O libs connecto to ceph-R osd' etc | 08:35 |
| yoctozepto | hrw: yeah, right worrying about it | 08:36 |
| hrw | mnasiadka: can you spend some time on cleaning queue of backports and priority changes? | 08:36 |
| rm_work | mgoddard: yeah i will end up *needing* to fix that, just in due course (since we DO use octavia here) | 08:36 |
| mgoddard | rm_work: that will be nice to see | 08:36 |
| rm_work | do you know offhand what the biggest ones are? | 08:36 |
| rm_work | I am assuming the HM network | 08:37 |
| mnasiadka | hrw: sure sir, running to Gerrit | 08:37 |
| mgoddard | rm_work: I know people who're using it who I could put you in touch with if that helps | 08:37 |
| yoctozepto | mnasiadka: that's the spirit | 08:37 |
| rm_work | yeah feel free to send people my wau | 08:37 |
| rm_work | *my way | 08:37 |
| mgoddard | rm_work: certificate generation is one, network config, possibly some keystone bits | 08:38 |
| rm_work | i thought we had cert-gen nailed down :( | 08:38 |
| rm_work | I guess not for the production case | 08:38 |
| mgoddard | rm_work: stackedsax and his colleagues might help | 08:38 |
| rm_work | the assumption is for production you will have some custom certs you need to use | 08:38 |
| rm_work | but I guess that can be handled the same as other custom file overrides | 08:38 |
| mgoddard | rm_work: yeah, they had to bake them into the amphora image | 08:39 |
| *** priteau has joined #openstack-kolla | 08:39 | |
| rm_work | errrrrr | 08:39 |
| rm_work | that's not... | 08:39 |
| rm_work | that's not good | 08:39 |
| mgoddard | well I'm sure stackedsax would be interested to hear why | 08:39 |
| rm_work | ok yeah, i should definitely talk to them at some point to work out what the real issue is, lol | 08:39 |
| stackedsax | yeah, we were about to start in on that work this week. | 08:39 |
| rm_work | why does your nickname sound familiar... | 08:40 |
| mgoddard | stackedsax: we're talking about octavia | 08:40 |
| rm_work | did we work together before and I'm just blanking? | 08:40 |
| mgoddard | stackedsax: even though it sounds similar to the internal TLS stuff :) | 08:40 |
| stackedsax | rm_work: Rackspace? Yahoo!? | 08:40 |
| rm_work | well I was at RAX | 08:40 |
| rm_work | and am now at Yahoo | 08:40 |
| rm_work | lol | 08:40 |
| rm_work | prolly RAX | 08:41 |
| rm_work | I was on the LBaaS team there | 08:41 |
| stackedsax | oh, that's totally it | 08:41 |
| hrw | mnasiadka: thank you | 08:41 |
| rm_work | k yeah | 08:41 |
| stackedsax | I ran Monitoring, Metrics, that stuff for a while in SF | 08:42 |
| rm_work | ahhh yeah | 08:42 |
| rm_work | I worked a bit with monitoring on BlueFlood | 08:42 |
| *** hamzaachi has joined #openstack-kolla | 08:42 | |
| rm_work | when we were integrating our dashboard | 08:42 |
| *** hamzaachi_ has joined #openstack-kolla | 08:42 | |
| stackedsax | I think we connected on some Metrics initiative to get more metrics out of LBaaS | 08:42 |
| rm_work | yep | 08:42 |
| stackedsax | hi! | 08:42 |
| rm_work | whereabouts are you now? | 08:43 |
| *** hamzaachi has quit IRC | 08:43 | |
| stackedsax | still on the West Coast. you? | 08:43 |
| rm_work | yeah, mostly | 08:43 |
| rm_work | i mean like, not RAX tho right? | 08:43 |
| stackedsax | god no | 08:43 |
| rm_work | lol | 08:43 |
| rm_work | i mean they don't run openstack anymore AFAIU so | 08:43 |
| rm_work | other than the remains of their public cloud that still fester | 08:43 |
| stackedsax | yeah, sad, really | 08:44 |
| stackedsax | but if you're west coast, perhaps we can connect tomorrow at a more thinking-appropriate time of day | 08:44 |
| *** gfidente has joined #openstack-kolla | 08:45 | |
| rm_work | eh, this is when I work mostly :D | 08:45 |
| rm_work | midnight-5am PST | 08:45 |
| stackedsax | 3rd shift? or you just like it? | 08:45 |
| rm_work | i'm usually on around 2-3pm PST tho too | 08:45 |
| rm_work | it's just when my body wants me to be awake | 08:46 |
| openstackgerrit | Merged openstack/kolla stable/rocky: Fix lower-constraints test fail due to failing dep build https://review.opendev.org/665427 | 08:46 |
| openstackgerrit | Merged openstack/kolla master: Move to releases.openstack.org https://review.opendev.org/664757 | 08:46 |
| * rm_work shrugs | 08:46 | |
| rm_work | and it helps me sync with folks in europe/asia | 08:46 |
| rm_work | yeah poke at me tomorrow tho if that works for you | 08:47 |
| stackedsax | got it. yeah, I made a note for 2-3pm to connect | 08:47 |
| stackedsax | will ping you then | 08:47 |
| stackedsax | thanks! | 08:47 |
| hrw | ok. CentOS/arm64 CI needs to change nodepool_mirror_host to nodepool_mirror_centos_url to have centos|altarch and to have epel working | 08:52 |
| hrw | but that has to wait for 2h as I go out | 08:52 |
| *** kevinluuuuu has joined #openstack-kolla | 08:54 | |
| hrw | or not ;D | 08:55 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Add CentOS/source aarch64 experimental job https://review.opendev.org/662456 | 08:56 |
| mnasiadka | ok, I think I went through all backports and priority ones | 08:56 |
| mnasiadka | Will look into the rest later | 08:56 |
| hrw | mnasiadka: thanks | 08:57 |
| *** k_mouza has joined #openstack-kolla | 08:59 | |
| * hrw out | 08:59 | |
| yoctozepto | mnasiadka: may you also take a look at some "lower priority" stuff of mine? I have a few half-approved changes that want to be backported too | 09:02 |
| *** sshnaidm|afk is now known as sshnaidm | 09:04 | |
| yoctozepto | https://review.opendev.org/#/q/owner:%22Rados%25C5%2582aw+Piliszek+%253Cradoslaw.piliszek%2540gmail.com%253E%22++change: | 09:04 |
| yoctozepto | I could not filter it on workflow nope and review yup | 09:04 |
| yoctozepto | but these are the ones I mean | 09:04 |
| yoctozepto | https://review.opendev.org/#/q/owner:%22Rados%25C5%2582aw+Piliszek+%253Cradoslaw.piliszek%2540gmail.com%253E%22++++status:pending+status:reviewed | 09:06 |
| yoctozepto | this is closer | 09:06 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: Add custom filters for checking services https://review.opendev.org/665922 | 09:29 |
| gfidente | guys the other day someone was asking why we didn't have ganesha in the centos/sig repo for ceph nautilus | 09:36 |
| gfidente | I just added it into nautilus-testing | 09:36 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Make Ceph upgrade check Ceph release to avoid EPERM https://review.opendev.org/665927 | 09:36 |
| openstackgerrit | pangliye proposed openstack/kolla master: add kapacitor https://review.opendev.org/665846 | 09:37 |
| gfidente | not sure why we swiched to the ceph repos though? | 09:38 |
| gfidente | was it just because ganesha was missing? or some other incompat? | 09:38 |
| yoctozepto | gfidente: ganesha | 09:38 |
| gfidente | yoctozepto hi, ok great so it is tagged into -testing now | 09:38 |
| gfidente | are you guys able to check if it everything works as intended? | 09:39 |
| yoctozepto | gfidente: I can put it on my todo, yeah | 09:40 |
| gfidente | yoctozepto wow that'd be great thanks! | 09:40 |
| openstackgerrit | pangliye proposed openstack/kolla-ansible master: add kapacitor https://review.opendev.org/665841 | 09:47 |
| openstackgerrit | Merged openstack/kolla master: base: do not use non-free repositories for Ubuntu https://review.opendev.org/665724 | 09:50 |
| openstackgerrit | Merged openstack/kolla stable/stein: Bump service versions for Stein https://review.opendev.org/660165 | 09:50 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Simplify handler conditionals https://review.opendev.org/647699 | 09:51 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Simplify handler conditionals https://review.opendev.org/647699 | 09:52 |
| openstackgerrit | Merged openstack/kolla stable/stein: Debian/Ubuntu: more python3 packages for binary https://review.opendev.org/664012 | 09:57 |
| openstackgerrit | Merged openstack/kolla master: Fix ownership of ironic inspector DHCP hostsdir https://review.opendev.org/664011 | 09:57 |
| openstackgerrit | Merged openstack/kolla master: Remove nova-consoleauth https://review.opendev.org/661251 | 09:58 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Stop duplicating Nova cells https://review.opendev.org/665618 | 09:59 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Make "Check if firewalld is installed" task quieter https://review.opendev.org/665539 | 10:00 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Fix docs - do not mention the old, deprecated CLI clients https://review.opendev.org/665522 | 10:00 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Fix docs - simply run the script instead of sourcing it https://review.opendev.org/665521 | 10:00 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Make bootstrap more idempotent https://review.opendev.org/665523 | 10:01 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Fix action_plugins python3 compatibility https://review.opendev.org/665193 | 10:01 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Configure coordination in default for cinder when redis is enabled https://review.opendev.org/665549 | 10:01 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Stop duplicating Nova cells https://review.opendev.org/665212 | 10:01 |
| openstackgerrit | Merged openstack/kolla-ansible stable/queens: Stop duplicating Nova cells https://review.opendev.org/665214 | 10:01 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/ocata: Disable publishing images on Ocata https://review.opendev.org/665742 | 10:10 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/ocata: Mark kolla-build-*-source jobs non-voting https://review.opendev.org/665935 | 10:10 |
| openstackgerrit | Zijian Guo proposed openstack/kolla-ansible master: Fix the redis_connection_string for osprofiler https://review.opendev.org/665937 | 10:13 |
| openstackgerrit | Zijian Guo proposed openstack/kolla-ansible master: Fix the redis_connection_string for osprofiler https://review.opendev.org/665937 | 10:17 |
| *** ktibi has quit IRC | 10:18 | |
| *** k_mouza has quit IRC | 10:39 | |
| nathharp | Morning all - I got hit by https://bugs.launchpad.net/kolla/+bug/1823818, and didn't catch it before it exhausted the RAM on our controllers | 10:39 |
| openstack | Launchpad bug 1823818 in kolla "Memory leak in some neutron agents" [Undecided,Confirmed] | 10:39 |
| nathharp | since restarting the affected neutron containers, we're now having unreliable network connections to instances | 10:40 |
| nathharp | any ideas? | 10:40 |
| *** schwicht has joined #openstack-kolla | 10:45 | |
| hrw | gfidente: thanks | 10:54 |
| gfidente | hrw hi I am trying a scratch build of 14.2.1 as well and if it passes I'll tag -testing that one as well | 10:54 |
| hrw | yoctozepto: https://buildlogs.centos.org/centos/7/storage/x86_64/ceph-nautilus/ has it | 10:54 |
| gfidente | hrw we're not the most proactive guys on those repos I know but we also consume them for tripleo/rdo and act mostly driven by necessitty | 10:55 |
| hrw | gfidente: I know. dealt with storage sig in past | 10:55 |
| hrw | gfidente: ceph-nfs/arm64 image built fine with storage ceph-nautilus + nfs-ganesha-ceph from buildlogs | 11:08 |
| *** kplant has joined #openstack-kolla | 11:14 | |
| *** hamzaachi_ has quit IRC | 11:15 | |
| hrw | gfidente: so once it migrate to mirrors.centos.org we may switch back | 11:19 |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Ingest non-standard Monasca logs https://review.opendev.org/665950 | 11:21 |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Increase log coverage for Monasca https://review.opendev.org/665951 | 11:21 |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Don't drop unmatched Kolla service logs https://review.opendev.org/665952 | 11:21 |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Format internal Fluentd logs https://review.opendev.org/665953 | 11:21 |
| *** mjturek has quit IRC | 11:23 | |
| hrw | dougsz: are there plans to update components in monasca-grafana? | 11:23 |
| hrw | dougsz: INFO:kolla.common.utils.monasca-grafana:.[91mnpm WARN deprecated minimatch@0.3.0: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue | 11:23 |
| hrw | dougsz: etc | 11:23 |
| hrw | INFO:kolla.common.utils.monasca-grafana:.[91mnpm WARN deprecated samsam@1.1.3: This package has been deprecated in favour of @sinonjs/samsam | 11:23 |
| *** mjturek has joined #openstack-kolla | 11:23 | |
| dougsz | hrw: thanks | 11:23 |
| dougsz | Wasaac is working on the rebase at the moment and might be able to fix a few of those warnings | 11:24 |
| hrw | dougsz: thx | 11:24 |
| hrw | dougsz: just asking as monasca is not buildable on arm64 anyway | 11:24 |
| dougsz | To be honest, it's best kept behind a firewall regardless given it's an old version. We should probably put a warning about that. | 11:25 |
| dougsz | hrw: yeah, longer term, the fork should go away (hopefully) | 11:25 |
| hrw | dougsz: and maybe one day phantomjs will land on arm64 | 11:25 |
| dougsz | hmm, so you can't build regular Grafana for arm64? | 11:26 |
| *** cah_link1 has joined #openstack-kolla | 11:29 | |
| *** cah_link has quit IRC | 11:30 | |
| *** luksky has quit IRC | 11:30 | |
| hrw | dougsz: grafana is in package and we use grafana/arm64 in production | 11:31 |
| dougsz | cool | 11:31 |
| *** cah_link1 has quit IRC | 11:31 | |
| hrw | dougsz: monasca wants libfdkafka in version which does not exist in repos so no confluent-kafka python module so no monasca | 11:32 |
| *** cah_link has joined #openstack-kolla | 11:32 | |
| hrw | dougsz: https://review.opendev.org/665662 just make it fail faster so I do not have to check why monasca-* failed | 11:32 |
| mnasiadka | everytime I read about Monasca on this channel - I'm starting to believe it's some sort of a black unicorn on broken wheels :) | 11:33 |
| mnasiadka | hi dougsz | 11:33 |
| *** harsha_ has joined #openstack-kolla | 11:34 | |
| *** ktibi has joined #openstack-kolla | 11:34 | |
| *** Wasaac has quit IRC | 11:36 | |
| hrw | mnasiadka: kolla covers zyllion of projects. we care about subsets (usually partially covering each other). I care about small one, ignore rest and mention those which fail ;D | 11:36 |
| *** dave-mccowan has joined #openstack-kolla | 11:36 | |
| *** Wasaac has joined #openstack-kolla | 11:38 | |
| openstackgerrit | Merged openstack/kolla master: Add upgrade note for nova-placement-api removal https://review.opendev.org/657587 | 11:40 |
| hrw | mgoddard: https://review.opendev.org/#/c/665668/ - may you? | 11:41 |
| mgoddard | hrw: what's the main reason for pulling them into the base image? build time? disk space? | 11:44 |
| hrw | mgoddard: build time mostly. | 11:44 |
| mgoddard | hrw: does it have a large impact? | 11:44 |
| hrw | mgoddard: on non-x86 all those modules need to be built as binary wheels can not be fetched from pypi | 11:45 |
| hrw | mgoddard: on fast 46-core server it gave me few minutes for 'try to build everything' build. not checked how it impacts CI | 11:45 |
| *** schwicht has quit IRC | 11:46 | |
| mgoddard | hrw: I see. I guess it makes sense. The main drawback is that it makes fatter images since not all images need all those deps. Although if they're all built together they can share layers | 11:47 |
| hrw | mgoddard: they all share layers | 11:48 |
| hrw | I hope that https://review.opendev.org/640099 author will appear one day and continue | 11:48 |
| mgoddard | hrw: yeah, unless you do CI/CD | 11:48 |
| hrw | that squash stuff was nice | 11:48 |
| mgoddard | hrw: I thought someone tested it and it worked already? | 11:49 |
| hrw | mgoddard: I played with it and results were nice. | 11:50 |
| mgoddard | hrw: have you tried using squash without that patch? I think it works already | 11:50 |
| hrw | mgoddard: iirc without it we have 1 layer per image | 11:51 |
| hrw | so nova-compute = 1 layer, nova-libvirt = 1 layer etc | 11:51 |
| hrw | but have to check | 11:51 |
| mgoddard | hrw: I'm sure someone said they checked and it does that already | 11:52 |
| hrw | ok | 11:52 |
| mgoddard | docker history will show | 11:52 |
| hrw | INFO:kolla.common.utils:Image squash is enabled and "docker-squash" version is b'1.0.7' | 11:52 |
| hrw | I see a space for patch ;d | 11:53 |
| hrw | INFO:kolla.common.utils.base:Parent lastest layer is: sha256:7a51de8a65d533b6706fbd63beea13610e5486e49141610e553a3e784c133a37 | 11:53 |
| hrw | lastest - is that a word at all? | 11:53 |
| hrw | ah. archaic english | 11:55 |
| mnasiadka | now we'll start a series of patches to fix ,,englishness''? :) | 11:55 |
| hrw | mgoddard: current squash support does not work at all | 11:56 |
| hrw | INFO New squashed image ID is 9cbb631487c04368b252502313612f310de0d9d7ff05b5895ed042ce5386d3f2\n2019-06-18 11:53:31,785 root INFO Image registered in Docker daemon as ceph-squash/centos-binary-base:8.1.0\n2019-06-18 11:53:32,028 root INFO Removing old sha256:b51bac4b4bac319cb0e1de103f5fcd2fdec496a4d2691edb425e56b8aa40a3ab image...\n2019-06-18 11:53:32,032 root | 11:56 |
| hrw | ERROR 409 Client Error: Conflict ("conflict: unable to delete b51bac4b4bac (cannot be forced) - image has dependent child images")\n2019-06-18 11:53:32,032 root ERROR Execution failed, consult logs above. If you think this is our fault, please file an issue: https://github.com/goldmann/docker-squash/issues, thanks!\n' | 11:56 |
| openstackgerrit | Merged openstack/kolla master: Add sha256 in ceph fetch keys https://review.opendev.org/655623 | 11:57 |
| mgoddard | hrw: ok, sounds like the problem that was described | 11:57 |
| *** luksky has joined #openstack-kolla | 12:02 | |
| *** cah_link has quit IRC | 12:03 | |
| *** cah_link has joined #openstack-kolla | 12:03 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/kolla master: Fedora support for python3 package testing https://review.opendev.org/624838 | 12:04 |
| hrw | heh. bifrost is on xenial still with CI | 12:07 |
| *** hamzaachi_ has joined #openstack-kolla | 12:10 | |
| mnasiadka | well, can't we move to deploying ironic-standalone with kolla (bikolla stuff that mgoddard did) instead of using bifrost? | 12:13 |
| mnasiadka | and deprecate this bloody container image? :) | 12:13 |
| mgoddard | mnasiadka: not until kayobe stops using it :) | 12:13 |
| mgoddard | although we don't support ubuntu | 12:14 |
| mnasiadka | mgoddard: so stop for god's sake :) | 12:14 |
| mgoddard | mnasiadka: we will | 12:14 |
| mnasiadka | I know, just laughing :) | 12:14 |
| mgoddard | :p | 12:14 |
| mnasiadka | but we could update the kolla-ansible bifrost docs with how to do bikolla and say this is the recommended method | 12:15 |
| *** goldyfruit has joined #openstack-kolla | 12:15 | |
| hrw | INFO:kolla.common.utils.base:Successfully tagged ceph-squash2/centos-binary-base:8.1.0 | 12:16 |
| hrw | INFO:kolla.common.utils.base:Squashing from sha256:7a51de8a65d533b6706fbd63beea13610e5486e49141610e553a3e784c133a37 (tagged as: centos:7) | 12:16 |
| hrw | INFO:kolla.common.utils.base:Image squashed successfully | 12:16 |
| hrw | 640099 applied | 12:16 |
| mgoddard | mnasiadka: or add separate standalone docs and link to them | 12:16 |
| mnasiadka | also good idea | 12:16 |
| *** schwicht has joined #openstack-kolla | 12:17 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Make cron jobs generator command task in keystone non-changing https://review.opendev.org/665533 | 12:19 |
| openstackgerrit | Merged openstack/kolla-ansible master: Comment out all variables in globals.yml https://review.opendev.org/665534 | 12:19 |
| gfidente | yoctozepto hrw so it seems there will be a new nfs-ganesha repo in centos | 12:22 |
| gfidente | would that work better for kolla vs having ganesha directly in the ceph repos? | 12:22 |
| hrw | gfidente: will work better for tripleo guys using kolla for sure ;D | 12:23 |
| yoctozepto | gfidente: irrelevant as long as it works | 12:23 |
| yoctozepto | ;D | 12:23 |
| yoctozepto | have not sat down to make a change testing it yet eh | 12:23 |
| yoctozepto | busy doing other things | 12:23 |
| gfidente | and would that allow kolla go back to the centos repos vs ceph ? | 12:24 |
| yoctozepto | gfidente: I think we would prefer that as that was the plan until ganesha was discovered to be missing | 12:24 |
| mnasiadka | gfidente: most likely, so where are the packages currently so we can test them? | 12:25 |
| gfidente | yoctozepto ok great | 12:25 |
| gfidente | mnasiadka currently in nautilus-testing repo | 12:25 |
| gfidente | the nfs-ganesha repo being created | 12:25 |
| gfidente | as we speak | 12:25 |
| yoctozepto | gfidente: great | 12:25 |
| yoctozepto | can you link us? | 12:25 |
| yoctozepto | (when ready) | 12:26 |
| mnasiadka | ok, I see them | 12:26 |
| gfidente | so I think for now | 12:26 |
| mnasiadka | for now we can just test - any eta when it will be on the usual mirror? | 12:26 |
| yoctozepto | https://cbs.centos.org/repos/storage7-ceph-nautilus-testing/x86_64/os/Packages/ | 12:26 |
| yoctozepto | mnasiadka: I planned to submit a change using this so that Zuul can crunch it | 12:27 |
| yoctozepto | but you can take it if you are more free | 12:27 |
| hrw | gfidente: once it land in http://mirror.centos.org/{altarch,centos}/7/storage/{aarch64,x86-64,ppc64le}/ceph-nautilus/ dirs we may use it | 12:27 |
| gfidente | hrw so for now the plan is | 12:28 |
| * hrw -> digging in requirements stuff | 12:28 | |
| mnasiadka | yoctozepto: yeah, still I prefer to test it on some setup - and then we need to wait for getting it published on mirror.centos.org | 12:28 |
| yoctozepto | hrw, gfidente: yup, now only testing | 12:28 |
| gfidente | yoctozepto volunteered to help testing, which is great | 12:28 |
| gfidente | at that point if things look fine I can promote it -release and in a matter of days it should be in the mirrors | 12:28 |
| gfidente | then I can ping you guys back | 12:28 |
| gfidente | regaring the 'standalone' ganesha repo, which is probably a better long term plan | 12:28 |
| gfidente | maybe can be integrated later? | 12:28 |
| *** k_mouza has joined #openstack-kolla | 12:29 | |
| mnasiadka | I guess it can, but in Storage SIG meaning I guess it's like 2 years time? :) | 12:29 |
| *** schwicht has quit IRC | 12:29 | |
| gfidente | I am also trying a scratch build of 14.2.1 so hopefully we get that updated too | 12:29 |
| hrw | gfidente: 'standalone ganesha repo' does not have much sense when you have to link it with ceph anyway | 12:30 |
| gfidente | hrw link at build time? | 12:30 |
| gfidente | if you guys want to join #centos-devel | 12:31 |
| hrw | gfidente: nfs-ganesha-ceph without ceph repo will be linked to a) latest whatever, b) some random version c) whatever was in first centos7? | 12:31 |
| *** schwicht has joined #openstack-kolla | 12:31 | |
| gfidente | hrw yeah I want to make sure I understand the question correctly, you're thinking linking at nfs-ganesha-ceph build time? | 12:31 |
| yoctozepto | gfidente: hrw means ganesha depends on ceph anyway | 12:32 |
| yoctozepto | so people are gonna use ceph repo too | 12:32 |
| gfidente | yeah so last proposal was to have centos-release-ceph install by dependency nfs-ganesha | 12:32 |
| hrw | gfidente: storage sig builds ceph XYZ version and put in storage/ARCH/ceph-XYZ/ repo. then you build nfs-ganesha against ceph XYZ or XYZ-1? | 12:32 |
| mnasiadka | well, ganesha in our setup yeah - but also glusterfs guys use ganesha - with different FSALs, right? | 12:33 |
| yoctozepto | let's move this to centos-devel | 12:33 |
| *** goldyfruit has quit IRC | 12:38 | |
| yoctozepto | ok, going AFK again | 12:38 |
| yoctozepto | busy day | 12:38 |
| gfidente | thanks guys | 12:40 |
| gfidente | mnasiadka yes on gluster, so I think Requires solves that best for all ganesha, ceph and gluster | 12:41 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Add CentOS/source aarch64 experimental job https://review.opendev.org/662456 | 12:41 |
| mnasiadka | gfidente: I guess yes, but I hope on the ceph-nfs package, right? | 12:43 |
| gfidente | mnasiadka on the centos-release-ceph-nautilus repo | 12:43 |
| gfidente | sorry package | 12:43 |
| *** rgogunskiy has joined #openstack-kolla | 12:44 | |
| gfidente | so that when you install the nautilus repo, the ganesha repo is installed as well | 12:44 |
| mnasiadka | gfidente: ah, you meant the repo :) sure :) | 12:44 |
| gfidente | the repo yes, sorry | 12:44 |
| gfidente | Requires on the repo package, to also install the ganesha repo package | 12:45 |
| mnasiadka | Good! :) | 12:47 |
| *** priteau has quit IRC | 12:49 | |
| *** nathharp has quit IRC | 12:50 | |
| *** schwicht has quit IRC | 12:53 | |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Format internal Fluentd logs https://review.opendev.org/665953 | 12:56 |
| dougsz | mnasiadka: haha | 12:57 |
| *** schwicht has joined #openstack-kolla | 12:57 | |
| dougsz | hrw: what do we do about the missing librdkafka, this is an arm specific problem right? | 12:57 |
| hrw | dougsz: this is non-x86 specific | 12:58 |
| hrw | dougsz: pypi should be dropped, burnt, covered with soil, digged out, burnt again, poured with jet fuel, burnt and then done properly | 12:59 |
| hrw | now it is 'just upload whatever and we will host' | 12:59 |
| dougsz | or print a message saying: "non-x86 is not currently supported by Monasca" message | 12:59 |
| *** schwicht has quit IRC | 12:59 | |
| hrw | dougsz: s/by Monasca// and you have one of my bad days ;D | 13:00 |
| *** goldyfruit has joined #openstack-kolla | 13:00 | |
| dougsz | :) | 13:00 |
| hrw | dougsz: I do arm since 2004. since 2007 it is my daily job. aarch64 since 2012 | 13:01 |
| dougsz | I first started on the mini2440, but it's been along time since I fired that up :) | 13:05 |
| hrw | dougsz: s3c2440 with 3.5" screen? | 13:05 |
| dougsz | that's the one :) | 13:06 |
| hrw | dougsz: sharp zaurus sl5500 was my first arm device. sharp zaurus c760 first device I got due to my foss work. and then countless devices and developer boards | 13:07 |
| *** hamzaachi_ has quit IRC | 13:08 | |
| dougsz | hrw: indeed, there was a Cambrian explosion of developer boards, and weird closed source binaries to go with them | 13:09 |
| hrw | ;D | 13:09 |
| *** harsha_ has quit IRC | 13:15 | |
| openstackgerrit | Doug Szumski proposed openstack/kolla master: Mark non-x86_64 Monasca builds as not supported https://review.opendev.org/665978 | 13:18 |
| dougsz | hrw, something like ^ for now? | 13:18 |
| *** priteau has joined #openstack-kolla | 13:19 | |
| hrw | dougsz: personally I am not a fan of such checks | 13:19 |
| hrw | dougsz: and your check also needs fixing. you are in {% if install_type == binary %} already ;D | 13:20 |
| *** rgogunskiy has quit IRC | 13:21 | |
| dougsz | hrw: what do you propose as an alternative? adding support for arm64 isn't easy for me, we don't have any arm hardware for testing. | 13:23 |
| hrw | dougsz: leave as it is | 13:23 |
| dougsz | Let the user hit a build error? | 13:23 |
| hrw | dougsz: yes | 13:24 |
| hrw | dougsz: once we get to the supported/maybesupported/notsupported split of images we may handle it other way | 13:24 |
| dougsz | ok, if you're happy to leave it that's fine with me | 13:25 |
| hrw | other thing: Does someone here uses Docker <17.03? | 13:26 |
| hrw | I may have nice patch for squash stuff | 13:28 |
| *** schwicht has joined #openstack-kolla | 13:31 | |
| *** pcaruana has quit IRC | 13:37 | |
| *** skramaja has quit IRC | 13:44 | |
| openstackgerrit | Rafael Weingärtner proposed openstack/kolla-ansible master: Cloudkitty InfluxDB Storage backend via Kolla-ansible https://review.opendev.org/658418 | 13:47 |
| *** rafaelweingartne has joined #openstack-kolla | 13:49 | |
| rafaelweingartne | Hello Guys, I have worked on this PR: https://review.opendev.org/#/c/652157/. It was reviewed and it looks ready to get in. Could somebody help me reviewing it?We need that standardization process in components URL to enable us to push other improvements that we did in Kolla-ansible | 13:51 |
| rafaelweingartne | We would be happy to finish this one, and then to start contributing back some other exciting features that we have already developed :) | 13:52 |
| mgoddard | hi rafaelweingartne, it's in a tab in my browser, just haven't got to it yet | 13:53 |
| mgoddard | rafaelweingartne: it's above the 'requires a fresh cup of tea' size threshold | 13:53 |
| rafaelweingartne | :P | 13:53 |
| rafaelweingartne | sure thanks :) you have been helping quite a lot | 13:54 |
| *** pcaruana has joined #openstack-kolla | 13:54 | |
| mgoddard | rafaelweingartne: if you have time to do a few reviews, that would be great | 13:55 |
| *** k_mouza has quit IRC | 13:59 | |
| *** mchlumsky has joined #openstack-kolla | 14:00 | |
| rafaelweingartne | I can make time | 14:02 |
| rafaelweingartne | do you have some PRs that you would like to get reviewed? | 14:02 |
| openstackgerrit | Zijian Guo proposed openstack/kolla-ansible master: Fix the redis_connection_string for osprofiler and make it generic https://review.opendev.org/665937 | 14:03 |
| hrw | 1 file changed, 9 insertions(+), 20 deletions(-) | 14:04 |
| *** k_mouza has joined #openstack-kolla | 14:04 | |
| hrw | proper squash support without docker-squash ;D | 14:04 |
| hrw | too optimistic. code works but not quite ok | 14:09 |
| *** rgogunskiy has joined #openstack-kolla | 14:10 | |
| *** Luzi has quit IRC | 14:12 | |
| * yoctozepto is back near the keyboard | 14:15 | |
| yoctozepto | > <hrw> too optimistic. code works but not quite ok | 14:15 |
| yoctozepto | I will steal this explanation | 14:15 |
| yoctozepto | next time I explain how something does | 14:15 |
| hrw | docker image build has squash stuff since ages but it does not work once you have parallel builds | 14:15 |
| hrw | building 'ceph' images == base ok, ceph-base ok, ceph-* fail as layer vanished | 14:16 |
| yoctozepto | I see | 14:18 |
| hrw | and now for something completely different: centos/arm64 CI went to building phase. so repos got sorted out. | 14:20 |
| *** rafaelweingartne has quit IRC | 14:20 | |
| yoctozepto | hrw: congrats! | 14:21 |
| hrw | yoctozepto: thx | 14:21 |
| hrw | https://pastebin.com/PHkMJb0f | 14:23 |
| hrw | is whole squashing patch | 14:23 |
| *** icarusfactor has joined #openstack-kolla | 14:23 | |
| yoctozepto | yay: https://review.opendev.org/665927 | 14:24 |
| yoctozepto | EPERM fixed :D | 14:24 |
| yoctozepto | hrw: sometimes the simplest solutions are the best ;-) | 14:25 |
| *** icarusfactor has quit IRC | 14:25 | |
| *** factor has quit IRC | 14:25 | |
| *** icarusfactor has joined #openstack-kolla | 14:25 | |
| yoctozepto | love changes with deletions > additions which actually add features to software | 14:26 |
| yoctozepto | though they probably should have been refactoring and addition | 14:26 |
| yoctozepto | nevertheless, it is always fun to do more with less (I think less utility used this pun actually) | 14:26 |
| *** dpawlik has quit IRC | 14:27 | |
| hrw | yoctozepto: so now you ask first mon for it's ceph ver and then tell it to ignore older osds? | 14:27 |
| yoctozepto | yup, exactly | 14:27 |
| yoctozepto | since we theoretically manage it | 14:27 |
| yoctozepto | up to this point | 14:27 |
| hrw | yoctozepto: what if we have multiple mons? | 14:27 |
| yoctozepto | all must have been updated to the same release | 14:27 |
| yoctozepto | just explained ^ | 14:28 |
| hrw | and you update and then run that code? | 14:28 |
| yoctozepto | and if we do have them outside kolla then the next step will still fail | 14:28 |
| hrw | +2 | 14:29 |
| *** priteau has quit IRC | 14:29 | |
| yoctozepto | then back to stein and we have this job finally fixed | 14:30 |
| *** nathharp has joined #openstack-kolla | 14:36 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: cleanup image squashing https://review.opendev.org/666000 | 14:39 |
| openstackgerrit | Pierre Riteau proposed openstack/kolla stable/queens: Remove HTTP configuration shipped in openstack-nova-placement-api RPM https://review.opendev.org/666002 | 14:40 |
| hrw | started era of satan ;D | 14:40 |
| *** JamesBenson has joined #openstack-kolla | 14:40 | |
| yoctozepto | hrw: lol | 14:43 |
| yoctozepto | so you did | 14:43 |
| *** priteau has joined #openstack-kolla | 14:44 | |
| yoctozepto | fetch-zuul-cloner : Install zuul-cloner shim dependencies | 14:45 |
| yoctozepto | popular failure in the last few failures | 14:45 |
| JamesBenson | klindgren: regarding the gather facts on your etherpad, #6, you could speed that up with an ansible config to store the facts locally, correct? | 14:45 |
| *** cah_link has quit IRC | 14:48 | |
| evrardjp | hrw: congrats on getting 666000. | 14:48 |
| hrw | evrardjp: I am off for 5 days so 666666 is out of reach | 14:49 |
| evrardjp | it was the goal of my life, I guess I need to wait for 666666 | 14:49 |
| evrardjp | hahah | 14:49 |
| hrw | evrardjp: nice goal | 14:49 |
| evrardjp | I am off too sadly | 14:49 |
| evrardjp | or happily, I don't know. | 14:49 |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Format internal Fluentd logs https://review.opendev.org/665953 | 14:56 |
| *** priteau has quit IRC | 14:58 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: Add custom filters for checking services https://review.opendev.org/665922 | 15:01 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/stein: WIP: Try CentOS Ceph Nautilus repo again, now with ganesha https://review.opendev.org/666012 | 15:01 |
| *** spsurya has quit IRC | 15:01 | |
| yoctozepto | my first devilish change | 15:01 |
| yoctozepto | mgoddard: may you +2 W+1 https://review.opendev.org/#/c/665927/? ;-) | 15:02 |
| JamesBenson | klindgren: Have you tried Mitogen? https://networkgenomics.com/ansible/ I'd love to see if this works flawlessly on deployments as large as yours. | 15:09 |
| openstackgerrit | Giulio Fidente proposed openstack/kolla master: Move back to CentOS StorageSIG repos for Ceph and Ganesha https://review.opendev.org/666015 | 15:09 |
| hrw | yoctozepto: do not append '?' to url as it will end with 404. s/\?/ patch/ keeps sentence sane and allows to click and go to patch | 15:11 |
| mgoddard | yoctozepto: I gave +1, waiting for mnasiadka who knows more about ceph | 15:12 |
| hrw | see you monday folks | 15:15 |
| mgoddard | bye hrw, have a nice break | 15:19 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Run nova-status upgrade check after DB sync and migrations https://review.opendev.org/665643 | 15:20 |
| *** whoami-rajat has quit IRC | 15:21 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Fix conditionals in CI playbook https://review.opendev.org/648757 | 15:23 |
| *** ktibi has quit IRC | 15:23 | |
| *** ktibi has joined #openstack-kolla | 15:25 | |
| *** ktibi has quit IRC | 15:26 | |
| *** sqqqrly has joined #openstack-kolla | 15:29 | |
| sqqqrly | What steps are required to use a different cinder backend within a kolla deployment? | 15:30 |
| sqqqrly | This one in particular: https://support.infinidat.com/hc/en-us/articles/115003585865 | 15:31 |
| *** ivve has quit IRC | 15:33 | |
| *** zbr|ruck is now known as zbr|brb | 15:34 | |
| yoctozepto | hrw: my client has no problems, sorry :/ | 15:36 |
| *** hamzy_ has joined #openstack-kolla | 15:38 | |
| yoctozepto | gfidente: I see you were tired of waiting for me and created your own patch :P | 15:38 |
| gfidente | yoctozepto nah my patch won't test anything cause there isn't ceph 1.2-1 out yet | 15:39 |
| gfidente | that was meant to revert kolla to centos cbs repos | 15:39 |
| *** luksky has quit IRC | 15:39 | |
| gfidente | so they don't have same goal :D | 15:39 |
| *** lemko has quit IRC | 15:39 | |
| *** kklimonda has quit IRC | 15:39 | |
| *** hamzy has quit IRC | 15:40 | |
| *** dave-mccowan has quit IRC | 15:40 | |
| *** lemko has joined #openstack-kolla | 15:40 | |
| *** kklimonda has joined #openstack-kolla | 15:41 | |
| yoctozepto | gfidente: ah, right, did not click on it :D | 15:42 |
| *** goldyfruit has quit IRC | 15:43 | |
| gfidente | yoctozepto we can keep your patch though | 15:43 |
| gfidente | so there is history of testing it | 15:43 |
| gfidente | needs to wait for 1.2-1 anyway | 15:43 |
| gfidente | tty s! | 15:44 |
| *** gfidente has quit IRC | 15:45 | |
| *** dking_desktop has joined #openstack-kolla | 15:47 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: Test: try change in ganesha https://review.opendev.org/666030 | 15:47 |
| yoctozepto | aaaand he's gone | 15:48 |
| *** hamzy_ is now known as hamzy | 15:48 | |
| klindgren | JamesBenson, Yes, I switched thedeployment over to running under mitogen 2.6 on ansible 2.7 it works, but is no faster than non-mitogen. My guess is because most of the time is spent doing merge_config and/or skips on evaluations on the config generation peice. Neither of which are speedup by mitogen, specifically custom modules are not done through mitogens black magic. | 15:53 |
| klindgren | sorry 0.2.6 and ansible 2.7.10 | 15:54 |
| JamesBenson | klindgren: You've probably done this, but these are my ansible.cfg settings: https://gist.github.com/JamesOBenson/4e2121d9f747fb21de458097b1bb621c | 15:55 |
| *** goldyfruit has joined #openstack-kolla | 15:55 | |
| JamesBenson | But I'm sure kolla itself could speed up some of these processes.... | 15:55 |
| JamesBenson | Of course, the forks would be higher for you... | 15:56 |
| klindgren | Yea, I enabled fact_caching to a jsonfile as well, we use 40 forks, pipelining is on, host_key_checking we do do, and I also played around with the internal_poll_interval (which didn't really seem to do anything) | 15:56 |
| *** gfhellma has joined #openstack-kolla | 15:56 | |
| klindgren | https://paste.ubuntu.com/p/4cbnPGw34N/ | 15:58 |
| klindgren | is our default ansible.cfg, on the actual playbook run, we overide the number of forks, as well as add in the mitogen modules. | 15:59 |
| *** pcaruana has quit IRC | 16:01 | |
| *** igordc has joined #openstack-kolla | 16:03 | |
| openstackgerrit | Merged openstack/kolla master: Remove crane image https://review.opendev.org/663942 | 16:03 |
| *** rgogunskiy has quit IRC | 16:10 | |
| *** rpittau is now known as rpittau|afk | 16:14 | |
| *** vishwanathj has joined #openstack-kolla | 16:22 | |
| *** k_mouza has quit IRC | 16:24 | |
| *** nathharp has quit IRC | 16:26 | |
| *** gfhellma_ has joined #openstack-kolla | 16:27 | |
| *** rgogunskiy has joined #openstack-kolla | 16:27 | |
| vishwanathj | dougsz I came across this link that had info on how to enable central logging using EFK https://github.com/inc0/kolla-ansible-workshop/blob/master/troubleshooting.md .... I am looking next to get info(best practices)on how to plan for storage requirements for logs....yesterday mgoddard mentioned that you are more knowledgable about this....are there any links that you could share? Thanks | 16:30 |
| *** gfhellma has quit IRC | 16:30 | |
| *** yan0s has quit IRC | 16:31 | |
| dougsz | vishwanathj, there is also Monasca which is another option to the simple EFK stack | 16:31 |
| dougsz | Obviously it depends on your deployment, and how much activity you see, but as a very rough guide we see about 10GB of logs per week on a medium size deployment (~100 bare metal nodes). | 16:33 |
| dougsz | Out of the box, an Elasticsearch index is created every day, so purging logs is easy | 16:34 |
| vishwanathj | dougsz ok, Is there any log rotation setting that ensures we do not run out of disk space once we have setup an initial storage size? One thing, we have made sure is NOT TO enable debug logs | 16:36 |
| *** zbr|brb is now known as zbr|ruck | 16:38 | |
| dougsz | vishwanathj: All the services write their logs out locally to file (kolla docker logs volumes), and there is a cron container which rotates them. For elasticsearch there is currently no support for a retention policy. It would be a nice feature to contribute. | 16:39 |
| dougsz | Good move on turning off debug logs if you don't need them. The figure I quoted above is with debug mode off, and I have no doubt it would be 1-2 orders of magnitude larger with global debug mode on. | 16:41 |
| openstackgerrit | Gaëtan Trellu proposed openstack/kolla-ansible master: Fix default deployment of freezer, use mariadb. https://review.opendev.org/661485 | 16:42 |
| dougsz | vishwanathj: Elasticsearch curator might be worth a look for retention management. | 16:43 |
| vishwanathj | dougsz thanks for the info, the reason, I asked about log ROTATION was, it would require no manual intervention and there is no need to worry about storage size not being sufficent....was wondering if kolla deploy had such a setting that could be tuned | 16:44 |
| vishwanathj | dougsz will take a look at Elasticsearch curator | 16:44 |
| *** pcaruana has joined #openstack-kolla | 16:45 | |
| *** Wasaac has quit IRC | 16:46 | |
| *** Wasaac has joined #openstack-kolla | 16:48 | |
| dougsz | vishwanathj: We've seen crashed services generate 10s of GBs per day, so it's worth factoring in a good amount of overhead into your storage calculations | 16:49 |
| *** igordc has quit IRC | 16:49 | |
| vishwanathj | dougsz thanks for that pointer | 16:52 |
| *** gfhellma_ has quit IRC | 16:52 | |
| vishwanathj | dougsz do you think it would be useful for kolla to provide a setting which in turn would allow for configuring a setting on elasticsearch that does something similar to log rotation....Is that what you were referring to as a nice feature to contribute above | 16:55 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Run nova-status upgrade check after DB sync and migrations https://review.opendev.org/665643 | 16:56 |
| dougsz | vishwanathj: If you could set something like `log_retention_period_days: 90`, and something would make sure that happened, that would be a nice feature in my opinion | 16:57 |
| vishwanathj | dougsz thanks for confirming my understanding | 16:57 |
| dougsz | You're welcome, there is a meeting on IRC tomorrow if you wanted to discuss with the core team | 16:58 |
| vishwanathj | dougsz what time is the meeting? | 16:58 |
| dougsz | 3pm UTC I think | 16:58 |
| *** Wasaac has quit IRC | 16:59 | |
| vishwanathj | ok, is it on the same channel | 16:59 |
| dougsz | yeah | 16:59 |
| dougsz | (https://wiki.openstack.org/wiki/Meetings/Kolla) | 17:00 |
| vishwanathj | At this time, I plan on attending and raising the topic we discussed | 17:00 |
| *** altlogbot_2 has quit IRC | 17:00 | |
| dougsz | vishwanathj, sounds good, thanks | 17:00 |
| vishwanathj | dougsz appreciate your time and guidance on next steps | 17:00 |
| dougsz | np! | 17:00 |
| *** gfhellma has joined #openstack-kolla | 17:01 | |
| *** altlogbot_2 has joined #openstack-kolla | 17:01 | |
| *** irclogbot_3 has quit IRC | 17:01 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Do not create Keystone domain-specific config directory if useless https://review.opendev.org/665423 | 17:02 |
| *** irclogbot_3 has joined #openstack-kolla | 17:03 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: Do not create Keystone domain-specific config directory if useless https://review.opendev.org/666059 | 17:04 |
| *** Wasaac has joined #openstack-kolla | 17:10 | |
| *** Sravan has joined #openstack-kolla | 17:11 | |
| mgoddard | sqqqrly: how familiar are you with kolla-ansible? | 17:15 |
| mgoddard | sqqqrly: https://docs.openstack.org/kolla-ansible/latest/admin/advanced-configuration.html#openstack-service-configuration-in-kolla | 17:16 |
| mgoddard | sqqqrly: you can add a custom config file, /etc/kolla/config/cinder.conf | 17:17 |
| *** gfhellma has quit IRC | 17:18 | |
| *** gfhellma has joined #openstack-kolla | 17:18 | |
| *** dougsz has quit IRC | 17:18 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Enable CI jobs for ceph upgrade, NFV, cinder LVM https://review.opendev.org/666068 | 17:29 |
| *** Wasaac has quit IRC | 17:29 | |
| *** gfhellma has quit IRC | 17:32 | |
| *** igordc has joined #openstack-kolla | 17:35 | |
| *** fvollero has quit IRC | 17:35 | |
| *** whoami-rajat has joined #openstack-kolla | 17:37 | |
| sqqqrly | mgoddard: I am getting pretty good kolla. I will next try to figure out what goes in cinder.conf. Thanks! | 17:37 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/stein: Use td-agent-gem to install fluentd monasca plugin on ubuntu https://review.opendev.org/666074 | 17:38 |
| *** Sravan has quit IRC | 17:39 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/rocky: Use td-agent-gem to install fluentd monasca plugin on ubuntu https://review.opendev.org/666075 | 17:39 |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix mongo command for supporting other port https://review.opendev.org/665538 | 17:40 |
| *** Sravan has joined #openstack-kolla | 17:40 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/rocky: [neutron] Do not migrate all subprojects https://review.opendev.org/666076 | 17:40 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/stein: Fix ownership of ironic inspector DHCP hostsdir https://review.opendev.org/666077 | 17:43 |
| yoctozepto | mgoddard: tripleo checks fail due to removed crane | 17:48 |
| yoctozepto | my suspicions confirmed, their jobs did not track the changes but branches | 17:48 |
| mgoddard | yoctozepto: doh | 17:48 |
| mgoddard | mwhahaha: ^ | 17:48 |
| yoctozepto | they are pretty useless this way | 17:48 |
| yoctozepto | as they check many times the same code and do not prevent real problems | 17:49 |
| mgoddard | yoctozepto: yeah, seems that way | 17:49 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/queens: Fix lower-constraints test fail due to failing dep build https://review.opendev.org/665432 | 17:51 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/stein: Add sha256 in ceph fetch keys https://review.opendev.org/666082 | 17:53 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/rocky: Add sha256 in ceph fetch keys https://review.opendev.org/666083 | 17:53 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/queens: Add sha256 in ceph fetch keys https://review.opendev.org/666084 | 17:53 |
| yoctozepto | mgoddard: do you know how to make gerrit (if possible) show changes in master that have not been backported to stein? | 17:56 |
| mgoddard | yoctozepto: I don't know anything that fancy | 17:57 |
| mgoddard | I just go through patches periodically and update the pointer on the whiteboard | 17:57 |
| *** Sravan has quit IRC | 17:57 | |
| yoctozepto | mgoddard: what whiteboard? | 17:58 |
| mgoddard | https://etherpad.openstack.org/p/KollaWhiteBoard | 17:58 |
| *** Sravan has joined #openstack-kolla | 17:59 | |
| *** markmcclain has quit IRC | 18:00 | |
| mgoddard | yoctozepto: if you search for the change ID you can see which branches it's in, e.g. Id9019298482c040be05f540e71dacfdf0bd77469 | 18:01 |
| yoctozepto | ok, I see, thanks | 18:02 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Add unit test for keystone fernet cron generator https://review.opendev.org/666086 | 18:02 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: Add unit test for keystone fernet cron generator https://review.opendev.org/666087 | 18:02 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/queens: Add unit test for keystone fernet cron generator https://review.opendev.org/666088 | 18:02 |
| *** markmcclain has joined #openstack-kolla | 18:02 | |
| *** Wasaac has joined #openstack-kolla | 18:03 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Fix keystone fernet key rotation scheduling https://review.opendev.org/666090 | 18:04 |
| *** Sravan has quit IRC | 18:04 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: Fix keystone fernet key rotation scheduling https://review.opendev.org/666093 | 18:06 |
| *** Sravan has joined #openstack-kolla | 18:06 | |
| *** dpawlik has joined #openstack-kolla | 18:06 | |
| *** Wasaac has quit IRC | 18:08 | |
| yoctozepto | eh, finally checking nfs ganesha | 18:09 |
| yoctozepto | http://logs.openstack.org/30/666030/1/check/kolla-ansible-ubuntu-source-ceph/9bc2aad/secondary1/logs/docker_logs/ceph_nfs.txt.gz | 18:09 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/queens: Fix keystone fernet key rotation scheduling https://review.opendev.org/666095 | 18:09 |
| yoctozepto | and it turns out we were providing a failing container :D | 18:09 |
| *** gfhellma has joined #openstack-kolla | 18:09 | |
| yoctozepto | yet another fix to do | 18:09 |
| mwhahaha | mgoddard: which patch? I'll fix. It should pull in zuul changes | 18:09 |
| mgoddard | yoctozepto: ^ | 18:09 |
| mgoddard | mwhahaha: this one removed crane, but I think you know that: https://review.opendev.org/663942 | 18:10 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/stein: WIP: Try CentOS Ceph Nautilus repo again, now with ganesha https://review.opendev.org/666012 | 18:10 |
| yoctozepto | mwhahaha: moment | 18:11 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/queens: nova: Fix DBNotAllowed during compute startup https://review.opendev.org/666096 | 18:11 |
| mwhahaha | I mean which patch is failing | 18:11 |
| mwhahaha | If it's one of those branch jobs we have a patch to remove those from the template as they were improperly added | 18:11 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Add ansible_nodename (system hostname) to /etc/hosts https://review.opendev.org/666097 | 18:12 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: Add ansible_nodename (system hostname) to /etc/hosts https://review.opendev.org/666098 | 18:12 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/queens: Add ansible_nodename (system hostname) to /etc/hosts https://review.opendev.org/666099 | 18:12 |
| yoctozepto | mwhahaha: https://review.opendev.org/666068 | 18:13 |
| yoctozepto | yes, those -branch jobs | 18:13 |
| yoctozepto | all changes fail now | 18:13 |
| yoctozepto | because crane was removed | 18:13 |
| yoctozepto | (and the crane change did not fail it) | 18:13 |
| yoctozepto | what's more interesting | 18:14 |
| yoctozepto | crane was not removed from stein | 18:14 |
| yoctozepto | but it still fails it | 18:14 |
| *** Sravan has quit IRC | 18:14 | |
| yoctozepto | so it looks like two two run against our current master each time | 18:14 |
| yoctozepto | mwhahaha, mgoddard | 18:14 |
| mgoddard | yoctozepto: that sounds a bit busted | 18:15 |
| yoctozepto | mgoddard: oh, it does | 18:15 |
| mwhahaha | sshnaidm: FYI -^ | 18:15 |
| mwhahaha | I'll look deeper once I get back to my desk | 18:15 |
| mgoddard | mwhahaha: sure. The jobs are non-voting so it doesn't block us necessarily | 18:17 |
| yoctozepto | yup, only looks bad and wasteful | 18:18 |
| *** lemko has quit IRC | 18:18 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Update service_provider for FWaaS v2 https://review.opendev.org/666118 | 18:19 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Elevate privileges for copying gnocchi policy.json https://review.opendev.org/666119 | 18:19 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: Elevate privileges for copying gnocchi policy.json https://review.opendev.org/666120 | 18:20 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Fixes VMTP deploys. https://review.opendev.org/666121 | 18:20 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: Fixes VMTP deploys. https://review.opendev.org/666122 | 18:20 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Fix monasca grafana organisation check https://review.opendev.org/666123 | 18:21 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: Fix monasca grafana organisation check https://review.opendev.org/666124 | 18:21 |
| *** Sravan has joined #openstack-kolla | 18:21 | |
| *** vishwanathj has quit IRC | 18:22 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Support multi-region discovery of Nova cells https://review.opendev.org/666125 | 18:23 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: Support multi-region discovery of Nova cells https://review.opendev.org/666126 | 18:23 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/queens: Support multi-region discovery of Nova cells https://review.opendev.org/666127 | 18:23 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Hide logs when looping over passwords https://review.opendev.org/666128 | 18:24 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: Hide logs when looping over passwords https://review.opendev.org/666129 | 18:24 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/queens: Hide logs when looping over passwords https://review.opendev.org/666130 | 18:24 |
| *** Sravan has quit IRC | 18:24 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Fix Blazar Nova aggregate in multi-region setup https://review.opendev.org/666131 | 18:24 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: Fix Blazar Nova aggregate in multi-region setup https://review.opendev.org/666132 | 18:24 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/queens: Fix Blazar Nova aggregate in multi-region setup https://review.opendev.org/666133 | 18:24 |
| *** Sravan has joined #openstack-kolla | 18:25 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: [heat] Multi-region support for bootstrap https://review.opendev.org/666134 | 18:25 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: [heat] Multi-region support for bootstrap https://review.opendev.org/666135 | 18:25 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/queens: [heat] Multi-region support for bootstrap https://review.opendev.org/666136 | 18:25 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Add blazar to fluentd aggregation https://review.opendev.org/666137 | 18:25 |
| *** rgogunskiy has quit IRC | 18:25 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: Add blazar to fluentd aggregation https://review.opendev.org/666138 | 18:26 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/queens: Add blazar to fluentd aggregation https://review.opendev.org/666139 | 18:26 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Use become for all docker tasks https://review.opendev.org/666140 | 18:28 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Check for 'become' in tasks that use Docker in pep8 https://review.opendev.org/666141 | 18:28 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Fix issue finding custom, host specific plugins https://review.opendev.org/666142 | 18:29 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: Fix issue finding custom, host specific plugins https://review.opendev.org/666143 | 18:29 |
| *** Sravan has quit IRC | 18:29 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Fix mongo command for checking replication status https://review.opendev.org/666144 | 18:30 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: Fix mongo command for checking replication status https://review.opendev.org/666145 | 18:31 |
| *** Sravan has joined #openstack-kolla | 18:31 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Fix placement log perms in config.json https://review.opendev.org/666146 | 18:31 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Fix mongo command for supporting other port https://review.opendev.org/666148 | 18:34 |
| sshnaidm | mwhahaha, seems like it rebases on https://review.opendev.org/#/c/624838/ | 18:34 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: Fix mongo command for supporting other port https://review.opendev.org/666149 | 18:34 |
| mgoddard | Phew, backporting finished... | 18:35 |
| sshnaidm | mwhahaha, I don't know much about this job, but seems like patch should be updated.. | 18:35 |
| *** vishwanathj has joined #openstack-kolla | 18:35 | |
| openstackgerrit | wes hayutin proposed openstack/kolla stable/stein: WIP: Fedora support for python3 package testing https://review.opendev.org/663409 | 18:36 |
| igordc | apparently horizon is broken today on both binary:master and source:master deployments, for different reasons.. wondering if someone else noticed or if I did something wrong (stein:source was fine with the same configs, and was attempted after) | 18:40 |
| yoctozepto | igordc: broken in what way? | 18:43 |
| openstackgerrit | wes hayutin proposed openstack/kolla master: Fedora support for python3 package testing https://review.opendev.org/624838 | 18:43 |
| *** pcaruana has quit IRC | 18:45 | |
| igordc | yoctozepto, on one case the issue is initially that the WSGIScriptAlias in horizon.conf is pointing to the python2.7 path but the horizon container only has python3.6 (this is the source:master case) | 18:46 |
| igordc | yoctozepto, on the other case the WSGIScriptAlias is fine but horizon apparently can't talk to keystone because it's not available on 127.0.0.1 (only on the mgmt ip address)... long story short it seems that local_settings.py does not get updated with what's described in local_settings (where the correct ip address is) (this is the binary:master case) | 18:48 |
| openstackgerrit | Merged openstack/kolla master: openstack-base: add several popular python modules https://review.opendev.org/665668 | 18:48 |
| igordc | yoctozepto, source:stein no issues whatsoever / same config and steps | 18:49 |
| openstackgerrit | Merged openstack/kolla master: Disable and remove OracleLinux CI jobs https://review.opendev.org/664217 | 18:50 |
| yoctozepto | igordc: ok, you mean you did a local deployment and experienced this... would you mind creating a bug report in our launchpad bugtracker? | 18:51 |
| yoctozepto | https://bugs.launchpad.net/kolla-ansible/+filebug | 18:51 |
| igordc | yoctozepto, yep | 18:52 |
| yoctozepto | also, we should probably test it in CI, ping mgoddard who may know something | 18:52 |
| yoctozepto | (it = horizon) | 18:52 |
| mnasiadka | mgoddard: there is a way to check what changes did not have cherry picks to old branches through the gerrit API, so if you want I can write some ugly script ;) | 18:55 |
| mnasiadka | mgoddard: second thing we should have some backporting policy... currently sometimes I see we backport everything everywhere ;) | 18:58 |
| *** hamzaachi has joined #openstack-kolla | 19:00 | |
| yoctozepto | mnasiadka: +1, my feelings exactly | 19:02 |
| yoctozepto | mnasiadka: you have some link to that api docs? | 19:03 |
| mnasiadka | yoctozepto: https://review.opendev.org/Documentation/rest-api.html | 19:03 |
| *** pcaruana has joined #openstack-kolla | 19:05 | |
| *** luksky has joined #openstack-kolla | 19:05 | |
| yoctozepto | mnasiadka: thanks | 19:05 |
| mnasiadka | ok, enough for the day | 19:05 |
| yoctozepto | have a good night | 19:06 |
| *** vishwanathj has quit IRC | 19:06 | |
| *** Sravan has quit IRC | 19:08 | |
| openstackgerrit | Gaëtan Trellu proposed openstack/kolla-ansible master: Fix default deployment of freezer, use mariadb. https://review.opendev.org/661485 | 19:12 |
| mwhahaha | yoctozepto, mgoddard: ah the fedora 28 jobs are a bit special cause we're carrying a patch to get them working. It's conflicting on applications hence the job failures. Thanks for the heads up but feel free to ignore them for the moment | 19:13 |
| openstackgerrit | Alex Schultz proposed openstack/kolla master: Fedora support for python3 package testing https://review.opendev.org/624838 | 19:13 |
| *** hamzaachi has quit IRC | 19:20 | |
| yoctozepto | mwhahaha: you are Alex Schultz? | 19:22 |
| mwhahaha | yes | 19:22 |
| yoctozepto | cool, I replied to your last comment on Fedora | 19:23 |
| mwhahaha | yea i had a WIP but someone kept removing it. maybe -2 it? | 19:24 |
| yoctozepto | mwhahaha: maybe they won't remove DNM :D | 19:26 |
| yoctozepto | though -2 is fine too | 19:26 |
| yoctozepto | but that requires a core member | 19:26 |
| *** JamesBenson has quit IRC | 19:26 | |
| yoctozepto | let mgoddard know, he reads backlog | 19:26 |
| *** ktibi has joined #openstack-kolla | 19:26 | |
| *** JamesBenson has joined #openstack-kolla | 19:27 | |
| *** ktibi has quit IRC | 19:31 | |
| *** JamesBenson has quit IRC | 19:34 | |
| *** JamesBenson has joined #openstack-kolla | 19:36 | |
| *** hamzaachi has joined #openstack-kolla | 19:38 | |
| *** Sravan has joined #openstack-kolla | 19:39 | |
| *** Sravan has quit IRC | 19:43 | |
| *** kplant has quit IRC | 19:44 | |
| *** vishwanathj has joined #openstack-kolla | 19:54 | |
| *** ivve has joined #openstack-kolla | 19:55 | |
| *** gfhellma_ has joined #openstack-kolla | 19:58 | |
| *** vishwanathj has quit IRC | 19:59 | |
| *** gfhellma has quit IRC | 20:02 | |
| JamesBenson | so interesting question here: Cluster is beyond repair but ceph works. Could I issue a ceph "rbd snap create .." command to snapshot a VM, download it and then upload it to another cluster and deploy? | 20:02 |
| *** rgogunskiy has joined #openstack-kolla | 20:07 | |
| *** sqqqrly has quit IRC | 20:09 | |
| *** rgogunskiy has quit IRC | 20:14 | |
| klindgren | are you going to specifically create a port for that server so the IP matches as well? | 20:14 |
| *** rgogunskiy has joined #openstack-kolla | 20:14 | |
| klindgren | I have never lost a whole cloud, and we've been running openstack for like 6 years in production. | 20:15 |
| klindgren | basically your state is in the DB, plus whatever is actually stored for the VM's, which sounds like all your vm's boot from ceph. | 20:16 |
| klindgren | if ceph is fine, then all of the rest of your state is stored in the DB, if you get the DB online, then the control plane should come online? | 20:17 |
| *** gfhellma has joined #openstack-kolla | 20:20 | |
| *** gfhellma_ has quit IRC | 20:23 | |
| JamesBenson | yeah, but having a really hard time getting the db back up... | 20:33 |
| JamesBenson | So we had two clouds go down from a dirty shutdown, the one I got back up and running... now i'm working on another and this one isn't good... | 20:34 |
| JamesBenson | I tried manually this way: https://bugs.launchpad.net/kolla-ansible/+bug/1712087 | 20:34 |
| openstack | Launchpad bug 1712087 in kolla-ansible "MariaDB in HA does not come up after kolla-ansible stop" [Undecided,Opinion] | 20:34 |
| *** rgogunskiy has quit IRC | 20:34 | |
| JamesBenson | and we have kolla 4.0.2 there... | 20:35 |
| JamesBenson | so it's an older one. | 20:35 |
| JamesBenson | it seems like I can get 2 nodes up but the third isn't and none of them take as the master | 20:36 |
| JamesBenson | any thoughts klindgren? | 20:38 |
| *** kplant has joined #openstack-kolla | 20:42 | |
| klindgren | We dont use mariadb/galera clustering for our clouds. We use external mysql servers with active/passive and But my thought would be find the server that has what you believe is the most up-to-date data on it and is still a somewhat functioning mysql instance | 20:45 |
| klindgren | convert that node from a galera cluster into a standalone mariadb server - https://serverfault.com/questions/861222/transform-galera-cluster-into-single-node-mariadb-server | 20:46 |
| klindgren | update your ckolla_configuration to tell kolla-ansible that mysql is externally managed: https://github.com/openstack/kolla-ansible/blob/33ee19611824a5660176787ddd64e149e0d69a38/doc/source/reference/databases/external-mariadb-guide.rst | 20:47 |
| klindgren | That should atleast get you back online, and a state where you can deploy | 20:48 |
| klindgren | then I would do whatever you need to do to attempt to reform the galera cluster | 20:48 |
| klindgren | for our mysql server we use active/passive, with one server replicating to the other server. We also have ip's that we manage between the servers so that we always have a common target to point out. | 20:49 |
| klindgren | In our specific case another team manages the mysql servers, so they handle the replication/backups/failover/patching for us. | 20:50 |
| JamesBenson | hahaha, I manage everything. | 20:50 |
| JamesBenson | last week I upgraded hardware... | 20:50 |
| *** dpawlik has quit IRC | 20:52 | |
| *** Wasaac has joined #openstack-kolla | 20:54 | |
| klindgren | Why are you having to do a deployment to get things running again anyway? Did the unclean shutdown blow up a bunch of the containers and you basically need to redeploy the containers, to get functional code? - I get the galera was not coming up, | 20:54 |
| JamesBenson | well not doing a deployment, just trying to get it back up and running so I can get access to the VM's. | 20:55 |
| JamesBenson | the DB is the only bit that is having issues. | 20:56 |
| *** whoami-rajat has quit IRC | 20:56 | |
| *** Wasaac has quit IRC | 20:58 | |
| *** hamzaachi has quit IRC | 21:02 | |
| *** hamzaachi has joined #openstack-kolla | 21:26 | |
| *** Sravan has joined #openstack-kolla | 21:30 | |
| *** JamesBenson has quit IRC | 21:46 | |
| *** ivve has quit IRC | 21:48 | |
| *** Wasaac has joined #openstack-kolla | 21:51 | |
| *** Wasaac has quit IRC | 21:56 | |
| *** gfhellma_ has joined #openstack-kolla | 22:00 | |
| *** gfhellma has quit IRC | 22:04 | |
| *** schwicht has quit IRC | 22:05 | |
| *** calbers has quit IRC | 22:06 | |
| *** calbers has joined #openstack-kolla | 22:08 | |
| *** mchlumsky has quit IRC | 22:09 | |
| kplant | klindgren: i didn't know that doc existed, thanks for sharing it | 22:24 |
| *** gfhellma_ has quit IRC | 22:25 | |
| *** gfhellma_ has joined #openstack-kolla | 22:28 | |
| *** hamzaachi has quit IRC | 22:29 | |
| *** icarusfactor has quit IRC | 22:34 | |
| *** icarusfactor has joined #openstack-kolla | 22:34 | |
| *** schwicht has joined #openstack-kolla | 22:44 | |
| *** luksky has quit IRC | 22:47 | |
| *** schwicht has quit IRC | 22:52 | |
| *** schwicht has joined #openstack-kolla | 22:55 | |
| *** JamesBenson has joined #openstack-kolla | 22:59 | |
| *** JamesBenson has quit IRC | 23:04 | |
| *** Sravan has quit IRC | 23:15 | |
| *** Sravan has joined #openstack-kolla | 23:16 | |
| *** Sravan has quit IRC | 23:22 | |
| *** calbers has quit IRC | 23:25 | |
| *** Sravan has joined #openstack-kolla | 23:25 | |
| *** calbers has joined #openstack-kolla | 23:26 | |
| *** schwicht has quit IRC | 23:27 | |
| *** Sravan has quit IRC | 23:29 | |
| *** Sravan has joined #openstack-kolla | 23:30 | |
| *** gfhellma_ has quit IRC | 23:31 | |
| *** zhanglong has joined #openstack-kolla | 23:36 | |
| *** Sravan has quit IRC | 23:40 | |
| *** gfhellma_ has joined #openstack-kolla | 23:42 | |
| *** Sravan has joined #openstack-kolla | 23:42 | |
| *** Sravan has quit IRC | 23:47 | |
| *** gfhellma_ has quit IRC | 23:49 | |
| *** gfhellma_ has joined #openstack-kolla | 23:50 | |
| *** gfhellma_ has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!