Wednesday, 2021-01-27

openstackgerritMerged openstack/openstack-ansible master: Bump ansible-base and OpenStack collections  https://review.opendev.org/c/openstack/openstack-ansible/+/77178000:41
*** cshen has joined #openstack-ansible00:45
*** cshen has quit IRC00:49
*** macz_ has quit IRC01:02
*** rh-jelabarre has quit IRC01:23
*** cshen has joined #openstack-ansible01:33
*** cshen has quit IRC01:38
*** cloudnull has quit IRC01:56
*** cloudnull has joined #openstack-ansible01:59
*** spatel has joined #openstack-ansible03:12
*** cshen has joined #openstack-ansible03:43
*** cshen has quit IRC03:48
*** jamesden_ has quit IRC04:08
*** logan- has quit IRC04:08
*** janno has quit IRC04:08
*** PrinzElvis has quit IRC04:08
*** zigo has quit IRC04:08
*** spatel has quit IRC04:11
*** janno has joined #openstack-ansible04:14
*** PrinzElvis has joined #openstack-ansible04:14
*** zigo has joined #openstack-ansible04:14
*** manti has quit IRC04:20
*** csmart has quit IRC04:20
*** fridtjof[m] has quit IRC04:20
*** masterpe has quit IRC04:20
*** ioni has quit IRC04:20
*** jamesden_ has joined #openstack-ansible04:34
*** logan- has joined #openstack-ansible04:34
*** jamesden_ has quit IRC04:36
*** logan- has quit IRC04:36
*** jamesden_ has joined #openstack-ansible04:38
*** logan- has joined #openstack-ansible04:38
*** ioni has joined #openstack-ansible04:49
*** jamesden_ has quit IRC04:54
*** logan- has quit IRC04:54
*** jamesden_ has joined #openstack-ansible04:57
*** logan- has joined #openstack-ansible04:57
*** ioni has quit IRC05:01
*** spatel has joined #openstack-ansible05:29
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-ansible05:33
*** spatel has quit IRC05:33
*** csmart has joined #openstack-ansible05:39
*** manti has joined #openstack-ansible05:40
*** cshen has joined #openstack-ansible05:44
*** fridtjof[m] has joined #openstack-ansible05:45
*** cshen has quit IRC05:49
*** cshen has joined #openstack-ansible06:00
*** cshen has quit IRC06:04
*** ioni has joined #openstack-ansible06:07
*** SiavashSardari has joined #openstack-ansible06:43
*** gyee has quit IRC06:44
*** cshen has joined #openstack-ansible07:02
*** cshen has quit IRC07:07
*** miloa has joined #openstack-ansible07:16
*** cshen has joined #openstack-ansible07:20
*** cshen has quit IRC07:26
*** cshen has joined #openstack-ansible07:56
*** andrewbonney has joined #openstack-ansible08:08
*** MickyMan77 has left #openstack-ansible08:18
*** MickyMan77 has joined #openstack-ansible08:18
*** yasemind has joined #openstack-ansible08:22
*** jbadiapa has joined #openstack-ansible08:27
*** rpittau|afk is now known as rpittau08:30
*** maharg101 has joined #openstack-ansible08:41
*** tosky has joined #openstack-ansible08:43
openstackgerritAndrew Bonney proposed openstack/openstack-ansible-galera_server stable/victoria: Bring db setup vars in line with other roles  https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/77255008:51
jrossermorning08:51
*** spatel has joined #openstack-ansible09:01
*** spatel has quit IRC09:05
*** yasemind has quit IRC09:10
*** SiavashSardari has quit IRC09:10
*** SiavashSardari has joined #openstack-ansible09:13
*** admin0 has joined #openstack-ansible09:26
admin0morning \o09:26
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_tempest master: Use internal endpoint for tempestconf and respect tempest_keystone_interface_insecure  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/76996609:35
*** yasemind has joined #openstack-ansible09:56
openstackgerritMerged openstack/openstack-ansible master: Add more explantion for variables for external ceph integration  https://review.opendev.org/c/openstack/openstack-ansible/+/77227010:10
*** dtantsur|afk is now known as dtantsur10:18
*** gshippey has joined #openstack-ansible10:37
*** spatel has joined #openstack-ansible10:49
*** spatel has quit IRC10:54
openstackgerritMerged openstack/openstack-ansible-rsyslog_client stable/ussuri: Update role for Ubuntu Focal and Centos-8  https://review.opendev.org/c/openstack/openstack-ansible-rsyslog_client/+/77141010:57
*** SiavashSardari has quit IRC11:08
*** ianychoi_ has joined #openstack-ansible11:09
*** yasemind has quit IRC11:09
*** ianychoi__ has quit IRC11:12
*** yasemind has joined #openstack-ansible11:23
*** SiavashSardari has joined #openstack-ansible12:25
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_tempest master: Use internal endpoint for tempestconf and respect tempest_keystone_interface_insecure  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/76996612:30
openstackgerritMerged openstack/openstack-ansible master: Install rsync for CentOS bootstrap  https://review.opendev.org/c/openstack/openstack-ansible/+/77249512:57
*** rh-jelabarre has joined #openstack-ansible12:58
openstackgerritAndrew Bonney proposed openstack/openstack-ansible master: Add to bind-to-mgmt upgrade release note for older deployments  https://review.opendev.org/c/openstack/openstack-ansible/+/77268813:06
openstackgerritAndrew Bonney proposed openstack/openstack-ansible stable/victoria: Add to bind-to-mgmt upgrade release note for older deployments  https://review.opendev.org/c/openstack/openstack-ansible/+/77269113:15
*** gaudenz has joined #openstack-ansible13:17
*** SiavashSardari has quit IRC13:22
*** yasemind has quit IRC13:23
*** d34dh0r53 has quit IRC13:27
*** d34dh0r53 has joined #openstack-ansible13:28
*** spatel has joined #openstack-ansible13:28
openstackgerritJonathan Rosser proposed openstack/openstack-ansible stable/victoria: Install rsync for CentOS bootstrap  https://review.opendev.org/c/openstack/openstack-ansible/+/77266513:31
*** spatel has quit IRC13:33
*** yasemind has joined #openstack-ansible13:36
gaudenzHi13:57
*** cshen has quit IRC13:58
gaudenzI'm a bit unsure about the current state of central logging in OSA. It looks like the systemd-journal-remote facility is now disabled, but the changes to convert some services to log to the journal were not reverted.13:58
gaudenzAlso it looks like some services were never converted to log to the journal.13:59
*** spatel has joined #openstack-ansible13:59
jrosserthe default is that everything should be logged to the journal14:00
jrosserif you've found things that are not it would be worth making some bugs on launchpad for those14:00
gaudenzSo the current default state should be that everything logs to the journal and there is no central logging?14:00
*** spatel has quit IRC14:01
jrossernot by default, as folk tend to have their own preferred solution, graylog / ELK / .....14:01
*** spatel has joined #openstack-ansible14:01
*** spotz has joined #openstack-ansible14:03
gaudenzI see, some services I thought were not converted to use the journal were converted in later releases. I'm currently in the process of ugprading to Rocky, there nova, glance and neutron still use syslog14:03
jrosserit would be good to get significantly beyond rocky fairly quickly14:08
jrosserfrom an OSA perspective we no longer run any meaningful CI for rocky or stein14:09
spateljrosser: i am running very large cluster on stein :D14:10
spatelI think i have to take some action14:10
spatelnot sure how difficult it would be to forklift whole cluster from centos to ubuntu14:12
jrosserif you introduce at minimum one ubuntu infra node with a repo container on it, you should be able to add ubuntu computes14:13
spatelhmm! interesting, so add ubuntu repo and tell your new ubuntu compute to use that repo to rollout right?14:14
spatelthat would be my next project to move from centos to ubunut because we are running 500 compute nodes on that cloud and it would be fun fun...14:16
jrosserpretty much yes, just using an ubuntu host for all the venv/wheel stuff we looked at yesterday14:16
gaudenzWe are planning to upgrade to ussuri or victoria fairly soon. This way I would like to settle on the way logging is going to be setup in the future.14:16
jrosserspatel: more care needed when you decide to roll the control plane centos->ubuntu though, might take a practice run at that first somewhere safe14:17
spateloh yes.. lab lab lab (i am not going to touch otherwise)14:18
spateljrosser: why are we not adding option in journalctl to forward logs to rsyslog if folks already have rsyslog setup using "ForwardToSyslog=yes" option in journalctl.conf14:19
jrosserbecasue it is broken in systemd14:20
jrosseroh sorry14:20
jrosseri guess thats again something that you can do in your host preparation14:20
jrosserall journals -> syslog may not really be a thing in the scope of OSA, but up to the deployer?14:20
jrosserOSA tries to be leaving these choices open14:21
gaudenzAt least that might be a transition path until the remote journal logging is fixed.14:22
*** tosky has quit IRC14:35
*** tosky has joined #openstack-ansible14:36
spatelThat is what i am going to use for interim fix to ship logs using journalctl -> rsyslog14:38
*** cshen has joined #openstack-ansible14:40
jrosseri think it's unlikley that we will merge any code to do with syslog stuff14:42
jrosserdo whatever is needed as part of the pre-OSA setup14:42
spatelunderstand14:42
*** tosky has quit IRC14:50
*** tosky has joined #openstack-ansible14:50
CeeMacjrosser: do the newer release still use journal-upload and journal-remote to centralise the journals on log-hosts14:51
jrosserthe code is there but it is disabled14:51
jrosserCeeMac: https://github.com/openstack/openstack-ansible/blob/master/releasenotes/notes/disable-journald-remote-33d3b43e538786ed.yaml14:53
CeeMacoh because of the systemd bug where pruning the log folders wasn't working?14:55
CeeMaci remember looking at the now14:55
*** irclogbot_2 has quit IRC14:55
CeeMacs/the/that14:55
spatelSo it looks like this was fixed some time between v239 and v244. Nice!14:56
spatelQuoting those comments14:56
jrosserso bionic doesnt have the fix, but focal should14:57
jrosserthat could specatularly simplify deploying something like the ELK stack for journal collection14:58
*** irclogbot_1 has joined #openstack-ansible14:58
jrosseri've got journalbeat on each host right now which is a lot of stuff to deploy14:58
spateljournalbeat each host and containers right?15:03
spatelI am debating between journalbeat vs rsyslog for my centralized graylog15:04
mgariepyspatel, journal2gelf  ?15:12
openstackgerritAndrew Bonney proposed openstack/openstack-ansible-lxc_container_create master: Fix formatting of LXC container config files  https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/77270615:21
*** spatel has quit IRC15:24
*** yasemind has quit IRC15:25
CeeMaci was probably going to roll with graylog15:27
CeeMaci've debated going with elk, alongside the elk metrics15:27
CeeMacbut graylog feels more appropriate15:28
CeeMacbut yeah if focal has the patched systemd that would be good!15:28
CeeMacone ive upgrade >_<15:28
CeeMacs/one/once15:28
*** spatel has joined #openstack-ansible15:53
spatelmgariepy: doesn't it need go interpreter?15:53
*** macz_ has joined #openstack-ansible16:12
*** macz_ has joined #openstack-ansible16:13
mgariepyspatel, no https://github.com/systemd/journal2gelf/blob/master/journal2gelf16:16
spatelmgariepy: really, so just sinle python script to ship logs16:17
mgariepyi'm not sure what does journalbeat do ship16:18
mgariepyin graylog16:18
mgariepybut journal2gelf does a good job.16:18
spatelgraylog is nothing but ELK behind the walls16:19
mgariepyhttps://github.com/openstack/openstack-ansible-ops/blob/master/graylog/graylog-forward-logs.yml16:19
spatelI got confused in SystemdJournal2Gelf which need Go package.16:20
spatelend of the days they all going to ship logs but looking for something easy which can last for next few years and not lots of overhead16:21
mgariepynow i am also.16:21
mgariepylol16:21
CeeMacspatel: technically, kinda, but also not i think16:23
CeeMaci know it uses elastic and mongo, and kibana16:23
CeeMacbut i figured the log processing with a bit better then logstash or fluentd16:24
*** rpittau is now known as rpittau|afk16:24
spatelCeeMac: yes, i am using both GUI interface graylog GUI + i have kibana installed16:24
CeeMacfrom what i've read anyway, but the internet is full of conflicting opinions :D16:24
spatelIts all depend how much logs your are pumping, I don't think openstack produce lots of logs so anything you deploy should be fine.16:27
spatelbut when it comes to millions of log line per second then you need to think about client side processing vs server side for parsing etc..16:27
spatelWe have massive ELK running (not for openstack) which has 15 data nodes and millions of transaction per second16:28
spatelwith 500 compute nodes my graylog producing this rate - https://ibb.co/Bfm4Hq916:31
CeeMacwell, i was going to multipurpose it for the other infrastructure and switch stacks etc too16:32
jrosserif you deploy the whole ELK stack the data rate gets out of hand16:36
*** CeeMac has quit IRC16:37
spatelwhat do you mean deploying whole ELK stack?16:37
*** PrinzElvis has quit IRC16:37
*** NobodyCam has quit IRC16:37
*** alanmeadows has quit IRC16:37
*** hindret has quit IRC16:37
*** mwhahaha has quit IRC16:37
*** gshippey has quit IRC16:37
*** mnaser has quit IRC16:37
*** johnsom has quit IRC16:38
*** strattao has quit IRC16:38
*** Open10K8S has quit IRC16:38
*** jungleboyj has quit IRC16:38
*** rpittau|afk has quit IRC16:38
*** fyx has quit IRC16:38
*** guilhermesp has quit IRC16:38
*** simondodsley has quit IRC16:38
*** mubix has quit IRC16:38
*** nicolasbock has quit IRC16:38
*** sri_ has quit IRC16:38
*** bl0m1 has quit IRC16:38
*** andrewbonney has quit IRC16:38
*** gixx has quit IRC16:38
*** CeeMac has joined #openstack-ansible16:38
*** NobodyCam has joined #openstack-ansible16:39
*** johnsom has joined #openstack-ansible16:39
*** andrewbonney has joined #openstack-ansible16:39
*** gixx has joined #openstack-ansible16:39
*** cyberpear has quit IRC16:40
*** gouthamr has quit IRC16:40
*** jrosser has quit IRC16:40
*** gundalow has quit IRC16:40
*** simondodsley has joined #openstack-ansible16:40
*** rpittau|afk has joined #openstack-ansible16:41
*** alanmeadows has joined #openstack-ansible16:42
*** hindret has joined #openstack-ansible16:42
*** jungleboyj has joined #openstack-ansible16:42
*** guilhermesp has joined #openstack-ansible16:42
*** mubix has joined #openstack-ansible16:44
*** mwhahaha has joined #openstack-ansible16:44
*** Open10K8S has joined #openstack-ansible16:45
*** strattao has joined #openstack-ansible16:45
*** gshippey has joined #openstack-ansible16:45
*** sri_ has joined #openstack-ansible16:45
*** mnaser has joined #openstack-ansible16:45
*** d34dh0r53 has quit IRC16:46
*** d34dh0r53 has joined #openstack-ansible16:46
*** d34dh0r53 has quit IRC16:47
spatelDo you guys using rabbitMQ HA queue for all notifications.* queue?16:47
*** d34dh0r53 has joined #openstack-ansible16:48
spatelI think its kind of overhead if you don't care about notifications16:48
*** fyx has joined #openstack-ansible16:50
*** nicolasbock has joined #openstack-ansible16:50
*** cyberpear has joined #openstack-ansible16:54
*** jrosser has joined #openstack-ansible16:55
*** nicolasbock has quit IRC16:55
*** PrinzElvis has joined #openstack-ansible16:56
*** bl0m1 has joined #openstack-ansible16:56
*** viks____ has joined #openstack-ansible16:57
*** gundalow has joined #openstack-ansible16:57
*** nicolasbock has joined #openstack-ansible16:57
*** johanssone has quit IRC16:58
*** johanssone has joined #openstack-ansible17:01
admin0how to make swift use ceph for its backend ?17:12
*** miloa has quit IRC17:15
*** gundalow has quit IRC17:17
*** nicolasbock has quit IRC17:18
*** viks____ has quit IRC17:18
*** fyx has quit IRC17:18
*** fyx has joined #openstack-ansible17:19
*** gundalow has joined #openstack-ansible17:19
*** nicolasbock has joined #openstack-ansible17:19
*** hindret has quit IRC17:21
*** andrewbonney has quit IRC17:21
*** andrewbonney has joined #openstack-ansible17:24
*** hindret has joined #openstack-ansible17:34
*** hindret has quit IRC17:39
*** b1tsh1ft3r has joined #openstack-ansible17:47
b1tsh1ft3rRunning into this error when launching an instance from a 1.5GB QCOW2 image on train 20.2.117:48
b1tsh1ft3rVolume f181e16d-926d-47ba-beec-87af0d013d2d did not finish being created even after we waited 21 seconds or 8 attempts. And its status is error.17:49
*** hindret has joined #openstack-ansible17:49
b1tsh1ft3rRaw images work just fine, but seems QCOW2 images take a bit longer and fail often17:49
b1tsh1ft3rAny thoughts? I havnt tweaked with any timeout settings and its all default deployment configs currently for nova/cinder17:50
*** cshen_ has joined #openstack-ansible17:57
*** tosky has quit IRC17:58
*** tosky has joined #openstack-ansible17:58
*** cshen has quit IRC18:00
*** andrewbonney has quit IRC18:05
*** mgariepy has quit IRC18:06
spatelwithout logs hard to answer18:07
*** strattao has quit IRC18:17
*** gixx has quit IRC18:17
*** fyx has quit IRC18:17
*** bl0m1 has quit IRC18:17
*** guilhermesp has quit IRC18:17
*** nicolasbock has quit IRC18:17
*** mnaser has quit IRC18:17
*** cyberpear has quit IRC18:17
*** mgariepy has joined #openstack-ansible18:18
*** gixx has joined #openstack-ansible18:18
*** strattao has joined #openstack-ansible18:18
*** nicolasbock has joined #openstack-ansible18:18
*** mnaser has joined #openstack-ansible18:18
*** fyx has joined #openstack-ansible18:18
*** guilhermesp has joined #openstack-ansible18:18
*** bl0m1 has joined #openstack-ansible18:19
*** cyberpear has joined #openstack-ansible18:19
*** gyee has joined #openstack-ansible18:23
*** gixx has quit IRC18:24
*** strattao has quit IRC18:24
*** guilhermesp has quit IRC18:25
*** fyx has quit IRC18:25
*** gixx has joined #openstack-ansible18:26
*** mnaser has quit IRC18:27
*** bl0m1 has quit IRC18:27
*** cyberpear has quit IRC18:27
*** nicolasbock has quit IRC18:27
*** guilhermesp has joined #openstack-ansible18:31
*** fyx has joined #openstack-ansible18:31
*** mnaser has joined #openstack-ansible18:32
*** strattao has joined #openstack-ansible18:32
*** bl0m1 has joined #openstack-ansible18:33
*** nicolasbock has joined #openstack-ansible18:33
*** cyberpear has joined #openstack-ansible18:34
*** dtantsur is now known as dtantsur|afk18:44
*** guilhermesp has quit IRC18:54
*** strattao has quit IRC18:54
*** fyx has quit IRC18:55
*** cyberpear has quit IRC18:55
*** bl0m1 has quit IRC18:55
*** nicolasbock has quit IRC18:55
*** mnaser has quit IRC18:55
*** manti has quit IRC18:57
*** manti has joined #openstack-ansible18:57
*** maharg101 has quit IRC19:03
*** gouthamr_ has joined #openstack-ansible19:18
*** strattao has joined #openstack-ansible19:19
*** mnaser has joined #openstack-ansible19:19
*** guilhermesp has joined #openstack-ansible19:20
*** fyx has joined #openstack-ansible19:20
*** nicolasbock has joined #openstack-ansible19:20
*** cyberpear has joined #openstack-ansible19:21
*** bl0m1 has joined #openstack-ansible19:22
*** strattao has quit IRC19:40
*** gouthamr_ has quit IRC19:40
*** mnaser has quit IRC19:40
*** fyx has quit IRC19:40
*** guilhermesp has quit IRC19:40
*** nicolasbock has quit IRC19:40
*** cyberpear has quit IRC19:40
*** bl0m1 has quit IRC19:40
*** strattao has joined #openstack-ansible19:41
*** guilhermesp has joined #openstack-ansible19:42
*** fyx has joined #openstack-ansible19:42
*** bl0m1 has joined #openstack-ansible19:42
*** mnaser has joined #openstack-ansible19:42
*** nicolasbock has joined #openstack-ansible19:45
*** cyberpear has joined #openstack-ansible19:45
CeeMacb1tsh1ft3r: it might be worth enabling debug in the cinder service so you can check the traceback20:18
*** masterpe has joined #openstack-ansible20:26
masterpeWe try to install some additional compute nodes, but we get a pip error "ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE" on the nova client.20:26
*** maharg101 has joined #openstack-ansible20:59
*** maharg101 has quit IRC21:04
*** masterpe has quit IRC21:50
*** csmart has quit IRC21:50
*** manti has quit IRC21:50
*** fridtjof[m] has quit IRC21:51
*** ioni has quit IRC21:51
*** b1tsh1ft3r has quit IRC21:56
*** macz_ has quit IRC22:02
*** janno has quit IRC22:29
*** zigo has quit IRC22:29
*** janno has joined #openstack-ansible22:30
*** zigo has joined #openstack-ansible22:30
*** spatel has quit IRC22:43
*** macz_ has joined #openstack-ansible22:45
*** cshen_ has quit IRC22:45
*** klamath_atx has quit IRC22:54
*** cshen has joined #openstack-ansible23:03
*** fridtjof[m] has joined #openstack-ansible23:24
*** fridtjof[m] has quit IRC23:25
*** csmart has joined #openstack-ansible23:36
*** gshippey has quit IRC23:56

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!