Friday, 2021-03-26

fungiso if all your servers are connecting from behind a single many-to-one nat, then they'll all get persisted to a single backend00:00
jralbertyeah, I guessed that.00:01
jralbertwhich means we're running around beating up one node at a time00:01
fungias for your log, it does look like there was maybe a large burst of 61 at 22:28, which would correlate to the latest impact we observed00:02
jralbertokay, well we're going to target github from here on out in hopes of a) succeeding, and b) not crunching the infrastructure any more00:02
fungithanks for your help on this, it's really appreciated, and hopefully we can figure out a way to get our git server farm a little more tolerant of patterns like this in the near future (we've got some plans which should allow us to take advantage of a shared storage solution, but it requires some not-quite-there-yet features in the git software we've been relying on)00:05
*** luksky has quit IRC00:05
jralbertThank you likewise, and I'm sorry for the impact this has had on your infrastructure!00:06
jralbertWe're I think about as big as we can get without reconsidering our deployment practices, so that's probably on the books in the near future one way or the other00:06
*** admin0 has joined #openstack-ansible00:08
fungijralbert: no apologies needed, we're all in this together! i want you to succeed as much as you want us to ;)00:15
jralbertThat's what I love about working with this community. Being able to get directly in contact with the people who make it happen at every level, and knowing that there's constructive collaboration to be had, is a massive strength of OpenStack. Thanks!00:17
fungiit's also why i stay involved! ;)00:18
*** tosky has quit IRC00:26
*** mensis has joined #openstack-ansible00:31
*** mensis has quit IRC00:45
*** spatel_ has joined #openstack-ansible01:40
*** spatel_ is now known as spatel01:40
*** jamesdenton has quit IRC01:49
*** jamesden_ has joined #openstack-ansible01:49
*** cp- has joined #openstack-ansible02:06
*** spatel has quit IRC02:25
*** spatel_ has joined #openstack-ansible02:25
*** spatel_ is now known as spatel02:25
*** macz_ has joined #openstack-ansible02:44
*** macz_ has quit IRC02:48
*** macz_ has joined #openstack-ansible03:25
*** macz_ has quit IRC03:30
*** evrardjp has quit IRC03:40
*** evrardjp has joined #openstack-ansible03:41
*** macz_ has joined #openstack-ansible03:46
*** macz_ has quit IRC03:51
*** chandan_kumar has joined #openstack-ansible03:58
*** chandan_kumar is now known as chkumar|ruck03:58
*** macz_ has joined #openstack-ansible04:07
*** macz_ has quit IRC04:11
*** shyamb has joined #openstack-ansible04:54
*** shyamb has quit IRC05:14
*** spatel has quit IRC05:14
*** yasemind has joined #openstack-ansible05:47
*** shyamb has joined #openstack-ansible05:51
*** macz_ has joined #openstack-ansible06:12
*** macz_ has quit IRC06:17
*** jralbert has quit IRC06:19
*** luksky has joined #openstack-ansible06:45
*** shyamb has quit IRC07:05
*** shyamb has joined #openstack-ansible07:05
*** rpittau|afk has quit IRC07:05
*** NobodyCam has quit IRC07:06
*** rpittau|afk has joined #openstack-ansible07:06
*** NobodyCam has joined #openstack-ansible07:06
jonhermorning, so i was looking at my pipeline in 782837 and i'm not sure what i should do next there, at first i figured it was a fluke with all the infra issues going on the same time but seems it's not?  i also noticed that cloudkitty is still running the openstack-ansible-role-jobs and not the integrated test07:14
jonheri think os_cloudkitty should be updated to the same list of jobs as adjutant that was recently adoped to OSA?07:16
jrosserjonher: yes, exactly. If you make that patch to os_cloudkitty have a 'Depends-On' line with your changes to openstack-ansible it should test it all together07:18
*** dpawlik6 has joined #openstack-ansible07:22
jonherok, i'll push a review for changing the zuul config to same as what adjutant has, unless you also feel like there's a need for the meta distro jobs07:25
jonheras for 782837 is that something i need to fix i os_cloudkitty? https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_035/782837/1/check/openstack-ansible-functional-ubuntu-focal/03514e3/logs/ara-report/results/546.html#msg07:28
jrosserif you are going to move away from those old tests to the ones run via openstack-ansible i would ignore that07:30
*** macz_ has joined #openstack-ansible07:31
openstackgerritJonathan Herlin proposed openstack/openstack-ansible-os_cloudkitty master: Test using integrated build  https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/78321907:33
jonher^ ok thanks for wise words :)07:33
*** macz_ has quit IRC07:35
*** crazzy has joined #openstack-ansible07:40
*** shyamb has quit IRC07:43
*** jbadiapa has joined #openstack-ansible07:48
*** shyamb has joined #openstack-ansible07:49
jrosserjonher: that will be extremely easy to reproduce locally, an AIO with the env var setting SCENARIO=aio_metal_cloudkitty will do exactly what that CI job will do07:54
jrosserpatches applied first, of course07:55
jrosserthere is an easy way to do that off the gerrit menu "3 dots" top right of the screen "Download patch"07:56
jonhergood to know, i think noonedeadpunk was looking at the gnocchi requirements as they were causing issues when i was testing OSA AIO against master for my larger changes in os_cloudkitty to migrate to the new config format and such08:00
*** rpittau|afk is now known as rpittau08:09
*** shyamb has quit IRC08:09
jrossernoonedeadpunk_: did you read back the stuff ^^^^ up there about jralbert deployment hitting the opendev git servers very hard?08:37
jrosseri confirmed that with --limit to a single compute node that the build host is still resolved correctly http://paste.openstack.org/show/803941/08:50
jrosserthats for a V deployment, if you have a T environment would be interesting to double check that08:50
*** andrewbonney has joined #openstack-ansible09:02
ebbexjrosser: could i get your thoughts on what's holding xenial back on this one? https://review.opendev.org/c/openstack/openstack-ansible-tests/+/78279909:05
ebbexit looks to be something with setuptools. It seems to be using v54, and i don't think it should go beyond 44.x for python209:07
*** shyamb has joined #openstack-ansible09:07
ebbexI tried pinning it in tempest_pip_packages to <45 but that didn't seem to have any effect. and it works on bionic without it.09:09
jrosserits likley because bionic is python 3.609:10
jrosseroh well hmm09:11
jrosserthis is the last patch to os_tempest on rocky https://github.com/openstack/openstack-ansible-os_tempest/commit/02b5549ac3d85910f809623a59e0a3879b43905e09:13
jrosserwhich feels in the same area09:13
jrosserbut on older branches like that i don't think thats sufficient to cap setuptools09:15
jrosserthe reason that tempest_pip_packages is not having any effect is that the plugins are installed in a seperate task https://github.com/openstack/openstack-ansible-os_tempest/blob/stable/rocky/tasks/tempest_install.yml#L181-L19709:16
ebbexaha. maybe i can use pip_install_developer_constraints?09:17
CeeMacjonher: my colleague and I have been looking at integrating the os-cloudkitty role recently as well if you want to compare notes. We focussed on getting the inventory details and playbook aligned for non-aio deployments so it you're making progress on the aio that would be good fit?09:19
jrosserebbex: it looks like for whatever reason, at this point the global-requirement-pins file is not accounted for09:20
jrosserCeeMac: are you doing it for billing type things?09:20
CeeMacjrosser: yes that was the plan09:21
jrosserI think jonher has an almost-ready patch for the openstack-ansible repo09:21
CeeMacyeah i've been following the chat.  Id made some notes for master but we'd been testing with a retrograde to stein which has been working well so far09:22
CeeMacincluding haproxy config integrated into osa09:22
jonheryep just saw your comment jrosser, i'll upload the patch to OSA in a bit, just need to figure out the dependson part from the docs. 783219 depends-on 782837 to function09:23
jonheri'll be pushing a separate commit to openstack/cloudkitty later on as well, did some cleanup in there and migrated to the new config format, but i want to get it added to main OSA repo first to verify they are good as i don't have another env to run it in currently09:25
jrosserjonher: so theres two things there, if you have two patches in the same repo when one needs another, you rebase the second one on top of the first09:25
openstackgerritEbbex proposed openstack/openstack-ansible-tests stable/rocky: Update get-pip url  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/78279909:25
jrosserso that they are consecutive in the git history, and gerrit/zuul understands that and they automatically get tested with the first patch present in the context of the second one09:25
jonherok, so i basically rebase my review of zuul changes to the venv change, so that it runs the new test09:26
jrosserhowever, there is a second situation where you are going to need a patch to the openstack-ansible repo to make things go properly in the os_cloudkitty test, "cross-repo" if you like09:26
jrosserat that point you have to add a "Depends-On: <patch-url>" line to your commit message to indicate that09:26
jrosserand then a similar thing will happen, when the patch to os_cloudkitty is run, it will bring in the change to the openstack-ansible repo at the same time09:27
jonheri'll get on that now then09:28
jrossertheres a really handy view here https://zuul.openstack.org/status where you can look at the running jobs, and see the relationship between the patches being tested together09:28
jrosseron that page you can also watch progress of the jobs in real-time, and view the console output in your browser09:29
jonheryes that much i know at least :)09:29
jrosserawesome :)09:29
*** tosky has joined #openstack-ansible09:30
CeeMacjonher: should I send you what I have around os_cloudkitty osa role or do you think you have it pretty much nailed already?09:31
jonherfeel free to send me what you have, i think i've got it pretty much to spec by reading the docs but i have not tried any of the changes so far. first step for me was to get the role running at all09:32
openstackgerritJonathan Herlin proposed openstack/openstack-ansible-os_cloudkitty master: Test using integrated build  https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/78321909:33
openstackgerritJonathan Herlin proposed openstack/openstack-ansible master: Integrate cloudkitty  https://review.opendev.org/c/openstack/openstack-ansible/+/78323609:36
*** amalrajgenocidex has joined #openstack-ansible10:10
*** macz_ has joined #openstack-ansible10:10
*** macz_ has quit IRC10:14
amalrajgenocidexHi All, I installd barbican-ui using openstack-ansible10:19
amalrajgenocidexbut showing up on horizon10:20
amalrajgenocidexshowing error "Could not process panel group barbican: Dashboard with slug "barbican" is not registered."  in horizon log10:20
amalrajgenocidexany idea?10:20
CeeMacjonher: this is what we pulled together http://paste.openstack.org/show/803944/10:28
CeeMacwe used this as a template for stein, and adjusted the appropriate sections for where stein was different, but all of the inventory stuff is the same, and the playbook I think shouldn't need adjusting10:29
jonherthank you, i will have a look and compare to my changes10:38
*** macz_ has joined #openstack-ansible10:52
*** macz_ has quit IRC10:56
*** gokhani has joined #openstack-ansible11:13
gokhanihi jonher and CeeMac, we are using cloudkitty on our pike environment with our billing service. May be our pike cloudkitty role help you. https://github.com/b3lab/openstack-ansible-os_cloudkitty/tree/stable/pike/extras. you can see necessary files in extras dir. I also will work on cloudkity in a month. We will integrate in our victoria11:17
gokhanienvironment. we have used gnocchi as collector in pike environment, but now we will use monasca.11:17
CeeMacgokhani: thanks I'll take a look.  We're using gnocchi to but I'm keeping half an eye on monasca for the future.  I think the osa os_monasca role has been mothballed though for a couple of releases. Is that right jrosser11:23
gokhaniCeeMac, my teammate mensis (he is not there now) is working on os_monasca role. we can run it on our environment. he will contribute it ASAP. I think In a 2 week, he will contribute it.11:25
CeeMacoh nice11:25
CeeMacrafaelweingartne seems to be driving some good change in the cloudkitty project as well so hopefully this will all keep momentum going :)11:27
gokhaniit is great, previously cloudkitty have been maintained by objectif libre. now I don't know they are active or not11:30
CeeMacyeah, i think they slowed down, now there are new guys taking on11:31
CeeMachttps://opendev.org/openstack/governance/commit/56442877d64dfd24453936562c1e10099dff0bf411:31
*** macz_ has joined #openstack-ansible11:34
*** owalsh has quit IRC11:35
CeeMaci should probably have made an honorable mention to priteau given he's sitting in this channel too :)  They're all being very helpful and positive over in their channel :)11:36
*** macz_ has quit IRC11:38
gokhaniit is good to know it is maintaining again:)11:38
CeeMacyes11:38
gokhaniwhen I work on cloudkitty, ı can share my experiences. I hope this will happen asap11:43
gokhaniCeeMac, https://review.opendev.org/c/openstack/project-config/+/765806 , now it is waiting for mensis commits.11:46
*** owalsh has joined #openstack-ansible11:55
*** tbarron has joined #openstack-ansible11:58
*** shyamb has quit IRC12:09
*** cloudnull has quit IRC12:14
*** jbadiapa has quit IRC12:14
openstackgerritEbbex proposed openstack/openstack-ansible-tests stable/rocky: Update get-pip url  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/78279912:16
*** cloudnull has joined #openstack-ansible12:31
*** arxcruz has joined #openstack-ansible12:39
*** gokhani has quit IRC12:43
*** gokhani has joined #openstack-ansible12:49
*** amalrajgenocidex has quit IRC12:53
*** jbadiapa has joined #openstack-ansible12:53
openstackgerritJonathan Herlin proposed openstack/openstack-ansible master: Integrate cloudkitty  https://review.opendev.org/c/openstack/openstack-ansible/+/78323613:02
*** spatel_ has joined #openstack-ansible13:03
*** spatel_ is now known as spatel13:03
openstackgerritEbbex proposed openstack/openstack-ansible-tests stable/rocky: Remove temporary overrides  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/78330413:23
*** jbadiapa has quit IRC13:32
*** jbadiapa has joined #openstack-ansible13:34
spatelnoonedeadpunk_ around13:45
spateli just build new lab and hit this - http://paste.openstack.org/show/803948/13:45
spatelOn ubuntu focal13:45
spatelI have install prettytable but just saying if anything missing here13:45
*** rpittau is now known as rpittau|afk13:50
*** gokhani has quit IRC13:53
*** gokhani has joined #openstack-ansible13:55
jrosserspatel: you copy it to /usr/local/bin yourself?14:04
jrosserdoes it do the same if you run it from /opt/openstack-ansible/...14:05
spatelroot@os-osa:/opt/openstack-ansible/playbooks# ls -l /usr/local/bin/inventory-manage.py14:05
spatellrwxrwxrwx 1 root root 50 Mar 26 04:36 /usr/local/bin/inventory-manage.py -> /opt/openstack-ansible/scripts/inventory-manage.py14:05
jonherroot@aio1:/opt/openstack-ansible# ./scripts/inventory-manage.py  gives the same result on AIO: ModuleNotFoundError: No module named 'prettytable'14:08
jrosserit should run from the /opt/ansible-runtime venv and prettytable hopefully is installed there14:09
jrosserso installing with apt, or with the system python is not correct14:09
jrosserit should be installed as part of bootstrap-ansible, like here https://zuul.opendev.org/t/openstack/build/4df6d1ffbb954384a67028087a5f38da/log/job-output.txt#3092-309314:11
jrosserspatel: you can do `source /opt/ansible-runtime/bin/activate` then `pip freeze` to see if prettytable is installed there14:16
ebbex"ImportError: No module named posixpath" is this new for tox?14:18
*** yasemind has quit IRC14:24
*** Guest84754 has quit IRC14:28
*** Guest84754 has joined #openstack-ansible14:30
*** macz_ has joined #openstack-ansible14:31
ebbexfg14:35
*** macz_ has quit IRC14:36
spateljrosser i checked in source /opt/ansible-runtime/bin/activate  and did pip3 list (i didn't find prettytable)14:38
jrosserthat’s really interesting14:39
jrosserI think at the moment prettytable is an indirect dependency of other things we install14:40
jrosserso we don’t have it specifically in a requirements file14:40
spateljrosser check this out if i missing something - http://paste.openstack.org/show/803955/14:40
jrosseroooh I wonder if it comes in with ARA14:42
jrosserwhich you would not install outside a zuul job14:42
spatelhmm14:42
jrosserthat would make sense14:43
jrosserlooks like adding `prettytable>=0.7.2` in /opt/openstack-ansible/requirements.txt and re-doing bootstrap-ansible would be my first guess at a fix14:48
spateljrosser its not part of this list openstack_ansible.egg-info/requires.txt14:48
*** macz_ has joined #openstack-ansible14:48
spatelits not there in my repo /opt/openstack-ansible/requirements.txt14:49
spateljrosser it was there earlier so who removed that?14:49
jrossernot sure, it's still in the repo https://github.com/openstack/openstack-ansible/blame/master/requirements.txt14:50
spatel3 weeks ago when i install fresh OSA it was there but last night from master branch its missing14:50
spateli can't see here - /opt/openstack-ansible/requirements.txt14:50
spatelhere too -  https://github.com/openstack/openstack-ansible/blame/master/requirements.txt14:51
jrosserwell, i guess /opt/openstack-ansible should be a git repo on your system, so you should be able to use normal git commands to see if the file is deleted/changed14:52
spatelcheck this out - https://opendev.org/openstack/openstack-ansible/src/branch/master/requirements.txt14:52
spateljrosser - https://review.opendev.org/c/openstack/openstack-ansible/+/775126/1/requirements.txt14:55
spatellook like we removed it14:55
spatelThis causes the requirements-check job to fail as a result of14:55
spateluncapping PrettyTable in14:55
spatelmay be my knowledge is limited here :D14:58
*** kopecmartin has joined #openstack-ansible15:01
*** kopecmartin has left #openstack-ansible15:01
jrosserspatel: make a patch to put it back i'd say15:06
spatelwhat version ? prettytable>=0.7. ?15:06
spatelor same version what we removed15:06
spatelputting back PrettyTable<0.8,>=0.7.1 # BSD15:07
jrosseri think that may fail else we'd have not removed it in the first place15:07
spatelhmm15:08
jrosseri think what i suggested before `PrettyTable>=0.7.2`15:09
spatelok lets me add PrettyTable>=0.7.2 and will see if it pass or fail15:10
*** cloudnull has quit IRC15:17
openstackgerritSatish Patel proposed openstack/openstack-ansible master: Re-adding PrettyTable dependency  https://review.opendev.org/c/openstack/openstack-ansible/+/78331815:18
spateljrosser let me know if wording doesn't make sense15:18
jrosserlooks ok - thanks for the patch15:19
spatelnp15:19
spateljamesden_ around?15:24
spatelI am setting up my OVN lab and encounter this issue - neutron-server[12832]: 2021-03-26 15:22:33.999 12832 ERROR neutron.plugins.ml2.managers [-] No type driver for tenant network_type: geneve. Service terminated!15:24
spatelI am installing multi-node OSA using OVN15:24
spatelI do have tenant_network_types = geneve,vlan in ml215:25
spatelhmm it was missing here - type_drivers = flat,vlan,vxlan,local15:27
spatelinteresting15:27
spatelalso [ovn] section is missing in ml2_config.ini15:28
*** cloudnull has joined #openstack-ansible15:35
*** chkumar|ruck is now known as raukadah15:41
*** sc has joined #openstack-ansible15:46
*** jamesden_ has quit IRC15:50
*** jamesdenton has joined #openstack-ansible15:51
spateljrosser in this doc its saying create this file /etc/openstack_deploy/group_vars/network_hosts but network_hosts isn't a group in inventory so how does it work?15:51
*** cloudnull has quit IRC15:52
*** cloudnull has joined #openstack-ansible15:55
jrosserspatel: i have network_hosts as a group here15:57
spatelhmm let me check again15:57
spatelwhy i can't see it here python3 /usr/local/bin/inventory-manage.py -G15:59
jrosserspatel: for my not-OVN stuff it's where the neutron L3 agents are16:06
spatelhmm! let me check if i missed something or OVN behave differently16:07
*** dwilde has quit IRC16:09
spatelhmm i can see network_hosts on my all other non-OVN deployment but not with OVN16:12
spatelvery interesting thing, i am running aio-ovn on single machine and seeing this - http://paste.openstack.org/show/803962/16:13
spateloh this is metal deployment16:13
spatelnot container16:14
spateljrosser did you see this issue before - http://paste.openstack.org/show/803963/16:27
jrosserspatel: https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/78304716:29
spateloh16:30
spatellet me patch by hand and re-run16:31
spatelquick question for you how did you find that pretty quickly :)16:34
*** gyee has joined #openstack-ansible16:40
*** d34dh0r53 has joined #openstack-ansible16:41
CeeMacspatel: because he's a machine! He remembers all of the things! :D16:46
spateli have no doubt :)16:50
spateljamesdenton i have question if you see this17:39
spatelon compute nodes "ovn-sbctl show" just hanging not returning anything17:40
*** d34dh0r53 has quit IRC17:40
*** d34dh0r53 has joined #openstack-ansible17:40
*** d34dh0r53 has quit IRC17:42
*** d34dh0r53 has joined #openstack-ansible17:43
openstackgerritMerged openstack/openstack-ansible-os_senlin master: Remove references to unsupported operating systems  https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/77974918:02
gokhanithere is a minor bug in os_trove role. backup_strategy need to change from InnoBackupEx to innobackupex >> https://github.com/openstack/openstack-ansible-os_trove/blob/master/templates/trove-guestagent.conf.j2#L6918:03
*** jamesdenton has quit IRC18:09
*** jamesdenton has joined #openstack-ansible18:10
openstackgerritEbbex proposed openstack/openstack-ansible-tests stable/rocky: Remove temporary overrides  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/78330418:27
openstackgerritEbbex proposed openstack/openstack-ansible-tests stable/rocky: Pin virtualenv to version prior to 20.2.2  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/78341318:27
openstackgerritEbbex proposed openstack/openstack-ansible-lxc_hosts stable/rocky: Determine latest base image available  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/78273418:29
*** andrewbonney has quit IRC18:47
openstackgerritEbbex proposed openstack/openstack-ansible-tests stable/rocky: Pin virtualenv to version prior to 20  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/78341318:52
openstackgerritEbbex proposed openstack/openstack-ansible-tests stable/rocky: Remove temporary overrides  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/78330418:52
*** yasemind has joined #openstack-ansible19:03
openstackgerritSatish Patel proposed openstack/openstack-ansible-os_neutron master: Fixing doc typo  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/78346119:12
openstackgerritEbbex proposed openstack/openstack-ansible-os_tempest stable/rocky: Pin tempest-plugins to last tag supporting py2.7  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/78299019:47
*** gokhani has quit IRC19:49
spatelFinally my multi-node OVN working now!!20:24
spatelnext step is to setup SRIOV or DPDK and run some serious load to check L3 router performance and overall performance.20:26
*** spatel has quit IRC20:36
*** yasemind has quit IRC20:54
*** jamesdenton has quit IRC21:36
*** jamesden_ has joined #openstack-ansible21:36
*** jbadiapa has quit IRC22:59
*** luksky has quit IRC23:14
*** gyee has quit IRC23:31
*** macz_ has quit IRC23:32
*** gyee has joined #openstack-ansible23:32
*** tosky has quit IRC23:39
*** macz_ has joined #openstack-ansible23:48
*** macz_ has quit IRC23:53

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!