Friday, 2019-03-01

ianwhow did ansible break testinfra?  isn't is a separate process?00:00
ianwoohhh, no i get it, it talks to the hosts over ansible, that's right00:00
clarkbianw: I read it as testinfra using ansible programmatically then breaking00:01
clarkbnow ansible will probably say that isn't a stable interface00:01
clarkb(but it points to a need for such an interface and that people are doing it anyway)00:02
clarkbhttps://github.com/philpep/testinfra/blob/master/testinfra/utils/ansible_runner.py#L105-L117 is the code that is broken with newer ansible00:03
*** wolverineav has quit IRC00:03
*** wolverineav has joined #openstack-infra00:04
*** hwoarang has quit IRC00:06
clarkbhttps://github.com/ansible/ansible/commit/7e92ff823e2bf97ee33c1b7ff87bb8acecb5ae77#diff-bdd6c847fae8976ab8a7259d0b583f34 is what broke it. I think it is just called parser now not base_parser00:07
*** hwoarang has joined #openstack-infra00:07
corvusperhaps testinfra could/should use https://github.com/ansible/ansible-runner00:07
clarkboh neat00:08
clarkbya if the goal there is a stable itnerface it probably should00:08
*** wolverineav has quit IRC00:08
corvusyeah, that's what they wrote it for.  that's what awx is using to run ansible now.  zuul would have used it if it had been invented when we wrote 2.5.  and if we ever have to overhaul how we run ansible, that might be the time to switch.00:09
corvus(note, btw, what's doing the merging on that repo)00:10
*** yamamoto_ has quit IRC00:10
clarkbis it bad that I immediately scream in horror when I see it uses expect?00:10
* clarkb has bad memories of using perl expect to configure cisco switches00:10
clarkbalso neat zuul merging commits to that00:11
*** slaweq has joined #openstack-infra00:11
clarkbit is interesting that they went with the process boundary and licensed it apache200:11
clarkbI'll file a bug on testinfra if there isn't one already00:12
*** yamamoto has joined #openstack-infra00:14
clarkbhttps://github.com/philpep/testinfra/issues/401#issuecomment-468491634 and done00:15
*** slaweq has quit IRC00:15
ianwclarkb: hrm, what's with https://github.com/philpep/testinfra/pull/40400:17
ianw[LOOKING FOR SOMEONE TO TAKEOVER :-] replace ansible base_parser with create_base_parser #40400:18
clarkbianw: the individual that wrote the initial PR is looking for someone to take over fixing the bug I think00:18
*** tosky has quit IRC00:18
clarkbthey asked the maintainer for help00:18
clarkb(bceause it turned out the fix was more complicated than initially assumed)00:18
ianwyeah, more than just a rename00:18
*** yamamoto has quit IRC00:20
ianwoh dear, yeah that's getting right into the middle of ansible cli stuff00:20
ianwi'm not sure if the devel job pulls in testinfra from master to test any changes00:21
clarkbI don't think it does00:21
openstackgerritMerged openstack-infra/infra-specs master: letsencrypt spec  https://review.openstack.org/58728300:22
ianwno, just ara, openstacksdk  & ansible.  i think that might be a good first step ...00:22
openstackgerritIan Wienand proposed openstack-infra/project-config master: Add philpep/testinfra to Zuul  https://review.openstack.org/64022900:26
*** mriedem is now known as mriedem_away00:26
openstackgerritMerged openstack-infra/system-config master: Set ansible_python_interpreter for zuul-preview  https://review.openstack.org/64022100:27
*** dpawlik has joined #openstack-infra00:30
*** wolverineav has joined #openstack-infra00:31
openstackgerritIan Wienand proposed openstack-infra/system-config master: Add testinfra master to -devel job  https://review.openstack.org/64023100:31
*** hwoarang has quit IRC00:32
*** rascasoft has joined #openstack-infra00:32
*** hwoarang has joined #openstack-infra00:34
*** dpawlik has quit IRC00:34
*** trown is now known as trown|outtypewww00:39
*** rascasoft has quit IRC00:41
*** yamamoto has joined #openstack-infra00:46
*** yamamoto has quit IRC00:50
*** strigazi has quit IRC00:55
*** strigazi has joined #openstack-infra00:56
openstackgerritClark Boylan proposed openstack-infra/storyboard master: Update docs on how to run the tests locally  https://review.openstack.org/64023300:57
openstackgerritClark Boylan proposed openstack-infra/storyboard master: Update docs on how to run the tests locally  https://review.openstack.org/64023300:59
*** dpawlik has joined #openstack-infra01:02
*** whoami-rajat has joined #openstack-infra01:04
*** strigazi has quit IRC01:05
*** strigazi has joined #openstack-infra01:05
*** dpawlik has quit IRC01:07
*** slaweq has joined #openstack-infra01:11
*** hamzy has joined #openstack-infra01:11
*** slaweq has quit IRC01:16
pabelangerianw: clarkb: yah, some folks in ansible org recently found out license concern around testinfra. Somebody hacked up ansible-runner as replacement, but don't think it was pushed up01:17
pabelangerwill check tomorrow01:17
clarkbApparently it was being worked on01:19
pabelangerthis is an issue for ansible/molecule also, which is now operated by ansible01:19
ianwpabelanger: ok, cool, LMN, I just half-hacked ansible-runner in ...01:20
ianwi think one problem may be it's api wants a list of hosts from the inventory01:20
clarkbiamw there was a response to my comment01:20
clarkbsaying it is almost done?01:20
*** wolverineav has quit IRC01:21
pabelangerianw: just pinged somebody, but likely won't hear anything until tomorrow01:22
ianwoh, ok cool.  can't see any code, but interested how "get_hosts" will work if it doesn't want to duplicate ansible's inventory parsing01:22
ianwit's easy enough to do with "from ansible.inventory.manager import InventoryManager" but i think that's the root of the problem anyway :)01:22
pabelangerianw: https://github.com/ansible/ansible-runner/issues/20001:23
pabelangerah, jctanner already commenting01:23
pabelangerthat was who did the initial work in ansible org01:24
corvusianw: +3 on the add repo change01:24
*** ijw has quit IRC01:24
*** ijw has joined #openstack-infra01:25
ianwcorvus: cool, well whatever comes out of the above, we can test it :)01:25
*** wolverineav has joined #openstack-infra01:26
*** wolverineav has quit IRC01:26
*** wolverineav has joined #openstack-infra01:27
pabelangerianw: the PR above has a gist of the ansible-runner code for test-infra, I think once the ansible-runner bug is fixed, we can propose the update to test-infra01:27
pabelangerthen you'd be the hero of ansible/molecule :)01:27
*** rascasoft has joined #openstack-infra01:28
ianwoh, right ... https://gist.github.com/jctanner/512ec07171d95e61a5b7fd5f9103a301 is the actual code01:29
ianwand the answer to my question seems to be "cmd = 'ansible-inventory -i %s' % self.host_list"01:30
*** ijw has quit IRC01:30
openstackgerritMerged openstack-infra/project-config master: Add philpep/testinfra to Zuul  https://review.openstack.org/64022901:30
pabelangerianw: yup01:31
*** jamesmcarthur has joined #openstack-infra01:32
openstackgerritIan Wienand proposed openstack-infra/puppet-graphite master: Fix django wsgi typo  https://review.openstack.org/64023701:34
*** jamesmcarthur has quit IRC01:35
*** jamesmcarthur has joined #openstack-infra01:36
*** rascasoft has quit IRC01:40
*** jillr has left #openstack-infra01:51
*** wolverineav has quit IRC01:54
*** wolverineav has joined #openstack-infra01:57
*** wolverineav has quit IRC01:57
*** wolverineav has joined #openstack-infra01:57
*** ijw has joined #openstack-infra01:58
*** yamamoto has joined #openstack-infra02:00
*** jamesmcarthur has quit IRC02:01
*** tkajinam has quit IRC02:02
*** tkajinam has joined #openstack-infra02:03
*** jamesmcarthur has joined #openstack-infra02:04
*** tkajinam_ has joined #openstack-infra02:05
*** bhavikdbavishi has joined #openstack-infra02:06
*** roman_g has quit IRC02:07
*** tkajinam has quit IRC02:07
*** tkajinam_ has quit IRC02:10
*** tkajinam has joined #openstack-infra02:10
*** jamesmcarthur has quit IRC02:11
*** jamesmcarthur has joined #openstack-infra02:12
*** rascasoft has joined #openstack-infra02:16
*** jamesmcarthur has quit IRC02:20
*** jamesmcarthur has joined #openstack-infra02:20
*** ijw has quit IRC02:21
*** markvoelker has quit IRC02:26
*** rascasoft has quit IRC02:26
ianwpabelanger: well, i got it sort of working, and made a common pull request i hope we can work from -> https://github.com/philpep/testinfra/pull/41002:37
ianwi hope that can be updated by others but i get lost with github permissions :/02:38
*** jamesmcarthur has quit IRC02:39
*** mriedem_away has quit IRC02:40
*** wolverineav has quit IRC02:40
*** gyee has quit IRC02:42
*** hongbin has joined #openstack-infra02:46
*** rlandy has quit IRC02:49
openstackgerritMerged openstack-infra/puppet-graphite master: Fix django wsgi typo  https://review.openstack.org/64023702:53
*** sdake has joined #openstack-infra02:55
*** dpawlik has joined #openstack-infra03:03
*** yamamoto has quit IRC03:04
*** dpawlik has quit IRC03:08
*** slaweq has joined #openstack-infra03:11
*** psachin has joined #openstack-infra03:11
*** ramishra has joined #openstack-infra03:14
*** apetrich has quit IRC03:15
*** hwoarang has quit IRC03:15
*** slaweq has quit IRC03:15
*** hwoarang has joined #openstack-infra03:16
*** dave-mccowan has joined #openstack-infra03:16
*** bhavikdbavishi has quit IRC03:18
*** harlowja has quit IRC03:27
*** markvoelker has joined #openstack-infra03:27
*** rascasoft has joined #openstack-infra03:27
*** dpawlik has joined #openstack-infra03:35
*** hwoarang has quit IRC03:36
*** hwoarang has joined #openstack-infra03:38
*** janki has joined #openstack-infra03:39
*** dpawlik has quit IRC03:40
*** yamamoto has joined #openstack-infra03:48
*** markvoelker has quit IRC04:00
*** sthussey has quit IRC04:03
*** dave-mccowan has quit IRC04:04
*** bhavikdbavishi has joined #openstack-infra04:06
*** sdake has quit IRC04:08
*** ramishra has quit IRC04:09
*** ramishra has joined #openstack-infra04:09
*** hwoarang has quit IRC04:09
*** yamamoto has quit IRC04:10
*** slaweq has joined #openstack-infra04:11
*** wolverineav has joined #openstack-infra04:11
*** hwoarang has joined #openstack-infra04:15
*** slaweq has quit IRC04:16
ianwTags: <, c, l, a, s, s, , ', a, n, s, i, b, l, e, ., u, t, i, l, s, ., s, e, n, t, i, n, e, l, ., S, e, n, t, i, n, e, l, ', >04:16
ianwdmsimard: ^ http://logs.openstack.org/31/640231/1/check/system-config-run-base-ansible-devel/9c80e90/hosts/bridge.openstack.org/ara-report/result/beb3c691-9a12-41b7-bcc6-a373d5921cd3/04:17
*** wolverineav has quit IRC04:17
ianwinfra-root: if you can poke at http://graphite.opendev.org/ ... I think it's ready for production.  If I don't hear anything, on my Monday when it's nice and quiet I'll look at updating dns and doing a final sync to make it live04:26
*** hwoarang has quit IRC04:30
*** hwoarang has joined #openstack-infra04:35
*** hongbin has quit IRC04:40
openstackgerritIan Wienand proposed openstack-infra/system-config master: [dnm] run devel job against testinfra with ansible-runner  https://review.openstack.org/64025604:41
*** wolverineav has joined #openstack-infra04:42
*** ykarel|away has joined #openstack-infra04:47
*** ykarel|away is now known as ykarel04:47
*** yamamoto has joined #openstack-infra04:49
*** yamamoto has quit IRC04:54
*** markvoelker has joined #openstack-infra04:57
*** raukadah is now known as chandankumar05:09
*** markvoelker has quit IRC05:31
*** dpawlik has joined #openstack-infra05:36
*** dpawlik has quit IRC05:41
*** wolverineav has quit IRC05:43
*** hwoarang has quit IRC05:44
*** hwoarang has joined #openstack-infra05:46
openstackgerritIvoline Ngong proposed openstack-infra/storyboard master: Updated_at content modified  https://review.openstack.org/63884005:50
*** yamamoto has joined #openstack-infra05:53
*** yamamoto has quit IRC05:58
*** wolverineav has joined #openstack-infra06:05
*** slaweq has joined #openstack-infra06:11
*** dpawlik has joined #openstack-infra06:12
*** slaweq has quit IRC06:15
*** dpawlik has quit IRC06:17
*** flaper87 has joined #openstack-infra06:21
*** markvoelker has joined #openstack-infra06:28
*** hwoarang has quit IRC06:50
*** hwoarang has joined #openstack-infra06:51
*** armax has quit IRC06:54
*** wolverineav has quit IRC06:57
*** markvoelker has quit IRC07:00
*** quiquell|off is now known as quiquell07:01
*** yamamoto has joined #openstack-infra07:08
*** slaweq has joined #openstack-infra07:11
*** yamamoto has quit IRC07:13
*** slaweq has quit IRC07:15
*** dpawlik has joined #openstack-infra07:15
*** pgaxatte has joined #openstack-infra07:21
ianwpabelanger: packing it in now, but https://github.com/philpep/testinfra/pull/410/ is actually I think really close.  only two tests fail, that don't use the shell module07:22
ianwi think it has something to do with the output chunking ansible-runner does when it captures stdout.  it seems to overflow it's buffer or somehow drop stuff, and read() on the stdout fd it provides just starts half-way through the response07:23
ianwit's the type of thing mordred loves to debug ;)07:23
*** kopecmartin|off is now known as kopecmartin07:23
*** wolverineav has joined #openstack-infra07:25
*** e0ne has joined #openstack-infra07:28
*** slaweq has joined #openstack-infra07:32
ianwspeaking of mordred .. possible tox siblings doesn't work for github projects?07:32
ianwhttp://logs.openstack.org/56/640256/1/check/system-config-run-base-ansible-devel/6d0eda6/ara-report/result/bffb27b7-af76-4662-9803-2ce02f4b6446/07:32
ianwhttps://review.openstack.org/640256 -- adding testinfra master from github to the ansible-devel system-config job07:33
*** jtomasek has joined #openstack-infra07:37
*** sdake has joined #openstack-infra07:44
*** yamamoto has joined #openstack-infra07:46
*** hwoarang has quit IRC07:49
*** hwoarang has joined #openstack-infra07:50
*** auristor has quit IRC07:53
*** quiquell is now known as quiquell|brb07:54
*** markvoelker has joined #openstack-infra07:57
*** kjackal has joined #openstack-infra08:09
*** wolverineav has quit IRC08:10
*** ginopc has joined #openstack-infra08:11
*** apetrich has joined #openstack-infra08:12
*** auristor has joined #openstack-infra08:15
*** tkajinam has quit IRC08:18
*** ccamacho has joined #openstack-infra08:19
*** sdake has quit IRC08:23
*** gfidente has joined #openstack-infra08:24
*** quiquell|brb is now known as quiquell08:26
*** markvoelker has quit IRC08:31
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Implied branch matching must use fullmatch  https://review.openstack.org/64027208:33
*** iurygregory has joined #openstack-infra08:36
*** mcgiggler has joined #openstack-infra08:39
*** tosky has joined #openstack-infra08:42
*** jpena|off is now known as jpena08:48
*** ramishra has quit IRC08:52
*** jpich has joined #openstack-infra08:52
*** sdake has joined #openstack-infra08:54
*** yamamoto has quit IRC09:04
*** yamamoto has joined #openstack-infra09:05
*** panda|ruck|off is now known as panda|ruck09:05
openstackgerritRiju Khatri proposed openstack-infra/storyboard-webclient master: allows editing for any length of story title  https://review.openstack.org/64018709:09
*** IvensZambrano has joined #openstack-infra09:10
*** florianf has joined #openstack-infra09:10
*** roman_g has joined #openstack-infra09:11
*** priteau has joined #openstack-infra09:11
*** kashyap has left #openstack-infra09:15
*** wolverineav has joined #openstack-infra09:17
openstackgerritRiju Khatri proposed openstack-infra/storyboard-webclient master: Adds task title to new story modal  https://review.openstack.org/63996209:17
*** wolverineav has quit IRC09:22
*** ykarel is now known as ykarel|lunch09:27
*** markvoelker has joined #openstack-infra09:28
fnordahlconfig-core: would anyone have a spare moment for creating and importing the repositories in https://review.openstack.org/#/c/639066/ ?09:30
*** bhavikdbavishi has quit IRC09:32
*** bhavikdbavishi has joined #openstack-infra09:32
*** dtantsur|afk is now known as dtantsur09:37
*** derekh has joined #openstack-infra09:42
*** sshnaidm is now known as sshnaidm|off10:00
*** markvoelker has quit IRC10:01
*** electrofelix has joined #openstack-infra10:02
*** ykarel|lunch is now known as ykarel10:03
*** ttx has quit IRC10:07
*** ttx has joined #openstack-infra10:08
AJaegerfnordahl: want to review https://review.openstack.org/639831 as well? That needs your PTL +1.10:15
*** owalsh has quit IRC10:17
*** sdake has quit IRC10:20
*** eandersson has quit IRC10:20
*** e0ne has quit IRC10:22
*** e0ne has joined #openstack-infra10:23
zbrAJaeger: ianw: can you please look at https://review.openstack.org/#/c/639951/ on bindep.10:30
AJaegerzbr: I'm not a bindep core...10:30
*** owalsh has joined #openstack-infra10:31
zbrahh, right forgot. But i hope I addressed your original comment.10:34
zbrpabelanger: ^10:34
zbris kinda key feature that we need in bindep in order to install right packages on fedora/centos10:34
*** yamamoto has quit IRC10:45
*** bhavikdbavishi has quit IRC10:46
*** kazsh has quit IRC10:46
*** kazsh has joined #openstack-infra10:52
*** ramishra has joined #openstack-infra10:57
fnordahlAJaeger: looking10:58
*** markvoelker has joined #openstack-infra10:58
*** luizbag has joined #openstack-infra11:00
fnordahlAJaeger: re the review you linked I put a -1 on that as it is not ready to be added yet.  We will reach out to the contributor and discuss further steps.11:05
*** IvensZambrano has quit IRC11:05
AJaegerfnordahl: fine with me - thanks.11:05
*** e0ne has quit IRC11:11
*** IvensZambrano has joined #openstack-infra11:11
*** e0ne has joined #openstack-infra11:14
*** tbarron has joined #openstack-infra11:21
*** yamamoto has joined #openstack-infra11:21
openstackgerritMerged openstack-infra/project-config master: Add ``ceph-rbd-mirror`` charm and dependencies  https://review.openstack.org/63906611:22
fnordahlAJaeger: awesome, thank you for your prompt response and help.  Sorry for the inconvenience of the unreviewed governance and project-config changes11:24
*** IvensZambrano is now known as snevi11:24
sneviAJaeger: Hi, my team is working on a feature and we created a new repo change in https://review.openstack.org/#/c/638633/ not sure if we are missing another step to get the repo created. Is tat something you could help us with?11:27
*** bhavikdbavishi has joined #openstack-infra11:29
*** markvoelker has quit IRC11:30
*** yamamoto has quit IRC11:32
*** yamamoto has joined #openstack-infra11:35
*** yamamoto has quit IRC11:35
*** yamamoto has joined #openstack-infra11:43
*** yamamoto has quit IRC11:47
*** e0ne has quit IRC11:49
*** kjackal has quit IRC11:49
*** EmilienM is now known as EvilienM11:58
openstackgerritMerged openstack-infra/project-config master: Add rsd-virt-for-nova project  https://review.openstack.org/63863312:03
*** jchhatbar has joined #openstack-infra12:06
*** janki has quit IRC12:07
*** dave-mccowan has joined #openstack-infra12:10
*** yamamoto has joined #openstack-infra12:11
*** dave-mccowan has quit IRC12:15
*** panda|ruck is now known as panda|ruck|lunch12:16
*** priteau has quit IRC12:17
*** ykarel is now known as ykarel|afk12:18
*** ldnunes has joined #openstack-infra12:25
*** markvoelker has joined #openstack-infra12:28
*** kjackal has joined #openstack-infra12:38
*** yamamoto has quit IRC12:50
*** e0ne has joined #openstack-infra12:50
*** jpena is now known as jpena|lunch12:51
*** wolverineav has joined #openstack-infra12:54
*** wolverineav has quit IRC12:59
*** markvoelker has quit IRC13:01
*** panda|ruck|lunch is now known as panda|ruck13:04
*** trown|outtypewww is now known as trown13:16
*** rh-jelabarre has joined #openstack-infra13:17
*** kgiusti has joined #openstack-infra13:26
*** yamamoto has joined #openstack-infra13:27
*** yamamoto has quit IRC13:30
*** yamamoto has joined #openstack-infra13:30
*** ekultails has joined #openstack-infra13:30
*** mriedem has joined #openstack-infra13:32
*** eharney has quit IRC13:33
*** rlandy has joined #openstack-infra13:34
dtantsurhi folks! can someone give us some hints why publish jobs stopped running after https://github.com/openstack/ironic-python-agent/commit/cf1127e03c6069ca3b117ff0d05b9e083656fb99 ?13:35
openstackgerritRoman Gorshunov proposed openstack-infra/irc-meetings master: Update Airship meeting time  https://review.openstack.org/64035913:35
dtantsurit's not that they fail, I can no longer see runs on master at all in http://status.openstack.org/openstack-health/#/job/ironic-python-agent-buildimage-tinyipa13:35
dtantsuroh, I think I see them in http://zuul.openstack.org/builds?job_name=ironic-python-agent-buildimage-tinyipa13:37
AJaegersnevi: the repo is created now and you can use it, see http://git.openstack.org/cgit/openstack/rsd-virt-for-nova/ . The only missing part is adding the original reporter to the ACLs, an infra-root will need to do this for you. Then it's all self-service ;)13:38
mnaserinfra-root: there seems to be some NODE_FAILURE around the flavor we use for nested virt.. does anyone have a clue?13:38
fricklerdtantsur: there was an issue with log processing, so the last runs may not be have made their way into logstash correctly13:39
dtantsurah, I see. anyway, we know the problem now.13:39
fricklermnaser: where do you see this? happy to dig into some logs if you point me where to look13:40
mnaserim pinging the containers team.. appparently they've seen it often13:41
*** efried is now known as fried_rice13:45
AJaegerfrickler: did you see snevi's request above? Could you help, please?13:47
fricklersnevi: AJaeger: infra-root: added helena.mcgough@intel.com to rsd-virt-for-nova-core and rsd-virt-for-nova-release (re https://review.openstack.org/#/c/638633/)13:47
fricklerAJaeger: I just did that ;)13:47
AJaegerthanks13:48
pabelangerfrickler: adding it to statusbot is also helpful, gives a place we can look outside of irc13:48
sneviAJaeger: Frickler: Thanks you very much for the collaboration!!13:49
*** jamesmcarthur has joined #openstack-infra13:51
*** ykarel|afk is now known as ykarel13:52
pabelangerzbr: left comment on bindep review13:52
fricklerpabelanger: I was tagging i-r only to make it easy to see that the earlier request has been resolved. the action itself should be traceable in the gerrit audit log for those groups, so I don't think duplicating that in the status log will be useful13:53
*** jpena|lunch is now known as jpena13:54
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: [WIP] Add AuthorizationRule Configuration Object, claim validation  https://review.openstack.org/63985513:54
openstackgerritIvoline Ngong proposed openstack-infra/storyboard-webclient master: Sort search results by updated_at by default  https://review.openstack.org/63869013:55
*** bhavikdbavishi has quit IRC13:55
zbrpabelanger: replied. is not about presence of tool, is about official packager.13:56
zbrwe should not care if optional package managers are installed, only to document which one is the distro one.13:57
zbrpabelanger: mainly to replicate what is exposed by ansible_pkg_mgr variable.13:57
*** markvoelker has joined #openstack-infra13:58
*** sdake has joined #openstack-infra14:02
pabelangerzbr: I think the official pkg manager on rhel8 is yum, thats why I asked.  To keep backwards compat with rhel 7 things, trying to find docs but having a hard time14:02
*** sthussey has joined #openstack-infra14:03
*** eernst has joined #openstack-infra14:08
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: [WIP] Add AuthorizationRule Configuration Object, claim validation  https://review.openstack.org/63985514:08
*** TheJulia is now known as needssleep14:14
*** ykarel_ has joined #openstack-infra14:16
*** mcgiggler has quit IRC14:17
zbrpabelanger: not really, yum executable is only a placeholder for dnf in r8, i double checked now.14:17
*** snevi has quit IRC14:17
*** snevi has joined #openstack-infra14:17
*** cmurphy is now known as cmorpheus14:17
*** eernst has quit IRC14:17
pabelangerzbr: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8-beta/html/8.0_beta_release_notes/new-features#software_management_214:18
pabelangeraccording to ansible human, yum is still the default, so I think we should keep bindep using that on rhel814:18
pabelanger(maybe centos-8 too)14:18
*** ykarel has quit IRC14:19
*** jchhatbar has quit IRC14:19
*** sdake has quit IRC14:19
*** eharney has joined #openstack-infra14:19
zbrpabelanger: what does ansible returns on it?14:20
pabelangerzbr: checking that now14:20
evrardjpcan I have a few votes on https://review.openstack.org/#/c/639779/ ?14:21
evrardjpthank you!14:21
evrardjp:D14:21
*** sdake has joined #openstack-infra14:21
*** sdake has quit IRC14:24
*** jpich has quit IRC14:24
*** chandankumar is now known as raukadah14:26
*** jpich has joined #openstack-infra14:27
*** ykarel_ is now known as ykarel14:29
*** markvoelker has quit IRC14:31
pabelangerzbr: it seems ansible will use /usr/bin/yum, but facts will return dnf. It is pretty confusing right now. So, given the docs for rhel8 say yum is the default, I think we need to do the same in bindep14:31
pabelangerwhich, yum4 really is dnf14:32
pabelangerit is a mess :(14:32
zbrpabelanger: this will totally break the entire purpose of the feature.14:32
zbrpabelanger: we rely on getting "dnf" on it, not yum. so we rely on ansible behavior, we have tons of code relying on this.14:33
*** ccamacho has quit IRC14:33
AJaegerzbr, so, rename it - do you really need dnf/yum - or do you want "Recent Red Hat family member"14:33
zbrlike "if dnf install python3-foo else install python-foo" kind of logic.14:33
*** ccamacho has joined #openstack-infra14:34
pabelangerbut on rhel8, the default is yum install python3-foo14:36
pabelangeryou optionally could use dnf for that14:36
pabelangerthe main reason, was to keep backwards compat for all the user legacy bash scripts14:37
pabelangerI left a -1 for now, mostly to get ianw input. But happy to revist.14:37
pabelangerbut, don't think we should update bindep to dnf, if not the default pkgmgr14:37
*** kjackal has quit IRC14:37
zbrpabelanger: can you do a quick bj session?14:38
*** sdake has joined #openstack-infra14:39
pabelangernot currently, have another meeting coming up14:39
pabelangerhappy to keep discussing it here14:39
AJaegerpabelanger: the change is not using dnf for installation - bindep does not install anything. It's only an atom to add to bindep.txt.14:40
*** roman_g has quit IRC14:40
pabelangerAJaeger: right, but14:41
pabelanger# dnf covers CentOS>=8, Fedora>=28, RHEL>=814:41
pabelangerhttps://review.openstack.org/#/c/639951/8/bindep/tests/bindep.txt14:42
*** roman_g has joined #openstack-infra14:42
pabelangerthat is not true, yum also covers RHEL>814:42
pabelangerso, we either need to support both yum / dnf for rhel814:42
pabelangeror maybe default to yum on rhel814:42
AJaegerpabelanger, zbr, I think saying dnf when you mean "CentOS>=8, Fedora>=28, RHEL>=8" is confusing.14:43
AJaegerwe wouldn't have this discussion if the atom would be called "centos8_fedora_28_rhel_8_or_newer"14:44
pabelangeryah14:44
AJaegerbut I'm bad in naming ;(14:44
pabelangerI basically think, platform:redhat platform:dnf and platform:redhat platform:yum would match RHEL814:45
pabelangerbecause, both pkgmgr are avaiable14:45
*** bnemec is now known as beekneemech14:46
*** luizbag has quit IRC14:46
AJaegerso, in other words: Let's rename...14:46
zbrpabelanger: lets make ansible do the first step, and fix the ansible_pkg_mgr,... we can follow with bindep after ;)14:49
zbri am writing a document about the issue now, so we can all work on a good solution14:49
*** ccamacho has quit IRC14:51
*** quiquell is now known as quiquell|off14:51
*** ccamacho has joined #openstack-infra14:53
*** snevi has quit IRC14:54
*** snevi has joined #openstack-infra14:54
*** tobias-urdin is now known as tobeass-urdin15:03
*** sdake has quit IRC15:10
*** ekultails has quit IRC15:14
*** wolverineav has joined #openstack-infra15:15
*** roman_g has quit IRC15:17
*** wolverineav has quit IRC15:20
*** bhavikdbavishi has joined #openstack-infra15:20
openstackgerritCamila Moura proposed openstack-infra/storyboard master: Fixes the Python 3 dependencies and fixes compatibility issues in migration  https://review.openstack.org/63971615:22
*** dpawlik has quit IRC15:24
openstackgerritCamila Moura proposed openstack-infra/storyboard master: Fixes the Python 3 dependencies and iterator  https://review.openstack.org/63971615:28
*** markvoelker has joined #openstack-infra15:28
*** ekultails has joined #openstack-infra15:31
openstackgerritCamila Moura proposed openstack-infra/storyboard master: Fixes the Python 3 dependencies and iterator  https://review.openstack.org/63971615:31
*** chason_ has quit IRC15:36
*** jistr is now known as jistr|mtg15:38
*** e0ne has quit IRC15:38
*** e0ne has joined #openstack-infra15:39
*** chason has joined #openstack-infra15:41
*** armax has joined #openstack-infra15:44
*** chason has quit IRC15:46
*** betherly has joined #openstack-infra15:48
*** chason has joined #openstack-infra15:52
*** betherly has quit IRC15:53
*** kjackal has joined #openstack-infra15:55
*** sdake has joined #openstack-infra15:58
*** markvoelker has quit IRC16:01
*** jistr|mtg is now known as jistr16:02
*** ykarel is now known as ykarel|away16:03
*** sdake has quit IRC16:03
*** sdake has joined #openstack-infra16:04
*** kjackal has quit IRC16:13
*** e0ne has quit IRC16:14
*** e0ne has joined #openstack-infra16:15
*** e0ne has quit IRC16:17
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Open port 80 on zuul-preview  https://review.openstack.org/64042516:21
*** kmalloc is now known as needscoffee16:22
*** roman_g has joined #openstack-infra16:22
*** yamamoto has quit IRC16:23
*** sdake has quit IRC16:23
openstackgerritJames E. Blair proposed openstack-infra/zone-opendev.org master: Switch www.opendev.org to load-balanced gitea farm  https://review.openstack.org/64042816:25
corvusmordred, clarkb, fungi: ^ those two very simple changes fill me with hope and anticipation16:26
*** yamamoto has joined #openstack-infra16:26
*** roman_g has quit IRC16:26
*** roman_g has joined #openstack-infra16:27
clarkbthis just occurred to me reviewing the www CNAME change, do we need to worry about lack of ipv6 there for ipv6 only clouds (I suppose those clouds will ipv4 through nat and maybe that is good enough since zuul pushes code into them now and they shouldn't fetch from the gitfarm)16:28
corvusclarkb: yes, also, we expect ipv6 before we really go into production.16:29
clarkb++16:29
clarkbI'll work on a the change to have gerrit replicate to those new gitea servers shortly16:29
corvusclarkb: thanks!16:30
*** yamamoto has quit IRC16:31
clarkblooks like gitea-lb01 is redirecting to https://opendev.org?16:32
corvusclarkb: yeah, if you hit it on http16:33
corvushttps should work as expected16:33
openstackgerritMerged openstack-infra/zuul-jobs master: Handle multiple docker images with the same repository  https://review.openstack.org/63977916:36
*** psachin has quit IRC16:38
openstackgerritClark Boylan proposed openstack-infra/system-config master: Replicate to the standalone gitea severs  https://review.openstack.org/64043116:38
clarkbinfra-root I think ^ is what we want for gitea stanalone server replication16:38
corvusclarkb: imma gonna edit the commit msg16:40
clarkbsure16:40
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Replicate to the standalone gitea severs  https://review.openstack.org/64043116:41
mordredcorvus: I was just about to start working on porting the project creation stuff - did that already get done?16:42
corvusclarkb: lgtm.  i marked it wip while we wait on mordred's task to update project creation :)16:42
mordredok. cool. :)16:42
mordredjust making sure16:42
corvus++16:42
*** pgaxatte has quit IRC16:45
*** ykarel|away has quit IRC16:45
*** ykarel|away has joined #openstack-infra16:46
openstackgerritMerged openstack-infra/zone-opendev.org master: Switch www.opendev.org to load-balanced gitea farm  https://review.openstack.org/64042816:48
*** jtomasek has quit IRC16:52
*** ijw has joined #openstack-infra16:55
*** ijw has quit IRC16:55
*** ijw has joined #openstack-infra16:56
dmsimardcorvus: regarding ^ I noticed that www.opendev.org properly redirects to gitea but "opendev.org" redirects me to "https://38.108.68.64/", is that patch intended to fix that ?16:57
*** ginopc has quit IRC16:57
*** markvoelker has joined #openstack-infra16:58
clarkbdmsimard: that IP addr is the gitea running in k8s16:59
openstackgerritJames E. Blair proposed openstack-infra/zone-opendev.org master: Switch opendev.org to load-balanced gitea farm  https://review.openstack.org/64043816:59
corvusdmsimard: nope.  but that one ^ is.  thanks.  :)16:59
dmsimardcorvus: cool, ++16:59
corvusand actually... we should probably have www.opendev.org point to something that redirects to opendev.org17:00
clarkbcorvus: I don't think @ can be a cname17:00
corvusthis is why we have review :)17:00
corvusi don't think we want to end up with people cloning https://www.opendev.org/openstack/nova/17:00
clarkbcorvus: haproxy may be able to do that, but we'd have to terminate ssl in haproxy17:01
corvusclarkb: re cname -- why not?  (which rule does this violate)17:01
*** kaiokmo has joined #openstack-infra17:01
corvusclarkb: ah, it's the no other resource type rule17:02
clarkbcorvus: its the CNAME uniquenss rule17:02
corvuscan't have cname + ns17:02
clarkbyup17:02
*** iurygregory has quit IRC17:02
openstackgerritJames E. Blair proposed openstack-infra/zone-opendev.org master: Switch opendev.org to load-balanced gitea farm  https://review.openstack.org/64043817:04
*** gyee has joined #openstack-infra17:04
clarkbmnaser: NODE_FAILURE means that all clouds that could provide a label failed to do so17:05
clarkbmnaser: we currently have our retries per cloud set to 3. So for each cloud region that can provide a label we try three times before moving on to the next region. If no regions were able to satisfy the request then you get a NODE_FAILURE17:06
*** yamamoto has joined #openstack-infra17:06
*** ykarel|away has quit IRC17:06
openstackgerritJames E. Blair proposed openstack-infra/zone-opendev.org master: Remove www name  https://review.openstack.org/64044417:07
corvusclarkb: ^ i think maybe we'll need to point it to a static apache server somewhere; i say we pull it until that happens17:07
clarkbwfm +2'd both changes17:08
clarkbI did quickly pull up haproxies docs and it can do redirects for us if we want to have it terminate ssl (at least for that name)17:09
*** luizbag has joined #openstack-infra17:09
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Create git repos on gitea servers before gerrit  https://review.openstack.org/64044517:10
mordredcorvus: how does that look? ^^17:10
*** yamamoto has quit IRC17:11
*** bhavikdbavishi has quit IRC17:11
clarkbinfra-root https://review.openstack.org/#/c/630391/6 switches all our dev servers (and some names that I don't think actually exist but we can clean that up in a followup) to puppet4. I'm happy to watch that go in today if we can get reviews on it17:11
openstackgerritMerged openstack-infra/zone-opendev.org master: Switch opendev.org to load-balanced gitea farm  https://review.openstack.org/64043817:12
openstackgerritMerged openstack-infra/zone-opendev.org master: Remove www name  https://review.openstack.org/64044417:12
mordredcorvus: also - do you have an env setup already that would make testing that locally easy? (I should get one, but I do not currently have one setup)17:13
*** kopecmartin is now known as kopecmartin|off17:15
corvusmordred: nothing special, i'm just running a gitea docker container like this: http://paste.openstack.org/show/746612/17:17
*** bhavikdbavishi has joined #openstack-infra17:17
corvusmordred: though that should be "image: gitea/gitea"17:17
clarkboh that reminds me port 222 is the ssh port on the new servers right?17:17
clarkbI need a new patch on my chagne then17:17
corvusclarkb: ha, yes :)17:18
*** snevi has quit IRC17:18
mordredcorvus: that doesn't have a mysql with it though, right? just uses built-in sqlite or something?17:18
mordredI should probably docker-compose the docker-compose - cause it's the running of the sql query I'm the most worried about17:19
mordredbecause quoting17:19
corvusoh yeah.  that's correct.17:19
corvusmordred: you can probably crib from the one in system-config.17:19
corvusmordred: tiny suggestion inline on patch17:21
*** panda|ruck is now known as panda|ruck|off17:21
openstackgerritClark Boylan proposed openstack-infra/system-config master: Replicate to the standalone gitea severs  https://review.openstack.org/64043117:21
corvusmordred: er, actually, i updated that suggestion, and now i'm a little unsure :)17:22
mordredyeah - I was about to ask if we needed to hit the 300017:22
corvusi think so, and it's ssl, so i don't know if that's going to work17:22
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Create git repos on gitea servers before gerrit  https://review.openstack.org/64044517:22
corvusmordred: i'll dig into that if you want to keep working on the local mariadb testing17:23
mordredcorvus: maybe it needs to be https://localhost:300017:23
openstackgerritMerged openstack-infra/system-config master: Open port 80 on zuul-preview  https://review.openstack.org/64042517:25
*** gfidente has quit IRC17:26
corvusmordred: okay, i think we have to use "https://localhost:3000" and add "validate_certs: false" to all the uri entries17:26
mordredcorvus: k. doing that. I'm also making one change to the sql command17:27
corvusi'll go update my rename patch to do the same :)17:28
mordredcorvus: I tested the sql using ansible to run a different command against gitea01.opendev.org17:28
clarkbcan we tell gitea to also listen on http but only do that on locahost so we don't expose it? I guess that isn't any better than https localhost with no cert validation17:28
mordredso the plumbing at least works17:28
corvusclarkb: i don't think so.  and yeah, i don't think it's worth it.17:28
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Create git repos on gitea servers before gerrit  https://review.openstack.org/64044517:29
mordredcorvus: I thnik that might actually be solid now17:29
clarkbwhat are repo types 2,3,5,7?17:30
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Add gitea to project rename playbook  https://review.openstack.org/64021817:30
clarkbthe prime repos17:31
*** markvoelker has quit IRC17:31
*** trown is now known as trown|lunch17:31
corvusit's a repo_unit type17:31
corvusi don't remember exactly which is which, but it's things like wiki, issues, etc17:31
clarkbgotcha17:32
corvus7 is issues, you can see because we're tweaking it there17:32
corvusso here's a thing...17:32
*** roman_g has quit IRC17:32
corvusthe swagger api didn't support setting these things, so we used sql17:33
corvusthe swagger api also doesn't support renaming or transferring.  but instead of using sql for that, i hit the UI web routes17:33
mordredcorvus: we should make the some patches to update the swagger api for all of those things :)17:34
corvusi think that's the better choice for renaming -- that's a complex procedure that touches lots of sql data.  it will certainly be easier to keep up with any form parameter changes than it would be to keep up with schema changes17:34
mordredyou know - next time we want to write some go17:34
corvuswe could probably adapt the repo_unit sql queries to use the same ui web route approach that i did for renaming17:35
corvusmordred: yes, that is definitely the right thing to do long-term17:35
clarkbso they aren't defining the api with swagger just the api clients?17:35
mordredcorvus: I think I'm comfortable with the sql route for this for now - it's already there and whatnot17:35
clarkbiirc swagger will do both sides if you let it then there isn't a mismatch17:35
corvusmordred: yeah.  i think i am too.  if it gives us problems, we could switch it, but if it works, it's fine, and we can just leave it until there's a swagger api.17:35
mordred++17:36
corvusclarkb: afaict, the swagger api is completely different than the UI17:36
*** ramishra has quit IRC17:37
corvusso when you push the "update" button on the settings page, it hits /org/repo/settings.  when you use the swagger api, it's at /api/v1/repo/yadda/yadda17:37
dtantsurhi folks! can someone please help me understanding what we're doing wrong in http://zuul.openstack.org/build/4319b34ff84c4bc789b950214db650f3 ?17:37
clarkbcorvus: huh17:37
dtantsurthis is a part of the attempt to move our publish job to zuul v317:37
mordreddtantsur: you didn't pay me enough money17:38
dtantsurmordred: I can exchange the chocolate medal you promised me for reviewing the Image nightmare in the SDK :)17:38
clarkbdtantsur: http://logs.openstack.org/5b/5b6bf0b6c86aad352db271cf530a6321ad4248eb/post/ironic-python-agent-buildimage-tinyipa/4319b34/ara-report/reports/7178e260-15c1-4f00-9b5a-5dad4352f301.html shows zero tasks run and that playbook is what does the sync to the executor looks like17:39
clarkbdtantsur: I'm not sure why that is happening but would explain the dir not existing17:39
dtantsurclarkb: this also raised my attention.. what "hosts" should I put for it? maybe that's the reason?17:39
dtantsurthe zuul v2 variant had hosts:primary, so we cargo-culted it17:40
clarkbdtantsur: your job specifies the nodeset it wants to use which sets the inventory names17:40
*** betherly has joined #openstack-infra17:40
clarkbin this case you might just want 'all' ?17:40
fungidtantsur: for a single-node job you can usually just get by with "all" for anything running on the job node17:40
corvuserm17:41
fungiand "localhost" for anything running on the executor server17:41
corvusis that an auto-converted playbook?17:41
dtantsurcorvus: not really. it was converted manually by people who have little clue about zuul v3 :)17:41
corvusdtantsur: throw it out :)17:41
dtantsurLOL17:42
* dtantsur throws his laptop out the window17:42
corvusnow it works!17:42
dtantsur\o/17:42
dtantsurso, if you don't value your eyes too much, the resulting post playbook is https://github.com/openstack/ironic-python-agent/blob/master/playbooks/ironic-python-agent-buildimage/post.yaml17:42
corvusdtantsur: that looks like it was at least based on something that the autoconvert script made.  the thing to know about that is that it produces complete gibberish, and no one understands how it works (including its authors).  and it can usually be replaced with one line of yaml anyway.17:43
fungibut yeah, "primary" is generally a reference to legacy nodesets where we used that to emulate the old style of nodepool multi-node groups which had a separate primary node17:43
dtantsurcorvus: I'm eager to learn this one line of yaml17:44
corvusyeah, that's the trick17:44
corvuslemme dig around a bit and understand the job more17:44
dtantsurcorvus: tl;dr build ironic-python-agent images and push them to tarballs.o.o17:45
*** sdake has joined #openstack-infra17:45
clarkbmordred: couple of thoughts for minor improvements on https://review.openstack.org/#/c/640445/3 but +2'd as should work as is17:46
corvusdtantsur: any reason you don't build the image in check?17:46
dtantsurcorvus: and publish from check?17:46
corvusdtantsur: no, just to see if it buids17:47
dtantsurcorvus: ah. yes, we do it as part of the CI runs.17:47
dtantsurbut we only publish in post17:47
* mordred needs to afk for a few real quick to get a sandwich before a phone call in an hour - will BRB17:49
*** betherly has quit IRC17:50
clarkbfrickler: if you are still around, can you review https://review.openstack.org/#/c/639740/ this is the edit secrets file command script change17:50
corvusdtantsur: so for that post playbook which isn't running -- http://logs.openstack.org/5b/5b6bf0b6c86aad352db271cf530a6321ad4248eb/post/ironic-python-agent-buildimage-tinyipa/4319b34/ara-report/17:51
corvusdtantsur: can you walk me through that?17:51
*** florianf has quit IRC17:51
corvusdtantsur: like, what is it that the run playbook produces, and what is it you want to end up on the fileserver?17:51
dtantsurcorvus: they key part is this bunch of files https://github.com/openstack/ironic-python-agent/blob/master/playbooks/ironic-python-agent-buildimage/run.yaml#L42-L44 that should end up in https://tarballs.openstack.org/ironic-python-agent/tinyipa/files/17:52
dtantsurcorrection: UPLOAD_TAR should end up in https://tarballs.openstack.org/ironic-python-agent/tinyipa/, UPLOAD_RAW - in files17:53
dtantsurthis was supposed to be done by https://github.com/openstack/ironic-python-agent/blob/master/playbooks/ironic-python-agent-buildimage/post.yaml#L10-L30 (and it worked with old style jobs)17:53
*** betherly has joined #openstack-infra17:54
corvusdtantsur: it's unfortunate that mordred left, because he and pabelanger were working on a simplified method of fetching artifacts from remote hosts, however, that's in the fetch-output role, which runs in the base job post playbook, which runs *after* the artifact upload playbook in openstack-artifacts.yaml.  so i do not know how they intend for that to be used.17:55
*** mriedem is now known as mriedem_lunch17:56
* dtantsur knew that mordred was to blame17:56
dtantsurcorvus: do you think that changing to hosts:all will at least provide a relief?17:56
corvusdtantsur: absent an explanation there, i think the shortest path to working is the direction clarkb was heading -- get that post playbook running again by using hosts:all17:56
dtantsurokay, let us try this17:57
corvusdtantsur: that's still a really scary playbook.  i hope later we can update it so that it just moves some files into {{ansible_user_dir}}/artifacts and then somehow use fetch-output to pull it to the executor.17:58
dtantsurI'll be happy to, but some guidance will be required17:59
corvusyeah, i don't know the plan either :(17:59
clarkbinfra-root https://review.openstack.org/#/c/636734/ is a change that I don't want to get completely lost (even if we decide not to go with it). It will reduce the number of github api roundtrips we make by stopping our constant retrieval of api rate limit data17:59
corvusi think maybe fetch-output works for logs, but probably hasn't been used for artifacts yet18:00
dtantsurthanks anyway corvus18:00
* dtantsur has proposed a patch and now has to go18:00
*** dtantsur is now known as dtantsur|afk18:00
*** harlowja has joined #openstack-infra18:01
*** ykarel|away has joined #openstack-infra18:03
*** jpena is now known as jpena|off18:03
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Create git repos on gitea servers before gerrit  https://review.openstack.org/64044518:04
corvusmordred, clarkb: ^ fixed the docs error18:04
*** jaosorior has quit IRC18:04
clarkbinfra-root Zhenguo Niu confirms that nimble is no longer needed and that we should feel free to do the pypi ownership swap. I'll be reaching out to devin directly to get details about pypi users and can add them as owner as soon as I have that info18:05
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Clone project-config to /opt on gitea servers  https://review.openstack.org/64047918:06
corvusmordred, clarkb: ^ and that's a minor followup to mordred's change18:06
*** IvensZambrano has joined #openstack-infra18:07
*** betherly has quit IRC18:07
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Create git repos on gitea servers before gerrit  https://review.openstack.org/64044518:07
corvussorry, just adding task headers18:07
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Clone project-config to /opt on gitea servers  https://review.openstack.org/64047918:08
*** ijw has quit IRC18:08
*** eharney has quit IRC18:09
clarkbcmorpheus: if you have a moment for a puppet review, your input on https://review.openstack.org/#/c/639442/ would be helpful. Not sure if there is a simpler way to do that in puppet18:09
*** jpich has quit IRC18:10
*** wolverineav has joined #openstack-infra18:10
*** irclogbot_1 has joined #openstack-infra18:10
*** wolverineav has quit IRC18:10
*** wolverineav has joined #openstack-infra18:10
*** betherly has joined #openstack-infra18:11
*** IvensZambrano has quit IRC18:13
cmorpheusclarkb: hmm it's weird that path        => '/usr/local/bin:/usr/bin:/bin/ doesn't just work18:14
clarkbinfra-root while we've got DNS context paged in https://review.openstack.org/#/c/622620/2 should make reboots of our name servers more reliable18:15
clarkbcmorpheus: ya I think the issue is in production pip is installed by puppet and by then puppet has already cached PATH contents?18:15
openstackgerritMerged openstack-infra/system-config master: Document kerberos stash file requirement  https://review.openstack.org/63877918:15
*** betherly has quit IRC18:15
fungisomething about how the pip provider decides how/where to invoke pip, i think?18:16
fungii used to know some of the internals there because of our pip3 provider fork18:16
*** betherly has joined #openstack-infra18:17
fungithere were some fancy rules which determined the name and path to pip (since it also might not just be called "pip")18:17
clarkbfungi: in this case we exec pip so not going through that code18:17
clarkbbut I think it likely is a similar problem18:17
fungiahh18:18
funginot using a pip package provider then18:18
clarkbcorvus: thanks I'll reboot one of the ns servers after that is applied to triple check it works as freedesktop says18:20
*** jamesmcarthur has quit IRC18:20
*** jamesmcarthur has joined #openstack-infra18:20
cmorpheusi wonder if using the environment option to set PATH would work better18:20
*** derekh has quit IRC18:22
mordredcorvus: do you want me to switch the defaults from tmp to opt still?18:22
*** wolverineav has quit IRC18:23
mordredcorvus: oh - i see - you did that as a followup18:24
*** jamesmcarthur has quit IRC18:24
*** wolverineav has joined #openstack-infra18:25
*** mrhillsman is now known as mrhillsman_lunch18:28
*** markvoelker has joined #openstack-infra18:28
*** betherly has quit IRC18:30
*** ijw has joined #openstack-infra18:31
clarkbmwhahaha: looks like requirements tarball publishing jobs are running successfully on the master branch now18:34
mwhahahayea saw that18:34
clarkbI don't see chagnes for the stable branches so I'll push them up now18:34
mwhahahakolla got broke by some other neutron requirements thing18:34
mwhahahahttps://review.openstack.org/#/c/639872/18:34
mwhahahaseems to have broken a few other projects as well18:35
clarkbocata, pike, queens and rocky branches in requirements all have that fix cherry picked and proposed now18:37
*** trown|lunch is now known as trown18:37
*** betherly has joined #openstack-infra18:37
*** betherly has quit IRC18:42
*** yboaron_ has quit IRC18:42
fungigonna go grab lunch, then come back and hopefully get started writing up the git redirect sites change. bbiab18:43
*** betherly has joined #openstack-infra18:45
*** ijw has quit IRC18:46
prometheanfireclarkb: guess it was needed :P18:46
*** bhavikdbavishi has quit IRC18:46
clarkbprometheanfire: ya18:47
*** ykarel|away has quit IRC18:48
*** ijw has joined #openstack-infra18:50
*** wolverineav has quit IRC18:50
openstackgerritMerged openstack-infra/system-config master: Update nsd systemd unit deps  https://review.openstack.org/62262018:51
corvusmordred: http://site.926bb0aaddad4bc3853269451e115dcb.openstack.zuul-preview.opendev.org/18:51
corvusJpMaxMan: ^18:51
*** electrofelix has quit IRC18:52
JpMaxManwoah!18:52
JpMaxMannice :)18:52
corvusthat came from https://review.openstack.org/63592418:52
corvusstill a little bit more to do to plug it all in so that zuul reports that url18:52
*** Adri2000 has joined #openstack-infra18:53
corvusbut all the pieces are in place :)18:53
JpMaxManthat's great news18:53
JpMaxManvery cool little tool!!18:53
*** wolverineav has joined #openstack-infra18:53
AJaegercorvus: woot!18:53
*** yamamoto has joined #openstack-infra18:54
*** betherly has quit IRC18:57
mordredcorvus: \o/18:59
*** yamamoto has quit IRC18:59
*** sdake has quit IRC19:00
clarkbrereviwing the zuul-preview stuff, the zuul-preview image must install apache right?19:01
corvusclarkb: yes19:01
clarkb(we aren't running apache as a separte image)19:01
*** betherly has joined #openstack-infra19:01
*** markvoelker has quit IRC19:01
*** betherly has quit IRC19:07
*** ijw has quit IRC19:08
*** e0ne has joined #openstack-infra19:12
*** wolverineav has quit IRC19:12
*** kjackal has joined #openstack-infra19:21
*** rfolco|rover has quit IRC19:21
*** luizbag has quit IRC19:24
*** ijw has joined #openstack-infra19:25
*** betherly has joined #openstack-infra19:26
*** mriedem_lunch is now known as mriedem19:28
*** mrhillsman_lunch is now known as mrhillsman19:29
*** wolverineav has joined #openstack-infra19:30
*** betherly has quit IRC19:30
*** e0ne has quit IRC19:31
openstackgerritRiju Khatri proposed openstack-infra/storyboard-webclient master: resolves task 26745 of story 2003883 : fixed project group view checkboxes  https://review.openstack.org/64049619:32
*** ijw has quit IRC19:33
*** auristor has quit IRC19:34
*** eharney has joined #openstack-infra19:35
*** auristor has joined #openstack-infra19:36
*** betherly has joined #openstack-infra19:39
*** tosky has quit IRC19:42
*** tosky has joined #openstack-infra19:43
mordredclarkb: that's right - because apache actually spawns zuul-preview as a subprocess and talks to it over stdin/stdout19:43
mordredclarkb: so apache is actually the main entrypoint of the container / what the container is running19:44
mordredand zuul-preview is part of the confif19:44
mordredconfig19:44
*** dmellado has quit IRC19:44
*** dmellado has joined #openstack-infra19:45
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Return npm site location as artifact  https://review.openstack.org/64049819:45
corvusmordred: https://opendev.org/ exists!19:46
corvusmordred: i've approved https://review.openstack.org/64044519:47
mordredcorvus: fingers crossed!19:48
*** betherly has quit IRC19:48
*** irclogbot_1 has quit IRC19:50
*** ijw has joined #openstack-infra19:55
openstackgerritMerged openstack-infra/puppet-etherpad_lite master: Bump eplite version in rspec tests  https://review.openstack.org/61100019:55
*** wolverineav has quit IRC19:56
*** rlandy is now known as rlandy|brb19:57
*** wolverineav has joined #openstack-infra19:57
*** markvoelker has joined #openstack-infra19:58
*** ijw has quit IRC20:00
*** irclogbot_1 has joined #openstack-infra20:01
*** wolverineav has quit IRC20:02
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Remove Sign In link from gitea  https://review.openstack.org/64050320:02
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Remove Sign In link from gitea  https://review.openstack.org/64050320:02
dmsimardcorvus: gitea is empty for me, it wasn't earlier today though20:02
clarkbdmsimard: yes 640445 will address that (as well as my followup to replicate to the new servers)20:03
dmsimard++ you guys are way ahead of me :p20:03
mordredcorvus: ^^ I think that should do20:03
mordredit20:04
*** e0ne has joined #openstack-infra20:05
fungiokay, i'm back and probably ready to get some stuff done. trying to catch up now20:06
fungiooh, netlify progress!20:07
clarkbinfra-root ns2 has bene updated with the systemd fix. Any opposition to me rebootingthat server now to triple check ti comes up with a running named?20:07
fungiall the exciting stuff i almost miss by being a slave to sustenance20:07
mordredclarkb: ++20:07
mordredfungi: stupid sustenance20:07
* fungi notes https://opendev.org/ is one character longer than the equivalent gh url. hopefully properly free and open software is worth typing one more character20:10
mordredfungi: my biggest issue is that when my fingers start typing open they have a habit of then typing stack20:11
corvusyep.  i am going through retraining now.20:11
fungithere was some astronomical cost related to registering open.dev or the osf would have considered that as an option for us20:11
*** EvilienM is now known as EmilienM20:14
*** betherly has joined #openstack-infra20:16
mordredoh I'm sure that one was spendy20:16
openstackgerritMerged openstack-infra/system-config master: Create git repos on gitea servers before gerrit  https://review.openstack.org/64044520:16
mordredBOOM20:16
openstackgerritMerged openstack-infra/system-config master: Clone project-config to /opt on gitea servers  https://review.openstack.org/64047920:17
*** sdake has joined #openstack-infra20:18
clarkbmy big issue is tab is far less useful for me20:18
*** sdake has quit IRC20:18
clarkbok rebooting ns2.opendev.org now20:18
fungiclarkb: we can probably register opentab.org if it helps you20:20
*** betherly has quit IRC20:21
mordredfungi: could we just start selling Tab instead of building clouds?20:21
fungithe 80s will thank us20:22
mordredfungi: you may or may not find it amusing that your nick is currently Tab colored in my irc client20:22
clarkbhrm nsd didn't start because it couldn't bind to those interfaces20:22
clarkbnow I wonder if network-online.target is just fubar on ubuntu20:23
*** sdake has joined #openstack-infra20:24
clarkbaccording to the logs ens3 was configured before nsd starts20:26
*** derekh has joined #openstack-infra20:26
clarkboh but the ens3 logs only show ipv4 being online20:27
clarkbI wonder if network-online is any ip version is up and running20:27
clarkband then we fail when we try to bind to ipv6 address20:27
*** wolverineav has joined #openstack-infra20:28
clarkbmaybe it would be simpler to not unbound on those servers and listen globally?20:28
clarkbI'll have to think about that more20:28
fungimordred: come back to me when it's moxie-colored20:29
clarkbDevin says they aren't ready to take over the nimble name now but will reach out to me when they are20:30
*** markvoelker has quit IRC20:31
fungiclarkb: hrm, is unbound trying to bind to interfaces other than loopback? or nsd trying to bind to loopback?20:31
fungian alternative would be to configure one of them to not use actual port 53 and redirect on the relevant interface with iptables (1:1 nat basically)20:32
clarkbfungi: nsd is trying to bind to our actual public interfaces because nsd is on localhost so you can't bind :: as nsd20:32
clarkber because unbound is on localhost20:32
fungiright20:32
clarkband if we could bind to :: we wouldn't have to worry about network startup because :: is magical in the linux kernel and just works (tm)20:33
fungii know that was the original idea. curious where the conflict is20:33
clarkbthe conflict is nsd starts before we have networking fully up20:33
clarkbso fails to start20:33
*** wolverineav has quit IRC20:33
openstackgerritMerged openstack-infra/zuul-jobs master: Return npm site location as artifact  https://review.openstack.org/64049820:33
*** betherly has joined #openstack-infra20:33
clarkbTo address this I switched the unit to start after network-online.target and reading the docs my undertanding is that is only waiting for ipv4 to compe up20:34
fungioh, i see, it tries to start before our public interface is configured?20:34
clarkbpossibly because we wait for the RA to happen every 30 seconds or whatever it is so our ipv6 addr isn't there yet20:34
clarkbfungi: yes20:34
openstackgerritJames E. Blair proposed openstack-infra/zuul-preview master: Use artifact name rather than log_url  https://review.openstack.org/64050920:35
*** rfolco has joined #openstack-infra20:37
*** betherly has quit IRC20:37
*** betherly has joined #openstack-infra20:38
fungiclarkb: do we listen by interface name or by address?20:40
mnasereh20:41
mnaseras someone who has been transforming all infra to ipv20:41
mnaseripv620:41
clarkbfungi: ipaddress20:42
*** sdake has quit IRC20:42
mnaseri strongly suggest configuring things as static and disabling autoconf20:42
mnaserit has been so flakey for me :(20:42
clarkbmnaser: via config drive data?20:42
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Add windmill-config project  https://review.openstack.org/64051020:42
fungiglean can do that for us at least, right?20:42
*** yamamoto has joined #openstack-infra20:42
mnaserclarkb: yeah.  plop it right into /etc/network/interfaces or whatever20:43
fungion an unrelated note, were we expecting to do something to hide the "sign in" link at https://opendev.org/ ?20:43
mnaserwhat we do is actually we have an ansible role that gets ansible_ipv4_default or whatever and creates an ifcfg file from it20:43
openstackgerritJames E. Blair proposed openstack-infra/zuul-preview master: Use artifact name rather than log_url  https://review.openstack.org/64050920:43
clarkbfungi: it can if we boot with config drive. I think up to date cloud init can too but that isn't present on xenail and older by default. unsure of bionic20:43
clarkbfungi: yes there are some chagnes above20:43
clarkbfungi: https://review.openstack.org/64050320:43
fungithanks!!!20:44
* fungi feels so out of touch lately20:44
fungiwow, "Remove Sign In link from gitea" involves adding 126 lines of html. clearly they didn't intend to make this optional :/20:46
*** betherly has quit IRC20:47
*** yamamoto has quit IRC20:48
clarkbfungi: I think we are replacing the html20:48
clarkband then we'll bake it in via an image rebuild20:48
fungiyeah20:49
fungithat's what it looks like to me anyway20:50
fungihence my "not optional" comment20:50
*** rlandy|brb is now known as rlandy20:51
*** wolverineav has joined #openstack-infra20:51
*** kgiusti has left #openstack-infra21:03
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Fix typo in docker push role  https://review.openstack.org/64051521:06
*** jamesmcarthur has joined #openstack-infra21:07
corvusfungi: yes, but they *did* intend for us to replace the html as needed :)21:08
*** jamesmcarthur has quit IRC21:09
*** jamesmcarthur has joined #openstack-infra21:10
*** betherly has joined #openstack-infra21:11
corvusclarkb: can you re-review https://review.openstack.org/638794 ?21:11
clarkbcorvus: done. And now my turn for food21:12
fungicorvus: fair point21:13
*** jamesmcarthur has quit IRC21:14
*** ldnunes has quit IRC21:15
openstackgerritMerged opendev/base-jobs master: Save registry logs  https://review.openstack.org/63879421:19
*** betherly has quit IRC21:23
*** betherly has joined #openstack-infra21:27
*** markvoelker has joined #openstack-infra21:28
*** jamesmcarthur has joined #openstack-infra21:29
*** ijw has joined #openstack-infra21:32
mordredclarkb, corvus: what am I missing:21:34
mordred2019-03-01 21:24:47,587 p=11773 u=root |  fatal: [gitea04.opendev.org]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'file'. Error was a <class 'ansible.errors.AnsibleError'>, original message: could not locate file in lookup: /opt/project_config/gerrit/projects.yaml"}21:34
corvusmordred: my first thought is: file lookups are local21:34
mordredbut I see /opt/project_config ...21:35
mordredOH21:35
mordredyeah21:35
mordredso really this can skip cloning project_config and can just read from the one on bridge21:35
corvusmordred: yeah, i think so21:35
mordredfix coming21:35
mordredcorvus: tahnks - I would not have gotten there for a while21:35
corvusmordred: i've hit it enough it's a reflex21:36
mordredcorvus: while you're fixing thigns for me - do you know where my other cast iron skillet is?21:36
*** reed has joined #openstack-infra21:36
*** betherly has quit IRC21:36
*** irclogbot_1 has quit IRC21:36
corvusmordred: check in the fire21:36
reedit's been a while :)21:36
* reed waves21:36
corvusreed: howdy!21:37
*** betherly has joined #openstack-infra21:39
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Remember that lookups are local  https://review.openstack.org/64053721:39
mordredcorvus: ^^21:40
openstackgerritMerged openstack-infra/zuul-jobs master: Fix typo in docker push role  https://review.openstack.org/64051521:40
corvusmordred: is that going to update it to master each time it runs?21:41
mordredcorvus: yes?21:47
mordredcorvus: we might want force: yes added like we do for system-config21:48
*** tobiash has quit IRC21:49
*** tobiash has joined #openstack-infra21:51
*** betherly has quit IRC21:52
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Remember that lookups are local  https://review.openstack.org/64053721:55
mordredcorvus: how about that instead?21:55
*** betherly has joined #openstack-infra21:55
*** markvoelker has quit IRC22:01
*** wolverineav has quit IRC22:04
*** wolverineav has joined #openstack-infra22:05
*** jamesmcarthur has quit IRC22:07
*** betherly has quit IRC22:08
*** jamesmcarthur has joined #openstack-infra22:08
*** rlandy has quit IRC22:10
mordredclarkb, fungi: ^^ feel like a +A?22:11
clarkbmordred: looking (sorry I've been nerd sniped using zuul-preview rust version to learn about moving variables and borrowing and all that fun stuff. It turns out that closures are a really good playground for that)22:12
clarkbmordred: did the other hosts need that clone to not be local?22:13
clarkboh no we just needed the ref22:13
mordredclarkb: yah. so we didnt' clone because we didnt' need to22:14
clarkbmordred: left a comment you might want to address (but +2'd if you want to followup)22:15
mordredclarkb: I'll just followup22:16
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Update playbook doc comment for correctness  https://review.openstack.org/64054522:17
*** rh-jelabarre has quit IRC22:18
*** kjackal has quit IRC22:19
openstackgerritJosef Wells proposed openstack-infra/zuul master: quickstart: web and others wait on mysql to start  https://review.openstack.org/64054822:22
*** e0ne has quit IRC22:24
*** ijw has quit IRC22:25
*** ijw has joined #openstack-infra22:29
*** yamamoto has joined #openstack-infra22:31
*** wolverineav has quit IRC22:35
*** yamamoto has quit IRC22:35
*** jamesmcarthur has quit IRC22:40
*** jamesmcarthur has joined #openstack-infra22:44
*** ajo_ has joined #openstack-infra22:45
*** sadin999_ has joined #openstack-infra22:45
*** jamespage_ has joined #openstack-infra22:45
*** zaro_ has joined #openstack-infra22:45
*** guilhermesp__ has joined #openstack-infra22:45
*** abelur_ has joined #openstack-infra22:45
*** odyssey4me_ has joined #openstack-infra22:45
*** beisner_ has joined #openstack-infra22:45
*** mwhahaha_ has joined #openstack-infra22:45
*** jbryce_ has joined #openstack-infra22:45
*** seongsoocho_ has joined #openstack-infra22:45
*** betherly has joined #openstack-infra22:46
clarkbshould I remove the WIP from https://review.openstack.org/#/c/640431/ ?22:46
clarkbthat won't take effect until we restart gerrit so might want to get that in place concurrently with the work to create the projects?22:47
corvusclarkb: we need the projects to be created before we replicate22:47
corvussame as git.o.o22:47
*** ijw_ has joined #openstack-infra22:47
*** Tengu_ has joined #openstack-infra22:47
mordredcorvus: yeah - but I think he's saying we have to restart gerrit for it to go live - so we could go ahead and land the patch22:47
corvusand if gerrit is restarted before then, we have a big cleanup job.  i'd rather wait22:48
*** ijw has quit IRC22:48
mordred++22:48
clarkbI'm fine with waiting.22:48
*** jamesmcarthur has quit IRC22:48
mordredI can go ahead and review it though22:48
*** wolverineav has joined #openstack-infra22:50
*** eharney has quit IRC22:52
*** Tengu has quit IRC22:52
*** zaro has quit IRC22:52
*** HenryG has quit IRC22:52
*** odyssey4me has quit IRC22:52
*** nicolasbock has quit IRC22:52
*** mwhahaha has quit IRC22:52
*** vkmc has quit IRC22:52
*** xgerman has quit IRC22:52
*** eglute has quit IRC22:52
*** sadin999 has quit IRC22:52
*** lamt has quit IRC22:52
*** seongsoocho has quit IRC22:52
*** guilhermesp has quit IRC22:52
*** jbryce has quit IRC22:52
*** melwitt has quit IRC22:52
*** beisner has quit IRC22:52
*** ajo has quit IRC22:52
*** Ng has quit IRC22:52
*** abelur has quit IRC22:52
*** jamespage has quit IRC22:52
*** cjloader has quit IRC22:52
*** sadin999_ is now known as sadin99922:52
*** guilhermesp__ is now known as 18WAAFJGJ22:52
*** abelur_ is now known as abelur22:52
*** beisner_ is now known as beisner22:52
*** ajo_ is now known as ajo22:52
*** seongsoocho_ is now known as seongsoocho22:52
*** odyssey4me_ is now known as odyssey4me22:52
*** zaro_ is now known as zaro22:52
*** jbryce_ is now known as jbryce22:52
*** jamespage_ is now known as jamespage22:52
*** melwitt has joined #openstack-infra22:53
*** vkmc has joined #openstack-infra22:53
*** mwhahaha_ is now known as mwhahaha22:53
*** kazsh has quit IRC22:55
*** Ng has joined #openstack-infra22:58
*** kazsh has joined #openstack-infra22:59
*** eharney has joined #openstack-infra22:59
*** markvoelker has joined #openstack-infra22:59
*** betherly has quit IRC22:59
*** jesusaur has quit IRC23:03
fungiyeah, safe to review, not really safe to approve far in advance23:05
fungibut we have some control over when it takes effect23:05
clarkb640537 should head to the gate soonish and when that merges we should create projects in gitea aiui23:06
clarkbthen we can either restart gerrit late today or do it on monday?23:06
*** betherly has joined #openstack-infra23:06
*** frickler has quit IRC23:06
*** frickler has joined #openstack-infra23:06
*** rascasoft has quit IRC23:07
clarkbinfra-root anyone else want to review https://review.openstack.org/#/c/630391/6 to do more puppet 4, https://review.openstack.org/639740 script for passwords/secrets editing, https://review.openstack.org/636734 reduce number of github requests that zuul does23:07
fungioh, wow, i took an unvoluntary friday nap and nearly missed a wayward reed!23:08
*** mriedem is now known as mriedem_afk23:08
*** jesusaur has joined #openstack-infra23:08
clarkbI'm going to pop out for a bit now but will be back to watch project creation and/or that set of chagnes above23:08
fungi(for the record, clarkb's rust excuse was waaay better)23:09
clarkbfungi I took the script and removed the to_string I thought wasnt necessary then went down the rabbit hole of learning why it was necessary23:10
*** betherly has quit IRC23:11
fungisounds like a marvellous lesson i won't learn until i attempt to do the same at some point23:12
*** jesusaur has quit IRC23:14
*** betherly has joined #openstack-infra23:16
*** ekultails has quit IRC23:20
mordredit's important to know why to_string is necessary23:23
*** jesusaur has joined #openstack-infra23:23
*** betherly has quit IRC23:25
clarkb17 minutes to merging the project creation fix23:29
*** betherly has joined #openstack-infra23:29
*** markvoelker has quit IRC23:31
*** hwoarang has quit IRC23:39
*** betherly has quit IRC23:43
openstackgerritMerged openstack-infra/system-config master: Remember that lookups are local  https://review.openstack.org/64053723:45
clarkbmordred: ^ ansible run after the current should include that23:48
clarkbwill start at 00:00UTC23:48
*** hwoarang has joined #openstack-infra23:49
*** wolverineav has quit IRC23:55
*** wolverineav has joined #openstack-infra23:55
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Fix buildset registry  https://review.openstack.org/64055723:56
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Fix docker image retag  https://review.openstack.org/64055823:56
corvusclarkb, fungi, mordred: ^ that's a bit of a redo on the buildset registry thing.  it turns out that the second issue (640558) caused the buildset registry proxy to fail to serve an image at all, which is what alerted me to the issue corrected by 640557.23:57
corvuswe need to land both of those, and then i can delete the current zuul-preview image on dockerhub, then we should be able to land that change and push it up.23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!