| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/diskimage-builder master: Move common ubuntu environment setting to ubuntu-common element https://review.openstack.org/563731 | 00:04 |
|---|---|---|
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/diskimage-builder master: Remove redundant sources change/update https://review.openstack.org/563739 | 00:05 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/diskimage-builder master: Add ubuntu-systemd-container operating-system element https://review.openstack.org/563748 | 00:05 |
| *** tosky has quit IRC | 00:09 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Move prepare-zanata-client to o-z-j https://review.openstack.org/525760 | 00:09 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Move prepare-zanata-client to o-z-j https://review.openstack.org/525760 | 00:19 |
| *** longkb has joined #openstack-infra | 00:29 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for mysql opportunistic test bug 1793364 https://review.openstack.org/603874 | 00:36 |
| openstack | bug 1793364 in OpenStack Compute (nova) "mysql db opportunistic unit tests timing out intermittently in the gate (bad thread switch?)" [High,Confirmed] https://launchpad.net/bugs/1793364 | 00:36 |
| *** gyee has quit IRC | 00:42 | |
| *** imacdonn has quit IRC | 00:49 | |
| *** hongbin_ has joined #openstack-infra | 01:00 | |
| *** admcleod has quit IRC | 01:00 | |
| *** yankcrime has quit IRC | 01:00 | |
| *** dalvarez has quit IRC | 01:00 | |
| *** cloudnull has quit IRC | 01:00 | |
| *** vabada has quit IRC | 01:00 | |
| *** tobias-urdin has quit IRC | 01:00 | |
| *** Guest58757 has joined #openstack-infra | 01:04 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/diskimage-builder master: Add ubuntu-systemd-container operating-system element https://review.openstack.org/563748 | 01:04 |
| *** admcleod has joined #openstack-infra | 01:05 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/diskimage-builder master: ubuntu-common: Update default DIB_RELEASE to bionic https://review.openstack.org/603940 | 01:08 |
| *** mriedem_away has quit IRC | 01:08 | |
| *** imacdonn has joined #openstack-infra | 01:09 | |
| *** slaweq has joined #openstack-infra | 01:11 | |
| *** imacdonn has quit IRC | 01:14 | |
| *** vkmc|afk is now known as vkmc | 01:15 | |
| *** imacdonn has joined #openstack-infra | 01:15 | |
| *** slaweq has quit IRC | 01:16 | |
| *** yumiriam has quit IRC | 01:23 | |
| *** jamesmcarthur has joined #openstack-infra | 01:25 | |
| *** mrsoul has joined #openstack-infra | 01:25 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for ansible ssh rsync fail bug 1793370 https://review.openstack.org/603878 | 01:57 |
| openstack | bug 1793370 in OpenStack-Gate ""Collect sphinx build html" fails with "rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]" on limestone nodes" [Undecided,Confirmed] https://launchpad.net/bugs/1793370 | 01:57 |
| *** jamesmcarthur has quit IRC | 02:05 | |
| *** armax has joined #openstack-infra | 02:05 | |
| *** dtruong has quit IRC | 02:10 | |
| *** jamesmcarthur has joined #openstack-infra | 02:22 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: update query 1449136 to match some more queries https://review.openstack.org/603905 | 02:26 |
| *** ijw has quit IRC | 02:30 | |
| *** jamesmcarthur has quit IRC | 02:32 | |
| *** jamesmcarthur has joined #openstack-infra | 02:39 | |
| *** sshnaidm has joined #openstack-infra | 02:42 | |
| *** sshnaidm is now known as sshnaidm|pto | 02:42 | |
| *** eernst has quit IRC | 02:56 | |
| *** eernst has joined #openstack-infra | 03:00 | |
| *** hongbin_ has quit IRC | 03:02 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/elastic-recheck master: convert docs to PTI https://review.openstack.org/559396 | 03:10 |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** eernst has quit IRC | 03:15 | |
| *** slaweq has quit IRC | 03:16 | |
| *** eernst has joined #openstack-infra | 03:19 | |
| *** eernst_ has joined #openstack-infra | 03:20 | |
| *** eernst has quit IRC | 03:20 | |
| SpamapS | does anybody else use gertty in gnome terminal? I can't get F-keys to work at all in Ubuntu 18.04 on gnome terminal | 03:24 |
| *** eernst_ has quit IRC | 03:25 | |
| dmsimard | infra-root: can we mute the channel by default in #openstack-registered ? I've seen too many typos leading to people writing their nickserv password on there. | 03:29 |
| dmsimard | With the topic and a message from freenode supplying https://freenode.net/kb/answer/registration, we're probably covered | 03:31 |
| *** jamesmcarthur has quit IRC | 03:32 | |
| *** bobh has quit IRC | 03:45 | |
| *** udesale has joined #openstack-infra | 03:53 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move coverage jobs to voting check job https://review.openstack.org/432836 | 03:56 |
| *** bobh has joined #openstack-infra | 03:58 | |
| SpamapS | n/m.. hardware problem | 04:00 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move coverage jobs to voting check job https://review.openstack.org/432836 | 04:00 |
| * SpamapS just discovered that MS natural keyboards default to the stupid extended key meanings not F keys | 04:00 | |
| *** jamesmcarthur has joined #openstack-infra | 04:00 | |
| StevenK | SpamapS: There's an Fn button, a'la laptop keyboards? | 04:03 |
| *** bobh has quit IRC | 04:03 | |
| *** ykarel has joined #openstack-infra | 04:05 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul master: Add a zuul-local tool for testing playbooks https://review.openstack.org/603967 | 04:07 |
| SpamapS | StevenK: there's an [F]Lock button, that I have to press when I want my keyboard to be un-stupid | 04:07 |
| SpamapS | only once per boot | 04:07 |
| SpamapS | but I have to remember to hit it | 04:07 |
| SpamapS | until I get a good keyboard | 04:07 |
| *** ijw has joined #openstack-infra | 04:10 | |
| *** bobh has joined #openstack-infra | 04:10 | |
| ianw | Failed connect to mirror.bhs1.ovh.openstack.org:80; No route to host | 04:10 |
| ianw | hrm | 04:10 |
| *** ijw has quit IRC | 04:14 | |
| *** ramishra has joined #openstack-infra | 04:15 | |
| ianw | a whole bunch of my jobs failed due to that, but it seems intermittent | 04:15 |
| *** bobh has quit IRC | 04:15 | |
| *** dayou_ has joined #openstack-infra | 04:19 | |
| *** bobh has joined #openstack-infra | 04:21 | |
| *** bobh has quit IRC | 04:26 | |
| AJaeger | SpamapS: isn't there a way to tell your bootup scripts to do the right thing for the keyboard? | 04:27 |
| AJaeger | ianw, want to +2 the coverage change again - https://review.openstack.org/432836 ? | 04:27 |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/603282/ and https://review.openstack.org/603199 (will recheck once 603282 is in), please? | 04:28 |
| *** bobh has joined #openstack-infra | 04:30 | |
| *** yamamoto has quit IRC | 04:35 | |
| *** yamamoto has joined #openstack-infra | 04:35 | |
| *** yamamoto has quit IRC | 04:36 | |
| *** bobh has quit IRC | 04:38 | |
| *** olivierb has joined #openstack-infra | 04:48 | |
| *** olivierb has quit IRC | 04:49 | |
| *** jamesmcarthur has quit IRC | 04:49 | |
| ianw | hrm, apache on mirror01.bhs1.ovh.openstack.org_access.log seems very suspiciously quiet | 04:54 |
| ianw | i can get to it fine, but it looks like requests from OVH can't? | 04:55 |
| ianw | Connecting to mirror01.bhs1.ovh.openstack.org (mirror01.bhs1.ovh.openstack.org)|158.69.80.87|:80... failed: No route to host. | 04:56 |
| ianw | yep ... gosh darn it | 04:57 |
| ianw | i guess as step 1 i'll reboot the mirror | 04:57 |
| *** janki has joined #openstack-infra | 05:02 | |
| *** auristor has quit IRC | 05:03 | |
| *** jaosorior has quit IRC | 05:03 | |
| *** jamesmcarthur has joined #openstack-infra | 05:03 | |
| ianw | nope :( | 05:04 |
| ianw | step 2, i'll actually shut it down, see if it gets a new hypervisor or something | 05:04 |
| *** auristor has joined #openstack-infra | 05:05 | |
| *** jamesmcarthur has quit IRC | 05:08 | |
| ianw | no luck either | 05:08 |
| *** jamesmcarthur has joined #openstack-infra | 05:09 | |
| *** yamamoto has joined #openstack-infra | 05:10 | |
| *** jamesmcarthur has quit IRC | 05:14 | |
| *** jamesmcarthur has joined #openstack-infra | 05:20 | |
| *** ykarel has quit IRC | 05:20 | |
| ianw | trying to talk to ovh support but it's not going well | 05:23 |
| *** jamesmcarthur has quit IRC | 05:24 | |
| ianw | i can't authenticate fully because i don't know the billing address, so they can't really help me. not that i blame them as i'm sure they get lots of crap | 05:25 |
| ianw | ohhhh "A maintenance operation is scheduled for the BHS1 region on 19th September 2018, between 03:00a.m. and 08:00a.m. UTC-4" | 05:26 |
| ianw | wait wait wait, this was the region clarkb paused ... hrm, this is starting to look more suspicious | 05:27 |
| *** jtomasek has joined #openstack-infra | 05:32 | |
| *** Guest58757 is now known as cloudnull | 05:32 | |
| *** alexchadin has joined #openstack-infra | 05:39 | |
| *** ykarel has joined #openstack-infra | 05:40 | |
| *** quique|rover|off is now known as quiquell|rover | 05:45 | |
| *** gfidente has joined #openstack-infra | 05:49 | |
| *** jamesmcarthur has joined #openstack-infra | 05:52 | |
| *** aojea has joined #openstack-infra | 05:53 | |
| *** chkumar|off is now known as chkumar|ruck | 05:53 | |
| AJaeger | ianw: yes, that was the region we paused - and restarted. | 05:55 |
| AJaeger | dpawlik: are you still around and can help with BHS1? | 05:56 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Disable OVH BHS1 region https://review.openstack.org/603988 | 05:57 |
| ianw | AJaeger: ^ i think we should probably do this for now. it's just making the gate even worse. | 05:57 |
| ianw | as mentioned, i managed to raise a ticket with them. i suspect rebuilding the mirror may help ... but i don't want to delete the existing mirror in case it can be helpful in debugging it | 05:58 |
| AJaeger | ianw: if it's that bad, I suggest you force merge directly - enqueue it directly or eve nadd a +Verified yourself as admin. | 05:59 |
| AJaeger | ianw: really sad with that high load but might be the best for now. | 06:00 |
| *** hashar has joined #openstack-infra | 06:00 | |
| ianw | usually project-config changes go in quickly cause they're separate, i'll give it a little | 06:00 |
| ianw | i might try spinning up a new mirror and see if it communicates ... i bet it will, and will probably be what we require in the end anyway | 06:01 |
| AJaeger | ianw: the check queue will take ages ;( | 06:01 |
| AJaeger | ianw: that's why I suggested to at least promote directly to gate... | 06:02 |
| ianw | hrm, yeah ok, i can do that ... | 06:03 |
| AJaeger | youngest change with running jobs is 30 mins old - so we would have to wait for check queue alone 30 min (I thought it was worse - yesterday it was several hours) | 06:03 |
| ianw | i think that did it | 06:04 |
| AJaeger | yep, thanks | 06:05 |
| ianw | i'm bringing up mirror02.bhs1.ovh.openstack.org now ... we can see if hosts can get to that | 06:06 |
| AJaeger | thanks, ianw | 06:06 |
| *** alexchadin has quit IRC | 06:08 | |
| *** holser_ has joined #openstack-infra | 06:09 | |
| ianw | Last exception: error creating clone | 06:11 |
| ianw | either there's something else wrong, or we don't have enough quota | 06:11 |
| *** jamesmcarthur has quit IRC | 06:12 | |
| ianw | hrm, it seems we have heaps of quota | 06:16 |
| openstackgerrit | Merged openstack-infra/project-config master: Disable OVH BHS1 region https://review.openstack.org/603988 | 06:16 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/elastic-recheck master: convert docs to PTI https://review.openstack.org/559396 | 06:22 |
| AJaeger | #status log disabled bhs1.ovh again since mirror is not reachable | 06:23 |
| openstackstatus | AJaeger: finished logging | 06:23 |
| *** jaosorior has joined #openstack-infra | 06:25 | |
| *** diablo_rojo has quit IRC | 06:25 | |
| ianw | ok, it looks like a can not launch a "hg-7" flavor node in OVH; it even fails via their webui | 06:29 |
| ianw | 215dc5d2-94ac-417b-8676-c6009d82839e & 860f0b36-c160-4ae7-b3c9-3e822f8710f4 for the record | 06:32 |
| ianw | i also can not launch an eg-7 flavor ... these are all given an asterisk "* older generation" so i'm guessing that might be related ... | 06:33 |
| *** quiquell|rover is now known as quique|rover|brb | 06:36 | |
| *** dpawlik_ has joined #openstack-infra | 06:38 | |
| *** dpawlik has quit IRC | 06:39 | |
| ianw | ARRRGGHHH. shade thinks it has an ipv6 address, but the actual host doesn't. launch.py then bails out trying to ping6 | 06:43 |
| *** dpawlik_ has quit IRC | 06:45 | |
| *** dpawlik has joined #openstack-infra | 06:45 | |
| *** slaweq has joined #openstack-infra | 06:47 | |
| ianw | ok, all for nothing it seems. even the newly built mirror can't be accessed from another host in bhs1 | 06:47 |
| ianw | i'll add info to 603988 just to keep it together | 06:47 |
| prometheanfire | which packages? http://logs.openstack.org/46/602446/4/check/openstack-infra-base-integration-gentoo-17-0-systemd/3cb5709/job-output.txt.gz#_2018-09-20_05_25_29_294672 | 06:57 |
| ianw | prometheanfire: ^ unsurpisingly that ran in ovh-bhs1, which is the region we've just disabled because the mirror is broken | 06:59 |
| prometheanfire | well, gentoo doesn't use mirrors | 07:00 |
| prometheanfire | see one task up :P | 07:00 |
| *** rcernin has quit IRC | 07:02 | |
| ianw | prometheanfire: interesting, does "package:" in ansible work on gentoo? | 07:04 |
| ianw | http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/tests/configure-mirrors.yaml#n10 | 07:04 |
| ianw | that test should probably be skipped if that message above triggers ... | 07:04 |
| prometheanfire | emacs matches against.... virtual/emacs app-editors/emacs | 07:04 |
| prometheanfire | ya | 07:06 |
| prometheanfire | DIB removes the portage 'cache' on build to shrink the image | 07:06 |
| prometheanfire | it's optional though | 07:06 |
| prometheanfire | You can set the GENTOO_PORTAGE_CLEANUP environment variable to False to prevent cleaning up portage repositories (including overlays). This makes the image bigger. | 07:06 |
| ianw | or you need to do the gentoo equivalent of "apt-get update"? | 07:07 |
| ianw | i'm pretty sure we have a package refresh like that in the base playbooks, somewhere | 07:07 |
| prometheanfire | emerge-webrsync | 07:08 |
| prometheanfire | or emaint sync -a | 07:08 |
| prometheanfire | or emerge --sync | 07:08 |
| prometheanfire | how often are the images built? | 07:09 |
| *** quique|rover|brb is now known as quiquell|rover | 07:09 | |
| ianw | daily, but i'd suggest finding where we do the package refresh in the base playbooks and adding that personally | 07:10 |
| prometheanfire | yarp | 07:10 |
| *** ginopc has joined #openstack-infra | 07:11 | |
| prometheanfire | heh, well, at least the ansible module to see if installed will work if it matches more than one thing | 07:11 |
| prometheanfire | https://github.com/ansible/ansible/blob/a8d4bf86421d151d8df7132e8e87d04b6662f45a/lib/ansible/modules/packaging/os/portage.py#L233-L237 | 07:11 |
| prometheanfire | I don't see an update in project-config/nodepool/elements | 07:13 |
| *** ykarel is now known as ykarel|lunch | 07:13 | |
| prometheanfire | same for openstack-zuul-jobs (just some legacy stuff) | 07:14 |
| *** jtomasek has quit IRC | 07:15 | |
| ianw | there's handlers for it, see https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/configure-mirrors/handlers/main.yaml | 07:16 |
| *** e0ne has joined #openstack-infra | 07:16 | |
| prometheanfire | ah, zuul-jobs | 07:16 |
| prometheanfire | those work even if on other distros? | 07:17 |
| ianw | i'd suggest either wiring up handlers for it ...which seems a little odd as the mirror isn't actually being configured, but also seems useful so you *can* install packages later ... or properly skip trying to install the package if we're not setting up the distro | 07:18 |
| ianw | those handlers are called from the appropriate distro places | 07:18 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Warn which patchset are going to be obsoleted. https://review.openstack.org/456937 | 07:20 |
| *** olivierb has joined #openstack-infra | 07:21 | |
| prometheanfire | ianw: I feel like this is too simple... | 07:24 |
| openstackgerrit | Matthew Thode proposed openstack-infra/zuul-jobs master: update Gentoo cache https://review.openstack.org/604003 | 07:24 |
| prometheanfire | it's gonna need at least one more thing | 07:26 |
| *** ginopc has quit IRC | 07:31 | |
| ianw | i'm heading out. will check back if we get any response on the OVH thing in a few hours | 07:31 |
| *** ginopc has joined #openstack-infra | 07:32 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/openstack-zuul-jobs master: add Gentoo jobs and fix install test https://review.openstack.org/602439 | 07:32 |
| prometheanfire | ianw: ^ is my other fix | 07:32 |
| openstackgerrit | Matthew Thode proposed openstack-infra/openstack-zuul-jobs master: use this to test gentoo zuul changes https://review.openstack.org/602446 | 07:33 |
| prometheanfire | anyway, nn, feel free to change my shitty ansible if needed | 07:34 |
| *** dayou_ has quit IRC | 07:42 | |
| *** jpena|off is now known as jpena | 07:47 | |
| *** jpich has joined #openstack-infra | 07:48 | |
| *** jaosorior has quit IRC | 08:12 | |
| *** janki has quit IRC | 08:13 | |
| *** yankcrime has joined #openstack-infra | 08:17 | |
| *** ykarel|lunch is now known as ykarel | 08:20 | |
| *** janki has joined #openstack-infra | 08:20 | |
| *** tosky has joined #openstack-infra | 08:21 | |
| *** jamesmcarthur has joined #openstack-infra | 08:24 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Avoid UnicodeEncodeError on python 2 https://review.openstack.org/583535 | 08:25 |
| *** jchhatbar has joined #openstack-infra | 08:27 | |
| *** janki has quit IRC | 08:28 | |
| *** jamesmcarthur has quit IRC | 08:28 | |
| *** jchhatba_ has joined #openstack-infra | 08:29 | |
| *** jchhatbar has quit IRC | 08:32 | |
| *** wznoinsk_ is now known as wznoinsk | 08:52 | |
| *** jaosorior has joined #openstack-infra | 08:58 | |
| *** Bhujay has joined #openstack-infra | 09:00 | |
| *** Tahvok has left #openstack-infra | 09:19 | |
| *** jchhatba_ has quit IRC | 09:31 | |
| *** janki has joined #openstack-infra | 09:31 | |
| *** derekh has joined #openstack-infra | 09:37 | |
| *** sshnaidm|pto has quit IRC | 09:41 | |
| *** dtantsur|afk is now known as dtantsur | 09:43 | |
| *** janki has quit IRC | 09:45 | |
| *** janki has joined #openstack-infra | 09:51 | |
| *** niedbalski has quit IRC | 09:53 | |
| *** jaosorior has quit IRC | 09:53 | |
| *** jlvillal has quit IRC | 09:54 | |
| *** jlvillal has joined #openstack-infra | 09:54 | |
| *** jchhatbar has joined #openstack-infra | 10:00 | |
| *** janki has quit IRC | 10:03 | |
| *** pcaruana has joined #openstack-infra | 10:04 | |
| *** jtomasek has joined #openstack-infra | 10:09 | |
| *** roman_g has joined #openstack-infra | 10:09 | |
| *** dalvarez has joined #openstack-infra | 10:16 | |
| *** alexchadin has joined #openstack-infra | 10:27 | |
| *** alexchadin has quit IRC | 10:28 | |
| *** alexchadin has joined #openstack-infra | 10:28 | |
| *** ssbarnea|bkp has joined #openstack-infra | 10:32 | |
| *** alexchadin has quit IRC | 10:33 | |
| *** ijw has joined #openstack-infra | 10:34 | |
| ssbarnea|bkp | can someone review/merge a minor devstack-gate fix? https://review.openstack.org/#/c/570546/ - is already 3mo old and no reason to keep it open. | 10:35 |
| *** ijw has quit IRC | 10:38 | |
| *** vtapia has quit IRC | 10:41 | |
| *** jlvillal has quit IRC | 10:50 | |
| *** jlvillal has joined #openstack-infra | 10:53 | |
| *** jchhatbar is now known as janki | 10:56 | |
| *** jpena is now known as jpena|lunch | 10:57 | |
| *** jaosorior has joined #openstack-infra | 10:58 | |
| *** udesale has quit IRC | 10:58 | |
| *** ansmith has quit IRC | 11:05 | |
| *** ijw has joined #openstack-infra | 11:09 | |
| *** ijw has quit IRC | 11:13 | |
| *** pcaruana has quit IRC | 11:15 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Revert "Disable OVH BHS1 region" https://review.openstack.org/604058 | 11:19 |
| *** pcaruana has joined #openstack-infra | 11:20 | |
| *** pbourke has quit IRC | 11:25 | |
| *** pbourke has joined #openstack-infra | 11:26 | |
| *** alexchadin has joined #openstack-infra | 11:28 | |
| *** pcaruana has quit IRC | 11:32 | |
| *** alexchadin has quit IRC | 11:33 | |
| *** longkb has quit IRC | 11:35 | |
| *** pcaruana has joined #openstack-infra | 11:39 | |
| *** ijw has joined #openstack-infra | 11:45 | |
| *** ijw has quit IRC | 11:49 | |
| *** pcaruana has quit IRC | 11:50 | |
| *** weshay_pto is now known as weshay | 12:01 | |
| ssbarnea | ianw: i refreshed the https://review.openstack.org/#/c/583535/ and now you should be able to reproduce the bug by commenting the fix in git-review main(). | 12:03 |
| *** priteau has joined #openstack-infra | 12:04 | |
| *** alexchadin has joined #openstack-infra | 12:08 | |
| *** fresta has joined #openstack-infra | 12:14 | |
| *** fresta_ has quit IRC | 12:15 | |
| *** boden has joined #openstack-infra | 12:15 | |
| *** rlandy has joined #openstack-infra | 12:16 | |
| egonzalez | hi, we have 3 kolla jobs in post merge queue from 28h ago, and other 7 in the periodic and stable-periodic jobs | 12:22 |
| egonzalez | probably will need to be killed or re schedule | 12:22 |
| *** trown|outtypewww is now known as trown | 12:23 | |
| *** fresta_ has joined #openstack-infra | 12:26 | |
| *** fresta has quit IRC | 12:26 | |
| AJaeger | egonzalez: post is no problem - we run always with HEAD, we can just wait... | 12:32 |
| *** tpsilva has joined #openstack-infra | 12:34 | |
| *** yamamoto has quit IRC | 12:42 | |
| *** yamamoto has joined #openstack-infra | 12:42 | |
| *** Bhujay has quit IRC | 12:43 | |
| *** ijw has joined #openstack-infra | 12:44 | |
| *** kgiusti has joined #openstack-infra | 12:44 | |
| *** Bhujay has joined #openstack-infra | 12:44 | |
| *** ansmith has joined #openstack-infra | 12:44 | |
| *** Bhujay has quit IRC | 12:45 | |
| *** fresta has joined #openstack-infra | 12:45 | |
| *** fresta_ has quit IRC | 12:45 | |
| *** Bhujay has joined #openstack-infra | 12:45 | |
| ssbarnea | jaosorior: any insights regarding what is happening with the abandon/revive game? | 12:46 |
| jaosorior | ssbarnea: trying to free up resources to get relevant patches merged, these patches should hopefully help with the timeouts that are clogging up the gate. | 12:47 |
| jaosorior | ssbarnea: that was being discussed in #tripleo | 12:47 |
| *** alexchadin has quit IRC | 12:48 | |
| *** alexchadin has joined #openstack-infra | 12:48 | |
| *** ijw has quit IRC | 12:49 | |
| *** janki has quit IRC | 12:50 | |
| ssbarnea | jaosorior: thanks, i missed that (too many msg to follow). indeed i never seen such a long queue. | 12:55 |
| *** fresta has quit IRC | 13:01 | |
| *** alexchadin has quit IRC | 13:01 | |
| fungi | ianw: not sure if anybody's mentioned it yet, but we set clouds.yaml to force ipv4 for ovh because the nova api returns ipv6 addresses for instances which the instances themselves can't autoconfigure due to lack of route announcements and nothing in metadata/configdrive for them | 13:03 |
| fungi | we also hadn't been putting their v6 addresses in dns i don't think, otherwise we'd have to manually add static v6 address configuration for their interfaces | 13:04 |
| *** mriedem has joined #openstack-infra | 13:06 | |
| fungi | jaosorior: zuul admins have the ability to reorder the gate pipeline contents to move important fixes to the top, if you let us know which ones you need reshuffled | 13:06 |
| *** _ari_ has joined #openstack-infra | 13:08 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Use zk connection passed to OpenStack driver https://review.openstack.org/603428 | 13:09 |
| jaosorior | fungi: well that's good to know! | 13:09 |
| jaosorior | fungi: I already purged a ton of patches... so... for now stuff seems to be fine. But I'll keep that in mind, thank you! | 13:09 |
| *** hrubi has quit IRC | 13:10 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Move OpenStack leak code into driver https://review.openstack.org/603429 | 13:11 |
| *** hrubi has joined #openstack-infra | 13:11 | |
| *** sthussey has joined #openstack-infra | 13:13 | |
| *** dmsimard has quit IRC | 13:15 | |
| *** quiquell|rover is now known as quique|rover|lch | 13:15 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: emit-job-header: fix print node information https://review.openstack.org/602290 | 13:18 |
| *** gfidente has quit IRC | 13:21 | |
| *** gfidente has joined #openstack-infra | 13:25 | |
| *** udesale has joined #openstack-infra | 13:27 | |
| *** jpena|lunch is now known as jpena | 13:29 | |
| *** yamamoto has quit IRC | 13:29 | |
| *** jistr is now known as jistr|call | 13:32 | |
| Diabelko | I've been looking at https://github.com/openstack-infra/project-config/blob/master/zuul.d/pipelines.yaml#L102 and then at Gerrit docs and I want to ask one question | 13:33 |
| Diabelko | what is the difference between ref-updated and change-merged in those two pipelines (post and promote)? | 13:33 |
| Diabelko | is this because ref-updated also works for manual code pushes outside of Gerrit? | 13:33 |
| Diabelko | (not via review, but git push) | 13:33 |
| *** dtruong has joined #openstack-infra | 13:36 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fix on reassign attendee tix https://review.openstack.org/604090 | 13:36 |
| mordred | Diabelko: ref-updated is also fired when a tag is made - the two events also contain slightly different information | 13:39 |
| mordred | change-merged doesn't contain the merge commit ref in it, because it's an event reporting on the change itself - while ref-updated does | 13:39 |
| Diabelko | oh, I see, thanks! | 13:40 |
| mordred | but ref-updated is just about refs and doesn't contain any information about a change, so for the promote pipeline it's more useful to trigger on change-merged so the change-id can be used as a key to look up an artifact produced in a different pipeline | 13:40 |
| *** ijw has joined #openstack-infra | 13:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add yamlgroup inventory plugin https://review.openstack.org/602385 | 13:45 |
| *** alexchadin has joined #openstack-infra | 13:46 | |
| *** e0ne has quit IRC | 13:48 | |
| *** ijw has quit IRC | 13:49 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Warn which patchset are going to be obsoleted. https://review.openstack.org/456937 | 13:51 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Warn which patchset are going to be obsoleted. https://review.openstack.org/456937 | 13:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove open-vm-tools from servers https://review.openstack.org/596254 | 13:54 |
| *** quique|rover|lch is now known as quiquell|rover | 13:55 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fix on reassign attendee tix https://review.openstack.org/604090 | 13:56 |
| *** chkumar|ruck is now known as chkumar|off | 13:56 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: rewrite interface in react https://review.openstack.org/591604 | 14:02 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add build page https://review.openstack.org/597024 | 14:03 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add job page https://review.openstack.org/597048 | 14:03 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add config-errors notifications drawer https://review.openstack.org/597147 | 14:03 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add change status page https://review.openstack.org/599472 | 14:03 |
| *** bobh has joined #openstack-infra | 14:04 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fix issue on track extra question https://review.openstack.org/604099 | 14:05 |
| *** ijw has joined #openstack-infra | 14:08 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Use include_tasks instead of include https://review.openstack.org/604101 | 14:10 |
| *** dpawlik has quit IRC | 14:11 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove ssh v1 options from sshd_config https://review.openstack.org/593648 | 14:11 |
| *** ijw has quit IRC | 14:13 | |
| *** smarcet has joined #openstack-infra | 14:13 | |
| *** smarcet has left #openstack-infra | 14:14 | |
| *** smarcet has joined #openstack-infra | 14:14 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fix issue on track extra question https://review.openstack.org/604099 | 14:15 |
| *** alexchadin has quit IRC | 14:21 | |
| mriedem | clarkb: the categorization rate actually went down somehow :( http://status.openstack.org/elastic-recheck/data/integrated_gate.html | 14:24 |
| mriedem | hmm, http://status.openstack.org/elastic-recheck/ is out of date | 14:25 |
| *** alexchadin has joined #openstack-infra | 14:25 | |
| mriedem | Data Last Updated: Wed Sep 19 2018 20:00:01 GMT-0500 (Central Daylight Time) | 14:25 |
| openstackgerrit | Simon Westphahl proposed openstack-infra/nodepool master: Re-register missing nodes in static driver https://review.openstack.org/600084 | 14:25 |
| openstackgerrit | Simon Westphahl proposed openstack-infra/nodepool master: Implement liveness check for static nodes https://review.openstack.org/601513 | 14:25 |
| *** Tim_ok has joined #openstack-infra | 14:27 | |
| *** smarcet has quit IRC | 14:34 | |
| *** jistr|call is now known as jistr | 14:34 | |
| *** rascasoft has quit IRC | 14:35 | |
| *** dpawlik has joined #openstack-infra | 14:41 | |
| *** dpawlik has quit IRC | 14:45 | |
| *** yamamoto has joined #openstack-infra | 14:48 | |
| *** Bhujay has quit IRC | 14:51 | |
| *** dmsimard has joined #openstack-infra | 14:54 | |
| *** e0ne has joined #openstack-infra | 14:56 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/openstack-zuul-jobs master: Remove ovsdbapp job https://review.openstack.org/604114 | 14:56 |
| *** Swami has joined #openstack-infra | 14:57 | |
| *** smarcet has joined #openstack-infra | 14:58 | |
| *** yamamoto has quit IRC | 15:03 | |
| *** alexchadin has quit IRC | 15:04 | |
| *** dklyle has joined #openstack-infra | 15:04 | |
| *** quiquell|rover is now known as quique|rover|off | 15:05 | |
| *** ykarel is now known as ykarel|away | 15:05 | |
| *** gfidente has quit IRC | 15:05 | |
| *** gfidente has joined #openstack-infra | 15:07 | |
| *** olivierb has quit IRC | 15:08 | |
| *** jtomasek has quit IRC | 15:10 | |
| *** eernst has joined #openstack-infra | 15:16 | |
| *** jamesmcarthur has joined #openstack-infra | 15:18 | |
| *** ykarel|away has quit IRC | 15:18 | |
| *** anteaya has joined #openstack-infra | 15:18 | |
| *** eernst has quit IRC | 15:21 | |
| *** dave-mccowan has quit IRC | 15:25 | |
| clarkb | mriedem: ok I'll take a look to see why it is stuck | 15:26 |
| openstackgerrit | Merged openstack-infra/zuul master: web: add /{tenant}/projects and /{tenant}/project/{project} routes https://review.openstack.org/550979 | 15:27 |
| clarkb | ianw: AJaeger the ipv6 issue is known on ovh. Neutron knows about the ipv6 address but config-drive/metadata service do not | 15:28 |
| AJaeger | clarkb: so, what does that mean with accessing the mirror? Jobs failed with that... | 15:29 |
| AJaeger | clarkb: don't know more details, ianw did the hard work | 15:29 |
| AJaeger | corvus, mordred, zuul experts, could you look at https://review.openstack.org/#/c/603554/ , please? What advise can you give for running periodic master-only jobs in-repo? | 15:29 |
| *** pbourke has quit IRC | 15:30 | |
| clarkb | AJaeger: ianw fungi I'm guessing this is a side effect of the upgrade, we should double check security groups since its a networking issue and security groups are often the cause of those problems. If security groups look fine I think we should see if dpawlik or amorin can help | 15:32 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: added new endpoint delete my presentation https://review.openstack.org/604130 | 15:36 |
| mordred | clarkb: so - tell me about CD issues? | 15:37 |
| mordred | clarkb: (whenever) | 15:38 |
| clarkb | mordred: ya just a sec, I'm double checking the ovh mirror. We still have the old one up and running which we probably want to keep for now so that ovh can debug the networking disconnect (but I also need to check security groups still) | 15:39 |
| clarkb | I'll do security groups after talking CD issues since the fire is out | 15:39 |
| mordred | nod | 15:39 |
| *** agopi has quit IRC | 15:39 | |
| clarkb | mordred: so there are two things I've run into. The first is that a lot of our base pre run stuff assumes it is running on actual nodes not the executor in a nil nodeset job. This means that we don't remove the master ssh key from the ssh agent when running with nil nodeset. We also don't log an inventory which first impression is we don't need the inventory there is no inventory except that all our | 15:41 |
| clarkb | variables and job inheritances and all that is in the inventory file | 15:41 |
| clarkb | the second is that bridge.o.o is set up such that ssh root login is only allowed from puppetmaster and bridge. So we cannot ssh in from the zuul executor as root to the bridge with the current setup | 15:41 |
| clarkb | My thought on this one is we already know how to create users and manage them, maybe we should just create a zuul user on bridge.o.o? | 15:42 |
| clarkb | The first issue is something we should jsut figure out how to generally improve for all nil nodeset jobs. The mechanics of that are tricky with 'all' excluding localhost, but it is doable (maybe with a base job dedicated to nil nodest jobs) | 15:43 |
| mordred | clarkb: yah - on the second one - and then just put a become: yes in the mix yeah? | 15:43 |
| clarkb | the second is a matter of configuration preference of infra-rooters | 15:44 |
| clarkb | mordred: ya that is what I am thinking | 15:44 |
| *** yumiriam has joined #openstack-infra | 15:44 | |
| *** sshnaidm|pto has joined #openstack-infra | 15:44 | |
| mordred | that wfm ... for the first ... yeah, we might want to work up a specific base job | 15:45 |
| mordred | clarkb: oh - do we not log our inventory bcause it's a task that takes all: ? | 15:45 |
| clarkb | mordred: I think it is that plus ansible doesn't set the inventory variable when running against localhost so we don't actually know where it is (but we can maybe hand wave around that because its running under zuul) | 15:46 |
| mordred | yah | 15:47 |
| clarkb | mordred: that is also why the ssh-agent cleanup of the master key doesn't happen | 15:47 |
| clarkb | the role is run against 'all' then the cleanup is delegated to localhost in that role | 15:47 |
| mordred | so in that case, maybe we can just change the invocation to be hosts: localhost | 15:47 |
| clarkb | I have some wip changes up around this stuff one sec | 15:47 |
| mordred | cool | 15:47 |
| clarkb | https://review.openstack.org/603240 and https://review.openstack.org/603241 I don't think that will work as is, but should be close to what we need | 15:48 |
| *** anteaya has quit IRC | 15:49 | |
| *** uberjay has quit IRC | 15:50 | |
| *** uberjay has joined #openstack-infra | 15:50 | |
| mordred | clarkb: it's close - I think there is a sequencing issue, becaues add-build-sshkey re-adds the temp ssh key after removing things | 15:51 |
| mordred | but split out the re-add happens before the removal | 15:51 |
| mordred | other than that, I think it's great | 15:51 |
| clarkb | mordred: ya, I had to update the ssh-add pattern so that we add with the stdin identifier for that key too (so it won't be removed) | 15:52 |
| clarkb | I'm not sure if that is a concern though. I know we want the master key removed before the actual job starts (so jobs can't ssh into other jobs) | 15:52 |
| *** fresta has joined #openstack-infra | 15:52 | |
| clarkb | not sure if we need that sequencing to be tighter within the base pre playbook | 15:52 |
| *** ssbarnea|bkp has quit IRC | 15:53 | |
| clarkb | if anyone is wondering the security group rule list osc command doesn't show you the direction of the rule | 15:54 |
| *** smarcet has quit IRC | 15:54 | |
| logan- | maybe it would be good to have a virtual 'localhost' nodeset label that can add localhost inventory hosts | 15:55 |
| logan- | so instead of requesting a node, just add it to inventory with ansible_connection: local | 15:55 |
| mordred | clarkb: ah - gotcha | 15:56 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config master: Remove ovsdbapp legacy jobs https://review.openstack.org/604137 | 15:56 |
| clarkb | logan-: I like that, then its not a weird nil nodeset but a nodest of localhost and things work | 15:56 |
| clarkb | unsure if zuul can do that for us, but if it can that might be a better way to express this | 15:57 |
| logan- | yeah the fake localhost item in ansible has some weird things that work differently than other inventory items, like ansible_python_interpreter will look at whatever venv you have ansible running in instead of the regular system python like any other inventory host by default | 15:57 |
| logan- | so i try to avoid using it for anything except tasks relating directly to the ansible run itself | 15:58 |
| logan- | and instead add a separate host entry with ansible_connection: local that is joined to groups, etc | 15:58 |
| *** gyee has joined #openstack-infra | 15:58 | |
| clarkb | logan-: zuul does some things to prevent ansible from running arbitrarily against localhost which likely makes this complicated | 15:59 |
| logan- | gotcha | 15:59 |
| clarkb | but I imagine it is doable with some work | 15:59 |
| clarkb | I don't see anything in the security groups that would affect internal traffic but not external | 16:01 |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config master: Adjust metalsmith ACL to match ones of an official project https://review.openstack.org/604143 | 16:01 |
| clarkb | our ingress rules either specify None as the source range or 0.0.0.0/0 and ::/0 | 16:01 |
| clarkb | which should apply to both my desktop and other hosts within the cloud | 16:02 |
| clarkb | I guess I should check security groups for egress in the other tenant | 16:02 |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config master: Adjust metalsmith ACL to match ones of an official project https://review.openstack.org/604143 | 16:02 |
| *** rascasoft has joined #openstack-infra | 16:03 | |
| clarkb | security groups on the other side look fine to me as well | 16:04 |
| mordred | clarkb: the arbitrary localhost stuff is skipped for trusted things - so it should work fine ... | 16:06 |
| mordred | but it might not be as much of a win to make a localhost nodeset, since that might imply to people they could use it for more arbitrary things | 16:06 |
| clarkb | mordred: oh interesting, but not all nil nodeset jobs would be trusted | 16:06 |
| mordred | yah | 16:06 |
| clarkb | yesterday roman_g was looking at tagging docker iamges with nil nodest job | 16:07 |
| clarkb | (as an example) | 16:07 |
| mordred | yah- and that would be potentially fine since it doesn't execute local shell commands | 16:07 |
| mordred | I guess I'm saying I'm not sure which thing will be better - a nil-nodeset base job, updating the exisitng one to work for both, having a localhost nodeset, or some combination | 16:08 |
| mordred | all have potential pros and cons | 16:08 |
| mordred | and we might just need to try all three :) | 16:08 |
| clarkb | 158.69.65.118 with netmask 255.255.224.0 what is top end of that range | 16:08 |
| * clarkb does octal maths | 16:08 | |
| clarkb | 158.69.95.255 | 16:09 |
| clarkb | our mirror is 158.69.80.87 | 16:09 |
| clarkb | so our VM thinks the mirror is in the l2 range and is directly attached | 16:10 |
| clarkb | however memory says all our hosts in OVH has a /32 previously and had to route | 16:10 |
| * clarkb checks an older host in ovh | 16:10 | |
| clarkb | yup older host has netmast of 255.255.255.255 | 16:11 |
| *** ginopc has quit IRC | 16:11 | |
| clarkb | and is in that same /19 and works | 16:11 |
| clarkb | ok old and new hosts have the same gateway set | 16:13 |
| clarkb | the only difference is the netmask | 16:13 |
| clarkb | any objection to me setting the netmast on the new host and potentially breaking networking to that host (its a nodepool machine we should be able to boot a new one if we need to) | 16:14 |
| clarkb | acutally before I do that I should check the dhcp lease details to see where the netmask is coming from | 16:14 |
| mordred | clarkb: I do not have any objection to that | 16:17 |
| *** anteaya has joined #openstack-infra | 16:17 | |
| mordred | clarkb: we might want to report the netmask change to our ovh friends though - maybe it is fallout from the upgrade? | 16:17 |
| clarkb | the /19 is in the config drive network data at least (I'm actually not seeing any confirmation dhcp was run at all so working backward from what glean would see to try and figure that out) | 16:17 |
| clarkb | mordred: ya I'm guessing this is unintentional breakage | 16:17 |
| clarkb | mordred: if setting it to a /32 fixes we'll definitely want to let them know | 16:18 |
| *** derekh has quit IRC | 16:18 | |
| mordred | ++ | 16:19 |
| mordred | dpawlik was in channel yesterday but doesn't seem to be here today | 16:20 |
| clarkb | we statically configure the interface with glean | 16:20 |
| clarkb | (whcih is fine just wanting to make sure I understand the mechanisms at play here) | 16:21 |
| clarkb | I'm going to set the netmask to 255.255.255.255 which may break my ssh connection | 16:21 |
| clarkb | but we can boot another if we need to | 16:21 |
| clarkb | ya that broke it | 16:22 |
| clarkb | I've pinged amorin | 16:24 |
| *** ykarel|away has joined #openstack-infra | 16:25 | |
| clarkb | I'll leave a followup on the change that ianw has/had since that is where we've collected notes so far | 16:29 |
| *** jamesmcarthur has quit IRC | 16:30 | |
| clarkb | one more datapoint, the old hosts seem to use dhcp not static configuration | 16:31 |
| *** aojea has quit IRC | 16:32 | |
| *** agopi has joined #openstack-infra | 16:33 | |
| *** holser_ has quit IRC | 16:33 | |
| mordred | clarkb: dhcp is more better | 16:33 |
| *** anteaya has quit IRC | 16:34 | |
| clarkb | https://review.openstack.org/#/c/603988/1 has my notes on it now | 16:35 |
| clarkb | fungi: ^ you may find those interesting and possibly you might even recognize the fix here? | 16:35 |
| *** diablo_rojo has joined #openstack-infra | 16:36 | |
| *** Swami has quit IRC | 16:36 | |
| fungi | thanks, catching up, was heads-down composing the ml combining announcement | 16:36 |
| *** hashar is now known as hasharDinner | 16:37 | |
| fungi | mmm, so a /32 mask is sort of a strange beast since it's undefined whether the machine should use arp to find the gateway's mac (as it's already outside the mask) | 16:39 |
| fungi | it's possible the kernel falls back to sending arp requests on all interfaces in this case | 16:39 |
| fungi | or that we may need to set an interface route defining how to reach the gateway | 16:40 |
| *** Bhujay has joined #openstack-infra | 16:40 | |
| *** e0ne has quit IRC | 16:41 | |
| *** Bhujay has quit IRC | 16:41 | |
| clarkb | ya I'm guessing something along those lines is why setting it from 255.255.255.224 to 255.255.255.255 broke things | 16:41 |
| fungi | i wonder if the behavior change is the result of dhcp being down through lease expiration, and was still down when the instance was rebooted causing it to fall back on the instance metadata instead? | 16:41 |
| *** Bhujay has joined #openstack-infra | 16:42 | |
| clarkb | fungi: I think glean is configuring statically because the network_data json blob on the config drive doesn't tell it to use dhcp | 16:42 |
| *** trown is now known as trown|lunch | 16:42 | |
| fungi | oh, did we switch to using glean on these? i thought they were cloud-init up until at least recently | 16:42 |
| *** radez has quit IRC | 16:42 | |
| clarkb | fungi: all of the nodepool nodes are glean | 16:43 |
| *** radez has joined #openstack-infra | 16:43 | |
| clarkb | my comparison was only of the nodepool nodes, not the mirror nodes ( I guess I didn't make that clear on the gerrit comment) | 16:43 |
| fungi | oh, sorry, thought we were talking about the mirror specifically, but i guess routing packets between the mirror and the nodepool nodes is at issue | 16:44 |
| clarkb | fungi: ya the mirror and the new nodepool node are in the same l2 range based on the netmask on the nodepool nodes | 16:44 |
| clarkb | old nodepool nodes had /32 netmask so would've routed to the mirror | 16:44 |
| *** sshnaidm|pto has quit IRC | 16:44 | |
| clarkb | new ones try to arp for it and say sorry no can do | 16:45 |
| fungi | so anyway, it's possible glean needs to learn to include an interface route for the gateway on the interface it thinks should be capable of reaching it | 16:45 |
| fungi | but i'm guessing it's that they ceased including an equivalent interface route in the metadata or dhcp leases (or both) | 16:45 |
| clarkb | assuming 255.255.255.255 is the fix | 16:46 |
| clarkb | I guess I should check if new mirror with config drive did something different | 16:46 |
| clarkb | fungi: 158.69.64.1 0.0.0.0 255.255.255.255 UH 0 0 0 ens3 is the tpye of route you expect? | 16:47 |
| *** jpich has quit IRC | 16:48 | |
| clarkb | 158.69.64.0 0.0.0.0 255.255.224.0 U 0 0 0 eth0 is what we had prior to breaking networking on the nodepool node | 16:48 |
| clarkb | which would be the issue I think | 16:48 |
| fungi | having a hard time parsing. can you provide `ip ro sh` output? | 16:48 |
| clarkb | default via 158.69.64.1 dev ens3 and 158.69.64.1 dev ens3 scope link | 16:48 |
| *** smarcet has joined #openstack-infra | 16:48 | |
| clarkb | basically that says route to your gateway via this specific device | 16:48 |
| clarkb | which we didn't have when the netmask was larger | 16:49 |
| fungi | yeah, the "scope link" is important | 16:49 |
| clarkb | config drive is configuring the new nodes to boot and dhcp | 16:49 |
| clarkb | http://paste.openstack.org/show/730461/ is the network data that caused glean to do not dhcp | 16:50 |
| fungi | so basically the default route is through 158.69.64.1 and the route to 158.69.64.1 is then specified as being through the ens3 device | 16:50 |
| fungi | since it can't be inferred from the netmask on that interface | 16:51 |
| clarkb | I think there are a couple bugs here based on that. 1) network data should specify dhcp if that is how config drive is configured and things are working (also dhcp should just be preferred) 2) the network data does not match with the data provided by dhcp | 16:51 |
| clarkb | I think if OVH fixes 1) we'll be fine with glean, it will configure dhcp and everything will work | 16:52 |
| clarkb | fixing 2) is more of a sanity thing so when you debug everything looks correct | 16:52 |
| *** udesale has quit IRC | 16:52 | |
| clarkb | now to figure out what data glean needs to understand dhcp should be used | 16:53 |
| clarkb | "type": "ipv4" should be "type": "ipv4_dhcp" | 16:54 |
| clarkb | if you do a subnet show on our ipv4 subnet range enable_dhcp is set to false | 16:56 |
| clarkb | whcih is why the type isn't ipv4_dhcp in the metadata | 16:56 |
| clarkb | infra-root AJaeger ^ I think the root of our problems is this. We probably need to talk to ovh about figuring out what they expect their networking to look like, fix the metadata as appropriate then fix glean as necessary to catch up | 16:58 |
| mordred | clarkb: ++ | 17:00 |
| clarkb | worth noting that the metadata info for the interface matches the subnet show details pretty closely | 17:00 |
| clarkb | so there appears to be a mismatch between how ovh has configured neutron and how we are expected to consume the networking resources | 17:01 |
| clarkb | I'm going to reboot my nodepool test node, then run dhclient manually which should get it working | 17:02 |
| clarkb | and that should tie up the confirmation of this is wht thing | 17:02 |
| *** ramishra has quit IRC | 17:02 | |
| clarkb | s/wht/the/ | 17:02 |
| *** yamamoto has joined #openstack-infra | 17:03 | |
| *** jamesmcarthur has joined #openstack-infra | 17:04 | |
| fungi | in unrelated news, the virtualenv tool was basically abandonware but one of the regular contributors to tox has volunteered and taken over maintenance of it. if anybody had any outstanding prs for fixing things in virtualenv this is probably a good time to refresh them | 17:04 |
| *** jamesmcarthur has quit IRC | 17:05 | |
| *** jamesmcarthur has joined #openstack-infra | 17:05 | |
| * fungi is unsure whether we had any hanging out there getting ignored | 17:05 | |
| *** Swami has joined #openstack-infra | 17:05 | |
| clarkb | dhclient gets a lease with netmask 255.255.255.255 | 17:06 |
| fungi | also, i need to go grab a late lunch and run some errands, but will be back in fairly short order | 17:06 |
| clarkb | it doesn't update the existing interface config though, but I think I'm satisfied without that happening | 17:06 |
| clarkb | fungi: I don't think we have any virtualenv requests. The big thing was having it update setuptools and pip and wheel automaticalyl which it has done for a while | 17:07 |
| clarkb | (helps with confusion over things don't work but its a fresh venv) | 17:07 |
| fungi | cool, thanks! | 17:07 |
| panda | is /etc/ci/mirror_info.sh directly baked into nodes images ? if not, at which point in the job is it called ? | 17:09 |
| *** jpena is now known as jpena|off | 17:09 | |
| clarkb | panda: it is written by a zuul pre step somewhere. Let me find it | 17:09 |
| clarkb | mriedem: the e-r cron's are running but not updating the files | 17:10 |
| panda | clarkb: thanks, I tried to git grep in zuul-jobs and openstack-zuul-jobs, nothing is calling the mirror-info role | 17:10 |
| clarkb | mriedem: I ran it in the foreground and got no output. going to try doing it step by step and see if I get anything | 17:10 |
| logan- | panda: https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/roles/mirror-info/ | 17:10 |
| logan- | oh | 17:10 |
| clarkb | ya thats the role and the base job in project-config runs it | 17:10 |
| panda | yep, I thinkg that's the role, I just can't find when in the process it's called | 17:10 |
| logan- | http://codesearch.openstack.org/?q=mirror-info&i=nope&files=&repos=project-config | 17:11 |
| clarkb | panda: its project-config/playbooks/base/pre.yaml as part of the base job | 17:11 |
| *** harlowja has joined #openstack-infra | 17:12 | |
| clarkb | mriedem: strace implies 500 errors talking to elasticsaerch? maybe one of our queries is malformed. I'll check elasticsearch logs | 17:12 |
| clarkb | [Failed to deserialize exception response from stream]; nested: OutOfMemoryError[Java heap space] unhappy java maybe | 17:13 |
| mriedem | clarkb: i probably need to do a pass on cleaning up old stale queries | 17:14 |
| mriedem | that could help | 17:14 |
| mriedem | it's been months since i've done one | 17:14 |
| mriedem | i'll do that after lunch | 17:15 |
| notmyname | fungi: is the -operators list included in the ML merge? | 17:16 |
| clarkb | mriedem: thanks. I don't see anything on the server side that points to the recent 500 errors strace seems to get | 17:16 |
| clarkb | mriedem: I'll keep digging on this end | 17:16 |
| notmyname | fungi: disregard. apparently, I can't read | 17:17 |
| panda | clarkb: so the base job is always involved, even if configurations are not explicitly inheriting from it ? | 17:18 |
| clarkb | panda: the base job is implied if you do not specify a specific parent | 17:19 |
| clarkb | the sudoers message from status was me, sorry for the noise | 17:19 |
| panda | clarkb: got it, thanks. thanks logan- | 17:19 |
| clarkb | (I was su'd to recheck user to debug running the script and tried to sudo like a derp) | 17:19 |
| *** yamamoto has quit IRC | 17:20 | |
| *** anteaya has joined #openstack-infra | 17:21 | |
| clarkb | mriedem: rerunning now with -v which seems to be a bit more verbose | 17:25 |
| clarkb | mriedem: it stops with 2018-09-20 17:25:12 DEBUG [ergraph ] Starting query for bug 1793364 | 17:29 |
| openstack | bug 1793364 in OpenStack Compute (nova) "mysql db opportunistic unit tests timing out intermittently in the gate (bad thread switch?)" [High,Confirmed] https://launchpad.net/bugs/1793364 | 17:29 |
| clarkb | mriedem: no traceback though. Maybe its a broken query ? | 17:29 |
| clarkb | mriedem: that was one of the queries added yesterday. My best guess is it is broken somehow, but reading it I don't see why | 17:30 |
| clarkb | mriedem: figured it out. Its OOMing the e-r process due to the size of the results | 17:31 |
| clarkb | kernel is killing it | 17:32 |
| *** Bhujay has quit IRC | 17:34 | |
| *** dtruong has quit IRC | 17:34 | |
| clarkb | mriedem: looks like that error matches some tripleo log files which end up being the whole log file for the event because they make a log file of grepped errors :/ | 17:36 |
| clarkb | mriedem: I think we can fix this by contraining the query a bit more rather than just tags=console do something like filename=job-output.txt? | 17:37 |
| openstackgerrit | Tom Barron proposed openstack-infra/irc-meetings master: Update manila team meeting chair https://review.openstack.org/604169 | 17:37 |
| *** anteaya has quit IRC | 17:38 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Updated user profile UI https://review.openstack.org/604172 | 17:39 |
| *** dtantsur is now known as dtantsur|afk | 17:40 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/elastic-recheck master: Contrain query to prevent e-r from OOMing https://review.openstack.org/604173 | 17:41 |
| clarkb | mriedem: ^ | 17:41 |
| mordred | notmyname: reading is hard | 17:44 |
| notmyname | mordred: you'd think I'd have noticed after I got 4 copies of the message on all the various lists :-) | 17:44 |
| mriedem | clarkb: ack thanks +W | 17:46 |
| *** dave-mccowan has joined #openstack-infra | 17:46 | |
| *** yamamoto has joined #openstack-infra | 17:50 | |
| clarkb | mriedem: the interesting thing is I think that points to this bug affecting users outside of unittests | 17:50 |
| clarkb | Ideally we'd track that too but the giant file as single event in elasticsearch just isn't working for e-r right now ;) | 17:51 |
| *** anteaya has joined #openstack-infra | 17:52 | |
| *** yamamoto has quit IRC | 17:56 | |
| *** smarcet has quit IRC | 18:00 | |
| *** jamesmcarthur has quit IRC | 18:00 | |
| *** gfidente has quit IRC | 18:00 | |
| clarkb | mriedem: is there a way for a cloud to not provide data in config drive? | 18:02 |
| clarkb | mriedem: specifically network_data.json because networking is being provided out of band of neutron so the neutron provided config drive information is just wrong | 18:02 |
| *** slaweq has quit IRC | 18:03 | |
| *** amorin has joined #openstack-infra | 18:04 | |
| amorin | hey guys | 18:04 |
| clarkb | amorin: hello. I just asked mriedem if it is possible to mask out network data on the config drives (since it is not accurate in the bhs1 case) | 18:05 |
| clarkb | (haven't gotten a response yet) | 18:05 |
| amorin | k | 18:05 |
| clarkb | amorin: to clarify for the channel in OVH the expectation is that we will use DHCP to configure interfaces on OVH VMs right? | 18:05 |
| amorin | yes | 18:05 |
| amorin | our DHCP is supposed to give you a /32 ip | 18:05 |
| amorin | with a next hop gateway | 18:06 |
| amorin | and everything is supposed to be fine with that | 18:06 |
| clarkb | for anyone following along http://paste.openstack.org/show/730461/ is the content of the network data in the config drive which has type: ipv4 which means don't use dhcp (ipv4_dhcp is the type to use with dhcp) | 18:06 |
| *** trown|lunch is now known as trown | 18:06 | |
| clarkb | prior to the ovh upgrade we didn't get any network data in the config drive | 18:06 |
| amorin | the weird thing is that it has always been like that | 18:06 |
| clarkb | and glean fell back to dhcp as a result. Wondering if we can go back to that since the network data is misleading now | 18:06 |
| amorin | so It means OpenStack is newton is giving something new to your instance | 18:07 |
| amorin | through metadata | 18:07 |
| clarkb | amorin: ya the difference was before the config drive data had no networkign information so we fell back to dhcp. But now it explicitly says not to use dhcp | 18:07 |
| clarkb | amorin: yup | 18:07 |
| *** jamesmcarthur has joined #openstack-infra | 18:07 | |
| amorin | what if we set DHCP on the subnet to true? | 18:07 |
| amorin | would it help? | 18:07 |
| clarkb | amorin: yes I think that would result in network data telling glean to use dhcp and it should work. The problem then is I think neutron will try to do dhcp for you and you don't want that? | 18:08 |
| amorin | exactly | 18:08 |
| amorin | we disabled DHCP because of that, but we were not aware about the issue with the network metadata stuff | 18:08 |
| amorin | so, I wont enable that right now, but at least, I can talk with the team about the issue | 18:09 |
| amorin | and maybe we can figure a solution to have the setting to True | 18:09 |
| clarkb | ok, I'm hoping that mriedem knows of some configuration item we can use to make both glean and ovh happy :) | 18:09 |
| amorin | :p | 18:09 |
| *** e0ne has joined #openstack-infra | 18:10 | |
| clarkb | amorin: thansk for popping in, I think we understand the problem now and its just a matter of figuring out a solution. I've got to go eat lunch here in a minute but I'm hoping mriedem will show up soon | 18:13 |
| clarkb | mordred: ^ you will likely be interested from a glean perspective as well | 18:14 |
| amorin | clarkb: ok, I may be afk also at it's evening in my timezone | 18:14 |
| clarkb | amorin: understood | 18:15 |
| mriedem | clarkb: was away, did you get your answer? | 18:16 |
| clarkb | mriedem: did not | 18:16 |
| clarkb | mriedem: basically ovh is managing VM networking via DHCP external to neutron | 18:16 |
| clarkb | mriedem: so the network data supplied to config drive by neutron/nova is just wrong | 18:16 |
| mriedem | config drive network data is wrong | 18:16 |
| clarkb | mriedem: is there a way to tell nova to not supply that info to the VMs? | 18:16 |
| mriedem | well, force_config_drive defaults to False but that doesn't prevent the user from requesting it | 18:17 |
| mriedem | it could also be built into the images | 18:17 |
| mriedem | as metadata to say gimme config drive goodies | 18:17 |
| *** hasharDinner is now known as hashar | 18:17 | |
| mriedem | are you looking to just nix the network_data part of the config drive? | 18:17 |
| clarkb | mriedem: right we explicitly ask for config drive data (which we do want, we just want to prevent the inaccurate info from showing up) | 18:17 |
| clarkb | yup | 18:17 |
| clarkb | we also use the same images on all our clouds | 18:18 |
| clarkb | but we could probably hack something terrible into glean to deal with this based on ip ranges | 18:18 |
| clarkb | or other info | 18:18 |
| mordred | clarkb: reading | 18:18 |
| mordred | clarkb: ugh, I certainly don't want to hack something terrible into glean to deal with this basd on ip ranges | 18:19 |
| mriedem | well we have config_drive_skip_versions for hiding some ec2 versions, but i'm not aware of something to configure hiding or neuturing the network data json | 18:19 |
| *** anteaya has quit IRC | 18:20 | |
| clarkb | mriedem: ok thansk for checking | 18:20 |
| mordred | clarkb: so - just to repeat back to make sure I grok ... | 18:20 |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config master: Add openstack/neutron-interconnection to neutron https://review.openstack.org/599429 | 18:20 |
| mordred | clarkb: dhcp is turned off in neutron, which is causing the network info in config-drive say "no dhcp, please to use static ips" | 18:21 |
| panda | is there a way to override completely a pre-run, and not use the playbooks from the inherited jobs definitions ? | 18:21 |
| mordred | clarkb: but dhcp IS to be used in ovh and exists and is correct, yes? | 18:21 |
| *** smarcet has joined #openstack-infra | 18:22 | |
| mordred | panda: no, I do not believe there is | 18:22 |
| clarkb | mordred: correct. Additionally the static info is not correct beucase it is based on the neutron subnet resource data which isn't what you would get from dhcp | 18:22 |
| clarkb | mordred: I think this would be fine if those two aligned with each other but they don't (and I'm not sure they can based on how OVH uses /32s) | 18:22 |
| mordred | clarkb: we *could* try one of mriedem's suggestions and configure config-drive to off in nodepool for ovh | 18:22 |
| clarkb | mordred: we wouldn't be able to login to those VMs as root anymore if we do that | 18:22 |
| clarkb | mordred: we would have to ssh via zuul executors | 18:23 |
| mordred | oh - because ssh keys | 18:23 |
| panda | mordred: make sense, thanks. | 18:23 |
| mordred | clarkb: well - maybe that's a better tradeoff for now than non-working networking until we can find a better workaround? | 18:24 |
| clarkb | mordred: possibly | 18:24 |
| clarkb | I can boot an instance after lunch with config drive disabled | 18:24 |
| mordred | clarkb: the only other thing I can think of is to define a metadata key/value that glean looks for as a flag to tell it to ignore network info | 18:24 |
| clarkb | then see if I can ssh in and if networking works | 18:24 |
| clarkb | mordred: maybe it could be a type override | 18:25 |
| mordred | clarkb: I doubt you'll be able to ssh in because of keys - you'd need to boot from an image with keys baked in | 18:25 |
| clarkb | we set it to ipv4_dhcp and it follows that path | 18:25 |
| mordred | clarkb: yah | 18:25 |
| clarkb | mordred: ya our zuul/nodepool key is baked in | 18:25 |
| clarkb | just going to confirm I can make that work prior to using it | 18:25 |
| clarkb | oh except we remove that key when jobs run | 18:25 |
| clarkb | hrm | 18:25 |
| mordred | clarkb: we could start putting infra keys in place in the base pre-run | 18:26 |
| clarkb | we would be giving up the ability to hold nodes | 18:26 |
| mordred | rather than via keypairs | 18:26 |
| mordred | we've got that code in an ansible role already | 18:26 |
| clarkb | mordred: maybe do both and then the period of time we can't ssh into the node is limited to only ovh | 18:26 |
| mordred | yah | 18:26 |
| clarkb | I'm going to eat lunch and think about this | 18:26 |
| mordred | we rarely need to ssh in to nodepool nodes that *haven't* run jobs yet | 18:27 |
| mordred | clarkb: I just ate lunch - I recommend it | 18:27 |
| clarkb | mordred: ya the two cases are really hold node and hand it off and cloud is really broken we need to debug | 18:27 |
| clarkb | for the first having the job do it works for the second we can boot a different image with our key baked in | 18:27 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove ssh v1 options from sshd_config https://review.openstack.org/593648 | 18:29 |
| *** tosky has quit IRC | 18:32 | |
| clarkb | I wonder if we could set an arbitrary type on the subnet | 18:32 |
| clarkb | so that when it ends up in the config drive glean doesnt understand it and does dhcp | 18:33 |
| *** dtruong has joined #openstack-infra | 18:35 | |
| mtreinish | fungi, clarkb: my spamassassin has been flagging some gerrit emails (it seems to be getting pretty frequent) as spam in part because of no rDNS. I could have sworn that used to be setup | 18:37 |
| mtreinish | here's an example message: http://paste.openstack.org/show/730470/ (with the spamassassin annotations) | 18:37 |
| mtreinish | I've setup a whitelist in the meantime, but I wanted to get to the bottom of what changed. (I'm assuming it's because I moved and changed isp and broke something adapting everything for that) | 18:38 |
| clarkb | ipv4 at least resolves forward and back | 18:39 |
| clarkb | we do cname review.o.o to review01.o.o | 18:40 |
| clarkb | maybe that confuses spamassassin? | 18:40 |
| *** ykarel|away has quit IRC | 18:41 | |
| mtreinish | clarkb: it looks like it wants an A record: https://wiki.apache.org/spamassassin/Rules/RDNS_NONE | 18:43 |
| mtreinish | but the last Recieved header is from review01.o.o so it should be checking that | 18:43 |
| mtreinish | hmm, I just logged into my mailserver and review01.o.o resolves both ways. So what is it complaining about? | 18:45 |
| *** Tim_ok has quit IRC | 18:47 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Cleanup old queries for fixed/invalid bugs https://review.openstack.org/604189 | 18:53 |
| mtreinish | oh maybe it's complaining about my server, apparently it doesn't resolve correctly in reverse | 18:54 |
| clarkb | spamassassin is checking the local dns lookup too? | 18:54 |
| mordred | that seems aggressive | 18:55 |
| mtreinish | I don't think so, but that's the only thing I saw in the Received headers that didn't have a valid rDNS | 18:55 |
| fungi | mtreinish: i'm using sa on my mta as well. i'll compare | 18:55 |
| fungi | X-Spam-Status: No, score=-2.7 required=5.0 tests=AWL,BAYES_00,SPF_PASS autolearn=ham autolearn_force=no version=3.4.1 | 18:56 |
| fungi | quite the discrepancy, same sa version | 18:57 |
| clarkb | I'm booting an ovh node without config drive now | 18:57 |
| clarkb | double checking glean does the right thing in that case and that I can ssh in as zuul with the master key | 18:57 |
| clarkb | if that works then ya I think we should consider running those nodes without config drives until we can find a better long term solution | 18:58 |
| fungi | mtreinish: possible because i'm using the autowhitelist and bayesian learning mechanisms, it's skipping a lot of checks which would otherwise fail | 18:58 |
| fungi | oh, here's one with a missing to: header | 18:59 |
| clarkb | mordred: ok that mostly works. Where it will break for us is multinode testing because the hostname isn't properly configured and we rely on hostnames and /etc/hosts and all that with nova and live migration | 19:00 |
| fungi | X-Spam-Status: No, score=-2.2 required=5.0 tests=AWL,BAYES_00,MISSING_HEADERS,SPF_PASS autolearn=no autolearn_force=no version=3.4.1 | 19:00 |
| mtreinish | I definitely have the bayesian learning turned on, don't know about autowhitelist though | 19:00 |
| clarkb | mordred: our best bet may be to set metadata that tells glean to fallback to dhcop | 19:00 |
| *** dtantsur|afk has quit IRC | 19:01 | |
| *** dtantsur has joined #openstack-infra | 19:01 | |
| clarkb | I've double checked taht the dhcp lease isn't setting a hostname either (it isn't) | 19:01 |
| *** e0ne has quit IRC | 19:03 | |
| *** ediardo has joined #openstack-infra | 19:04 | |
| fungi | mtreinish: i do agree that having no to: or cc: header but including a reply-to: is sort of broken. possible newer gerrit will improve on that | 19:04 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: convert docs to PTI https://review.openstack.org/559396 | 19:06 |
| mtreinish | I think I figured it, all emails are getting flagged no rdns on my server | 19:09 |
| *** smarcet has quit IRC | 19:09 | |
| mtreinish | I think it is because I got lazy in my workaround for my new isp blocking inbound port 25 and just setup an iptables forward rule from an outside server to redirect it to my local mail server on a different port | 19:10 |
| mtreinish | instead of doing a proper mail relay, and it realizes the hostnames don't match | 19:11 |
| ediardo | Hi o/... I have this annoying problem that I just cant find a solution to it. When other contributors want to add me as a reviewer they see this: Eddie Ramirez <eddie.ramirez@intel.com> does not identify a registered user or group ... | 19:11 |
| tbarron | so gouthamr is better than I am at remembering to update the #openstack-manila topic, what is needed to get him rights with chanserv to do this? | 19:11 |
| *** dpawlik has joined #openstack-infra | 19:12 | |
| ediardo | Any ideas on how could I fix this or where could I go for more help? | 19:12 |
| mtreinish | that coupled with the other rules, like the no to: or cc: is tipping it over the spam threshold | 19:12 |
| *** dtantsur has quit IRC | 19:13 | |
| fungi | ediardo: looks like you have two accounts in gerrit with that same e-,ail address | 19:13 |
| fungi | one from 2015-12-16 and another from 2016-02-10 | 19:13 |
| fungi | ediardo: older account 9155 is associated with username:ediardo while newer account 20505 has no ssh/api username | 19:14 |
| fungi | account 9155 is also the only one with ssh keys uploaded | 19:15 |
| ediardo | yes | 19:15 |
| ediardo | That's the one I used in the past | 19:15 |
| *** dpawlik has quit IRC | 19:16 | |
| fungi | ediardo: looks like you might have done an e-mail address change in launchpad/ubuntuone sso related signing in with a gmail address at some point? | 19:16 |
| *** dtantsur has joined #openstack-infra | 19:17 | |
| ediardo | ok, just went to launchpad website and saw gmail as the default one | 19:17 |
| fungi | ediardo: right now if you go to https://review.openstack.org/#/settings/ what Account ID number do you see? 9155 or 20505 | 19:17 |
| ediardo | Just set it back to intel.com | 19:17 |
| ediardo | 9155 | 19:18 |
| fungi | okay, i've marked the newer 20505 account invalid and this should resolve the problem. let us know if you run into any other issues you think might be related | 19:19 |
| ediardo | perfect, thanks a lot fungi! | 19:19 |
| *** ssbarnea|bkp2 has joined #openstack-infra | 19:20 | |
| fungi | tbarron: /msg chanserv access #openstack-manila add gouthamr +Aefortv | 19:20 |
| fungi | tbarron: let me know if it says you don't have permission or something | 19:21 |
| tbarron | fungi: ty! no permission issue so far as I can see | 19:22 |
| tbarron | gouthamr: would you try tweaking the topic in #openstack-manila when you have a chance? | 19:23 |
| *** jamesmcarthur has quit IRC | 19:25 | |
| fungi | tbarron: to be fair, +t is all that was really needed, but i took your request to mean you were interested in just promoting him to full channel operator permissions | 19:26 |
| tbarron | fungi: you read me rightly, thanks | 19:26 |
| fungi | this way he can pitch in on whatever same channel management tasks you can | 19:26 |
| tbarron | fungi: no reason for a spof on our team | 19:27 |
| openstackgerrit | Clark Boylan proposed openstack-infra/glean master: Add option to ignore config drive interfaces info https://review.openstack.org/604193 | 19:29 |
| clarkb | mordred: ^ totally untested | 19:29 |
| clarkb | mordred: but I think that will still configure the hostname like we need. Where its a little weird is it will not configure the dns server(s) but typically that is provided by dhcp so I think thiat is the right approach | 19:30 |
| *** e0ne has joined #openstack-infra | 19:30 | |
| clarkb | mordred: also in reading glean we could really stand to refactor it a bit | 19:32 |
| clarkb | the control flow for writing network configs in particular is painful in places | 19:32 |
| *** jamesmcarthur has joined #openstack-infra | 19:33 | |
| *** jamesmcarthur has quit IRC | 19:33 | |
| clarkb | After writing that I think I need a few minutes away from computers | 19:34 |
| *** e0ne has quit IRC | 19:34 | |
| *** panda is now known as panda|dinner | 19:34 | |
| *** amorin has quit IRC | 19:35 | |
| *** amorin has joined #openstack-infra | 19:38 | |
| *** jtomasek has joined #openstack-infra | 19:42 | |
| *** jamesmcarthur has joined #openstack-infra | 19:42 | |
| *** ssbarnea|bkp2 has quit IRC | 19:43 | |
| *** e0ne has joined #openstack-infra | 19:43 | |
| mordred | clarkb: yes. I believe we very much need to refactor it | 19:45 |
| *** jtomasek has quit IRC | 19:47 | |
| *** slaweq has joined #openstack-infra | 19:53 | |
| *** yamamoto has joined #openstack-infra | 19:58 | |
| AJaeger | config-core, OpenStackSDK is ready with with python3-first, could you review https://review.openstack.org/597579 , please? | 20:01 |
| *** yamamoto has quit IRC | 20:06 | |
| clarkb | mordred: this manual interface type must be ironic specific? | 20:07 |
| *** slaweq has quit IRC | 20:08 | |
| fungi | wow, current distutils-sig megathread is going off the rails. pipenv maintainers suggesting that since pip can't be imported as a stable api by pipenv we should expect that pip will fall by the wayside because pipenv will need to reimplement replacements for pip internals and pipenv is clearly the future for installing python packages. pip maintainers... disagree on this point | 20:11 |
| *** priteau has quit IRC | 20:13 | |
| *** Tim_ok has joined #openstack-infra | 20:17 | |
| *** jtomasek has joined #openstack-infra | 20:18 | |
| clarkb | as a non pipenv user I think I agree with pip on that one | 20:21 |
| *** panda|dinner is now known as panda | 20:24 | |
| fungi | a lot of it hinges on the fact that dstufft's pipfile idea got picked up by pipenv but hasn't been implemented in pip yet | 20:24 |
| clarkb | mordred: in trying to refactor this stuff I've found my patch to skip configuration is broken | 20:25 |
| *** bobh has quit IRC | 20:25 | |
| clarkb | mordred: so thats good I guess | 20:25 |
| *** bobh_ has joined #openstack-infra | 20:25 | |
| *** jtomasek has quit IRC | 20:27 | |
| *** owalsh has joined #openstack-infra | 20:27 | |
| *** ansmith has quit IRC | 20:29 | |
| gouthamr | fungi tbarron: ty, tested and chanserv accepts my commands on #openstack-manila | 20:30 |
| AJaeger | clarkb: want to +2A the coverage change now, please? It's after PTG... I rebased https://review.openstack.org/432836 to resolve conflicts | 20:34 |
| mordred | clarkb: yay! | 20:34 |
| fungi | gouthamr: glad it worked! let us know if you have any questions | 20:35 |
| gouthamr | fungi: +1 | 20:36 |
| AJaeger | mordred: , OpenStackSDK is ready with with python3-first, could you review https://review.openstack.org/597579 for project-config , please? | 20:38 |
| mordred | AJaeger: done - thanks! | 20:41 |
| *** dtruong has quit IRC | 20:47 | |
| *** holser_ has joined #openstack-infra | 20:49 | |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for OpenStackSDK repositories https://review.openstack.org/597579 | 20:51 |
| ianw | clarkb: catching up ... wow :/ this also possibly explains a bit about what's going on with ipv6 in OVH -- where launching the node openstacksdk thinks the hosts have ipv6 addresses, but they don't | 20:52 |
| dtroyer | I would love to get some eyes on https://review.openstack.org/599048 and https://review.openstack.org/599054 to wrap them up from PTG week… | 20:55 |
| clarkb | ianw: ya | 20:56 |
| *** trown is now known as trown|outtypewww | 20:56 | |
| clarkb | ianw: fwiw I'm working on a series of refactors to glean to make these easier to comprehend (at least for debian to start because I understand e/n/i) and hopelly that makes it easier to have glean handle this | 20:56 |
| clarkb | ianw: my patch to have glean handle it that I have already pushed is wrong | 20:56 |
| clarkb | does anyone know why manual mode configures bond details? that doesn't seem very manual to me | 20:58 |
| clarkb | meh we can keep them for backwad compat | 21:02 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Contrain query to prevent e-r from OOMing https://review.openstack.org/604173 | 21:05 |
| *** bobh_ has quit IRC | 21:06 | |
| clarkb | fungi: is hw-mac-address a valid token in debian /e/n/i config? | 21:10 |
| clarkb | hwaddress seems valid, glean sets hw-mac-address in some cases and the internets are failing me | 21:11 |
| *** agopi is now known as agopi|brb | 21:12 | |
| clarkb | AJaeger: did we send email about that? | 21:13 |
| clarkb | (sorry too many fires in the last couple days to keep everything straight) | 21:13 |
| openstackgerrit | Clark Boylan proposed openstack-infra/glean master: Use common function for debian bond mode https://review.openstack.org/604221 | 21:15 |
| openstackgerrit | Clark Boylan proposed openstack-infra/glean master: Check same debian interface path everywhere https://review.openstack.org/604222 | 21:15 |
| openstackgerrit | Clark Boylan proposed openstack-infra/glean master: Manage the debian interface header in one place https://review.openstack.org/604223 | 21:15 |
| openstackgerrit | Clark Boylan proposed openstack-infra/glean master: Consistent debian interface control flow https://review.openstack.org/604224 | 21:15 |
| openstackgerrit | Clark Boylan proposed openstack-infra/glean master: Debian interface config set bond once https://review.openstack.org/604225 | 21:15 |
| clarkb | mordred: ianw ^ I find that to be much more readable, but still not great | 21:15 |
| clarkb | it is a start to undersatnding it at least | 21:15 |
| openstackgerrit | Clark Boylan proposed openstack-infra/glean master: Add option to ignore config drive interfaces info https://review.openstack.org/604193 | 21:17 |
| clarkb | I think ^ actually does what we want now but is untested | 21:18 |
| *** agopi|brb has quit IRC | 21:18 | |
| *** ansmith has joined #openstack-infra | 21:19 | |
| *** hashar has quit IRC | 21:20 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: update Gentoo cache https://review.openstack.org/604003 | 21:22 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Cleanup old queries for fixed/invalid bugs https://review.openstack.org/604189 | 21:23 |
| *** Tim_ok has quit IRC | 21:23 | |
| ianw | clarkb: all the bond stuff, etc; that's not actually required for OVH, right? | 21:23 |
| clarkb | ianw: no, this is me trying to make that configuration flow readable so that I understand how to write https://review.openstack.org/604193 properly | 21:24 |
| clarkb | ianw: the diff on the two patchsets of that chagne will show you how my understanding of the code changed :) | 21:24 |
| clarkb | ianw: I got annoyed I couldn't understand it as written so took a machete to it. I think I achieved what is hopefully non controversial readability improvements (much less repetition for example) | 21:25 |
| clarkb | but also simpler control flow as there are fewer branches now | 21:25 |
| openstackgerrit | Merged openstack-infra/nodepool master: Re-register missing nodes in static driver https://review.openstack.org/600084 | 21:26 |
| clarkb | I'll update that function name | 21:26 |
| ianw | clarkb: cool; i've certainly noted a lot of what you've just fixed up. nothing like a crisis to help spur things along :) | 21:29 |
| openstackgerrit | Merged openstack-infra/nodepool master: Implement liveness check for static nodes https://review.openstack.org/601513 | 21:31 |
| ianw | this page on ipv6 https://docs.ovh.com/au/en/dedicated/network-ipv6/ is a bit ... "If you are using an OVH-provided OS template to install your server, you will see that you already have the first (MAIN) IPv6 configured right out of the box." | 21:31 |
| openstackgerrit | Merged openstack-infra/system-config master: Use include_tasks instead of include https://review.openstack.org/604101 | 21:33 |
| clarkb | ianw: it is impressive how rebase conflicty that one edit is :) | 21:35 |
| clarkb | I tried to order things such that they made sense to reviewers but it is all intertwined in the end | 21:35 |
| *** e0ne has quit IRC | 21:36 | |
| *** ssbarnea|bkp2 has joined #openstack-infra | 21:38 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/glean master: Use common function for debian bond mode https://review.openstack.org/604221 | 21:40 |
| openstackgerrit | Clark Boylan proposed openstack-infra/glean master: Check same debian interface path everywhere https://review.openstack.org/604222 | 21:40 |
| openstackgerrit | Clark Boylan proposed openstack-infra/glean master: Manage the debian interface header in one place https://review.openstack.org/604223 | 21:40 |
| openstackgerrit | Clark Boylan proposed openstack-infra/glean master: Consistent debian interface control flow https://review.openstack.org/604224 | 21:40 |
| openstackgerrit | Clark Boylan proposed openstack-infra/glean master: Debian interface config set bond once https://review.openstack.org/604225 | 21:40 |
| fungi | clarkb: `man interfaces` mentions hwaddress and not hw-mac-address (i've never seen the latter, maybe that's for rh configs instead?) | 21:40 |
| clarkb | ianw ^ hopefully I didn't screw that up | 21:40 |
| clarkb | fungi: its in the debian interface config. I think its a screw up and they meant hwaddress | 21:40 |
| clarkb | but that is getting into changing behavior vs just refactoring so I need to be more careful with it | 21:41 |
| clarkb | ianw: the diffs lgtm | 21:42 |
| clarkb | fungi: https://review.openstack.org/#/c/604225/2/glean/cmd.py line 835 for the example | 21:42 |
| fungi | man vlan-interfaces | 21:42 |
| fungi | seems that's not installed on my systems | 21:43 |
| fungi | maybe obsolete | 21:43 |
| clarkb | aha hw-mac-addresses is valid | 21:43 |
| fungi | https://manpages.debian.org/jessie/vlan/vlan-interfaces.5.en.html#EXTRA_IFACE_OPTIONS | 21:43 |
| clarkb | why is that different than hwaddress | 21:44 |
| clarkb | I think that means we need to add that to the static config though | 21:44 |
| clarkb | since right now you can only set the mac addr on a dhcp config'd interface | 21:45 |
| *** bobh has joined #openstack-infra | 21:45 | |
| clarkb | and maybe we even set it on the manual interface so that we can cut down on the configuration. STarts to push the definition of a manual interface but meh cloud | 21:45 |
| panda | is there a way to specify a on-abort: playbook to do some custom cleanup on the jobs when the job is aborted ? | 21:49 |
| clarkb | panda: no, aborted jobs are killed immediately by zuul. The reason for that is zuul no longer needs their results | 21:49 |
| *** jamesmcarthur has quit IRC | 21:50 | |
| panda | clarkb: mmhh, ok thanks. | 21:51 |
| *** jamesmcarthur has joined #openstack-infra | 21:51 | |
| clarkb | panda: jobs are aborted when a new patchset shows up (and so old patchset can never merge now) or if zuul has to rebuild a global state for the gate after a test has failed and a change is being evicted | 21:51 |
| clarkb | panda: are you trying to do something with these aborted jobs? you should be able to ignore them | 21:52 |
| *** jamesmcarthur has quit IRC | 21:53 | |
| *** bobh has quit IRC | 21:53 | |
| clarkb | prometheanfire: are we still using the write_gentoo_interfaces path in glean or is that all networkd now? I don't think we'll remove it (backward compat) but will help inform where to focus on cleaning stuff up | 21:53 |
| panda | clarkb: we create and use external resources while running some third party job. When jobs are aborted, we don't have any change to properly clean up those resources. | 21:53 |
| panda | clarkb: We'll find another way. | 21:54 |
| *** dtruong has joined #openstack-infra | 21:54 | |
| clarkb | panda: I think this was why the old tripleo testing didn't the mutual locking thing with gearman. Then if one side went away the other knew the resources were no longer locked | 21:55 |
| *** jamesmcarthur has joined #openstack-infra | 21:55 | |
| *** boden has quit IRC | 21:56 | |
| clarkb | ianw: the rh code in glean is actually a lot easier to follow. Only problem is it doesn't support static ipv6 (but I'll ignore that for now as we don't need that in any clouds) | 21:57 |
| *** kgiusti has left #openstack-infra | 21:58 | |
| ianw | clarkb: except maybe ovh? | 21:58 |
| *** ssbarnea|bkp2 has quit IRC | 21:59 | |
| *** bobh has joined #openstack-infra | 21:59 | |
| *** bobh has quit IRC | 21:59 | |
| jbryce | can i ask a dumb list administration question? i'm trying to approve one of my own messages that has been queued for moderation on foundation@lists.o.o, but there's so much spam in the moderation queue that the page keeps timing out when i submit the form. is there anyway to make this work? | 21:59 |
| fungi | wow, i don't think i've seen a list with so much spam in the moderation queue that the interface times out loading | 22:00 |
| *** holser_ has quit IRC | 22:00 | |
| jbryce | the interface loads, but on form submission, it times out | 22:00 |
| clarkb | ianw: ya, but we've only supported ipv4 there anyway, but also you have to get the ipv6 info from the neutron api, it isn't in the config drive :( | 22:00 |
| *** jento has quit IRC | 22:01 | |
| clarkb | ianw: prometheanfire I don't think my patch for ovh to ignore interfacese will work for the networkd code | 22:01 |
| fungi | jbryce: there is an alternative approval mechanism list admins can perform by sending e-mail, but with the -owner addresses getting spammed too and so black-holed i'm not sure how to go about that... there's also a command-line list management utility i can try, i think | 22:01 |
| *** jento has joined #openstack-infra | 22:03 | |
| jbryce | fungi: i'm also fine discard all the queued messages and then resending my message if that's an easier pass or possible from the command-line | 22:03 |
| jbryce | path* | 22:03 |
| *** ruhe has quit IRC | 22:03 | |
| clarkb | prometheanfire: ianw nevermind I see it now | 22:03 |
| fungi | oh, looks like i can delete them from the heldmsg directory according to https://wiki.list.org/DOC/How%20do%20I%20handle%20list%20moderation%20from%20the%20command-line%3F | 22:04 |
| *** DinaBelova has quit IRC | 22:04 | |
| clarkb | ianw: ignoring it being untested, do you think the approach of setting nova metadata (which I I don't think we can actually do arbitrarily in nodepool so that needs fixing too) in https://review.openstack.org/#/c/604193/2/glean/cmd.py is a reasonable one to take here? | 22:05 |
| jbryce | fungi: buh-lete away | 22:05 |
| *** ruhe has joined #openstack-infra | 22:06 | |
| *** DinaBelova has joined #openstack-infra | 22:06 | |
| fungi | 9497 messages held! wow | 22:06 |
| fungi | that's just the foundation@ ml | 22:06 |
| *** jamesmcarthur has quit IRC | 22:06 | |
| fungi | i should be able to isolate your post with a quick grep and blow away the others | 22:07 |
| *** jamesmcarthur has joined #openstack-infra | 22:07 | |
| *** yamamoto has joined #openstack-infra | 22:08 | |
| fungi | for ~some~ definition of "quick" that is | 22:08 |
| jbryce | yeah with the owner alerts off because of spam i haven't checked the moderation queue in a long while | 22:08 |
| *** dtruong has quit IRC | 22:09 | |
| *** dtruong has joined #openstack-infra | 22:09 | |
| prometheanfire | clarkb: yes, gentoo CAN be networkd, but typically isn't (see this, only one set is systemd/networkd) http://distfiles.gentoo.org/experimental/amd64/openstack/ | 22:10 |
| prometheanfire | clarkb: anything I need to review? | 22:10 |
| clarkb | prometheanfire: https://review.openstack.org/#/c/604193/2 mostly double checking that that should work on all the distros because we'll set dhcp on any interfaces not listed in the config drive | 22:10 |
| ianw | clarkb: does cloud-init not hit it because it doesn't look at the config-drive? | 22:12 |
| prometheanfire | clarkb: looks good, since that's pre-backend selection | 22:12 |
| prometheanfire | ianw: cloud-init falls back to config-drive | 22:12 |
| *** jamesmcarthur has quit IRC | 22:13 | |
| prometheanfire | so they may hit it if network-config fails | 22:13 |
| clarkb | I think their cloud init must be hard configured to use dhcp | 22:13 |
| prometheanfire | or that | 22:13 |
| clarkb | prometheanfire: its actually completely configurable | 22:13 |
| fungi | jbryce: okay, check again and yours should be the only one in the moderation queue now | 22:13 |
| clarkb | which is one of the big issues with cloud init it does all the things and people get carried away | 22:13 |
| *** jamesmcarthur has joined #openstack-infra | 22:13 | |
| prometheanfire | ++ | 22:13 |
| jbryce | fungi: checking now....still seems to have quite a few messages in the queue.... | 22:14 |
| clarkb | like lets just reformat that volume you have ... | 22:14 |
| clarkb | how helpful of you:( | 22:14 |
| prometheanfire | really just a hook into a config-management system is all that's needed imo (+ networking) | 22:15 |
| prometheanfire | while people are looking at glean, mind looking at https://review.openstack.org/#/c/493443/ (simple) | 22:16 |
| clarkb | prometheanfire: all that is needed imo is set the ssh key and assume dhcp | 22:16 |
| clarkb | but people don't like dhcp for some reason or another | 22:16 |
| fungi | jbryce: i'll see if i can figure out why that is. the .pck files for them have all been deleted except for yours | 22:16 |
| ianw | fungi / clarkb : apropos looking at cloud servers, another thing that i think is different is you have to be root on bridge for /etc/clouds which seems like unnecessary privilege escalation when i'm just getting server details | 22:16 |
| prometheanfire | it's another failure mode | 22:16 |
| prometheanfire | also, can someone delete the gentoo image, I think I'm ready for the next round | 22:17 |
| clarkb | ianw: ya, talking at the PTG corvus seemed to strongly prefer the simplicity of root/sudo for that | 22:17 |
| clarkb | ianw: since it avoids needing to curate groups across all the things | 22:17 |
| ianw | yeah, but it also means i'm root & using openstack client from a venv in my homedir to query server status etc | 22:18 |
| jbryce | fungi: ok. maybe there's another way you can help me achieve what i ultimately was trying to do. i want to post the pdf of the slides from the board meeting publicly. it's a 3MB file which exceeds the wiki limit. i was attached it to my message (which is why it got moderated). any thoughts on some other place i can post it that is globally accessible? | 22:19 |
| fungi | jbryce: i've tried removing yours as well since you seemed okay with re-sending. does refreshing the moderation page look any better? | 22:19 |
| clarkb | ianw: ya... but our ansible is basically that too :/ | 22:19 |
| *** jamesmcarthur has quit IRC | 22:20 | |
| *** jamesmcarthur has joined #openstack-infra | 22:20 | |
| ianw | clarkb: so just trying to understand ... 158.69.64.10 is a good example because it's a replacement mirror i booted in OVH yesterday when i thought the problem may have been their networking interface etc | 22:22 |
| jbryce | fungi: it's not even loading for me now | 22:22 |
| fungi | it may need a new post to hit the moderation queue | 22:23 |
| jbryce | perhaps it's trying to find it's way through the newly empty space | 22:23 |
| *** jamesmcarthur has quit IRC | 22:23 | |
| ianw | clarkb: in meta-data, that has local-ipv4 as it's address : 158.69.64.10 | 22:23 |
| clarkb | ianw: the problem is the netmask and routes | 22:23 |
| clarkb | the ip address is correct, but it puts it on a /19 if looking at config drive which does not work | 22:23 |
| clarkb | if you dhcp (whihc yuor replacement mirror did) it gets a /32 and the funny route for the gateway | 22:24 |
| *** yamamoto has quit IRC | 22:24 | |
| ianw | but why did it dhcp? it seems like cloud-init would have picked it up out of metadata? | 22:24 |
| clarkb | cloud init doesn't necessarily use metadata | 22:25 |
| clarkb | its configurable and we used the cloud's vm image | 22:25 |
| fungi | jbryce: the mailing list seems like probably the best option (though, oof, 3mb is a pretty sizeable message and it'll probably be more like 3.5 with encoding). alternatively i can try to get the pdf onto the wiki... we turned off pdf uploads a while back because search engines index them so wiki spammers were trying to abuse that to sneak past content filters | 22:25 |
| ianw | right, that's what i'm looking at, and i'm not seeing it being told to use dhcp via a "network:" in the config, etc | 22:25 |
| clarkb | ianw: actually I wonder if the VM image has /e/n/i preconfigured to dhcp | 22:25 |
| clarkb | in which case cloud init would ignore it | 22:26 |
| clarkb | similar to how glean ignores preconfigured interfaces? | 22:26 |
| ianw | theres /etc/network/interfaces.d/cat 50-cloud-init.cfg which says "# This file is generated from information provided by | 22:27 |
| ianw | # the datasource." | 22:27 |
| *** rh-jelabarre has quit IRC | 22:28 | |
| jbryce | fungi: yay...it worked. thanks for the assist | 22:29 |
| clarkb | ianw: docs /var/lib/dhclient show a lease? | 22:29 |
| fungi | jbryce: thanks for helping me learn that if deleting files manually from a huge moderation queue, we need a new message to land in the queue before the webui will load it! | 22:29 |
| ianw | clarkb: oh, well that file has "iface ens3 inet dhcp" ... but why did it decide to do that, when it has ip address info in meta-data? | 22:30 |
| ianw | that's sort of what i'm wondering, if there's some magic hidden ... something ... that we can determine this by ... | 22:30 |
| clarkb | huh | 22:33 |
| mordred | panda, clarkb: there was discussion a little while ago with corvus and SpamapS about adding a "cleanup" phase - I'm not sure where the thinking got on that ... I also know that fatih has requested the ability to prevent some things from being aborted because over in his world they have some tests that run like a week on real hardware so the cost of an abort is pretty high | 22:35 |
| mordred | panda, clarkb: all of that to say - collecting the use-case for cleanup-related-to-abort would be helpful as we think about if/how to implement such a feature | 22:35 |
| ianw | heh, it's been so long since i looked at cloud-init i still have a bzr checkout of it | 22:36 |
| mordred | ianw: nice | 22:36 |
| panda | mordred: how do I contribute to the collection of use-cases | 22:36 |
| *** rlandy is now known as rlandy|bbl | 22:37 | |
| *** dmsimard has quit IRC | 22:37 | |
| mordred | panda: that's a great question, I can't find a story on this topic currently, so maybe for now just file a story at https://storyboard.openstack.org/#!/project/openstack-infra/zuul | 22:41 |
| clarkb | mordred: for fatih's use case I think you'd construct the pipeline to not be abortable (eg post) | 22:42 |
| clarkb | for the other an always run this cleanup could be useful | 22:42 |
| mordred | yah. the cleanup usecase is the one I was thikning of more | 22:42 |
| panda | mordred: hope is enough https://storyboard.openstack.org/#!/story/2003823 | 22:46 |
| mordred | panda: yes - thanks! | 22:48 |
| *** spsurya has quit IRC | 22:48 | |
| *** david-lyle has joined #openstack-infra | 22:49 | |
| *** spsurya has joined #openstack-infra | 22:50 | |
| *** dklyle has quit IRC | 22:51 | |
| *** eernst has joined #openstack-infra | 22:52 | |
| *** rcernin has joined #openstack-infra | 22:53 | |
| *** eernst has quit IRC | 22:54 | |
| *** ssbarnea|bkp2 has joined #openstack-infra | 22:55 | |
| panda | is it normal to have a job queued since 18 minutes at this hour ? :( | 22:58 |
| openstackgerrit | Adrian Turjak proposed openstack-infra/project-config master: Start migrating Adjutant to Storyboard https://review.openstack.org/604233 | 23:01 |
| adriant | ^ beyond that, what else must I do to start being able to create tasks in storyboard for Adjutant? | 23:03 |
| * adriant wants to get away from launchpad | 23:03 | |
| mriedem | gmann: clarkb: fungi: mtreinish: so based on http://status.openstack.org/elastic-recheck/#1783405 i think we might want to promote https://review.openstack.org/#/c/603900 | 23:04 |
| mriedem | once/if gmann approves of course | 23:04 |
| mriedem | because tempest-full is branchless and going to be failing / timing out on all projects that run with the integrated-gate on all branches | 23:05 |
| panda | 30 minutes queued ... | 23:09 |
| fungi | mriedem: can do. gmann: let us know when you've approved it and we can bump it to the top of the integrated gate queue | 23:09 |
| fungi | adriant: do you have any bugs in lp that you want imported? | 23:09 |
| *** mriedem is now known as mriedem_away | 23:10 | |
| panda | wow 120 patches in check, that's probably the reason | 23:10 |
| fungi | adriant: see https://docs.openstack.org/infra/storyboard/migration.html if you haven't already | 23:10 |
| adriant | fungi: we have 2 bugs only one of which is open, so I can move that manually, and... blueprints we don't migrate right? | 23:10 |
| adriant | so that's manual as well? | 23:11 |
| fungi | adriant: correct | 23:11 |
| *** slaweq has joined #openstack-infra | 23:11 | |
| adriant | cool, yeah, I just then need projects created on storyboard with myself getting access and I'll handle the rest | 23:11 |
| fungi | in that case once 604233 gets approved, merges and you wait a few minutes you can just do that | 23:12 |
| adriant | fantastic | 23:12 |
| *** dpawlik has joined #openstack-infra | 23:12 | |
| fungi | but i recommend reading that documentation page if you haven't already | 23:12 |
| adriant | I did, but mostly seemed not too useful since i'm not migrating a lot of bugs | 23:13 |
| adriant | I'll be updating docs/launchpad and locking stuff once all is manually moved | 23:14 |
| fungi | cool, just making sure you'd seen it | 23:14 |
| adriant | np, and thanks :) | 23:14 |
| adriant | looking forward to using Storyboard | 23:14 |
| fungi | my pleasure! | 23:14 |
| clarkb | ianw: sorry I've had to context switch in paperwork duties post travel (still can't file my expenses because not all have posted :( ) any luck figuring out what forces cloud init to dhcp there? | 23:15 |
| *** slaweq has quit IRC | 23:16 | |
| ianw | clarkb: so i think that the presence of http://169.254.169.254/latest/meta-data/local-ipv4 doesn't mean anything to cloud-init | 23:16 |
| ianw | a network/interfaces/macs/mac/local-ipv4s might | 23:16 |
| *** dpawlik has quit IRC | 23:16 | |
| ianw | the meta-data specs say : The private IPv4 address of the instance. In cases where multiple network interfaces are present, this refers to the eth0 device (the device for which the device number is 0). | 23:17 |
| ianw | considering you can't seem to set a subnet, etc ... i wonder if this is basically a big red herring | 23:17 |
| clarkb | grepping for dhcp in /var/lib/cloud shows nothing | 23:17 |
| ianw | i also note that harlowja has a copyright on the file looking at this :) | 23:18 |
| clarkb | ianw: actually I wonder if cloud init just assumes dhcp | 23:18 |
| clarkb | I know old cloud init didn't support the static conf that rax wanted for a long time | 23:18 |
| clarkb | and that static conf is basically what we are running with glean now | 23:18 |
| fungi | cloud-init will look for the network data in the configdrive and via the ec2-style metadata service | 23:19 |
| ianw | this is the crux of it i think -> https://git.launchpad.net/cloud-init/tree/cloudinit/sources/DataSourceEc2.py#n500 | 23:20 |
| fungi | or rather the openstack metadata service which masquerades in part as an ec2 metadata service | 23:20 |
| prometheanfire | anyone mind deleting the gentoo image? | 23:20 |
| clarkb | ianw: ya that seems to say use dhcpv4 if we have public interface? | 23:21 |
| clarkb | (and similar for ipv6) | 23:21 |
| ianw | fungi / clarkb: the meta-data looks like -> http://paste.openstack.org/show/730491/ ... note there's no "network/" directory. so it sees nothing, and falls back to dhcp afacit | 23:21 |
| clarkb | got it | 23:21 |
| clarkb | prometheanfire: do we need to delete it or build a new one? | 23:22 |
| ianw | i think deleting it implies building another one these days | 23:22 |
| prometheanfire | clarkb: just build a new one | 23:22 |
| clarkb | it does if we don't have one new enough otherwise | 23:22 |
| prometheanfire | last time someone jsut deleted it | 23:22 |
| clarkb | prometheanfire: building on nb02 | 23:23 |
| fungi | yeah, no voting jobs use that node type yet | 23:23 |
| fungi | so i simply did a nodepool dib-image-delete on them | 23:24 |
| prometheanfire | cool | 23:24 |
| ianw | ok, http://169.254.169.254/latest/meta-data/local-ipv4 is i guess a red-herring. it's not intended for network configuration in any way, but more as "this is where you'll be able to ping me" | 23:26 |
| ianw | ergo it not having associated subnet masks, etc | 23:26 |
| *** andreykurilin has quit IRC | 23:34 | |
| *** andreykurilin has joined #openstack-infra | 23:35 | |
| *** Diabelko has quit IRC | 23:36 | |
| *** rcernin has quit IRC | 23:36 | |
| *** rcernin has joined #openstack-infra | 23:36 | |
| *** panda is now known as panda|off | 23:39 | |
| *** gyee has quit IRC | 23:41 | |
| *** Diabelko has joined #openstack-infra | 23:41 | |
| *** jamesmcarthur has joined #openstack-infra | 23:43 | |
| *** Swami has quit IRC | 23:45 | |
| *** jamesmcarthur has quit IRC | 23:46 | |
| *** jamesmcarthur_ has joined #openstack-infra | 23:46 | |
| *** dtruong has quit IRC | 23:46 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/glean master: Add option to ignore config drive interfaces info https://review.openstack.org/604193 | 23:46 |
| clarkb | yay for testing | 23:46 |
| clarkb | ianw: if we decide to go with ^ we should build some images with that and test it in ovh | 23:47 |
| *** dtruong has joined #openstack-infra | 23:47 | |
| clarkb | I'm running out of day (have to go cook dinner) but can pick that up tomorrow probably | 23:47 |
| ianw | clarkb: yeah, sorry going over ground you've covered just to context switch myself in. the netmask in config-drive does appear totally different to what you get via dhcp | 23:49 |
| ianw | if you boot even cloud-init with config-drive, it fails same way | 23:49 |
| ianw | i feel like this is probably something ovs would like to fix before they roll out whatever upgrade they've done across all regions ... | 23:50 |
| ianw | ovh even | 23:51 |
| *** sthussey has quit IRC | 23:52 | |
| clarkb | ianw: ya amorin was here earlier (but is in european timezones I think, maybe you want to catch up in your evening) and it seems their intent isn't to have that exposed | 23:52 |
| clarkb | but new neutron/nova are doing it and mriedem didn't know of a way to tell it to not expose that | 23:53 |
| harlowja | ianw, i am everywhereeee, lol | 23:55 |
| ianw | clarkb: ahhh, ok. so that explains why the config drive on the host booted a week ago has *no* network_data.json and hence didn't have this problem | 23:56 |
| ianw | ok, i think i'm caught up :) | 23:57 |
| clarkb | ianw ya and fundamentally its a mismatch in how neutron and nova operate and how they run their networking out if band of neutron | 23:57 |
| *** dpawlik has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!