| *** Wryhder has joined #openstack-infra | 00:00 | |
| *** cgoncalves has quit IRC | 00:00 | |
| *** Lucas_Gray has quit IRC | 00:00 | |
| *** icey has quit IRC | 00:00 | |
| *** Wryhder is now known as Lucas_Gray | 00:01 | |
| *** sgw has quit IRC | 00:01 | |
| *** armax has quit IRC | 00:07 | |
| ianw | sorry late start due to bureaucracy ... looking at nb now | 00:09 |
|---|---|---|
| *** yamamoto has joined #openstack-infra | 00:15 | |
| fungi | bureaucracy, the popular board game from milton bradley | 00:18 |
| *** yamamoto has quit IRC | 00:20 | |
| ianw | https://imgur.com/a/BFnjYLY <- outside the house i just moved out of ... the one where i couldn't see across the road for the smoke a few weeks ago | 00:22 |
| ianw | the same cable it took the isp weeks and week to figure out how to connect to when it wasn't under a tree, too | 00:24 |
| fungi | good timing! | 00:25 |
| ianw | TASK [puppet : run puppet] completes in like 1 second on this new nb03 | 00:39 |
| ianw | something is clearly up but i'm not sure where the logs are | 00:39 |
| clarkb | syslog | 00:40 |
| clarkb | its probably not doing much at this point. I think i got everything installed | 00:40 |
| *** goldyfruit has quit IRC | 00:42 | |
| ianw | yeah, i think that's right .. .but the deamon isn't starting | 00:43 |
| ianw | Feb 10 00:41:36 nb03 puppet-user[2725]: Finished catalog run in 19.82 seconds | 00:44 |
| ianw | i don't know why irqbalance is flooding the logs | 00:44 |
| *** goldyfruit has joined #openstack-infra | 00:44 | |
| ianw | ImportError: cannot import name 'DependencyWarning' ... that's the issue now i gues | 00:45 |
| ianw | that comes from inside keystoneauth1/session.py but suggests requests is at fault | 00:46 |
| openstackgerrit | Merged opendev/system-config master: Run apt-get autoremove after rm cloud-init https://review.opendev.org/706624 | 00:50 |
| ianw | ok, i don't want to think about it too much, but a manual pip reinstall of requests seems to get nodepool-builder daemon going further | 00:51 |
| ianw | ahh, now zookeeper can't talk ... and i bet that's the firewall | 00:51 |
| ianw | i think it's time to switch the dns and then we can sort out fw rules | 00:52 |
| clarkb | ++ | 00:53 |
| *** d34dh0r53 has quit IRC | 01:04 | |
| *** cloudnull has quit IRC | 01:04 | |
| *** cloudnull has joined #openstack-infra | 01:05 | |
| *** d34dh0r53 has joined #openstack-infra | 01:05 | |
| *** slaweq has joined #openstack-infra | 01:05 | |
| ianw | clarkb: did you remove nb03 A entry at some point? | 01:10 |
| ianw | ... don't worry sorry | 01:13 |
| ianw | for some reason, ctrl-f would not find it in the rax dns webpage | 01:13 |
| ianw | #status log switched nb03.openstack.org dns from old server in london (213.146.141.47) to new us-based host (139.178.85.141) | 01:14 |
| clarkb | ^F behaves weord there | 01:15 |
| ianw | yeah, scrolling through 400 entries is fun | 01:15 |
| *** openstackstatus has joined #openstack-infra | 01:16 | |
| *** ChanServ sets mode: +v openstackstatus | 01:16 | |
| ianw | #status log switched nb03.openstack.org dns from old server in london (213.146.141.47) to new us-based host (139.178.85.141) | 01:18 |
| openstackstatus | ianw: finished logging | 01:18 |
| ianw | (i also restarted statusbot) | 01:18 |
| ianw | anyone mind if i cycle zk01, zk02, zk03 in order to refresh their iptables rules? | 01:26 |
| clarkb | should be fine particularly if done in sequence | 01:27 |
| *** yamamoto has joined #openstack-infra | 01:28 | |
| *** yamamoto has quit IRC | 01:33 | |
| ianw | hrrm, i have restarted the zk hosts, and nb03 (new) still can't seem to talk to them | 01:38 |
| ianw | # curl https://ifconfig.me | 01:50 |
| ianw | 139.178.85.226 | 01:50 |
| *** Lucas_Gray has quit IRC | 01:50 | |
| ianw | yeah, if i try to ssh to a remote host from nb03 I see a rejection for "user root 139.178.85.226 port 30309 [preauth]" | 01:56 |
| ianw | there must be some sort of nat going on? but i can ping it's "public" ip address? | 01:57 |
| ianw | and indeed ssh into it @ 139.178.85.141 ... this is weird | 01:58 |
| *** yamamoto has joined #openstack-infra | 02:01 | |
| *** yamamoto has quit IRC | 02:06 | |
| clarkb | huh | 02:13 |
| *** yamamoto has joined #openstack-infra | 02:15 | |
| ianw | i can't see this 226 address in any of the routers, subnets, etc. i'm officially stumped | 02:18 |
| ianw | kevinz: ^ in short; nb03 is working at 139.178.85.141, but somehow external connections seem to come from 139.178.85.226. our firewalls don't understand this, so the host is currently not working | 02:19 |
| ianw | notes updated in https://storyboard.openstack.org/#!/story/2007195 | 02:19 |
| *** gagehugo has joined #openstack-infra | 02:39 | |
| kevinz | ianw: Hi, 226 is the totally cluster router for this cluster | 02:49 |
| kevinz | ianw: let me chek | 02:49 |
| kevinz | ianw: I will check the router configuration | 02:51 |
| *** gagehugo has quit IRC | 02:51 | |
| *** yamamoto has quit IRC | 02:51 | |
| *** yamamoto has joined #openstack-infra | 02:51 | |
| *** gagehugo has joined #openstack-infra | 02:51 | |
| ianw | kevinz: it's sort of like NAT ... but not? | 03:01 |
| kevinz | ianw: router here is sort of like NAT | 03:02 |
| rm_work | anyone happen to have a copy of `CentOS-8-GenericCloud-8.0.1905-1.x86_64.qcow2` sitting around? :D | 03:07 |
| rm_work | it's gone from the known universe AFAICT but I need a copy | 03:07 |
| ianw | i got a rhel 8.0-1854 but not centos8 sorry | 03:08 |
| *** jamesmcarthur has joined #openstack-infra | 03:10 | |
| rm_work | T_T | 03:27 |
| *** yamamoto has quit IRC | 03:38 | |
| *** jamesmcarthur has quit IRC | 03:40 | |
| *** tkajinam has quit IRC | 03:40 | |
| *** yamamoto has joined #openstack-infra | 03:41 | |
| *** tkajinam has joined #openstack-infra | 03:41 | |
| *** jamesmcarthur has joined #openstack-infra | 03:44 | |
| *** zxiiro has quit IRC | 03:45 | |
| *** psachin has joined #openstack-infra | 03:45 | |
| *** jamesmcarthur has quit IRC | 03:49 | |
| *** sgw has joined #openstack-infra | 03:53 | |
| *** yamamoto has quit IRC | 03:57 | |
| *** sgw has quit IRC | 04:08 | |
| *** udesale has joined #openstack-infra | 04:09 | |
| *** sgw has joined #openstack-infra | 04:26 | |
| *** ykarel|away is now known as ykarel | 04:27 | |
| *** ramishra has joined #openstack-infra | 04:29 | |
| *** yamamoto has joined #openstack-infra | 04:31 | |
| *** ricolin has joined #openstack-infra | 04:33 | |
| *** ykarel is now known as ykarel|afk | 04:34 | |
| *** yamamoto has quit IRC | 04:40 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Migrate AFS publishing to mirror-update.opendev.org https://review.opendev.org/706731 | 04:49 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add project.tarballs to periodic AFS release https://review.opendev.org/706732 | 04:49 |
| *** ykarel|afk is now known as ykarel | 04:56 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Publish tarballs artifacts to AFS in parallel https://review.opendev.org/706733 | 05:03 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Remove tarball artifact publishing to static site https://review.opendev.org/706734 | 05:03 |
| kevinz | ianw: finished. Pls check again, One rules in router has been wrong configured. | 05:09 |
| kevinz | ianw: ubuntu@test:~$ curl https://ifconfig.me | 05:09 |
| kevinz | 139.178.85.145 | 05:09 |
| ianw | kevinz: ooohh, looking good, i can telnet to the right place | 05:10 |
| kevinz | ianw: ace | 05:10 |
| ianw | yep, it's talking | 05:10 |
| ianw | i'm just going to reboot it and see if nodepool-buidler starts of its own accord | 05:11 |
| ianw | yay, it did \o/ ! it's building now | 05:12 |
| ianw | so let's give it some time and see if it uploads correctly ... i wouldn't mind betting something like this was the problem with the uk->us connections anyway? | 05:12 |
| ianw | i don't think it's bad we moved it ... it was only our zk firewalls rejecting the host that tipped us off, it would be hard to regonise on a testing node | 05:15 |
| kevinz | ianw: sure np, hope it works fine. I will check if uk-> us https works | 05:18 |
| kevinz | ianw: uk->us https connection still has issue :( | 05:21 |
| *** yamamoto has joined #openstack-infra | 05:26 | |
| ianw | ok; first builds take a while as it caches all of git | 05:27 |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #openstack-infra | 05:34 | |
| *** dSrinivas has joined #openstack-infra | 05:39 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Remove tarball artifact publishing to static site https://review.opendev.org/706734 | 05:40 |
| *** raukadah is now known as chkumar|rover | 05:44 | |
| *** lmiccini has joined #openstack-infra | 06:30 | |
| *** cgoncalves has joined #openstack-infra | 06:31 | |
| *** cgoncalves has quit IRC | 06:31 | |
| *** cgoncalves has joined #openstack-infra | 06:32 | |
| *** dpawlik has joined #openstack-infra | 06:44 | |
| *** dpawlik has quit IRC | 06:50 | |
| *** dpawlik has joined #openstack-infra | 06:55 | |
| ianw | looks like linaro-us has images uploaded now, so that bit is working | 06:57 |
| ianw | there's hundreds of "failed" images .. i'm trying to delete them with nodepool dib-image-delete | 06:58 |
| kevinz | ianw: why a lot of failed images in nodepool? | 07:02 |
| ianw | i don't exactly know why they're in that state in the backing db | 07:04 |
| *** jtomasek has joined #openstack-infra | 07:06 | |
| *** jtomasek_ has joined #openstack-infra | 07:09 | |
| *** jtomasek has quit IRC | 07:12 | |
| *** yamamoto has quit IRC | 07:16 | |
| *** dpawlik has quit IRC | 07:16 | |
| *** yamamoto has joined #openstack-infra | 07:18 | |
| *** dpawlik has joined #openstack-infra | 07:19 | |
| *** pgaxatte has joined #openstack-infra | 07:20 | |
| kevinz | ianw: so now we can move to instance launching? | 07:22 |
| *** yamamoto_ has joined #openstack-infra | 07:29 | |
| *** yamamoto has quit IRC | 07:33 | |
| *** tosky has joined #openstack-infra | 07:35 | |
| *** dciabrin has joined #openstack-infra | 07:37 | |
| dciabrin | morning o/ | 07:38 |
| *** ykarel is now known as ykarel|lunch | 07:38 | |
| *** Tengu has quit IRC | 07:44 | |
| *** Tengu has joined #openstack-infra | 07:46 | |
| *** ociuhandu has joined #openstack-infra | 07:47 | |
| *** slaweq has quit IRC | 07:53 | |
| *** imacdonn has quit IRC | 07:53 | |
| *** imacdonn has joined #openstack-infra | 07:53 | |
| *** ociuhandu has quit IRC | 08:07 | |
| *** ociuhandu has joined #openstack-infra | 08:08 | |
| *** slaweq has joined #openstack-infra | 08:09 | |
| *** iurygregory has joined #openstack-infra | 08:10 | |
| *** yamamoto_ has quit IRC | 08:10 | |
| *** ociuhandu has quit IRC | 08:14 | |
| *** dchen has quit IRC | 08:14 | |
| *** gfidente has joined #openstack-infra | 08:20 | |
| *** tkajinam has quit IRC | 08:23 | |
| *** yamamoto has joined #openstack-infra | 08:26 | |
| *** rakhmerov has joined #openstack-infra | 08:27 | |
| *** tesseract has joined #openstack-infra | 08:30 | |
| *** carli has joined #openstack-infra | 08:31 | |
| *** ykarel|lunch is now known as ykarel | 08:33 | |
| openstackgerrit | M V P Nitesh proposed openstack/diskimage-builder master: efibootmgr is need to create boot entries https://review.opendev.org/671456 | 08:34 |
| *** ralonsoh has joined #openstack-infra | 08:38 | |
| *** rpittau|afk is now known as rpittau | 08:41 | |
| *** jpena|off is now known as jpena | 08:54 | |
| *** jberg-dev has joined #openstack-infra | 09:00 | |
| *** lucasagomes has joined #openstack-infra | 09:00 | |
| *** dtantsur|afk is now known as dtantsur | 09:01 | |
| mnaser | hi infra | 09:15 |
| mnaser | i think the opensuse mirrors @ vexxhost (or just mirrors in general) seem borked? | 09:15 |
| mnaser | [repo-update|http://mirror.sjc1.vexxhost.openstack.org/opensuse/update/leap/15.1/oss/] Valid metadata not found at specified URL | 09:15 |
| mnaser | https://09dae708dd145b62560d-fa5c157bc6c850b5e031899614317800.ssl.cf5.rackcdn.com/703572/5/check/openstack-ansible-deploy-aio_metal-opensuse-15/c631209/job-output.txt | 09:15 |
| mnaser | nope, not juts mirrors at our side | 09:17 |
| mnaser | [repo-update|http://mirror.regionone.limestone.openstack.org/opensuse/update/leap/15.1/oss/] Valid metadata not found at specified URL | 09:17 |
| *** hashar has joined #openstack-infra | 09:19 | |
| *** apetrich has joined #openstack-infra | 09:21 | |
| ianw | mnaser: it seems like we're mirroring what's upstream ok ... either that mirror is out of sync, or something wrong with it's upstream (http://files.openstack.org/mirror/logs/rsync-mirrors/opensuse.log) | 09:24 |
| mnaser | poop | 09:26 |
| frickler | infra-root: ze12 has oomed and is failing for finger connections. I'm not restarting it yet in case someone wants to dig further | 09:28 |
| ianw | frickler: yeah, i think we know it happens, i'm not sure there would be much investigation, i'd probably restart | 09:28 |
| frickler | ianw: so just restart zuul-executor or the whole node? I haven't done this in a while | 09:29 |
| ianw | frickler: stop the executor, and wait (quite a bit it will feel like) till everything disappears, then restart is what i do | 09:29 |
| frickler | ianw: o.k., I'll give it a try, thx | 09:30 |
| openstackgerrit | Sorin Sbarnea proposed openstack/project-config master: List oooq members as irc OPs https://review.opendev.org/706767 | 09:37 |
| *** derekh has joined #openstack-infra | 09:38 | |
| ianw | kevinz: i think that linaro-us will just start taking nodes now the images are updated -> https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl03.openstack.org.yaml#L414 | 09:41 |
| *** sshnaidm has quit IRC | 09:41 | |
| *** yamamoto has quit IRC | 09:41 | |
| *** sshnaidm has joined #openstack-infra | 09:42 | |
| *** ociuhandu has joined #openstack-infra | 09:42 | |
| frickler | ianw: kevinz: according to nodepool, there are 6 nodes stuck deleting in linaro-london http://grafana.openstack.org/d/vcvSN5Siz/nodepool-linaro?orgId=1 | 09:44 |
| kevinz | frickler: yes, image deletion has a problem in Linaro-London(upload is fine). I'm working on it | 09:45 |
| frickler | kevinz: not images, nodes. might be nodepool having a different view than nova, though | 09:46 |
| kevinz | frickler: OK, let me check | 09:46 |
| *** ociuhandu has quit IRC | 09:47 | |
| frickler | kevinz: FYI, this is what nodepool thinks it has currently. I didn't find any errors for the delete attempts, though. http://paste.openstack.org/show/789356/ | 09:50 |
| *** roman_g has joined #openstack-infra | 09:50 | |
| frickler | seems peculiar that those 6 nodes all are debian-stretch-arm64, not buster | 09:51 |
| kevinz | frickler: thx, I've removed them now, os-jobs can create more instances. I will boot 1 instance for debian-stretch-arm64 to reproduce it | 09:52 |
| *** ociuhandu has joined #openstack-infra | 09:54 | |
| frickler | well, only one patch in the queue currently, will need to wait for kolla ppl to wake up and submit fresh patches I guess ;) | 09:54 |
| *** ociuhandu has quit IRC | 09:55 | |
| *** jpena is now known as jpena|brb | 09:58 | |
| roman_g | clarkb: thank you! | 10:06 |
| *** pkopec has joined #openstack-infra | 10:16 | |
| *** Tengu has quit IRC | 10:17 | |
| *** Tengu has joined #openstack-infra | 10:18 | |
| *** hashar has quit IRC | 10:20 | |
| *** ociuhandu has joined #openstack-infra | 10:27 | |
| *** Tengu has quit IRC | 10:31 | |
| *** ociuhandu has quit IRC | 10:32 | |
| *** Tengu has joined #openstack-infra | 10:40 | |
| *** jpena|brb is now known as jpena | 10:43 | |
| *** hashar has joined #openstack-infra | 10:44 | |
| amotoki | hi, build-openstack-releasenotes job in stable/rocky is broken due to python2 venv refers to requirements master. | 10:44 |
| amotoki | https://zuul.opendev.org/t/openstack/build/fac75293255f49aba167225e4b636cd5 | 10:44 |
| amotoki | in stable/rocky (and other old branches) a project template release-notes-jobs is used and it uses python2. | 10:46 |
| amotoki | ensure-sphinx role checks out the master branch of requirments repo. | 10:46 |
| amotoki | what is the good way to fix the failure? | 10:46 |
| openstackgerrit | Onong Tayeng proposed openstack/project-config master: Remove networking-vpp entry https://review.opendev.org/706790 | 10:46 |
| *** udesale has quit IRC | 10:48 | |
| amotoki | correction: build-reno-releasenotes job in zuul-jobs checks out the master branch of requirments repo using override-checkout. | 10:48 |
| *** yamamoto has joined #openstack-infra | 10:51 | |
| kevinz | frickler: sure, thx | 10:53 |
| *** yamamoto has quit IRC | 10:54 | |
| *** yamamoto has joined #openstack-infra | 10:55 | |
| *** ociuhandu has joined #openstack-infra | 10:59 | |
| *** yamamoto has quit IRC | 11:00 | |
| AJaeger_ | amotoki: build-openstack-releasenotes job should run on *master* completely | 11:02 |
| *** ociuhandu has quit IRC | 11:03 | |
| *** ociuhandu has joined #openstack-infra | 11:03 | |
| amotoki | AJaeger_: so, where do you think python interpreter should be configured? | 11:05 |
| amotoki | or do you suggest to use build-openstack-releasenotes-python3 in older stable brnaches likes stable/rocky? | 11:05 |
| amotoki | it happens due to the mismatch that build-openstack-releasenotes uses python2 but it refers to requirements mater which has dropped py2 support. | 11:07 |
| amotoki | as a result, an error that reno==3.0.0 is not found for py2 in build-openstack-releasenotes job. | 11:07 |
| amotoki | perhaps it affects all stable branches of all repos which run the releasenotes job in py2. | 11:08 |
| frickler | #status log restarted zuul-executor on ze12 after it had OOMed | 11:09 |
| openstackstatus | frickler: finished logging | 11:09 |
| *** dirk has joined #openstack-infra | 11:14 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Authorization rules: add templating https://review.opendev.org/705193 | 11:14 |
| *** Lucas_Gray has joined #openstack-infra | 11:17 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: WIP: Store unparsed branch config in Zookeeper https://review.opendev.org/705716 | 11:21 |
| *** yamamoto has joined #openstack-infra | 11:27 | |
| *** ociuhandu has quit IRC | 11:32 | |
| *** yamamoto has quit IRC | 11:32 | |
| *** ociuhandu has joined #openstack-infra | 11:32 | |
| *** ociuhandu has quit IRC | 11:38 | |
| *** ociuhandu has joined #openstack-infra | 11:41 | |
| *** hashar has quit IRC | 11:56 | |
| *** nicolasbock has joined #openstack-infra | 12:01 | |
| *** yamamoto has joined #openstack-infra | 12:07 | |
| *** rpittau is now known as rpittau|bbl | 12:08 | |
| *** ociuhandu has quit IRC | 12:10 | |
| *** ociuhandu has joined #openstack-infra | 12:11 | |
| *** yamamoto has quit IRC | 12:12 | |
| *** ociuhandu has quit IRC | 12:15 | |
| *** ykarel is now known as ykarel|afk | 12:24 | |
| *** udesale has joined #openstack-infra | 12:28 | |
| *** adriant has quit IRC | 12:34 | |
| *** adriant has joined #openstack-infra | 12:35 | |
| *** jpena is now known as jpena|lunch | 12:35 | |
| AJaeger_ | amotoki: let me check the change. | 12:35 |
| AJaeger_ | amotoki: oh, seems we didn't think that completly through initally - and should switch to build-openstack-releasenotes-python3 everywhere indeed. Or change the job to use python3 everywhere. | 12:39 |
| *** rosmaita has joined #openstack-infra | 12:41 | |
| amotoki | AJaeger_: Although I am not sure it can in zuul configuration, what we need is to check the master branch of the target repo only and we can use the corresponding branch of requirements repo. | 12:41 |
| AJaeger_ | amotoki: I don't have time to dig into this right now since I'm travelling, would be great if somebody could do that. Looking at https://opendev.org/zuul/zuul-jobs/src/branch/master/zuul.d/python-jobs.yaml#L326, it seems safe to switch the job template from python 2 to python 3 | 12:41 |
| AJaeger_ | amotoki: or your master branch of *both* | 12:41 |
| AJaeger_ | amotoki: so, I'm considering to change https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/project-templates.yaml#L233 to use python3 - so, basically do the same as : release-notes-jobs-python3 | 12:43 |
| amotoki | AJaeger_: yeah, I know. perhaps it should work. | 12:43 |
| amotoki | AJaeger_: okay, I will propose a fix and see what kind of responses we get. | 12:44 |
| AJaeger_ | amotoki: try locally to use the new template - and if that one works, propose to change the template and let's discuss in review. I'd like to have clarkb or fungi thinnk this through with us - my initial impression: This is the right way forward. | 12:44 |
| AJaeger_ | amotoki: Thanks! | 12:44 |
| amotoki | AJaeger_: if we need more than this, I would like to wait for someone to pick this up. | 12:44 |
| *** electrofelix has joined #openstack-infra | 12:44 | |
| AJaeger_ | yep | 12:45 |
| *** yamamoto has joined #openstack-infra | 12:45 | |
| *** artom has joined #openstack-infra | 12:45 | |
| *** ociuhandu has joined #openstack-infra | 12:48 | |
| *** yamamoto has quit IRC | 12:49 | |
| *** rlandy has joined #openstack-infra | 12:50 | |
| *** ociuhandu has quit IRC | 12:53 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Authorization rules: add templating https://review.opendev.org/705193 | 13:00 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Authorization rules: add templating https://review.opendev.org/705193 | 13:00 |
| *** rpittau|bbl is now known as rpittau | 13:12 | |
| *** vesper11 has quit IRC | 13:15 | |
| *** vesper has joined #openstack-infra | 13:15 | |
| *** rh-jelabarre has joined #openstack-infra | 13:17 | |
| *** artom has quit IRC | 13:20 | |
| *** aedc_ has joined #openstack-infra | 13:20 | |
| fungi | morning meeting is over, i'll try to catch up and see if i can think of anything there | 13:21 |
| *** aedc has quit IRC | 13:23 | |
| fungi | ianw: i've found the only reliable way to search in rackspace's dns management interface is to first scroll to the very bottom since it seems to do on-demand loading with some sort of pagination so you have to keep scrolling to the bottom in that subframe until it's all paged in | 13:23 |
| *** jaosorior has joined #openstack-infra | 13:25 | |
| *** gary_perkins has joined #openstack-infra | 13:26 | |
| *** ykarel|afk is now known as ykarel | 13:26 | |
| fungi | amotoki: AJaeger_: yeah, switching the template to use the python3 version of the job seems like the most reasonable next (and hopefully last) step | 13:27 |
| amotoki | fungi: thanks for checking. I am preparing a patch for openstack-zuul-jobs right now. | 13:27 |
| openstackgerrit | Akihiro Motoki proposed openstack/openstack-zuul-jobs master: Use python3 in release-notes-jobs https://review.opendev.org/706825 | 13:29 |
| *** aedc_ has quit IRC | 13:30 | |
| *** jpena|lunch is now known as jpena | 13:30 | |
| *** aedc_ has joined #openstack-infra | 13:33 | |
| *** Lucas_Gray has quit IRC | 13:34 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Offload repo reset to processes https://review.opendev.org/706827 | 13:34 |
| *** Lucas_Gray has joined #openstack-infra | 13:35 | |
| *** rfolco has joined #openstack-infra | 13:42 | |
| AJaeger_ | amotoki: can you test that this does the right thing to confirm the theory? | 13:45 |
| *** priteau has joined #openstack-infra | 13:50 | |
| *** sgw has quit IRC | 13:52 | |
| *** lpetrut has joined #openstack-infra | 13:55 | |
| *** yamamoto has joined #openstack-infra | 13:59 | |
| *** Lucas_Gray has quit IRC | 14:03 | |
| *** yamamoto has quit IRC | 14:04 | |
| *** Lucas_Gray has joined #openstack-infra | 14:08 | |
| *** zxiiro has joined #openstack-infra | 14:08 | |
| AJaeger_ | amotoki: I've convinced myself it's correct - still ,tell us after rechecking the other job, please ;) | 14:09 |
| amotoki | AJaeger_: I just proposed a patch to test it https://review.opendev.org/706500 | 14:10 |
| AJaeger_ | amotoki: is that really the correct URL? | 14:11 |
| frickler | AJaeger_: I'd say https://review.opendev.org/706838 is what amotoki meant | 14:14 |
| *** lbragstad has quit IRC | 14:14 | |
| AJaeger_ | frickler, thanks that looks like it | 14:15 |
| AJaeger_ | that one is passing - so all fine as expected! Yeah | 14:16 |
| *** Lucas_Gray has quit IRC | 14:17 | |
| fungi | i think we meant to make that switch and then forgot about it | 14:19 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Use python3 in release-notes-jobs https://review.opendev.org/706825 | 14:19 |
| mordred | morning all! anything fun and exciting to look at this morning? | 14:21 |
| mordred | fungi: how's coastal living? | 14:21 |
| fungi | mordred: we're above water for the moment, situation normal | 14:22 |
| mordred | \o/ | 14:22 |
| fungi | i let #openstack-release know about the release notes fix merging | 14:22 |
| *** lbragstad has joined #openstack-infra | 14:23 | |
| AJaeger_ | thanks | 14:23 |
| *** aedc_ has quit IRC | 14:24 | |
| *** yamamoto has joined #openstack-infra | 14:32 | |
| *** ociuhandu has joined #openstack-infra | 14:34 | |
| *** Goneri has joined #openstack-infra | 14:35 | |
| *** yamamoto has quit IRC | 14:37 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Implement basic github checks API workflow https://review.opendev.org/705168 | 14:39 |
| *** icey_ is now known as icey | 14:43 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Implement basic github checks API workflow https://review.opendev.org/705168 | 14:44 |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Implement basic github checks API workflow https://review.opendev.org/705168 | 14:47 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Cap virtualenv to <20.0.0 https://review.opendev.org/706860 | 14:47 |
| *** yamamoto has joined #openstack-infra | 14:48 | |
| *** xek has joined #openstack-infra | 14:50 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Implement basic github checks API workflow https://review.opendev.org/705168 | 14:52 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Uncap virtualenv https://review.opendev.org/706871 | 15:02 |
| *** eharney has joined #openstack-infra | 15:04 | |
| *** KeithMnemonic has joined #openstack-infra | 15:06 | |
| *** lpetrut has quit IRC | 15:08 | |
| *** ykarel is now known as ykarel|afk | 15:09 | |
| roman_g | Hello team. Error "NODE_FAILURE in 0s" - what could that be and where to look for logs? Link to job information shows nothing interesting (http 404): https://review.opendev.org/#/c/706830/ | 15:16 |
| fungi | roman_g: that normally only happens when nodepool is unable to satisfy a node request anywhere in any of its configured providers | 15:21 |
| fungi | looks like that wants to get 4 nodes with the ubuntu-bionic-32G label | 15:23 |
| fungi | i'll check how much quota we have for thise | 15:23 |
| roman_g | fungi: no, 1x ubuntu-bionic-32G and 3 ubuntu-bionic-expanded https://review.opendev.org/#/c/706821/2/zuul.d/nodesets.yaml | 15:24 |
| fungi | ahh | 15:25 |
| fungi | yep, i misread | 15:25 |
| fungi | let me make sure there's a single provider which can make all of those | 15:25 |
| roman_g | name: airship-kna1; region-name: 'Kna1'; cloud: airship-citycloud | 15:27 |
| fungi | yeah, looks like this pool offers both: https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl02.openstack.org.yaml#L438-L452 | 15:27 |
| fungi | so now to check logs on nl02 and see if we're getting quota errors or anything | 15:28 |
| roman_g | please :) | 15:28 |
| fungi | i've found where it logged declining the node request... | 15:38 |
| fungi | 2020-02-10 15:08:57,908 DEBUG nodepool.driver.NodeRequestHandler[nl02-11076-PoolWorker.airship-kna1-main]: Declining node request 300-0007329039 because node type(s) [ubuntu-bionic-32G,ubuntu-bionic-expanded,ubuntu-bionic-expanded,ubuntu-bionic-expanded] not available | 15:38 |
| fungi | so now i just need to figure out why it's decided those aren't available | 15:39 |
| fungi | oh, wait, i should be looking at PoolWorker.airship-kna1-airship not PoolWorker.airship-kna1-main | 15:42 |
| *** ykarel|afk is now known as ykarel|away | 15:42 | |
| *** armax has joined #openstack-infra | 15:44 | |
| fungi | still basically the same level of detail though... | 15:47 |
| fungi | 2020-02-10 15:03:06,533 DEBUG nodepool.driver.NodeRequestHandler[nl02-11076-PoolWorker.airship-kna1-airship]: Declining node request 300-0007329039 because node type(s) [ubuntu-bionic-32G] not available | 15:47 |
| corvus | fungi, roman_g: nodepool config has a "B" on the end of the label. like 32GB not 32G | 15:48 |
| corvus | here's the list of labels: http://zuul.openstack.org/labels | 15:49 |
| fungi | oh, yep! i see https://review.opendev.org/#/c/706821/2/zuul.d/nodesets.yaml is inconsistent | 15:50 |
| *** jamesmcarthur has joined #openstack-infra | 15:50 | |
| fungi | it says 32G in one place and 32GB in another | 15:50 |
| fungi | so that seems to be the reason. nodepool is rejecting the request for a ubuntu-bionic-32G node because it doesn't know you meant a ubuntu-bionic-32GB node | 15:51 |
| corvus | silly nodepool | 15:51 |
| clarkb | I don't need to change anything in the nodepool config then right? just the job consumption | 15:52 |
| fungi | looks that way | 15:53 |
| fungi | i think it's just 706821 needs updating | 15:53 |
| *** carli has quit IRC | 15:56 | |
| openstackgerrit | Merged zuul/zuul master: Cap virtualenv to <20.0.0 https://review.opendev.org/706860 | 15:57 |
| dulek | Is it possible to debug "Unknown configuration error" errors from Zuul? | 16:00 |
| clarkb | dulek: can you share a link? | 16:00 |
| dulek | clarkb: https://review.opendev.org/#/c/682531 | 16:00 |
| clarkb | my hunch is invalid yaml if it can't say more about where or how it failed | 16:01 |
| dulek | clarkb: Alright! | 16:01 |
| clarkb | <<: *base_devstack_localrc | 16:02 |
| clarkb | that line catches the eye | 16:02 |
| dulek | clarkb: Well, the validator is fine with this. | 16:02 |
| clarkb | I'm not sure if < is valid key name start | 16:02 |
| dulek | clarkb: It's special key meaning that it should unpack the anchor there while allowing me to inject another key. | 16:02 |
| corvus | voluptuous.error.MultipleInvalid: expected dict for dictionary value @ data['host-vars']['controller'] | 16:03 |
| corvus | it looks like that's the error, but it got caught up in a second error in the exception handle which masked it | 16:03 |
| corvus | i'll see if i can find/fix the second bug | 16:03 |
| corvus | i still don't actually see what's wrong with that, sorry | 16:05 |
| dulek | That looks fairly okay to me, let's see what happens if I start resolving those anchors. | 16:05 |
| dulek | corvus: Had it said which file it has the problem with? | 16:05 |
| fungi | yeah, it does at least look like data['host-vars']['controller'] would be a dict | 16:05 |
| corvus | normally it would, but that didn't make it into the fallback exception handler | 16:06 |
| fungi | both files seem consistent in the structure of the anchor contents there | 16:06 |
| dulek | I'll try a few other ways, let's see. | 16:07 |
| *** ociuhandu has quit IRC | 16:07 | |
| corvus | fwiw, i'm not knocking clarkb's suspcions of the << ... we haven't seen that a lot, and it could still somehow be the underlying cause | 16:07 |
| *** ociuhandu has joined #openstack-infra | 16:08 | |
| clarkb | that was my uneducated guess, juggling yaml parsing with a meeting :) | 16:08 |
| dulek | corvus, fungi: Now I replaced all the references with "<<" syntax. Looks like it worked, there's a different error. | 16:08 |
| dulek | BTW note that initial version of the patch was failing too and it had no "<<" as keys. | 16:08 |
| *** yamamoto has quit IRC | 16:12 | |
| *** ociuhandu has quit IRC | 16:12 | |
| *** chkumar|rover is now known as raukadah | 16:15 | |
| *** ociuhandu has joined #openstack-infra | 16:17 | |
| *** gyee has joined #openstack-infra | 16:18 | |
| *** eernst has joined #openstack-infra | 16:18 | |
| *** ociuhandu has quit IRC | 16:21 | |
| *** ociuhandu has joined #openstack-infra | 16:22 | |
| *** udesale has quit IRC | 16:24 | |
| *** rh-jelabarre has quit IRC | 16:26 | |
| *** rh-jelabarre has joined #openstack-infra | 16:32 | |
| mordred | clarkb: is https://review.opendev.org/#/c/697589/ still a thing we want? | 16:34 |
| *** mattw4 has joined #openstack-infra | 16:34 | |
| clarkb | mordred: I think we do already set that on the puppet managed stuff so this brings ansible managed in line | 16:34 |
| mordred | kk | 16:35 |
| clarkb | so yes, but probably not urgent? | 16:35 |
| mordred | clarkb: I was looking at open system-config changes and noticed it | 16:35 |
| mordred | clarkb: if you get bored today ... https://review.opendev.org/#/c/705878 and parents | 16:35 |
| clarkb | I'd be happy to have it off my in flight list :) | 16:35 |
| *** jamesmcarthur has quit IRC | 16:40 | |
| *** lmiccini has quit IRC | 16:41 | |
| *** jamesmcarthur has joined #openstack-infra | 16:41 | |
| roman_g | fungi: corvus: thank you! | 16:42 |
| fungi | roman_g: you're welcome. corvus has far better eyes than i do apparently | 16:43 |
| clarkb | my eyes are terrible. The other day I typed a = and thought I had a - and was seeing double | 16:44 |
| clarkb | (I'm ok I see the optometrist regularly for my visual issues) | 16:44 |
| mordred | we can't all have corvus eyes | 16:44 |
| *** tosky has quit IRC | 16:48 | |
| *** psachin has quit IRC | 16:49 | |
| *** yamamoto has joined #openstack-infra | 16:52 | |
| *** priteau has quit IRC | 16:56 | |
| *** jamesmcarthur has quit IRC | 16:57 | |
| *** jamesmcarthur has joined #openstack-infra | 16:57 | |
| *** yamamoto has quit IRC | 16:58 | |
| *** ociuhandu has quit IRC | 17:00 | |
| *** lucasagomes has quit IRC | 17:01 | |
| *** pkopec has quit IRC | 17:03 | |
| *** pkopec has joined #openstack-infra | 17:06 | |
| *** rpittau is now known as rpittau|afk | 17:07 | |
| *** ociuhandu has joined #openstack-infra | 17:08 | |
| openstackgerrit | Merged opendev/system-config master: Add docs for deleting an AFS volume https://review.opendev.org/703470 | 17:09 |
| openstackgerrit | Merged opendev/system-config master: Add irc bots for Multi-Arch SIG https://review.opendev.org/703325 | 17:09 |
| clarkb | I'm going to grab breakfast, but then I'll look into why this mirrors LE cert isn't being refreshed | 17:22 |
| clarkb | (if someone else is already looking at that let me know) | 17:22 |
| fungi | clarkb: the rackspace one we got alerted to? i haven't had time to look yet either, but happy we're now checking those | 17:25 |
| clarkb | ya | 17:27 |
| clarkb | the LE files were updated february 4 which would've been 30 days pre expiration as expected | 17:28 |
| *** pgaxatte has quit IRC | 17:28 | |
| clarkb | my browser sees an old cert | 17:29 |
| clarkb | fungi: is it possible that reloading apache isn't actually sufficient? | 17:29 |
| clarkb | I'm going to try manually running systemctl reload apache2 then will check the cert | 17:30 |
| clarkb | ok that seems to have fixed it | 17:31 |
| *** ociuhandu has quit IRC | 17:31 | |
| clarkb | maybe the reload handler didn't fire for some reason? | 17:32 |
| clarkb | (note I had to use a new browser process so that my old tcp connection wasn't reused) | 17:32 |
| *** evrardjp has quit IRC | 17:34 | |
| *** evrardjp has joined #openstack-infra | 17:34 | |
| *** dtantsur is now known as dtantsur|afk | 17:34 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:35 | |
| *** ociuhandu has joined #openstack-infra | 17:36 | |
| clarkb | I want to say we've seen flaky handler firing before | 17:38 |
| clarkb | I guess we keep monitoring with certcheck and its a simple fix for the few that have this happen (and if it becomes persistent maybe we use another method) | 17:38 |
| *** ociuhandu has quit IRC | 17:41 | |
| fungi | yeah, hard to say | 17:41 |
| fungi | also possible the reload handler doesn't actually do what we think, but i did dissect the service reload implementation in ansible and it seems to call what we want on the platforms we care about | 17:42 |
| clarkb | ya I'm pretty sure that it ends up doing the equivalent of systemctl reload apach2 on systemd platforms | 17:43 |
| fungi | and the initscript on sysv platforms maps reload to the graceful restart we want | 17:44 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Allow more k8s actions in untrusted context https://review.opendev.org/706940 | 17:45 |
| *** electrofelix has quit IRC | 17:49 | |
| *** tesseract has quit IRC | 17:52 | |
| gouthamr | hi, seeing sporadic POST_FAILUREs on jobs gating openstack/manila - https://zuul.opendev.org/t/openstack/build/630680abc25d462ca216da4367719af5 (rsync error) | 17:54 |
| gouthamr | wondering if you're aware, and i shouldn't recheck for a bit? :) | 17:54 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Allow more k8s actions in untrusted context https://review.opendev.org/706940 | 17:55 |
| clarkb | that looks like the rax leaked IPs problem | 17:55 |
| *** gfidente is now known as gfidente|afk | 17:55 | |
| clarkb | it would be cool if ansible rsync could use the control persist process in ansible | 17:56 |
| *** openstackstatus has quit IRC | 17:57 | |
| *** openstack has joined #openstack-infra | 17:58 | |
| *** ChanServ sets mode: +o openstack | 17:58 | |
| clarkb | gouthamr: I think what has happened there is rackspace has assigned an IP address twice and depending on which host's arp is current winning you either get the correct host or the wrong host | 17:58 |
| clarkb | gouthamr: there tends not to be a quick fix for that (we digthrough logstash and colelct a set of IPs for them to check every so often) | 17:59 |
| *** Darcidride has joined #openstack-infra | 17:59 | |
| *** derekh has quit IRC | 18:00 | |
| gouthamr | clarkb: oh... makes sense | 18:00 |
| gouthamr | clarkb: thank you for the explanation! | 18:01 |
| clarkb | speaking of logstash it seems much happierwith the lowered retention | 18:01 |
| clarkb | e-r is mostly up to date now | 18:01 |
| *** eernst has quit IRC | 18:06 | |
| mordred | woot | 18:06 |
| *** jpena is now known as jpena|off | 18:09 | |
| *** jaosorior has quit IRC | 18:16 | |
| fungi | gouthamr: yeah, we see it with varying frequency in different providers, but apparently nova can be a little "leaky" and for whatever reason lose track of instances, but think the ip address is freed up and then reassign it | 18:16 |
| fungi | it would be cool if there was a way for openstack to test for a response on a presumed-available ip address before assigning it. i've seen dhcp servers which did that in the past (ping the ip address you're about to put in a lease and if there's a response from other hardware addresses than the requester, skip it) | 18:18 |
| fungi | actually i think the implementation i'm remembering uses arp who-has rather than icmp echo-request | 18:19 |
| fungi | ipv4 link-local assignment is also supposed to work that way, if i remember correctly | 18:20 |
| fungi | granted, the probes would need to come from something within the same broadcast domain, which the relevant systems may not have an interface in | 18:21 |
| JayF | fungi: are you familiar with `arping`? | 18:26 |
| JayF | it's a tool to explicitly look for ip address conflicts that might fit well for that desire | 18:27 |
| fungi | JayF: yep, have used it often in years past | 18:30 |
| fungi | i suspect the trick is having an interface to emit from which is under (either direct or indirect) control of the service performing address assignment | 18:31 |
| fungi | openstack is probably too decoupled to make that easy | 18:31 |
| *** sgw has joined #openstack-infra | 18:34 | |
| *** ramishra has quit IRC | 18:37 | |
| *** yamamoto has joined #openstack-infra | 18:41 | |
| *** yamamoto has quit IRC | 18:46 | |
| *** hwoarang has quit IRC | 18:49 | |
| *** hwoarang has joined #openstack-infra | 18:52 | |
| clarkb | I think in the neutron case you'd use netns exec to run the commands on the correct interfaces | 19:00 |
| clarkb | its probably not too difficult if you can sort out where to run the command | 19:01 |
| *** eharney has quit IRC | 19:07 | |
| clarkb | the ubuntu-xenial arm64 image has successfully been uploaded to linaro-us | 19:09 |
| *** eharney has joined #openstack-infra | 19:09 | |
| *** ralonsoh has quit IRC | 19:10 | |
| openstackgerrit | Merged zuul/zuul master: Allow more k8s actions in untrusted context https://review.opendev.org/706940 | 19:10 |
| clarkb | oh we've got the other images uploaded there now too, but we hide them in all of the older failred uploaded | 19:11 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Add new airship citycloud region to grafana https://review.opendev.org/706958 | 19:14 |
| *** artom has joined #openstack-infra | 19:17 | |
| *** yamamoto has joined #openstack-infra | 19:18 | |
| *** yamamoto has quit IRC | 19:24 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Allow template lookup in untrusted context https://review.opendev.org/706963 | 19:27 |
| *** jamesmcarthur has quit IRC | 19:34 | |
| *** igordc has joined #openstack-infra | 19:35 | |
| *** stevebaker has joined #openstack-infra | 19:39 | |
| *** jamesmcarthur has joined #openstack-infra | 19:43 | |
| *** jamesmcarthur has quit IRC | 19:52 | |
| *** igordc has quit IRC | 19:53 | |
| *** jamesmcarthur has joined #openstack-infra | 19:57 | |
| *** yamamoto has joined #openstack-infra | 20:10 | |
| *** Lucas_Gray has joined #openstack-infra | 20:10 | |
| *** dklyle has quit IRC | 20:19 | |
| *** david-lyle has joined #openstack-infra | 20:19 | |
| *** eharney has quit IRC | 20:22 | |
| *** pkopec has quit IRC | 20:36 | |
| *** Lucas_Gray has quit IRC | 20:36 | |
| *** yamamoto_ has joined #openstack-infra | 20:39 | |
| *** yamamoto has quit IRC | 20:42 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: llow template lookup in untrusted context https://review.opendev.org/706963 | 20:44 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Allow template lookup in untrusted context https://review.opendev.org/706963 | 20:45 |
| *** yamamoto_ has quit IRC | 20:52 | |
| *** jamesmcarthur has quit IRC | 20:53 | |
| *** yamamoto has joined #openstack-infra | 20:54 | |
| ianw | infra-root: the static to afs publishing is all ready to go, but before we'd start, i'd like to do some better tracking of our afs release process per https://review.opendev.org/#/c/706176/ ... reviews appreciated | 20:56 |
| ianw | i think it's better to start tracking before we have an issue | 20:58 |
| mordred | ++ tracking good | 20:59 |
| corvus | ianw: did we reverse the decision to avoid virtualenvs on servers? | 21:00 |
| *** jamesmcarthur has joined #openstack-infra | 21:00 | |
| mordred | ianw: the script in https://review.opendev.org/#/c/706176/6/playbooks/roles/afs-release/files/release-volumes.py is mostly a copy of the existing one with some statsd added, yeah? - also corvus' question | 21:01 |
| ianw | mordred: yes, a few other things but mostly that, mentioned in the changelog | 21:02 |
| ianw | corvus: i wasn't aware we had a blanket ban on virtualenv's sorry. this could use packages | 21:03 |
| fungi | ianw: on a related note, i'd appreciate a look through topic:airship-docs | 21:03 |
| ianw | fungi: will do | 21:04 |
| fungi | trying to make sure the new docs.airshipit.org site follows the new process | 21:04 |
| corvus | ianw: it's mostly a group cognizance issue -- if we aren't expecting one, it's hard for us to know that's how a program is to be run. i think either system-installed packages or containers (which i think is the group-favored way to move forward, but we don't have a container "cron-job" example yet) are the things we're collectively expecting. i'll leave a comment. | 21:05 |
| clarkb | in general system level installs are less and less useful due to changes in pip | 21:06 |
| clarkb | the pyyaml issue on nb03 is an example of the problems we have | 21:06 |
| clarkb | maybe change the shebang line to make it clear that it is expected to run out of the venv? | 21:06 |
| corvus | or maybe use containers? | 21:06 |
| corvus | i mean my view is that we have 2 ways the group expects python programs to be run: system installed or containers. | 21:07 |
| corvus | and we want to move everything to containers | 21:07 |
| corvus | so maybe use one of those before going a third route? especially since we've rejected it in the past? | 21:07 |
| clarkb | I guess I see the container route as a pretty big investment particularly for a simple script. It would work though | 21:08 |
| corvus | yes. also, the system-installed packages will work just fine for this. | 21:08 |
| ianw | i hear the feedback and i think that for this case, starting with system installed is probably the right place, since the dependencies will be available | 21:08 |
| clarkb | (note the containers also install packages at system level in their fs, but avoid the problems we have on non containers because they are minimal enough to not have conflicting python packages) | 21:08 |
| corvus | so my personal recommendation is just use system packages for this, and defer containerization | 21:09 |
| ianw | i'm not opposed to the container approach, but i wouldn't like to make the static publishing work become dependent on our first use of a script container cron job | 21:09 |
| corvus | ya | 21:09 |
| ianw | ok, it sounds like we're in agreement | 21:09 |
| fungi | system packages for this sound good to me too | 21:10 |
| mordred | I also agree - and I'm also not 100% sure we've proved out doing AFS operations from inside of a container yet either | 21:10 |
| clarkb | mordred: in this case it doesn't do afs directly. It sshes to a place it can do -localauth to avoid auth timeouts | 21:11 |
| clarkb | but ya system packages should work fine | 21:11 |
| mordred | ah - yes - good point | 21:11 |
| corvus | well, it runs 'vos examine' | 21:11 |
| corvus | i don't know if that requires talking to the local cache manager or not | 21:11 |
| clarkb | oh right it checks if it needs to release | 21:11 |
| mordred | one gotcha for us all - we should still be careful to not mix system packages and pip installed packages. that should be fine here - and I think the "use containers or system packages" above takes care of it overall | 21:11 |
| mordred | but as we roll things out - we should take care that we don't do both things | 21:12 |
| corvus | mordred: ++ | 21:12 |
| corvus | out of curiosity, what does a containerized cronjob look like? a crontab entry that does "podman run --rm opendevorg/release-volumes" ? | 21:13 |
| mordred | corvus: I think I'd vote for that, yes | 21:13 |
| mordred | (as how a containerized cronjob should look) | 21:14 |
| *** jamesmcarthur_ has joined #openstack-infra | 21:14 | |
| corvus | yeah, just hypothesizing since i haven't thought much about it. doesn't seem terrible. | 21:14 |
| corvus | after this is all working, that sounds like it might be fun to migrate. | 21:14 |
| mordred | corvus: I thnik the ugly part would be figure out mounts | 21:14 |
| corvus | yeah, a bunch of -v args i'm guessing | 21:15 |
| mordred | corvus: which might want to make it be a something something podman-compose something | 21:15 |
| clarkb | and managing a bunch of dockerfiles for these things | 21:15 |
| mordred | clarkb: the dockerfiles part I think we've got - we've got that pattern in system-config and it's pretty easy to deal wit | 21:15 |
| *** jtomasek_ has quit IRC | 21:15 | |
| mordred | but I could imagine a cron invocation with a billion --volume args might get taxing | 21:16 |
| corvus | or maybe it's just one -v for /afs | 21:16 |
| mordred | maybe so! | 21:16 |
| mordred | these are all questions to which I do not have the answers | 21:17 |
| *** jamesmcarthur has quit IRC | 21:17 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Allow for periodic afs releases from mirror-update https://review.opendev.org/706176 | 21:18 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Migrate AFS publishing to mirror-update.opendev.org https://review.opendev.org/706731 | 21:18 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add project.tarballs to periodic AFS release https://review.opendev.org/706732 | 21:18 |
| ianw | corvus: it would be an odd container though, as it would depend on a working mounted /afs on the host system, it seems like an abstraction violation | 21:19 |
| ianw | but i'm not sure about openafs clients in containers | 21:20 |
| TheJulia | is http://zuul.opendev.org/t/openstack/status working for folks? | 21:20 |
| mordred | ianw: it's a concept I think we're gonna want to tee up some experimentation around - if for no other reason so that we know how it does and doesn't work | 21:20 |
| corvus | ianw: i'm not seeking ideological purity with containers :) | 21:20 |
| mordred | ianw: I'm not super concerned about abstraction violation myself ... yeah, that ^^ | 21:20 |
| Shrews | TheJulia: yes, and hi! :) | 21:21 |
| TheJulia | Shrews: o/ | 21:21 |
| mordred | TheJulia: also works for me | 21:21 |
| TheJulia | Okay, maybe my web browser has decided it needs a vacation... or a restart | 21:21 |
| mordred | in case it matters - I'm connecting over ipv6 | 21:21 |
| fungi | i too am fetching it over ipv6, but it's loading fine for me | 21:22 |
| mordred | TheJulia: my browser has been misbehaving a lot recently | 21:22 |
| clarkb | it works for me and I'm hitting it via ipv4 | 21:23 |
| TheJulia | yeah, looks like I can traceroute to it over v6 and that is what the sockets show as trying to use for my browser | 21:23 |
| *** rfolco has quit IRC | 21:23 | |
| * TheJulia goes and wraps up a rebase before giving the browser a nice fresh restart | 21:23 | |
| mordred | TheJulia: maybe it needs a fluffy bunny rabbit | 21:25 |
| *** Darcidride has quit IRC | 21:28 | |
| TheJulia | mordred: so it wants me to put my desktop in the back yard for when the rabbits appear at night? | 21:29 |
| TheJulia | I guess... this machine could use an upgrade... | 21:30 |
| mordred | TheJulia: yes. that is my very unscientific guess as to what's wrong | 21:33 |
| TheJulia | Very well then | 21:36 |
| *** xek has quit IRC | 21:36 | |
| *** rfolco has joined #openstack-infra | 21:37 | |
| *** yamamoto has quit IRC | 21:39 | |
| fungi | rabbit reboot ribbit | 21:42 |
| *** jamesmcarthur_ has quit IRC | 21:45 | |
| *** Goneri has quit IRC | 21:46 | |
| openstackgerrit | Merged zuul/zuul master: Allow template lookup in untrusted context https://review.opendev.org/706963 | 21:51 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Allow for periodic afs releases from mirror-update https://review.opendev.org/706176 | 21:52 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Migrate AFS publishing to mirror-update.opendev.org https://review.opendev.org/706731 | 21:52 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add project.tarballs to periodic AFS release https://review.opendev.org/706732 | 21:52 |
| *** diablo_rojo has quit IRC | 22:02 | |
| *** diablo_rojo has joined #openstack-infra | 22:02 | |
| sean-k-mooney | ianw: dumb question but does devstack work on rhel7 specificly rhel7.7 not centos | 22:07 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Uncap virtualenv https://review.opendev.org/706871 | 22:07 |
| *** yamamoto has joined #openstack-infra | 22:10 | |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Add gcp-authdaemon to Zuul https://review.opendev.org/706989 | 22:11 |
| ianw | sean-k-mooney: not the master branch, as that's moved on to python3, but the older branches *should* | 22:13 |
| sean-k-mooney | so i am trying to figure out how to test nova and cyborg patches on a host im being lent with an fpga | 22:13 |
| sean-k-mooney | it would have to be python 3 since the patche are still under review | 22:14 |
| *** dpawlik has quit IRC | 22:14 | |
| sean-k-mooney | so im trying to see if i need to just bite the bullet and install ubuntu or if can leave the os untouched | 22:15 |
| fungi | ianw: i think rhel 7.7 includes python3 directly (not via software collections) now, doesn't it? | 22:15 |
| clarkb | fungi: it does | 22:15 |
| mordred | mnaser: does openstack-ansible install on rhel ? ^^ | 22:15 |
| sean-k-mooney | it has 3.6 installed ya | 22:15 |
| clarkb | I'm not sure that that has been tested at all though | 22:15 |
| *** yamamoto has quit IRC | 22:15 | |
| clarkb | (with devstack I mean) | 22:15 |
| mordred | yeah- devstack + rhel doesn't seem to be high on anyone's list to work on | 22:15 |
| fungi | oh, i expect not unless someone did a manual poc | 22:15 |
| sean-k-mooney | well i have done python3 install on centos | 22:15 |
| sean-k-mooney | but rhel... | 22:15 |
| mordred | sean-k-mooney: funny story - at the grizzly summit we spoke with RH and Rackspace - and between the two of them, both interested in doing so, nobody could figure out how to donate rhel quota to infra | 22:16 |
| mordred | yay cloud | 22:16 |
| sean-k-mooney | as in licences for rhel | 22:17 |
| ianw | the problem will be that devstack on centos has a heavy dependency on RDO packages to actually do anything. so while you can probably make it work, if anything goes wrong, the people who care are going to be very limited | 22:17 |
| mordred | yup. figuring out how to manage that was apparently intractible | 22:17 |
| *** dmsimard is now known as dmsimard|off | 22:17 | |
| sean-k-mooney | ya also the subscription manager hates men and makes me sad anytime i touch it | 22:17 |
| sean-k-mooney | *hate me | 22:17 |
| mordred | sean-k-mooney: it's a subscription manager whose job is to prevent you from doing things - of course it does | 22:18 |
| fungi | it made me sad back in the old rhn_register days too | 22:18 |
| fungi | i'm just glad i haven't had to touch it for going on a decade | 22:18 |
| sean-k-mooney | its why i do all my dev on ubuntu or centos | 22:18 |
| mordred | sean-k-mooney: s/I/people/ | 22:19 |
| ianw | mordred: i've never pursued it because i imagine the infra community wouldn't be thrilled about managing something like software-key donation | 22:20 |
| sean-k-mooney | hehe yes but when i joined redhat i used to try and do all my dev on centos to at least be clsoe. untel python3 started ot be funky on it so i move to ubunutu | 22:20 |
| ianw | afaik nobody has pursued it as a 3rd-party-ci situation, where it probably would be more appropriate | 22:20 |
| sean-k-mooney | ianw: the funny thing is that you can get developer licences now | 22:20 |
| mordred | ianw: yah - like, solid devstack on centos jobs would be a great first step - but nobody really ever got that solidly off the ground - and also the RDO thing | 22:21 |
| sean-k-mooney | still too much effort however | 22:21 |
| fungi | they last time i looked at the developer licenses program, it didn't cover cloud images | 22:21 |
| fungi | hopefully they've since extended it | 22:21 |
| clarkb | sean-k-mooney: the dev licenses explicitly forbid our use case though | 22:21 |
| sean-k-mooney | clarkb: of couse it does | 22:21 |
| clarkb | fungi: they are for personal use only too | 22:21 |
| sean-k-mooney | ah right | 22:22 |
| * mordred is happy we don't use software that requires license keys | 22:22 | |
| fungi | the tripleo team has been using them in some places, and arguing that the stated use case (to assist with open source development) broadly allows their use in a ci system | 22:22 |
| sean-k-mooney | fungi: that is in the RDO ci however irght | 22:23 |
| fungi | i brought it up as a question on one of the reviews in gerrit | 22:23 |
| * mordred AFKs for a minute | 22:23 | |
| sean-k-mooney | not first party and i think they are moving back to centos8 | 22:23 |
| clarkb | fungi: I read the terms recently and you get one install for one machine | 22:23 |
| sean-k-mooney | when ever they manage to make building modules with python 3 work | 22:23 |
| clarkb | mordred: when you get back are we able to do gitea upgrades? | 22:24 |
| fungi | aha, it was on https://review.opendev.org/669223 but that change was later reworked to use centos instead anyway | 22:25 |
| sean-k-mooney | ianw: anyway thanks. ill think about other ways to achive my aim. | 22:25 |
| fungi | https://review.opendev.org/#/c/669223/8/molecule/install-deps/molecule.yml | 22:25 |
| fungi | the argument was that by being a docker image and not actually "booting" the license restrictions did not apply | 22:26 |
| sean-k-mooney | im not sure that is actully true but also dont really want to find out | 22:27 |
| fungi | sure, just glad the change ultimately switched to a free distro | 22:27 |
| sean-k-mooney | yep | 22:27 |
| clarkb | I'm having a hard time finding the usage terms for the dev license now though | 22:28 |
| ianw | fungi: looking at the topic:airship-docs it looks like the testing of the vhost is blocked by the other change with ajaeger's comments? | 22:28 |
| *** dchen has joined #openstack-infra | 22:28 | |
| *** slaweq has quit IRC | 22:29 | |
| fungi | ianw: yep, just want to be sure that we recommend using the tox docs job from openstack/openstack-zuul-jobs instead of the one from opendev/base-jobs | 22:30 |
| fungi | will both of them work interchangeably as far as the promote job is concerned? | 22:30 |
| ianw | fungi: ahh, i dunno ... https://review.opendev.org/#/c/706598/3/zuul.d/jobs.yaml i wonder if we should mount that as "docs.airshipit.org" ? | 22:32 |
| fungi | i see now that they won't, as the parent of the promote job refers specifically to openstack-tox-docs for the artifact rather than using a generic provides | 22:32 |
| fungi | ianw: then if they add another site later we're stuck putting it in a different volume or in a confusingly-named volume | 22:33 |
| fungi | the project/ tree was originally names of project communities' base domains | 22:33 |
| ianw | yeah, alright, and we have precidence with opendev.org/docs | 22:33 |
| fungi | we can always mount sub-volumes under that too if we want | 22:34 |
| fungi | but alternatively, this gives them a nice native way to have https://airshipit.org/docs/ later if they start also publishing their project site in afs | 22:34 |
| sean-k-mooney | actully that reminds me. i had a conversation today about way to reuse upstream zuul jobs internally. | 22:35 |
| fungi | sean-k-mooney: that's an intentional aspect of zuul v3's design, though it can require extra care depending on the job and how you intend to consume it | 22:36 |
| sean-k-mooney | do people know of a way to disable tox creating a venv and just running the test with site packages | 22:36 |
| ianw | fungi: you've got me thinking now that the top-level host-names in project/ are a mis-step | 22:36 |
| fungi | ianw: we can always move them around later if we want, i didn't want to be too picky about your plan | 22:37 |
| sean-k-mooney | fungi: basically we have a downstream hack that just uses send to remove the install command a deps lines which is ... icky but i was wondering if there was a way we could support it upstream | 22:37 |
| sean-k-mooney | *sed | 22:37 |
| clarkb | I think you can change the toxconfig to use system packages | 22:38 |
| clarkb | but it will still install the current package | 22:38 |
| sean-k-mooney | well we want to install the project under test. just not its deps | 22:38 |
| clarkb | ya I think if you enable system site pcakages then you'll get that behavior | 22:39 |
| fungi | we used to do that for many of our projects | 22:39 |
| fungi | for example so nova could get system-packaged python-libvirt | 22:40 |
| sean-k-mooney | ya i think we could basically install the rpms and pip install the project under test then add skip_install=True stie_pacakages=true | 22:40 |
| fungi | back in the days when a number of our python deps were not yet installable with pip | 22:40 |
| clarkb | I think you can't set skip_install=True | 22:40 |
| ianw | fungi: the extant openstack ones could be moved transparently? make a /afs/.openstack.org/project/openstack.org directory, and then mount the volumes under that (they remain mounted where they are too), update apache configs, then remove the old mounts? | 22:40 |
| clarkb | but maybe that works with site packages (not sure) | 22:40 |
| fungi | ianw: yeah, probably, if folks think it's important to do so for consistency | 22:41 |
| sean-k-mooney | clarkb: well i was thinking of use a task in the pre playbook to install it first | 22:41 |
| openstackgerrit | Merged openstack/project-config master: Add gcp-authdaemon to Zuul https://review.opendev.org/706989 | 22:41 |
| ianw | fungi: yeah, i had just never considered /afs/.openstack.org/project/openstack.org before about 5 minutes ago :) i wish i did :) | 22:41 |
| *** slaweq has joined #openstack-infra | 22:41 | |
| sean-k-mooney | but ya i was just wondering if there was a way to remove the downstream hacking of the tox.ini | 22:41 |
| sean-k-mooney | sould like no | 22:41 |
| sean-k-mooney | *sounds like no | 22:43 |
| fungi | ianw: i've updated https://review.opendev.org/706599 now per AJaeger_'s suggestion | 22:44 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Install kubectl/oc into executor container image https://review.opendev.org/706995 | 22:44 |
| fungi | in the future we probably could use a more generic, less openstacky version of this publication pipeline | 22:44 |
| ianw | fungi: i think the docs/ directory needs to be made | 22:45 |
| fungi | oh, the job won't make it on its own? | 22:45 |
| ianw | i do not think so | 22:46 |
| *** openstackgerrit has quit IRC | 22:46 | |
| ianw | but we can try it and see | 22:46 |
| fungi | i see | 22:46 |
| *** slaweq has quit IRC | 22:46 | |
| fungi | i suppose it'll come down to how and where tar -x is being called | 22:46 |
| fungi | oh, wait, but we untar and then rsync right? | 22:47 |
| fungi | so it more likely depends on rsync's options | 22:47 |
| fungi | which are probably hidden behind an ansible synchronize | 22:47 |
| ianw | fungi: docs are actually uploaded using our bespoke library implementation | 22:50 |
| fungi | oh, because of our handling of .root-marker and stuff | 22:52 |
| ianw | yeah, that was what i renamed in https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-afs-roots | 22:52 |
| ianw | if we could loop back on https://review.opendev.org/#/c/706176/ it's using packages now and passed ci | 22:54 |
| *** openstackgerrit has joined #openstack-infra | 22:56 | |
| openstackgerrit | Merged openstack/project-config master: Add a job to promote docs.airshipit.org content https://review.opendev.org/706598 | 22:56 |
| *** eharney has joined #openstack-infra | 23:02 | |
| clarkb | roman_g: looks like the more recent node failure you got was due to 2020-02-10 18:09:59,821 ERROR nodepool.NodeLauncher: [node: 0014470682] Request 300-0007334237: Detailed error for node 9eaed9df-c455-4487-bfc2-34fb0197cd2f: No valid host was found. There are not enough hosts available. | 23:03 |
| clarkb | roman_g: I believe that means the nova instance could not schedule those nodes | 23:04 |
| clarkb | (and nodepool tried three times) | 23:04 |
| ianw | fungi: https://storyboard.openstack.org/#!/story/2006598 # 38730 to track moving existing sites | 23:04 |
| fungi | ahh, yeah, that came up in the tc meeting last week | 23:05 |
| *** artom has quit IRC | 23:09 | |
| clarkb | infra-root ^ fyi on the airship test nodes | 23:09 |
| fungi | basically we had enough room in the various quota limits, but nova didn't have enough compute hosts it could schedule at least one of those flavors on? | 23:12 |
| fungi | in which case i guess that's a problem in citycloud? | 23:12 |
| clarkb | yes, that is how I read the error message | 23:13 |
| * clarkb is writing an email now | 23:19 | |
| *** jamesmcarthur has joined #openstack-infra | 23:22 | |
| clarkb | and now I've sent email with details to the airship crew | 23:24 |
| *** nicolasbock has quit IRC | 23:26 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: web: link to index.html if index_links is set https://review.opendev.org/705585 | 23:29 |
| *** ociuhandu has joined #openstack-infra | 23:30 | |
| *** nicolasbock has joined #openstack-infra | 23:34 | |
| *** ociuhandu has quit IRC | 23:35 | |
| *** rh-jelabarre has quit IRC | 23:37 | |
| *** jamesmcarthur has quit IRC | 23:39 | |
| ianw | kevinz: hey, when you're around see prior discussion on linaro-london images; i've filed https://storyboard.openstack.org/#!/story/2007195 #38731 with details to track | 23:41 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Install kubectl/oc into executor container image https://review.opendev.org/706995 | 23:43 |
| adriant | what is the version of sqlite installed on the VMs that run our tests in Zuul? | 23:47 |
| *** jaicaa has quit IRC | 23:47 | |
| *** jaicaa has joined #openstack-infra | 23:48 | |
| ianw | adriant: that's going to depend on what type of node you run on, but probably https://packages.ubuntu.com/bionic/sqlite | 23:48 |
| clarkb | adriant: it will be whatever the jobs install | 23:48 |
| clarkb | ya in most cases likely the distro version | 23:48 |
| adriant | 2.8! | 23:49 |
| adriant | ouch | 23:49 |
| adriant | that's like from 2015... | 23:49 |
| adriant | https://sqlite.org/chronology.html | 23:50 |
| adriant | wait, thats 2 not 3 | 23:50 |
| clarkb | adriant: it likely comes from debian which is where ubuntu picks it up | 23:50 |
| *** jamesmcarthur has joined #openstack-infra | 23:50 | |
| adriant | wow | 23:51 |
| adriant | 2005-12-19 2.8.17 | 23:51 |
| adriant | jesus | 23:51 |
| adriant | that's old | 23:51 |
| clarkb | its also possible the distro version is different than upstream version? | 23:51 |
| adriant | maybe | 23:51 |
| openstackgerrit | Merged opendev/system-config master: Allow for periodic afs releases from mirror-update https://review.opendev.org/706176 | 23:51 |
| clarkb | ya ubuntu splits it https://packages.ubuntu.com/bionic-updates/sqlite3 | 23:51 |
| clarkb | you can install either 2 or 3 looks like | 23:52 |
| adriant | I'm getting hit by it because of my failure here: https://review.opendev.org/#/c/706723/2 | 23:52 |
| adriant | so I can install 3 by adding it to bindep? | 23:52 |
| *** jamesmcarthur has quit IRC | 23:52 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Install kubectl/oc into executor container image https://review.opendev.org/706995 | 23:53 |
| clarkb | well I think 3 is probably installed but 3.22 is still too old according to your error message | 23:53 |
| adriant | yeah | 23:53 |
| adriant | I seem to have the latest on my machine | 23:53 |
| adriant | which is why I never hit that error before during testing | 23:53 |
| adriant | and it seems a new error with django 2.2. | 23:54 |
| adriant | may just have to rework my migrations :( | 23:54 |
| fungi | you likely want to use the sqlite3 package yeah | 23:54 |
| fungi | ahh, you reached that conclusion already | 23:55 |
| adriant | and even then 3.22 may be too old for the code as is | 23:55 |
| adriant | I have... 3.30.1 locally | 23:55 |
| fungi | yeah, sqlite 2->3 was an abi break, if memory serves, so debian (and thus ubuntu) makes both available | 23:55 |
| adriant | but my OS is a rolling release | 23:55 |
| fungi | i have 3.30.1 installed locally from debian on my systems too | 23:56 |
| adriant | that said, 3.22 is pretty old | 23:56 |
| clarkb | take it up with the distros :) | 23:56 |
| *** sgw has quit IRC | 23:56 | |
| clarkb | looks like it was about 3 months old at time of bionic's release | 23:56 |
| ianw | fungi: https://zuul.opendev.org/t/openstack/build/934a48743da44206b60454d8ef31238d so airship testing will need some docs published i guess, but otherwise LGTM. i guess you can drive some initial publishing of the docs then we can follow-up with the webserver? | 23:56 |
| clarkb | which isn't that egregious | 23:56 |
| *** eernst has joined #openstack-infra | 23:57 | |
| fungi | 3.22.0 was tagged 2018-01-22 so just a few months before ubuntu 18.04 lts was released, well into its release freeze i expect | 23:57 |
| fungi | yeah, what clarkb said | 23:57 |
| adriant | I know, I'm not entirely serious about it being old :P | 23:57 |
| fungi | ubuntu bionic is nearly 2 years old, and ubuntu makes lts releases every 2 years, so the one in 20.04 lts will likely be much newer | 23:57 |
| fungi | looks like they've got 3.30.1 going into it presently | 23:58 |
| clarkb | you can alsorun your jobs on another platform | 23:58 |
| adriant | that's an option | 23:59 |
| adriant | I know the migrations work against mariaDB | 23:59 |
| adriant | so I don't particularly want to change them if I can avoid it | 23:59 |
| *** slaweq has joined #openstack-infra | 23:59 | |
| adriant | and they do work against newer sqlite :/ | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!