| clarkb | anyone know why my zuul zuul-build-image job immediately fails on 702503? | 00:02 |
|---|---|---|
| clarkb | that change is only editing an rst file so likely isn't related to the change | 00:02 |
| clarkb | the parent chnage had the zuul-build-image job timeout | 00:03 |
| clarkb | is the immediate fail related to that because it is looking for the parent image artifact? | 00:03 |
| *** rfolco has quit IRC | 00:06 | |
| clarkb | I've rechecked the parent in an attempt to get that to go green then will recheck the child to test this theory | 00:06 |
| zbr | docs changed? lots 404 now: https://www.google.com/search?client=firefox-b-d&q=zuul+job+definition+requires | 00:13 |
| clarkb | zbr: they were reorganized, google should catch up after their next reindex | 00:14 |
| clarkb | I wonder if we can ask google to do that more quickly | 00:14 |
| zbr | yes you can | 00:14 |
| zbr | but you need few tricks, https://stackoverflow.com/questions/9466360/how-to-request-google-to-re-crawl-my-website | 00:15 |
| zbr | proof of ownership | 00:15 |
| zbr | ideally the weserver should have had added few HTTP redirects. | 00:16 |
| zbr | i need to go, is past midnight.. | 00:16 |
| clarkb | ya we can add an htaccess file with redirects | 00:17 |
| clarkb | re the image build jobs that seems to have been it | 01:10 |
| *** jamesmcarthur has joined #zuul | 01:37 | |
| *** michael-beaver has quit IRC | 01:40 | |
| *** jamesmcarthur has quit IRC | 01:44 | |
| clarkb | looks like the nodepool-zuul-functional job is flaky | 01:45 |
| clarkb | it seems to be breaking running manage ansible command | 01:46 |
| clarkb | not going to be able to look furhte rtonight but thought I'd call it out | 01:46 |
| *** sgw has quit IRC | 01:48 | |
| *** jamesmcarthur has joined #zuul | 02:25 | |
| *** jamesmcarthur_ has joined #zuul | 02:28 | |
| *** jamesmcarthur has quit IRC | 02:28 | |
| *** bhavikdbavishi has joined #zuul | 02:30 | |
| pabelanger | corvus: thanks! | 02:43 |
| *** rlandy has quit IRC | 03:20 | |
| *** sgw has joined #zuul | 03:45 | |
| *** jamesmcarthur_ has quit IRC | 04:08 | |
| *** jamesmcarthur has joined #zuul | 04:10 | |
| *** bhavikdbavishi has quit IRC | 04:36 | |
| *** jamesmcarthur has quit IRC | 04:43 | |
| *** jamesmcarthur has joined #zuul | 04:44 | |
| *** zxiiro has quit IRC | 04:48 | |
| *** jamesmcarthur has quit IRC | 04:50 | |
| *** jamesmcarthur has joined #zuul | 05:15 | |
| *** jamesmcarthur has quit IRC | 05:23 | |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #zuul | 05:34 | |
| *** jamesmcarthur has joined #zuul | 05:57 | |
| *** jamesmcarthur has quit IRC | 06:02 | |
| *** mugsie has quit IRC | 06:11 | |
| *** mugsie has joined #zuul | 06:14 | |
| *** bolg has joined #zuul | 06:15 | |
| yoctozepto | corvus: thanks! | 06:22 |
| *** bolg has quit IRC | 06:22 | |
| *** bolg has joined #zuul | 06:25 | |
| *** saneax has joined #zuul | 06:34 | |
| *** jamesmcarthur has joined #zuul | 06:36 | |
| *** jamesmcarthur has quit IRC | 06:42 | |
| *** dmsimard has quit IRC | 06:50 | |
| *** dmsimard has joined #zuul | 06:55 | |
| *** jamesmcarthur has joined #zuul | 07:38 | |
| *** jamesmcarthur has quit IRC | 07:43 | |
| *** avass has joined #zuul | 08:04 | |
| *** saneax has quit IRC | 08:14 | |
| *** tosky has joined #zuul | 08:19 | |
| *** arxcruz|off is now known as arxcruz | 08:20 | |
| *** swest has joined #zuul | 08:34 | |
| *** swest has quit IRC | 08:38 | |
| *** sshnaidm|afk is now known as sshnaidm | 08:38 | |
| *** swest has joined #zuul | 08:39 | |
| *** jamesmcarthur has joined #zuul | 08:39 | |
| *** jamesmcarthur has quit IRC | 08:45 | |
| *** jpena|off is now known as jpena | 08:50 | |
| *** dmellado has quit IRC | 08:51 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Add spec for scale out scheduler https://review.opendev.org/621479 | 08:55 |
| *** dmellado has joined #zuul | 08:55 | |
| *** bolg has quit IRC | 09:05 | |
| *** swest has quit IRC | 09:07 | |
| *** swest has joined #zuul | 09:30 | |
| *** jamesmcarthur has joined #zuul | 09:41 | |
| *** pcaruana has joined #zuul | 09:42 | |
| *** jamesmcarthur has quit IRC | 09:46 | |
| *** swest has quit IRC | 10:15 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option https://review.opendev.org/702275 | 10:26 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: JWT drivers: Deprecate RS256withJWKS, introduce OpenIDConnect https://review.opendev.org/701972 | 10:34 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option https://review.opendev.org/702275 | 10:34 |
| *** jamesmcarthur has joined #zuul | 10:42 | |
| *** jamesmcarthur has quit IRC | 10:46 | |
| *** pcaruana has quit IRC | 11:12 | |
| *** swest has joined #zuul | 11:19 | |
| *** bolg has joined #zuul | 11:21 | |
| *** swest has quit IRC | 11:24 | |
| *** swest has joined #zuul | 11:24 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Document the buildsets endpoint https://review.opendev.org/702127 | 11:30 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support dynamic badges https://review.opendev.org/702128 | 11:30 |
| *** jamesmcarthur has joined #zuul | 11:43 | |
| *** jamesmcarthur has quit IRC | 11:48 | |
| *** mgoddard has quit IRC | 11:55 | |
| *** mgoddard has joined #zuul | 12:01 | |
| *** bolg has quit IRC | 12:02 | |
| *** swest has quit IRC | 12:02 | |
| *** mgoddard has quit IRC | 12:05 | |
| *** rfolco has joined #zuul | 12:06 | |
| *** sshnaidm is now known as sshnaidm|afk | 12:09 | |
| *** mgoddard has joined #zuul | 12:15 | |
| *** swest has joined #zuul | 12:17 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option https://review.opendev.org/702275 | 12:18 |
| *** jpena is now known as jpena|lunch | 12:33 | |
| *** swest has quit IRC | 12:43 | |
| yoctozepto | hey Zuulers, Zuul docs got shrunk or /me went blind | 12:43 |
| yoctozepto | https://zuul-ci.org/docs/zuul/index.html | 12:43 |
| yoctozepto | ah, they are reorganized | 12:43 |
| *** jamesmcarthur has joined #zuul | 12:44 | |
| yoctozepto | "search is your friend" doh | 12:44 |
| *** bolg has joined #zuul | 12:45 | |
| *** pcaruana has joined #zuul | 12:49 | |
| *** jamesmcarthur has quit IRC | 12:49 | |
| yoctozepto | quick question - does Zuul merge dictionary vars key by key? | 12:53 |
| yoctozepto | I mean - if you have a dictionary as a var, is it merged key-by-key | 12:54 |
| yoctozepto | or replaced | 12:54 |
| yoctozepto | when inheriting | 12:54 |
| *** rlandy has joined #zuul | 13:05 | |
| *** bolg has quit IRC | 13:07 | |
| *** swest has joined #zuul | 13:11 | |
| *** saneax has joined #zuul | 13:11 | |
| *** jamesmcarthur has joined #zuul | 13:15 | |
| *** jamesmcarthur has quit IRC | 13:24 | |
| *** bolg has joined #zuul | 13:24 | |
| *** jamesmcarthur has joined #zuul | 13:24 | |
| *** bolg has quit IRC | 13:30 | |
| *** swest has quit IRC | 13:31 | |
| *** bolg has joined #zuul | 13:33 | |
| *** jpena|lunch is now known as jpena | 13:35 | |
| *** jamesmcarthur has quit IRC | 13:36 | |
| *** sgw has quit IRC | 13:37 | |
| *** bolg has quit IRC | 13:41 | |
| avass | yoctozepto: Variables are overridden according to: https://zuul-ci.org/docs/zuul/references/config.html#attr-job.vars | 13:43 |
| *** jamesmcarthur has joined #zuul | 13:47 | |
| *** swest has joined #zuul | 13:51 | |
| *** bolg has joined #zuul | 13:52 | |
| *** mnaser has quit IRC | 13:54 | |
| *** mnaser has joined #zuul | 13:55 | |
| *** swest has quit IRC | 13:57 | |
| *** sgw has joined #zuul | 13:59 | |
| *** rfolco has quit IRC | 14:00 | |
| *** rfolco has joined #zuul | 14:01 | |
| *** bolg has quit IRC | 14:02 | |
| *** bolg has joined #zuul | 14:04 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option https://review.opendev.org/702275 | 14:18 |
| yoctozepto | avass: that's what I thought until https://review.opendev.org/702132 | 14:35 |
| yoctozepto | it clearly merged the two dicts | 14:35 |
| yoctozepto | not replaced | 14:35 |
| *** sshnaidm|afk is now known as sshnaidm | 14:39 | |
| *** jamesmcarthur has quit IRC | 14:40 | |
| AJaeger | yoctozepto: the docs say "merge": This means a variable definition with the same name will override a previously defined variable, but new variable names will be added to the set of defined variables. | 14:42 |
| AJaeger | what was different for you? | 14:42 |
| AJaeger | yoctozepto: ah, you have a dict as variable? | 14:42 |
| *** electrofelix has joined #zuul | 14:49 | |
| corvus | it's a dictionary merge (key-by-key). i guess the docs could be more clear. the first sentence is attempting to say dictionaries merge, the second is attempting to say what happens to each leaf node. | 14:51 |
| mnaser | https://review.opendev.org/#/q/topic:collect-container-logs -- anyone mind reviewing the nodepool and zuul-registry there (and perhaps system-config too if you have access? :)) | 14:53 |
| * mnaser is trying to clear ever growing review queue | 14:54 | |
| yoctozepto | AJaeger, corvus: yup and yup, that's the behavior I am seeing there | 14:54 |
| yoctozepto | it's actually nice since I can minimize my case to really just disabling tls for now :-) | 14:55 |
| yoctozepto | and any uselessly deployed services | 14:55 |
| mnaser | corvus: do you think it would be ok for you to self approve stack starting at https://review.opendev.org/#/c/702052/3 ? IMHO we don't have a lot of k8s reviewers (and I think they are ok changes and so does tristanC) | 14:56 |
| corvus | yoctozepto: yep that's the idea | 14:58 |
| *** saneax has quit IRC | 14:58 | |
| corvus | mnaser: yep, i think that stack is gtg; let's ask tristanC and/or mordred to look at https://review.opendev.org/702101 oh it looks like tristanC just did. :) | 14:59 |
| mnaser | :) | 14:59 |
| mnaser | that way i can try starting to consume that too | 15:00 |
| mnaser | speaking of there's still that inotify patch, i honestly havent gotten around cleaning that up, which means we have a full scheduler restart on tenant config changes | 15:00 |
| corvus | cool. my zuul-for-gerrit work is paused while we work out some more google account stuff. but it's still running the tip of that stack. | 15:01 |
| corvus | mnaser: re https://review.opendev.org/689560 i think mhu just did some similar work | 15:01 |
| tristanC | corvus: would you know why https://review.opendev.org/#/c/702106/6/playbooks/zuul-operator-functional-k8s/pre.yaml fails with 'directory /run/user/1000 does not exist' here: https://zuul.opendev.org/t/zuul/build/bf5cf937edc64b46a0a15be45e2f966a/console#1/0/68/ubuntu-xenial | 15:02 |
| mnaser | tristanC: /me looks at url and suggest to start blindly using bionic at least and see how that works | 15:02 |
| mnaser | corvus: cool, i havent been keeping up with changes as much, things are flying by quickly these days :< | 15:03 |
| *** rfolco is now known as rfolco|bbl | 15:03 | |
| corvus | tristanC: yeah i think that's a weird podman installation error. it means the user id mapping stuff isn't correct. i think mnaser's suggestion of bionic is probably the place to start | 15:04 |
| corvus | tristanC: oh wait, that's not the error i thought it was | 15:04 |
| corvus | tristanC: that's for use-buildset-registry... but yeah, maybe that's still a xenial vs bionic error | 15:05 |
| *** avass has quit IRC | 15:06 | |
| tristanC | ok, thanks, let's see if using bionic helps | 15:07 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Bump ansible-lint to 4.2.0 https://review.opendev.org/702679 | 15:07 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Replace existing operator tasks with the new application https://review.opendev.org/702106 | 15:07 |
| openstackgerrit | Merged zuul/zuul-helm master: Add option to manage secrets outside of helm https://review.opendev.org/702052 | 15:13 |
| openstackgerrit | Merged zuul/zuul-helm master: Add Zuul charts https://review.opendev.org/700460 | 15:14 |
| openstackgerrit | Merged zuul/zuul-helm master: Allow tenant config file to be managed externally https://review.opendev.org/702057 | 15:14 |
| openstackgerrit | Merged zuul/zuul-helm master: Allow override of service type for zuul-web https://review.opendev.org/702101 | 15:14 |
| *** bhavikdbavishi has joined #zuul | 15:14 | |
| *** jamesmcarthur has joined #zuul | 15:22 | |
| tristanC | I'm connected to sip:6001@pbx.openstack.org for the zuul-operator discussion | 15:28 |
| clarkb | I'm about to be connected. Maybe a minite or two late | 15:30 |
| corvus | mnaser, mordred: reminder ^ | 15:31 |
| mnaser | connecting.. | 15:33 |
| * fungi realized he still needed to put the battery back into my phone and boot it up, will listen into whatever's still being discussed | 15:38 | |
| *** bolg has quit IRC | 15:42 | |
| corvus | https://zuul-ci.org/docs/zuul/references/developer/specs/kubernetes-operator.html | 15:54 |
| *** zxiiro has joined #zuul | 15:58 | |
| *** bhavikdbavishi has quit IRC | 16:01 | |
| mnaser | https://kopf.readthedocs.io/en/latest/ | 16:02 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option https://review.opendev.org/702275 | 16:02 |
| openstackgerrit | Merged zuul/zuul master: Fix nits from initial reorg https://review.opendev.org/702532 | 16:06 |
| mnaser | https://kopf.readthedocs.io/en/latest/embedding/ | 16:06 |
| zbr | what to do about opensuse-tumbleween job? ok to remove? see https://zuul.opendev.org/t/zuul/build/32f5180d6d4043b3bfec0ce083ed8584 | 16:15 |
| clarkb | zbr: we should consider making it non voting. The job is a good canary for things changing in linux, but iirc we allow its mirroring to fail (due to size and available bw) and that causes this error you see | 16:17 |
| zbr | clarkb: ok to make it nv in same change or you want separated? | 16:18 |
| clarkb | Maybe separate would be good if we look into why it is non voting later | 16:19 |
| zbr | sure | 16:19 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Making zuul-jobs-test-multinode-roles-opensuse-tumbleweed nv https://review.opendev.org/702693 | 16:23 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Bump ansible-lint to 4.2.0 https://review.opendev.org/702679 | 16:24 |
| AJaeger | zbr: use the tool to regenerate the project-stanza so that the opensuse-tumbleweed job does not run in gate | 16:25 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: JWT drivers: Deprecate RS256withJWKS, introduce OpenIDConnect https://review.opendev.org/701972 | 16:26 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option https://review.opendev.org/702275 | 16:26 |
| pabelanger | doh, browser crashed | 16:30 |
| *** rishabhhpe has joined #zuul | 16:31 | |
| zbr | AJaeger: tx for the hint. I wonder why update-test-platforms is not part of linting. | 16:31 |
| zbr | pabelanger: since last two weeks fifrefox is crashing 4-5 times a day for me, i did not had time to look into but annoying. | 16:31 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Extract project config YAML into ref docs https://review.opendev.org/701977 | 16:32 |
| pabelanger | I guess meeting rapped up? | 16:32 |
| clarkb | I've approved https://review.opendev.org/#/c/701282/ | 16:32 |
| clarkb | pabelanger: yup. corvus will send out a summary | 16:32 |
| pabelanger | ack | 16:32 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Replace existing operator tasks with the new application https://review.opendev.org/702106 | 16:32 |
| corvus | homework is to look into kopf :) | 16:33 |
| rishabhhpe | Hi all, We are facing a issue while spawning an instance from the DIB image built . I had pasted the error log and nodepool.yaml here -: http://paste.openstack.org/show/788419/ .. please check and let me know if i am missing anything in configuration .. i had tried both with public network and shared but stil having the same issue | 16:33 |
| pabelanger | rishabhhpe: firewall blocking port 22? | 16:34 |
| rishabhhpe | how can i check that ? because i am not able to ping that ip also from the host devstack machine | 16:37 |
| clarkb | rishabhhpe: look at your openstack security groups | 16:38 |
| clarkb | and any other firewalls you may have | 16:38 |
| pabelanger | +1 | 16:38 |
| clarkb | another thing to check is the server console log via the openstack api | 16:38 |
| pabelanger | yah, I'd make sure glean is getting IP info | 16:39 |
| clarkb | should tell you if the server has booted and if it is running services (though it will be quite verbose) | 16:39 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Making zuul-jobs-test-multinode-roles-opensuse-tumbleweed nv https://review.opendev.org/702693 | 16:39 |
| *** openstackgerrit has quit IRC | 16:43 | |
| *** pcaruana has quit IRC | 16:45 | |
| rishabhhpe | clarkb: pabelanger: we are using default security group for this .. but for time being i had allowed the ingress for port 22 from all IP ranges will see if it works out | 16:47 |
| *** openstackgerrit has joined #zuul | 16:48 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Stop compressing files during intermediate steps part 2 https://review.opendev.org/701282 | 16:48 |
| zbr | i need to optionally enable docker (install-docker) for any of the tox jobs without reparenting them (as I would loose their own customizations), how can I achieve this? | 16:48 |
| pabelanger | rishabhhpe: we usually allow all ingress for nodepool nodes, then firewall of things as needed on node side | 16:49 |
| pabelanger | via DIB | 16:49 |
| *** clayg has quit IRC | 16:49 | |
| *** clayg has joined #zuul | 16:49 | |
| zbr | i was considering adding a "extra_roles" argument which tells "tox" base job to run these roles. makes sense? better ideas? | 16:49 |
| zbr | zuul has single parent inheritance, so I am looking for ways to compose stuff without exploding the job definitions. | 16:50 |
| corvus | zbr: can you use a pre-playbook? | 16:50 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option https://review.opendev.org/702275 | 16:50 |
| tristanC | zbr: creates new 'docker-tox-*' (or 'podman-tox-*') that parents to their 'tox-*' equivalent and adds a pre run playbook to install the container runtime? | 16:50 |
| corvus | tristanC used more better words :) | 16:51 |
| rishabhhpe | pabelanger: well i had choosen the cidr which is matching our host ip only ..but still getting the same error ... can u suggest me the way how i can check the things for firewall as u suggested above . | 16:52 |
| tristanC | mnaser: corvus: switching to bionic fixed the job, you can see the operator spawing zuul control plane here: https://zuul.opendev.org/t/zuul/stream/271de02c139b4b9faf78b1931f3a65a9?logfile=console.log | 16:54 |
| corvus | tristanC: \o/ | 16:55 |
| pabelanger | rishabhhpe: you need to log into vm console, and check if IP is attached | 16:55 |
| rishabhhpe | pabelanger: i tried that way too but it is not allowing me logged in to console also | 16:56 |
| tristanC | mnaser: tobiash: and here is where you would do the queue dump when zuul-scheduler is restarted: https://review.opendev.org/#/c/702106/8/roles/zuul/tasks/main.yaml | 16:56 |
| *** themroc has quit IRC | 16:56 | |
| rishabhhpe | pabelanger: http://paste.openstack.org/show/788422/ please find this log | 17:00 |
| zbr | tristanC: i am looking for a way to avoid duplicating each such job because I already have (py27/35/36/37/38) variants and *some* users of these jobs may also need docker, or podman, or both. I do not think that creating new jobs for each of these matrix combination does scale well. | 17:01 |
| pabelanger | rishabhhpe: I've never seen that before | 17:02 |
| pabelanger | what is it from? | 17:02 |
| zbr | thus is why i was looking for a variable that can be used by the user when he adds a tox-py36 job but he also needs a running docker with it. | 17:02 |
| rishabhhpe | from openstack console i am trying to connect to spawned VM console | 17:02 |
| pabelanger | sounds like something is up with cloud | 17:03 |
| pabelanger | likely need to contact admins | 17:03 |
| zbr | this should allow us to on/off that feature, without creating new jobs. | 17:03 |
| openstackgerrit | Clément Mondion proposed zuul/nodepool master: add requirements for json logging https://review.opendev.org/702704 | 17:03 |
| clarkb | pabelanger: rishabhhpe another thing to consider is if using glean you need config drive | 17:04 |
| clarkb | also compare against booting a default cloud image in the same network with same project | 17:04 |
| pabelanger | ah, yup | 17:04 |
| pabelanger | good thing to check | 17:05 |
| clarkb | zbr: I think what we've found is that being explicit is often a good thing | 17:05 |
| clarkb | zbr: so yes it requires new jobs, but then it is clear to users how to consume jobs with those features | 17:05 |
| clarkb | and you don't have to do any special configuration, you simply use the job | 17:06 |
| tristanC | zbr: i'm not sure to understand the use-case, but if a job requires something to work, it's often better to create an explicite variant using parenting instead of implicitely adds toggle in every project pipeline config | 17:06 |
| zbr | clarkb: and if I want a job that has both pdoman and docker installed, I created a docker-podman-tox-pyXY job? | 17:06 |
| clarkb | zbr: or you create some middle group balance between having too much installed and too little | 17:07 |
| clarkb | zbr: like maybe you have a "containers-tox" set of jobs and install all the related container tooling | 17:07 |
| zbr | in fact I already know 3 such features: docker, podman and libvirt. | 17:07 |
| corvus | jobs are free. we don't charge you for them. | 17:07 |
| zbr | yeah :D | 17:08 |
| *** gmann has quit IRC | 17:09 | |
| *** gmann has joined #zuul | 17:10 | |
| pabelanger | +1 for layered jobs | 17:10 |
| * Shrews googles json logging | 17:13 | |
| clarkb | Shrews: oslo logging implements a python logging handler for it | 17:14 |
| clarkb | if you need examples | 17:14 |
| zbr | thanks * for feedback on job feature expansion, you convinced me to create new jobs instead of using variables. | 17:15 |
| Shrews | clarkb: i'm not sure we want to make a new requirements.txt entry for every conceivable form of logging someone might want to use, though. or maybe we do? i'm undecided on it, but leaning toward the former | 17:15 |
| clarkb | Shrews: I don't think we should use oslo.logging if that is what you mean. But I think the actual implementation is pretty simple if we wanted to vendor it | 17:16 |
| Shrews | clarkb: no, i think they want this: https://github.com/madzak/python-json-logger | 17:16 |
| clarkb | Shrews: hits basically take the input to the logging function and instead of interpolating vars into a string you make a json dict with them instead | 17:16 |
| clarkb | *it is basically | 17:16 |
| Shrews | not oslo | 17:16 |
| Shrews | referring to https://review.opendev.org/702704 | 17:17 |
| clarkb | also json logging is pretty common these days | 17:17 |
| clarkb | it makes indexing logs much simpler | 17:18 |
| Shrews | that's fine. my worry is adding a new requirement for every conceivable library someone might want to use | 17:18 |
| clarkb | right, that is why I mentioned an implementation should be simple if we want to go that route | 17:18 |
| clarkb | looks like that lib lets you customize the format of the output. I bet that is most of the code in the lib | 17:19 |
| rishabhhpe | pabelanger: clarkb: i am using config drive as true only ..clarkb: i tried booting a default image also and still the same issue is coming first of all i am not able to attach floating IP to it moreover cannot ssh in to the instance , | 17:21 |
| *** arxcruz is now known as arxcruz|off | 17:29 | |
| *** guilhermesp has quit IRC | 17:31 | |
| zbr | AJaeger: clarkb: please look again at https://review.opendev.org/#/c/702693/ | 17:32 |
| *** guilhermesp has joined #zuul | 17:32 | |
| zbr | i hope you like my "dirty" check | 17:32 |
| *** evrardjp has quit IRC | 17:34 | |
| *** evrardjp has joined #zuul | 17:34 | |
| rishabhhpe | clarkb: pabelanger: i had pasted the booting log for vm here http://paste.openstack.org/show/788425/ | 17:41 |
| fungi | Shrews: clarkb: a middle ground might be to declare those in an "extras" entry in the package metadata | 17:42 |
| * mordred has arrived at his hotel - apologizes for screwing up timezone math somehow and missing the operator call | 17:43 | |
| clarkb | fungi: ya, or simply suggest people install those items alongside zuul independently. I think we do that for yappi | 17:43 |
| fungi | Shrews: clarkb: you could also make it a soft dependency and only use it if present | 17:43 |
| fungi | right, something like that | 17:43 |
| corvus | mordred: i'll send out a quick summary email later; we had a good level setting call, have some things to look into, and will reconvene next week | 17:43 |
| corvus | i think we made progress | 17:43 |
| fungi | Shrews: clarkb: we do that with statsd too, right? | 17:43 |
| mordred | corvus: awesome. I will not be on a plane next week | 17:44 |
| Shrews | clarkb: fungi: that's what i suggested in my response (and especially makes sense to do so independently since they want a specific version) | 17:44 |
| fungi | mordred: good, a week is a long time to spend on a plane | 17:44 |
| clarkb | fungi: I think we require the lib install for that but don't do anything unless it is configured? | 17:44 |
| mordred | corvus: I was looking at kopf briefly based on scrollback | 17:44 |
| mordred | fungi: 30 hours is already a long time to spend on a plane - so yeah, a week would be much worse :) | 17:44 |
| pabelanger | rishabhhpe: yah, clean isn't booting properly | 17:45 |
| pabelanger | glean* | 17:45 |
| Shrews | clarkb: fungi: right, we don't require a statsd, but do need to require the lib since we actually use it | 17:45 |
| fungi | mordred: at least for me, 30 hours on a plane would feel like a week | 17:45 |
| Shrews | s/use it/have code to use it/ | 17:45 |
| fungi | ahh, right | 17:45 |
| clarkb | pabelanger: usually it writes a bit more about what it is doing like mounting the config drive then writing files etc | 17:46 |
| fungi | well, in the json-logging case, seems like you could try to import it, and then use it if that doesn't raise an exception. adding it to a [json-logging] extra for the zuul package would just be additional convenience? | 17:46 |
| rishabhhpe | pabelanger: clarkb: so in that case what needs to be done from my end ? | 17:46 |
| clarkb | rishabhhpe: I would confirm that the config drive is used (do a server show on it and check the value) | 17:47 |
| pabelanger | clarkb: I don't think systemd has started it, which makes me things dependency order | 17:47 |
| tristanC | mordred: i also had a look at kopf, that would work nicely. But is there any 'stable' operator already written with it? | 17:49 |
| rishabhhpe | clarkb: please see the output for openstack server show http://paste.openstack.org/show/788427/ | 17:50 |
| clarkb | rishabhhpe: | config_drive | | that tells me there is no config drive | 17:53 |
| clarkb | it should say True | 17:53 |
| clarkb | iirc | 17:54 |
| rishabhhpe | so it means the DIB image i need to built again and try the same ? | 17:54 |
| tristanC | in the meantime, i'm adding tls cert for zuul gearman and scheduler queues dump/restore logic to the current zuul-operator | 17:54 |
| fungi | rishabhhpe: the configdrive property is something you tell nodepool to request from nova | 17:55 |
| clarkb | and I thought it was on by default now. Is it possible the cloud doesn't offer config drive? | 17:55 |
| rishabhhpe | OK i got it so config drive is missing from the dib level only | 17:55 |
| fungi | i dunno, but we set it like https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl01.openstack.org.yaml#L63 | 17:55 |
| *** jpena is now known as jpena|off | 17:57 | |
| fungi | rishabhhpe: possibly either missing in nova's configuration for your openstack environment, or missing from your nodepool launcher configuration | 17:57 |
| fungi | i don't think dib has anything to do with that | 17:57 |
| rishabhhpe | fungi: and clarkb: i will try to built the image again and how to check this at nova level ? i had set local.conf file but not sure whether i had given anything related to config drive in it | 17:58 |
| clarkb | rishabhhpe: you don't need to build the image again | 17:58 |
| clarkb | config drive is a node instance boot time option | 17:58 |
| clarkb | independent of the image | 17:58 |
| fungi | rishabhhpe: https://docs.openstack.org/nova/queens/user/config-drive.html | 17:58 |
| rishabhhpe | how to check where it is missing can i put a info at dib image creation level ? | 17:58 |
| fungi | er, that's a link to an older release but should still be the same | 17:59 |
| clarkb | rishabhhpe: see fungi's link for how we set it https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl01.openstack.org.yaml#L63 | 17:59 |
| fungi | also i meant to link to https://docs.openstack.org/ironic/latest/install/configdrive.html | 17:59 |
| fungi | for how to configure nova | 17:59 |
| clarkb | rishabhhpe: I thought that was set by default but you can try setting that value to be sure | 17:59 |
| fungi | no, wait, that's ironic | 18:00 |
| fungi | i'm really not doing great with the documentation links today, sorry :/ | 18:00 |
| fungi | https://docs.openstack.org/nova/latest/admin/config-drive.html | 18:00 |
| fungi | THAT's what i meant to link | 18:00 |
| rishabhhpe | fungi: ok .. so after changing these params do i need to recycle the nova service ? | 18:02 |
| fungi | rishabhhpe: well, what that document says (summarizing) is that if you make sure genisoimage is installed on all the compute hosts then configdrive should just work | 18:03 |
| clarkb | fungi: that is a fairly recent chagne though, if the cloud is older then you need to explicitly enable config drive | 18:03 |
| fungi | ahh, yeah, so depends on what openstack/nova release you're running | 18:04 |
| fungi | it seems to at least also be relevant for the train release | 18:04 |
| fungi | that page does not appear in the stein release documentation for nova, so not sure if it's true for releases older than train | 18:06 |
| *** electrofelix has quit IRC | 18:07 | |
| fungi | https://docs.openstack.org/nova/stein/configuration/config.html#DEFAULT.force_config_drive suggests that it's also the case on stein | 18:11 |
| fungi | and back as far as pike at least, i think | 18:13 |
| *** reiterative has quit IRC | 18:15 | |
| fungi | according to dansmith in #openstack-nova it's been available by default for "a long time" | 18:15 |
| clarkb | ah ok I must be thinking of the force option then | 18:16 |
| clarkb | since that defaults to false forcing users to request a config drive | 18:16 |
| clarkb | I thought the plan was to invert that | 18:16 |
| clarkb | since there isn't a good reason not to have a config drive | 18:16 |
| fungi | well, we've seen one corner case where it's problematic | 18:17 |
| clarkb | oh? | 18:17 |
| fungi | xen's limit on the number of attached block devices for a guest | 18:17 |
| fungi | configdrive eats one | 18:17 |
| fungi | when you're limited to 16 and already consuming one for the rootfs and one for an ephemeral disk, configdrive is the difference between 14 and 13 cinder volumes you can attach | 18:18 |
| clarkb | ya so you can force it off | 18:18 |
| clarkb | at least that was my argument | 18:18 |
| fungi | right, and it's admittedly an unusual case | 18:18 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Update sdk to v0.13.0 and install dhall-to-json https://review.opendev.org/702103 | 18:25 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Import dhall-operator https://review.opendev.org/702104 | 18:25 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Import zuul application from dhall-zuul https://review.opendev.org/702105 | 18:25 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Replace existing operator tasks with the new application https://review.opendev.org/702106 | 18:25 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Generate TLS certificats for the gearman service https://review.opendev.org/702716 | 18:25 |
| pabelanger | ugh, something with DIB and centos-8 is now failing | 18:26 |
| pabelanger | http://paste.openstack.org/show/788433/ | 18:26 |
| clarkb | pabelanger: centos 8.1 just happened | 18:26 |
| clarkb | or some large set up package updates anyway | 18:27 |
| pabelanger | oh, yah | 18:27 |
| pabelanger | that would do it | 18:27 |
| pabelanger | great | 18:27 |
| pabelanger | clarkb: that would make sense why glibc is complaining | 18:28 |
| rishabhhpe | fungi: and clarkb: now even after changing the nova.conf file and restarting the service i tried building the default image cirros but still not getting the config drive output populated in that openstack server show o/p : http://paste.openstack.org/show/788432/ booting log -: http://paste.openstack.org/show/788431/ | 18:28 |
| clarkb | pabelanger: I'm guessing that /etc/yum.repos.d is no longer part of new centos? | 18:28 |
| clarkb | pabelanger: thats just a guess though | 18:28 |
| clarkb | pabelanger: you probably need to trap into that build and look at it | 18:28 |
| *** rfolco|bbl is now known as rfolco | 18:28 | |
| pabelanger | clarkb: exactly how I wanted to spend my day today | 18:29 |
| clarkb | pabelanger: I mean you can probably ignore it for a while | 18:29 |
| clarkb | your existing 8.0 image will continue to work >_> | 18:30 |
| pabelanger | that's the thing, they don't | 18:30 |
| *** webknjaz has quit IRC | 18:30 | |
| *** samccann has quit IRC | 18:30 | |
| *** evgenyl has quit IRC | 18:30 | |
| *** ChrisShort has quit IRC | 18:30 | |
| *** zxiiro has quit IRC | 18:30 | |
| *** donnyd has quit IRC | 18:30 | |
| *** tdasilva has quit IRC | 18:30 | |
| *** kmalloc has quit IRC | 18:30 | |
| *** wxy-xiyuan has quit IRC | 18:31 | |
| *** maxamillion has quit IRC | 18:31 | |
| *** tributarian has quit IRC | 18:31 | |
| *** dcastellani has quit IRC | 18:31 | |
| clarkb | oh huh | 18:31 |
| *** mnaser has quit IRC | 18:31 | |
| pabelanger | http://paste.openstack.org/raw/788436/ | 18:31 |
| pabelanger | glibc wants to update, and conflicts | 18:31 |
| fungi | rishabhhpe: and you added config-drive: true to the diskimages entry for it in your providers list in the nodepool launcher's nodepool.yaml file? | 18:31 |
| *** donnyd has joined #zuul | 18:31 | |
| *** iamweswilson has quit IRC | 18:31 | |
| *** Shrews has quit IRC | 18:31 | |
| *** stevthedev has quit IRC | 18:31 | |
| fungi | rishabhhpe: like we do in https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl01.openstack.org.yaml#L63 | 18:32 |
| *** openstackstatus has quit IRC | 18:32 | |
| *** wxy-xiyuan has joined #zuul | 18:32 | |
| *** webknjaz has joined #zuul | 18:33 | |
| *** kmalloc has joined #zuul | 18:33 | |
| *** ChrisShort has joined #zuul | 18:33 | |
| *** maxamillion has joined #zuul | 18:33 | |
| *** samccann has joined #zuul | 18:33 | |
| *** tdasilva has joined #zuul | 18:33 | |
| *** dcastellani has joined #zuul | 18:33 | |
| *** johnsom has quit IRC | 18:33 | |
| *** tributarian has joined #zuul | 18:33 | |
| *** tosky has quit IRC | 18:33 | |
| *** mnaser has joined #zuul | 18:33 | |
| *** iamweswilson has joined #zuul | 18:34 | |
| *** Shrews has joined #zuul | 18:35 | |
| *** johnsom has joined #zuul | 18:35 | |
| clarkb | pabelanger: we last built our centos-8 image 17 hours ago | 18:37 |
| clarkb | we are about 8 hours away from failure I think | 18:37 |
| pabelanger | yah | 18:38 |
| *** sshnaidm is now known as sshnaidm|afk | 18:38 | |
| pabelanger | my last build is 01:02:45:50 | 18:38 |
| *** ChrisShort has quit IRC | 18:38 | |
| pabelanger | but, surprised openstack isn't seeing conflict | 18:38 |
| *** samccann has quit IRC | 18:38 | |
| clarkb | pabelanger: our mirror hasn't updated yet | 18:39 |
| pabelanger | ah, right | 18:39 |
| clarkb | I think that should happen momentarily though | 18:39 |
| clarkb | (the sync shoudl be running right now) | 18:39 |
| rishabhhpe | fungi: yes i tried that way but still the config-drive is not getting populated .. later i tried the solution recommended by clarkb: that is also not populating the config drive for new instances | 18:40 |
| *** kmalloc has quit IRC | 18:40 | |
| *** donnyd has quit IRC | 18:40 | |
| pabelanger | heh, I can get build logs from centos-8-0000000148 because latest image is at centos-8-0000000684 and log rotated deleted them | 18:41 |
| *** donnyd has joined #zuul | 18:43 | |
| *** tdasilva has quit IRC | 18:43 | |
| rishabhhpe | fungi: clarkb: please find booting log for instance in which i hard coded the value for config drive and still not able to login to it -: http://paste.openstack.org/show/788439/ | 18:43 |
| clarkb | pabelanger: remote: https://review.opendev.org/702722 DNM debugging centos 8.1 updates bit of a brute force attempt to reproduce | 18:43 |
| pabelanger | ++ | 18:43 |
| fungi | rishabhhpe: that looks like you're using cloud-init instead of glean, but it does at least seem to indicate that it sees a configdrive attached | 18:44 |
| *** jamesmcarthur has quit IRC | 18:44 | |
| clarkb | pabelanger: https://review.opendev.org/702721 | 18:44 |
| pabelanger | Nice! | 18:45 |
| rishabhhpe | fungi: so in that case what changes can be done from my end to make it working ? | 18:46 |
| clarkb | fungi: rishabhhpe I think cloud init is using metadata service there, not config drive | 18:49 |
| rishabhhpe | clarkb: i had not set anything related to metadata at dib level and also at instance creation level although for the log which i shared with you is the custom instance which i spawned through cli using command openstack server create --config-drive true --imageubuntu-bionic-1578956527 --flavor m1.large --network shared --key-name nodepool MYINSTANCE | 18:52 |
| clarkb | rishabhhpe: if you nova show that instance do you see config-drive | true? | 18:52 |
| rishabhhpe | let me check | 18:53 |
| rishabhhpe | i think nova show and openstack server show will give the same info ? if it is correct yes it is showing the config-drive true | 18:53 |
| clarkb | yes that is the same info | 18:54 |
| clarkb | alright so you need to configure nodepool to do the same for the nodepool instalces. That is what https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl01.openstack.org.yaml#L63 shows | 18:54 |
| rishabhhpe | yeah i will do that too . but my concern is still i am not able to log in to the instance .. hopefully i will get the same error again for ssh connection | 18:55 |
| fungi | clarkb: the reason i thought it was using configdrive is "finished at Wed, 15 Jan 2020 18:35:52 +0000. Datasource DataSourceConfigDrive [net,ver=2][source=/dev/sr0]." | 18:56 |
| fungi | not sure where it would get /dev/sr0 from otherwise | 18:56 |
| clarkb | fungi: ya rishabhhpe confirmed it was config drive (I say metadata crawler and assumed metadata service but that might just be a generic get data from anywhere tool) | 18:57 |
| *** samccann has joined #zuul | 18:58 | |
| rishabhhpe | fungi: clarkb: had pasted the output here http://paste.openstack.org/show/788440/ | 18:59 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: install-kubernetes: disable docker userland proxy https://review.opendev.org/702723 | 19:01 |
| *** sugaar has quit IRC | 19:02 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: DNM: test without docker userland proxy https://review.opendev.org/702724 | 19:02 |
| openstackgerrit | Merged zuul/zuul master: Add note about GitHub Action limitations https://review.opendev.org/702503 | 19:08 |
| *** sshnaidm|afk is now known as sshnaidm | 19:09 | |
| *** kmalloc has joined #zuul | 19:19 | |
| *** rishabhhpe has quit IRC | 19:21 | |
| *** saneax has joined #zuul | 19:22 | |
| pabelanger | clarkb: upload centos-8 image now, to see if boots | 19:22 |
| pabelanger | patch worked | 19:22 |
| clarkb | pabelanger: cool, its been approved and I expect when ianw's day starts we can think about a dib release | 19:23 |
| *** openstackstatus has joined #zuul | 19:23 | |
| *** ChanServ sets mode: +v openstackstatus | 19:23 | |
| pabelanger | ++ | 19:23 |
| pabelanger | clarkb: corvus: speaking of release, I take it out fallout on zuul restart yesterday (had to #dadops early) | 19:24 |
| pabelanger | was none | 19:24 |
| clarkb | ya I think we can also do zuul and nodepool releases today | 19:24 |
| pabelanger | if so, do you think we can cut new zuul release also? | 19:24 |
| pabelanger | great | 19:24 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Dockerfile: add support for arbritary uid https://review.opendev.org/650246 | 19:25 |
| *** rishabhhpe has joined #zuul | 19:26 | |
| rishabhhpe | Sorry if i missed some chat because i was disconnected due to network -- any findings regarding this fungi: clarkb: had pasted the output here http://paste.openstack.org/show/788440/ | 19:28 |
| clarkb | rishabhhpe: that instance has config drive now, can you ping it? | 19:29 |
| clarkb | did the console log change with the availability of config drive? | 19:29 |
| rishabhhpe | let me check | 19:30 |
| *** ChrisShort has joined #zuul | 19:31 | |
| tobiash | rishabhhpe: dib by default configures cloud init to not use configdrive. To use config drive with a dib built image it must be configured accordingly | 19:31 |
| clarkb | tobiash: this image is using glean | 19:31 |
| clarkb | (at least an earlier paste showed glean failing and no config drive) | 19:31 |
| tobiash | ah ok, then forget what I wrote | 19:32 |
| rishabhhpe | clarkb: log is still not updated and also i am not able to ping the IP | 19:33 |
| *** themroc has joined #zuul | 19:36 | |
| pabelanger | clarkb: ianw: so far, centos8.1 DIB looks fine, our jobs are green again | 19:41 |
| pabelanger | (zuul.a.c) | 19:41 |
| rishabhhpe | clarkb: is there anyother way we can troubleshoot this ? | 19:42 |
| fungi | rishabhhpe: do you have access to the console log? (i.e. `openstack console log show <uuid>`) | 19:45 |
| rishabhhpe | let me check | 19:46 |
| rishabhhpe | fungi: yes but it is showing the same output which i pasted above | 19:47 |
| fungi | okay | 19:50 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: DNM: test without docker userland proxy https://review.opendev.org/702724 | 19:51 |
| rishabhhpe | fungi: so in this case what can be done to rectify it ? | 19:53 |
| fungi | rishabhhpe: by "same output which i pasted above" you mean... which paste? there were a bunch and the last time i saw you pasted anything looking like a boot log for an image using glean was http://paste.openstack.org/show/788425/ over two hours ago | 19:54 |
| fungi | still the same behavior where it ends at the really long message with glean trying to start for 5+ minutes? | 19:55 |
| rishabhhpe | yes still the same error is coming | 19:58 |
| clarkb | fungi: rishabhhpe a good next step may be to try booting one of opendev's images? we know those work and this can help isolate the problem to the cloud side or the image side | 20:00 |
| fungi | so it just says "A start job is running for Glean for interface ens3" over and over? | 20:00 |
| rishabhhpe | clarkb: how to do that ? | 20:01 |
| rishabhhpe | fungi: yes it is saying the same getting out of it and deleting the instance | 20:01 |
| clarkb | rishabhhpe: https://nb01.openstack.org/images/ubuntu-bionic-0000098242.qcow2 you can download that image, upload it to glance, then manually boot it | 20:01 |
| clarkb | rishabhhpe: you should only use that for testing as our zuul can log into it | 20:01 |
| clarkb | rishabhhpe: also you might try booting your image outside of nodepool manually | 20:02 |
| clarkb | maybe nodepool's timeouts are too sort | 20:02 |
| clarkb | *short | 20:02 |
| fungi | though 5 minutes is a long time for glean to have to wait | 20:02 |
| clarkb | fungi: ya | 20:02 |
| rishabhhpe | let me try that clarkb | 20:03 |
| *** sugaar has joined #zuul | 20:05 | |
| rishabhhpe | fungi: yes but still i am not able to figure out why it is happening | 20:05 |
| rishabhhpe | fungi: clarkb: visibility for my image is kept to private can this cause the problem ? | 20:08 |
| clarkb | corvus: pabelanger I think nodepool tag should be 3.11.0 on 9036dd7159126c8ca3df46d1ab07c7e865e350d4 (bug fixes and aws driver features as well as /ready endpoint added). Zuul should be 3.15.0 on e6d8b210cc416ed494b0b0248404e3e6d7ce337c (bug fixes and docs changes but also behavior changes around files matchers and auth rules specifications. THis could possibly be a 3.14.1) | 20:09 |
| clarkb | rishabhhpe: no that means your user can use it but not other users | 20:09 |
| rishabhhpe | ok | 20:09 |
| pabelanger | looking | 20:09 |
| pabelanger | clarkb: wfm, also find with version numbers. Either seem to work | 20:11 |
| *** jamesmcarthur has joined #zuul | 20:19 | |
| *** rishabhhpe has quit IRC | 20:22 | |
| *** wxy-xiyuan has quit IRC | 20:35 | |
| *** armstrongs has joined #zuul | 20:42 | |
| corvus | clarkb: zuul 3.15.0 lgtm | 20:50 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Re-elect James Blair as project lead https://review.opendev.org/701267 | 20:51 |
| tristanC | mnaser: it seems like the k8s service setup by install-kubernetes doesn't let deployment service able to connect to itself... are you still using a dedicated service/deployment for gearman? | 20:52 |
| *** armstrongs has quit IRC | 20:52 | |
| corvus | tristanC: can you elaborate on that question? what can't connect to itself? | 20:54 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option https://review.opendev.org/702275 | 20:58 |
| clarkb | corvus: I'm guessing its zuul talking to its own pod on port 4730 | 20:58 |
| clarkb | (weird that that wouldn't be allowed) | 20:59 |
| corvus | so if the zuul-scheduler is configured to fork a gearman, the scheduler process can't connect to it? | 20:59 |
| corvus | (presumably it's still configured to connect using the pod name and k8s dns) | 21:00 |
| tristanC | corvus: this fix attempt has the details: https://review.opendev.org/702723 , though that doesn't seems to work | 21:03 |
| tristanC | i had the exact same problem with okd v3.x, and using --userland-proxy=false solved the issue | 21:04 |
| tristanC | (and with cri-o/cni, the bridge needs something called 'hairpinMode' to be set at true) | 21:04 |
| mordred | wow | 21:05 |
| tristanC | this is for when the scheduler tries to reach the gearman service through the kube-proxy nat | 21:05 |
| corvus | tristanC: i think install-kubernetes does have a crio option, so we may need to add a conditional to 702723 if that ends up being the fix | 21:05 |
| tristanC | that could also be fixed by either providing 'server=localhost' in the scheduler zuul.conf, or by using a dedicated deployment for the gearman service | 21:05 |
| *** jamesmcarthur has quit IRC | 21:06 | |
| tristanC | i remember the later has been discussed here, but i don't remember if it was recommended or not to run the gearman service outside of the scheduler pod | 21:07 |
| pabelanger | I'm a fan of using loopback for config options, when possible | 21:07 |
| corvus | tristanC: i did not run into this problem in gke; do you think this only shows up in minikube? | 21:07 |
| pabelanger | that said, we also do external gear service for zuul.a.c, seems to work as expected. Not specific reason, other to say wanted to see how it worked | 21:08 |
| tristanC | corvus: yes, it probably only happens in mini* setup where the kube-proxy dnat seems to be confused | 21:09 |
| mordred | I think I remember mnaser saying he was doing external gearman too - but I also don't remember a specific reason | 21:09 |
| corvus | yeah, i don't think it's unreasonable to run a separate gear pod. but i'd love it if our test environment was like prod and we didn't have to make a decision based on making it work in test (when it works in prod) | 21:09 |
| *** jamesmcarthur has joined #zuul | 21:10 | |
| mordred | is it possible that option tristanC added to minikube isn't taking hold because we otherwise edit docker.conf for other reasons? | 21:10 |
| tristanC | corvus: you mean zuul.opendev doesn't have a dedicated gear service? | 21:10 |
| corvus | tristanC: it does not, but that's probably not what i meant to say. | 21:11 |
| corvus | tristanC: i mean that the helm charts i just deployed in gke are running and do not have a dedicated gear service either | 21:12 |
| corvus | tristanC: and i also meant that since this seems only to show up in minikube, i think the best fix is there -- so if 702723 works, i like it. | 21:12 |
| mordred | tristanC: why do you say it doesn't seem to work? | 21:13 |
| mordred | it and the test patch are both green? | 21:14 |
| tristanC | corvus: agreed, let me see if i can make the fix work | 21:14 |
| tristanC | mordred: the tests currently just wait 4 minutes, they don't check services are running. In that case they do, it's just the scheduler seems to be stuck waiting for gearman | 21:15 |
| *** saneax has quit IRC | 21:15 | |
| mordred | tristanC: nod | 21:15 |
| *** jamesmcarthur has quit IRC | 21:15 | |
| *** jamesmcarthur has joined #zuul | 21:15 | |
| mordred | tristanC: well - it may be off-base, but I'd check that our other roles aren't editing docker.conf in a way that's conflicting with that cmdline option - we might just have to edit docker.conf directly and restart | 21:16 |
| mordred | tristanC: but that's me waving my arms wildly | 21:16 |
| corvus | tristanC: a command like this may be helpful: kubectl wait --for=condition=Ready pod/upstream-dockertest --timeout=60s | 21:16 |
| corvus | mordred: what's docker.conf? | 21:16 |
| corvus | (we edit a lot of docker-related files, but that name isn't ringing a bell) | 21:17 |
| tristanC | corvus: thanks, that is helpful indeed :) | 21:17 |
| mordred | corvus: daemon.json is what I was thinking of I think :) | 21:18 |
| tristanC | mordred: perhaps minikube doesn't restart or set docker command line if it's already running | 21:18 |
| mordred | tristanC: yah... something like that was what I was thinking - since we do docker setup ourselves, maybe minikube just doesn't touch it enough | 21:18 |
| corvus | mordred, tristanC: use-buildset-registry will update daemon.json, and i think we run it after the minikube install | 21:19 |
| corvus | use-buildset-registry *should* non-destructively edit the config though | 21:19 |
| corvus | and it does restart docker | 21:19 |
| mordred | corvus: I agre that use-buildset-registry looks to edit non-destructively | 21:20 |
| tristanC | what if minikube doesn't set docker-opt to daemon.json... | 21:21 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: DNM: debug docker command line options https://review.opendev.org/702752 | 21:23 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: use-buildset-registry: disable docker userland proxy https://review.opendev.org/702753 | 21:25 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: DNM: test without docker userland proxy https://review.opendev.org/702724 | 21:27 |
| corvus | tristanC: reading minikube source, it looks like the --docker-opt flags are written to a systemd drop-in unit | 21:29 |
| *** igordc has joined #zuul | 21:29 | |
| corvus | tristanC: so i don't think we'd expect it to show up in daemon.json, but rather that systemd unit file (and it should show up on the docker command line) | 21:29 |
| tristanC | corvus: in that case, there may be another issue... i hope 702752 will tell us if the command line gets correctly applied | 21:30 |
| corvus | tristanC: i think it might be writing it to /lib/systemd/system/docker.service | 21:30 |
| tristanC | the dockerd command line doesn't seems to have the new opt in https://1a7fdfed9af44c951d10-48a55ed00d52551138f2a4465777f377.ssl.cf5.rackcdn.com/702752/1/check/zuul-jobs-test-install-kubernetes-docker/287dc6c/job-output.txt | 21:38 |
| *** themroc has quit IRC | 21:39 | |
| corvus | tristanC: that's very different from what i see here https://github.com/kubernetes/minikube/blob/master/pkg/provision/buildroot.go#L133 | 21:41 |
| corvus | maybe that code isn't being run | 21:41 |
| tristanC | corvus: that's also what i assumed in the initial feature pr: https://github.com/kubernetes/minikube/pull/1260/files | 21:41 |
| mordred | corvus, tristanC: in our case docker is related to containerd - is minikube still targetting and older pre-containerd version of docker? | 21:42 |
| mordred | cause I agree - that go code does not match what I see there | 21:43 |
| tristanC | alright, the daemon.json addition seems to work, here is the zuul-scheduler pod loading https://opendev.org/zuul/zuul-jobs ! https://bd7fd1da913c58e03b7d-5f33f4a8f6999785c5e66684a945b77a.ssl.cf2.rackcdn.com/702724/3/check/zuul-operator-functional-k8s/2c09428/docker/k8s_scheduler_zuul-scheduler-0_default_6d706c7f-9381-4a91-9965-fe2e679c6f25_0.txt | 21:56 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: use-buildset-registry: disable docker userland proxy https://review.opendev.org/702753 | 21:57 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Import dhall-operator https://review.opendev.org/702104 | 21:58 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Import zuul application from dhall-zuul https://review.opendev.org/702105 | 21:58 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Replace existing operator tasks with the new application https://review.opendev.org/702106 | 21:58 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Generate TLS certificats for the gearman service https://review.opendev.org/702716 | 21:58 |
| tristanC | and https://review.opendev.org/702716 should show the same thing, but using tls cert for the gearmon service | 21:59 |
| *** jamesmcarthur has quit IRC | 22:06 | |
| *** jamesmcarthur has joined #zuul | 22:16 | |
| *** jamesmcarthur has quit IRC | 22:19 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add Openshift SCC and functional test https://review.opendev.org/702758 | 22:23 |
| fungi | anybody have an opinion on whether https://review.opendev.org/701435 is how we should be handling implemented specs? | 22:33 |
| pabelanger | fungi: I have a slight preference to do what we did in infra-specs. | 22:35 |
| pabelanger | https://specs.openstack.org/openstack-infra/infra-specs/ for others | 22:36 |
| fungi | i don't mind that approach, but if we do then we need to alter some of the warnings/admonitions | 22:36 |
| pabelanger | agree | 22:36 |
| fungi | for example making the ones in the implemented specs state that the details they outline may not match what was implemented | 22:37 |
| fungi | 701435 was suggested as the simplest solution, so i figured i'd start there and get the conversation going anyway | 22:38 |
| pabelanger | Yah, I wouldn't -1 it | 22:38 |
| fungi | that way if we do something more complicated, we have a clear explanation of why we didn't do the simple thing | 22:38 |
| pabelanger | if there was another option, I kinda like how we did it with infra-specs | 22:38 |
| fungi | please leave a comment on the change to that effect. i feel like that's a bit of additional process, so would want the benefits stated (in zuul we've tended to shy away from process for the sake of process, in favor of simple solutions) | 22:41 |
| pabelanger | ack | 22:42 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add Openshift SCC and functional test https://review.opendev.org/702758 | 22:46 |
| *** igordc has quit IRC | 23:09 | |
| *** igordc has joined #zuul | 23:10 | |
| clarkb | corvus: re releases were you indicating I should make them or were you going to do that? | 23:27 |
| *** tdasilva has joined #zuul | 23:37 | |
| *** tobiash has quit IRC | 23:38 | |
| *** tobiash has joined #zuul | 23:39 | |
| corvus | clarkb: i'm happy either way -- i'm about to eod, so if you want to go ahead, that wfm. or i could do it tomorrow. | 23:40 |
| clarkb | I probably won't get to it today either | 23:40 |
| corvus | clarkb: cool, we'll armwrestle tomorrow then | 23:40 |
| clarkb | kk | 23:41 |
| corvus | fungi: +2 but i'll refrain from a +W right now; let's get some more feedback on that. zuul-maint please see https://review.opendev.org/701435 | 23:43 |
| openstackgerrit | Merged zuul/zuul master: Re-elect James Blair as project lead https://review.opendev.org/701267 | 23:45 |
| fungi | corvus: yup, i'm in no hurry to merge it, that's about how we all get work done so the more input the better | 23:46 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!