| openstackgerrit | Merged zuul/nodepool master: Add nodepool_debug flag to openstack functional jobs https://review.opendev.org/669939 | 00:15 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] functional testing: test journal-to-console element https://review.opendev.org/669787 | 00:21 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Enable debug logs for openstack-functional tests https://review.opendev.org/672412 | 00:23 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] functional testing: test journal-to-console element https://review.opendev.org/669787 | 00:23 |
| *** igordc has quit IRC | 01:04 | |
| *** wxy-xiyuan has joined #zuul | 01:11 | |
| corvus | mordred: i think the solution in https://review.opendev.org/672606 will be "good enough", but maybe the subquery idea would pay off for the job_name case? | 01:34 |
| *** bjackman has joined #zuul | 02:47 | |
| *** bhavikdbavishi has joined #zuul | 02:51 | |
| *** bhavikdbavishi1 has joined #zuul | 02:54 | |
| *** bhavikdbavishi has quit IRC | 02:55 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:55 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Functional testing: add journal-to-console element https://review.opendev.org/669787 | 03:35 |
| *** yolanda has quit IRC | 04:21 | |
| *** yolanda has joined #zuul | 04:22 | |
| *** jank has joined #zuul | 04:39 | |
| *** pcaruana has joined #zuul | 04:44 | |
| *** pcaruana has quit IRC | 04:56 | |
| *** swest has joined #zuul | 05:05 | |
| *** pcaruana has joined #zuul | 06:21 | |
| yoctozepto | corvus, Shrews, mordred: thanks for handling, I went to bed and failed to say goodbye :-( | 06:37 |
| *** rlandy has joined #zuul | 06:46 | |
| *** jpena|off is now known as jpena | 06:52 | |
| *** jpena is now known as jpena|mtg | 06:53 | |
| -openstackstatus- NOTICE: The git service on opendev.org is currently down. | 06:53 | |
| *** ChanServ changes topic to "The git service on opendev.org is currently down." | 06:53 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 06:56 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Add Authorization Rules configuration https://review.opendev.org/639855 | 06:58 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Web: plug the authorization engine https://review.opendev.org/640884 | 06:59 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.opendev.org/641099 | 06:59 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: authentication config: add optional token_expiry https://review.opendev.org/642408 | 06:59 |
| *** jamesmcarthur has joined #zuul | 07:04 | |
| *** rlandy is now known as rlandy|mtg | 07:07 | |
| flaper87 | what zuul services need to be restarted when the tenant definition is changed? Just web? | 07:20 |
| flaper87 | https://github.com/openstack/project-config/blob/master/zuul/main.yaml <- this file here | 07:20 |
| *** jank has quit IRC | 07:21 | |
| flaper87 | mmh, apparently the scheduler | 07:27 |
| *** jank has joined #zuul | 07:50 | |
| *** jank has quit IRC | 07:53 | |
| *** panda has quit IRC | 08:28 | |
| *** tosky has joined #zuul | 08:30 | |
| *** panda has joined #zuul | 08:31 | |
| -openstackstatus- NOTICE: Services at opendev.org like our git server and at openstack.org are currently down, looks like an outage in one of our cloud providers. | 08:36 | |
| *** ChanServ changes topic to "Services at opendev.org like our git server and at openstack.org are currently down, looks like an outage in one of our cloud providers." | 08:36 | |
| *** sshnaidm has quit IRC | 08:38 | |
| *** sshnaidm has joined #zuul | 08:42 | |
| *** ChanServ changes topic to "Discussion of the project gating system Zuul | Website: https://zuul-ci.org/ | Docs: https://zuul-ci.org/docs/ | Source: https://git.zuul-ci.org/ | Channel logs: http://eavesdrop.openstack.org/irclogs/%23zuul/ | Weekly updates: https://etherpad.openstack.org/p/zuul-update-email" | 08:43 | |
| -openstackstatus- NOTICE: The problem in our cloud provider has been fixed, services should be working again | 08:43 | |
| *** jamesmcarthur has quit IRC | 08:48 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Return dependency cycle failure to user https://review.opendev.org/672487 | 09:12 |
| *** swest has quit IRC | 09:13 | |
| *** swest has joined #zuul | 09:14 | |
| *** lennyb has joined #zuul | 09:16 | |
| *** hashar has joined #zuul | 09:48 | |
| *** hwangbo has quit IRC | 09:50 | |
| *** bhavikdbavishi has quit IRC | 09:52 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Fix reference pipelines syntax coloration for Pagure driver https://review.opendev.org/672677 | 09:54 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Add reference pipelines file for Gerrit driver https://review.opendev.org/672683 | 10:12 |
| *** hashar has quit IRC | 10:15 | |
| *** AshBullock has joined #zuul | 10:23 | |
| AshBullock | Hey guys, in the openshift example there is a role variable called openshift_pods, is there a similar variable exposed when using the kubernetes driver? As we need a var that has the spun up pod names so we can run kubectl thanks | 10:45 |
| AshBullock | this is the openshift example https://www.softwarefactory-project.io/tech-preview-using-openshift-as-a-resource-provider.html | 10:45 |
| AshBullock | and the prepare workspace role I was looking at https://review.opendev.org/#/c/631402/3/roles/prepare-workspace-openshift/tasks/main.yaml | 10:46 |
| *** hashar has joined #zuul | 11:32 | |
| *** pcaruana has quit IRC | 11:42 | |
| *** bhavikdbavishi has joined #zuul | 11:42 | |
| *** igordc has joined #zuul | 11:43 | |
| sshnaidm | is there a correct way to inherit job from two parents? | 11:52 |
| tristanC | sshnaidm: yes, you can define two variants with different parent | 12:19 |
| sshnaidm | tristanC, do you have any example how to do it? | 12:20 |
| tristanC | sshnaidm: like so: http://paste.openstack.org/show/754848/ | 12:20 |
| sshnaidm | tristanC, cool, thanks | 12:20 |
| *** pcaruana has joined #zuul | 12:22 | |
| tristanC | AshBullock: can't find openshift_pods in the blog post, what do you need it for? | 12:22 |
| sshnaidm | tristanC, do you know if cleanup is already available in o.o and rdo zuuls? https://review.opendev.org/#/c/662147/3/doc/source/user/config.rst | 12:24 |
| *** hashar has quit IRC | 12:25 | |
| tristanC | sshnaidm: looking at http://zuul.opendev.org/t/zuul/status, at the bottom it says the commit short sha that the scheduler is running | 12:27 |
| tristanC | sshnaidm: which should includes the cleanup phases | 12:27 |
| tristanC | sshnaidm: rdo is running tagged version, we would need a new zuul version | 12:28 |
| *** hashar has joined #zuul | 12:28 | |
| AshBullock | tristanC got a little further now, have the pod name and trying to run an exec command to create a directory in the pod, we are receiving this error: "main -> localhost | error: You must be logged in to the server (Unauthorized)", we have the task running localhost, here is our config: http://paste.openstack.org/show/754849/ | 12:28 |
| AshBullock | any idea what I'm doing wrong? | 12:29 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Improve SQL query performance in some cases https://review.opendev.org/672606 | 12:31 |
| tristanC | AshBullock: is this using eks? and is your nodepool and zuul services from outside the cluster? | 12:33 |
| AshBullock | yes and yes | 12:34 |
| AshBullock | it looks like it is not correctly assuming the zuul-worker role | 12:34 |
| AshBullock | so is throwing a non authorised | 12:35 |
| AshBullock | we can see the role and role bindings created | 12:35 |
| tristanC | AshBullock: perhaps the service account and token provided by nodepool needs some extra configuration to be usable by zuul with eks | 12:36 |
| AshBullock | where is that config set? | 12:36 |
| tristanC | iirc eks enforces special auth requirement for external api call | 12:36 |
| tristanC | AshBullock: the namespace is configured like so: https://opendev.org/zuul/nodepool/src/branch/master/nodepool/driver/kubernetes/provider.py#L155 | 12:37 |
| AshBullock | we are able to create the namespace and pod, it just seems we are unable to run the exec command | 12:37 |
| AshBullock | so it must be able to run commands against the cluster | 12:37 |
| AshBullock | what we're seeing is from a playbook when it assumes the zuul-worker user it is getting a permissions error | 12:38 |
| tristanC | AshBullock: nodepool and zuul doesn't use the same auth, so there may an issue with the zuul's token | 12:38 |
| tristanC | AshBullock: here is how we configure nodepool for our openshift integration test: https://softwarefactory-project.io/cgit/software-factory/sf-ci/tree/roles/health-check/openshift/tasks/config_repo_nodepool_configuration.yml | 12:39 |
| tristanC | AshBullock: and here is how we test zuul job running in pods: https://softwarefactory-project.io/cgit/software-factory/sf-ci/tree/roles/health-check/openshift/tasks/demo_project_zuul_configuration.yml | 12:39 |
| AshBullock | where in the code does the zuul worker token get set? | 12:40 |
| AshBullock | the secondary kube config that zuul uses ? | 12:40 |
| AshBullock | for the zuul-worker user | 12:40 |
| AshBullock | as our nodepool kube config has the correct aws auth command but I guess the zuul one does not | 12:41 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Add reference pipelines file for Github driver https://review.opendev.org/672712 | 12:41 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Add change replacement field in doc for start-message https://review.opendev.org/665974 | 12:44 |
| *** bjackman has quit IRC | 13:00 | |
| *** bhavikdbavishi has quit IRC | 13:07 | |
| AshBullock | I'm thinking I need to add the get-token logic that eks requires to the zuul-worker kube config, something like "aws eks get-token --cluster-name zuul-eks --region eu-west-1" to generate a token | 13:09 |
| *** bhavikdbavishi has joined #zuul | 13:10 | |
| *** jhesketh has quit IRC | 13:22 | |
| *** jhesketh has joined #zuul | 13:26 | |
| flaper87 | what zuul services need to be restarted when the tenant definition is changed? Just scheduler? Is there a way to reload the tenant configs without restarting the service? | 13:31 |
| pabelanger | flaper87: yup, I believe it is zuul-scheduler reload | 13:33 |
| pabelanger | or kill -HIP pid | 13:33 |
| pabelanger | kill -HUP | 13:33 |
| fungi | yeah, no restart needed | 13:44 |
| fungi | it can just reload the config while remaining up | 13:44 |
| fungi | restarts should only be necessary to replace zuul software or its python dependencies | 13:45 |
| mhu | Hi! In a multi-executor setup, is it possible to "tie" nodes to an executor? The use case would be running jobs on baremetal instances living in different geographical zones | 13:57 |
| AshBullock | i've tracked down the .kube config file in the zuuls working directory, this has a token set, but we need to pass in the aws token generation command, we've tried templating over it but the file seems to be protected, is there a way to set kube configuration to override the default? | 13:57 |
| mhu | so you want the executor closest to a given node to run the playbook | 13:57 |
| tristanC | mhu: iirc, in nodepol we can set an executor-zone to nodepool node, and then we can configure a zuul executor service to run job for that zone | 13:59 |
| tristanC | AshBullock: here is how the zuul executor service writes down the kube config: https://opendev.org/zuul/zuul/src/branch/master/zuul/executor/server.py#L1692 | 14:00 |
| mhu | tristanC, thanks! | 14:00 |
| tristanC | corvus: running the zuul-operator test playbooks with openshift, the operator pod starts successfully: http://paste.openstack.org/show/754853/ | 14:09 |
| tristanC | corvus: makes me wonder if the install-kubernetes correctly setup the operator framework... Could I start an openshift based integration job? | 14:10 |
| corvus | tristanC: yeah, but i think we should have both, so i think we need to get the k8s one working | 14:20 |
| corvus | tristanC: jeliu was looking into it and said it looked like it was running locally | 14:20 |
| corvus | tristanC: but what do you mean "install-kubernetes correctly setup the operator framework" ? | 14:21 |
| tristanC | corvus: looking at the pod logs in ci: http://logs.openstack.org/76/672576/1/check/zuul-operator-functional-k8s/2d0c442/ara-report/result/946eb628-8523-4d9b-b38b-d75267586010/ | 14:22 |
| tristanC | corvus: it looks like something is missing, logs shouldn't contains an usage output | 14:22 |
| tristanC | it does says "Watches established.", but then it fails to go into operator mode (e.g. select a leader, "starting to serve" | 14:23 |
| corvus | tristanC: agreed -- | 14:23 |
| corvus | tristanC: agreed -- but also, it looks like maybe the output from the 2 containers are overwriting there? | 14:24 |
| corvus | tristanC: because the first lines of the usage output are missing | 14:24 |
| tristanC | corvus: the second container (named ansible) should be silent as watch are triggered by crd yet | 14:24 |
| corvus | tristanC: check this out: http://logs.openstack.org/95/670395/7/check/zuul-operator-functional-k8s/2d74a66/ara-report/ | 14:24 |
| corvus | tristanC: jeliu split that up so we have the ouput from each container separately | 14:25 |
| corvus | tristanC: so the operator container emitted help text, and the ansible container emitted logs about 'watches' | 14:26 |
| corvus | tristanC: what made the watches output? | 14:26 |
| tristanC | corvus: i see, well with openshift it seems to work out of the box: http://paste.openstack.org/show/754853/ | 14:26 |
| tristanC | i guess we'll want both k8s & openshift integration test, so i could write one quickly to at least get the expected output in ci | 14:27 |
| corvus | tristanC: yep, that might help us get the k8s one working | 14:27 |
| corvus | i also held a node if we want to look into it | 14:27 |
| corvus | tristanC: i can add your key if you want | 14:28 |
| corvus | i didn't have time to look yesterday :| | 14:28 |
| corvus | tristanC: what does the ansible container do? | 14:29 |
| corvus | (i was expecting only one container (the operator container) and for it to run a daemon which responded to k8s crd changes) | 14:30 |
| tristanC | corvus: the ansible container is running ansible-runner and you'll have the task json output logs from it | 14:31 |
| tristanC | corvus: the operator container are info logs about the operator process, *iirc* | 14:31 |
| *** jeliu_ has joined #zuul | 14:32 | |
| corvus | jeliu_: good morning! tristanC and i were just talking about the operator | 14:32 |
| tristanC | jeliu_: here is what i got using a local oc cluster up setup: http://paste.openstack.org/show/754853/ | 14:33 |
| corvus | jeliu_: you can catch up on what we were talking about here: http://eavesdrop.openstack.org/irclogs/%23zuul/%23zuul.2019-07-25.log | 14:34 |
| corvus | tristanC, jeliu_: that's still really weird that http://logs.openstack.org/95/670395/7/check/zuul-operator-functional-k8s/2d74a66/ara-report/result/1859a953-bbc8-4afe-901a-64723b115dea/ is missing some lines of output | 14:39 |
| tristanC | corvus: it maybe that the command doesn't display an header and goes straight to cli options | 14:40 |
| tristanC | perhaps a badly designed failure mode... | 14:41 |
| corvus | tristanC: well, when jeliu_ ran it on his local k8s, he got something like this: https://etherpad.openstack.org/p/zuul_commands | 14:41 |
| corvus | tristanC: i guess it could be in the failure mode we're seeing, there's no error or header, but that would be weird too | 14:42 |
| tristanC | jeliu_: how did you deploy the local k8s? | 14:42 |
| corvus | tristanC: aiui, he's done it both via a local build of k8s, and then also with minikube | 14:48 |
| corvus | if i open up a shell on the ansible container and try to run the operator, i get: Error: Get https://10.96.0.1:443/api?timeout=32s: dial tcp 10.96.0.1:443: connect: no route to host | 14:55 |
| corvus | (and then the full operator help output) | 14:55 |
| *** dkehn has joined #zuul | 15:00 | |
| *** dkehn has left #zuul | 15:01 | |
| *** dkehn has joined #zuul | 15:01 | |
| corvus | tristanC, jeliu_: what if there's a firewall issue on our test nodes? | 15:05 |
| *** rlandy|mtg has quit IRC | 15:05 | |
| *** jpena|mtg is now known as jpena|off | 15:07 | |
| corvus | clarkb, mordred, fungi: ^ do you remember that weird docker firewall issue we ran into? | 15:07 |
| clarkb | it was ipv6 specific I think (since docker doesnt manage ipv6 by default?) | 15:08 |
| fungi | also docker wants to manage your firwewall unless you explicitly tell it hands off, right? | 15:08 |
| corvus | oh i thought it was something about how docker set the default policy of one of the chains? | 15:08 |
| corvus | it sets forward to drop | 15:09 |
| corvus | our firewall rules are still in place on the input chain, but it's last; so i'm not sure if it's an issue or not | 15:10 |
| corvus | there's no telnet in the zuul-operator image; we should add that :) | 15:11 |
| jeliu_ | corvus: I can help with that! | 15:17 |
| openstackgerrit | James E. Blair proposed zuul/zuul-operator master: WIP: test operator / iptables https://review.opendev.org/672755 | 15:17 |
| corvus | jeliu_: thanks! | 15:17 |
| corvus | clarkb, fungi: ^ do you think "/etc/init.d/iptables-persistent stop" will be enough to remove our iptables rules as a factor? | 15:18 |
| clarkb | corvus: I cant remember if that does a drop of the rulesit knows about | 15:19 |
| fungi | that file doesn't seem to exist on bionic | 15:19 |
| fungi | oh, right, /etc/init.d/netfilter-persistent | 15:19 |
| corvus | it's a xenial node | 15:19 |
| clarkb | you may need to edit the rules files and start to load an emptier ruleset (then restart dockerd so it replaces ots rules) | 15:19 |
| corvus | i put that in before the docker install | 15:20 |
| fungi | runs `/usr/sbin/netfilter-persistent stop` to flush rules | 15:20 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/672756 | 15:21 |
| fungi | if you follow the execution path through the plugin scripts, it basically does `/sbin/iptables -P $chain ACCEPT` for chains INPUT FORWARD OUTPUT | 15:21 |
| fungi | so essentially opens the rulset up wide | 15:22 |
| fungi | that ought to work | 15:22 |
| fungi | though i was looking at the bionic version. probably similar on xenial just simpler because that predates the netfilter plugins mechanism | 15:22 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/672756 | 15:25 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Assure ensure-tox installs latest tox version https://review.opendev.org/672760 | 15:39 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Improve SQL query performance in some cases https://review.opendev.org/672606 | 15:39 |
| AJaeger | corvus: https://review.opendev.org/#/c/670133/ for zuul-jobs (skipping test-setup.sh in pep8) is now 14days old. Want to merge it and followup on zuul-announce? | 15:43 |
| corvus | AJaeger: done. took me a minute to double check everything :) | 15:45 |
| AJaeger | thanks | 15:47 |
| *** altlogbot_2 has quit IRC | 15:48 | |
| corvus | mordred, tristanC, jeliu_: any more thoughts on the spec https://review.opendev.org/659180 ? are you ready for me to start poking other folks to review it? | 15:50 |
| *** altlogbot_3 has joined #zuul | 15:51 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-operator master: WIP: test operator / iptables https://review.opendev.org/672755 | 15:53 |
| mordred | corvus: I think it's ready for poking | 15:53 |
| openstackgerrit | Merged zuul/zuul-jobs master: Skip test-setup.sh in pep8 jobs https://review.opendev.org/670133 | 15:57 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Assure ensure-tox installs latest tox version https://review.opendev.org/672760 | 15:58 |
| *** hwangbo has joined #zuul | 16:09 | |
| corvus | fungi, clarkb: hrm that didn't work: http://logs.openstack.org/55/672755/2/check/zuul-operator-functional-k8s/471465a/ara-report/result/39faf412-62d6-4ea2-9959-3e626d679204/ | 16:09 |
| corvus | http://logs.openstack.org/55/672755/2/check/zuul-operator-functional-k8s/471465a/ara-report/result/49ee5cf7-856c-4d65-858c-dc1f1a141148/ | 16:10 |
| fungi | that's after flushing via iptables-persistent stop? | 16:10 |
| corvus | yeah -- second link is the stop command | 16:11 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Allow ensure-tox to upgrade tox version https://review.opendev.org/672760 | 16:12 |
| fungi | and the iptables -L is a couple seconds later, yeah | 16:12 |
| openstackgerrit | James E. Blair proposed zuul/zuul-operator master: WIP: test operator / iptables https://review.opendev.org/672755 | 16:13 |
| corvus | brute force ^ | 16:13 |
| *** mattw4 has joined #zuul | 16:14 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/672756 | 16:16 |
| tristanC | corvus: the spec lgtm | 16:16 |
| corvus | SpamapS, tobiash, flaper87: can you look over https://review.opendev.org/659180 soonish? i think it's close to ready to merge | 16:17 |
| *** mattw4 has quit IRC | 16:23 | |
| *** mattw4 has joined #zuul | 16:23 | |
| flaper87 | corvus: well re-review tomorrow! Hope to make it in time before it merges | 16:25 |
| flaper87 | :D | 16:25 |
| corvus | tristanC, jeliu_: it looks like it was the firewall: http://logs.openstack.org/55/672755/3/check/zuul-operator-functional-k8s/54283ff/ara-report/result/d1174d83-4096-456b-86ff-2dc4da5c27f4/ | 16:34 |
| corvus | jeliu_: if you want to incorporate https://review.opendev.org/672755 into your change, i think that's good for now | 16:36 |
| mordred | corvus: awesome | 16:36 |
| corvus | we should probably add a role to zuul-jobs to clear out the firewall | 16:36 |
| corvus | or, maybe talk with the opendev sysadmins about maybe possibly dropping the firewall from test nodes... | 16:36 |
| *** AshBullock has quit IRC | 16:36 | |
| mordred | corvus: want me to extract that and make one? I'm at a lull where I could do that | 16:37 |
| corvus | mordred: yeah, that'd be great; i have to afk for few | 16:37 |
| mordred | kk | 16:37 |
| fungi | the risk with not firewalling by default is we have folks installing things like open recursive dns resolvers and elasticsearch api servers in their jobs, which get picked up by miscreants sweeping the internet looking for exploitable systems and then used in coordinated ddos attacks all before the builds conclude | 16:39 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: install-openshift: bump version to 3.11.0 https://review.opendev.org/672785 | 16:40 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add clear-firewall role https://review.opendev.org/672786 | 16:41 |
| fungi | (note this is not a theoretical risk, i've fielded a number of complaints from our providers in the past because folks running complex jobs took it upon themselves to turn off the firewalling in them since it was too hard to troubleshoot) | 16:41 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/672756 | 16:41 |
| fungi | (in particular, it was deployment projects trying to do things with containers: kolla, openstack-helm, tripleo... i guess there's a common reason) | 16:43 |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: DNM: test openshift version bump https://review.opendev.org/672788 | 16:43 |
| Shrews | corvus: you don't need a separate index for just uuid in https://review.opendev.org/672606 | 16:46 |
| * mordred agrees with Shrews | 16:47 | |
| Shrews | mordred: see? i haven't forgotten *everything* | 16:47 |
| Shrews | :) | 16:47 |
| Shrews | as much as tried to | 16:48 |
| Shrews | corvus: it's because uuid is the _first_ column used in the index. if it were the buildset_id, that would be different | 16:49 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/672756 | 16:50 |
| *** chandankumar is now known as raukadah | 16:52 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Add telnet to Docker Image https://review.opendev.org/672791 | 16:53 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Add telnet to Docker Image https://review.opendev.org/672791 | 16:56 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: install-openshift: bump version to 3.11.0 https://review.opendev.org/672785 | 16:57 |
| *** igordc has quit IRC | 16:58 | |
| *** igordc has joined #zuul | 16:58 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: DNM: test openshift version bump https://review.opendev.org/672788 | 16:58 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/672756 | 17:10 |
| clarkb | fungi: ya I would not be comfortable with removing the firewall | 17:11 |
| clarkb | I think instead we need to figure out how to make the fireawll ignore docker/k8s | 17:11 |
| clarkb | (perhaps with a local only addr that is open?) | 17:12 |
| clarkb | well docker is supposed to do it for you | 17:13 |
| clarkb | not sure about k8s | 17:13 |
| clarkb | corvus: out of curiousity why "clear out the firwall" and not "update firwall so it works with k8s"? | 17:15 |
| corvus | clarkb, fungi: okay, we can just do the clear firewall thing in these jobs for now. my suggestion re opendev was mostly in service of aligning our testing platform with vanilla installations. but i'm not going to defend that hill. :) | 17:15 |
| corvus | clarkb: if you have an idea of how to do that, great -- we can even test that it work now. | 17:16 |
| corvus | clarkb: but from my pov, minikube+k8s want to run the firewall, so i want to get out of their way and let them | 17:16 |
| corvus | clarkb: it's more or less the same conclusion we came to when we looked at running k8s in opendev | 17:16 |
| clarkb | corvus: (this is from memory several months old so results may vary) I believe the issue is that k8s has internal network ranges that we'll block by default. Much like multinode testing we want t add rules that say those ranges can talk to that range on all ports | 17:17 |
| corvus | (but s/ansible-openstack/minikube/) | 17:17 |
| clarkb | I don't know what ranges minikube uses, but if we can identify those and allow all of them to talk to each other I expect it will be happier | 17:17 |
| corvus | clarkb: it's beyond my understanding. my feeling is that clearing out the firewall is okay in this instance (it's being replaced by a sufficiently robust firewall). and that's even desirable (because it's good to have the testing infra be similar to prod, and we don't want to have to eliminate the firewall as a cause of every problem we have). so even if that approach would work, i'm still not sure it's | 17:21 |
| corvus | what we want in the long run. | 17:21 |
| corvus | Shrews, mordred: replied | 17:22 |
| mordred | Shrews: we aren't smart | 17:22 |
| clarkb | it is too bad it isn't easier to identify what is "external" networking in a cloud sense as we could change the ruleset to say "don't let any of that in except for ssh" and freely allow everything else to talk | 17:22 |
| corvus | yeah, i mean, i'm not happy about the way docker/k8s interact with the host :) | 17:23 |
| clarkb | Another option is to lean more heavily on security groups | 17:24 |
| clarkb | supposedly they have vastly improved the performance of how those rules are applied so I won't get 3am phone calls anymore | 17:24 |
| clarkb | That is the proper way to express "don't let external in" in the openstack clodu context I think | 17:24 |
| clarkb | But historically it caused the cloud to break when we booted hundreds of nodes an hour | 17:25 |
| corvus | clarkb: i'm confused -- if you're concerned about people abusing our test nodes -- we give them root. | 17:25 |
| clarkb | re security groups? | 17:26 |
| corvus | yeah | 17:26 |
| clarkb | the issue was it thrashed the mysql database causing it to stop being performant which meant none of the cloud apis functioned and I got a phone call from the cloud operator saying please make it stop | 17:26 |
| corvus | okay this doesn't sound like a #zuul convo | 17:26 |
| clarkb | well its realted to zuul in that that is one way we could potentially remove the on host firewalls as you suggested here | 17:26 |
| corvus | how does security groups help us remove the on-host firewall? | 17:27 |
| clarkb | we would use the security groups to prevent open dns resolvers on the internet and remove the on host firwalls | 17:27 |
| clarkb | basically use security groups to block that traffic instead of iptables | 17:27 |
| corvus | what do open dns resolvers have to do with this? | 17:27 |
| clarkb | corvus: see fungi's concern | 17:27 |
| *** igordc has quit IRC | 17:28 | |
| clarkb | 16:39:04 fungi | the risk with not firewalling by default is we have folks installing things like open recursive dns resolvers and elasticsearch api servers in their jobs | 17:28 |
| corvus | yeah i see that | 17:28 |
| fungi | i think wiping the firewall off the test node is fine in this case, as long as people are mindful that there's not necessarily any preexisting protection in place any longer and it's up to them to secure what they're installing in jobs | 17:28 |
| corvus | do you think i just wrote a job that installes an open dns resolver? | 17:28 |
| clarkb | corvus: no but other people do? | 17:28 |
| corvus | okay, we're going to have to talk about this later | 17:28 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/672756 | 17:30 |
| fungi | with my opendev sysadmin hat on, we provide this mitigation by default but we don't really have a means of preventing jobs from turning it off. disabling that protection isn't something which should be taken lightly, but there are plenty of other ways for folks to cause similar problems with jobs | 17:30 |
| clarkb | fungi: yup. My suggestion is we can replace the host firewalls globally with security groups, then we can remove the iptables rules across the board as corvus suggested | 17:31 |
| fungi | replacing the firewall which is there with a different firewall seems reasonable, which is what is being proposed in this particular job | 17:31 |
| clarkb | Then jobs don't have to worry about the firwall and we still mitigate the risks | 17:31 |
| Shrews | mordred: corvus: lol! | 17:31 |
| fungi | discussion of security groups in the opendev donor environments is a much bigger topic, and better suited to #openstack-infra or #opendev | 17:32 |
| Shrews | corvus: sorry. i made assumptions based on the tests you had me perform last night. so this is really your fault if you think about it :-P | 17:33 |
| webknjaz | Hey folks! I recall you do automatic releases to PyPI. Now you can have a proper project-scoped upload token instead of putting a password there. | 17:47 |
| webknjaz | Ref: https://discuss.python.org/t/pypi-security-work-multifactor-auth-progress-help-needed/1042/31 | 17:47 |
| webknjaz | N.B. If you have 2FA enabled on PyPI, eventually you'll be forced to switch to using tokens. | 17:47 |
| fungi | webknjaz: how does that work with unattended twine upload? | 17:47 |
| fungi | webknjaz: ahh, i see, thanks | 17:48 |
| webknjaz | I haven't tested it yet, but I assume that you'll need to replace password with that | 17:48 |
| webknjaz | lemme check | 17:49 |
| fungi | so we need a human to generate a token for uploading... i still don't see where it's more secure than having a dedicated account for uploading but maybe this is useful to people who use the same account for webui on warehouse too | 17:49 |
| fungi | looks like the announcement about that just hit the pypa-dev ml as well | 17:52 |
| tristanC | corvus: there is the same issue with openshift, the opendev default firewall prevent access to 172.30.1.1:5000 (the internal registry) | 17:54 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/672756 | 17:54 |
| webknjaz | fungi: just one acc less to maintain.. | 17:56 |
| clarkb | webknjaz: well I don't think we'd use a personal accoutn for zuul automated uploads | 17:57 |
| fungi | right, we'll use a dedicated account anyway | 17:58 |
| fungi | it'll just be an extra step now to use that account to generate a perpetual api token | 17:58 |
| webknjaz | FWIW: use `@token` for a username and the token itself for a password when using Twine: https://github.com/pypa/warehouse/issues/994#issuecomment-512634222 | 17:58 |
| *** jeliu_ has quit IRC | 17:58 | |
| fungi | we'll want to test whether that's compatible with how privileges are delegated, since we expect to use one set of credentials to upload for multiple projects, including projects which don't exist yet or need access delegated at a later time | 17:59 |
| fungi | anyway, the announcement doesn't indicate there's any breaking change on the way for how the job is currently defined | 18:01 |
| fungi | as long as two-factor authentication isn't enabled for the account used, the indication is that it will continue to be able to upload via username+password with such an account | 18:02 |
| fungi | only accounts with two-factor authentication enabled are hinted at being forced to use api tokens to perform uploads in the future | 18:03 |
| *** jeliu_ has joined #zuul | 18:03 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Allow ensure-tox to upgrade tox version https://review.opendev.org/672760 | 18:03 |
| clarkb | fungi: and I'm guessing that is beacuse twine doesn't know how to 2fa so you ahve to 2fa to get a limited time/scope token | 18:03 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/672756 | 18:05 |
| fungi | twine could be adapted to do interactive 2fa fairly easily. the bigger challenge is distutils, which is baked into python's stdlib, so a backward-compatible solution is needed there regardless | 18:05 |
| clarkb | ah | 18:05 |
| *** mattw4 has quit IRC | 18:11 | |
| *** mattw4 has joined #zuul | 18:11 | |
| *** jamesmcarthur has joined #zuul | 18:17 | |
| openstackgerrit | Merged zuul/zuul master: Improve SQL query performance in some cases https://review.opendev.org/672606 | 18:18 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/672756 | 18:21 |
| *** igordc has joined #zuul | 18:22 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Allow ensure-tox to upgrade tox version https://review.opendev.org/672760 | 18:30 |
| webknjaz | fungi: I'm pretty sure nobody will ever try to change distutils there. AFAIR it's scheduled to be removed from stdlib. | 18:37 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Allow ensure-tox to upgrade tox version https://review.opendev.org/672760 | 18:42 |
| fungi | webknjaz: yep, in... like... 3.9 | 18:43 |
| fungi | but older interpreters//stdlib will be in use for many, many years to come | 18:43 |
| webknjaz | `setup.py upload` is highly discouraged anyway so no need to care about it | 18:44 |
| webknjaz | not even sure whether still works | 18:44 |
| webknjaz | *it | 18:44 |
| clarkb | yup, we are sort of why twine exists :) | 18:44 |
| fungi | it's discouraged, but still supported (which is why the announcement mentions using api tokens with distutils) | 18:45 |
| clarkb | dstufft found out we were using curl instead of setup.py upload (because setup.py upload is scary for a few reasons) and that was the genesis of twine | 18:45 |
| *** fdegir has quit IRC | 18:45 | |
| webknjaz | ah | 18:46 |
| *** fdegir has joined #zuul | 18:46 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/672756 | 18:51 |
| mordred | setup.py is scary for many reasons | 18:54 |
| mordred | regardless of which subcommand it runs | 18:54 |
| * mordred quickly gets back off soapbox | 18:55 | |
| *** jeliu_ has quit IRC | 19:06 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: [WIP] Verify Operator Pod Running https://review.opendev.org/670395 | 19:08 |
| *** igordc has quit IRC | 19:09 | |
| *** jeliu_ has joined #zuul | 19:10 | |
| *** tosky has quit IRC | 19:10 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Allow ensure-tox to upgrade tox version https://review.opendev.org/672760 | 19:13 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/672756 | 19:15 |
| *** bhavikdbavishi has quit IRC | 19:18 | |
| *** jeliu_ has quit IRC | 19:23 | |
| *** igordc has joined #zuul | 19:25 | |
| *** igordc has quit IRC | 19:32 | |
| *** jamesmcarthur has quit IRC | 19:40 | |
| *** jamesmcarthur has joined #zuul | 19:41 | |
| *** armstrongs has joined #zuul | 19:43 | |
| *** jamesmcarthur has quit IRC | 19:46 | |
| armstrongs | Hey, we got eks up and running with zuul today. We had to patch the executor server.py to do this amoung some other things we were keen to document the steps for this as other users will be keen on this too. What would be the best way to do this? | 19:47 |
| fungi | armstrongs: a commit (or commits) with your patches pushed up for review would be a great start | 19:56 |
| fungi | that should hopefully simplify the instructions | 19:56 |
| armstrongs | Cool was also wanting to document the kubernetes pre stuff a little more as a new page. So was wondering where that should sit. As we had to ask a tonne of questions and pester you guys. So was thinking could do an end to end guide for newbies | 20:00 |
| fungi | if we add a third guide for newcomers (in addition to the quick-start and from-scratch guides) we'll likely need some way to disabmiguate them | 20:01 |
| *** michael-beaver has joined #zuul | 20:02 | |
| fungi | would this fit as an alternate ending for from-scratch? | 20:02 |
| armstrongs | This wasn't to replace them more a kubernetes driver section to supplement them | 20:02 |
| armstrongs | So I think there was good coverage on the from scratch with static driver. But I had to ask questions on the nodepool aws and kubernetes ones. So thought could do those maybe as part of the from scratch with links? | 20:05 |
| clarkb | maybe a mini guide for each nodepool driver as the step beyond quickstart | 20:06 |
| clarkb | "go here if using openstack, here if using k8s, etc" | 20:06 |
| armstrongs | Yup was thinking that | 20:07 |
| tristanC | perhaps in the zuul/doc/source/user/howtos/ section? | 20:07 |
| Shrews | we have the quickstart guide already divided up based on nodepool driver. seems logical to at least add *something* there | 20:07 |
| Shrews | i.e. https://zuul-ci.org/docs/zuul/admin/zuul-from-scratch.html#nodepool | 20:08 |
| *** igordc has joined #zuul | 20:08 | |
| Shrews | but i could also see something going into nodepool itself, too | 20:08 |
| armstrongs | Yeah was thinking 2 more links there | 20:09 |
| *** jamesmcarthur has joined #zuul | 20:11 | |
| Shrews | armstrongs: that makes the most sense to my brain. perhaps go with that and let's see how it looks? | 20:12 |
| armstrongs | Thanks will do | 20:12 |
| Shrews | armstrongs: thx for the offer to do it | 20:12 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add clear-firewall role https://review.opendev.org/672786 | 20:15 |
| *** armstrongs has quit IRC | 20:16 | |
| *** jamesmcarthur has quit IRC | 20:19 | |
| *** jeliu_ has joined #zuul | 20:28 | |
| *** jamesmcarthur has joined #zuul | 20:30 | |
| *** zbr_ has quit IRC | 20:35 | |
| *** zbr has joined #zuul | 20:37 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Update testing section https://review.opendev.org/672820 | 20:37 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add clear-firewall role https://review.opendev.org/672786 | 20:41 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Allow ensure-tox to upgrade tox version https://review.opendev.org/672760 | 20:49 |
| *** jamesmcarthur has quit IRC | 21:00 | |
| *** jamesmcarthur has joined #zuul | 21:01 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Allow ensure-tox to upgrade tox version https://review.opendev.org/672760 | 21:02 |
| corvus | mordred, jeliu_: looks good! http://logs.openstack.org/95/670395/8/check/zuul-operator-functional-k8s/68f041f/ara-report/result/3f3b7bbb-6daa-4dba-93c5-fb226f78ebc6/ and http://logs.openstack.org/95/670395/8/check/zuul-operator-functional-k8s/68f041f/ara-report/result/73b10032-3c4b-4647-a41e-5ed412a335a3/ | 21:08 |
| mordred | corvus: woot! | 21:09 |
| jeliu_ | corvus: sweet! finally running | 21:09 |
| corvus | mordred: i'll go ahead and fix up the linters issue and add a test job | 21:09 |
| corvus | jeliu_: and i made a suggestion for an update to your change if you want to take a look | 21:10 |
| *** zbr has quit IRC | 21:11 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: [WIP] Verify Operator Pod Running https://review.opendev.org/670395 | 21:12 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add clear-firewall role https://review.opendev.org/672786 | 21:13 |
| mordred | corvus: awesome! | 21:13 |
| corvus | that should be gtg now | 21:13 |
| *** jamesmcarthur has quit IRC | 21:13 | |
| mordred | corvus: we should maybe get someone other than us to +2 that one | 21:14 |
| corvus | clarkb: ^ :) | 21:14 |
| clarkb | I'll look | 21:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Update testing section https://review.opendev.org/672820 | 21:17 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Add note to clear-firewall docs https://review.opendev.org/672829 | 21:20 |
| clarkb | corvus: mordred ^ do you think something like that makes sense there? | 21:20 |
| clarkb | I'm happy for it to be a follow on (I'm appriving the parent now) | 21:20 |
| corvus | i'd prefer to be as neutral as possible in zuul-jobs | 21:24 |
| clarkb | ok | 21:26 |
| *** zbr has joined #zuul | 21:26 | |
| corvus | maybe something more along the lines of "you may want to consult with your sysadmin blah blah policy" or something might accomplish what you want while not being opinionated? | 21:27 |
| clarkb | might also be worth noting that ansible has an iptables module but I don't think it can express "clear everything" | 21:27 |
| clarkb | corvus: let me a try a version of that | 21:27 |
| corvus | i think i'm tripping up on the "best option may be to modify the rules" because, wearing my opendev hat, i don't think we've ever promised that the "openstack-INPUT" chain is a stable interface :) | 21:28 |
| *** pcaruana has quit IRC | 21:28 | |
| corvus | maybe i'm overthinking it though. maybe that's okay. it doesn't preclude "add your accept rule to the start of INPUT" | 21:29 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Add note to clear-firewall docs https://review.opendev.org/672829 | 21:30 |
| clarkb | something like that maybe? | 21:30 |
| corvus | clarkb: +2. i would also probably +2 PS1 at this point too. | 21:30 |
| *** zbr has quit IRC | 21:32 | |
| *** panda has quit IRC | 21:34 | |
| *** panda has joined #zuul | 21:34 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add clear-firewall role https://review.opendev.org/672786 | 21:34 |
| *** jamesmcarthur has joined #zuul | 21:46 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: [WIP] Verify Operator Pod Running https://review.opendev.org/670395 | 21:48 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add note to clear-firewall docs https://review.opendev.org/672829 | 21:50 |
| *** jamesmcarthur has quit IRC | 21:51 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: [WIP] Verify Operator Pod Running https://review.opendev.org/670395 | 21:55 |
| *** jeliu_ has quit IRC | 22:05 | |
| *** hwangbo has quit IRC | 22:17 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Remember tab location on build page https://review.opendev.org/672836 | 22:29 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Use base 1 line number anchors in log view https://review.opendev.org/672837 | 22:33 |
| *** jamesmcarthur has joined #zuul | 22:38 | |
| *** jamesmcarthur has quit IRC | 22:44 | |
| mordred | corvus: https://www.npmjs.com/package/react-lazylog | 22:52 |
| mordred | corvus: I was reading the convo between you and tristanC in https://review.opendev.org/#/c/671906/4/web/src/reducers/logfile.js@30 and it made me want to go trolling around a little bit | 22:54 |
| *** jamesmcarthur has joined #zuul | 23:20 | |
| corvus | mordred: that seems like it might be really close to what we want, but i'm not sure about things like adding support for severity | 23:20 |
| corvus | and the whole "only show me > debug" lines is apparently a well-liked feature of osla? | 23:21 |
| corvus | maybe that's something that could be added? | 23:22 |
| corvus | i really like the chunked responses and " | 23:22 |
| corvus | Able to load large files upwards of 100MB without crashing the browser" | 23:22 |
| *** jamesmcarthur has quit IRC | 23:24 | |
| corvus | mordred: i wonder how close we could get the search/filtering feature of that to mimic it? | 23:24 |
| corvus | like, have a button which does a search for "(?!DEBUG)" | 23:25 |
| corvus | mordred: i was just starting on importing some of the osla stuff, but since you found that, i'm going to clean up what i've done and push it up, and i think next step should be to replace the existing thing with that and see what it does | 23:26 |
| corvus | i'm going to EOD, so you or tristanC are welcome to try that tomorrow before i wake up if you want :) | 23:26 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Parse log file in action module https://review.opendev.org/672839 | 23:30 |
| corvus | mordred, tristanC: ^ if you want to stick lazylog on that and see how it looks, ++; if you don't get to it first, i can try that tomorrow. | 23:31 |
| *** tjgresha has quit IRC | 23:31 | |
| *** sshnaidm is now known as sshnaidm|off | 23:43 | |
| *** hashar has quit IRC | 23:43 | |
| *** armstrongs has joined #zuul | 23:48 | |
| *** jamesmcarthur has joined #zuul | 23:50 | |
| *** jamesmcarthur has quit IRC | 23:55 | |
| *** smcginnis has quit IRC | 23:56 | |
| *** armstrongs has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!