| *** pierrepr1netti has joined #oooq | 01:53 | |
| *** pierreprinetti has quit IRC | 01:56 | |
| *** aakarsh has joined #oooq | 02:00 | |
| *** dsneddon has quit IRC | 02:04 | |
| *** dsneddon has joined #oooq | 02:17 | |
| *** dsneddon has quit IRC | 02:22 | |
| *** rlandy|bbl is now known as rlandy | 02:37 | |
| *** dsneddon has joined #oooq | 02:45 | |
| *** rfolco has quit IRC | 03:05 | |
| rlandy | weshay: still around? | 03:11 |
|---|---|---|
| weshay | rlandy, alo | 03:11 |
| rlandy | weshay: left you instructions for the secret in the google doc | 03:12 |
| weshay | ah rock on.. :) | 03:12 |
| weshay | thank you | 03:12 |
| rlandy | wasn't sure if you wanted it submitted yet | 03:12 |
| weshay | rlandy, will probably happen tomorrow | 03:12 |
| rlandy | the output will be in the format you see in the secrets file | 03:12 |
| rlandy | zuul.d/secrets.yaml | 03:12 |
| weshay | k cool.. thanks for getting that | 03:13 |
| rlandy | that is all :) | 03:13 |
| weshay | cool cool.. get out of here :) | 03:13 |
| rlandy | I'm busy python'ing | 03:13 |
| rlandy | behind the curve here | 03:13 |
| weshay | ha.. me too :) | 03:18 |
| weshay | trying to get container-prep time into the cockpit | 03:18 |
| *** rlandy has quit IRC | 03:38 | |
| *** ykarel|away has joined #oooq | 03:39 | |
| *** gkadam has joined #oooq | 03:58 | |
| *** gkadam has quit IRC | 03:58 | |
| *** skramaja has joined #oooq | 03:59 | |
| *** dsneddon has quit IRC | 03:59 | |
| *** dsneddon has joined #oooq | 04:01 | |
| *** dsneddon has quit IRC | 04:05 | |
| *** ykarel|away is now known as ykarel | 04:11 | |
| *** soniya29 has joined #oooq | 04:23 | |
| *** ratailor has joined #oooq | 04:40 | |
| *** surpatil has joined #oooq | 04:43 | |
| *** weshay has quit IRC | 04:52 | |
| *** holser has joined #oooq | 05:08 | |
| *** holser has quit IRC | 05:12 | |
| *** dsneddon has joined #oooq | 05:34 | |
| *** dsneddon has quit IRC | 05:40 | |
| *** ykarel has quit IRC | 05:43 | |
| *** ksambor has joined #oooq | 05:57 | |
| *** ykarel has joined #oooq | 06:03 | |
| *** apetrich has quit IRC | 06:08 | |
| *** Tengu has quit IRC | 06:16 | |
| *** Tengu has joined #oooq | 06:17 | |
| *** jfrancoa has joined #oooq | 06:19 | |
| *** Tengu has quit IRC | 06:20 | |
| *** Tengu has joined #oooq | 06:21 | |
| *** jfrancoa has quit IRC | 06:23 | |
| *** Tengu has quit IRC | 06:28 | |
| *** kopecmartin|off is now known as kopecmartin | 06:36 | |
| *** holser has joined #oooq | 06:41 | |
| *** Tengu has joined #oooq | 06:42 | |
| *** Tengu has quit IRC | 06:44 | |
| *** zbr is now known as zbr|ruck | 06:48 | |
| *** Tengu has joined #oooq | 06:50 | |
| *** bogdando has joined #oooq | 06:57 | |
| *** ccamacho has joined #oooq | 07:13 | |
| *** marios has joined #oooq | 07:15 | |
| *** tosky has joined #oooq | 07:18 | |
| *** udesale has joined #oooq | 07:20 | |
| *** yolanda has quit IRC | 07:21 | |
| chandankumar | arxcruz|rover: kopecmartin soniya29 surpatil I have added the tempest planning meeting Agenda, please add/modify it https://etherpad.openstack.org/p/tripleo-tempest-squad-meeting | 07:21 |
| *** amoralej|off is now known as amoralej | 07:23 | |
| *** yolanda has joined #oooq | 07:23 | |
| *** yolanda has quit IRC | 07:32 | |
| *** dsneddon has joined #oooq | 07:36 | |
| *** apetrich has joined #oooq | 07:41 | |
| *** jpena|off is now known as jpena | 07:42 | |
| *** yolanda has joined #oooq | 07:47 | |
| *** ykarel is now known as ykarel|lunch | 07:52 | |
| *** soniya29 has quit IRC | 07:54 | |
| *** dsneddon has quit IRC | 08:07 | |
| *** apetrich has quit IRC | 08:07 | |
| zbr|ruck | arxcruz|rover: what do you know about r8 cont build failure https://review.opendev.org/#/c/683200/1 | 08:11 |
| zbr|ruck | apparently this had 100% failure rate since yesterday | 08:12 |
| zbr|ruck | https://review.rdoproject.org/zuul/builds?job_name=tripleo-build-containers-rhel-8-master | 08:12 |
| zbr|ruck | i see something "error: Failed to merge in the changes." | 08:13 |
| zbr|ruck | damn, there is a rhel8 branch | 08:13 |
| zbr|ruck | who is in charge of keeping that branch updated? | 08:14 |
| zbr|ruck | joys of using cherry picks in ci | 08:15 |
| chandankumar | zbr|ruck: I think I can help there | 08:22 |
| zbr|ruck | alredy fixed that CR | 08:23 |
| zbr|ruck | there were two CRs from folco that went outdated, rebased them. | 08:23 |
| zbr|ruck | now my question is why we cannot merge them | 08:24 |
| chandankumar | zbr|ruck: we are waiting for CentOS 8 and python3 swift stuff | 08:25 |
| arxcruz|rover | zbr|ruck: i saw yesterday failing, but was focusing on the master | 08:27 |
| *** dsneddon has joined #oooq | 08:34 | |
| *** ykarel|lunch is now known as ykarel | 08:50 | |
| arxcruz|rover | zbr|ruck: the jobs are timeing out :/ | 08:52 |
| *** dsneddon has quit IRC | 09:01 | |
| *** dsneddon has joined #oooq | 09:06 | |
| *** dsneddon has quit IRC | 09:10 | |
| *** derekh has joined #oooq | 09:27 | |
| *** dtantsur|afk is now known as dtantsur | 09:27 | |
| *** apetrich has joined #oooq | 09:41 | |
| *** apetrich has quit IRC | 09:45 | |
| zbr|ruck | panda: check https://review.opendev.org/#/c/683200/ | 09:50 |
| panda | zbr|ruck: oh, that's why al the retries fail anyway | 09:56 |
| *** apetrich has joined #oooq | 09:59 | |
| marios | zbr|ruck: panda where can i put the code i want on the 'host' before molecule is invoked there.. .any ideas/? problem is i'm trying to do 'become' with 'delegate_to' localhost, from the molecule vm, and it fails http://logs.rdoproject.org/02/22002/17/check/rdo-tox-molecule/5f37347/tox/reports.html | 10:03 |
| *** jaosorior has quit IRC | 10:03 | |
| marios | zbr|ruck: panda i realise it wont work as we can't sudo from the molecule vm on the localhost... so i am trying to understand where should i do that setup on th elocalhost, before molecule is invoked (these are the tasks i want to run https://review.rdoproject.org/r/#/c/22002/17/ci-scripts/infra-setup/roles/promoter/tasks/main.yml ) | 10:04 |
| panda | marios: you are running molecule from a host and you want to do something as root on that host ? | 10:05 |
| marios | panda: yeah we need to setup something on the host for the local registry (since that will be runnin on the localhost where we have the docker) | 10:07 |
| marios | panda: certificates and such | 10:07 |
| panda | marios: the ansible_user must be setup to be able to sudo as root passwordless, and without a controlling terminal | 10:08 |
| marios | panda: i see we have that there https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/promoter/molecule/Dockerfile.j2 (sudo from promoter_user) but | 10:10 |
| marios | panda: isn't that just for the molecule VM. | 10:10 |
| zbr|ruck | marios: that is what you got wrong: "no service should run on localhost". you want to configure something on the docker registry, but this registry is supposed to run on a container. | 10:10 |
| panda | marios: ansible is run from the host always, then acts on the containers as they were part of the inventory | 10:11 |
| marios | panda: the problem is even if the promoter_user has sudo on the molecule vm ("centos7" host) - i need sudo on the localhost. | 10:11 |
| zbr|ruck | assume you cannot configure anything on your localhost | 10:11 |
| marios | panda: zbr|ruck i think doing it like this, with delegate_to and sudo, doesn't make sense. the molecule vm shouldn't be able to sudo onto the localhost. hence my question, where should i put that setup instead. it needs to be before the molecule invocation i think, like host setup step? can't find that | 10:11 |
| marios | 13:10 < zbr|ruck> marios: that is what you got wrong: "no service should run on localhost". you want to configure something on the docker registry, but this registry is supposed to run on a container. | 10:12 |
| marios | zbr|ruck: we are using a trick so that when we start containersduring the molecule run, like the container registry, it is actually running on localhost | 10:12 |
| marios | zbr|ruck: cos we can't run docker on a docker container | 10:12 |
| marios | zbr|ruck: so the registry IS going to run on the localhost | 10:12 |
| zbr|ruck | the registry is a container itself | 10:13 |
| marios | zbr|ruck: yes but it is executing on localhost not on the centos7 vm | 10:14 |
| marios | zbr|ruck: (cos we have the DOCKER_HOST: unix:///tmp/docker.sock | 10:14 |
| marios | zbr|ruck: so i spin up a container during my test, which is running in the localhost. and i need to configure some stuff for that container which requires sudo | 10:14 |
| zbr|ruck | forget about sudo localhost, I will not support anything that tries to do this. this was once of the selling points of molecule, and will totally break some usecases. | 10:15 |
| marios | zbr|ruck: sure fine ^ thats what i'm asking though where does it go instead | 10:15 |
| zbr|ruck | imagine that you are asking "pytest to do sudo", openstack-infra will tell you "nope" | 10:15 |
| zbr|ruck | that is almost the same thing | 10:15 |
| marios | zbr|ruck: agree 'delegate_to: localhost" and become: true is bad | 10:15 |
| marios | zbr|ruck: so.? | 10:15 |
| zbr|ruck | happy to hear that. | 10:16 |
| marios | zbr|ruck: not sure where then? | 10:16 |
| zbr|ruck | there are two exceptions but this is well controled and outside molecule power: bindep and installation of docker. | 10:16 |
| zbr|ruck | you can think this as "zuul-managed" | 10:17 |
| zbr|ruck | zuul runs bindep and installs docker, as these are both needed by molecule job. | 10:17 |
| panda | zbr|ruck: the alternative | 10:17 |
| zbr|ruck | but once the job starts, it will never be able to sudo, by design. | 10:17 |
| zbr|ruck | panda: i need to understand what we needed to do as root in the first place | 10:18 |
| zbr|ruck | bj session? maybe it would easier to understand | 10:18 |
| *** soniya29 has joined #oooq | 10:27 | |
| *** brault has joined #oooq | 10:47 | |
| *** jpena is now known as jpena|lunch | 11:01 | |
| *** panda is now known as panda|lunch | 11:03 | |
| *** dsneddon has joined #oooq | 11:07 | |
| *** apetrich has quit IRC | 11:07 | |
| chandankumar | marios: panda|lunch please have a look https://review.rdoproject.org/r/#/c/22441/ and https://review.rdoproject.org/r/#/c/22442/ when free, thanks ! | 11:14 |
| *** ccamacho has quit IRC | 11:21 | |
| *** jaosorior has joined #oooq | 11:27 | |
| *** dsneddon has quit IRC | 11:31 | |
| * chandankumar needs to leave now | 11:33 | |
| chandankumar | see ya on monday | 11:33 |
| *** chandankumar is now known as raukadah | 11:33 | |
| zbr|ruck | bye see you on tuesday! and arxcruz|rover do not lose hope! ;) | 11:40 |
| arxcruz|rover | zbr|ruck: what is hope? baby don't hurt me | 11:41 |
| zbr|ruck | yeah, music usually helps (if you don't follow my spotify playlist) | 11:41 |
| *** dsneddon has joined #oooq | 11:42 | |
| arxcruz|rover | lol | 11:42 |
| *** dmellado has quit IRC | 11:45 | |
| *** jaosorior has quit IRC | 11:47 | |
| *** dsneddon has quit IRC | 11:47 | |
| *** dmellado has joined #oooq | 11:48 | |
| *** apetrich has joined #oooq | 11:51 | |
| *** udesale has quit IRC | 11:52 | |
| *** jpena|lunch is now known as jpena | 11:53 | |
| *** udesale has joined #oooq | 11:54 | |
| *** dsneddon has joined #oooq | 12:01 | |
| *** weshay has joined #oooq | 12:03 | |
| *** dsneddon has quit IRC | 12:06 | |
| arxcruz|rover | weshay: hey boss | 12:09 |
| weshay | hey | 12:09 |
| arxcruz|rover | weshay: the openstack-periodic-master queue was 17 hours, Zoli is taking a look, i also told him to kill the rest of the jobs on queue, because it wouldn't promote anyway | 12:11 |
| weshay | dang | 12:13 |
| weshay | thanks | 12:13 |
| weshay | in mtg fyi | 12:13 |
| weshay | slow | 12:13 |
| arxcruz|rover | weshay: also, for some reason, the overcloud deploy is timeing out | 12:15 |
| arxcruz|rover | i open a lp, try to dig into the logs | 12:15 |
| *** panda|lunch is now known as panda | 12:16 | |
| *** akahat has joined #oooq | 12:19 | |
| *** amoralej is now known as amoralej|lunch | 12:19 | |
| *** ratailor has quit IRC | 12:25 | |
| *** skramaja has quit IRC | 12:37 | |
| *** jaosorior has joined #oooq | 12:38 | |
| *** rlandy has joined #oooq | 12:38 | |
| *** apetrich has quit IRC | 12:42 | |
| *** rfolco has joined #oooq | 12:46 | |
| *** aakarsh has quit IRC | 12:47 | |
| *** soniya29 has quit IRC | 12:52 | |
| arxcruz|rover | weshay: tempest meeting ? | 13:01 |
| weshay | arxcruz|rover, ya.. few minutes late | 13:01 |
| marios | rlandy sync call? | 13:02 |
| rlandy | ack | 13:02 |
| marios | zbr|ruck: (if you are available? ^^ about molecule should be in your calendar no worries if busy) | 13:02 |
| weshay | arxcruz|rover, running long | 13:07 |
| arxcruz|rover | weshay: ? | 13:07 |
| *** dsneddon has joined #oooq | 13:08 | |
| *** brault has quit IRC | 13:13 | |
| *** dsneddon has quit IRC | 13:13 | |
| *** aakarsh has joined #oooq | 13:15 | |
| *** amoralej|lunch is now known as amoralej | 13:27 | |
| weshay | kopecmartin, http://logs.rdoproject.org/13/683313/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/486808b/ | 13:30 |
| *** apetrich has joined #oooq | 13:39 | |
| ykarel | zbr|ruck, arxcruz|rover are you guys aware of issue in rocky/queens promotion jobs already | 13:39 |
| ykarel | ? | 13:39 |
| arxcruz|rover | ykarel: which one ? | 13:39 |
| arxcruz|rover | lol | 13:39 |
| arxcruz|rover | ykarel: i submit a patch to skip the security tests on rocky | 13:39 |
| weshay | surpatil, ping | 13:39 |
| weshay | hello | 13:39 |
| surpatil | hello | 13:40 |
| arxcruz|rover | ykarel: didn't touch on queens yet, because of master | 13:40 |
| ykarel | arxcruz|rover, all jobs failing at dumb-init | 13:40 |
| surpatil | weshay, | 13:40 |
| weshay | surpatil, https://tree.taiga.io/project/tripleo-ci-board/us/1070 | 13:40 |
| *** soniya29 has joined #oooq | 13:41 | |
| weshay | soniya29, :) | 13:41 |
| arxcruz|rover | ykarel: nope, i did not check that, can you sent me the log ? | 13:41 |
| arxcruz|rover | ykarel: http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens/2f13e34/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz ? | 13:41 |
| ykarel | arxcruz|rover, nope | 13:42 |
| ykarel | arxcruz|rover, queens:_ https://trunk-primary.rdoproject.org/api-centos-queens/api/civotes_detail.html?commit_hash=1769de3549e81a965ed93bca3cbbd9daa9a5b87a&distro_hash=6927fa105876d7e42878338baf1a753a5fe2e828 | 13:42 |
| ykarel | rocky:- https://trunk-primary.rdoproject.org/api-centos-rocky/api/civotes_detail.html?commit_hash=8c202730982e8f627f4bfedd59160b0e4f352f6a&distro_hash=1a4bc69cb4df0dc41b0d5213d3637666d922bc0c | 13:43 |
| soniya29 | weshay, :) | 13:43 |
| ykarel | arxcruz|rover, in queens see non-ovb jobs | 13:43 |
| ykarel | and in rocky both ovb and non-ovb failing same issue | 13:43 |
| ykarel | so in queens ovb there seems different issue | 13:43 |
| *** soniya29 has quit IRC | 13:44 | |
| *** surpatil has quit IRC | 13:47 | |
| ykarel | arxcruz|rover, https://review.opendev.org/#/c/681876/ patch has caused the issue in queens, and similar patch in rocky affected rocky jobs | 13:47 |
| *** weshay is now known as weshay|ruck | 13:47 | |
| *** apetrich has quit IRC | 13:51 | |
| *** dsneddon has joined #oooq | 13:55 | |
| *** dsneddon has quit IRC | 14:00 | |
| rlandy | panda: marios: rfolco: https://review.rdoproject.org/r/#/c/22348/ | 14:04 |
| rlandy | please comment | 14:07 |
| *** rascasoft has quit IRC | 14:10 | |
| weshay|ruck | panda, hey | 14:14 |
| weshay|ruck | ready when you are | 14:14 |
| weshay|ruck | panda, https://meet.google.com/ana-ebsw-tvu | 14:15 |
| rfolco | rlandy, tripleo-common task is https://tree.taiga.io/project/tripleo-ci-board/task/1293?kanban-status=1447274 | 14:15 |
| rfolco | ?? | 14:15 |
| *** dsneddon has joined #oooq | 14:17 | |
| weshay|ruck | rlandy, did you brief the other folks on quay? | 14:20 |
| *** akahat has quit IRC | 14:20 | |
| rlandy | weshay|ruck: no - which folks? | 14:26 |
| rlandy | rfolco: ack - all yours | 14:27 |
| rfolco | rlandy, thanks!! | 14:27 |
| rlandy | weshay|ruck: are we going forward? do you want me to commit the secret? | 14:27 |
| weshay|ruck | rfolco, marios panda rlandy fyi.. we're testing the quay registry.. going to start pushing to docker.io and quay for master | 14:27 |
| rlandy | rfolco: actually I need you to check my me - pls see pvt | 14:27 |
| weshay|ruck | trying to increase perf | 14:27 |
| weshay|ruck | have a doc w/ details | 14:28 |
| weshay|ruck | anyone chatting w/ panda atm? | 14:28 |
| rlandy | we were | 14:28 |
| rlandy | ended call a few minutes ago | 14:29 |
| *** jaosorior has quit IRC | 14:30 | |
| marios | weshay|ruck: ack ... just testing for now right? like might we eventually drop docker for quay if it works? | 14:32 |
| * weshay|ruck suspsects some transition period for 1 job to run on quay etc.. it's all very new atm | 14:32 | |
| *** dtantsur is now known as dtantsur|afk | 14:33 | |
| arxcruz|rover | rfolco: so many emails... | 14:37 |
| rfolco | arxcruz|rover, google meet link | 14:37 |
| rlandy | weshay|ruck: https://review.rdoproject.org/r/22458 Add secret for new registry to push promoted containers | 14:43 |
| weshay|ruck | thanks | 14:44 |
| rlandy | weshay|ruck: that is config so it's a shot in the dark if it works - will have to merge to try | 14:45 |
| rlandy | https://review.rdoproject.org/r/#/c/22413/ | 14:47 |
| rlandy | panda: marios: sshnaidm|off: arxcruz|rover: ^^ pls confirm if want that merged now or not | 14:48 |
| *** udesale has quit IRC | 14:50 | |
| *** udesale has joined #oooq | 14:51 | |
| *** Vorrtex has joined #oooq | 14:56 | |
| *** dsneddon has quit IRC | 15:00 | |
| panda | rlandy: question about this | 15:06 |
| panda | rlandy: if the periodic in queens fails for example, and you need to fix it, is running periodic jobs as check enough to get the branch back on track ? | 15:07 |
| weshay|ruck | arxcruz|rover, based on what I see here, re: perf regression | 15:09 |
| weshay|ruck | http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-90d&to=now&var-influxdb_filter=job_name%7C%3D%7Ctripleo-ci-centos-7-containers-multinode&var-influxdb_filter=branch%7C%3D%7Cmaster&var-influxdb_filter=result%7C%3D%7CSUCCESS&fullscreen&panelId=16 | 15:09 |
| weshay|ruck | arxcruz|rover, I think you found the right bug :) | 15:09 |
| weshay|ruck | arxcruz++ | 15:09 |
| arxcruz|rover | :) | 15:10 |
| weshay|ruck | arxcruz|rover, the same handler is running in the undercloud | 15:15 |
| weshay|ruck | https://2a7fe70570a247a5c306-1d41175f1f59c744683b1a918bffa4f9.ssl.cf5.rackcdn.com/683040/1/check/tripleo-ci-centos-7-containers-multinode/7a1e062/logs/undercloud/home/zuul/install-undercloud.log.txt.gz | 15:15 |
| weshay|ruck | arxcruz|rover, however we have no idea how long it takes because it's not running w/ enough verbosity | 15:15 |
| arxcruz|rover | weshay|ruck: if we assume 25 minutes each one, the jobs should now run 50 minutes less right ? | 15:16 |
| weshay|ruck | I would say.. 30-50 ya | 15:16 |
| rlandy | panda: you acn always run testproject job | 15:20 |
| rlandy | essentially, we could run these jobs more often | 15:20 |
| rlandy | it's just we are adding a new release | 15:21 |
| rlandy | additional burden | 15:21 |
| rlandy | panda: so to answer your question, I think so, but open to debate here | 15:22 |
| *** Vorrtex has quit IRC | 15:25 | |
| rfolco | weshay|ruck, knock knock | 15:29 |
| rlandy | jpena: https://review.rdoproject.org/r/#/c/22458/2/zuul.d/secrets.yaml - better now? no newline? | 15:30 |
| marios | panda: the other cost (continuing the conversation earlier) is ofcourse iteration is not as easy (running local vs zuul) | 15:30 |
| panda | marios: yes | 15:31 |
| jpena | rlandy: aha, that looks fine now | 15:31 |
| panda | marios: I have a question for you. You're trying to secure the registry with your tasks. Have you tried to allow docker to contact those registries as insecure instead ? | 15:31 |
| rlandy | jpena: thanks | 15:32 |
| panda | marios: as in https://docs.docker.com/registry/insecure/ | 15:32 |
| marios | panda: yes if you mean --insecure-registries and also in /etc/docker/daemon.json ? | 15:32 |
| panda | marios: ah, yes | 15:32 |
| jpena | rlandy: I had the same issue a few days ago, so I now test secrets with a decrypt_secret tool I found | 15:32 |
| weshay|ruck | rfolco, https://meet.google.com/etf-xwxz-kwa?authuser=1 | 15:33 |
| marios | panda: am pretty sure i tried all the things. pushing *nanifests* to localhost:5000 works fine... but we can't do it from the molecule vm cos localhost:5000 doesn't resolve correctly. so the problem is we need certs as soon as we try to use an IP instead like the host IP for the registry | 15:33 |
| rlandy | jpena: can you share link to the decrypt_secret tool? I have messed up on creating secrets before and there is no good way of knowing ahead if time | 15:33 |
| rlandy | of | 15:33 |
| jpena | rlandy: I got it from https://review.opendev.org/639771 | 15:34 |
| rlandy | nice - thanks | 15:34 |
| jpena | it has to be run from the zuul server itself, I think | 15:34 |
| marios | panda: https://github.com/docker/cli/issues/1631 | 15:35 |
| *** kopecmartin is now known as kopecmartin|off | 15:35 | |
| *** apetrich has joined #oooq | 15:37 | |
| weshay|ruck | arxcruz|rover, fyi https://review.opendev.org/#/c/683425/1 | 15:38 |
| arxcruz|rover | weshay|ruck: ack | 15:49 |
| marios | panda: rlandy rfolco heh, in fact , if we're gonna run it as part of the main job then we don't even need the certs stuff https://review.rdoproject.org/r/#/c/22002/21/playbooks/staging.yml /me just realised | 15:49 |
| rlandy | is that a problem? | 15:50 |
| marios | panda: rlandy rfolco the certs stuff was so we could test it via molecule... but if were just running it on a local zuul machine then its fine | 15:50 |
| marios | rlandy: not at all, just saying its even easier (I included the certs stuff but just realised i don't need it at https://review.rdoproject.org/r/#/c/22002/21/playbooks/staging.yml | 15:50 |
| panda | marios: why not ? | 15:54 |
| panda | marios: because both registry are reachable at localhost ? | 15:55 |
| marios | panda: yeah cos we can push manifest from localhost to localhost with just --insecure | 16:00 |
| *** ykarel is now known as ykarel|away | 16:00 | |
| marios | panda: problem was for molecule case we had to use IP and that needs certs for manifest push | 16:00 |
| panda | marios: I think I can solve that | 16:04 |
| panda | marios: what were the IPs on the registries ? 192.168.0.x ? | 16:05 |
| marios | panda: when you start the registry container it is reachable via any local address.... localhost:5000 127.0: or 192.168.x.x:5000 depending on the host etc | 16:06 |
| marios | panda: i was trying to use the host IP . but there is no 'ip on the registry' | 16:06 |
| panda | marios: ok, patch incoming | 16:08 |
| *** Vorrtex has joined #oooq | 16:09 | |
| marios | panda: ... ok great :D i mean lets discuss on monday and decide what to pursue (am trying with the job for now/till then) | 16:10 |
| rlandy | panda: when you pass dlrn in https://github.com/rdo-infra/ci-config/blob/91b8b21c9c7c17f7ab224c54e67fac1493177684/ci-scripts/dlrnapi_promoter/dlrnapi_promoter.py#L31 - what are you actually passing | 16:11 |
| rlandy | dlrnapi_client.ApiClient? | 16:12 |
| panda | marios: ok, do you mind if I make your patch depends-on this https://review.rdoproject.org/r/22464 ? | 16:19 |
| panda | marios: and iterate a bit when you're gone ? | 16:19 |
| marios | panda: ack no worries, i am gona wrap up in a minute | 16:19 |
| marios | panda: i think v24 is my last update today can you let it run first | 16:20 |
| marios | panda: and thank you would be cool if it works | 16:20 |
| panda | rlandy: yes, dlrn is the client | 16:20 |
| panda | rlandy: the link is the target name | 16:20 |
| marios | panda: you think its cos of insecure in the molecule vm? | 16:21 |
| marios | panda: not sure it will help but worth testing it for sure | 16:21 |
| rlandy | nvm - got it | 16:22 |
| panda | marios: it's the deamon, it will refuse to contact insecure registries, it's a daemon option | 16:23 |
| marios | panda: fyi on the address added a comment | 16:23 |
| panda | marios: so we need to instrsuct the daemon on the host to use insecure registries for all those ips, it works with CIDR | 16:23 |
| marios | panda: but it doesn't refuse. i mean, we can use insecure.. rlandy is already using it. and we can push containers to it too. i can do that via molecule already | 16:23 |
| marios | panda: the problem is specific to 'docker manifest ...' | 16:23 |
| panda | so if we can configure the deamon on the host to allow insecure connection wth all the registries there, we can skip the whole cert part | 16:24 |
| * rlandy is clearly very anti-security | 16:24 | |
| marios | panda: like running it vanila docker run registry on localhost:5000 i can tag + push containers to it no problem via molecule | 16:24 |
| marios | panda: like rlandy is already doing | 16:24 |
| marios | 19:23 < marios> panda: the problem is specific to 'docker manifest ...' | 16:24 |
| panda | marios: by default , registries in localhost:500 are added to insecure | 16:24 |
| ykarel|away | rlandy, wouldn't be good if rocky/queens are moved to openstack-peiodic-weekend after they are green | 16:25 |
| panda | marios: https://github.com/docker/docker-ce/blob/e0e6de2e0ddf383ed5bf9099512e32afcf121eda/components/engine/registry/config.go#L162 | 16:25 |
| ykarel|away | currently rocky-15 days, queens 9 days | 16:25 |
| rlandy | ykarel|away; ack - patch not merged yet | 16:25 |
| panda | marios: we'll see | 16:25 |
| marios | panda: yeah | 16:25 |
| rlandy | until the board clears up | 16:25 |
| ykarel|away | rlandy, yes that' why i asked | 16:25 |
| panda | marios: I have some doubts, but would like to test. | 16:25 |
| marios | panda: i just went looking to find some logs to show you | 16:25 |
| rlandy | up to ruck/rover to merge that | 16:25 |
| marios | panda: like the output | 16:25 |
| ykarel|away | it already has +2s so | 16:25 |
| rlandy | ykarel|away: you can -1 it if you want - for now | 16:25 |
| marios | panda: sure, its late here too so gonna leave it for monday lets see how the test goes | 16:26 |
| ykarel|away | rlandy, i just commented as a suggestion | 16:26 |
| rlandy | k | 16:26 |
| ykarel|away | no negative vote | 16:26 |
| ykarel|away | anyway train pipeline is not ready yet | 16:26 |
| ykarel|away | so not much harm in continuing as it is currently | 16:26 |
| rlandy | ykarel|away: ok - I w-1 | 16:28 |
| rlandy | on https://review.rdoproject.org/r/#/c/22413/ | 16:28 |
| rlandy | safer that way | 16:28 |
| ykarel|away | ack | 16:28 |
| ykarel|away | Thanks | 16:28 |
| *** holser__ has joined #oooq | 16:29 | |
| *** jpena is now known as jpena|off | 16:31 | |
| *** holser has quit IRC | 16:31 | |
| *** bogdando has quit IRC | 16:36 | |
| *** marios has quit IRC | 16:41 | |
| *** ykarel|away has quit IRC | 16:41 | |
| weshay|ruck | rlandy, need a merge here https://review.rdoproject.org/r/#/c/22377/ | 16:41 |
| *** derekh has quit IRC | 16:42 | |
| rlandy | looking | 16:42 |
| weshay|ruck | re: sagi's comment https://review.opendev.org/#/c/683289/ | 16:43 |
| rlandy | done | 16:43 |
| weshay|ruck | thanks | 16:43 |
| raukadah | weshay|ruck: rlandy https://review.rdoproject.org/r/#/c/22441/ and https://review.rdoproject.org/r/#/c/22442/ | 16:47 |
| raukadah | few minor reviews | 16:48 |
| rlandy | raukadah: do we know this works now | 16:51 |
| raukadah | rlandy: yes | 16:51 |
| raukadah | rlandy: https://review.rdoproject.org/r/#/c/22304/ | 16:51 |
| raukadah | check last comment | 16:51 |
| raukadah | rlandy: for more info we need to wait till monday, there is an AI waiting on jpena|off | 16:52 |
| rlandy | raukadah: fine - ping me then | 16:52 |
| raukadah | rlandy: sure | 16:52 |
| * raukadah now sleep, I have wake up early tomorrow for this conference https://hasgeek.com/rootconf/2019-pune/ | 16:56 | |
| raukadah | rlandy: weshay|ruck did you get a chance to look at hackmd which I emailed related to podman & ceph-ansible gating? | 16:57 |
| raukadah | looking forward to have some comments & questions | 16:57 |
| weshay|ruck | raukadah, nope.. too much shit on fire | 16:57 |
| weshay|ruck | folks arxcruz|rover just found a HUGE per regression :) | 16:57 |
| weshay|ruck | perf | 16:57 |
| *** udesale has quit IRC | 16:57 | |
| weshay|ruck | arxcruz++ | 16:58 |
| weshay|ruck | potentially saving 30-50 min per run | 16:58 |
| *** holser__ has quit IRC | 16:58 | |
| *** ykarel|away has joined #oooq | 16:58 | |
| arxcruz|rover | :) | 16:59 |
| rlandy | yep | 17:00 |
| *** holser has joined #oooq | 17:01 | |
| *** holser has quit IRC | 17:05 | |
| rlandy | arxcruz|rover: what was the regression? | 17:15 |
| arxcruz|rover | rlandy: https://review.opendev.org/#/c/683402/5 | 17:15 |
| arxcruz|rover | rlandy: the delete the prepare_param patch was using ansible file module that was taking 30 minutes to delete the directory due the amount of files inside it | 17:16 |
| rlandy | wow - seriously? | 17:17 |
| rlandy | who knew that took time | 17:17 |
| rlandy | nice | 17:17 |
| arxcruz|rover | rlandy: yup | 17:17 |
| rlandy | so much weirdness | 17:18 |
| arxcruz|rover | rlandy: this patch was introduced one month ago | 17:18 |
| raukadah | arxcruz|rover: Thanks for recording the meeting, quite nice discussion | 17:25 |
| raukadah | arxcruz|rover: rlandy weshay|ruck good night see ya on monday with questions | 17:26 |
| arxcruz|rover | raukadah: it wasn't me | 17:26 |
| *** amoralej is now known as amoralej|off | 17:35 | |
| rlandy | raukadah: k - have a good weekend | 17:38 |
| *** tosky_ has joined #oooq | 17:48 | |
| *** holser has joined #oooq | 17:49 | |
| *** tosky is now known as Guest87259 | 17:51 | |
| *** tosky_ is now known as tosky | 17:51 | |
| *** Guest87259 has quit IRC | 17:52 | |
| *** holser has quit IRC | 17:54 | |
| *** Tengu has quit IRC | 18:07 | |
| *** Tengu has joined #oooq | 18:07 | |
| rlandy | panda: ok - added the dlrn promotion check - now looking at the dir compare | 18:16 |
| rlandy | so - you mentioned key for ssh | 18:16 |
| sshnaidm|off | rlandy, see comment on https://review.rdoproject.org/r/#/c/22377/ | 18:19 |
| rlandy | reading | 18:22 |
| *** dsneddon has joined #oooq | 18:23 | |
| rlandy | weshay|ruck: what do you wantt to do about https://review.opendev.org/#/c/683289/2/ci/environments/scenario004-standalone.yaml? | 18:23 |
| rlandy | switch case it? | 18:23 |
| *** dsneddon has quit IRC | 18:27 | |
| *** redrobot has quit IRC | 18:28 | |
| weshay|ruck | sorry was at lunch | 18:30 |
| weshay|ruck | rlandy, /me looks | 18:31 |
| rlandy | np - forgot about friday lunch | 18:31 |
| weshay|ruck | rlandy, hrm.. ya.. so it's still using docker http://logs.rdoproject.org/53/682853/4/openstack-check/tripleo-ci-rhel-8-scenario004-standalone-rdo/986c095/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 18:32 |
| *** Vorrtex has quit IRC | 18:34 | |
| rlandy | weshay|ruck: ack - you -2'ed sshnaidm|off's patch to change that | 18:34 |
| rlandy | so we have a problem | 18:34 |
| weshay|ruck | rlandy, we can try rebasing on top of sshnaidm|off's patch | 18:35 |
| rlandy | what about centos7? | 18:36 |
| weshay|ruck | rlandy, ya.. going to find out | 18:36 |
| weshay|ruck | rebased on top | 18:36 |
| rlandy | k - let's see | 18:37 |
| weshay|ruck | https://review.opendev.org/#/c/682853/ | 18:37 |
| *** holser has joined #oooq | 18:37 | |
| * weshay|ruck watches in zuul | 18:37 | |
| rlandy | panda: hmmm ... login creds to image server will limit where we run this | 18:37 |
| rlandy | the BeautifulSoup can run everywhere | 18:38 |
| rlandy | but will try the creds route for the local dir and skip the md5 sum compare | 18:38 |
| *** holser has quit IRC | 18:41 | |
| *** Vorrtex has joined #oooq | 18:50 | |
| weshay|ruck | rlandy, we're using beautifulSoup, what are we scraping? | 18:58 |
| *** ykarel|away has quit IRC | 19:17 | |
| rlandy | weshay|ruck: two places - where the promoter puts the images | 19:17 |
| rlandy | there is a link created from the promoted hash image to the promotion_target dir | 19:17 |
| rlandy | for the staging reproducer, I can get grab the link | 19:18 |
| weshay|ruck | ok.. | 19:18 |
| rlandy | for the images server, I'd need a username/password/key to get in the server | 19:18 |
| rlandy | but I can scrape this https://images.rdoproject.org/centos7/master/rdo_trunk/current-tripleo/ with no creds | 19:18 |
| rlandy | all it is the names and size of the files to compare | 19:19 |
| rlandy | the same tests needs to run on the staging and real promoter | 19:19 |
| rlandy | weshay|ruck: make sense? | 19:20 |
| weshay|ruck | rlandy, ok.. of course that worked w/ sagi's patch | 19:29 |
| weshay|ruck | http://logs.rdoproject.org/53/682853/5/openstack-check/tripleo-ci-rhel-8-scenario004-standalone-rdo/fadb98d/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 19:29 |
| weshay|ruck | didn't pass.. but we have containers | 19:30 |
| *** dsneddon has joined #oooq | 19:31 | |
| rlandy | one step better | 19:34 |
| rlandy | weshay|ruck: can we create two diff templates | 19:35 |
| rlandy | for centos7 and rehl8? | 19:35 |
| rlandy | rhel8 | 19:35 |
| weshay|ruck | rlandy, that's not the issue anymore | 19:38 |
| weshay|ruck | podman is working | 19:38 |
| weshay|ruck | not sure about docker yet.. as the tests havent started | 19:39 |
| weshay|ruck | but we'll see I guess | 19:39 |
| weshay|ruck | if docker fails.. on centos7 that is a good plan | 19:39 |
| rlandy | hmmmm ... well we exposed the server anyways | 19:43 |
| recheck | [gh-ansible] v2.9.0rc1 → https://github.com/ansible/ansible/releases/tag/v2.9.0rc1 | 20:06 |
| *** Vorrtex has quit IRC | 20:29 | |
| *** jtomasek has quit IRC | 20:32 | |
| *** redrobot has joined #oooq | 20:34 | |
| *** rfolco has quit IRC | 21:08 | |
| *** apetrich has quit IRC | 21:19 | |
| *** aakarsh has quit IRC | 21:20 | |
| arxcruz|rover | weshay|ruck: just checking, yatin seems to be right, the time there is the whole task, but the featureset001 rans in 3 hours... | 21:23 |
| arxcruz|rover | I don't know if it's related or not :( | 21:23 |
| weshay|ruck | arxcruz|rover, everything is running very long | 21:24 |
| arxcruz|rover | sorry, I was very happy that we might finally found the issue... | 21:24 |
| weshay|ruck | ya.. me too | 21:24 |
| weshay|ruck | it's kind of stupid to run prep containers twice | 21:24 |
| weshay|ruck | maybe we can talk to the tripleo guys about that bit | 21:24 |
| *** rlandy has quit IRC | 21:25 | |
| weshay|ruck | arxcruz|rover, doubt it's that simple though | 21:25 |
| weshay|ruck | we've been at this for 3 days now, and haven't made much of a dent | 21:25 |
| weshay|ruck | re: bringing down the job times | 21:25 |
| arxcruz|rover | weshay|ruck: i'll see if i can work on a script to collect the tasks time | 21:26 |
| arxcruz|rover | but tomorrow, it's almost midnight here, just came to check the results | 21:26 |
| weshay|ruck | arxcruz|rover, I have one | 21:26 |
| weshay|ruck | arxcruz|rover, maybe you can help me with it.. as it errors too often | 21:26 |
| *** pierrepr1netti is now known as pierreprinetti | 21:26 | |
| weshay|ruck | arxcruz|rover, it's not great atm https://review.rdoproject.org/r/#/c/22418/4/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf/zuulv3_job_builds.py | 21:27 |
| arxcruz|rover | weshay|ruck: nice, i'll take a look on that :) | 21:27 |
| weshay|ruck | arxcruz|rover, python2 only as the container is python2.. ./zuulv3_job_builds.py --type upstream --url http://zuul.openstack.org/ | 21:28 |
| weshay|ruck | also works if you just run w/o args | 21:28 |
| weshay|ruck | arxcruz|rover, although.. if you use a python virtenv.. #!/usr/bin/env python | 21:28 |
| weshay|ruck | not #!/usr/bin/python | 21:28 |
| weshay|ruck | container does not have /usr/bin/env | 21:29 |
| weshay|ruck | need to update the container | 21:29 |
| weshay|ruck | arxcruz|rover, oh.. and that is for the undercloud or standalone only atm | 21:29 |
| arxcruz|rover | ok | 21:30 |
| *** tosky has quit IRC | 23:28 | |
| *** dsneddon has quit IRC | 23:29 | |
| *** rfolco has joined #oooq | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!