| *** zbr has quit IRC | 00:09 | |
| *** rfolco has quit IRC | 01:54 | |
| *** rlandy|ruck|bbl is now known as rlandy|ruck | 02:11 | |
| rlandy|ruck | weshay|rover: interesting ... rhos-16 scenario004 log ... pacemaker.x86_64 2.0.2-3.el8 @rhel-8-for-x86_64-highavailability-rpms | 02:16 |
|---|---|---|
| rlandy|ruck | same as master | 02:17 |
| rlandy|ruck | https://sf.hosted.upshift.rdu2.redhat.com/logs/13/185113/1/check/tripleo-ci-rhel-8-scenario004-standalone-rhos-16/c83c938/ | 02:17 |
| *** rlandy|ruck has quit IRC | 02:34 | |
| *** apetrich has quit IRC | 03:08 | |
| *** brault has quit IRC | 03:13 | |
| *** brault has joined #oooq | 03:20 | |
| *** bhagyashris has joined #oooq | 03:40 | |
| *** udesale has joined #oooq | 03:41 | |
| *** udesale has quit IRC | 03:45 | |
| *** udesale has joined #oooq | 03:46 | |
| weshay|rover | bhagyashris, hello :) | 03:50 |
| bhagyashris | weshay|rover, Hello :) | 03:51 |
| weshay|rover | bhagyashris, welcome!! how's it going? | 03:51 |
| bhagyashris | weshay|rover, Thank you!. Yes it's going good... I have requested help desk to give me subscription for Red Hat university courses for certification. They marked my request as resolved and said it will take 5 working days.. so waiting for the access. | 03:53 |
| weshay|rover | agh k | 03:53 |
| weshay|rover | bhagyashris, if you want to buy / borrow a book.. you can expense it | 03:54 |
| weshay|rover | bhagyashris, you could ask yatin or chandan if they have a book on the rhcsa / rhce | 03:54 |
| bhagyashris | weshay|rover, okay i will check with them today. | 03:55 |
| weshay|rover | cool.. anything else to chat about? | 03:55 |
| weshay|rover | we can jump on a video call if you want | 03:55 |
| weshay|rover | bhagyashris, I have promised pooja and unmesh I would set up an irc bouncer for all of you | 03:56 |
| bhagyashris | weshay|rover, Nothing from my side, but if you have anything to tell me then that would great to have short call | 03:56 |
| weshay|rover | ok.. let's chat for a minute | 03:56 |
| bhagyashris | weshay|rover, sure | 03:57 |
| weshay|rover | bhagyashris, https://meet.google.com/tfy-mwmw-xvb | 03:57 |
| bhagyashris | weshay|rover, give me min | 03:58 |
| *** skramaja has joined #oooq | 04:17 | |
| *** ykarel|away has joined #oooq | 04:25 | |
| *** ykarel|away is now known as ykarel | 04:31 | |
| *** dsneddon_ has quit IRC | 04:33 | |
| *** dsneddon_ has joined #oooq | 04:58 | |
| *** surpatil has joined #oooq | 05:15 | |
| *** epoojad1 has joined #oooq | 05:17 | |
| *** ratailor has joined #oooq | 05:30 | |
| *** udesale has quit IRC | 05:43 | |
| *** udesale has joined #oooq | 05:43 | |
| *** udesale has quit IRC | 05:44 | |
| *** udesale has joined #oooq | 05:44 | |
| *** saneax has joined #oooq | 05:57 | |
| *** dsneddon_ has quit IRC | 06:12 | |
| *** marios has joined #oooq | 06:17 | |
| *** marios has quit IRC | 06:20 | |
| *** marios has joined #oooq | 06:23 | |
| *** ksambor has joined #oooq | 06:27 | |
| *** dsneddon_ has joined #oooq | 06:37 | |
| *** dsneddon_ has quit IRC | 06:42 | |
| *** saneax has quit IRC | 06:44 | |
| *** saneax has joined #oooq | 06:44 | |
| marios | scen7 is trolling me there is no other explanation https://review.opendev.org/#/c/675313/ | 06:54 |
| *** udesale has quit IRC | 07:03 | |
| *** udesale has joined #oooq | 07:04 | |
| *** jfrancoa has joined #oooq | 07:11 | |
| *** ksambor has quit IRC | 07:18 | |
| *** d0ugal has quit IRC | 07:28 | |
| *** dsneddon_ has joined #oooq | 07:38 | |
| *** apetrich has joined #oooq | 07:43 | |
| *** akahat has joined #oooq | 07:43 | |
| *** dsneddon_ has quit IRC | 07:44 | |
| *** jpena|off is now known as jpena | 07:46 | |
| *** tosky has joined #oooq | 07:52 | |
| *** ksambor has joined #oooq | 08:08 | |
| *** tesseract has joined #oooq | 08:11 | |
| *** epoojad1 has quit IRC | 08:13 | |
| *** dsneddon_ has joined #oooq | 08:21 | |
| *** dsneddon_ has quit IRC | 08:25 | |
| *** zbr has joined #oooq | 08:30 | |
| *** tosky has quit IRC | 08:32 | |
| *** dsneddon_ has joined #oooq | 08:33 | |
| *** zbr has quit IRC | 08:35 | |
| *** zbr has joined #oooq | 08:36 | |
| *** tosky has joined #oooq | 08:37 | |
| *** dsneddon_ has quit IRC | 08:38 | |
| *** amoralej|off is now known as amoralej | 08:38 | |
| *** dtantsur|afk is now known as dtantsur | 08:48 | |
| *** d0ugal has joined #oooq | 08:50 | |
| *** bogdando has joined #oooq | 09:03 | |
| *** chem has joined #oooq | 09:09 | |
| arxcruz | jpena: can you check zuul log why https://review.rdoproject.org/r/23642 the second job is failing ? | 09:20 |
| jpena | arxcruz: sure | 09:21 |
| arxcruz | thanks | 09:21 |
| jpena | arxcruz: you mean the tripleo-ceph-integration-rhel-8-standalone-featureset016 job, right? | 09:21 |
| arxcruz | jpena: the test patch is https://review.rdoproject.org/r/#/c/23642/ | 09:22 |
| arxcruz | i set scenario001 and scenario004 jobs depending on rpm build | 09:22 |
| arxcruz | the first is running, the second is in error state | 09:22 |
| jpena | ah ok | 09:23 |
| *** ykarel is now known as ykarel|lunch | 09:23 | |
| jpena | aha, got it | 09:27 |
| jpena | arxcruz: see my last comment on https://review.rdoproject.org/r/23386 | 09:28 |
| jpena | there was a typo in the definition | 09:28 |
| *** sshnaidm|off is now known as sshnaidm|rover | 09:30 | |
| *** dsneddon_ has joined #oooq | 09:34 | |
| arxcruz | jpena: jesus christ! lol | 09:35 |
| *** dsneddon_ has quit IRC | 09:38 | |
| *** irclogbot_3 has quit IRC | 09:39 | |
| *** irclogbot_0 has joined #oooq | 09:40 | |
| chandankumar | arxcruz: https://review.rdoproject.org/r/#/c/23386/38/zuul.d/ceph-ansible.yaml@58 is there any patch pending there? | 09:40 |
| *** tosky has quit IRC | 09:41 | |
| arxcruz | chandankumar: i would like to finish the test with testproject first before get it merged | 09:42 |
| arxcruz | and have ykarel|lunch blessing on those | 09:42 |
| chandankumar | arxcruz: ok | 09:42 |
| *** tosky has joined #oooq | 09:42 | |
| *** ksambor has quit IRC | 09:42 | |
| *** apetrich has quit IRC | 09:47 | |
| *** jaosorior has joined #oooq | 09:51 | |
| *** tosky_ has joined #oooq | 09:52 | |
| *** tosky has quit IRC | 09:54 | |
| *** tosky has joined #oooq | 09:58 | |
| *** apetrich has joined #oooq | 09:59 | |
| *** tosky_ has quit IRC | 10:01 | |
| *** tosky_ has joined #oooq | 10:05 | |
| *** tosky has quit IRC | 10:08 | |
| *** rfolco has joined #oooq | 10:08 | |
| *** soniya29 has joined #oooq | 10:09 | |
| *** tosky has joined #oooq | 10:14 | |
| *** tosky_ has quit IRC | 10:17 | |
| *** tosky_ has joined #oooq | 10:21 | |
| *** tosky has quit IRC | 10:24 | |
| *** panda has quit IRC | 10:24 | |
| *** ratailor_ has joined #oooq | 10:26 | |
| *** ratailor has quit IRC | 10:27 | |
| *** panda has joined #oooq | 10:29 | |
| *** tosky has joined #oooq | 10:34 | |
| *** dsneddon_ has joined #oooq | 10:35 | |
| *** tosky_ has quit IRC | 10:36 | |
| *** soniya29 has quit IRC | 10:37 | |
| *** dsneddon_ has quit IRC | 10:39 | |
| *** tosky_ has joined #oooq | 10:43 | |
| *** tosky has quit IRC | 10:45 | |
| *** ykarel|lunch has quit IRC | 10:46 | |
| *** tosky_ has quit IRC | 10:46 | |
| *** tosky has joined #oooq | 10:46 | |
| *** bhagyashris has quit IRC | 10:54 | |
| *** ratailor__ has joined #oooq | 10:55 | |
| *** ratailor_ has quit IRC | 10:57 | |
| *** tosky has quit IRC | 11:01 | |
| *** tosky has joined #oooq | 11:01 | |
| *** tosky_ has joined #oooq | 11:05 | |
| *** tosky has quit IRC | 11:06 | |
| *** chem is now known as chem|brb | 11:07 | |
| *** tosky_ has quit IRC | 11:09 | |
| *** tosky has joined #oooq | 11:09 | |
| *** jaosorior has quit IRC | 11:10 | |
| *** ksambor has joined #oooq | 11:10 | |
| zbr | chandankumar: marios please add a +W to https://review.opendev.org/#/c/692440/ | 11:11 |
| marios | ack zbr | 11:14 |
| *** ksambor has quit IRC | 11:15 | |
| *** ksambor has joined #oooq | 11:15 | |
| *** ykarel|lunch has joined #oooq | 11:15 | |
| *** ksambor has quit IRC | 11:16 | |
| *** udesale has quit IRC | 11:17 | |
| *** tosky has quit IRC | 11:17 | |
| *** ykarel|lunch is now known as ykarel | 11:18 | |
| *** chem|brb is now known as chem | 11:20 | |
| *** tosky has joined #oooq | 11:23 | |
| bogdando | hi folks | 11:24 |
| bogdando | this rework of container image uploader https://review.opendev.org/#/c/687288/ improves the CI cases somewhat | 11:24 |
| bogdando | PTAL | 11:24 |
| bogdando | weshay|rover: ^^ | 11:24 |
| bogdando | I provided benchmarking results as well | 11:24 |
| bogdando | https://docs.google.com/document/d/1H-UYr2_hqCHZqlOCDJ95aMwcwGb6NOF4BxCprz9Yn7U/edit#heading=h.85nezhhbr3ot | 11:25 |
| bogdando | tl;dr 10/36% faster and connects docker.io 20-39% less | 11:26 |
| marios | bogdando: nice adding to my reviews list | 11:26 |
| *** ksambor has joined #oooq | 11:33 | |
| *** ksambor has quit IRC | 11:33 | |
| *** dsneddon_ has joined #oooq | 11:35 | |
| *** ksambor has joined #oooq | 11:36 | |
| *** ksambor has quit IRC | 11:36 | |
| *** dsneddon_ has quit IRC | 11:40 | |
| *** surpatil has quit IRC | 11:40 | |
| *** tosky_ has joined #oooq | 11:47 | |
| *** ykarel_ has joined #oooq | 11:49 | |
| *** tosky has quit IRC | 11:49 | |
| *** tosky has joined #oooq | 11:50 | |
| *** ykarel has quit IRC | 11:51 | |
| *** tosky_ has quit IRC | 11:53 | |
| Tengu | bogdando: "podman rm -fa" and "podman rmi -fa" will drop all containers and all images - no need to list them. That's a new feature compared to docker :). | 11:54 |
| Tengu | and find has "-delete" instead of the -exec rm :3 | 11:54 |
| bogdando | Tengu: nice to know thanks! ) | 11:55 |
| bogdando | I knew some folks will start reading it from the bottom up | 11:55 |
| Tengu | :] | 11:59 |
| Tengu | when I can get some numbers and benchmarks, I love understanding the process ;) | 11:59 |
| *** surpatil has joined #oooq | 12:00 | |
| *** dsneddon_ has joined #oooq | 12:00 | |
| sshnaidm|rover | Tengu, bogdando where do you use "podman rmi -fa" ? | 12:02 |
| weshay|rover | sshnaidm|rover, hey | 12:02 |
| sshnaidm|rover | weshay|rover, hi | 12:02 |
| sshnaidm|rover | weshay|rover, mtg? | 12:02 |
| bogdando | sshnaidm|rover: for the aforementioned benchmark scenarios teardown steps | 12:02 |
| weshay|rover | sshnaidm|rover, ya.. just pouring a cup of coffee | 12:02 |
| weshay|rover | be there in a sec | 12:02 |
| bogdando | Tengu: note that some cases require docker distribution image filtered out | 12:03 |
| sshnaidm|rover | weshay|rover, if you want to switch to it, tell me | 12:03 |
| sshnaidm|rover | weshay|rover, "to meet" | 12:03 |
| weshay|rover | https://meet.google.com/diq-fpug-jxe?authuser=1 | 12:04 |
| weshay|rover | chandankumar, https://docs.google.com/document/d/1o-jj5RnP4eJsifWn1O1HKu9VoU1RPKHqj17Pc8rx2iM/edit | 12:05 |
| Tengu | sshnaidm|rover: in the doc provided by bogdando. | 12:10 |
| Tengu | bogdando: care to let ppl comment maybe? | 12:12 |
| chandankumar | weshay|rover: regarding Storage / Upgrades DFG and I checked with christin and he told it will not impact any issue related to rings but having a upgrade job would be good | 12:13 |
| weshay|rover | sshnaidm|rover, https://docs.openstack.org/sushy-tools/latest/ | 12:14 |
| weshay|rover | https://docs.openstack.org/sushy/latest/contributor/index.html#contributing | 12:15 |
| weshay|rover | https://github.com/openstack/sushy | 12:15 |
| weshay|rover | https://docs.openstack.org/sushy-tools/latest/user/dynamic-emulator.html#systems-resource-driver-openstack | 12:15 |
| weshay|rover | https://docs.google.com/document/d/1ghuoh2vLerzuYHDcHH_3t5efYHm79owImOnn8-iyUVM/edit | 12:17 |
| weshay|rover | sshnaidm|rover, ^ | 12:17 |
| bogdando | Tengu: done | 12:17 |
| Tengu | bogdando: thanks! adding some comments, feel free to drop them if not relevant. | 12:19 |
| bogdando | Tengu: thanks, appreciated | 12:19 |
| Tengu | bogdando: well, thank you for your work in getting better perfs, AND getting numbers | 12:19 |
| *** jpena is now known as jpena|lunch | 12:26 | |
| *** amoralej is now known as amoralej|lunch | 12:27 | |
| chandankumar | weshay|rover: Do we want to include collect-logs discussion also in the google docs? | 12:28 |
| *** ykarel_ is now known as ykarel|afk | 12:28 | |
| weshay|rover | chandankumar, if you want to fill that out.. sure.. but I don't have the details | 12:30 |
| chandankumar | weshay|rover: I have covered in my report | 12:30 |
| chandankumar | will keep some stuff seperate | 12:30 |
| chandankumar | here is my report https://hackmd.io/3XPZZF6-T_CEjnPLMX48pQ | 12:30 |
| Tengu | bogdando: sooo if I understand correctly, your patch makes the whole thing even faster than docker-ditribution? | 12:30 |
| chandankumar | weshay|rover: rest of the stuff looks good. | 12:34 |
| weshay|rover | ok.. cool | 12:34 |
| weshay|rover | chandankumar, /me reads yours | 12:34 |
| *** ssbarnea has quit IRC | 12:38 | |
| *** ratailor__ has quit IRC | 12:39 | |
| zbr | every day i make new discoveries, not all good: https://github.com/openstack/tripleo-ansible/blob/master/scripts/run-local-test#L60-L62 | 12:49 |
| *** ykarel_ has joined #oooq | 12:49 | |
| zbr | WTF is someone wiping my local wheel cache? what other surprised should I expect? | 12:50 |
| *** rlandy has joined #oooq | 12:50 | |
| *** ykarel|afk has quit IRC | 12:52 | |
| sshnaidm|rover | weshay|rover, you can change the nick :) | 12:52 |
| *** ykarel_ is now known as ykarel|afk | 12:53 | |
| *** rlandy is now known as rlandy|ruck | 12:54 | |
| *** weshay|rover is now known as weshay | 12:54 | |
| weshay | thanks | 12:54 |
| rlandy|ruck | sshnaidm|rover: do you want to switch over as we have a week left? | 12:54 |
| sshnaidm|rover | rlandy|ruck, ya, np | 12:55 |
| *** sshnaidm|rover is now known as sshnaidm|ruck | 12:55 | |
| *** rlandy|ruck is now known as rlandy|over | 12:55 | |
| *** rlandy|over is now known as rlandy|rover | 12:56 | |
| weshay | panda, howdy.. we should follow up today /me wants to hear your thoughts | 12:56 |
| rlandy|rover | not that it makes much difference though | 12:56 |
| sshnaidm|ruck | rlandy|rover, :D | 12:56 |
| weshay | re: the component pipeline | 12:56 |
| rlandy|rover | we're both on on our timeslots | 12:56 |
| sshnaidm|ruck | rlandy|rover, wanna sync? | 12:56 |
| rlandy|rover | sshnaidm|ruck: sure in 5 - just reading through email | 12:57 |
| *** jaosorior has joined #oooq | 12:57 | |
| sshnaidm|ruck | rlandy|rover, ack | 12:57 |
| rfolco | panda, marios my understanding from yesterday discussion: | 12:57 |
| rfolco | https://drive.google.com/file/d/1p9UFJNFKKKiOQB7N08TTHfdox1HghL35/view | 12:57 |
| chandankumar | weshay: please let me know anything I can improve in the report | 12:57 |
| rfolco | https://drive.google.com/file/d/1Ry8Mlvl-XqdqxOshtQo4NYf6jbK6I9ce/view | 12:57 |
| weshay | chandankumar, looked good :) send it | 12:58 |
| chandankumar | weshay: sure | 12:58 |
| weshay | I'm going to send mine as well | 12:58 |
| rfolco | panda, marios: can you please take a look and tell if I got this right? | 12:58 |
| weshay | chandankumar++ | 12:58 |
| marios | rfolco: ack | 12:58 |
| marios | rfolco: we discuss on call today | 12:59 |
| rfolco | marios, cool | 12:59 |
| marios | rfolco: thanks always nicer to discuss with a picture :) | 12:59 |
| rfolco | chandankumar, weshay, bug triage or not bug triage, trick or treat? | 13:00 |
| *** surpatil has quit IRC | 13:00 | |
| weshay | rfolco, chandankumar let's chat for a few.. then I need to speak w/ panda | 13:00 |
| rfolco | k | 13:00 |
| weshay | rfolco, chandankumar https://meet.google.com/unv-qweu-tdp?authuser=1 | 13:01 |
| weshay | panda, you around? | 13:02 |
| rlandy|rover | sshnaidm|ruck: k - ready when you are | 13:02 |
| *** soniya29 has joined #oooq | 13:03 | |
| sshnaidm|ruck | rlandy|rover, https://meet.google.com/jzy-uzmx-xsa | 13:03 |
| *** dsneddon_ has quit IRC | 13:03 | |
| rlandy|rover | matbu: hi sshnaidm|ruck and I wanted to touch base on the train upgrades test - we got pinged about the card yesterday. if there's any progress or anything you need from our side let us know - thanks | 13:08 |
| rlandy|rover | https://trello.com/c/f1inJq4M/1205-cixlp1850983tripleociproa-train-upgrades-periodic-tripleo-ci-centos-7-standalone-upgrade-train-is-failing-error-configuring-swif | 13:08 |
| panda | weshay: yes | 13:09 |
| weshay | panda, ok.. maybe in 5-10 min | 13:10 |
| *** soniya29 has quit IRC | 13:22 | |
| chandankumar | report sent | 13:25 |
| weshay | panda, https://meet.google.com/gam-npur-zyi?authuser=1 | 13:26 |
| weshay | chandankumar, thanks | 13:26 |
| *** jpena|lunch is now known as jpena | 13:27 | |
| sshnaidm|ruck | rlandy|rover, we need to do some maintenance in sova: http://logs.rdoproject.org/openstack-periodic-24hr/review.rdoproject.org/rdo-infra/ci-config/master/sova-tracking-jobs/8f98661/job-output.txt | 13:27 |
| panda | weshay: isn't there a community meeting in 3 minutes ? | 13:27 |
| weshay | oh crap.. sorry.. you are right | 13:28 |
| sshnaidm|ruck | rlandy|rover, I'm gonna restart the cockpit right now, will be unavailable for a while | 13:28 |
| rlandy|rover | sshnaidm|ruck: ok | 13:28 |
| weshay | panda, ok.. let's see what's going on there | 13:28 |
| *** amoralej|lunch is now known as amoralej | 13:39 | |
| *** ksambor has joined #oooq | 13:41 | |
| marios | my connection is coming/going on google meet :/ | 13:41 |
| marios | rejoining... | 13:41 |
| *** ksambor has quit IRC | 13:43 | |
| mjturek | is the ci community sync happening? | 13:44 |
| weshay | mjturek, yes | 13:45 |
| *** rlandy|rover is now known as rlandy|rover|mtg | 13:45 | |
| panda | mjturek: yes, but not in the classic bluejeans link | 13:45 |
| mjturek | weshay panda can you link me? | 13:45 |
| weshay | https://meet.google.com/bqx-xwht-wky?authuser=1 | 13:45 |
| mjturek | baha ^ | 13:46 |
| *** Goneri has joined #oooq | 13:48 | |
| bogdando | Tengu: not really, docker distribution only used for local2 storage | 13:49 |
| bogdando | I have no data to compare how fast those are | 13:50 |
| bogdando | but some deduplication layer info presents | 13:50 |
| bogdando | Tengu: I compare data for unpatched image-serve vs the patched | 13:50 |
| Tengu | bogdando: hmm there are numbers for local2 :). but anyway, it seems to be pretty fast, at least faster than current code. | 13:52 |
| bogdando | Tengu: yes, those are control numbers to check if no layers was lost | 13:52 |
| bogdando | Tengu: yea, faster and does better cross-linking, also less times connects external registries, like docker.io | 13:53 |
| Tengu | yup | 13:53 |
| Tengu | sounds good :) | 13:53 |
| bogdando | also, I saw some large layers ~200m are getting 8-10 times re-fetched currently, that's the main bug was targeted for addressing | 13:53 |
| bogdando | now solved! | 13:54 |
| Tengu | :) | 13:54 |
| weshay | mjturek, these guys are diving into unrelated work from multi-arch | 13:54 |
| mjturek | yeaaah baha and I will reach out later | 13:54 |
| weshay | mjturek, ok.. anything specific we can address atm? | 13:56 |
| mjturek | weshay: we're getting a false return code of 1, causing the build to be marked as failure. The failure is due to not being able to reach a mirror https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1737/logs/logs/buildah-builds/kolla-vUE2xO/docker/cinder/cinder-api/cinder-api-build.log | 13:57 |
| mjturek | but it succeeds and the container builds and publishes fine | 13:58 |
| mjturek | it's always a random container it happens to | 13:58 |
| mjturek | in short we have a false failure that we need to overcome to finally publish | 13:58 |
| *** rlandy|rover|mtg is now known as rlandy|rover | 13:59 | |
| weshay | hrm.. we have similiar issues from time to time w/ mirrors and proxies | 13:59 |
| weshay | perhaps our ruck/rovers can help | 13:59 |
| *** dsneddon_ has joined #oooq | 14:00 | |
| *** ykarel|afk is now known as ykarel | 14:02 | |
| mjturek | okay fair enough, this is consistent, let me know if you have an idea on who can help | 14:02 |
| sshnaidm|ruck | mjturek, Could not resolve host: mirror1.centos.org; Unknown error | 14:02 |
| sshnaidm|ruck | mjturek, it doesn't exist | 14:03 |
| rlandy|rover | sshnaidm|ruck: I'm going to merge this rhos-16 scenario004 job https://code.engineering.redhat.com/gerrit/#/c/185113 | 14:03 |
| *** ssbarnea has joined #oooq | 14:03 | |
| mjturek | sshnaidm|ruck fair enough but we try another mirror and succeed | 14:03 |
| mjturek | should we really be failing? | 14:03 |
| sshnaidm|ruck | mjturek, this host just doesn't exist | 14:04 |
| sshnaidm|ruck | mjturek, why is it there | 14:04 |
| mjturek | I have no idea | 14:05 |
| baha | Right, we're not putting it there | 14:05 |
| *** dsneddon_ has quit IRC | 14:05 | |
| sshnaidm|ruck | need to check that | 14:05 |
| mjturek | baha sshnaidm|ruck: hmmm a little more digging and it doesn't seem to be the cause of the false failure | 14:07 |
| mjturek | I have no idea why we're returning 1 at the end of the build. | 14:07 |
| sshnaidm|ruck | rfolco, take a look please at mail, EmilienM should have sent you invite about this paunch stuff, can you schedule a mtg for all of us? | 14:07 |
| sshnaidm|ruck | mjturek, this build seems to finish successfully | 14:08 |
| sshnaidm|ruck | mjturek, where do you see exit code 1? | 14:08 |
| mjturek | sshnaidm|ruck here's the exit code https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1737/logs/logs/build.log | 14:09 |
| chandankumar | marios: sorry I need to leave little early, will follow the taiga board | 14:09 |
| marios | ack chandankumar | 14:09 |
| *** ykarel_ has joined #oooq | 14:19 | |
| *** ykarel has quit IRC | 14:22 | |
| rlandy|rover | sshnaidm|ruck: going to ping on https://bugs.launchpad.net/tripleo/+bug/1851847 | 14:29 |
| openstack | Launchpad bug 1851847 in tripleo "rhel-8-scenario004 fails to deploy standalone - error while resolving custom fact \"rabbitmq_nodename\": undefined method `[]' for nil:NilClass"" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy) | 14:29 |
| sshnaidm|ruck | rlandy|rover, ack | 14:30 |
| *** ksambor has joined #oooq | 14:31 | |
| *** TrevorV has joined #oooq | 14:39 | |
| weshay | paging mr panda | 14:44 |
| weshay | mr panda come in please | 14:45 |
| weshay | sorry.. is | 14:46 |
| weshay | 熊猫 | 14:46 |
| weshay | there? | 14:46 |
| panda | weshay: ? | 14:47 |
| weshay | panda, just want to chat for 5 min https://meet.google.com/zeq-sdhy-ryg?authuser=1 | 14:47 |
| *** dtrainor has quit IRC | 14:50 | |
| *** dtrainor has joined #oooq | 14:50 | |
| *** ykarel_ is now known as ykarel|afk | 14:59 | |
| *** tosky_ has joined #oooq | 15:00 | |
| *** tosky has quit IRC | 15:00 | |
| *** dsneddon_ has joined #oooq | 15:01 | |
| *** tosky_ is now known as tosky | 15:01 | |
| sshnaidm|ruck | rfolco, shouldn't be a promoter call now? | 15:04 |
| *** dsneddon_ has quit IRC | 15:06 | |
| *** ksambor has quit IRC | 15:14 | |
| rfolco | sshnaidm|ruck, I'm sorry, we made it right after community one. I apologize, I did not notice you weren't in the community call. | 15:22 |
| *** rfolco is now known as rfolco|brbr | 15:22 | |
| *** rfolco|brbr is now known as rfolco|brb | 15:22 | |
| panda | rfolco|brb: we'll have to redo it. I was in another meeting ... | 15:25 |
| mjturek | sshnaidm|ruck: did you get a chance to look at the exit code thing I linked? | 15:26 |
| panda | uh, quay.io is now open source. | 15:27 |
| panda | marios: rfolco|brb | 15:27 |
| marios | panda: o/ | 15:27 |
| panda | what's Michal nick ? | 15:27 |
| marios | panda: were you really in another meeting? /me was wondering why you were so quiet ! | 15:27 |
| marios | mpryc i think? | 15:27 |
| marios | but not sure | 15:27 |
| panda | marios: it's migi.\ | 15:29 |
| marios | panda: tx | 15:29 |
| chandankumar | jpena: panda: Do we want to replace rdo registry with quay open source version? | 15:29 |
| jpena | chandankumar: we definitely want to replay rdo registry with something else. Not sure if quay will be the answer (need to investigate) | 15:30 |
| chandankumar | jpena: ack, it would be good to include that as a part of reproducer so that we can test registry secrets locally earlier it was a pain | 15:31 |
| *** tosky has quit IRC | 15:31 | |
| *** tosky has joined #oooq | 15:32 | |
| panda | chandankumar: let's do it. Fire up a shared tmux session | 15:33 |
| chandankumar | panda: code is not yet available so we cannot do anything | 15:34 |
| panda | uhm | 15:35 |
| panda | marios: do you have a summary of the promoter sync ? | 15:36 |
| marios | panda: no | 15:37 |
| panda | marios: any action item ? | 15:37 |
| marios | panda: yes "do it" | 15:38 |
| marios | panda: most significant thing for me was what i shared re the 'hashes' 17:24 <panda> anything new ? did you update the US ? | 15:38 |
| marios | 17:26 <marios> nothing spectacular, added note there https://tree.taiga.io/project/tripleo-ci-board/task/1385?kanban-status=1447274 | 15:38 |
| marios | panda: lets talk in one place | 15:40 |
| marios | panda: so i don't know why that call was named 'promoter' indeed we were talking about the component ci pipeline | 15:40 |
| marios | panda: we didn't discuss promoter actually | 15:40 |
| marios | rfolco|brb: ^ sshnaidm|ruck ^ | 15:40 |
| panda | rfolco|brb: marios there was a meeting called "promoter" supposed to start 40 minutes ago | 15:41 |
| marios | panda: yeah i know sshnaidm|ruck was asking about it and i tried joining like 7 mins passed but noone there | 15:41 |
| marios | panda: am not sure if it was confusion about the call topic or we just missed a call :D | 15:42 |
| marios | panda: but we were all on a call that overlapped so maybe that's why the confusion | 15:42 |
| panda | all right | 15:42 |
| marios | panda: rfolco|brb maybe we should move that promoter call tomorrow then ... i think we wanted to jump on the box and see what we can do re the new promoter | 15:42 |
| panda | I'll propose something for tomorrow. | 15:42 |
| panda | marios: or marios, you're good a proposing | 15:43 |
| marios | panda: why would i deny you that pleasure. i insist | 15:43 |
| sshnaidm|ruck | mjturek, I couldn't see the failure there, seems like something internal in kolla.. Was there second run of the job?\ | 15:47 |
| sshnaidm|ruck | mjturek, is it reproducible? | 15:47 |
| mjturek | sshnaidm|ruck: it happens every build at a random time | 15:58 |
| sshnaidm|ruck | mjturek, which job is it? | 15:58 |
| mjturek | https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/ | 15:58 |
| mjturek | sshnaidm|ruck any idea what log might have relevant info? | 15:59 |
| marios | weshay: do we have a call now? i have it in my calendar but its a new one you sent yesterday ("Component testing ci project") ... but seems to be for a wider group than ours? | 16:01 |
| *** dsneddon_ has joined #oooq | 16:02 | |
| amoralej | rlandy|rover, wrt the problem with scenario004, maybe you already discussed but the problem in latest executions of periodic looks a different one | 16:02 |
| amoralej | not https://bugs.launchpad.net/tripleo/+bug/1851847 | 16:02 |
| openstack | Launchpad bug 1851847 in tripleo "rhel-8-scenario004 fails to deploy standalone - error while resolving custom fact \"rabbitmq_nodename\": undefined method `[]' for nil:NilClass"" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy) | 16:02 |
| * rlandy|rover checks | 16:03 | |
| rlandy|rover | as in a different failure | 16:04 |
| rlandy|rover | we didn't really conclude that discussion | 16:04 |
| rlandy|rover | still looks to me like we are failing out on some combination of pacemaker/haproxy | 16:05 |
| mjturek | rfolco|brb: Now that we have the buildah-builds folder working, do you see why we're getting a return code of 1? https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1737/logs/logs/ | 16:05 |
| *** ykarel|afk is now known as ykarel | 16:05 | |
| rlandy|rover | amoralej: ^^ I'll change the bug description because that's not the fundamental error | 16:05 |
| amoralej | rlandy|rover, i'd say the problem in periodic is with pacemaker/mysql | 16:06 |
| *** dsneddon_ has quit IRC | 16:07 | |
| rlandy|rover | I think we are more concenred with periodic than check atm | 16:07 |
| rlandy|rover | because we have not promoted rhel-8 master in a long time | 16:07 |
| sshnaidm|ruck | mjturek, difficult to say, I see errors like "scriptlet failed, exit status 1" but it doesn't seem as critical: https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1739/logs/logs/containers-consolidated-builds.log | 16:08 |
| amoralej | rlandy|rover, i think problem in check can be related to versions missmatch | 16:08 |
| amoralej | and a promotion will fix it | 16:08 |
| amoralej | i'd try to fix promotion | 16:08 |
| amoralej | i just added some comment | 16:09 |
| rlandy|rover | amoralej: totally - once we get a promotion through check should deal with the problem | 16:09 |
| amoralej | and i'd see the key error here is "podman(galera-bundle-podman-0)[80315]: ERROR: Error: error checking path "/var/log/mariadb": stat /var/log/mariadb: no such file or directory" | 16:09 |
| rlandy|rover | which is why I am only concerned with getting scenario004 and ovb passing in the promotion job | 16:09 |
| rlandy|rover | amoralej: is it worth repinning some of the puppet versions? | 16:11 |
| amoralej | not sure | 16:11 |
| *** ksambor has joined #oooq | 16:12 | |
| amoralej | it'd be good to get help from pidone | 16:12 |
| amoralej | i wouldn't know what to repin | 16:12 |
| amoralej | puppet-mysql is still pinned | 16:12 |
| rlandy|rover | ok - let me see who I can reach on #rhos-pidone channel | 16:13 |
| rlandy|rover | it's possible bandini is still working on it | 16:14 |
| rlandy|rover | amoralej: I don't think it sits with your team any more | 16:14 |
| amoralej | ok | 16:15 |
| *** jaosorior has quit IRC | 16:18 | |
| *** ksambor has quit IRC | 16:18 | |
| sshnaidm|ruck | rlandy|rover, maybe worth to promote rhel master w/o 004? | 16:20 |
| mjturek | sshnaidm|ruck: If all of the containers are building, I don't know what would be wrong | 16:20 |
| sshnaidm|ruck | mjturek, there is systemerror in the build.log | 16:21 |
| sshnaidm|ruck | mjturek, and seems like it comes from tripleoclient | 16:21 |
| mjturek | right, but it doesn't prevent any container from building | 16:21 |
| rlandy|rover | sec - updating bug report | 16:21 |
| sshnaidm|ruck | mjturek, but seems like that fails the script | 16:21 |
| sshnaidm|ruck | mjturek, I'd ask on #tripleo | 16:21 |
| mjturek | sshnaidm|ruck will do | 16:22 |
| weshay | nice marios++ | 16:23 |
| weshay | https://review.opendev.org/#/c/675313/ | 16:23 |
| rlandy|rover | ok - so https://bugs.launchpad.net/tripleo/+bug/1851847 has a more accurate description | 16:23 |
| openstack | Launchpad bug 1851847 in tripleo "periodic-tripleo-ci-rhel-8-scenario004-standalone-master fails to deploy standalone - pacemaker/mysql" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy) | 16:23 |
| weshay | sshnaidm|ruck, if you guys want to promote w/o 004 that's fine w/ me atm | 16:24 |
| rlandy|rover | sshnaidm|ruck: we would to promote w/o sc004 and ovb | 16:24 |
| rlandy|rover | weshay: ^^ | 16:24 |
| rlandy|rover | and then I expect the check tests will fail | 16:24 |
| rlandy|rover | but with the same error as periodic | 16:24 |
| rlandy|rover | we can do it though | 16:24 |
| rlandy|rover | update scenario 1, 2, 3 | 16:24 |
| rlandy|rover | nothing to lose | 16:25 |
| sshnaidm|ruck | rlandy|rover, exactly | 16:25 |
| weshay | hrm.. is the ovb issue the same as 004 | 16:25 |
| rlandy|rover | weshay: I think it's related | 16:25 |
| rlandy|rover | see the deploy log | 16:25 |
| sshnaidm|ruck | rlandy|rover, weshay at least we'll have 1 issue to fight with, not 2 | 16:25 |
| weshay | rlandy|rover, ok.. please just flip them back after the promotion | 16:26 |
| rlandy|rover | sshnaidm|ruck: you can tell me what you think about the ovb error | 16:27 |
| rlandy|rover | overcloud-controller-0 : ok=233 changed=141 unreachable=0 failed=1 skipped=454 rescued=0 ignored=0 | 16:27 |
| sshnaidm|ruck | rlandy|rover, link? | 16:27 |
| rlandy|rover | http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/7257166/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 16:27 |
| zbr | sshnaidm|ruck: are you aware of some logging issue that look like KeyError: 'user' -- https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_d37/693858/1/check/tripleo-ansible-centos-7-role-addition/d37a7b9/job-output.txt | 16:27 |
| rlandy|rover | fails out on the main controller node | 16:27 |
| rlandy|rover | 019-11-12 10:42:55 | "Error: unable to find resource 'redis-bundle'", | 16:28 |
| rlandy|rover | 2019-11-12 10:42:55 | "Error: unable to find resource 'haproxy-bundle'", | 16:28 |
| zbr | not sure if these are specific to this job but running locally work fine for me. | 16:28 |
| chandankumar | sshnaidm|ruck: rlandy|rover panda weshay we need to setup a pre interview prep for mon interview? | 16:29 |
| chandankumar | or do we want to go adhoc? | 16:29 |
| rlandy|rover | chandankumar" let's just hire him and skip the interview :) | 16:30 |
| chandankumar | rlandy|rover: it depends on weshay's decision :-) | 16:30 |
| weshay | vhat? | 16:31 |
| chandankumar | weshay: skiping the mon interview part and go ahead and hire. | 16:31 |
| weshay | panda chandankumar, rlandy|rover sshnaidm|ruck 5 min chat about the interview | 16:31 |
| weshay | I think it's you four | 16:31 |
| weshay | heh | 16:31 |
| *** tesseract has quit IRC | 16:32 | |
| chandankumar | weshay: link? | 16:32 |
| weshay | https://meet.google.com/zwd-rxdx-vta?authuser=1 | 16:32 |
| weshay | I'll be brief | 16:32 |
| sshnaidm|ruck | zbr, I saw this, wanted to ask you :) | 16:36 |
| zbr | sshnaidm|ruck: we need to narrow it down. | 16:36 |
| zbr | i am not sure where it comes from, if is specific to zuul, python, ansible or molecule. | 16:37 |
| *** marios is now known as marios|out | 16:41 | |
| *** jpena is now known as jpena|off | 16:46 | |
| *** skramaja has quit IRC | 16:46 | |
| *** marios|out has quit IRC | 16:49 | |
| chandankumar | Thanks everyone :-) | 16:50 |
| sshnaidm|ruck | rlandy|rover, it doesn't look well: http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/7257166/logs/undercloud/var/log/extra/podman/containers/neutron_api/stdout.log.txt.gz | 16:53 |
| rlandy|rover | will look in a sec - creating interview doc | 16:54 |
| sshnaidm|ruck | rfolco|brb, did you get EmilienM mail? | 16:54 |
| chandankumar | sshnaidm|ruck: just an odd question, what are the fs where we run ipv6 jobs? | 16:55 |
| weshay | chandankumar, sshnaidm|ruck rlandy|rover panda sent the resume | 16:55 |
| sshnaidm|ruck | chandankumar, fs? | 16:55 |
| chandankumar | featureset | 16:55 |
| sshnaidm|ruck | ykarel, do we miss some plugin/package there? http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/7257166/logs/undercloud/var/log/extra/podman/containers/neutron_api/stdout.log.txt.gz | 16:55 |
| sshnaidm|ruck | chandankumar, it's 035 iirc | 16:56 |
| sshnaidm|ruck | chandankumar, we run it in periodic | 16:56 |
| chandankumar | sshnaidm|ruck: do we have a standalone version of that? as in upstream alot of work done on IPv6 side it would be good have coverage in check job also so asked | 16:57 |
| sshnaidm|ruck | chandankumar, not that I'm aware of, only ovb. But in standalone there is no multiple network or network isolation, not sure what to test there | 16:57 |
| ykarel | sshnaidm|ruck, looks like some config issue | 16:58 |
| sshnaidm|ruck | chandankumar, ipv6 comes to the picture when it's used in ovb network setup | 16:58 |
| sshnaidm|ruck | chandankumar, there is nothing uses it in standalone | 16:58 |
| chandankumar | sshnaidm|ruck: is it possible to simulate in standlone if upstream infra supports ipv6 network? | 16:59 |
| sshnaidm|ruck | chandankumar, you don't choose where to run your jobs in upstream, though | 16:59 |
| chandankumar | sshnaidm|ruck: all queries pops up because of that https://governance.openstack.org/tc/goals/selected/train/ipv6-support-and-testing.html | 17:00 |
| rfolco|brb | sshnaidm|ruck, which email you are referring to? | 17:00 |
| * chandankumar not sure it is important for tripleo have coverage in upstream CI | 17:00 | |
| sshnaidm|ruck | rfolco|brb, about paunch meeting | 17:01 |
| sshnaidm|ruck | chandankumar, nice, I'm not sure we have ipv6 on overcloud there, worth to check, but usually 035 is the great candidate for that | 17:01 |
| chandankumar | sshnaidm|ruck: how much time fs035 takes? if less than 3 hrs, we can enable in upstream? | 17:02 |
| *** dtantsur is now known as dtantsur|afk | 17:02 | |
| sshnaidm|ruck | chandankumar, it's running in periodics because it takes long time, exactly as fs001 ovb job | 17:02 |
| chandankumar | topic for next tripleo meeting | 17:02 |
| sshnaidm|ruck | chandankumar, and nothing was broken in ipv6 in last 2 years | 17:03 |
| *** dsneddon_ has joined #oooq | 17:03 | |
| sshnaidm|ruck | chandankumar, so for reducing load on rdo cloud we put it on periodic only, and still nothing is broken :) | 17:03 |
| ykarel | sshnaidm|ruck, i see the same error in passing job as well | 17:03 |
| *** Tengu has quit IRC | 17:03 | |
| ykarel | that auth url one | 17:03 |
| sshnaidm|ruck | chandankumar, I wouldn't put it in check pipeline again, it doesn't worth it | 17:03 |
| chandankumar | sshnaidm|ruck: ok | 17:03 |
| ykarel | https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/255228c/logs/undercloud/var/log/extra/podman/containers/neutron_api/stdout.log.txt.gz | 17:03 |
| sshnaidm|ruck | ykarel, hmm.. so neutron api just doesn't work? | 17:04 |
| chandankumar | sshnaidm|ruck: thanks for the info, it answers my queries | 17:04 |
| ykarel | sshnaidm|ruck, if i see neutron logs, it's working https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/255228c/logs/undercloud/var/log/containers/neutron/server.log.txt.gz | 17:05 |
| *** Tengu has joined #oooq | 17:05 | |
| ykarel | but stdouts it's not, seems stdouts ones are non fatal | 17:05 |
| chandankumar | sshnaidm|ruck: on fs035, I will work on temepst side, to add appropriate tempest test and config to test the stuff | 17:05 |
| *** Tengu has quit IRC | 17:06 | |
| *** Tengu has joined #oooq | 17:07 | |
| sshnaidm|ruck | chandankumar, ack, I think we still run there ipv4 tests | 17:07 |
| chandankumar | https://opendev.org/openstack/tempest/src/branch/master/.zuul.yaml#L281 - needs to got the appropriate ipv6 tests and add there as well as move that fs to os_tempest | 17:08 |
| sshnaidm|ruck | chandankumar, it's weird that it comes only now, I remember writing ipv6 tempest tests yet in 2014 | 17:08 |
| *** dsneddon_ has quit IRC | 17:08 | |
| sshnaidm|ruck | ykarel, yeah, seems like a red herring | 17:09 |
| *** bogdando has quit IRC | 17:09 | |
| sshnaidm|ruck | chandankumar, hehe https://review.opendev.org/#/q/owner:sshnaidm%2540redhat.com+status:merged+project:openstack/tempest | 17:10 |
| *** ykarel is now known as ykarel|away | 17:11 | |
| chandankumar | sshnaidm|ruck: :-) nice, /me needs to take a look | 17:12 |
| chandankumar | rlandy|rover: sagi is expert in tempest :-) | 17:13 |
| chandankumar | proof is above | 17:13 |
| sshnaidm|ruck | chandankumar, not anymore :P | 17:13 |
| * rlandy|rover rejoins conversation here | 17:13 | |
| rlandy|rover | chandankumar: sshnaidm|ruck: panda: pls see email with hackmd link | 17:13 |
| chandankumar | rlandy|rover: yup, saw it , will act in morning :-) | 17:14 |
| chandankumar | rlandy|rover: thank you rlandy :-) | 17:14 |
| rlandy|rover | sshnaidm|ruck: sorry - to go back - fs001 - failure is different? | 17:15 |
| * rlandy|rover has been concentrating on scenario004 | 17:15 | |
| sshnaidm|ruck | rlandy|rover, trying to understand, there are unrelated issues.. | 17:15 |
| rlandy|rover | bt both do need to pass | 17:15 |
| chandankumar | rlandy|rover: sshnaidm|ruck arxcruz panda rfolco|brb zbr sent the ptg report on tripleo-ci-internal please ask queries so that we can work together to clear the AI | 17:15 |
| *** Tengu has quit IRC | 17:16 | |
| rlandy|rover | k | 17:17 |
| *** Tengu has joined #oooq | 17:18 | |
| chandankumar | sshnaidm|ruck: got the approriate regex for fs035 https://opendev.org/openstack/tempest/src/branch/master/tox.ini#L244 | 17:18 |
| chandankumar | smoke|ipv6|test_network_v6\ | 17:19 |
| chandankumar | may be using regex like v6 will do the job | 17:19 |
| chandankumar | envoking smoke tests will eat too much time | 17:20 |
| chandankumar | will get the patches up tomorrow | 17:25 |
| chandankumar | weshay: sshnaidm|ruck: do we need a RHEL8 job also based on fs035 ipv6? | 17:26 |
| weshay | chandankumar, meh | 17:26 |
| weshay | let's keep what we have until we start removing jobs | 17:27 |
| chandankumar | weshay: ok | 17:27 |
| weshay | chandankumar, replacing centos ovb jobs in train / master w/ rhel is fine by me | 17:27 |
| weshay | but we have a lot of working going on now.. and I don't want to stress the team | 17:27 |
| rlandy|rover | pld hold off until we get rhel sorted | 17:28 |
| rlandy|rover | rhel 8 master is a mess | 17:28 |
| weshay | chandankumar, so do as you wish there but consult your local ruck / rover first | 17:28 |
| rlandy|rover | train is fine though | 17:28 |
| weshay | ha | 17:28 |
| weshay | rlandy|rover, and I are of one mind | 17:28 |
| rlandy|rover | weshay: can we start promoting rhel 8 train with the old promoter | 17:28 |
| chandankumar | weshay: I will try to move current fs035 on os_tempest with proper ipv6 tempest tests on centos is this ok rlandy|rover ? | 17:29 |
| weshay | rlandy|rover, when things do slow down.. tearing out fs002 from train/master would be awesome | 17:29 |
| rlandy|rover | we had initially set it aside for rfolco|brb and friends | 17:29 |
| *** chandankumar is now known as rrrrrrrrrrrr | 17:29 | |
| * rlandy|rover kills rrrrrrrrrrrr | 17:30 | |
| rlandy|rover | r nicks are off limits | 17:30 |
| rrrrrrrrrrrr | not cats sitting on my keyword | 17:30 |
| * rfolco|brb gets popcorn | 17:30 | |
| *** rfolco|brb is now known as rfolco | 17:30 | |
| rrrrrrrrrrrr | rlandy|rover: kill -9 command not found | 17:31 |
| sshnaidm|ruck | rrrrrrrrrrrr, when we sort out rhel master and as periodic job too | 17:31 |
| rlandy|rover | lol | 17:31 |
| sshnaidm|ruck | rrrrrrrrrrrr, sudo kill rrrrrrrrrrrr | 17:31 |
| rrrrrrrrrrrr | sshnaidm|ruck: you donot have permission to operate this operation contact weshay | 17:31 |
| rlandy|rover | that's the spirit - pull rank | 17:32 |
| sshnaidm|ruck | when I use sudo I'm weshay | 17:32 |
| weshay | ? | 17:32 |
| sshnaidm|ruck | weshay, privileges escalation | 17:33 |
| rrrrrrrrrrrr | sshnaidm|ruck: need a CIX | 17:33 |
| zbr | sshnaidm|ruck: how about this https://zuul.opendev.org/t/openstack/build/236553c6a6d84d0788518f449f89bb18 | 17:34 |
| * rrrrrrrrrrrr applies kill -9 rrrrrrrrrrrr | 17:34 | |
| *** rrrrrrrrrrrr is now known as chandankumar | 17:34 | |
| zbr | these errors are so ftrustrating because i do not have any way to login to that node to see what mess is there, locally works well... | 17:35 |
| sshnaidm|ruck | zbr, yeah, the failure is interesting | 17:35 |
| sshnaidm|ruck | zbr, more errors and more red color.. | 17:36 |
| rlandy|rover | kick the test in rdocloud and hold the node | 17:36 |
| zbr | i need to go now, but i read the backlog | 17:36 |
| sshnaidm|ruck | zbr, can you rebase on https://review.opendev.org/#/c/688574/ ? | 17:37 |
| sshnaidm|ruck | zbr, ok, I'll do it | 17:37 |
| chandankumar | see ya tomorrow | 17:38 |
| *** chandankumar is now known as raukadah | 17:38 | |
| rlandy|rover | sshnaidm|ruck: I wonder if we kick ovb on master rhel-8 without ha if we would get a pass | 17:39 |
| raukadah | sorry again r | 17:39 |
| rlandy|rover | raukadah: it's fine - I'll get over it | 17:39 |
| *** dsneddon_ has joined #oooq | 17:39 | |
| rlandy|rover | one controller one compute no pacemaker | 17:40 |
| sshnaidm|ruck | rlandy|rover, mm.. not sure we have such featureset | 17:40 |
| *** akahat has quit IRC | 17:41 | |
| rlandy|rover | we don;t but it would take pacemaker out of the equation | 17:41 |
| rlandy|rover | we have the nodeset | 17:41 |
| sshnaidm|ruck | rlandy|rover, until it's fixed I'd rather make it non-voting or even remove from check | 17:41 |
| rlandy|rover | we could have to override the deploy | 17:41 |
| rlandy|rover | sshnaidm|ruck: let's try the promotion | 17:42 |
| rlandy|rover | I assume it will fail | 17:42 |
| rlandy|rover | at least it will be updated | 17:42 |
| sshnaidm|ruck | rlandy|rover, need to change criteria | 17:42 |
| rlandy|rover | sshnaidm|ruck: ack - getting on promotion server | 17:42 |
| sshnaidm|ruck | rlandy|rover, and remove 001 and sceanrio004 | 17:42 |
| rlandy|rover | weshay: ^^ want to join tmate ? | 17:42 |
| sshnaidm|ruck | rlandy|rover, why on server? | 17:42 |
| sshnaidm|ruck | rlandy|rover, not in repo? | 17:43 |
| rlandy|rover | sshnaidm|ruck: the old promoter is locked | 17:43 |
| rlandy|rover | it doesn;t update | 17:43 |
| sshnaidm|ruck | rlandy|rover, oh, right | 17:43 |
| sshnaidm|ruck | ack | 17:43 |
| rlandy|rover | you have to update the server itself | 17:43 |
| rlandy|rover | very safe :) | 17:43 |
| sshnaidm|ruck | rlandy|rover, I can help, let's do tmate | 17:43 |
| rlandy|rover | sshnaidm|ruck: k - it's simple but just to double check so I don;t mess up | 17:44 |
| sshnaidm|ruck | rlandy|rover, sure | 17:44 |
| *** ykarel|away has quit IRC | 17:44 | |
| *** dsneddon_ has quit IRC | 17:46 | |
| *** chem is now known as chem|eod | 17:46 | |
| rlandy|rover | sshnaidm|ruck: isn't it another holiday again today - elections? | 17:55 |
| sshnaidm|ruck | rlandy|rover, where? | 17:55 |
| rlandy|rover | israel | 17:56 |
| sshnaidm|ruck | rlandy|rover, in IL? we already had 2 elections in a row :D | 17:56 |
| rlandy|rover | ha thought it was happening again | 17:56 |
| sshnaidm|ruck | rlandy|rover, there is a time to avoid 3rd one, but not much left.. | 17:56 |
| rlandy|rover | ok - whatever | 17:56 |
| sshnaidm|ruck | turned to be a national sport | 17:57 |
| rlandy|rover | weshay: read back - ack on fs002 - adding to ruck/rover etherpad | 17:58 |
| sshnaidm|ruck | rlandy|rover, what's about 002? | 17:58 |
| rlandy|rover | <weshay> rlandy|rover, when things do slow down.. tearing out fs002 from train/master would be awesome | 17:58 |
| rlandy|rover | let's get rhel-8 right first | 17:59 |
| sshnaidm|ruck | rlandy|rover, but then need to point build images jobs to upload them to tripleo-ci-testing | 18:01 |
| rlandy|rover | not doing that noe | 18:07 |
| rlandy|rover | now | 18:07 |
| rlandy|rover | lunch brb | 18:07 |
| sshnaidm|ruck | rlandy|rover, yeah, let's do it in 2 weeks | 18:09 |
| weshay | zbr, when you open bugs | 18:30 |
| weshay | https://bugs.launchpad.net/tripleo/+bug/1852213 | 18:30 |
| openstack | Launchpad bug 1852213 in tripleo "pre-commit is not fully running on triple-ansible" [Medium,Confirmed] - Assigned to Sorin Sbarnea (ssbarnea) | 18:30 |
| weshay | status = triaged, importance can be what ever... always fill out the milestone | 18:30 |
| *** amoralej is now known as amoralej|off | 18:33 | |
| *** dsneddon_ has joined #oooq | 18:43 | |
| rlandy|rover | checking rhel 8 promotion | 18:47 |
| *** dsneddon_ has quit IRC | 18:48 | |
| rlandy|rover | File "/home/centos/ci-config/ci-scripts/dlrnapi_promoter/dlrnapi_promoter.py", line 167, in tag_containers | 18:48 |
| rlandy|rover | stderr=subprocess.STDOUT).split("\n") | 18:48 |
| rlandy|rover | File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output | 18:48 |
| rlandy|rover | raise CalledProcessError(retcode, cmd, output=output) | 18:48 |
| rlandy|rover | CalledProcessError: Command '[u'env', u'ANSIBLE_LOG_PATH=/home/centos/promoter_logs/container-push/20191112-183945.log', u'RELEASE=master', u'COMMIT_HASH=66d1776f1b992d3b5f593240f4a9bfa75e572f76', u'DISTRO_HASH=ae355860ad6402be31e6b265d9f6a6cafbbc876d', u'FULL_HASH=66d1776f1b992d3b5f593240f4a9bfa75e572f76_ae355860', u'PROMOTE_NAME=current-tripleo', u'SCRIPT_ROOT=/home/centos/ci-config/', u'DISTRO_NAME=rhel', | 18:48 |
| rlandy|rover | u'DISTRO_VERSION=8', u'ansible-playbook', u'/home/centos/ci-config/ci-scripts/container-push/container-push.yml']' returned non-zero exit status 2 | 18:48 |
| weshay | hrm | 18:48 |
| rlandy|rover | at least it tried to promote | 18:49 |
| weshay | rlandy|rover, k.. few min /me looks w/ you | 18:49 |
| weshay | need to fix one thing | 18:49 |
| rlandy|rover | weshay: wrt a solution no - it will need to go to CIX tomorrow to close out | 18:49 |
| rlandy|rover | bug has additional notes now | 18:49 |
| rlandy|rover | failed: [localhost] (item=openstack-base) => {"ansible_loop_var": "item", "changed": false, "item": "openstack-base", "msg": "Error pulling image trunk.registry.rdoproject.org/tripleomaster/rhel-binary-openstack-base:66d1776f1b992d3b5f593240f4a9bfa75e572f76_ae355860 - 500 Server Error: Internal Server Error (\"Get https://trunk.registry.rdoproject.org/v2/tripleomaster/rhel-binary-openstack-base/manifests/66d1776f1b992d3b5f59 | 18:50 |
| rlandy|rover | 3240f4a9bfa75e572f76_ae355860: unauthorized: authentication required\")"} | 18:50 |
| weshay | ya.. was going to point that out | 18:55 |
| weshay | rlandy|rover, there really is no need to pull containers to the promoter server for rhel 8 | 18:55 |
| weshay | rlandy|rover, let's wait for one more run | 18:56 |
| weshay | see if the auth error persists | 18:56 |
| rlandy|rover | weshay" ack - I don;t think anything else is up for promotion | 18:56 |
| rlandy|rover | so it shoudl run again soon | 18:56 |
| rlandy|rover | checking train | 18:56 |
| rlandy|rover | that may promote | 18:56 |
| rlandy|rover | both master an dtrain failed on fs039 | 18:58 |
| rlandy|rover | rerunning both | 18:59 |
| rlandy|rover | 2019-11-12 18:58:00,274 17784 INFO promoter Running: env ANSIBLE_LOG_PATH=/home/centos/promoter_logs/container-push/20191112-185800.log RELEASE=master COMMIT_HASH=66d1776f1b992d3b5f593240f4a9bfa75e572f76 DISTRO_HASH=ae355860ad6402be31e6b265d9f6a6cafbbc876d FULL_HASH=66d1776f1b992d3b5f593240f4a9bfa75e572f76_ae355860 PROMOTE_NAME=current-tripleo SCRIPT_ROOT=/home/centos/ci-config/ DISTRO_NAME=rhel DISTRO_VERSION=8 | 18:59 |
| rlandy|rover | ansible-playbook /home/centos/ci-config/ci-scripts/container-push/container-push.yml | 18:59 |
| rlandy|rover | weshay: ^^ running again | 18:59 |
| weshay | really? | 19:02 |
| weshay | oh I see now | 19:03 |
| weshay | :) | 19:03 |
| weshay | rlandy|rover, same issue | 19:03 |
| rlandy|rover | weshay: let's jump on the promoter | 19:05 |
| weshay | I'm there | 19:05 |
| rlandy|rover | joining | 19:06 |
| rlandy|rover | tmux? | 19:06 |
| weshay | I'm in | 19:06 |
| weshay | rlandy|rover, I think the registry is down | 19:08 |
| rlandy|rover | ha? checking | 19:08 |
| weshay | man.. maybe I will get permission to do that thing w/ containers we thought was bad | 19:08 |
| rlandy|rover | Application is not available | 19:09 |
| rlandy|rover | The application is currently not serving requests at this endpoint. It may not have been started or is still starting. | 19:09 |
| rlandy|rover | ugh | 19:09 |
| rlandy|rover | bad timing | 19:09 |
| rlandy|rover | pinging sf-ops | 19:09 |
| rlandy|rover | on the upside it may just work when registry is back | 19:10 |
| weshay | rlandy|rover, well.. not only that.. | 19:11 |
| weshay | rlandy|rover, /me wonders if this breaks check jobs | 19:12 |
| weshay | think it would | 19:12 |
| rlandy|rover | yeah but it hasn't been downs for weeks | 19:13 |
| rlandy|rover | because otherwise how did we promote anything???? | 19:14 |
| rlandy|rover | whatever - let's see what we get with a working reg | 19:14 |
| rlandy|rover | one problem at a time | 19:14 |
| *** dsneddon_ has joined #oooq | 19:20 | |
| weshay | rlandy|rover, is our baremetal doc in here? https://docs.openstack.org/tripleo-quickstart/latest/ | 19:28 |
| rlandy|rover | no - internal | 19:29 |
| sshnaidm|ruck | rlandy|rover, need to update ansible on out infra hosts.. will do it tomorrow maybe, after zbr fixed linters some stuff fails there | 19:39 |
| *** jaosorior has joined #oooq | 19:39 | |
| sshnaidm|ruck | because we have there 2.5 or kind of | 19:39 |
| sshnaidm|ruck | rlandy|rover, so till then repo changes might be not be reflected on hosts | 19:40 |
| rlandy|rover | sshnaidm|ruck: that's fine - the only critical one is the promoter | 19:40 |
| sshnaidm|ruck | need to update grafana for train too | 19:41 |
| sshnaidm|ruck | in cockpit I updated ansible, should be fine | 19:41 |
| rlandy|rover | that's fine | 19:41 |
| rlandy|rover | nothing critical atm | 19:41 |
| sshnaidm|ruck | rlandy|rover, out for today, see you tomorrow | 19:41 |
| rlandy|rover | sshnaidm|ruck: sure - see you then | 19:42 |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 19:42 | |
| rlandy|rover | oh gee - we got another problem | 19:47 |
| rlandy|rover | ovb | 19:47 |
| rlandy|rover | 2019-11-12 19:38:49.058033 | TASK [ovb-manage : Recover idnum] | 19:47 |
| rlandy|rover | 2019-11-12 19:38:49.590478 | primary | cat: /home/zuul/workspace/ovb/idnum: No such file or directory | 19:47 |
| rlandy|rover | 2019-11-12 19:39:00.104079 | primary | ERROR | 19:47 |
| rlandy|rover | 2019-11-12 19:39:00.104528 | primary | { | 19:47 |
| rlandy|rover | 2019-11-12 19:39:00.104658 | primary | "delta": "0:00:00.005345", | 19:47 |
| rlandy|rover | 2019-11-12 19:39:00.104735 | primary | "end": "2019-11-12 19:38:49.591838", | 19:47 |
| rlandy|rover | 2019-11-12 19:39:00.104800 | primary | "msg": "non-zero return code", | 19:48 |
| rlandy|rover | 2019-11-12 19:39:00.104862 | primary | "rc": 1, | 19:48 |
| rlandy|rover | 2019-11-12 19:39:00.104923 | primary | "start": "2019-11-12 19:38:49.586493" | 19:48 |
| rlandy|rover | 2019-11-12 19:39:00.104998 | primary | } | 19:48 |
| rlandy|rover | 2019-11-12 19:39:00.124714 | | 19:48 |
| rlandy|rover | 2019-11-12 19:38:07.028361 | TASK [Verify Login for docker] | 19:48 |
| rlandy|rover | 2019-11-12 19:38:07.115014 | primary | ERROR | 19:48 |
| rlandy|rover | 2019-11-12 19:38:07.115417 | primary | { | 19:48 |
| rlandy|rover | 2019-11-12 19:38:07.115520 | primary | "assertion": "\"unauthorized\" in registry_login_docker.results.0.stderr", | 19:48 |
| rlandy|rover | 2019-11-12 19:38:07.115590 | primary | "evaluated_to": false, | 19:48 |
| rlandy|rover | 2019-11-12 19:38:07.115653 | primary | "msg": "Role failed authentication for an Unknown reason." | 19:48 |
| rlandy|rover | 2019-11-12 19:38:07.115712 | primary | } | 19:48 |
| rlandy|rover | 2019-11-12 19:38:07.132589 | | 19:48 |
| rlandy|rover | oh - that may be registry | 19:48 |
| *** Goneri has quit IRC | 19:51 | |
| weshay | rlandy|rover, link? | 19:51 |
| rlandy|rover | https://review.rdoproject.org/r/#/c/21853/ | 19:52 |
| rlandy|rover | http://logs.rdoproject.org/53/21853/9/check/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-master/f23b7cb/ | 19:52 |
| rlandy|rover | weshay, registry login | 19:53 |
| rlandy|rover | let's try after registry is operational | 19:53 |
| weshay | rlandy|rover, if it's down at the end of our day.. let's bug + cix it | 19:54 |
| rlandy|rover | ack | 19:54 |
| rlandy|rover | I have faith though | 19:54 |
| *** jfrancoa has quit IRC | 19:59 | |
| rlandy|rover | hopefully the promoter will cycle again now | 20:08 |
| rlandy|rover | here we go again | 20:13 |
| *** jaosorior has quit IRC | 20:16 | |
| *** panda has quit IRC | 20:18 | |
| *** panda has joined #oooq | 20:21 | |
| rlandy|rover | weshay: can you access the registry now? | 20:45 |
| rlandy|rover | still returning Application is not available | 20:45 |
| * weshay checks | 20:46 | |
| rlandy|rover | the promotion looks like it's running though | 20:46 |
| weshay | rlandy|rover, same here | 20:46 |
| weshay | oh | 20:46 |
| rlandy|rover | clear cookies? | 20:46 |
| rlandy|rover | oh what? | 20:48 |
| rlandy|rover | ? | 20:52 |
| weshay | rlandy|rover, meh.. I think the web front end is down | 20:54 |
| weshay | it's not the cooooookies | 20:54 |
| rlandy|rover | I asked - we'll see | 20:54 |
| rlandy|rover | have no other good way to check progress | 20:54 |
| weshay | sshnaidm|afk, you da man :) https://blueprints.launchpad.net/tripleo/+spec/tripleo-operators-ansible | 20:58 |
| *** rfolco has quit IRC | 21:01 | |
| *** rfolco has joined #oooq | 21:01 | |
| *** Goneri has joined #oooq | 21:02 | |
| *** rfolco has quit IRC | 21:06 | |
| *** TrevorV has quit IRC | 21:11 | |
| rlandy|rover | promoter SUCCESS \0/ promoting (u'rhel', u'8') tripleo-ci-testing as current-tripleo ({'timestamp': 1573561233, 'distro_hash': 'ae355860ad6402be31e6b265│··································· | 21:43 |
| rlandy|rover | d9f6a6cafbbc876d', 'promote_name': 'tripleo-ci-testing', 'user': 'review_rdoproject_org', 'repo_url': 'http://trunk.rdoproject.org/rhel8-master/66/d1/66d1776f1b992d3b5f593240f4a9bfa75e572f76_│··································· | 21:43 |
| rlandy|rover | ae355860', 'full_hash': '66d1776f1b992d3b5f593240f4a9bfa75e572f76_ae355860', 'repo_hash': '66d1776f1b992d3b5f593240f4a9bfa75e572f76_ae355860', 'commit_hash': '66d1776f1b992d3b5f593240f4a9bfa7│··································· | 21:43 |
| rlandy|rover | 5e572f76'}) | 21:43 |
| rlandy|rover | OMG - finally | 21:43 |
| *** Goneri has quit IRC | 21:43 | |
| rlandy|rover | will be interesting to see what the check jobs do now | 21:45 |
| *** Goneri has joined #oooq | 21:46 | |
| *** jfrancoa has joined #oooq | 21:48 | |
| *** chem|eod has quit IRC | 21:48 | |
| *** Goneri has quit IRC | 21:57 | |
| *** Goneri has joined #oooq | 21:57 | |
| *** jfrancoa has quit IRC | 22:00 | |
| *** Goneri has quit IRC | 22:01 | |
| *** Goneri has joined #oooq | 22:01 | |
| rlandy|rover | 2019-11-12 22:44:13 | "Error: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Pacemaker::Property[haproxy-role-standalone]/Pcmk_property[property-standalone-haproxy-role]: Could not evaluate: backup_cib: Running: pcs cluster cib /var/lib/pacemaker/cib/puppet-cib-backup20191112-9-fqdbhh failed with code: 1 -> Error: unable to get cib", | 22:48 |
| rlandy|rover | a ha | 22:51 |
| rlandy|rover | dnf.exceptions.RepoError: Unknown repo: 'delorean-*-deps-a' | 22:51 |
| rlandy|rover | 2019-11-12T22:45:06Z CRITICAL Error: Unknown repo: 'delorean-*-deps-a' | 22:51 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!