| *** ysandeep|out is now known as ysandeep | 01:31 | |
| ysandeep | o/ good morning folks | 01:32 |
|---|---|---|
| dasm | ysandeep: \o | 01:56 |
| *** rlandy|bbl is now known as rlandy|out | 01:59 | |
| rlandy|out | hi ysandeep | 01:59 |
| ysandeep | good evening folks o/ | 01:59 |
| rlandy|out | pls review https://review.opendev.org/c/openstack/tripleo-ci/+/858768 for infra | 01:59 |
| ysandeep | ack | 02:00 |
| ysandeep | hmm, clark is removing 2.8 and 2.9, from where are we getting 2.12.9 here | 02:04 |
| ysandeep | 2022-09-21 16:42:35.357909 | TASK [emit-job-header : Print job information] | 02:04 |
| ysandeep | 2022-09-21 16:42:35.420589 | # Job Information | 02:04 |
| ysandeep | 2022-09-21 16:42:35.420903 | Ansible Version: 2.12.9 | 02:04 |
| ysandeep | https://d6341d649c554a5c4ee6-1f83b5006b42c57e1aaf7265351cee4a.ssl.cf5.rackcdn.com/858768/1/check/tripleo-ci-centos-9-scenario001-standalone/82786f2/job-output.txt | 02:04 |
| ysandeep | based on his commit - This changes removes the pins of jobs to those versions which | 02:04 |
| ysandeep | will cause the jobs to run against the current zuul tenant default | 02:04 |
| ysandeep | ansible (version 5) | 02:04 |
| ysandeep | ^^ shouldn't we get version 5 as per his commit msg. | 02:05 |
| ysandeep | ahh 2.8/2.9 is min version to use | 02:12 |
| ysandeep | commented for confirmation: https://review.opendev.org/c/openstack/tripleo-ci/+/858768/1#message-33c77d95538f5b7ae72c3242bc6c07078109b29b | 02:25 |
| *** ysandeep is now known as ysandeep|afk | 04:31 | |
| *** dasm is now known as dasm|off | 04:51 | |
| marios | \o | 05:12 |
| marios | akahat|ruck: all good? shout if need sthing | 07:18 |
| *** jpena|off is now known as jpena | 07:19 | |
| jm1 | moin :) | 07:25 |
| akahat|ruck | marios, o/ | 07:34 |
| akahat|ruck | marios, everything is fine for now.. Chasing with promotions upstream/downstream. | 07:34 |
| marios | ok akahat|ruck | 07:36 |
| marios | let me know if you want to discuss anything | 07:36 |
| akahat|ruck | marios, yes. sure. :) | 07:36 |
| *** amoralej|off is now known as amoralej | 07:37 | |
| marios | pojadhav: o/ | 07:43 |
| marios | pojadhav: see pvt when you have a minute | 07:43 |
| arxcruz | pojadhav hi, i won't participate on scrum today, i have dentist | 07:58 |
| pojadhav | arxcruz, ack thanks for update! | 07:59 |
| marios | arxcruz: join from dentist man come on! ;) | 08:08 |
| arxcruz | i can, but i will not be able to speak lol | 08:09 |
| marios | :) | 08:09 |
| Tengu | marios: heya! do you want someone else from CI to check on https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/857096 or may I request votes from my DF colleagues? | 08:12 |
| marios | thanks for checking Tengu - should be ok i think since you're already working with ysandeep|afk from ci team on that | 08:14 |
| *** ysandeep|afk is now known as ysandeep | 08:15 | |
| Tengu | marios: sounds good! I just answered your question on the other one. | 08:15 |
| ysandeep | marios, all jobs are green in testproject/on that patch - hopefully nothing will break. | 08:16 |
| ysandeep | Tengu++ | 08:16 |
| Tengu | :) | 08:16 |
| Tengu | that was a nice hunt. | 08:16 |
| Tengu | but with proper logging, everything's easier. | 08:16 |
| Tengu | that said... | 08:19 |
| Tengu | ysandeep: shouldn't we open the CI console stream for standalone as well? | 08:20 |
| Tengu | just asking.. and what about the ssh port for such case? | 08:20 |
| * Tengu worried | 08:20 | |
| Tengu | marios: you may have spot a good thing :) | 08:20 |
| Tengu | oh. it happens to have a standalone run. let's check | 08:21 |
| Tengu | er. not the right thing. | 08:21 |
| ysandeep | Tengu, console is working for standalone even we noticed some drops for 19885 port | 08:22 |
| Tengu | damned OVH network full of sh** | 08:22 |
| Tengu | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_02d/852808/13/check/tripleo-ci-centos-9-standalone/02dd30c/logs/undercloud/var/log/extra/dropped-packets.txt how are we supposed to find anything in there ?! | 08:22 |
| Tengu | ysandeep: if console is working on standalone, things should be fine. lemme check the actual rules. | 08:23 |
| Tengu | tcp dport 22 ct state new counter packets 151 bytes 9020 accept comment "003 accept ssh from all" | 08:23 |
| Tengu | perfect. | 08:23 |
| Tengu | ssh is open as expected | 08:23 |
| ysandeep | afair.. the drop for 19885(console) were SYN only | 08:24 |
| Tengu | I don't see how the console can work since the port isn't open......... | 08:24 |
| Tengu | yep, seems to be SYN only. | 08:24 |
| Tengu | oh. | 08:24 |
| Tengu | well. I think I know why it was working | 08:24 |
| Tengu | or still is | 08:24 |
| Tengu | the connection is a stream, created *before* the deploy actually starts. | 08:24 |
| Tengu | meaning: it's ESTABLISHED | 08:25 |
| Tengu | and we happen to have: ct state established,related counter packets 1569138 bytes 3737765498 accept comment "000 accept related established rules" | 08:25 |
| Tengu | so we're covered, unless the console stream cuts for #reason | 08:25 |
| Tengu | then, it will more than probably crash. | 08:25 |
| Tengu | *maybe* adding the rule, just in case... | 08:25 |
| ysandeep | Tengu, It should go in https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/standalone role | 08:27 |
| ysandeep | we use this template to create standalone_parameters.yaml: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/standalone/templates/standalone_config.yaml.j2 | 08:27 |
| ysandeep | maybe we can add it here | 08:27 |
| Tengu | ok. we can add the extrafirewall thingy in there. | 08:27 |
| Tengu | I'll do a new patch - the oooq-extras has just been nudged. | 08:28 |
| ysandeep | Tengu: thanks! | 08:30 |
| Tengu | ysandeep: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/858868 | 08:34 |
| Tengu | not 100% sure about the generated content, will wait for the CI to die. | 08:34 |
| ysandeep | Tengu, dropped packet - SRC=104.239.144.167 DST=198.72.124.73 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=35652 DF PROTO=TCP SPT=45938 DPT=19885 SEQ=2795943201 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A3A03424D0000000001030309) | 08:36 |
| ysandeep | SRC was not source_ci_network | 08:37 |
| Tengu | hmmmmm indeed. | 08:37 |
| Tengu | dang. where can we find this network now :/ | 08:37 |
| ysandeep | afaik.. I don't think we have that anywhere | 08:37 |
| Tengu | right... | 08:37 |
| Tengu | not a route, not a known host | 08:38 |
| ysandeep | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_02d/852808/13/check/tripleo-ci-centos-9-standalone/02dd30c/zuul-info/host-info.primary.yaml | 08:39 |
| ysandeep | ~~~ | 08:39 |
| ysandeep | SSH_CLIENT: 104.239.144.167 53988 22 | 08:39 |
| ysandeep | SSH_CONNECTION: 104.239.144.167 53988 198.72.124.73 22 | 08:39 |
| ysandeep | ~~~ | 08:39 |
| *** akahat|ruck is now known as akahat|ruck|lunch | 08:39 | |
| ysandeep | but I don't think we can pull these details from job | 08:39 |
| * ysandeep checks how we create host-info.primary.yaml | 08:40 | |
| Tengu | it's in ansible_env | 08:40 |
| Tengu | MAYBE? | 08:40 |
| ysandeep | Tengu, https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_02d/852808/13/check/tripleo-ci-centos-9-standalone/02dd30c/logs/undercloud/var/log/extra/dump_variables_hostvars.yaml | 08:40 |
| ysandeep | ~~~ | 08:41 |
| ysandeep | ansible_env: | 08:41 |
| Tengu | yeah | 08:41 |
| Tengu | it's in | 08:41 |
| Tengu | sooo.... errr... | 08:41 |
| Tengu | using https://docs.ansible.com/ansible/latest/collections/ansible/builtin/env_lookup.html maybe? | 08:42 |
| Tengu | and defaulting on the ci_network | 08:42 |
| ysandeep | I haven't tried that inside jinja.. not sure | 08:44 |
| Tengu | lemme do some local testing. | 08:45 |
| Tengu | and remote, since I'll need to get that SSH_CLIENT thing. | 08:45 |
| Tengu | AND parse it because it's not just the IP. | 08:45 |
| ysandeep | yeah, also we will need a condition to only do it for master | 08:45 |
| Tengu | ah, right. though... is it really important? | 08:46 |
| Tengu | I mean... it would just add the same thing in iptables. | 08:46 |
| Tengu | no harm. | 08:46 |
| ysandeep | yes, should not harm previous branches as well.. upto you to add/or not.. I was thinking about something like {% if release not in ['train', 'wallaby'] -%} | 08:49 |
| Tengu | interesting... in {% set foo = ... %} we can't use a string like "{{ foo | bar }}" | 08:50 |
| * ysandeep need to step out for lunch | 08:52 | |
| *** ysandeep is now known as ysandeep|lunch | 08:52 | |
| Tengu | ok, I think I get it. | 08:54 |
| *** akahat|ruck|lunch is now known as akahat|ruck | 09:31 | |
| *** ysandeep|lunch is now known as ysandeep | 10:01 | |
| marios | amoralej: o/ hey still have missing openstack-tempest-all are we waiting for something else ? | 10:16 |
| marios | amoralej: https://logserver.rdoproject.org/37/45037/4/check/periodic-tripleo-ci-build-containers-ubi-9-quay-push-zed/4534d3b/logs/container-builds/e0719dfb-25d3-43cb-b53a-39a4dea9b592/base/os/tempest/tempest-build.log | 10:16 |
| amoralej | mmm i expected it should be there | 10:16 |
| amoralej | lemme check | 10:16 |
| marios | amoralej: and repo there http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org:8080/rdo/centos9-zed/component/tempest/b3/9b/b39bbcc0bc5be39cc4d9e3cf2addf1924c8df49a_e68950ea | 10:16 |
| marios | ah hold one | 10:16 |
| marios | might be my bad amoralej | 10:16 |
| marios | i need to run the promotion again | 10:16 |
| marios | sorry :/ | 10:16 |
| amoralej | marios, yes, that :) | 10:17 |
| marios | i think i need to promote again the consistent->promoted-components->tripleo-ci-testing | 10:17 |
| marios | amoralej: ack | 10:17 |
| amoralej | it's in current https://trunk.rdoproject.org/centos9-zed/component/tempest/current/ | 10:17 |
| marios | thanks amoralej | 10:17 |
| amoralej | and consistent | 10:17 |
| afuscoar | marios: sorry, can u take a look again to this one? https://code.engineering.redhat.com/gerrit/c/openstack/rrcockpit/+/428675 idk if u have +2 | 10:18 |
| marios | afuscoar: i'll revisit but it is on the ci team review list so folks should be checking it today | 10:19 |
| afuscoar | that's perfect then, thanks marios! | 10:22 |
| *** rlandy|out is now known as rlandy|rover | 10:29 | |
| rlandy|rover | chandankumar: ysandeep: marios: hi - thanks for your reviews on https://review.opendev.org/c/openstack/tripleo-ci/+/858768 | 10:31 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 10:31 |
| rlandy|rover | we need to get a merge/fix in for opendev as zuul will update over the weekend | 10:31 |
| rlandy|rover | chandankumar: ysandeep: marios: can we meet at the usual review time today to discuss? this is time sensitive | 10:32 |
| rlandy|rover | ie: in about 45 mins | 10:32 |
| marios | rlandy|rover: sure works for me | 10:34 |
| ysandeep | ack | 10:34 |
| rlandy|rover | ty | 10:34 |
| rlandy|rover | akahat|ruck: hello - any news on downstream jenkins? | 10:35 |
| rlandy|rover | did you check with them? | 10:35 |
| rlandy|rover | also we need to promote wallaby today | 10:36 |
| akahat|ruck | rlandy|rover, o/ let's sync.. | 10:36 |
| akahat|ruck | rlandy|rover, yeah.. for promotion I'm running jobs and so far we are good. | 10:37 |
| rlandy|rover | akahat|ruck: did you talk with attila? | 10:37 |
| akahat|ruck | rlandy|rover, no. | 10:37 |
| rlandy|rover | if not, I need to talk to him first | 10:37 |
| rlandy|rover | ok - then I'll sync with you later | 10:37 |
| akahat|ruck | rlandy|rover, ok. | 10:37 |
| akahat|ruck | creating cix for 18.1 line. | 10:37 |
| akahat|ruck | 17.1 line * | 10:37 |
| arxcruz | dpawlik hey, check https://review.opendev.org/c/openstack/ci-log-processing/+/858373 | 10:46 |
| arxcruz | when you have time | 10:46 |
| arxcruz | frenzyfriday ˆ | 10:46 |
| * frenzyfriday looks | 10:47 | |
| *** ysandeep is now known as ysandeep|afk | 10:48 | |
| rlandy|rover | akahat|ruck: https://trunk.rdoproject.org/api-centos9-wallaby/api/civotes_agg_detail.html?ref_hash=3406eb4e4f80fb0dfb113893dc876677 | 10:51 |
| akahat|ruck | rlandy|rover, only fs35 is problem | 10:52 |
| akahat|ruck | It's quite unstable. | 10:52 |
| rlandy|rover | looks to be a promotable hash ... fs035 is failing but internal is passing - pls do this ... put in a temp criteria patch to switch fs035 internal for regulra fs035 | 10:52 |
| rlandy|rover | there is a real keystone issue going on | 10:52 |
| akahat|ruck | rlandy|rover, ack | 10:52 |
| rlandy|rover | pls do one more rerun of that job to check - with that hash | 10:52 |
| rlandy|rover | but put in the criteria patch first | 10:53 |
| rlandy|rover | akahat|ruck: pls dequeue and enqueue stable 1 | 10:58 |
| rlandy|rover | jobs will be skipped | 10:59 |
| rlandy|rover | restart that pls | 10:59 |
| akahat|ruck | rlandy|rover, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45215 | 11:01 |
| *** ysandeep|afk is now known as ysandeep | 11:01 | |
| akahat|ruck | testing here: https://code.engineering.redhat.com/gerrit/c/testproject/+/429379 | 11:02 |
| *** amoralej is now known as amoralej|lunch | 11:02 | |
| rlandy|rover | soniya29; had to move our 1-1 - sorry need to deal with a bunch of things now | 11:03 |
| soniya29 | rlandy|rover, ack, no problem | 11:03 |
| frenzyfriday | folks, the ER dashboard is up \O/ http://38.102.83.141/ | 11:09 |
| frenzyfriday | I'll start on the queries next so that we get some output in the next RR shift | 11:10 |
| *** frenzyfriday is now known as frenzyfriday|food | 11:11 | |
| marios | rlandy|rover: ysandeep: you have a meet or should i start one? here https://meet.google.com/ocx-hgrx-bjg | 11:15 |
| rlandy|rover | https://meet.google.com/vzm-nrah-qqf?authuser=0 | 11:15 |
| marios | chandankumar: ^^ | 11:15 |
| rlandy|rover | marios: ^^ | 11:15 |
| marios | :D | 11:15 |
| marios | k ... | 11:16 |
| rlandy|rover | on regular meeting | 11:16 |
| marios | ah right | 11:16 |
| rlandy|rover | chandankumar: ysandeep: ^^ | 11:16 |
| chandankumar | joining | 11:21 |
| chandankumar | https://pypi.org/project/ansible-core/#history | 11:28 |
| chandankumar | https://pypi.org/project/ansible/#history | 11:29 |
| *** ysandeep is now known as ysandeep|afk | 11:36 | |
| Tengu | ysandeep|afk: seems https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/858868 is doing the right job. I've added a comment in order to explain a bit the background and context. | 11:37 |
| Tengu | would be good to add to the review queue https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/858868 | 11:38 |
| reviewbot | I could not add the review to Review List | 11:38 |
| Tengu | bad robot | 11:38 |
| rlandy|rover | akahat|ruck: chandankumar: frenzyfriday|food: will you be available to sync after scrum for rr? | 11:39 |
| chandankumar | rlandy|rover: yup | 11:39 |
| akahat|ruck | rlandy|rover, yes. | 11:39 |
| rlandy|rover | akahat|ruck: pls requeue stable 1 | 11:40 |
| akahat|ruck | rlandy|rover, i've already did.. still the same issue? | 11:40 |
| rlandy|rover | it didn't start? | 11:40 |
| akahat|ruck | rlandy|rover, let me try it again. | 11:41 |
| rlandy|rover | akahat|ruck: let's sync and see what's going on | 11:42 |
| akahat|ruck | ok. it's now started. | 11:43 |
| akahat|ruck | rlandy|rover, https://meet.google.com/vri-uhts-dht | 11:43 |
| frenzyfriday|food | rlandy|rover, yep | 11:47 |
| *** amoralej|lunch is now known as amoralej | 11:59 | |
| *** frenzyfriday|food is now known as frenzyfriday | 12:00 | |
| akahat|ruck | rlandy|rover, https://review.rdoproject.org/r/c/testproject/+/45217 wallaby jobs return | 12:00 |
| akahat|ruck | wallaby jobs rerun * | 12:01 |
| * bhagyashris brb | 12:27 | |
| *** ysandeep|afk is now known as ysandeep | 12:35 | |
| ysandeep | Tengu, I have posted a testproject with standalone job: https://review.rdoproject.org/r/c/testproject/+/31954/97/.zuul.yaml to see if it works with nftables as well. | 12:42 |
| Tengu | ysandeep: oh, thanks! | 12:43 |
| Tengu | I was looking into it, but couldn't find the actual thing to add. lemme check | 12:43 |
| *** Tengu is now known as Guest1145 | 12:51 | |
| *** Tengu_ is now known as Tengu | 12:51 | |
| rcastillo | jm1 rlandy|rover running collection tests here https://review.rdoproject.org/r/c/testproject/+/45218 | 12:57 |
| rlandy|rover | rcastillo: on a million things atm | 12:58 |
| rlandy|rover | will chat with you later | 12:58 |
| rcastillo | rlandy|rover: ack | 12:58 |
| pojadhav | rlandy|rover, akahat|ruck, jm1 : scrum time | 13:01 |
| pojadhav | dasm|off, ^^ | 13:02 |
| jm1 | rlandy|rover, frenzyfriday: thanks for migrating over our hackmd space, esp. without having to move our notes 🥳 | 13:13 |
| soniya29 | reviewbot, please add in review list: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/857669 https://review.opendev.org/c/openstack/tripleo-quickstart/+/857671 | 13:23 |
| reviewbot | I could not add the review to Review List | 13:23 |
| reviewbot | I could not add the review to Review List | 13:23 |
| soniya29 | reviewbot, please add in review list: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/857669 | 13:24 |
| reviewbot | I could not add the review to Review List | 13:24 |
| frenzyfriday | soniya29, the bot is not working, I'll check tomorrow | 13:24 |
| marios | pojadhav: rcastillo: in 5 mins ok? | 13:24 |
| soniya29 | frenzyfriday, okay, no problem | 13:24 |
| rlandy|rover | akahat|ruck: rr sync | 13:24 |
| marios | pojadhav: rcastillo: should be a quick call | 13:24 |
| pojadhav | marios, sure | 13:24 |
| rcastillo | marios: yeah | 13:24 |
| marios | rcastillo: pojadhav: https://meet.google.com/uqn-oyfv-pzt whenever you're ready | 13:29 |
| akahat|ruck | rlandy|rover, o/ | 13:33 |
| akahat|ruck | rlandy|rover, i'm here.. we can sync. | 13:34 |
| frenzyfriday | akahat|ruck, we are on the scrum meet link | 13:34 |
| jpodivin | reviewbot: please add to review list https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/836752 | 13:35 |
| reviewbot | I could not add the review to Review List | 13:35 |
| frenzyfriday | jpodivin, sorry, the bot is not working, I'll check tomorrow | 13:35 |
| jpodivin | frenzyfriday: I see, thanks. | 13:36 |
| * ysandeep under heavy rainfall, out for the day. | 13:37 | |
| *** ysandeep is now known as ysandeep|out | 13:37 | |
| jm1 | rcastillo: thanks for the aoc @ tripleo multinode jobs :) | 13:52 |
| frenzyfriday | rlandy|rover, chandankumar new RR hackmd https://hackmd.io/5qVksz_VQHOTy6A8a-VhOA I have linked it to the main RR status hackmd | 14:08 |
| chandankumar | rlandy|rover: akahat|ruck ceph update testing https://code.engineering.redhat.com/gerrit/c/testproject/+/401805 a | 14:09 |
| rlandy|rover | frenzyfriday: chandankumar: thank you | 14:10 |
| akahat|ruck | chandankumar, thank you!! | 14:10 |
| *** dasm|off is now known as dasm | 14:17 | |
| dasm | o/ | 14:17 |
| rlandy|rover | jm1++++++++++++++++++++ on handling this aoc/zed branching effort | 14:17 |
| jm1 | rlandy|rover: thank you, but it that was a team effort, so rhos-bu++++ ;D | 14:18 |
| akahat|ruck | testing ceph_tag here: https://code.engineering.redhat.com/gerrit/c/testproject/+/429399 | 14:26 |
| chandankumar | see ya! | 14:33 |
| chandankumar | rlandy|rover: akahat|ruck please leave notes | 14:33 |
| rlandy|rover | chandankumar: ack - will do | 14:33 |
| marios | rcastillo: pojadhav: ysandeep|out: chandankumar: rlandy|rover: can we merge this one please review when you have time if there are issues will update https://review.rdoproject.org/r/c/config/+/45158 | 14:54 |
| rlandy|rover | marios: ack - can merge that | 14:58 |
| rlandy|rover | watching this: https://review.rdoproject.org/r/c/testproject/+/45225 Test corosync_token_timeout | 14:59 |
| rlandy|rover | sorry | 14:59 |
| rlandy|rover | https://review.opendev.org/c/openstack/tripleo-heat-templates/+/858889/ | 14:59 |
| marios | thanks rlandy|rover | 15:01 |
| marios | rlandy|rover: i'll merge in the mornign if still around | 15:01 |
| rlandy|rover | I voted | 15:01 |
| rlandy|rover | so you can merge when you get in | 15:01 |
| rlandy|rover | marios: chandankumar: ysandeep|out: going to merge https://review.opendev.org/c/openstack/tripleo-ci/+/858768/ - when https://review.opendev.org/c/openstack/tripleo-heat-templates/+/858889/ completes | 15:23 |
| dpawlik | arxcruz: will do review tomorrow | 15:27 |
| dpawlik | but it seems to be promising! | 15:27 |
| marios | ack rlandy|rover | 15:29 |
| * marios off | 15:29 | |
| marios | will check tomorrow | 15:29 |
| *** marios is now known as marios|out | 15:29 | |
| *** jpena is now known as jpena|off | 16:13 | |
| *** amoralej is now known as amoralej|off | 16:21 | |
| dasm | jm1: o/ recent merge on grafana 9.1.5 broke cockpit-ci.tripleo.org view: http://dashboard-ci.tripleo.org/d/YRJtmtNWk/cockpit/ | 16:22 |
| dasm | i know you've been using that. do you recall what was there? | 16:22 |
| dasm | frenzyfriday: o/ i'm seeing we have a cockpit instance: 10.0.111.235 it has your key associated | 16:47 |
| dasm | frenzyfriday: i tried ssh-ing but it didn't let me in. | 16:47 |
| dasm | frenzyfriday: can you give me an access to it? i want to cross check how and why nginx behaves like it does | 16:48 |
| frenzyfriday | dasm, yes, that is the development server that Pooja and the CRE team usually uses to verify grafana patches. It was set up manually. Can you pl send me your keys? I'll add them | 16:53 |
| frenzyfriday | dasm, are you facing any issure related to nginx? | 16:53 |
| dasm | frenzyfriday: default nginx config returns "welcome to nginx" page when accessed via IP | 16:54 |
| dasm | frenzyfriday: it needs to have "dashboard-ci.tripleo.org" Host provided to return grafana view | 16:55 |
| dasm | it's differet for this server tho. | 16:55 |
| dasm | i have 2 VMs: cs8 and cs9 deployed via podman, and i have the same thing | 16:55 |
| dasm | both shows default nginx page, and not grafana | 16:55 |
| dasm | so, i wanted to see how (and why) development server shows grafana. | 16:56 |
| dasm | it probably means it's changed | 16:56 |
| dasm | frenzyfriday: does it make sense? | 16:56 |
| dasm | frenzyfriday: long story short: podman works, but i don't have a way (except curl) to show it working | 16:56 |
| frenzyfriday | dasm, yeah, for the dev server I changed dashboard-ci.tripleo.org to the IP of the VM | 16:56 |
| dasm | mhm. ok, that explains it :) | 16:56 |
| frenzyfriday | can you try ssh now? ssh centos@10.0.111.235 | 16:57 |
| dasm | i'm in | 16:57 |
| dasm | thx | 16:57 |
| frenzyfriday | cool :) | 16:57 |
| frenzyfriday | the nginx stuff is under ~/ci-config/ci-scripts/infra-setup/roles/rrcockpit/files/ etc... | 16:59 |
| dasm | ack. i was just informed why this dev server works without Hostname | 17:00 |
| dasm | now I'm wondering if we could have something like that for staging/dev env | 17:00 |
| frenzyfriday | If you face any issues like "Are you trying to mount a directory onto a file" with nginx this is the solution: https://stackoverflow.com/questions/49214000/what-is-the-nginx-conf-error-directive-not-allowed-here | 17:00 |
| frenzyfriday | I saw it in the dev server. If you face the same in the new server I'll put up a patch to fix it | 17:01 |
| dasm | no issues with new deployments atm | 17:01 |
| frenzyfriday | cool | 17:01 |
| rlandy|rover | lunch - brb | 17:07 |
| * akahat|ruck offline | 17:20 | |
| rcastillo | jm1: collections tp just reported. containers-multinode and most ovb passed | 17:50 |
| rcastillo | https://review.rdoproject.org/r/c/testproject/+/45218 | 17:50 |
| rcastillo | don't think fs35 failure is related to collections | 17:50 |
| rcastillo | I dare say we could pull the pin without burning the house down | 17:52 |
| rcastillo | some light scorching at most :) | 17:52 |
| rlandy|rover | arxcruz: hey - did we just merge a tempest skiplist change? | 18:00 |
| rlandy|rover | pojadhav: ^^ | 18:01 |
| rlandy|rover | https://logserver.rdoproject.org/71/857671/9/openstack-check/tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020/5a99474/logs/undercloud/var/log/tempest/stestr_results.html.gz | 18:01 |
| rlandy|rover | https://review.opendev.org/c/openstack/tripleo-quickstart/+/857671/9/config/general_config/featureset020.yml | 18:02 |
| rlandy|rover | nah - just that patch | 18:02 |
| jm1 | dasm: where di you get this cockpit url from? what was supposed to be there? | 18:36 |
| jm1 | rcastillo: nice! fs35 failed with lp#1987092 so this is unrelated to aoc | 18:40 |
| dasm | jm1: it's in the nginx config. i saw you sharing that link in the past | 18:43 |
| jm1 | dasm: me? where? cannot find it in my irc logs | 18:45 |
| dasm | jm1: hmm.. i can't find it. welp. i thought it was you | 18:48 |
| dasm | actually i can't find it at all | 18:48 |
| dasm | it might mean, it's not used anymore | 18:49 |
| jm1 | dasm: found it, it is a change prior our rh time ;) https://github.com/rdo-infra/ci-config/blame/master/ci-scripts/infra-setup/roles/rrcockpit/files/nginx.conf.local | 18:49 |
| dasm | > sshnaidm committed on Jan 16, 2020 | 18:50 |
| dasm | hmm... yup. might be not relevant anymore | 18:50 |
| dasm | jm1: btw, podman: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45206 cc frenzyfriday | 18:51 |
| dasm | test instances are up and running, but they can be only curl'ed with '-H "Host: dashboard-ci.tripleo.org"' otherwise they're gonna return generic nginx landing pade. | 18:52 |
| dasm | *page | 18:52 |
| jm1 | dasm: when you checkout the commit where sagi added that url you will not find any reference to that cockpit url. you will find a cockpit dashboard though. but that is still available although it has a different url now. | 18:54 |
| dasm | jm1: when you open cockpit-ci.tripleo.org it says: "Dashboard not found" | 18:56 |
| jm1 | dasm: did you play with podman's kubernetes yaml support? wondering whether this is a proper replacement for docker compose | 18:57 |
| dasm | i thought you were using that link to reach out some resources | 18:57 |
| dasm | but now idk | 18:57 |
| dasm | i can update the link and that should be it, i think | 18:57 |
| dasm | this one: http://dashboard-ci.tripleo.org/d/Z4vLSmOGk/cockpit?orgId=1 | 18:57 |
| dasm | jm1: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45231 | 19:00 |
| jm1 | dasm: thanks, lets try that :) | 19:03 |
| jm1 | dasm: did you try podman's play kube? | 19:09 |
| dasm | nope | 19:10 |
| dasm | i know nothing about that | 19:10 |
| jm1 | dasm: would be good to know if that is a proper replacement for docker-compose.yml nowadays | 19:10 |
| dasm | i would consider that "next steps" | 19:10 |
| dasm | for now, i wanted to keep docker-compose around. | 19:11 |
| jm1 | dasm: ok, nevermind. we can test that in future | 19:11 |
| rcastillo | I've wondered about that | 19:11 |
| dasm | evolution not revolution :) | 19:11 |
| dasm | rcastillo: responded to your comments | 19:15 |
| jm1 | dasm: the hacks that you have to do to get podman compose working are.. quite heavy though | 19:15 |
| jm1 | dasm: did podman+docker-compose not work? | 19:16 |
| dasm | jm1: haven't tried podman-docker-compose | 19:16 |
| dasm | i can give it a try | 19:17 |
| jm1 | dasm: https://www.redhat.com/sysadmin/podman-docker-compose | 19:17 |
| rcastillo | I always had the impression that docker-compose would work just by enabling podman socket. Never tried it though | 19:17 |
| jm1 | rcastillo: yeah that is what the link about does | 19:17 |
| jm1 | *above | 19:18 |
| rcastillo | ah, I see | 19:18 |
| dasm | it should be straightforward swap. i'mma try it | 19:18 |
| dasm | i wasn't happy with podman-compose workaround. so maybe podman-docker-compose is gonna be better | 19:19 |
| dasm | still, not perfect, because we're messing up with two things: docker and podman in the same space | 19:20 |
| dasm | it makes confusion | 19:20 |
| jm1 | dasm: what do we mess up? its podman + podman socket + podman-docker + docker-compose. | 19:20 |
| dasm | docker-compose still being used: https://docs.ansible.com/ansible/latest/collections/community/docker/docker_compose_module.html | 19:20 |
| jm1 | dasm: if you want to reduce confusion you have to get rid of docker-compose.. and use kubernetes yaml / play kube | 19:20 |
| dasm | i don't want to do that now. too much intrusion with just one thing. | 19:21 |
| jm1 | dasm: but i agree, for now we should continue to use docker-compose simply because you did most of the work already | 19:21 |
| rcastillo | it's as confusing as the fact that we have a Dockerfile | 19:21 |
| dasm | i wanted to make it work with podman -- it was already an issue | 19:21 |
| dasm | rcastillo: true | 19:21 |
| dasm | docker-compose can't be used with docker anymore due to "quay.io" hub space. | 19:22 |
| jm1 | rcastillo: yeah. one could rename it to Containerfile but come on, Dockerfile has its benefits: you know where to search for help | 19:22 |
| rcastillo | exactly | 19:22 |
| jm1 | dasm:why? my local docker install has images from quay.io | 19:23 |
| jm1 | dasm: and i use it with docker-compose. works fine | 19:23 |
| dasm | do you use docker or podman? | 19:24 |
| jm1 | dasm: docker | 19:24 |
| dasm | hmm... | 19:24 |
| dasm | i thought frenzyfriday mentioned we could not use docker for quay resources | 19:24 |
| jm1 | dasm: the reason we had with docker and quay.io was due to centos7 | 19:24 |
| dasm | did i get it wrokg again? | 19:24 |
| dasm | c7 had older docker version? was that it? | 19:25 |
| jm1 | dasm: yeah, docker on c7 was/is too old. but more up to date docker works fine with quay.io | 19:25 |
| dasm | ah | 19:25 |
| dasm | i missed that them | 19:25 |
| dasm | *then | 19:25 |
| rcastillo | oh shoot, forgot about review meeting | 19:25 |
| dasm | so, tl'dr. keeping docker-compose.yml allows to use both podman and docker. | 19:26 |
| dasm | at least i hope so ;) | 19:26 |
| dasm | i prefer anything but docker | 19:26 |
| dasm | rcastillo: was short | 19:26 |
| dasm | not a lot of things to discuss | 19:26 |
| rcastillo | ok | 19:27 |
| jm1 | dasm: why? there is no downside to docker for our use case. cool kids use kubernetes nowadays anyway | 19:27 |
| rcastillo | cockpit on openshift when? | 19:28 |
| rcastillo | only half kidding | 19:28 |
| dasm | rcastillo: Soon(TM) https://wowwiki-archive.fandom.com/wiki/Soon | 19:28 |
| jm1 | rcastillo: pssst 🤫 | 19:28 |
| dasm | i'm not kidding tho :) | 19:28 |
| dasm | jm1: old gripe with docker, their split between community and enterprise and few other things | 19:29 |
| dasm | i would use k3s or k8s if that would make sense (no, it does not :P) | 19:29 |
| jm1 | dasm: come on, they build a wonderful toolset, never made any money and lost against kubernetes. let us be thankful for their great work | 19:30 |
| rcastillo | I like podman better technically but docker did lay the groundwork | 19:31 |
| dasm | the only thing what docker did was UI. lxc and lxd were around the same time | 19:31 |
| jm1 | dasm, rcastillo: we thought about getting access and quota for our openshift service hosted by red hat (what was its name?) and then move our services over there. this would relief us from the burden of maintaining vms and all that. but thats future work and definitely not done in a few minutes | 19:32 |
| dasm | Soon(TM) | 19:32 |
| rcastillo | that'd be pretty amazing | 19:33 |
| dasm | that'd be pretty menacing ;) | 19:33 |
| jm1 | dasm: their ui is definitely an improvement and they brought us registries aka docker hub. they made containers useable for a broad audience | 19:33 |
| jm1 | dasm: 😂 | 19:33 |
| jm1 | dasm: but your right, containers tech have been there before | 19:34 |
| jm1 | anyway, wife is calling, have to leave for today 😅 | 19:34 |
| dasm | good night jm1 | 19:34 |
| rcastillo | jm1: o/ | 19:34 |
| jm1 | dasm, rcastillo, oooq: have a good night :) | 19:34 |
| rlandy|rover | rcastillo: hey busy debugging a systemd issue bit pl ping when you are ready | 19:54 |
| rlandy|rover | to work on the tests for aoc | 19:54 |
| rcastillo | rlandy|rover: ready now | 19:55 |
| rlandy|rover | one sec | 19:55 |
| rlandy|rover | rcastillo: https://meet.google.com/vhp-phan-vdj?pli=1&authuser=0 | 20:03 |
| rlandy|rover | chandankumar: https://review.rdoproject.org/r/c/config/+/45232 Replace constraint_file link - connection timeouts | 20:41 |
| rlandy|rover | testing: https://review.rdoproject.org/r/c/testproject/+/31165/105/.zuul.yaml#9 | 20:41 |
| *** rlandy|rover is now known as rlandy|rover|biab | 21:33 | |
| * dasm => offline | 22:33 | |
| dasm | o/ | 22:33 |
| *** dasm is now known as dasm|off | 22:33 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!