| *** weshay has quit IRC | 00:44 | |
| *** dsneddon has quit IRC | 01:03 | |
| *** weshay has joined #oooq | 01:12 | |
| *** weshay has quit IRC | 02:03 | |
| *** rfolco has quit IRC | 02:06 | |
| *** apetrich has quit IRC | 02:10 | |
| *** weshay has joined #oooq | 02:12 | |
| *** aakarsh has joined #oooq | 02:18 | |
| *** brault has quit IRC | 02:19 | |
| *** weshay has quit IRC | 02:38 | |
| *** ykarel has joined #oooq | 02:39 | |
| *** aakarsh has quit IRC | 02:42 | |
| *** aakarsh has joined #oooq | 02:52 | |
| *** aakarsh has quit IRC | 03:04 | |
| *** Goneri has joined #oooq | 03:44 | |
| *** udesale has joined #oooq | 04:12 | |
| *** rlandy|bbl is now known as rlandy | 04:21 | |
| *** ratailor has joined #oooq | 04:34 | |
| *** ykarel has quit IRC | 04:42 | |
| *** ykarel has joined #oooq | 05:04 | |
| *** marios has joined #oooq | 05:11 | |
| *** soniya29 has joined #oooq | 05:13 | |
| *** Goneri has quit IRC | 05:24 | |
| *** Goneri has joined #oooq | 05:27 | |
| *** akahat has joined #oooq | 05:37 | |
| *** jfrancoa has joined #oooq | 05:52 | |
| *** jfrancoa has quit IRC | 05:56 | |
| ykarel | dtantsur|afk, arxcruz|rover 1 ovb job passed with ironicclient -3.1.0 https://review.rdoproject.org/r/#/c/22551/ | 06:03 |
|---|---|---|
| ykarel | and i am taking it in train with https://review.rdoproject.org/r/#/c/22588/ before it's updated in u-c | 06:03 |
| *** saneax has joined #oooq | 06:06 | |
| *** jfrancoa has joined #oooq | 06:13 | |
| *** Goneri has quit IRC | 06:17 | |
| *** skramaja has joined #oooq | 06:29 | |
| *** jaosorior has quit IRC | 06:30 | |
| *** holser has joined #oooq | 06:31 | |
| *** akahat has quit IRC | 06:40 | |
| *** tosky has joined #oooq | 06:41 | |
| *** udesale has quit IRC | 07:01 | |
| *** udesale has joined #oooq | 07:02 | |
| *** jaosorior has joined #oooq | 07:05 | |
| *** tesseract has joined #oooq | 07:05 | |
| soniya29 | marios, Hello | 07:21 |
| *** bogdando has joined #oooq | 07:21 | |
| marios | o/ soniya29 | 07:25 |
| *** pierrepr1netti is now known as pierreprinetti | 07:26 | |
| *** jpena|off is now known as jpena | 07:26 | |
| *** ccamacho has joined #oooq | 07:27 | |
| *** akahat has joined #oooq | 07:28 | |
| soniya29 | marios, I am trying to install cockpit locally, so when I run './development_Script.sh -s', it is consuming too much time. Is it expected behaviour or unusual? | 07:28 |
| *** brault has joined #oooq | 07:28 | |
| marios | soniya29: o/ hi i think the first time it pulls containers etc so it takes a while if i recall | 07:30 |
| marios | soniya29: i updated the readme with some info did you check there? | 07:30 |
| * marios finds | 07:30 | |
| marios | https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/rrcockpit soniya29 | 07:31 |
| marios | " Note it may take a while to load especially on first run. You will see that the containers are being pulled before starting them like:" | 07:31 |
| marios | soniya29: like is it hanging or is stuff happening just takes a long time? | 07:31 |
| *** brault has quit IRC | 07:33 | |
| soniya29 | marios, It is continuosly pulling containers, https://paste.fedoraproject.org/paste/~QAmFD4o3mm6EebowKA2Mg | 07:33 |
| marios | soniya29: try http://172.18.0.6:3000 in your browser | 07:35 |
| soniya29 | marios, It opened dashboard in the browser | 07:36 |
| marios | soniya29: so its running :) | 07:37 |
| marios | soniya29: read that it explains a bit https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/README.md | 07:37 |
| soniya29 | marios, thanks :), I thought something went wrong with my installation. | 07:38 |
| marios | soniya29: np :D i went through the exact same pain last time i tried to work there, which is why i updated the readme | 07:38 |
| marios | soniya29: maybe you can contribute by improving that further once you're done setting up and if you think of something missing/find some new issue not documented etc | 07:39 |
| soniya29 | marios, at second step, path to the development_script.sh isn't mentioned. I was wondering where exactly is the script location. | 07:43 |
| *** udesale has quit IRC | 07:44 | |
| marios | soniya29: did you clone ci-config? | 07:44 |
| *** ykarel is now known as ykarel|lunch | 07:44 | |
| marios | soniya29: its there https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/rrcockpit/files | 07:44 |
| *** udesale has joined #oooq | 07:44 | |
| soniya29 | marios, yeah I cloned it. But development script is in ci-scripts/infra-setup/roles/rrcockpit/files and maybe the new folk trying to install be at some other location, let's say ci-scripts/infra-setup. He has to do 'cd' to the ci-scripts/infra-setup/roles/rrcockpit/files and then hit './developemnt_script.sh' | 07:48 |
| *** soniya29 is now known as soniya29|lunch | 07:48 | |
| *** ksambor has quit IRC | 07:50 | |
| *** tesseract has quit IRC | 07:52 | |
| *** ksambor has joined #oooq | 07:52 | |
| *** tesseract has joined #oooq | 07:53 | |
| *** sshnaidm|afk is now known as sshnaidm | 08:02 | |
| *** amoralej|off is now known as amoralej | 08:12 | |
| *** soniya29|lunch has quit IRC | 08:13 | |
| *** apetrich has joined #oooq | 08:27 | |
| *** soniya29 has joined #oooq | 08:37 | |
| *** derekh has joined #oooq | 08:37 | |
| *** ykarel|lunch is now known as ykarel | 08:41 | |
| marios | panda: around? molecule test -s container-push fails locally on some weird error about the python interpreter http://paste.openstack.org/raw/779420/ seen that one? | 08:45 |
| marios | trying rdo box in a sec | 08:45 |
| *** chem has joined #oooq | 08:49 | |
| *** soniya29 has quit IRC | 08:51 | |
| arxcruz|rover | ykarel: cool, thanks for testing | 08:53 |
| arxcruz|rover | u da man | 08:53 |
| *** soniya29 has joined #oooq | 08:54 | |
| marios | panda: ah i see you're hardcoding it | 08:54 |
| panda | marios: not anymore | 09:02 |
| panda | marios: I added a variable there to test it locally | 09:02 |
| *** brault has joined #oooq | 09:02 | |
| panda | marios: ansible_python_interpreter: ${MOLECULE_INTERPRETER:-/home/zuul/test-python/bin/python} | 09:02 |
| panda | marios: so it can be tested both locally or in zuul | 09:03 |
| panda | marios: new PS coming soon, with some other tests. The integration test is now at the point in which the container-push is invoked, so it's time to merge something ... | 09:05 |
| *** dtantsur|afk is now known as dtantsur | 09:10 | |
| marios | panda: ack... i just removed it for my local test (I want to iterate on manifest include today | 09:10 |
| marios | panda: but i an issue with generate container list (see comment https://review.rdoproject.org/r/#/c/22445/17/ci-scripts/container-push/roles/containers-promote/tasks/main.yaml@53 | 09:11 |
| marios | panda: i will just iterate in zuul job if i have to but since its molecule the whole point is meant to be local | 09:11 |
| marios | panda: also needed pip install selinux | 09:12 |
| *** udesale has quit IRC | 09:13 | |
| *** udesale has joined #oooq | 09:17 | |
| zbr|ruck | marios: pip install selinux helps only inside virtualenvs, when you already have libselinux. | 09:17 |
| *** jaosorior has quit IRC | 09:17 | |
| marios | zbr|ruck: yes it was in venv | 09:18 |
| zbr|ruck | a warning regarding using molecule/ansible under centos: you must use only the official python version of the platform, the other one(s) *will not work*. | 09:18 |
| marios | zbr|ruck: so just like i ahve to manually install molecule/docker etc i have to do that too | 09:19 |
| zbr|ruck | i do have playbooks for most stuff | 09:20 |
| zbr|ruck | i am still worried that we really dont have real docker on centos8 | 09:21 |
| zbr|ruck | which is a problem for using molecule. so many reasons for not using centos 7 or 8 as a development platform. | 09:22 |
| zbr|ruck | i have much better testing coverage with fedora/macos/ubuntu/debian where I have docker, all python versions without selinux issues. hopefully we will address these at some point. | 09:24 |
| ykarel | zbr|ruck, arxcruz|rover debugging queens issue with https://review.rdoproject.org/r/#/c/22551/ | 09:28 |
| arxcruz|rover | ykarel: ack | 09:28 |
| ykarel | arxcruz|rover, it's not getting reproduced, so need to get node held and debug | 09:29 |
| ykarel | i will let you know once it fails and node get held | 09:29 |
| arxcruz|rover | ykarel: is remove_ovb_after_job hold the node? | 09:29 |
| arxcruz|rover | oh, it doesn't remove the node from nodepool, got it, last time i had this problem, only the main node was held, the others were deleted | 09:30 |
| arxcruz|rover | good to know :) | 09:30 |
| ykarel | arxcruz|rover, nope ^^ param will hold overcloud nodes, i requested jpena on #rdo to hold the nodepool node | 09:30 |
| ykarel | yes /me trying that remove param first time :) | 09:30 |
| panda | marios: new PS uploaded, it passes all my local tests. | 09:58 |
| *** tesseract has quit IRC | 09:59 | |
| *** tesseract has joined #oooq | 10:00 | |
| marios | panda: k will try it in a bit trying to change the curl to use manifest ispect first | 10:02 |
| marios | panda: it still failed for me on the same things locally | 10:48 |
| marios | panda: added comment | 10:48 |
| panda | marios: you're probably missing export MOLECULE_INTERPRETER= | 10:53 |
| panda | marios: pointing to the python to your local venv | 10:53 |
| panda | maybe this sould be more explicit , not sure where to put this warning though | 10:54 |
| marios | panda: sure but is that expected i.e. 'just somethign you have to do for local testing' so maybe we add a readme about that | 10:54 |
| marios | panda: that one is not a big problem but the containers list one is | 10:54 |
| marios | panda: but clearly it works in the zuul job so... not sure why the difference | 10:55 |
| *** brault has quit IRC | 10:55 | |
| panda | marios: checking it now. Can you paste the whole log ? Or the log with some context around ? | 10:55 |
| marios | panda: http://paste.openstack.org/raw/779429/ | 10:56 |
| chandankumar | soniya29: https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-master/08421f4/logs/stestr_results.html | 11:02 |
| panda | marios: looks like int your env the "mock registries" task fails in some way to push the base images, I'd rerun with molecule converge -s container-push -- -vv to increase verbosity and see what is not run in that task | 11:03 |
| *** amoralej is now known as amoralej|lunch | 11:04 | |
| *** jaosorior has joined #oooq | 11:05 | |
| chandankumar | arxcruz|rover: hello | 11:06 |
| chandankumar | arxcruz|rover: regarding parsing failed tests from fs021, I got a nice idea | 11:06 |
| chandankumar | since we have .html file in each job | 11:06 |
| chandankumar | just convert it to json and get the result | 11:06 |
| chandankumar | what do you say? | 11:06 |
| *** jpena is now known as jpena|lunch | 11:09 | |
| *** udesale has quit IRC | 11:14 | |
| *** derekh has quit IRC | 11:20 | |
| arxcruz|rover | chandankumar: hmmmmm will you be able to parse? did you saw the mess that that html is ? :D | 11:27 |
| arxcruz|rover | I mean, of course, All i need is the info in the cockpit, how you going to do it it's another history :P) | 11:28 |
| arxcruz|rover | if parse the html is better... fine | 11:28 |
| arxcruz|rover | using the unit 2 csv works pretty fine btw | 11:29 |
| arxcruz|rover | ykarel: asking just for asking, because you da man, but do you need help on the queens? I saw the job finishes | 11:29 |
| ykarel | arxcruz|rover, yes it reproduced, | 11:30 |
| ykarel | and i am not able to ssh it | 11:30 |
| ykarel | arxcruz|rover, i am thinking of enabling password in the image, upload and redeploy, and then access with password | 11:32 |
| ykarel | any other idea if u can recollect to shorten the process | 11:32 |
| ykarel | if you would like to access it, i can add your keys | 11:33 |
| chandankumar | arxcruz|rover: subunit2csv is broken | 11:38 |
| chandankumar | I need to spend some time on fixing it | 11:39 |
| marios | panda: can you please update that https://review.rdoproject.org/r/#/c/22445/19/ci-scripts/infra-setup/roles/promoter/molecule/container-push/playbook.yml | 11:43 |
| marios | panda: we need all the containers you push in test regsitry to also be tagged with {{ full_hash }}_x86_64 like rdo registry | 11:44 |
| panda | marios: ack | 11:44 |
| marios | panda: 2019-09-26 11:32:58.692075 | rdo-centos-7 | msg: 'Error pulling image localhost:5000/tripleomaster/centos-binary-base:82a1e08b25e42f1e8ea99d7c9b6957bb06423a34_dae2ef8_x86_64 - 404 Client Error: Not Found ("manifest for localhost:5000/tripleomaster/centos-binary-base:82a1e08b25e42f1e8ea99d7c9b6957bb06423a34_dae2ef8_x86_64 not found: manifest unknown: manifest unknown")' | 11:44 |
| panda | marios: do you want ppc too ? | 11:44 |
| marios | http://logs.rdoproject.org/02/22002/59/check/molecule-delegated/2d21292/job-output.txt.gz | 11:44 |
| marios | panda: thanks | 11:44 |
| marios | panda: yes ideally not all of them though so we can test the logic in manifest push | 11:45 |
| marios | panda: like leave 1/2 out for ppc | 11:45 |
| marios | panda: but for x86 we expect all | 11:45 |
| arxcruz|rover | ykarel: where you are not able to ssh? | 11:47 |
| ykarel | arxcruz|rover, overcloud nodes, same issue in the bug | 11:48 |
| arxcruz|rover | ykarel: you mean the ssh issue, not the get_endpoint issue right ? | 11:51 |
| ykarel | arxcruz|rover, ye right queens ssh issue, get_endpoint is in master | 11:52 |
| arxcruz|rover | sorry, i missunderstood :) | 11:52 |
| marios | panda: just saw its only 2 there so make on with ppc one without | 11:52 |
| marios | panda: btw if you're busy with somethign else let me know i'll update it | 11:52 |
| *** ratailor has quit IRC | 11:54 | |
| *** weshay has joined #oooq | 11:57 | |
| panda | marios: updating it now | 11:58 |
| ykarel | arxcruz|rover, np, i am trying an alternative instead of virt-customize, to inject root password:- https://github.com/openstack/tripleo-heat-templates/commit/724ba3a32f20349ed20093758a48ca1297a0534e#diff-768a7913933c61a5853abd6ed2dd1400 | 11:59 |
| weshay | arxcruz|rover, any luck w/ a reproducer on fs001 queens? | 11:59 |
| marios | panda: thx | 11:59 |
| marios | panda: would *love* to get a run on the manifest push before scrum | 11:59 |
| *** brault has joined #oooq | 12:01 | |
| *** rfolco has joined #oooq | 12:03 | |
| *** chandankumar has quit IRC | 12:04 | |
| panda | marios: sorry, adding the other tags broke the cleanup and I had to run some tests and change a few things | 12:12 |
| panda | marios: new PS uploaded | 12:13 |
| weshay | arxcruz|rover, ping me when you have a sec.. I have an update on queens | 12:13 |
| marios | panda: ack i got it running on the rdo vm via molecule test so not sure what is different locally... i added the tags and got an error | 12:13 |
| marios | panda: thanks checking now | 12:13 |
| arxcruz|rover | weshay: ping | 12:13 |
| *** chandankumar has joined #oooq | 12:14 | |
| zbr|ruck | weshay: i think that I managed to get the ball rolling regarding missing libselinux, i see progress in my inbox. | 12:14 |
| weshay | arxcruz|rover, hey morning.. can you please check a few other branches of tripleo and ovb jobs.. see if we're getting ips | 12:14 |
| weshay | arxcruz|rover, http://logs.rdoproject.org/11/684411/5/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/54e01f5/logs/overcloud-controller-0/var/log/extra/network.txt.gz and https://bugs.launchpad.net/tripleo/+bug/1845166 | 12:14 |
| openstack | Launchpad bug 1845166 in tripleo "[queens] [Periodic][check] OVB jobs failing in OC deploy when trying to ssh to nodes" [Critical,Triaged] | 12:14 |
| weshay | arxcruz|rover, /me gets coffee | 12:14 |
| marios | panda: thanks for update again but i only see tag where are you pushing now? | 12:16 |
| marios | panda: https://review.rdoproject.org/r/#/c/22445/21/ci-scripts/infra-setup/roles/promoter/molecule/container-push/playbook.yml | 12:16 |
| panda | marios: line 77, instead of pushing one by one, I push everything that I tagged | 12:17 |
| marios | panda: k thx trying | 12:17 |
| panda | marios: ah you're right | 12:18 |
| weshay | zbr|ruck, help me understand python2-libselinux and centos-8 | 12:18 |
| panda | marios: updating again | 12:18 |
| marios | panda: k | 12:19 |
| weshay | zbr|ruck, or are you working on centos-7 | 12:19 |
| *** derekh has joined #oooq | 12:19 | |
| zbr|ruck | weshay: yeah, in fact is a double issue: c7 missing one of them and c8 missing the other one. | 12:19 |
| zbr|ruck | not sure which one will be fixed fist. we will see. | 12:20 |
| panda | marios: done, feel free to take over so I'm not slowing you | 12:20 |
| zbr|ruck | shortly: keep. using fedora | 12:20 |
| weshay | zbr|ruck, arxcruz|rover you guys up for a 10min chat? | 12:20 |
| *** jpena|lunch is now known as jpena | 12:20 | |
| arxcruz|rover | weshay: sure boss | 12:20 |
| zbr|ruck | link? | 12:21 |
| weshay | https://meet.google.com/vts-bkoe-itj | 12:21 |
| marios | panda: thanks checking | 12:21 |
| sshnaidm | bogdando, please don' | 12:22 |
| sshnaidm | t https://review.opendev.org/#/c/683826/ | 12:22 |
| sshnaidm | bogdando, it doesn't work | 12:22 |
| bogdando | undone | 12:22 |
| *** amoralej|lunch is now known as amoralej | 12:23 | |
| weshay | zbr|ruck, libselinux-python-2.5-14.1.el7.x86_64 | 12:24 |
| *** derekh has quit IRC | 12:24 | |
| *** derekh has joined #oooq | 12:25 | |
| *** beagles is now known as beagles-mtg | 12:29 | |
| weshay | zbr|ruck, arxcruz|rover http://logs.rdoproject.org/32/618832/7/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/6c2fe25/logs/undercloud/ | 12:35 |
| weshay | last known good queens | 12:35 |
| ykarel | arxcruz|rover, hi | 12:35 |
| arxcruz|rover | ykarel: hallo | 12:35 |
| weshay | http://logs.rdoproject.org/32/618832/7/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/6c2fe25/logs/undercloud/var/log/extra/rpm-list.txt.gz | 12:36 |
| ykarel | arxcruz|rover, can you get me console access for overcloud node | 12:36 |
| weshay | openssh-7.4p1-21.el7.x86_64 | 12:36 |
| arxcruz|rover | ykarel: sorry ? | 12:36 |
| weshay | ykarel, do you have a queens fs001 recreate? | 12:36 |
| ykarel | weshay, yes | 12:36 |
| ykarel | arxcruz|rover, openstack console url show b9401f73-0553-4eaf-8efe-50b128f31227 | 12:36 |
| ykarel | ^^ server id of one of overcloud node | 12:37 |
| *** rlandy has joined #oooq | 12:37 | |
| jfrancoa | sshnaidm: hey, can you give me a hand with the reproducer? I'm trying to run it but can't get rid of this error: http://pastebin.test.redhat.com/800955 (in this execution I tried with -e cloud_networks=['private_nw'] because with the default ['private'] it also failed) | 12:37 |
| ykarel | weshay, ssh not accessible, so i deployed with passord | 12:37 |
| arxcruz|rover | ykarel: i'm not following you, sorry... | 12:38 |
| ykarel | arxcruz|rover, can u run the above command with openstack-nodepool tenant credentials | 12:38 |
| ykarel | that will give console url | 12:38 |
| arxcruz|rover | ykarel: just a sec | 12:38 |
| rfolco | panda, how do we include teardown playbook to run in the molecule destroy step ? | 12:39 |
| ykarel | arxcruz|rover, ack | 12:40 |
| panda | rfolco: add a destroy.yml in your molecule/scenario/ dir | 12:40 |
| arxcruz|rover | ykarel: jsut a sec, i don't have the credentials easy here with me and i'm in a meeting with weshay now | 12:40 |
| ykarel | arxcruz|rover, ack, may be panda can help with this: openstack console url show b9401f73-0553-4eaf-8efe-50b128f31227 | 12:41 |
| rfolco | panda, there is no converge.yml, how does molecule know playbook.yml is the converge then? | 12:41 |
| ykarel | panda, can u please help to get it for us ^^ | 12:41 |
| ykarel | from openstack-nodepool tenant | 12:42 |
| panda | rfolco: magice | 12:42 |
| panda | ykarel: yiou want the result here or in private ? | 12:43 |
| ykarel | panda, pm will work | 12:44 |
| weshay | ykarel, can we get on your queens job? | 12:46 |
| ykarel | weshay, yes | 12:46 |
| ykarel | share you pub key | 12:46 |
| arxcruz|rover | github.com/arxcruz.keys | 12:46 |
| weshay | ykarel, https://github.com/weshayutin.keys | 12:46 |
| weshay | zbr|ruck, https://github.com/sshnaidm/jcomparison | 12:47 |
| ykarel | weshay, arxcruz|rover ssh zuul@38.145.33.225 | 12:47 |
| sshnaidm | weshay, moving it today to ci-config repo | 12:47 |
| weshay | sshnaidm, cool thanks | 12:48 |
| sshnaidm | weshay, when back from ptos, we'll set up a new repo for it | 12:48 |
| ykarel | panda, Thanks | 12:50 |
| *** ccamacho has quit IRC | 12:51 | |
| weshay | ykarel, /me is suspicious of the network | 12:52 |
| weshay | trying ssh w/ verbose and it's hanging | 12:52 |
| ykarel | weshay, hmm, after some time it returns authentication failed | 12:53 |
| *** jtomasek_ has quit IRC | 12:53 | |
| *** jtomasek has joined #oooq | 12:56 | |
| weshay | https://opendev.org/openstack/tripleo-heat-templates/src/branch/stable/queens/ci/environments/network/multiple-nics/nic-configs/controller.yaml | 12:59 |
| rfolco | pinglist: marios, sshnaidm, weshay, panda, rlandy, arxcruz, rfolco, chandankumar, zbr, kopecmartin | 12:59 |
| rfolco | scrum in 1 min | 12:59 |
| weshay | mtu at 1350 | 12:59 |
| matbu|hfpto | Hey folks, quick question, Is oooq can be used on a RHEL 8 distro ? | 12:59 |
| matbu|hfpto | oops | 12:59 |
| *** matbu|hfpto is now known as matbu | 12:59 | |
| rfolco | scrum here https://meet.google.com/oiv-geho-mai?authuser=1 | 12:59 |
| sshnaidm | matbu, quickstart.sh? not sure anybody tried | 13:00 |
| ykarel | weshay, hmm this setting should be there from long but why issue started since few days only | 13:00 |
| matbu | sshnaidm: oki, i just tried but it failed :D | 13:00 |
| sshnaidm | matbu, so now we know :) | 13:00 |
| matbu | sshnaidm: hehe yep. I just asking because I might miss something | 13:01 |
| matbu | sshnaidm: but i can try to make it work | 13:01 |
| ykarel | weshay, token expired, can u get one generated again:- openstack console url show b9401f73-0553-4eaf-8efe-50b128f31227 | 13:01 |
| matbu | sound mostly matter of checking/installing packages | 13:01 |
| weshay | arxcruz|rover, https://opendev.org/openstack/tripleo-ci/src/branch/master/toci-quickstart/config/testenv/ovb.yml#L46 | 13:01 |
| chandankumar | rlandy: I fixed the emit release tests | 13:01 |
| chandankumar | taken too much time | 13:02 |
| rlandy | chandankumar: ok - will review after meeting | 13:02 |
| weshay | panda, you have a vnc console to an overcloud | 13:04 |
| weshay | ? | 13:04 |
| * weshay trying to connect to it | 13:05 | |
| panda | weshay: PM | 13:06 |
| *** udesale has joined #oooq | 13:16 | |
| *** chem has quit IRC | 13:29 | |
| amoralej | ykarel, arxcruz|rover weshay it may be a problem with mt | 13:31 |
| amoralej | mtu | 13:31 |
| amoralej | dalvarez is investigating | 13:31 |
| amoralej | with ovs | 13:31 |
| amoralej | wrt problem in queens | 13:31 |
| ykarel | ack /me watching , but why starting happening recently | 13:32 |
| ykarel | that would be interesting to know | 13:32 |
| ykarel | may be centos7.7 and some dep from queens | 13:32 |
| arxcruz|rover | ykarel: and whhy only on queens? | 13:32 |
| arxcruz|rover | mtu, what they are, where they live, what they eat, that and much more in the next chapters | 13:33 |
| ykarel | queens have different packages than other | 13:33 |
| amoralej | queens has ovs 2.9 | 13:33 |
| ykarel | so could be, but we will know soon, what causing it | 13:33 |
| amoralej | may be related | 13:33 |
| ykarel | yup | 13:33 |
| *** ccamacho has joined #oooq | 13:33 | |
| *** soniya29 has quit IRC | 13:43 | |
| panda | marios: you wanted to sync ? | 13:52 |
| rfolco | panda, molecule should remove molecule container on destroy step ? | 13:53 |
| marios | panda: yes can you do in like 10 mins quick break? | 13:53 |
| marios | panda: rfolco rlandy we want sync tomorrow @ 'scrum' time? | 13:53 |
| marios | panda: rfolco rlandy i'll send invite in a sec | 13:53 |
| rfolco | yes marios | 13:53 |
| rlandy | marios: ack | 13:53 |
| rlandy | thanks | 13:53 |
| panda | marios: you decide, I'm your humble slave | 13:53 |
| marios | panda: :D thanks please lets start in 7 mins will pm you in a sec for the room | 13:53 |
| *** akahat has quit IRC | 13:54 | |
| panda | rfolco: if you override the default destroy with your destroy.yml then you have to remove the container too | 13:54 |
| rlandy | marios: ^^ save that above line from panda | 13:54 |
| marios | rlandy: gonna frame it | 13:54 |
| rfolco | panda, it wasn't removing before my destroy.yml either | 13:54 |
| amoralej | arx ykarel weshay we are thinking in updating ovs to 2.11 in undercloud in that environment to see if the problem is centos7.7+ovs2.9+ovb | 13:56 |
| amoralej | is that fine? | 13:56 |
| rfolco | marios, call me for your party, I promise I behave well. | 13:56 |
| amoralej | or you want to check something before? | 13:56 |
| ykarel | amoralej, okk for me | 13:56 |
| ykarel | we can revert again | 13:56 |
| panda | rfolco: how do you call molecule ? | 13:56 |
| rfolco | panda, I am running this local, with molecule converge to be quicker for development | 13:57 |
| amoralej | revert ovs update... | 13:57 |
| amoralej | dunno | 13:57 |
| amoralej | may work | 13:57 |
| amoralej | :) | 13:57 |
| rfolco | panda, molecule destroy, molecule converge | 13:57 |
| ykarel | :D | 13:57 |
| amoralej | arxcruz|rover, ^ | 13:57 |
| rfolco | instead of molecule test or tox -e moulecule | 13:57 |
| rfolco | panda, ^ | 13:57 |
| panda | rfolco: and molecule destroy does not destroy ? | 13:58 |
| panda | rfolco: any errors ? | 13:58 |
| rfolco | no | 13:58 |
| rfolco | no | 13:58 |
| panda | rfolco: you're lying | 13:58 |
| rfolco | :'-( | 13:58 |
| panda | rfolco: how do you know it's not destroyed ? | 13:58 |
| rfolco | sudo docker ps -a ? | 13:58 |
| marios | ack rfolco | 13:59 |
| *** Vorrtex has joined #oooq | 13:59 | |
| weshay | amoralej, k.. can we see that in an rdo-info job? | 14:00 |
| weshay | I don't think ovb is there | 14:00 |
| weshay | amoralej, we did see a 7.7 ovb queens job pass.. at least once or twice | 14:01 |
| amoralej | mmm | 14:01 |
| weshay | but it's still a good lead | 14:01 |
| amoralej | not sure if we have a ovb in experimental | 14:01 |
| amoralej | for rdoinfo tbh | 14:01 |
| amoralej | need to check | 14:01 |
| amoralej | i'm joingin a mtg now | 14:01 |
| ykarel | weshay, a queens job passed with 7.7 | 14:04 |
| ykarel | the diffence there was it was running kernel from centos7.6 | 14:05 |
| ykarel | but yes failing has kernel from 7.7 | 14:05 |
| ykarel | with pass i meant it moved till tempest | 14:08 |
| ykarel | didn't failed at ssh | 14:08 |
| weshay | sshnaidm, marios https://review.opendev.org/685090 | 14:10 |
| ykarel | reference:- https://logs.rdoproject.org/10/587310/9/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/2f67ee3/logs/undercloud/etc/redhat-release.txt.g | 14:10 |
| rlandy | chandankumar: I think https://review.rdoproject.org/r/#/c/22549/ is uncontroversial and can probably go when your patches are in . fixing rdo-jobs patch | 14:11 |
| rlandy | https://review.rdoproject.org/r/#/c/22550/3/zuul.d/standalone-jobs.yaml | 14:14 |
| rlandy | ^^ needs review pls | 14:15 |
| marios | panda: did it drop you? i just dropped | 14:17 |
| panda | marios: you crashed my laptop | 14:19 |
| marios | panda: ack | 14:21 |
| weshay | arxcruz|rover, you on the console of ykarel's job? | 14:21 |
| rlandy | chandankumar: weshay: check jobs are removed, removed my -2 - https://review.rdoproject.org/r/#/q/topic:train-jobs+(status:open+OR+status:merged) ready for review | 14:26 |
| weshay | sshnaidm, where is ovb-setup.yml logged? | 14:26 |
| sshnaidm | weshay, logged..? | 14:27 |
| *** saneax has quit IRC | 14:28 | |
| arxcruz|rover | weshay: yes | 14:30 |
| weshay | arxcruz|rover, want to debug together? | 14:32 |
| weshay | arxcruz|rover, https://meet.google.com/tid-cmdj-jry | 14:33 |
| *** beagles-mtg is now known as beagles | 14:37 | |
| *** dalvarez has joined #oooq | 14:37 | |
| dalvarez | o/ | 14:37 |
| weshay | arxcruz|rover, ^ | 14:38 |
| weshay | jump on | 14:38 |
| amoralej | dalvarez, ykarel arxcruz|rover so what do you thing is better to troubleshoot | 14:38 |
| amoralej | update ovs in UC or reboot? | 14:38 |
| weshay | reboot? | 14:38 |
| amoralej | uc | 14:38 |
| amoralej | i think someone proposed that? | 14:39 |
| dalvarez | arxcruz|rover: was it you the one writing on tmux? :) | 14:39 |
| weshay | you want to reboot the undercloud? | 14:39 |
| ykarel | amoralej, i would prefer whichever would give us option to revert back to original | 14:39 |
| weshay | https://meet.google.com/tid-cmdj-jry?authuser=1 | 14:39 |
| ykarel | dalvarez, it was me | 14:39 |
| dalvarez | oh | 14:39 |
| dalvarez | ykarel: we can revert back the ovs thing | 14:39 |
| ykarel | dalvarez, they we can start with ovs update | 14:39 |
| dalvarez | ykarel: so you said that we have same environment, same ovs, same kernel working as well? | 14:39 |
| amoralej | ok, let me update ovs | 14:39 |
| ykarel | dalvarez, yes | 14:39 |
| amoralej | ykarel, also with br-ctlplane | 14:40 |
| amoralej | right? | 14:40 |
| amoralej | same ovs topology? | 14:40 |
| dalvarez | ykarel: amoralej ^ i was saying that maybe it's got to do with the ovs/kernel in the hypervisor where those instances are running | 14:40 |
| ykarel | amoralej, yes same | 14:40 |
| weshay | amoralej, ykarel https://review.opendev.org/#/c/618832/ | 14:40 |
| ykarel | amoralej, ssh zuul@38.145.34.241 | 14:40 |
| weshay | tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branchSUCCESS in 2h 30m 37s | 14:40 |
| dalvarez | ykarel: amoralej i disabled segmentation hw offloading and still saw packets larger than the mtu for some reason | 14:40 |
| * ykarel looks | 14:40 | |
| weshay | dalvarez, should be at 1350 right? | 14:41 |
| dalvarez | weshay: yeah, but still saw packets of 1600 or so | 14:41 |
| weshay | neutron.conf:130:global_physnet_mtu=1350 | 14:41 |
| dalvarez | the mss negotiation seemed ok | 14:41 |
| dalvarez | tcp wise | 14:42 |
| amoralej | so you mean that kernel in uc vs kernel in hypervisor issue? | 14:42 |
| ykarel | weshay, in that job, "kernel": "3.10.0-957.27.2.el7.x86_64", | 14:42 |
| ykarel | old kernel from centos 7.6 | 14:42 |
| ykarel | so it's different case | 14:42 |
| weshay | ah poop | 14:42 |
| weshay | ykarel, man.. oh so much for pretest | 14:43 |
| dalvarez | 13:30:12.690879 Out fa:16:3e:e9:e9:d8 ethertype IPv4 (0x0800), length 1564: (tos 0x0, ttl 64, id 10771, offset 0, flags [DF], proto TCP (6 | 14:43 |
| dalvarez | weshay: ^ | 14:43 |
| dalvarez | even though the mss looked ok in the handshake | 14:43 |
| dalvarez | anyways i turned off the segment hw offload and looks better but still bigger packets | 14:43 |
| dalvarez | ykarel: so the one that works is using older kernel? | 14:44 |
| amoralej | the different may be that the failing jobs were created with new kernel from the begining | 14:44 |
| ykarel | dalvarez, so before reboot, old kernel + ovs 2.9 + ssh to overcloud was working | 14:44 |
| amoralej | while ykarel's one was initially set with old one and then rebooted | 14:45 |
| ykarel | amoralej, yes exactly that's the difference | 14:45 |
| ykarel | yes ^^ true | 14:45 |
| amoralej | old kernel may imply different datapaths stored in te ovs even after reboot with new kernel? | 14:45 |
| amoralej | dalvarez, ^ | 14:45 |
| dalvarez | amoralej: no | 14:46 |
| dalvarez | if a new kernel is rebooted, the datapath from that kernel is used | 14:46 |
| dalvarez | unless we load the modules explicitly | 14:46 |
| dalvarez | which we should not | 14:46 |
| amoralej | i'm inclined to update ovs | 14:47 |
| amoralej | tbh | 14:47 |
| dalvarez | hmm | 14:47 |
| dalvarez | wait | 14:47 |
| dalvarez | ykarel: do you have access to that vm that is working? can you "lsmod |grep openvswitch" ? | 14:47 |
| ykarel | dalvarez, yes | 14:47 |
| ykarel | [zuul@undercloud ~]$ lsmod |grep openvswitch | 14:47 |
| ykarel | openvswitch 114838 5 | 14:47 |
| ykarel | nf_nat_ipv6 14131 1 openvswitch | 14:47 |
| ykarel | nf_defrag_ipv6 35104 2 openvswitch,nf_conntrack_ipv6 | 14:47 |
| ykarel | nf_nat_ipv4 14115 2 openvswitch,iptable_nat | 14:47 |
| ykarel | nf_nat 26583 5 nf_nat_redirect,openvswitch,nf_nat_ipv4,nf_nat_ipv6,nf_nat_masquerade_ipv4 | 14:47 |
| ykarel | nf_conntrack 139224 9 ip_vs,openvswitch,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_conntrack_ipv6 | 14:47 |
| ykarel | libcrc32c 12644 4 ip_vs,openvswitch,nf_nat,nf_conntrack | 14:47 |
| dalvarez | ok | 14:47 |
| dalvarez | and using 3.10.0-1062.1.1.el7.x86_64 ? | 14:48 |
| *** aakarsh has joined #oooq | 14:48 | |
| ykarel | dalvarez, yes | 14:49 |
| ykarel | [zuul@undercloud ~]$ uname -r | 14:49 |
| ykarel | 3.10.0-1062.1.1.el7.x86_64 | 14:49 |
| dalvarez | (undercloud) [zuul@undercloud ~]$ modinfo openvswitch ?····························· | 14:49 |
| dalvarez | filename: /lib/modules/3.10.0-1062.1.1.el7.x86_64/kernel/net/openvswitch/openvswitch.ko.xz | 14:49 |
| dalvarez | ykarel: ^ this module right? | 14:49 |
| amoralej | ykarel, you remember if we can run ovb in rdoinfo gate? | 14:49 |
| weshay | ykarel, that's what the test job used | 14:50 |
| weshay | "BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1062.1.1.el7.x86_64", | 14:50 |
| ykarel | amoralej, i think i have tried it earlier | 14:50 |
| amoralej | i remember we tried | 14:50 |
| amoralej | but don't remember if it worked | 14:50 |
| ykarel | amoralej, and there were some some issues iirc, but could be fixed | 14:50 |
| amoralej | to test it with 2.11 in a DNM | 14:51 |
| ykarel | weshay, yes | 14:51 |
| dalvarez | amoralej: i'd check also rdocloud (ie. maybe it fails when they nodes are on the same (or certain) hypervisors) | 14:51 |
| dalvarez | s/they/the | 14:51 |
| weshay | ykarel, http://logs.rdoproject.org/32/618832/7/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/6c2fe25/logs/overcloud-controller-0/var/log/extra/dump_variables_vars.json.txt.gz | 14:51 |
| amoralej | dalvarez, it fails always | 14:51 |
| weshay | ykarel, same kernel w/ a passing job | 14:51 |
| dalvarez | amoralej: always? weshay just linked this ^ passing | 14:52 |
| dalvarez | :? | 14:52 |
| amoralej | mm | 14:52 |
| dalvarez | and also it works when the kernel is upgraded | 14:52 |
| amoralej | always in periodic :) | 14:52 |
| dalvarez | hah is periodic somehow pinning certain rdocloud nodes? | 14:52 |
| amoralej | and yatin reproducer is in a different tenant | 14:53 |
| dalvarez | looks like mtu issue definitely | 14:53 |
| amoralej | may be related to tenant configuration? | 14:53 |
| *** aakarsh has quit IRC | 14:53 | |
| dalvarez | i can even ping with 2000B because the router namespace is fragmenting | 14:53 |
| ykarel | weshay, that is for overcloud, undercloud is 7.6 kernel | 14:53 |
| * ykarel rechecks | 14:54 | |
| weshay | ykarel, openstack-neutron-openvswitch-12.1.1-0.20190910010613.28f3e37.el7.noarch | 14:54 |
| weshay | same ovs package on the passing job | 14:55 |
| weshay | ykarel, ya.. the nodepool node may be old /me checks that job's undercloud | 14:55 |
| ykarel | weshay, yes | 14:56 |
| weshay | correct.. kernel on the nodepool node.. is "BOOT_IMAGE": "/boot/vmlinuz-3.10.0-957.27.2.el7.x86_64", | 14:56 |
| amoralej | http://logs.rdoproject.org/32/618832/7/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/6c2fe25/logs/undercloud/var/log/journal.txt.gz | 14:56 |
| amoralej | old kernel | 14:56 |
| weshay | ya | 14:56 |
| weshay | I don't think we can reboot the nodepool node into a new kernel at run time | 14:57 |
| weshay | that's just how it is :( | 14:57 |
| weshay | ykarel, amoralej perhaps in the future we need to build a test nodepool node w/ cr? | 14:58 |
| ykarel | yes that should help detect these issues | 14:58 |
| amoralej | weshay, much better we'll have centos streams | 14:58 |
| weshay | if this really ends up being the kernel | 14:58 |
| amoralej | ok | 15:00 |
| amoralej | so,,, my proposal | 15:00 |
| amoralej | 1. reboot as is | 15:00 |
| amoralej | 2. we may try to downgrade kernel | 15:00 |
| amoralej | and check what happens | 15:00 |
| amoralej | or | 15:00 |
| amoralej | 3. update ovs | 15:00 |
| dalvarez | amoralej: cant we just update ovs first? :) | 15:00 |
| amoralej | ok | 15:00 |
| weshay | that would be interesting.. to downgrade the kernel on the undercloud that yatin has held | 15:01 |
| dalvarez | but that'd just update the userspace | 15:01 |
| amoralej | let's try | 15:01 |
| amoralej | downgrade kernel vs upgrade ovs :) | 15:01 |
| amoralej | we can generate other reproducer quite easyily if needed | 15:01 |
| ykarel | but downgrading kernel would also need reboot i think | 15:03 |
| amoralej | i'll update ovs, if needed let's rebuild | 15:03 |
| amoralej | yes | 15:03 |
| amoralej | it does | 15:03 |
| amoralej | that's why i proposed to reboot first | 15:03 |
| amoralej | to check rebooting itself does not change | 15:03 |
| ykarel | yup that would give some lead to our theory | 15:04 |
| rlandy | chandankumar: I added a separate task for Train check jobs in RDO (marked it blocked) and moved the two cards for periodic train to 'ready for review' | 15:05 |
| rlandy | amoralej: when you have time, pls review https://review.rdoproject.org/r/#/c/22582/ - took a shot at adding the weirdo jobs for train promotions in ci.centos. | 15:09 |
| *** skramaja has quit IRC | 15:15 | |
| amoralej | weshay, we may get other token for the console? | 15:15 |
| amoralej | i think it expired | 15:15 |
| panda | I'm going to merge https://review.rdoproject.org/r/22538 | 15:17 |
| weshay | dalvarez, I think the overclouds are gone | 15:20 |
| dalvarez | weshay: yeah | 15:20 |
| dalvarez | looks like it :S | 15:20 |
| weshay | amoralej, or ykarel will have to spin up another env and hold | 15:20 |
| ykarel | ovecloud gone? | 15:21 |
| ykarel | how | 15:21 |
| chandankumar | rlandy: ack | 15:21 |
| ykarel | may be cleanup script delete, which delete overclouds older than 5 hours, right? | 15:23 |
| ykarel | or someone manually deleted it | 15:23 |
| weshay | ykarel, ya.. the reaper | 15:24 |
| weshay | script | 15:24 |
| ykarel | :( | 15:24 |
| weshay | could go turn that off | 15:24 |
| ykarel | iirc there was an option added to skip some nodes, but can't recall | 15:24 |
| weshay | rlandy, which infra server is the cleanup script running on these days? | 15:25 |
| weshay | hubbot? | 15:25 |
| sshnaidm | weshay, te-broker | 15:26 |
| weshay | sshnaidm, there is no instance in infra w/ that name anymore | 15:27 |
| sshnaidm | weshay, 38.145.33.166 | 15:29 |
| weshay | that's odd | 15:29 |
| weshay | oh.. that's in openstack-ifnra | 15:30 |
| weshay | sorry | 15:30 |
| weshay | openstack-nodepool rather | 15:30 |
| weshay | ya | 15:30 |
| weshay | :) | 15:30 |
| sshnaidm | weshay, it's weshay and whayutin users there :) | 15:30 |
| weshay | sshnaidm, rlandy so we don't have a way to easily exclude certain stacks / servers | 15:33 |
| sshnaidm | weshay, no | 15:33 |
| weshay | we're debugging a queens kernel issue | 15:33 |
| weshay | adding an exception list an ok idea? | 15:33 |
| rlandy | weshay: no | 15:39 |
| rlandy | adding a an exception list is ok - if we have the resources to exclude some | 15:40 |
| weshay | sshnaidm, ok by you? | 15:41 |
| weshay | I would assume exceptions are submitted and removed via gerrit | 15:41 |
| sshnaidm | weshay, yeah, fine by me | 15:42 |
| panda | zbr|ruck: arxcruz|rover merging a change that could impact the promoter server, I'll keep an eye for the next hour, but if it's acting weird you can revert the patch. | 15:44 |
| rlandy | panda: which patch is merging? | 15:46 |
| rlandy | https://review.rdoproject.org/r/#/c/22538/? | 15:46 |
| panda | rlandy: yes | 15:47 |
| panda | rlandy: I touch the production code only in two points | 15:47 |
| panda | rlandy: I add a raise to output any error at the end of the promotion, and install ansible 2.8 in the virtualenv | 15:47 |
| panda | oh well three points | 15:48 |
| rlandy | 2019-09-26 15:38:36.677398 | TASK [promoter : check if dlrn api actually finished a promotion process] | 15:48 |
| rlandy | 2019-09-26 15:38:37.726440 | rdo-centos-7 | ERROR | 15:48 |
| panda | I aslo added a daemon_reload if the dlrn-promoter service change | 15:48 |
| rlandy | ^^ failing | 15:48 |
| rlandy | non-voting - guess that's ok | 15:48 |
| panda | .... I knew it ... I make the promoter skip the last log line if an error occurs | 15:49 |
| panda | it never actually writes "FINISHED" if if fails with an error | 15:49 |
| panda | which makes sense in some way | 15:49 |
| panda | but without the raise, I could not see why it was failing bore | 15:49 |
| panda | the catchall except was making everything. | 15:50 |
| panda | masking | 15:50 |
| *** tosky has quit IRC | 15:52 | |
| *** dtantsur is now known as dtantsur|afk | 15:54 | |
| rlandy | panda: k think I am done with train stuff now | 15:57 |
| rlandy | back to promoter test | 15:57 |
| panda | rlandy: ok, want to +W https://review.rdoproject.org/r/22538 or I'll just selfishly sel merge ? | 16:01 |
| chandankumar | rlandy: did we sent upstream train job patch if not then I will do tomorrow with -W | 16:01 |
| rlandy | panda: I'll w+ it | 16:02 |
| panda | rlandy: thanks | 16:02 |
| rlandy | or at least review +2 it | 16:02 |
| rlandy | give you a hint of legitimacy | 16:02 |
| rlandy | chandankumar: ack - thanks | 16:02 |
| chandankumar | weshay: rlandy I have 3 public holiday in between 2-9 in india | 16:03 |
| rlandy | panda: ok - +2'ed - now you can self merge with confidence | 16:03 |
| rlandy | chandankumar: np - I have a bunch of holidays coming up | 16:04 |
| rlandy | out monday tues this coming week | 16:04 |
| rlandy | wed the next week | 16:04 |
| rlandy | and then mon and tues the two weeks following that | 16:04 |
| chandankumar | after that I will be attending pycon India from 11th Oct | 16:05 |
| rlandy | chandankumar: let's try get this complete as much as we can tomorrow | 16:05 |
| chandankumar | rlandy: yes | 16:05 |
| rlandy | so someone can pick it up in our absence | 16:05 |
| chandankumar | yup | 16:09 |
| rlandy | hmmm ... no W+ yet? | 16:11 |
| marios | panda: should be green now at https://review.rdoproject.org/r/#/c/22002/ but maybe issue with ppc for the test registries will continue tomorrow (delegated job is good now waiting for zuul to report it) | 16:11 |
| marios | ttyl | 16:11 |
| *** ykarel is now known as ykarel|afk | 16:12 | |
| *** jpena is now known as jpena|off | 16:12 | |
| rlandy | panda: question about keeping a test config file ... | 16:13 |
| *** marios has quit IRC | 16:13 | |
| rlandy | better to add the test parameters to the current staging config or keep a separate file | 16:13 |
| rlandy | noting some duplication | 16:13 |
| panda | rlandy: you can keep a file and the ovverides separated, then merge the two in the code ... | 16:14 |
| *** tesseract has quit IRC | 16:19 | |
| *** kopecmartin is now known as kopecmartin|off | 16:19 | |
| *** bogdando has quit IRC | 16:19 | |
| *** ykarel|afk has quit IRC | 16:50 | |
| *** udesale has quit IRC | 16:51 | |
| *** aakarsh has joined #oooq | 16:57 | |
| *** derekh has quit IRC | 17:01 | |
| *** Goneri has joined #oooq | 17:02 | |
| *** holser has quit IRC | 17:03 | |
| *** ykarel has joined #oooq | 17:05 | |
| *** aakarsh|2 has joined #oooq | 17:06 | |
| *** aakarsh has quit IRC | 17:08 | |
| *** weshay is now known as weshay_passport | 17:13 | |
| *** EmilienM is now known as MniLieEm | 17:15 | |
| *** MniLieEm is now known as EmilienM | 17:15 | |
| *** aakarsh|2 has quit IRC | 17:16 | |
| rlandy | panda: how do I refernce the dlrn host? | 17:29 |
| rlandy | from the staging enV? | 17:30 |
| rlandy | http://logs.rdoproject.org/48/22348/41/check/tripleo-ci-promotion-staging/61b95f9/job-output.txt.gz | 17:30 |
| *** brault has quit IRC | 17:32 | |
| arxcruz|rover | rlandy: weshay_passport panda sshnaidm https://review.opendev.org/#/c/683350/ please, +w this to fix rocky, the fs020 pass with it | 17:33 |
| *** aakarsh has joined #oooq | 17:37 | |
| *** Goneri has quit IRC | 17:44 | |
| *** matbu has quit IRC | 17:45 | |
| *** matbu has joined #oooq | 17:46 | |
| panda | rlandy: other than localhost:8080 or 127.0.0.1:8080 ? | 17:49 |
| rlandy | panda: told me no host with localhost:8080 | 17:49 |
| rlandy | I didn't try 127.0.0.1:8080 yet | 17:51 |
| rlandy | I added the http: to see if that helps | 17:51 |
| panda | rlandy: I have bandwwidht problem right now, so I did not look at that log yet | 17:52 |
| rlandy | panda: no worries - will debug it | 17:52 |
| rlandy | oh - I think it worked now | 17:53 |
| rlandy | 2019-09-26 17:52:50.186286 | rdo-centos-7 | Expected commit hash: 17234e9ab9dfab4cf5600f67f1d24db5064f1025 has not been promoted to.triple-ci-staging-promoted | 17:53 |
| rlandy | 2019-09-26 17:52:50.186349 | rdo-centos-7 | Traceback (most recent call last): | 17:53 |
| *** brault has joined #oooq | 17:53 | |
| rlandy | promoter errored | 17:53 |
| rlandy | that's right | 17:53 |
| *** brault has quit IRC | 17:54 | |
| rlandy | panda: ^^ I think it's working - it failed because promotion did not succeed | 17:58 |
| panda | rlandy: at least we have the negative test | 18:02 |
| rlandy | panda: baby steps | 18:02 |
| rlandy | better than before - no promotion - all green - this is fine | 18:04 |
| *** amoralej is now known as amoralej|off | 18:06 | |
| *** ykarel is now known as ykarel|away | 18:14 | |
| *** weshay_passport is now known as weshay | 18:45 | |
| *** Vorrtex has quit IRC | 18:52 | |
| *** dsneddon has joined #oooq | 18:53 | |
| *** Goneri has joined #oooq | 18:57 | |
| *** sshnaidm is now known as sshnaidm|off | 19:11 | |
| *** weshay has quit IRC | 19:18 | |
| *** Vorrtex has joined #oooq | 19:21 | |
| *** Vorrtex has quit IRC | 19:21 | |
| *** weshay has joined #oooq | 19:25 | |
| weshay | ugh.. | 19:26 |
| weshay | so silly | 19:26 |
| *** Vorrtex has joined #oooq | 19:27 | |
| rlandy | no passport? | 19:29 |
| rlandy | weshay: ^^? | 19:30 |
| weshay | not yet | 19:30 |
| rlandy | ugh - wow | 19:31 |
| weshay | I feel crazy right now | 19:33 |
| rlandy | maybe you're not meant to go | 19:33 |
| weshay | woot.. finally getting a laptop | 19:37 |
| weshay | rlandy, check it out https://docs.google.com/document/d/14f6zANeJeBTIu6xCYwq9p3vG4D9r0ixskZiPFIEmfVs/edit#heading=h.howov9rq5h3k | 19:43 |
| rlandy | well, well | 19:43 |
| rlandy | your influence has expanded | 19:45 |
| weshay | rlandy, that's not influence.. that is EmilienM trollign :) | 19:45 |
| weshay | trolling | 19:45 |
| EmilienM | lol | 19:46 |
| *** dtrainor has quit IRC | 19:49 | |
| *** dtrainor has joined #oooq | 19:56 | |
| *** Goneri has quit IRC | 20:04 | |
| *** holser has joined #oooq | 20:10 | |
| *** ykarel|away has quit IRC | 20:13 | |
| *** Goneri has joined #oooq | 20:20 | |
| *** holser has quit IRC | 20:22 | |
| *** Vorrtex has quit IRC | 20:40 | |
| weshay | rlandy, https://review.opendev.org/685177 | 20:44 |
| weshay | rlandy, ping to chat for a sec if it's not too late | 20:44 |
| rlandy | sure | 20:44 |
| rlandy | weshay: where do we even use that? | 20:46 |
| weshay | you're going to laugh :) | 20:46 |
| *** brault has joined #oooq | 20:46 | |
| rlandy | I could use a laugh - where? | 20:47 |
| weshay | https://meet.google.com/cnm-zyfp-owb?authuser=1 | 20:47 |
| *** brault has quit IRC | 20:50 | |
| rlandy | joining | 20:51 |
| rlandy | weshay: can you hear me? | 20:51 |
| rlandy | weshay: let me join again | 20:52 |
| *** ccamacho has quit IRC | 20:53 | |
| weshay | rlandy, http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/9f915d9/logs/reproducer-quickstart-deprecated/ | 20:57 |
| *** aakarsh has quit IRC | 21:15 | |
| weshay | rlandy, also https://review.opendev.org/#/c/685090/ | 21:25 |
| rlandy | ok | 21:26 |
| rlandy | I would have put localhost back | 21:26 |
| rlandy | but this is likely easier | 21:26 |
| rlandy | weshay: your install still going? | 21:27 |
| rlandy | after the skip? | 21:27 |
| weshay | yes | 21:27 |
| weshay | undercloud is installing | 21:27 |
| rlandy | wow - it really is the return of the dead | 21:27 |
| weshay | rlandy, also need to find where we pull the centos qcow2 and update that | 21:27 |
| *** jfrancoa has quit IRC | 21:29 | |
| rlandy | https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml#L130 | 21:29 |
| rlandy | weshay: ^^ that one? | 21:29 |
| rlandy | I added that for bm | 21:29 |
| rlandy | https://review.opendev.org/#/c/579154/ - must have needed that for some reason | 21:36 |
| rlandy | that creates the file we pass for nodes | 21:36 |
| weshay | rlandy, it's defined in create-zuul-based-reproducer/templates/launcher-playbook.yaml.j2 | 21:36 |
| *** matbu has quit IRC | 21:37 | |
| rlandy | https://review.opendev.org/#/c/636406/12/roles/prepare-node/tasks/main.yaml | 21:37 |
| rlandy | ^^ I think we need that | 21:38 |
| rlandy | which will gives us the vars zuul passes | 21:38 |
| weshay | rlandy, images are building :) | 21:40 |
| weshay | rlandy, there is life | 21:40 |
| rlandy | https://review.opendev.org/#/c/613414/1/roles/nodepool-setup/tasks/main.yml | 21:42 |
| rlandy | ^^ that was where the v3 reproducer work started | 21:42 |
| rlandy | https://tree.taiga.io/project/tripleo-ci-board/task/284 | 21:43 |
| rlandy | weshay: here we go ... https://review.opendev.org/#/c/614638/4/playbooks/tripleo-ci/run-v3.yaml | 21:46 |
| rlandy | https://tree.taiga.io/project/tripleo-ci-board/task/271 | 21:47 |
| rlandy | list of vars exported | 21:47 |
| rlandy | some of it merged | 21:47 |
| *** Goneri has quit IRC | 21:48 | |
| weshay | heh.. | 21:50 |
| weshay | seed planted | 21:50 |
| rlandy | let's see how far we got if we rebased | 21:54 |
| rlandy | if we bring this back, I really will laugh | 21:54 |
| rlandy | but it might actually be easier to keep this tested than to get others to run the zuul-based one | 21:55 |
| *** brault has joined #oooq | 21:58 | |
| *** brault has quit IRC | 22:13 | |
| rlandy | weshay: need to step away for a bit but I have traced some old reviews/tests - will show what wee still have | 22:23 |
| rlandy | after I rebase | 22:23 |
| *** rlandy is now known as rlandy|biab | 22:24 | |
| weshay | k | 22:25 |
| *** aakarsh has joined #oooq | 22:42 | |
| *** brault has joined #oooq | 22:55 | |
| *** brault has quit IRC | 23:00 | |
| *** brault has joined #oooq | 23:16 | |
| *** brault has quit IRC | 23:20 | |
| *** dsneddon has quit IRC | 23:52 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!