| *** rascasoft has joined #rdo | 00:01 | |
| *** rascasoft has quit IRC | 00:10 | |
| *** wolverineav has joined #rdo | 00:26 | |
| *** wolverineav has quit IRC | 00:30 | |
| *** rascasoft has joined #rdo | 00:34 | |
| *** rascasoft has quit IRC | 00:40 | |
| *** wolverineav has joined #rdo | 01:01 | |
| *** rascasoft has joined #rdo | 01:11 | |
| *** rascasoft has quit IRC | 01:19 | |
| *** rfolco has joined #rdo | 01:31 | |
| *** rascasoft has joined #rdo | 01:46 | |
| *** rascasoft has quit IRC | 01:52 | |
| *** wolverineav has quit IRC | 01:54 | |
| *** wolverineav has joined #rdo | 01:54 | |
| *** wolverineav has quit IRC | 02:02 | |
| *** wolverineav has joined #rdo | 02:25 | |
| *** wolverineav has quit IRC | 02:31 | |
| *** rascasoft has joined #rdo | 02:32 | |
| *** rascasoft has quit IRC | 02:41 | |
| *** rfolco has quit IRC | 02:59 | |
| *** rascasoft has joined #rdo | 03:05 | |
| *** wolverineav has joined #rdo | 03:11 | |
| *** rascasoft has quit IRC | 03:13 | |
| *** ykarel has joined #rdo | 03:36 | |
| *** rascasoft has joined #rdo | 03:43 | |
| *** rascasoft has quit IRC | 03:52 | |
| *** ramishra has joined #rdo | 04:05 | |
| *** udesale has joined #rdo | 04:08 | |
| *** wolverineav has quit IRC | 04:08 | |
| *** rascasoft has joined #rdo | 04:14 | |
| *** wolverineav has joined #rdo | 04:17 | |
| *** rascasoft has quit IRC | 04:24 | |
| *** ykarel has quit IRC | 04:33 | |
| *** satya4ever has joined #rdo | 04:41 | |
| *** ykarel has joined #rdo | 04:49 | |
| *** rascasoft has joined #rdo | 04:49 | |
| *** ykarel has quit IRC | 04:53 | |
| *** rascasoft has quit IRC | 05:00 | |
| *** rascasoft has joined #rdo | 05:13 | |
| *** imcleod has joined #rdo | 05:15 | |
| *** rascasoft has quit IRC | 05:21 | |
| *** wolverineav has quit IRC | 05:25 | |
| *** wolverineav has joined #rdo | 05:26 | |
| rdogerrit | Yatin Karel created rdo-infra/weirdo master: Pin ansible-lint to <= 3.5.0 https://review.rdoproject.org/r/18040 | 05:30 |
|---|---|---|
| rdogerrit | Yatin Karel proposed rdo-infra/ansible-role-weirdo-puppet-openstack master: Do not enable Ceph SIG repo in Fedora https://review.rdoproject.org/r/18037 | 05:32 |
| *** wolverineav has quit IRC | 05:39 | |
| *** rascasoft has joined #rdo | 05:46 | |
| *** imcleod has quit IRC | 05:49 | |
| *** agurenko has joined #rdo | 05:50 | |
| *** rascasoft has quit IRC | 05:53 | |
| *** redrobot has quit IRC | 05:55 | |
| *** wolverineav has joined #rdo | 06:15 | |
| *** wolverineav has quit IRC | 06:25 | |
| *** ykarel has joined #rdo | 06:28 | |
| *** paragn has joined #rdo | 06:37 | |
| *** radeks__ has joined #rdo | 06:39 | |
| *** radeks_ has joined #rdo | 06:46 | |
| *** radeks__ has quit IRC | 06:47 | |
| *** agurenko has quit IRC | 06:56 | |
| *** jbadiapa has joined #rdo | 06:57 | |
| *** rascasoft has joined #rdo | 06:57 | |
| *** rcernin has quit IRC | 07:00 | |
| *** numans has joined #rdo | 07:04 | |
| *** quiquell|off is now known as quiquell | 07:06 | |
| *** faizy98 has joined #rdo | 07:07 | |
| *** rascasoft has quit IRC | 07:07 | |
| quiquell | number80: you there ? | 07:08 |
| *** jfrancoa has joined #rdo | 07:09 | |
| *** rascasoft has joined #rdo | 07:10 | |
| *** ykarel_ has joined #rdo | 07:15 | |
| *** ykarel has quit IRC | 07:17 | |
| *** ykarel__ has joined #rdo | 07:20 | |
| *** aufi has joined #rdo | 07:21 | |
| *** ykarel_ has quit IRC | 07:23 | |
| *** ykarel_ has joined #rdo | 07:24 | |
| *** ykarel__ has quit IRC | 07:27 | |
| number80 | quiquell: yup? | 07:30 |
| *** ykarel_ is now known as ykarel|lunch | 07:31 | |
| ykarel|lunch | number80, welcome and Happy New Year | 07:31 |
| number80 | ykarel|lunch: Thanks, happy new year and wish you all good things :) | 07:32 |
| *** ykarel_ has joined #rdo | 07:35 | |
| *** ykarel|lunch has quit IRC | 07:38 | |
| quiquell | number80: hapy new year | 07:38 |
| number80 | :) | 07:38 |
| quiquell | number80: So I have a 2019 subunit thing for you :-) | 07:38 |
| number80 | listening | 07:38 |
| quiquell | number80: It's fixed upstream, after merge a pull request | 07:39 |
| * quiquell find for you | 07:39 | |
| quiquell | number80: https://github.com/testing-cabal/subunit/commit/58c3d364266e7550d3e00e34191eed7df2d36d13 | 07:39 |
| quiquell | what's the easies way to have this at our RPMs ? | 07:39 |
| quiquell | I see there are some patches of merged stuff in the fedora .spec file | 07:40 |
| number80 | quiquell: I can refresh the package with that patch | 07:40 |
| number80 | Yes, it was not included in latest release | 07:40 |
| quiquell | number80: Does it make senes to have a new release at launchpad ? | 07:40 |
| quiquell | https://launchpad.net/subunit | 07:41 |
| *** brault has joined #rdo | 07:41 | |
| quiquell | Latest is 2018-04-07 | 07:41 |
| quiquell | Also I see the .spec have patches for alread merged stuff | 07:42 |
| number80 | quiquell: it would be nice but up to the maintainer | 07:42 |
| quiquell | Robert Collins is the main maintener | 07:42 |
| quiquell | Was not able to contact himn | 07:42 |
| *** brault has quit IRC | 07:43 | |
| chandankumar | Happy new year #rdo :-) | 07:43 |
| quiquell | chandankumar: o/ | 07:45 |
| chandankumar | quiquell: \o | 07:45 |
| quiquell | number80: let's refresh the package so we have it and fix the f28 job for subunit report | 07:45 |
| quiquell | number80: We can try to contact later the mainteiner, what doyou think ? | 07:45 |
| number80 | works for me | 07:46 |
| *** arxcruz|next_yr is now known as arxcruz | 07:47 | |
| *** satya4ever is now known as sbulage|lunch | 07:54 | |
| *** slaweq has joined #rdo | 07:54 | |
| *** avivgt has joined #rdo | 07:57 | |
| *** matbu has joined #rdo | 07:58 | |
| *** rpittau has joined #rdo | 08:00 | |
| *** quiquell is now known as quiquell|brb | 08:04 | |
| *** jtomasek has joined #rdo | 08:06 | |
| *** ccamacho has joined #rdo | 08:11 | |
| *** tosky has joined #rdo | 08:14 | |
| *** jlibosva has joined #rdo | 08:18 | |
| *** faizy_ has joined #rdo | 08:23 | |
| *** faizy98 has quit IRC | 08:26 | |
| *** ykarel_ is now known as ykarel | 08:33 | |
| *** janki has joined #rdo | 08:37 | |
| ykarel | number80, hi | 08:39 |
| number80 | hi | 08:39 |
| ykarel | number80, we need updated version of rfc3896 | 08:39 |
| ykarel | i have filed https://bugzilla.redhat.com/show_bug.cgi?id=1662840 | 08:40 |
| openstack | bugzilla.redhat.com bug 1662840 in python-rfc3986 "python-rfc3986: Bump to latest i.e 1.2.0" [Unspecified,New] - Assigned to apevec | 08:40 |
| number80 | ack looking | 08:40 |
| ykarel | seems apevec is maiintainer there | 08:40 |
| ykarel | and jpena | 08:40 |
| number80 | I think they're both on PTO, so I'll deal with it | 08:41 |
| ykarel | number80, yes they are on PTO | 08:41 |
| ykarel | should i send PR? | 08:42 |
| ykarel | number80, also i have a doubt | 08:48 |
| ykarel | number80, i was trying to understand the Release naming for mariadb: https://github.com/rdo-common/mariadb/blob/common-rdo/mariadb.spec#L147 | 08:48 |
| ykarel | why it's %{?dist}.0.0.rdo2? | 08:48 |
| *** jpich has joined #rdo | 08:49 | |
| number80 | ykarel: it's a slight fork of Fedora package, we had agreed on that convention with dcibriani before | 08:50 |
| number80 | (and bandini if I remember correctly) | 08:50 |
| ykarel | number80, ack so it's not standard that we follow | 08:51 |
| ykarel | some exception, right | 08:51 |
| *** quiquell|brb is now known as quiquell | 08:52 | |
| ykarel | number80, also why not <release>.0.0.rdo2.%{?dist} ? vs <release>.%{?dist}.0.0.rdo2 | 08:54 |
| ykarel | actually while reading Fedora docs i could see some examples for former convention but not for later, so asking | 08:54 |
| number80 | usually, we tend to put %dist last, except for rebuilds because of build systems blunders (since Koji enforces NVR unicity) | 08:57 |
| quiquell | number80: Let me know when you have the review, so we can see if it worked | 08:57 |
| ykarel | number80, ack, makes sense, Thanks | 08:57 |
| number80 | quiquell: in progress, pushing a build in Fedora | 08:58 |
| quiquell | number80: cool thanks so much, It's the very last piece to have a working f28 job :-) | 09:00 |
| quiquell | (for now) | 09:00 |
| *** janki has quit IRC | 09:01 | |
| *** janki has joined #rdo | 09:01 | |
| *** holser_ has joined #rdo | 09:11 | |
| *** holser_ has quit IRC | 09:21 | |
| *** holser_ has joined #rdo | 09:22 | |
| ykarel | tosky, hi i tested sahara with python3 with some changes(https://review.openstack.org/#/c/625613/11/run_tests.sh@213) and tempest tests worked | 09:27 |
| ykarel | i saw you had some patches to make sahara work with python3 | 09:27 |
| ykarel | may be u are already aware of the issues | 09:27 |
| ykarel | if not can u take care of ^^, or if u want me to push patch let me know, i can do that | 09:28 |
| tosky | ykarel: there are some issues still | 09:29 |
| tosky | the current tempest tests will work, but the provisioning won't | 09:29 |
| ykarel | tosky, ack, i just tested poi scenaio003 | 09:29 |
| tosky | if that's working, I'm all for enabling it | 09:30 |
| ykarel | yes with those changes it's working | 09:30 |
| tosky | this fixes part of the issues with provisioning: https://review.openstack.org/#/c/626643/ | 09:30 |
| ykarel | tosky, ack | 09:31 |
| tosky | ykarel: and please send a patch for sahara-tests | 09:31 |
| tosky | so we can see for sure if they work also on python2 | 09:32 |
| tosky | oh, one patch is for sahara | 09:32 |
| ykarel | tosky, ok will send patch to sahara-tests, do i need to file bug also? | 09:32 |
| *** Son_Goku has quit IRC | 09:32 | |
| ykarel | yes one is in shara | 09:32 |
| ykarel | sahara also | 09:32 |
| ykarel | string_types --> binary_type | 09:33 |
| tosky | I think we have a generic "fix remaining python3 issues" story | 09:33 |
| tosky | let me find it; if it's the case, you can add new tasks | 09:34 |
| ykarel | ok will link that | 09:34 |
| tosky | uhm, no, there is the one created for the governance goal, but we are not really using it for specific issues | 09:35 |
| tosky | so feel free to create one, with two tasks: one for sahara and the other for sahara-tests | 09:35 |
| rdogerrit | Félix Enrique Llorente Pastora created rdo-infra/ansible-role-tripleo-ci-reproducer master: Add a base-minimal base job https://review.rdoproject.org/r/18041 | 09:36 |
| ykarel | tosky, ack | 09:37 |
| rdogerrit | Félix Enrique Llorente Pastora proposed rdo-infra/ansible-role-tripleo-ci-reproducer master: Add a base-minimal base job https://review.rdoproject.org/r/18041 | 09:37 |
| *** shardy has joined #rdo | 09:47 | |
| ykarel | tosky, while creating story, getting 400: POST /api/v1/stories/2004688: Invalid input for field/attribute story. Value: '2004688'. unable to convert to Story | 09:48 |
| *** fragatina has joined #rdo | 09:50 | |
| *** fragatina has quit IRC | 09:50 | |
| ykarel | looks like some bug, story is created though:https://storyboard.openstack.org/#!/story/2004688 | 09:50 |
| *** fragatina has joined #rdo | 09:51 | |
| *** fragatina has quit IRC | 09:51 | |
| *** fragatina has joined #rdo | 09:52 | |
| *** nijaba- has quit IRC | 09:52 | |
| tosky | ykarel: I've seen it, but I didn't pinpoint when it happens (maybe when you create a story with tasks assigned to different projects, or more tasks in general, but not sure) | 09:54 |
| ykarel | i tried with both, same projects, different projects, failed both times | 09:54 |
| *** nijaba has joined #rdo | 09:56 | |
| *** fragatina has quit IRC | 09:56 | |
| *** Son_Goku has joined #rdo | 09:58 | |
| *** faizy98 has joined #rdo | 10:03 | |
| ykarel | tosky, proposed patches to both sahara and sahara-tests | 10:04 |
| *** faizy_ has quit IRC | 10:06 | |
| *** faizy_ has joined #rdo | 10:07 | |
| *** faizy98 has quit IRC | 10:08 | |
| *** faizy98 has joined #rdo | 10:11 | |
| *** faizy_ has quit IRC | 10:12 | |
| tosky | ykarel: noticed them, thanks, let's see the gates | 10:14 |
| ykarel | ack | 10:14 |
| *** faizy98 has quit IRC | 10:16 | |
| ykarel | number80, opened PR for python-rfc3986: https://src.fedoraproject.org/rpms/python-rfc3986/pull-request/2 | 10:17 |
| ykarel | i had forked it earlier | 10:17 |
| tosky | ykarel: the patch for sahara.git has a first failure on the python3.6 unit test job (!) | 10:21 |
| ykarel | tosky, ack will look | 10:21 |
| *** tigalch has joined #rdo | 10:21 | |
| *** moguimar has joined #rdo | 10:24 | |
| *** factor has joined #rdo | 10:24 | |
| factor | If I wanted to install an older version of openstack which URLs would I use. | 10:26 |
| number80 | ykarel: ack | 10:26 |
| factor | If you go to rdoproject.org , it no longer works. | 10:27 |
| number80 | factor: unsupported ones? | 10:28 |
| ykarel | tosky, looks like some issue with kombu pypi build, local tox -epy36 not working | 10:28 |
| factor | yes | 10:28 |
| factor | number80, ^ | 10:28 |
| factor | Liberty | 10:28 |
| ykarel | tosky, it looks for 4.2.2 but it's not available here https://pypi.org/project/kombu/#history, seems deleted | 10:29 |
| ykarel | yesterday | 10:29 |
| tosky | yay! | 10:29 |
| ykarel | in ci may be working because of some cache | 10:29 |
| number80 | factor: we don't provide release package for older releases, but you can always write a repo file pointing to https://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-liberty/ | 10:30 |
| tosky | ykarel: there is a 4.2.2.post1 though | 10:30 |
| ykarel | tosky, yes but in u-c it looks for 4.2.2 | 10:31 |
| factor | Looking. | 10:31 |
| ykarel | in a venv, pip install kombu==4.2.2 fails | 10:31 |
| ykarel | in CI job, i see 4.2.2 is installed | 10:31 |
| number80 | factor: here's one you can use as a template => https://github.com/rdo-infra/rdo-release/blob/queens-rdo/rdo-testing.repo | 10:32 |
| tosky | ykarel: argh | 10:32 |
| number80 | s/queens/liberty/ should be enough | 10:32 |
| tosky | ykarel: that's for, uhm, -release? | 10:32 |
| number80 | Even better => https://github.com/rdo-infra/rdo-release/tree/liberty-rdo (this includes the GPG key used to signed liberty packages) | 10:32 |
| factor | number80, So I can still use the modern quickstart.sh with repos pointed toward liberty. | 10:33 |
| number80 | factor: it should work with quickstart.sh | 10:34 |
| ykarel | tosky, ? not understood ur question | 10:34 |
| number80 | just make sure to disable the release repo as they were removed for mirrors.centos.org | 10:34 |
| ykarel | number80, quickstart.sh has removed checks for older releases, so should face issues | 10:35 |
| ykarel | there was some recent cleanup for releases < newton | 10:35 |
| tosky | ykarel: I think that upper-constraints changes are handled by openstack-release | 10:35 |
| factor | hum | 10:35 |
| ykarel | tosky, yes | 10:35 |
| factor | Can I find the old quickstart.sh and it would be fine? | 10:36 |
| number80 | ykarel: ha, they removed them | 10:36 |
| factor | I m trying to install the older and new version to see the differences. | 10:36 |
| ykarel | tosky, kombu seems an external dep, so not managed from openstack/releases | 10:37 |
| number80 | factor: try this commit => https://github.com/openstack/tripleo-quickstart/commit/e356f978343a758f4791286e899f2287d6416294 | 10:37 |
| number80 | (maybe they removed some liberty stuff before without me finding, I'm not 100% sure | 10:38 |
| ykarel | so example this cleaned up libety stuff: https://review.openstack.org/#/c/619519/ | 10:39 |
| ykarel | https://review.openstack.org/#/q/topic:bug/1722602+(status:open+OR+status:merged) | 10:39 |
| number80 | Yep, the commit I'm pointing at was the one right before this review | 10:42 |
| ykarel | ack | 10:42 |
| *** gfidente has joined #rdo | 10:45 | |
| ykarel | tosky, in ci it's working from mirror | 10:47 |
| ykarel | tosky, like pip install --only-binary all --trusted-host mirror.mtl01.inap.openstack.org -i http://mirror.mtl01.inap.openstack.org/wheel/centos-7-x86_64/ kombu==4.2.2 working | 10:47 |
| tosky | but it will explode at some point, I suspect | 10:48 |
| ykarel | looking at http://mirror.mtl01.inap.openstack.org/wheel/centos-7-x86_64/kombu/ it seems older releases are kept: http://mirror.mtl01.inap.openstack.org/wheel/centos-7-x86_64/kombu/ | 10:48 |
| rdogerrit | Félix Enrique Llorente Pastora created rdo-infra/ansible-role-tripleo-ci-reproducer master: Don't pre-create logs dir https://review.rdoproject.org/r/18042 | 10:49 |
| *** fragatina has joined #rdo | 10:49 | |
| factor | Yes, im giving up on installing older version. | 10:50 |
| *** fragatina has quit IRC | 10:50 | |
| ykarel | and 4.2.2 don't exist at https://pypi.org/simple/kombu/ | 10:51 |
| *** fragatina has joined #rdo | 10:51 | |
| *** fragatina has quit IRC | 10:51 | |
| *** fragatina has joined #rdo | 10:53 | |
| rdogerrit | Merged rdo-infra/ansible-role-tripleo-ci-reproducer master: Add a base-minimal base job https://review.rdoproject.org/r/18041 | 10:53 |
| rdogerrit | Merged rdo-infra/ansible-role-tripleo-ci-reproducer master: Don't pre-create logs dir https://review.rdoproject.org/r/18042 | 10:55 |
| *** fragatina has quit IRC | 10:56 | |
| rdogerrit | Félix Enrique Llorente Pastora created rdo-infra/ansible-role-tripleo-ci-reproducer master: Add nodepool_type option for host and openstack https://review.rdoproject.org/r/18043 | 11:02 |
| rdogerrit | Félix Enrique Llorente Pastora proposed rdo-infra/ansible-role-tripleo-ci-reproducer master: WIP: Add nodepool_type option for host and openstack https://review.rdoproject.org/r/18043 | 11:02 |
| rdogerrit | Félix Enrique Llorente Pastora proposed rdo-infra/ansible-role-tripleo-ci-reproducer master: WIP: Add nodepool_type option for host and openstack https://review.rdoproject.org/r/18043 | 11:06 |
| rdogerrit | Félix Enrique Llorente Pastora proposed rdo-infra/ansible-role-tripleo-ci-reproducer master: WIP: Add nodepool_type option for host and openstack https://review.rdoproject.org/r/18043 | 11:07 |
| *** quiquell is now known as quiquell|brb | 11:11 | |
| ykarel | number80, apevec has merged the PR | 11:16 |
| ykarel | and i builded it https://koji.fedoraproject.org/koji/buildinfo?buildID=1176593 | 11:16 |
| ykarel | proposing rebuild in RDO | 11:16 |
| number80 | okie dokie | 11:17 |
| number80 | I was checking that we needn't to update any other packages in rawhide | 11:17 |
| *** udesale has quit IRC | 11:18 | |
| ykarel | number80, ack | 11:19 |
| rdogerrit | Yatin Karel created rdoinfo master: Rebuild python-rfc3986-1.2.0-1 from Fedora for stein https://review.rdoproject.org/r/18044 | 11:21 |
| ykarel | number80, ^^ let's see | 11:21 |
| *** agurenko has joined #rdo | 11:36 | |
| *** holser_ is now known as holser|lunch | 11:37 | |
| *** agurenko has quit IRC | 11:37 | |
| *** sbulage|lunch is now known as sbulage|brb | 11:38 | |
| panda | Is anyone feeling like reviewing ? https://review.rdoproject.org/r/17744 | 11:41 |
| rdogerrit | Félix Enrique Llorente Pastora proposed rdo-infra/ansible-role-tripleo-ci-reproducer master: WIP: Add nodepool_type option for host and openstack https://review.rdoproject.org/r/18043 | 11:44 |
| rdogerrit | Merged rdo-infra/ansible-role-tripleo-ci-reproducer master: WIP: Add nodepool_type option for host and openstack https://review.rdoproject.org/r/18043 | 11:47 |
| *** sshnaidm is now known as sshnaidm|afk | 11:48 | |
| rdogerrit | Félix Enrique Llorente Pastora created rdo-infra/ansible-role-tripleo-ci-reproducer master: Add missing files https://review.rdoproject.org/r/18045 | 11:52 |
| rdogerrit | Félix Enrique Llorente Pastora created rdo-infra/ansible-role-tripleo-ci-reproducer master: WIP: Add CI using using "host" nodepool https://review.rdoproject.org/r/18046 | 12:09 |
| *** quiquell|brb is now known as quiquell | 12:11 | |
| *** jchhatbar has joined #rdo | 12:12 | |
| rdogerrit | Merged rdo-infra/ansible-role-tripleo-ci-reproducer master: Add missing files https://review.rdoproject.org/r/18045 | 12:13 |
| rdogerrit | Félix Enrique Llorente Pastora proposed rdo-infra/ansible-role-tripleo-ci-reproducer master: WIP: Add CI using using "host" nodepool https://review.rdoproject.org/r/18046 | 12:14 |
| *** rpittau is now known as rpittau|lunch | 12:14 | |
| *** janki has quit IRC | 12:15 | |
| vkmc | number80, ykarel https://review.rdoproject.org/r/#/c/18036/ | 12:18 |
| vkmc | :) | 12:18 |
| vkmc | it's an easy one | 12:18 |
| ykarel | looking | 12:19 |
| vkmc | ykarel++ | 12:20 |
| ykarel | vkmc, any reason to not give permission to webservice? | 12:22 |
| ykarel | i see other ui package set permissions for apache user | 12:22 |
| ykarel | let me grab link | 12:23 |
| vkmc | ykarel, not that I know of... maybe it's for making it easier if you decide to use other that is not apache | 12:24 |
| factor | In the undercloud.conf it has the option -- heat_container_image= Question is : Where do I get this URL from? | 12:24 |
| vkmc | argh, that sounds confusing... again, I decided to make it that way so we don't add constrains | 12:26 |
| *** udesale has joined #rdo | 12:26 | |
| vkmc | unless there is a security concern, I considered that was the best way | 12:26 |
| rdogerrit | Félix Enrique Llorente Pastora proposed rdo-infra/ansible-role-tripleo-ci-reproducer master: WIP: Add CI using using "host" nodepool https://review.rdoproject.org/r/18046 | 12:27 |
| ykarel | vkmc, ack if the files itself dont' contain any secure information, don't think any security issue is there | 12:30 |
| vkmc | ykarel, no, they are just configuration files for manila ui | 12:31 |
| ykarel | vkmc, ack +2 | 12:34 |
| vkmc | ykarel, thx | 12:34 |
| rdogerrit | Félix Enrique Llorente Pastora proposed rdo-infra/ansible-role-tripleo-ci-reproducer master: WIP: Add CI using using "host" nodepool https://review.rdoproject.org/r/18046 | 12:35 |
| rdogerrit | Félix Enrique Llorente Pastora created rdo-infra/ansible-role-tripleo-ci-reproducer master: Do openstack stuff only at proper nodepool type https://review.rdoproject.org/r/18047 | 12:35 |
| *** rlandy has joined #rdo | 12:39 | |
| quiquell | ykarel: we can use only github accounts at https://review.rdoproject.org/auth/login/ ? | 12:40 |
| ykarel | quiquell, afaik yes | 12:40 |
| quiquell | ykarel: ack | 12:40 |
| ykarel | quiquell, if u are looking for software factory in general, i think there must be other ways to login though | 12:41 |
| quiquell | ykarel: juse one to create a fake account to test some stuff | 12:42 |
| rdogerrit | Félix Enrique Llorente Pastora proposed rdo-infra/ansible-role-tripleo-ci-reproducer master: WIP: Add CI using using "host" nodepool https://review.rdoproject.org/r/18046 | 12:42 |
| *** rlandy is now known as rlandy|rover | 12:42 | |
| ykarel | quiquell, ack | 12:42 |
| *** rfolco has joined #rdo | 12:43 | |
| *** sshnaidm|afk is now known as sshnaidm | 12:43 | |
| factor | Look slike heat is installed by default for TripleO , no need to enable it. | 12:45 |
| *** holser|lunch is now known as holser_ | 12:46 | |
| * factor Takes a break while installing. | 12:46 | |
| *** quiquell is now known as quiquell|lunch | 12:48 | |
| ykarel | number80, rebuild for python-rfc3896 failing | 12:48 |
| ykarel | even locally it's failing but can't get what's the error: bash validate_rebuild_fedora.sh cloud7-openstack-stein-el7 python-rfc3986-1.2.0-1.fc30 | 12:49 |
| ykarel | Task id is returned blank | 12:49 |
| rdogerrit | Merged rdo-infra/ansible-role-tripleo-ci-reproducer master: Do openstack stuff only at proper nodepool type https://review.rdoproject.org/r/18047 | 12:49 |
| ykarel | ok, locally failing with: AuthError: Unknown user: yatinkarel | 12:51 |
| ykarel | but need to check in CI, logs are not collected for workdir | 12:51 |
| *** jcoufal has joined #rdo | 12:52 | |
| *** imcleod has joined #rdo | 12:56 | |
| *** sbulage|brb is now known as sbulage | 12:59 | |
| *** dprince has joined #rdo | 13:05 | |
| *** rh-jelabarre has joined #rdo | 13:07 | |
| *** avivgt has quit IRC | 13:09 | |
| *** bcafarel has quit IRC | 13:10 | |
| rdogerrit | Yatin Karel created config master: Collect logs for job: legacy-validate-dependencies-rebuilds https://review.rdoproject.org/r/18048 | 13:11 |
| ykarel | number80, to collect logs from that job ^^ | 13:11 |
| number80 | excellent :) | 13:12 |
| *** Guest67237 has joined #rdo | 13:13 | |
| *** Guest67237 is now known as redrobot | 13:13 | |
| *** imcleod has quit IRC | 13:14 | |
| *** rpittau|lunch is now known as rpittau | 13:15 | |
| *** trown|outtypewww is now known as trown | 13:17 | |
| *** dougbtv has joined #rdo | 13:20 | |
| rdogerrit | Merged config master: Collect logs for job: legacy-validate-dependencies-rebuilds https://review.rdoproject.org/r/18048 | 13:20 |
| rdogerrit | rdo-trunk created rdoinfo master: CBS tags sync with uc for stein-candidate https://review.rdoproject.org/r/18049 | 13:23 |
| ykarel | number80, so logs are collected | 13:31 |
| ykarel | number80, Error: 2019-01-02 13:24:52,081 [ERROR] koji: SSLError: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1822) | 13:31 |
| ykarel | https://logs.rdoproject.org/44/18044/1/check/legacy-validate-dependencies-rebuilds/7fcdef8/build_env/python-rfc3986_meta/task_build_output | 13:31 |
| number80 | ha, likely certificate died | 13:31 |
| *** imcleod has joined #rdo | 13:31 | |
| ykarel | number80, ack, how to check certificate are valid | 13:32 |
| number80 | centos-cert -f <file> but we only upload an encrypted version | 13:33 |
| number80 | centos-cert -v -f <file> | 13:33 |
| ykarel | number80, ack, | 13:36 |
| ykarel | number80, i don't have those certificates, can u check | 13:36 |
| ykarel | is the secret: koji_centos, contains those certs? | 13:37 |
| number80 | yes, I generated a new one and will be submitting an update | 13:38 |
| ykarel | number80, ack, Thanks | 13:38 |
| *** sbulage is now known as sbulage|afk | 13:39 | |
| *** ne0h|nihilist has joined #rdo | 13:39 | |
| *** imcleod has quit IRC | 13:47 | |
| *** quiquell|lunch is now known as quiquell | 13:48 | |
| *** imcleod has joined #rdo | 13:49 | |
| *** mcornea has joined #rdo | 13:52 | |
| rdogerrit | Sagi Shnaidman created rdo-infra/ansible-role-tripleo-ci-reproducer master: Fix nodepool type check https://review.rdoproject.org/r/18050 | 13:52 |
| sshnaidm | quiquell, it's broken right now ^^ | 13:53 |
| quiquell | Argg | 13:53 |
| quiquell | we need to add some CI I am on it | 13:53 |
| rdogerrit | hguemar created config master: Update CBS certicate https://review.rdoproject.org/r/18051 | 13:53 |
| quiquell | I have this change at latest patchset I think | 13:53 |
| number80 | ykarel: ^ | 13:55 |
| ykarel | number80, ack, | 13:55 |
| rdogerrit | Merged rdo-infra/ansible-role-tripleo-ci-reproducer master: Fix nodepool type check https://review.rdoproject.org/r/18050 | 13:59 |
| rdogerrit | Merged openstack/manila-ui-distgit rpm-master: Add read permissions for others https://review.rdoproject.org/r/18036 | 14:01 |
| ykarel | number80, so only private_key changed, not server_ca_cert? | 14:03 |
| *** nhicher has joined #rdo | 14:04 | |
| ykarel | i see both listed under koji_centos secret | 14:04 |
| *** shardy has quit IRC | 14:07 | |
| number80 | ykarel: it doesn't change | 14:07 |
| ykarel | number80, ack | 14:07 |
| rdogerrit | Merged config master: Update CBS certicate https://review.rdoproject.org/r/18051 | 14:09 |
| rdogerrit | Rafael Folco proposed config master: Add f28 promotion job containers-build https://review.rdoproject.org/r/17738 | 14:10 |
| rdogerrit | Rafael Folco proposed config master: Add pure f28 promotion jobs (bm/containers) to periodic pipeline https://review.rdoproject.org/r/17740 | 14:11 |
| rdogerrit | Yatin Karel proposed rdoinfo master: Rebuild python-rfc3986-1.2.0-1 from Fedora for stein https://review.rdoproject.org/r/18044 | 14:12 |
| ykarel | number80, let's see ^^ | 14:12 |
| * ykarel will check from home | 14:12 | |
| *** ykarel is now known as ykarel|away | 14:12 | |
| *** ne0h|nihilist has quit IRC | 14:13 | |
| number80 | yyup | 14:16 |
| *** ykarel|away has quit IRC | 14:19 | |
| *** ramishra has quit IRC | 14:20 | |
| *** bcafarel has joined #rdo | 14:25 | |
| *** shardy has joined #rdo | 14:27 | |
| *** ykarel|away has joined #rdo | 14:34 | |
| *** mjturek has joined #rdo | 14:39 | |
| *** slaweq_ has joined #rdo | 14:41 | |
| number80 | mmm | 14:41 |
| number80 | is there a RDO cloud outage? | 14:41 |
| *** slaweq has quit IRC | 14:42 | |
| *** artom has joined #rdo | 14:48 | |
| *** ykarel|away is now known as ykarel | 14:51 | |
| *** bnemerryxmas is now known as bnemec | 14:51 | |
| *** Caterpillar has joined #rdo | 14:53 | |
| rdogerrit | Félix Enrique Llorente Pastora created config master: Add user to test tripleo ci reproducer https://review.rdoproject.org/r/18052 | 14:53 |
| ykarel | number80, scratch builds succeeded with new secret: https://cbs.centos.org/koji/taskinfo?taskID=662540 | 14:53 |
| ykarel | number80, are u seeing some issue? u asked about the Outage, i haven't seen an issue | 14:54 |
| number80 | ykarel: yep but I'd like to see a final build to be at ease :) | 14:54 |
| number80 | ykarel: we have a lot of queued jobs in the check gate | 14:54 |
| ykarel | number80, yup, jobs are looking for node | 14:54 |
| ykarel | tripleo jobs are in queued for long | 14:55 |
| number80 | Just wondering what was the issue | 14:55 |
| ykarel | weshay and team are looking into it | 14:55 |
| ykarel | seems issue with provider | 14:55 |
| quiquell | ykarel, number80: We need more users in the gerrit stream group https://review.rdoproject.org/r/#/c/18052/ | 14:59 |
| quiquell | Don't know if we can deactivate this so everywone can use that | 14:59 |
| number80 | quiquell: ack, if someone in this list also approves | 14:59 |
| number80 | sshnaidm, tristanC, rlandy|rover ^ | 15:00 |
| quiquell | Damn I need this so bad | 15:00 |
| quiquell | thanks number80 | 15:01 |
| ykarel | ack | 15:01 |
| number80 | quiquell: approved then, if they disagree, they can revert | 15:01 |
| *** slaweq_ has quit IRC | 15:01 | |
| quiquell | thanks! | 15:01 |
| rdogerrit | Merged config master: Add user to test tripleo ci reproducer https://review.rdoproject.org/r/18052 | 15:01 |
| sshnaidm | need to do it available for everyone | 15:01 |
| sshnaidm | like it's done in upstream | 15:02 |
| quiquell | number80: ^ do you know if it's possible at RDO to do that ? | 15:02 |
| *** slaweq_ has joined #rdo | 15:03 | |
| ykarel | i think there was some discussion with tristanC or jpena for this, so good to cross check with them | 15:04 |
| *** weshay|ruck has joined #rdo | 15:04 | |
| mjturek | o/ | 15:05 |
| mjturek | oh - is there a meeting today? | 15:06 |
| tosky | ykarel: the fix for requirements (with the proper version of kombu) was just merged, it was already on the way: https://review.openstack.org/#/c/627925/ | 15:08 |
| rdogerrit | wes hayutin created rdo-infra/ci-config master: add full tempest fs20 back as criteria https://review.rdoproject.org/r/18053 | 15:08 |
| ykarel | tosky, cook. Thanks for the info | 15:08 |
| ykarel | cook/cool | 15:08 |
| number80 | mjturek: nope, it was cancelled :) | 15:08 |
| ykarel | tosky, btw i revised sahara patch | 15:08 |
| mjturek | ahhh thanks number80 :) | 15:09 |
| tosky | ykarel: oh, I didn't notice that the gates passed | 15:09 |
| ykarel | yes | 15:10 |
| *** jrist has joined #rdo | 15:17 | |
| *** quiquell is now known as quiquell|off | 15:17 | |
| *** rfolco is now known as rfolco|lunch | 15:18 | |
| *** jrist has quit IRC | 15:22 | |
| *** jrist has joined #rdo | 15:26 | |
| rdogerrit | Merged rdo-jobs master: Add extra_tags into job vars for overcloud upgrade/update https://review.rdoproject.org/r/17962 | 15:28 |
| *** jfrancoa has quit IRC | 15:39 | |
| *** dprince has quit IRC | 15:47 | |
| *** Son_Goku has quit IRC | 15:48 | |
| *** ykarel is now known as ykarel|awat | 15:48 | |
| *** ykarel|awat is now known as ykarel|away | 15:48 | |
| rdogerrit | rdo-trunk created openstack/os-faults-distgit rocky-rdo: python-os-faults: failed to build ff78b92 https://review.rdoproject.org/r/18054 | 15:49 |
| rdogerrit | rdo-trunk created openstack/os-faults-distgit rpm-master: python-os-faults: failed to build ff78b92 https://review.rdoproject.org/r/18055 | 15:49 |
| *** dprince has joined #rdo | 15:50 | |
| rdogerrit | rdo-trunk created openstack/os-faults-distgit rpm-master: python-os-faults: failed to build ff78b92 https://review.rdoproject.org/r/18056 | 15:52 |
| *** udesale has quit IRC | 15:53 | |
| *** rfolco|lunch is now known as rfolco | 15:55 | |
| *** ykarel|away has quit IRC | 15:55 | |
| *** Son_Goku has joined #rdo | 16:03 | |
| *** kopecmartin is now known as kopecmartin|off | 16:04 | |
| *** aufi has quit IRC | 16:23 | |
| *** dprince has quit IRC | 16:26 | |
| *** ykarel|away has joined #rdo | 16:29 | |
| *** ttsiouts has joined #rdo | 16:36 | |
| rdogerrit | rdo-trunk created openstack/os-faults-distgit rpm-master: python-os-faults: failed to build ff78b92 https://review.rdoproject.org/r/18057 | 16:41 |
| *** jchhatbar has quit IRC | 16:43 | |
| rdogerrit | Merged rdo-infra/ci-config master: add full tempest fs20 back as criteria https://review.rdoproject.org/r/18053 | 16:54 |
| *** brault has joined #rdo | 16:58 | |
| rdogerrit | hguemar proposed openstack/os-faults-distgit rpm-master: python-os-faults: remove fuel network management driver https://review.rdoproject.org/r/17503 | 17:00 |
| rlandy|rover | dmsimard: do you know if it's possible to exclude certain providers for upstream check jobs? | 17:03 |
| dmsimard | rlandy|rover: providers ? | 17:03 |
| rlandy|rover | dmsimard: investigating at a failure in multinode jobs that looks only to occur with certain hosts | 17:04 |
| dmsimard | rlandy|rover: that's possible -- do you have a bug "signature" ? like, is there a specific error that is occurring ? | 17:04 |
| rlandy|rover | dmsimard: https://bugs.launchpad.net/tripleo/+bug/1810054 | 17:05 |
| openstack | Launchpad bug 1810054 in tripleo "mulitnode jobs failing on gathering facts from subnode-2" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy) | 17:05 |
| rlandy|rover | localhost | Hostname: centos-7-limestone-regionone-* | 17:05 |
| rlandy|rover | the failure doesn't occur with rax, etc. | 17:05 |
| dmsimard | well, I mean -- it's possible that there is an issue with a specific provider but it's not standard to avoid a specific provider | 17:05 |
| dmsimard | ok let me see | 17:05 |
| rlandy|rover | dmsimard: understandable - I am not sure whom I would contact for that provider and in the mean time the failure is resetting the gate | 17:06 |
| dmsimard | rlandy|rover: if you have a feeling there's something going on with a specific provider, it's typically a good idea to report it in #openstack-infra (with data to back it up) | 17:07 |
| dmsimard | I'm looking at it right now | 17:07 |
| *** Vorrtex has joined #rdo | 17:07 | |
| rlandy|rover | dmsimard: ack - we can switch the discussion to #openstack-infra ... let me know | 17:08 |
| dmsimard | rlandy|rover: yeah, I'll switch | 17:08 |
| rlandy|rover | k | 17:08 |
| *** Vorrtex has quit IRC | 17:12 | |
| *** rpittau has quit IRC | 17:15 | |
| rdogerrit | Giulio Fidente proposed rdoinfo master: deps: bump Ansible to 2.6.11 for Queens, Stein and Rocky https://review.rdoproject.org/r/14960 | 17:17 |
| *** dprince has joined #rdo | 17:18 | |
| dmsimard | rlandy|rover: if you look at some of the jobs, they have ran successfully on limestone before so it doesn't feel like a 100% failure rate | 17:21 |
| dmsimard | example query: build_name:"tripleo-ci-centos-7-containers-multinode" AND node_provider:"limestone-regionone" AND build_status:"SUCCESS" | 17:21 |
| *** jlibosva has quit IRC | 17:22 | |
| rlandy|rover | dmsimard: yes - sporadic - but any time there is a failure of this kind, it's limestone | 17:22 |
| dmsimard | rlandy|rover: agreed, logstash tells us that much | 17:23 |
| dmsimard | I'm trying to understand what is going on | 17:23 |
| rlandy|rover | dmsimard: thanks | 17:23 |
| *** tosky has quit IRC | 17:23 | |
| *** Vorrtex has joined #rdo | 17:24 | |
| dmsimard | rlandy|rover: the error is clearly occurring at the same place every time | 17:25 |
| dmsimard | After "tripleo-inventory" has run | 17:25 |
| dmsimard | Could there be an error in the inventory generation ? | 17:25 |
| rlandy|rover | and it gets through most of the test | 17:25 |
| rlandy|rover | so if inventory was a problem, we should hit that everywhere | 17:26 |
| dmsimard | clarkb was mentioning in #openstack-infra that limestone apparently runs ipv6 -- the ssh connection failure occurs on an ipv4 (not sure where that specific IP is coming from) | 17:26 |
| *** jpich has quit IRC | 17:26 | |
| rlandy|rover | yeah - I saw that - was thinking about if that could cause an issue | 17:26 |
| rlandy|rover | but the sporadic nature is weird | 17:27 |
| rlandy|rover | if ipv6 was a problem, we would see it every time we hit limestone | 17:27 |
| dmsimard | he also just mentioned that there was nested virtualization issues reported at limestone | 17:27 |
| rlandy|rover | ^^ that makes more sense | 17:27 |
| dmsimard | rlandy|rover: ara from zuul's perspective is not very useful because it ends up running a 3h long shell task when it runs to completion | 17:30 |
| dmsimard | ok, I found the nested one with the relevant info | 17:31 |
| dmsimard | sshnaidm, ssbarnea: is there any effort to get rid of ara html generation ? it's really not optimal | 17:31 |
| dmsimard | i.e, http://logs.openstack.org/90/623390/1/gate/tripleo-ci-centos-7-containers-multinode/81a6130/logs/ara_oooq/ | 17:31 |
| sshnaidm | dmsimard, what is the alternative? | 17:31 |
| rlandy|rover | looks like we start seeing this on 12/23 | 17:32 |
| dmsimard | sshnaidm: saving the ansible.sqlite database in an ara-report directory | 17:32 |
| sshnaidm | dmsimard, and then? | 17:32 |
| dmsimard | sshnaidm: openstack-ansible does it so they have ara-report at the root (which is ara from Zuul's ansible perspective) and then logs/ara-report/ for their perspective | 17:32 |
| dmsimard | sshnaidm: that's it | 17:32 |
| rlandy|rover | so something hit then | 17:32 |
| dmsimard | sshnaidm: instead of running "ara generate html" you run something like "cp ~/.ara/ansible.sqlite $WORKSPACE/logs/ara-report/ansible.sqlite" | 17:33 |
| sshnaidm | dmsimard, and it'll be generated automatically? | 17:33 |
| dmsimard | sshnaidm: it's not generated, it's loaded on the fly | 17:33 |
| dmsimard | sshnaidm: explained here: http://superuser.openstack.org/articles/scaling-ara-ansible/ | 17:33 |
| sshnaidm | dmsimard, sorry, can you please elaborate little bit more? | 17:33 |
| dmsimard | sshnaidm: there is a wsgi middleware which maps the URL to a database location on disk | 17:34 |
| *** imacdonn has joined #rdo | 17:34 | |
| dmsimard | hang on | 17:34 |
| dmsimard | oh no, github unicorn | 17:34 |
| rlandy|rover | yep - github unicorn here as well :( | 17:35 |
| rlandy|rover | trying to check changes from 12/23 | 17:35 |
| *** mvkr has quit IRC | 17:35 | |
| imacdonn | it's got a pretty rainbow mane, though | 17:36 |
| rlandy|rover | show how dependent we are on github | 17:36 |
| sshnaidm | https://www.githubstatus.com/ | 17:37 |
| *** mcornea has quit IRC | 17:42 | |
| dmsimard | rlandy|rover: to be fair I could have looked up what I was looking for on git.openstack.org, I just prefer the github ui :p | 17:45 |
| rdogerrit | Merged openstack/os-faults-distgit rpm-master: python-os-faults: remove fuel network management driver https://review.rdoproject.org/r/17503 | 17:45 |
| dmsimard | openstack-infra is working on standing up a gitea instance, though, that'll be much better | 17:46 |
| dmsimard | sshnaidm: so, what I wanted to show you was https://github.com/openstack-infra/puppet-openstackci/blob/master/templates/logs.vhost.erb#L88-L112 | 17:46 |
| ssbarnea | i like github UI code browser much more than ours, much easier to browse and share | 17:47 |
| dmsimard | ssbarnea: ours = ? | 17:47 |
| rlandy|rover | dmsimard: yep - was just checking possible changed to inventory - nothing for a while | 17:47 |
| rlandy|rover | and we run the tripleo-inventory command multiple times per job | 17:48 |
| ssbarnea | dmsimard: now there is a serious problem with the "middleware" approach: it prevents us from moving to a pure distributed storage (swift) | 17:49 |
| dmsimard | ssbarnea: yes, I'm aware -- the swift approach does rely on html generation | 17:49 |
| ssbarnea | ideally sqlite could be loaded without a middleware, as js has support working with sqlite | 17:49 |
| ssbarnea | still, this means all code in js and no python middleware.... | 17:49 |
| dmsimard | ssbarnea: in 1.0 things are decoupled properly and the web interface has it's own API client implementation (a bit like Zuul's web interface) | 17:50 |
| *** ttsiouts has quit IRC | 17:50 | |
| dmsimard | ssbarnea: html generation doesn't scale well at all -- putting it in swift is fine until you need to pay for it | 17:51 |
| ssbarnea | infra is working to move to swift, this will happen sooner than you may thingk | 17:51 |
| ssbarnea | i am not trying to defent the static html implementation, at all. | 17:51 |
| rdogerrit | hguemar proposed openstack/os-faults-distgit rpm-master: python-os-faults: failed to build ff78b92 https://review.rdoproject.org/r/18057 | 17:52 |
| sshnaidm | dmsimard, ok | 17:52 |
| sshnaidm | dmsimard, I'll make a patch, let's test it | 17:52 |
| dmsimard | ssbarnea: you know I'm root in infra right? :p | 17:52 |
| sshnaidm | dmsimard, do you have this config generated to show me? | 17:52 |
| dmsimard | ssbarnea: I'm well aware of the challenges and I haven't found the right solution yet in the context of 1.0 | 17:52 |
| dmsimard | sshnaidm: the only different thing from your perspective would be to remove "ara generate html /some/path" and replace it with "cp $database /some/path/ara-report/" | 17:53 |
| ssbarnea | if we store 2 files for each build in logs I will be really happy: one html file and one sqlite file. the rest should be "outsourced". | 17:53 |
| sshnaidm | dmsimard, I see, but I need this config for something different | 17:54 |
| ssbarnea | this lowers the load on storage and the minimizes the collection time. | 17:54 |
| dmsimard | ssbarnea: let's move to #ara :P | 17:55 |
| ssbarnea | indeed | 17:55 |
| *** mjturek has quit IRC | 17:57 | |
| *** mcornea has joined #rdo | 17:58 | |
| *** imacdonn has quit IRC | 17:59 | |
| *** imacdonn has joined #rdo | 17:59 | |
| sshnaidm | dmsimard, where should be "ara_report" path? | 18:00 |
| *** ykarel|away has quit IRC | 18:01 | |
| dmsimard | sshnaidm: anywhere you want | 18:01 |
| sshnaidm | dmsimard, and if I have 3 different sql files? | 18:02 |
| *** gfidente has quit IRC | 18:02 | |
| dmsimard | sshnaidm: why 3 different sql files ? | 18:02 |
| sshnaidm | dmsimard, because we track 3 different ansible runs/layers | 18:02 |
| sshnaidm | dmsimard, ansible in oooq, ansible in undercloud and ansible in overcloud.. | 18:03 |
| sshnaidm | dmsimard, should I use like "/logs/ara-ooq/ara-report/", "/logs/ara-undercloud/ara-report", etc ? | 18:03 |
| dmsimard | sshnaidm: the location of the databases and ara-report directories are not important -- so long as the directory is named "ara-report" which is sent to the application by the apache rewrite rules | 18:03 |
| dmsimard | sshnaidm: yeah, exactly | 18:03 |
| sshnaidm | dmsimard, ack | 18:04 |
| *** sshnaidm is now known as sshnaidm|afk | 18:05 | |
| *** trown is now known as trown|lunch | 18:10 | |
| *** panda is now known as panda|off | 18:15 | |
| dmsimard | rlandy|rover: I might have something. Still looking. | 18:20 |
| *** kgiusti has joined #rdo | 18:20 | |
| rlandy|rover | dmsimard: awesome | 18:21 |
| *** wolverineav has joined #rdo | 18:22 | |
| rdogerrit | hguemar proposed openstack/os-faults-distgit rpm-master: python-os-faults: failed to build ff78b92 https://review.rdoproject.org/r/18055 | 18:22 |
| *** wolverineav has quit IRC | 18:26 | |
| *** wolverineav has joined #rdo | 18:38 | |
| dmsimard | rlandy|rover: does tripleo attempt to use nested virtualization if it can ? | 18:39 |
| rlandy|rover | dmsimard: tripleo or tripleo jobs? if the question is on tripleo itself, I'd check with mwhahaha | 18:41 |
| *** wolverineav has quit IRC | 18:42 | |
| mwhahaha | dmsimard: it used to | 18:42 |
| mwhahaha | depends on the job i guess | 18:42 |
| mwhahaha | i know we hard coded it somewhere | 18:42 |
| * mwhahaha digs up review | 18:42 | |
| mwhahaha | https://review.openstack.org/#/c/623293/ | 18:42 |
| dmsimard | rlandy|rover: I mentioned it in #openstack-infra -- it could be a red herring but the failures from limestone appear to occur only on a specific processor model | 18:43 |
| mwhahaha | so for standalone we fixed it to be just qemu | 18:43 |
| dmsimard | rlandy|rover: I've seen two compute node models -- one that exposes the CPU as "Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)" and the other "Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz" | 18:43 |
| mwhahaha | not sure if muiltinode jobs were also adressed | 18:43 |
| dmsimard | I haven't been able to find a failure that used "Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz" -- all I've looked at were using the E3-13xx | 18:44 |
| dmsimard | 12xx* | 18:44 |
| mwhahaha | which bug are we talking about? | 18:44 |
| mwhahaha | the facts one? | 18:44 |
| *** wolverineav has joined #rdo | 18:45 | |
| dmsimard | mwhahaha: yeah https://bugs.launchpad.net/tripleo/+bug/1810054 | 18:46 |
| openstack | Launchpad bug 1810054 in tripleo "mulitnode jobs failing on gathering facts from subnode-2" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy) | 18:46 |
| rlandy|rover | I think we default to libvirt_type: qemu | 18:46 |
| mwhahaha | so multinode does try to | 18:46 |
| *** wolverineav has quit IRC | 18:46 | |
| mwhahaha | it uses se4t-libvirt-type | 18:46 |
| * mwhahaha digs up code | 18:46 | |
| mwhahaha | well multinode-undercloud does | 18:46 |
| mwhahaha | http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/playbooks/multinode-undercloud.yml#n91 | 18:46 |
| mwhahaha | not sure where that is specifically used | 18:47 |
| *** holser_ has quit IRC | 18:47 | |
| *** wolverineav has joined #rdo | 18:48 | |
| mwhahaha | though the failure seems to be befmore the virttype stuff | 18:50 |
| mwhahaha | ovs problems maybe? | 18:50 |
| dmsimard | sort of stabbing in the dark right now | 18:51 |
| mwhahaha | so 10.4.70.64 is the private ipv4 | 18:51 |
| mwhahaha | the boxes have public ipv6 | 18:52 |
| mwhahaha | http://logs.openstack.org/92/625692/2/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/b38e0b1/zuul-info/inventory.yaml | 18:52 |
| dmsimard | tl;dr -- jobs are /sometimes/ failing on limestone, they are not failing 100% of the time | 18:52 |
| mwhahaha | maybe firewall rules or something? | 18:52 |
| mwhahaha | are there any successful limestone multinode jobs? | 18:52 |
| dmsimard | I believe so | 18:52 |
| dmsimard | hang on | 18:52 |
| dmsimard | mwhahaha: there's hits for this in logstash: build_name:"tripleo-ci-centos-7-containers-multinode" AND node_provider:"limestone-regionone" AND build_status:"SUCCESS" | 18:53 |
| *** wolverineav has quit IRC | 18:53 | |
| mwhahaha | i know i first saw this over the weekend when i started approving some stuff | 18:55 |
| mwhahaha | i wonder when it started to fail | 18:55 |
| *** Caterpillar has quit IRC | 18:55 | |
| rlandy|rover | dmsimard: so probably one compute model is using kvm and one qemu | 18:55 |
| rlandy|rover | looks like we have our first failure 12/23 | 18:55 |
| mwhahaha | i don't see anything earlier than 12/29 | 18:55 |
| mwhahaha | wait i was only checking the gate | 18:56 |
| * mwhahaha adjusts | 18:56 | |
| mwhahaha | 12/25 | 18:56 |
| dmsimard | for the same job | 18:57 |
| rlandy|rover | http://logs.openstack.org/48/602248/19/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/6a14c18/job-output.txt#_2018-12-24_00_07_11_910528 | 18:57 |
| dmsimard | Failures: https://screenshotscdn.firefoxusercontent.com/images/42d00ec2-249d-464d-97a4-eb862274a3d8.png | 18:57 |
| rlandy|rover | 12/24 | 18:57 |
| dmsimard | Success: https://screenshotscdn.firefoxusercontent.com/images/df296dd8-30c5-4215-bf46-75503447e11e.png | 18:57 |
| *** wolverineav has joined #rdo | 18:59 | |
| mwhahaha | only think quickstart related that we merged seemed to be around podman/docker cli which this isn't even getting to that code yet | 18:59 |
| mwhahaha | dmsimard: wait, what's the mtu supposed to be in limestone? | 19:00 |
| mwhahaha | is it 1500? | 19:00 |
| dmsimard | I can look at what's the default | 19:00 |
| dmsimard | looks like eth0 is set to 1500 | 19:03 |
| dmsimard | br-ex 1350 | 19:03 |
| dmsimard | there's a job going on this node so hard to tell if it's vanilla | 19:04 |
| mwhahaha | we can't pull logs from the subnode :/ | 19:05 |
| mwhahaha | http://logs.openstack.org/92/625692/2/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/b38e0b1/logs/undercloud/var/log/extra/network.txt.gz seems to show that the iptables rules should be correct | 19:05 |
| mwhahaha | at least from the main node | 19:05 |
| mwhahaha | cause it has both itself and .64 added | 19:06 |
| mwhahaha | my thought might be that's it's an ovs thing | 19:06 |
| mwhahaha | because we've configured that correctly prior to this failing | 19:06 |
| mwhahaha | http://logs.openstack.org/92/625692/2/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/b38e0b1/job-output.txt.gz#_2018-12-29_20_23_53_325399 | 19:06 |
| mwhahaha | but we don't really dump that info :/ | 19:07 |
| *** dtantsur is now known as dtantsur|afk | 19:09 | |
| *** tigalch has left #rdo | 19:10 | |
| dmsimard | I'd really like to set up a autohold to better understand what is going on but it's hard to reproduce | 19:11 |
| dmsimard | so it'd end up catching totally unreleated failures | 19:11 |
| mwhahaha | rlandy|rover: why aren't the subnode-2 logs attempted to be fetched? | 19:11 |
| rlandy|rover | gathering facts | 19:12 |
| dmsimard | trying to figure out if it's possible to autohold for a specific provider | 19:12 |
| mwhahaha | oh so the same thing | 19:12 |
| rlandy|rover | we do that on the start of many roles | 19:12 |
| mwhahaha | so basically because node 1 can't ssh to node 2 | 19:12 |
| rlandy|rover | correct | 19:12 |
| rlandy|rover | there is no connection to node 2 | 19:12 |
| rlandy|rover | but only sometimes | 19:13 |
| mwhahaha | i wonder if there's some improved ovs thing we can try and grab during log collection that'll show something | 19:13 |
| rlandy|rover | should be in ara | 19:14 |
| rlandy|rover | checking | 19:14 |
| rlandy|rover | 2018-12-29 20:23:45.681658 | TASK [multi-node-bridge : Install openvswitch] | 19:14 |
| mwhahaha | so it's installed and configured | 19:15 |
| rlandy|rover | done as part of pre's | 19:15 |
| mwhahaha | right | 19:15 |
| mwhahaha | i mean printing the switch config or something | 19:15 |
| mwhahaha | ie ovs-ofctl show switch | 19:15 |
| rlandy|rover | 2018-12-29 20:23:53.325399 | TASK [multi-node-bridge : Ensure openvswitch is started] | 19:15 |
| rlandy|rover | 2018-12-29 20:23:54.303443 | primary | changed | 19:15 |
| rlandy|rover | 2018-12-29 20:23:54.357473 | secondary | changed | 19:15 |
| mwhahaha | or dump-tables | 19:15 |
| rlandy|rover | http://logs.openstack.org/92/625692/2/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/b38e0b1/zuul-info/zuul-info.secondary.txt | 19:16 |
| rlandy|rover | all unrechable | 19:16 |
| mwhahaha | that's the ipv6 stuff, not sure if related | 19:17 |
| mwhahaha | http://logs.openstack.org/92/625692/2/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/b38e0b1/zuul-info/zuul-info.primary.txt | 19:17 |
| mwhahaha | the problem is that' | 19:18 |
| mwhahaha | the problem is that's before we start doing stuff | 19:18 |
| mwhahaha | http://logs.openstack.org/92/625692/2/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/b38e0b1/logs/undercloud/var/log/extra/network.txt.gz | 19:18 |
| mwhahaha | shows that we've reconfigured with br-ex and such | 19:18 |
| rlandy|rover | http://logs.openstack.org/92/625692/2/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/b38e0b1/ara-report/ ... look at third pre | 19:20 |
| mwhahaha | yea that's why i was wondering if we could print like the peer status out of ovs in the log collection | 19:22 |
| mwhahaha | though seems odd that this would be provider specific | 19:22 |
| rlandy|rover | we could - that code is directly upstream zuul stuff | 19:23 |
| rlandy|rover | interesting that this started just recently | 19:24 |
| rlandy|rover | nothing new related afaik | 19:24 |
| rlandy|rover | we have had that bridge code in there for at least three months | 19:25 |
| *** trown|lunch is now known as trown | 19:25 | |
| mwhahaha | longer than that | 19:28 |
| rlandy|rover | shows no errors in bringing the bridge up/configuring | 19:29 |
| rlandy|rover | how long has limestone been around? | 19:31 |
| mwhahaha | forever | 19:32 |
| mwhahaha | so the thing is that zuul still has access to the secondary node after the ovs stuff is configrued | 19:32 |
| *** artom has quit IRC | 19:33 | |
| mwhahaha | wonder if the nodepool id_rsa is not there anymore? | 19:35 |
| mwhahaha | http://logs.openstack.org/92/625692/2/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/b38e0b1/logs/quickstart_files/hosts.txt.gz | 19:35 |
| mwhahaha | or the perms are bad maybe? | 19:37 |
| rlandy|rover | only on some nodes of limestone? | 19:37 |
| mwhahaha | not sure, unfortunately that error can be any auth related failure including connectivity i think | 19:37 |
| mwhahaha | only way to know would be to -vvvv the execution | 19:37 |
| rdogerrit | hguemar proposed openstack/os-faults-distgit rpm-master: python-os-faults: failed to build ff78b92 https://review.rdoproject.org/r/18056 | 19:38 |
| *** dneary has joined #rdo | 19:42 | |
| *** mcornea has quit IRC | 19:56 | |
| *** wolverineav has quit IRC | 19:56 | |
| *** wolverineav has joined #rdo | 19:57 | |
| *** Vorrtex has quit IRC | 19:58 | |
| *** mcornea has joined #rdo | 19:59 | |
| *** shardy has quit IRC | 20:00 | |
| *** kgiusti has left #rdo | 20:01 | |
| *** wolverineav has quit IRC | 20:01 | |
| *** kgiusti has joined #rdo | 20:01 | |
| *** radeks__ has joined #rdo | 20:02 | |
| *** radeks_ has quit IRC | 20:05 | |
| rdogerrit | Merged rdoinfo master: Rebuild python-rfc3986-1.2.0-1 from Fedora for stein https://review.rdoproject.org/r/18044 | 20:05 |
| *** Vorrtex has joined #rdo | 20:06 | |
| rlandy|rover | weshay|ruck: ^^ woohoo - to fix tempest failure - phase 1 | 20:13 |
| weshay|ruck | cool | 20:14 |
| dmsimard | rlandy|rover, weshay|ruck, mwhahaha: added a comment in https://bugs.launchpad.net/tripleo/+bug/1810054/comments/8 | 20:24 |
| openstack | Launchpad bug 1810054 in tripleo "mulitnode jobs failing on gathering facts from subnode-2" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy) | 20:24 |
| rlandy|rover | thanks - reading through | 20:24 |
| *** panda|off has quit IRC | 20:30 | |
| *** panda has joined #rdo | 20:33 | |
| *** mjturek has joined #rdo | 20:34 | |
| *** wolverineav has joined #rdo | 20:35 | |
| *** wolverineav has quit IRC | 20:40 | |
| dmsimard | rlandy|rover: I'm not going to spend more time on that for now, the elastic-recheck query should tell us if it reproduces despite the host-passthrough fix. If it doesn't reproduce, it means it was an issue with nested virtualization. | 20:41 |
| rlandy|rover | dmsimard: ack - I don't think there is much more to do atm | 20:42 |
| rlandy|rover | we've all spent a lot of time here today | 20:43 |
| rlandy|rover | will watch the results | 20:43 |
| dmsimard | intermittent and seemingly random failures are time consuming indeed :p | 20:43 |
| rlandy|rover | thanks for your help | 20:43 |
| *** mjturek has quit IRC | 20:45 | |
| *** jcoufal has quit IRC | 20:48 | |
| *** slaweq_ is now known as slaweq | 20:49 | |
| *** slaweq has quit IRC | 20:55 | |
| *** radeks__ has quit IRC | 21:00 | |
| *** wolverineav has joined #rdo | 21:07 | |
| *** jmlowe has quit IRC | 21:09 | |
| *** jmlowe has joined #rdo | 21:12 | |
| *** slaweq has joined #rdo | 21:12 | |
| *** slaweq has quit IRC | 21:17 | |
| *** mcornea has quit IRC | 21:17 | |
| *** mcornea has joined #rdo | 21:26 | |
| *** dprince has quit IRC | 21:26 | |
| *** kgiusti has left #rdo | 21:30 | |
| *** mjturek has joined #rdo | 21:39 | |
| *** mjturek has quit IRC | 21:42 | |
| *** Vorrtex has quit IRC | 21:43 | |
| *** mjturek has joined #rdo | 21:43 | |
| *** mjturek has quit IRC | 21:44 | |
| *** mjturek has joined #rdo | 21:46 | |
| *** factor has quit IRC | 21:49 | |
| *** mcornea has quit IRC | 21:53 | |
| *** trown is now known as trown|outtypewww | 22:00 | |
| *** ttsiouts has joined #rdo | 22:09 | |
| *** holser_ has joined #rdo | 22:10 | |
| *** cyberworm54 has quit IRC | 22:20 | |
| *** mvkr has joined #rdo | 22:23 | |
| *** wolverineav has quit IRC | 22:24 | |
| *** wolverineav has joined #rdo | 22:28 | |
| *** factor has joined #rdo | 22:30 | |
| *** tosky has joined #rdo | 22:30 | |
| *** jtomasek has quit IRC | 22:36 | |
| *** holser_ has quit IRC | 22:39 | |
| *** rcernin has joined #rdo | 22:49 | |
| *** wolverineav has quit IRC | 23:03 | |
| *** wolverineav has joined #rdo | 23:07 | |
| *** rascasoft has quit IRC | 23:10 | |
| *** holser_ has joined #rdo | 23:16 | |
| *** rlandy|rover is now known as rlandy|rover|bbl | 23:23 | |
| *** wolverineav has quit IRC | 23:35 | |
| *** wolverineav has joined #rdo | 23:40 | |
| *** wolverineav has joined #rdo | 23:40 | |
| *** Jeffrey4l has quit IRC | 23:41 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
