| opendevreview | Ian Wienand proposed openstack/project-config master: wheel: don't try to release Xenial ARM64 https://review.opendev.org/c/openstack/project-config/+/798236 | 01:58 |
|---|---|---|
| *** jpena|off is now known as jpena | 06:57 | |
| *** ykarel|away is now known as ykarel | 07:46 | |
| *** ykarel is now known as ykarel|lunch | 09:38 | |
| *** bhagyashris_ is now known as bhagyashris | 09:40 | |
| *** ykarel|lunch is now known as ykarel | 10:43 | |
| *** jpena is now known as jpena|lunch | 11:32 | |
| *** jpena|lunch is now known as jpena | 12:29 | |
| opendevreview | gnuoy proposed openstack/project-config master: Add Ceph Dashboard charm to OpenStack charms https://review.opendev.org/c/openstack/project-config/+/797911 | 12:32 |
| opendevreview | gnuoy proposed openstack/project-config master: Add Ceph Dashboard charm to OpenStack charms https://review.opendev.org/c/openstack/project-config/+/797911 | 13:02 |
| *** abhishekk is now known as akekane|home | 13:39 | |
| *** akekane|home is now known as abhishekk | 13:39 | |
| opendevreview | Merged openstack/project-config master: wheel: don't try to release Xenial ARM64 https://review.opendev.org/c/openstack/project-config/+/798236 | 13:42 |
| sean-k-mooney | o/ quick question we noticed https://meetings.opendev.org/irclogs/%23openstack-nova/latest.log.html is lagging more then normal | 13:45 |
| sean-k-mooney | anychance the sript/bot that updates that is stuck or need to be restarted | 13:45 |
| sean-k-mooney | its nomally only a minute or two behind at most but looks like its about 30 mins behind currenlty | 13:46 |
| clarkb | sean-k-mooney: it has to do with fs syncing. | 13:47 |
| clarkb | fungi: ^ do you know if your change to tweak settings around that landed and if so was the bot restarted with that config? | 13:47 |
| clarkb | sean-k-mooney: basically if we restart the bot it flushes everything to disk and it catches up, but were trying to determine if we can force it to be more aggressive under normal operation | 13:48 |
| sean-k-mooney | ah ok | 13:49 |
| sean-k-mooney | normally its not an issue bauzas was just going to link to it in a gerrit comment? | 13:49 |
| clarkb | it could also be a bug in limnoria. We have fixed at least one upstream so far | 13:49 |
| clarkb | but ya it has to do with fs syncs | 13:50 |
| sean-k-mooney | its not blocking anything critical in anycase just wanted to make sure it was not stuck without anyone knowing | 13:51 |
| clarkb | yup no worries. I'll have to defer to fungi on current status though as I was out most of last week | 13:51 |
| clarkb | there was at least one config change aimed at making the bot flush to disk more often | 13:51 |
| sean-k-mooney | is this the bot that was always used https://github.com/ProgVal/Limnoria | 13:52 |
| sean-k-mooney | or is limnoria new since we moved | 13:52 |
| clarkb | sean-k-mooney: no, we had been using supybot (limnoria is afork of supybot) but upgraded to get to python3 (though it was a long planned move the freenode stuff just pushed it along) | 13:52 |
| clarkb | basically we did a major upgrade of hte software and switched from python2 to python3 in the process | 13:53 |
| clarkb | the upstream is very response though. I already fixed one bug that they merged so far | 13:53 |
| sean-k-mooney | ya if you were going to do that at any point doing it then made sense | 13:53 |
| bauzas | fwiw, -nova log is now lagging for more than 50 mins | 13:56 |
| bauzas | last update is 1315 UTC | 13:57 |
| clarkb | yes we were just discussing it above | 13:57 |
| clarkb | it is a known problem with syncing to disk that we have been trying to run down in limnoria | 13:57 |
| clarkb | if we restart the bot it flushes and is happy. I wonder if maybe a manual sync on the system would do it? | 13:57 |
| bauzas | oh my bad, I thought you were mentioned it should back up quick | 13:58 |
| bauzas | mentioning* | 13:58 |
| clarkb | and by happy I mean things get flushed to disk and then we start the lag over again | 13:58 |
| clarkb | if out of band snycing works we might get away with a short cron to run sync for now | 13:58 |
| bauzas | I see | 13:59 |
| clarkb | oh neat I think the text logs may actually be up to date | 14:00 |
| clarkb | fungi: ^ in previous debugging did you notice if it was just html having problems? iirc the html is produced by a cronjob | 14:00 |
| bauzas | clarkb: you're right | 14:01 |
| bauzas | good catch | 14:01 |
| fungi | clarkb: pretty sure my changes merged, checking... but yes the bug i was addressing was a delay flushing the text logs, the html logs are created by a cronjob from them | 14:02 |
| clarkb | fungi: ok I think the text logs are flushing properly or at least on a much smaller lag for nova for now | 14:02 |
| clarkb | though it could be that since we only run the cronjob every 15 minutes that we had text lagging by 35 minutes then missed the cron job and have to wait for the next rotation | 14:03 |
| fungi | right, https://review.opendev.org/796513 merged | 14:03 |
| clarkb | though we don't have updated html after the 14:00 cron job (not sure if it has completed yet though) | 14:03 |
| fungi | i also don't know for sure that 796513 necessarily fixed it, that was just what we were trying | 14:04 |
| fungi | it seemed like a probable candidate for the problem, at least | 14:05 |
| sean-k-mooney | i assume the html conversion is just to make the lines linkable | 14:07 |
| clarkb | sean-k-mooney: yup | 14:08 |
| clarkb | that and the colors | 14:08 |
| sean-k-mooney | ack | 14:08 |
| sean-k-mooney | i was wondering could that be done client side with java script | 14:08 |
| sean-k-mooney | but not really imporant | 14:09 |
| clarkb | no, it is done in place on the system. Another thought is maybe we're serving data with cache set long enough we don't get more recent updates? | 14:09 |
| clarkb | I notice the text files don't open in our browser | 14:09 |
| clarkb | they did previously which means we may have lost header configuration compared to the old server | 14:09 |
| sean-k-mooney | no change with the debug console open | 14:09 |
| sean-k-mooney | that sens a do not cache hearder right | 14:10 |
| fungi | trying to figure out where the output from that cronjob gets logged | 14:10 |
| fungi | grepping CRON from /var/log/syslog indicates the last run was nearly an hour ago: | 14:11 |
| fungi | Jun 28 13:15:01 eavesdrop01 CRON[1911956]: (root) CMD (/usr/local/bin/docker-compose -f /etc/ircbot-docker/docker-compose.yaml exec -T ircbot bash -c "find /var/lib/limnoria/opendev/logs/ChannelLogger/oftc/ -mindepth 1 -maxdepth 1 -type d | xargs -n1 logs2html") | 14:11 |
| fungi | oh! | 14:12 |
| fungi | hah, cron timespec fail | 14:12 |
| fungi | 15 * * * * /usr/local/bin/docker-compose -f /etc/ircbot-docker/docker-compose.yaml exec -T ircbot bash -c "find /var/lib/limnoria/opendev/logs/ChannelLogger/oftc/ -mindepth 1 -maxdepth 1 -type d | xargs -n1 logs2html" | 14:12 |
| fungi | it's configured to run on the 15th minute of every hour :/ | 14:12 |
| clarkb | that would explain it | 14:12 |
| fungi | i'll work on a patch for that in a few, but we should see it update in ~2 minutes | 14:13 |
| clarkb | also that will fire in 2 minutes, we can double check that is the problem shortly | 14:13 |
| sean-k-mooney | oh lol ya | 14:13 |
| sean-k-mooney | cron has a / sysntax right e.g. 15/* or */15 | 14:14 |
| sean-k-mooney | to get it every 15 minutes | 14:14 |
| clarkb | yes | 14:14 |
| sean-k-mooney | i always have to look that up | 14:14 |
| fungi | right, that's what i plan to set, but yes i never can remember if it's integer division or modulus | 14:15 |
| sean-k-mooney | this is what i have used the last few times https://crontab.guru/ | 14:15 |
| *** ysandeep is now known as ysandeep|afk | 14:15 | |
| clarkb | the html log did update fwiw | 14:26 |
| clarkb | I think it is 15/* | 14:27 |
| clarkb | if I am reading man 5 crontab properly | 14:27 |
| fungi | */15 | 14:28 |
| fungi | 0 */4 1 * mon echo "run every 4th hour on the 1st and on every Monday" | 14:28 |
| fungi | according to `man 5 crontab` | 14:29 |
| clarkb | aha I misread then :) | 14:29 |
| clarkb | its eraly | 14:29 |
| fungi | Steps are also permitted after an asterisk, so if you want to say ``every two hours'', just use ``*/2''. | 14:30 |
| fungi | sean-k-mooney: bauzas: clarkb: ianw: ^ | 14:33 |
| fungi | er, https://review.opendev.org/798311 | 14:34 |
| fungi | forgot gerritbot doesn't echo system-config changes in this channel | 14:34 |
| sean-k-mooney | yes its */15 | 14:34 |
| clarkb | +2'd you can probably go ahead and approve it if you want | 14:34 |
| sean-k-mooney | i think if the step deviding into the * | 14:34 |
| sean-k-mooney | although its really a modules checking if the remainder is 0 | 14:35 |
| fungi | right, what i meant by modulus | 14:36 |
| sean-k-mooney | this is why i like https://crontab.guru/#*/15_*_*_*_* | 14:37 |
| sean-k-mooney | systemd have also adopted the same format for there timeres and they have a utility for checkign the time format too | 14:37 |
| sean-k-mooney | ah its systemd-analyze calendar | 14:40 |
| sean-k-mooney | but that a really recent addtion | 14:40 |
| sean-k-mooney | https://opensource.com/article/20/7/systemd-timers#testing-calendar-specifications | 14:41 |
| sean-k-mooney | fungi: lol why does gerrit say your Status: missing, presumed fed | 14:44 |
| sean-k-mooney | i assume that is a new feature since the upgrade | 14:44 |
| *** ysandeep|afk is now known as ysandeep | 14:44 | |
| fungi | sean-k-mooney: oh, yeah gerrit lets you set a status for your account now, like to say you're on vacation or something | 15:00 |
| fungi | i've simply made a humorous hitchhiker's guide to the galaxy reference in mine | 15:01 |
| fungi | https://hitchhikers.fandom.com/wiki/Lig_Lury_Jr#Disappearance | 15:03 |
| gmann | fungi: mnaser if you are around, few config patches need reviews- https://review.opendev.org/c/openstack/project-config/+/797777/2 https://review.opendev.org/c/openstack/project-config/+/797745 https://review.opendev.org/c/openstack/project-config/+/797784/2 https://review.opendev.org/c/openstack/project-config/+/797727 https://review.opendev.org/c/openstack/project-config/+/797781 | 15:38 |
| opendevreview | Brian Rosmaita proposed openstack/project-config master: Add project templates to rbd-iscsi-client https://review.opendev.org/c/openstack/project-config/+/798348 | 15:50 |
| *** ykarel is now known as ykarel|away | 15:52 | |
| *** abhishekk is now known as abhishekk|dinner | 15:55 | |
| *** ysandeep is now known as ysandeep|away | 15:56 | |
| *** jpena is now known as jpena|off | 16:00 | |
| *** abhishekk|dinner is now known as abhishekk|home | 16:28 | |
| *** abhishekk|home is now known as abhishekk | 16:28 | |
| *** stevebaker_ is now known as stevebaker | 20:08 | |
| opendevreview | Merged openstack/project-config master: Make explicit the network backend used in the CI jobs https://review.opendev.org/c/openstack/project-config/+/797454 | 20:55 |
| opendevreview | Merged openstack/project-config master: Remove governance-uc from infra https://review.opendev.org/c/openstack/project-config/+/797777 | 21:11 |
| opendevreview | Merged openstack/project-config master: Remove uc-recognition from infra https://review.opendev.org/c/openstack/project-config/+/797778 | 21:12 |
| opendevreview | Merged openstack/project-config master: End project gating for retiring scientific-wg repo https://review.opendev.org/c/openstack/project-config/+/797745 | 21:12 |
| opendevreview | Merged openstack/project-config master: Remove puppet-openstack-specs from infra https://review.opendev.org/c/openstack/project-config/+/797784 | 21:17 |
| opendevreview | Merged openstack/project-config master: Remove noop jobs for OpenStack-Ansible nspawn projects https://review.opendev.org/c/openstack/project-config/+/797727 | 21:17 |
| opendevreview | Kendall Nelson proposed openstack/ptgbot master: Add Container Image Build https://review.opendev.org/c/openstack/ptgbot/+/798025 | 21:49 |
| opendevreview | Merged openstack/project-config master: Update Zuul job semaphore usage https://review.opendev.org/c/openstack/project-config/+/798021 | 22:00 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Remove scientific-wg from infra https://review.opendev.org/c/openstack/project-config/+/797747 | 22:12 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Add noop job to remove content of puppet-openstack-specs repo https://review.opendev.org/c/openstack/project-config/+/798392 | 22:27 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Add noop job to remove content of puppet-openstack-specs repo https://review.opendev.org/c/openstack/project-config/+/798392 | 22:29 |
| opendevreview | Ghanshyam proposed openstack/project-config master: End project gate for puppet-openstack-specs https://review.opendev.org/c/openstack/project-config/+/798393 | 22:31 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Add noop job to remove content of puppet-openstack-specs repo https://review.opendev.org/c/openstack/project-config/+/798392 | 22:47 |
| opendevreview | Ghanshyam proposed openstack/project-config master: End project gate for puppet-openstack-specs https://review.opendev.org/c/openstack/project-config/+/798393 | 23:13 |
| opendevreview | Ghanshyam proposed openstack/project-config master: End project gate for puppet-openstack-specs https://review.opendev.org/c/openstack/project-config/+/798393 | 23:37 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!