| openstackgerrit | Evgeniy L proposed airship/treasuremap master: [DNM] Debug failures in airskiff gate related https://review.opendev.org/674238 | 00:05 |
|---|---|---|
| openstackgerrit | Evgeniy L proposed airship/treasuremap master: Stop systemd-resolved to fix DNS configuration https://review.opendev.org/674238 | 00:09 |
| openstackgerrit | Evgeniy L proposed airship/treasuremap master: Stop systemd-resolved to fix DNS configuration https://review.opendev.org/674238 | 00:09 |
| *** michael-beaver has quit IRC | 00:19 | |
| *** kranthikirang has quit IRC | 00:19 | |
| *** nishantkr has quit IRC | 00:29 | |
| *** aaronsheffield has quit IRC | 00:38 | |
| *** AlexNoskov has quit IRC | 00:44 | |
| *** rezroo has quit IRC | 00:45 | |
| *** altlogbot_1 has quit IRC | 00:46 | |
| *** altlogbot_1 has joined #airshipit | 00:46 | |
| *** kranthikirang has joined #airshipit | 01:03 | |
| *** kranthikirang has quit IRC | 01:04 | |
| *** kranthikirang has joined #airshipit | 01:05 | |
| *** timClicks has joined #airshipit | 01:06 | |
| *** kranthikirang has quit IRC | 01:09 | |
| *** kranthikirang has joined #airshipit | 01:10 | |
| *** kranthikirang has quit IRC | 01:15 | |
| *** timClicks has quit IRC | 02:48 | |
| *** gkadam has joined #airshipit | 03:33 | |
| *** gkadam has quit IRC | 03:40 | |
| *** sthussey has quit IRC | 05:12 | |
| *** phasespace has quit IRC | 05:55 | |
| *** jsuchome has joined #airshipit | 06:19 | |
| *** kskels has quit IRC | 06:41 | |
| *** dpawlik has joined #airshipit | 07:08 | |
| *** ianychoi has quit IRC | 07:46 | |
| *** ianychoi has joined #airshipit | 07:47 | |
| CobHead | When can I expect this to be merged? https://review.opendev.org/#/c/674238/ | 08:59 |
| CobHead | I want to get a demo up ASAP | 09:00 |
| CobHead | Actually, I'll just checkout that commit. | 09:05 |
| openstackgerrit | Dmitry Ukov proposed airship/airshipctl master: Implement imagegen cli command https://review.opendev.org/676121 | 09:34 |
| openstackgerrit | Dmitry Ukov proposed airship/airshipctl master: Inplement volume mount for container https://review.opendev.org/676141 | 09:34 |
| openstackgerrit | Dmitry Ukov proposed airship/airshipctl master: Add logic to image gen command https://review.opendev.org/676179 | 09:34 |
| roman_g | CobHead: CR +2 | 10:15 |
| CobHead | So, I checked out the commit, verified that the changes were indeed there and deployed to my usual Ubuntu 16.04 machine - it seems like the ucp-keystone-test container is sitll looping endlessly. I verified that systemd-resolved was inactive. | 10:28 |
| CobHead | I'll switch to Bionic there and see if anything changes. | 10:28 |
| *** gkadam has joined #airshipit | 10:38 | |
| CobHead | And the script crashed on Bionic, which was expected on my part - but I had a little hope. | 10:55 |
| *** gkadam is now known as gkadam-afk | 11:00 | |
| roman_g | срфтпу щт пфеуы шы дшьшеув ещ ишщтшс | 11:00 |
| roman_g | change on gates is limited to bionic | 11:00 |
| roman_g | Why ucp-keystone-test is looping - is another question. So, why? What's there in logs? | 11:01 |
| CobHead | The test succeeds, but it seems like nothing is returned from the container - so it runs until the deadline is exceeded, and then the pod is recreated -> http://paste.openstack.org/show/3cXl07VbLMJxyUD8vADH/ | 11:06 |
| CobHead | This continues on forever. | 11:07 |
| roman_g | What's in the logs of airship-ucp-keystone-test pod? | 11:10 |
| CobHead | Gonna need another re-deploy to find that again. Gonna take a while. | 11:11 |
| openstackgerrit | Dmitry Ukov proposed airship/airshipctl master: Add stdin support to container interface https://review.opendev.org/676105 | 11:17 |
| openstackgerrit | Dmitry Ukov proposed airship/airshipctl master: Implement imagegen cli command https://review.opendev.org/676121 | 11:17 |
| openstackgerrit | Dmitry Ukov proposed airship/airshipctl master: Inplement volume mount for container https://review.opendev.org/676141 | 11:17 |
| openstackgerrit | Dmitry Ukov proposed airship/airshipctl master: Add logic to image gen command https://review.opendev.org/676179 | 11:17 |
| openstackgerrit | Smruti Soumitra Khuntia proposed airship/promenade master: Enabling debug-report.sh to configure pods log duration. https://review.opendev.org/677448 | 11:19 |
| *** gkadam-afk is now known as gkadam | 11:42 | |
| openstackgerrit | varagini karthik proposed airship/porthole master: Chart/Dockerfile for Postgresql Utility https://review.opendev.org/675866 | 11:42 |
| *** roman_g has quit IRC | 11:43 | |
| openstackgerrit | varagini karthik proposed airship/porthole master: Chart/Dockerfile for Postgresql Utility https://review.opendev.org/675866 | 11:44 |
| *** roman_g has joined #airshipit | 11:46 | |
| openstackgerrit | varagini karthik proposed airship/porthole master: Zuul gates setup for Utility Containers https://review.opendev.org/675798 | 11:49 |
| CobHead | roman_g: http://paste.openstack.org/show/79chV2bDEsC5VVU4gBKh/ | 11:59 |
| CobHead | This is the log from the airship-ucp-keystone-test pod | 12:00 |
| CobHead | Interestingly, it seems to have completed the test, but is terminated and the pod is recreated in the script.. | 12:00 |
| roman_g | 2019-08-20 11:56:13.454 43 ERROR rally.task.engine [-] Unexpected exception during the workload execution: AuthenticationFailed: Failed to authenticate to http://keystone-api.ucp.svc.cluster.local:5000/v3 for user 'keystone-test' in project 'test': The request you have made requires authentication. | 12:01 |
| roman_g | test is failed | 12:01 |
| CobHead | Huh | 12:02 |
| CobHead | Anyways - now I didn't get the Deadline_Exceeded error in the script | 12:03 |
| CobHead | We'll see what happens this rerun | 12:03 |
| openstackgerrit | Sean Eagan proposed airship/armada master: Support builtin chart dependencies https://review.opendev.org/676831 | 12:09 |
| openstackgerrit | varagini karthik proposed airship/porthole master: Chart/Dockerfile for Postgresql Utility https://review.opendev.org/675866 | 12:09 |
| CobHead | http://paste.openstack.org/show/760312/ | 12:09 |
| CobHead | For some reason paste is not taking all the lines | 12:09 |
| *** henriqueof has quit IRC | 12:09 | |
| openstackgerrit | varagini karthik proposed airship/porthole master: Chart/Dockerfile for Postgresql Utility https://review.opendev.org/675866 | 12:10 |
| roman_g | https://paste.ubuntu.com/ has virtually no limit. I was not able to reach it. | 12:10 |
| CobHead | roman_g: https://paste.ubuntu.com/p/KpyqDPk6vK/ | 12:11 |
| CobHead | This time all report as passed | 12:12 |
| CobHead | Except I see that the Deadline_Exceeded error occurred in the script. | 12:12 |
| CobHead | I pasted that error here: https://paste.ubuntu.com/p/yvRbsGtSGp/ | 12:14 |
| roman_g | So it passes, but past deadline? | 12:14 |
| CobHead | It's possible, I guess. | 12:14 |
| CobHead | The deadline might be too hard on an AIAB? | 12:15 |
| CobHead | s/hard/short | 12:15 |
| roman_g | Could be. | 12:17 |
| CobHead | Skimming through the Max (sec) columns and summarizing it, it does not appear to be near the 300 sec deadline as described in the script tho.. | 12:17 |
| CobHead | Wait | 12:17 |
| CobHead | There's full durations, obviously (duh) | 12:17 |
| CobHead | Summarizing the full loads, it seems to be 371 secs | 12:21 |
| CobHead | So 71 secs beyond the deadline | 12:24 |
| CobHead | Wonder why the script doesn't terminate it the moment it actually hits the deadline tho - that would actually make more sense than terminate it and restarting after it has submitted the results back. | 12:25 |
| CobHead | Out of curiousity, I am going to give a new VM the max number of cores I can allocate at the moment (it currently has 6) to see if that will have an impact on the test. | 12:28 |
| CobHead | If it, by some miracle works now, then I suggest the actual requirements for AIAB be set higher, or the deadlines be set longer | 12:32 |
| *** gkadam has quit IRC | 12:32 | |
| CobHead | I guess the deadlines should be longer anyways(?) | 12:32 |
| roman_g | Well, doesn't hurt to set it to e.g. 600 seconds. | 12:34 |
| roman_g | I guess. | 12:34 |
| *** alexanderhughes has joined #airshipit | 12:41 | |
| *** mcriswell has joined #airshipit | 12:42 | |
| *** nishantkr has joined #airshipit | 12:45 | |
| openstackgerrit | Merged airship/spyglass-plugin-xls master: Upgrade yapf to 0.28 https://review.opendev.org/676748 | 12:49 |
| *** bh526r has joined #airshipit | 12:53 | |
| *** henriqueof has joined #airshipit | 12:56 | |
| *** zogger has joined #airshipit | 13:01 | |
| CobHead | roman_g: From where does Genesis get these timeouts? | 13:07 |
| CobHead | I want to change the source of it, not subsitute the template when it is already generate | 13:07 |
| evrardjp | o/ | 13:16 |
| CobHead | Nevermind - for the curious it's in under treasuremap/site/aiab/deployment/deployment-configuration.yaml | 13:16 |
| CobHead | Hi evrardjp o/ | 13:17 |
| evrardjp | how are things in here? | 13:17 |
| mattmceuen | o/ | 13:17 |
| *** kranthikirang has joined #airshipit | 13:19 | |
| alexanderhughes | \o | 13:20 |
| evrardjp | as usual, figuring out the webext url because my brain always forgets it | 13:24 |
| evrardjp | and my email search foo is not strong | 13:24 |
| openstackgerrit | Merged airship/porthole master: Enable usage of calicoq utility https://review.opendev.org/675656 | 13:29 |
| *** kranthikirang has quit IRC | 13:31 | |
| *** kranthikirang has joined #airshipit | 13:31 | |
| *** aaronsheffield has joined #airshipit | 13:33 | |
| *** ab2434_ has joined #airshipit | 13:34 | |
| *** kranthikirang has quit IRC | 13:36 | |
| CobHead | evrardjp: Debugging AIAB. Seems like 300 is too short of a timeout wrt tests @ 8 cores @2.4GHz + 25GB RAM, thus leading to the eternal loop. | 13:37 |
| CobHead | I've set all 300 timeouts to 600 now - rerun in progress. | 13:38 |
| CobHead | It also seems to me that the deadline counter starts when the pod is initializing (and that takes some time too.) - thus adding to the total runtime of the test. | 13:40 |
| *** kranthikirang has joined #airshipit | 13:40 | |
| *** sthussey has joined #airshipit | 13:42 | |
| *** kranthikirang has quit IRC | 13:45 | |
| CobHead | I've learned a valuable lesson today - do not trust Zuul if the project in question contains timeout(s) on the tests, as Zuul is vastly more powerful than my current setup. | 13:45 |
| nishantkr | Hello everyone! Meeting starts in another 15 minutes. Please add any topics you would like to discuss - https://etherpad.openstack.org/p/airship-meeting-2019-08-20 | 13:45 |
| *** ian-pittwood has joined #airshipit | 13:45 | |
| *** jamesgu_ has joined #airshipit | 13:45 | |
| *** jamesgu__ has joined #airshipit | 13:45 | |
| *** kranthikirang has joined #airshipit | 13:46 | |
| *** jamesgu__ has quit IRC | 13:46 | |
| CobHead | Is anyone not not active as a contributor to the project allowed to join? :] | 13:47 |
| CobHead | Double not there.. Disregard first. | 13:47 |
| nishantkr | CobHead: everyone is welcomed to join the meeting and feel free to add any topics onto the etherpad you would like to discuss :) | 13:50 |
| sthussey | @cobhead joining the meeting makes you a contributor | 13:51 |
| CobHead | Thanks :) | 13:51 |
| nishantkr | #startmeeting airship | 14:00 |
| openstack | Meeting started Tue Aug 20 14:00:08 2019 UTC and is due to finish in 60 minutes. The chair is nishantkr. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| *** openstack changes topic to " (Meeting topic: airship)" | 14:00 | |
| openstack | The meeting name has been set to 'airship' | 14:00 |
| nishantkr | #topic Rollcall | 14:00 |
| *** openstack changes topic to "Rollcall (Meeting topic: airship)" | 14:00 | |
| alexanderhughes | o/ | 14:00 |
| howell | o/ | 14:00 |
| CobHead | o/ | 14:00 |
| nishantkr | Hello everyone!! | 14:00 |
| openstackgerrit | Merged airship/deckhand master: Upgrade six to 1.12 https://review.opendev.org/677272 | 14:00 |
| evgenyl | hi! | 14:00 |
| nishantkr | Here is the Agenda for today - https://etherpad.openstack.org/p/airship-meeting-2019-08-20 | 14:00 |
| ian-pittwood | o/ | 14:00 |
| mattmceuen | o/ | 14:00 |
| sthussey | here | 14:00 |
| nishantkr | Please add anything you would like to discuss | 14:00 |
| jemangs | o/ | 14:00 |
| nishantkr | let's give it a min before we start | 14:01 |
| dwalt | o/ | 14:01 |
| *** seaneagan has joined #airshipit | 14:01 | |
| nishantkr | ok let's go for it !! | 14:02 |
| nishantkr | #topic Announcements | 14:02 |
| *** openstack changes topic to "Announcements (Meeting topic: airship)" | 14:02 | |
| nishantkr | Airship Working committe conducted it's first meeting yesterday. mattmceuen sent out any email yesterday regarding the same but for anyone who might have missed that you could find all the details regarding the meeting here - https://wiki.openstack.org/wiki/Airship/Airship-WC | 14:02 |
| nishantkr | some key highlights of that meeting- | 14:03 |
| hogepodge | Congratulations to the new WC! | 14:03 |
| mattmceuen | ty! :) | 14:03 |
| nishantkr | 1) you would be noticing this meeting being hosted by one of the WC members going forward thus sharing the responsibility and workload that has so far been fantastically managed by mattmceuen alone | 14:03 |
| *** henriqueof has quit IRC | 14:03 | |
| nishantkr | 2) A special election would be held after 3 months from the first election held for WC to fill out the fifth seat. Since the 2 member per company quota is already filled up by AT&T and Ericsson, we would be looking for a 5th member from another company. Please look out for the announcement and news on the election date 3 months (approx) from now | 14:04 |
| nishantkr | 3) We have planned to gather feedback from the community in the form of survey and also have a new section "community feedback" in the etherpad used for this meeting. Please provide your feedback in that section and help us improve as a community. | 14:05 |
| *** DanCrank has joined #airshipit | 14:05 | |
| nishantkr | Please provide your feedback on gathering the feedback as well and let us know if you have any other ideas. | 14:05 |
| nishantkr | mattmceuen, dwalt and kskels - Please add if I may have missed something | 14:06 |
| mattmceuen | just one more thing on #2 -- we will run the special election plan by the TC to make sure they're good with it, since they own our governance in general | 14:06 |
| sthussey | I thought the governance said that WC owned governance? | 14:07 |
| mattmceuen | hmm, if so we're done :D maybe I'm remembering the details wrong, will double check | 14:07 |
| openstackgerrit | Dan Crank proposed airship/promenade master: Fixes/updates for webhook-apiserver https://review.opendev.org/665761 | 14:07 |
| dwalt | I'd also like to add that the last fifteen minutes are reserved as an open forum for questions -- anyone is welcome to attend the meeting! | 14:07 |
| sthussey | https://www.irccloud.com/pastebin/8493LHQF/ | 14:07 |
| nishantkr | ok thanks for pointing that out mattmceuen and sthussey | 14:08 |
| mattmceuen | good catch sthussey! | 14:08 |
| nishantkr | thank you dwalt | 14:08 |
| nishantkr | ok then moving on | 14:08 |
| nishantkr | mattmceuen: you have the next item in the Announcement. please go for it. | 14:08 |
| mattmceuen | In case anyone missed it: https://about.att.com/story/2019/att_dell_opensource5g.html | 14:09 |
| mattmceuen | There was a press release from Dell & AT&T around collaborating on 5G technology, and a big part of that was that Dell is going to be getting into Airship development in a big way | 14:09 |
| mattmceuen | So that's great to hear - welcome to anyone from Dell who is here or will be joining over the coming weeks! | 14:10 |
| *** pramchan has joined #airshipit | 14:10 | |
| mattmceuen | excited for you to be joining us | 14:10 |
| mattmceuen | that's all from me nishantkr | 14:10 |
| nishantkr | That's exciting news. Thanks for sharing mattmceuen. | 14:10 |
| nishantkr | #topic Package management for Airship python projects | 14:10 |
| *** openstack changes topic to "Package management for Airship python projects (Meeting topic: airship)" | 14:10 | |
| nishantkr | ian-pittwood: go for it !! | 14:11 |
| ian-pittwood | Thanks! | 14:11 |
| hogepodge | The candidate pool for the upcoming WC election is a bit larger now ;-) | 14:11 |
| ian-pittwood | So as you all know, right now we are just using opendev git repos for all of airship | 14:11 |
| mattmceuen | +1 hogepodge. Let's get those commits in! | 14:11 |
| ian-pittwood | We aren't doing any packaging that I know of | 14:11 |
| ian-pittwood | Recently I've encountered a lot of issues dealing with using only these git repos for our dependencies. Pip doesn't really play well with VCS dependencies it seems. I'm curious how everyone would feel about possibly packaging and uploading our airship projects onto PyPI or some other index | 14:12 |
| ian-pittwood | Thoughts? | 14:12 |
| pramchan | If we have python package like sushy-tools if we need to call fro go is that via RESTfull calls only or there are other options to call go to python directly? | 14:13 |
| ian-pittwood | pramchan: sorry I'm a little confused by that. Are you asking how we'll be calling python from golang in the future? Or something else? | 14:14 |
| pramchan | yes | 14:14 |
| sthussey | @ian-pittwood i agree w/ pypi | 14:14 |
| nishantkr | ian-pittwood: that sounds good to me unless anyone has any other thoughts on it. | 14:14 |
| sthussey | but someone will need to set up the plumbing | 14:15 |
| sthussey | and more, we will need to adopt some versioning | 14:15 |
| pramchan | Let me hold this while we get pypi cleared | 14:15 |
| ian-pittwood | pramchan: It would be essentially the same as far as I know. The main difference is that using packages instead of buillding from VCS will make dependency installation easier. There won't be any difference in really how we use it | 14:15 |
| ian-pittwood | Yeah who would manage an Airship PyPI account? | 14:15 |
| CobHead | Question - you are talking about packaging the minor/major releases of airship and uploading to PyPi, right? | 14:15 |
| *** spilla has joined #airshipit | 14:16 | |
| ian-pittwood | Or would we tag along with OpenStack's account?? | 14:16 |
| *** StaceyF has joined #airshipit | 14:16 | |
| ian-pittwood | Well that's actually a thing I've thought about CobHead. We can do the major/minor releases, but we can also make dev pre-releases | 14:16 |
| ian-pittwood | So for every change we make that is merged between releases, we can still install a version of the package with those changes | 14:16 |
| ian-pittwood | Thanks to the magic of pbr | 14:17 |
| mattmceuen | ian-pittwood: re the pypi account -- management of airship shared infrastructure credentials is one of the things on the WC backlog we didn't get to yesterday - should hopefully have a suggestion next week | 14:17 |
| ian-pittwood | So instead of just having v0.1.0, v0.2.0, etc... We can also install v0.1.0dev28 | 14:17 |
| *** zogger has quit IRC | 14:18 | |
| ian-pittwood | mattmceuen so this is an action item for WC? | 14:18 |
| *** kskels has joined #airshipit | 14:18 | |
| CobHead | I see | 14:18 |
| mattmceuen | action item to to figure out, not necessarily to own :) | 14:18 |
| ian-pittwood | Haha well I can help if need be. I did some research yesterday to make sure I understood the process | 14:19 |
| *** jezogwza has joined #airshipit | 14:19 | |
| mattmceuen | account management was a broader thing that sthussey brought up last week | 14:19 |
| *** cheng1 has quit IRC | 14:19 | |
| mattmceuen | ++ thanks! | 14:19 |
| nishantkr | yep, i will note the action item on me for now and we can discuss the process on the WC and irc meeting next week. | 14:19 |
| sthussey | you'll need to convert some of the modules to pbr | 14:19 |
| ian-pittwood | Ok, one more quick question on this | 14:19 |
| ian-pittwood | What's up with promenade's dependency management? Why doesn't it use a requirements.txt? | 14:20 |
| nishantkr | #action nishantkr Airship python project packaging with pypi | 14:20 |
| *** michael-beaver has joined #airshipit | 14:20 | |
| ian-pittwood | I can foresee it becoming a problem for us when it comes to packaging | 14:21 |
| pramchan | #info https://pypi.org/project/python-openstackclient/ is it different from what airship uses? | 14:21 |
| *** AlexNoskov has joined #airshipit | 14:21 | |
| *** cheng1 has joined #airshipit | 14:21 | |
| nishantkr | ian-pittwood: I think it's more readable in the sense we just use direct dependencies in promenade, hence we have separate requirements file as a best practice maybe? | 14:22 |
| alexanderhughes | I think converting to a release management system is going to be a deeper discussion one that might be better suited to the Airship design calls if we can pull some time from Airship2.0 talks. Might save us some headaches from the lessons we learn now too with 2.0 | 14:22 |
| ian-pittwood | nishantkr the problem is that's not helpful for other projects attempting to install promenade. As far as I know, pip only searches for requirements.txt | 14:23 |
| nishantkr | ideally requirements.txt could have pinned version vs unpinned version package which could help in upgrading those unpinned packages easily | 14:23 |
| alexanderhughes | the problem with pinned packages, especially in projects that are consumed by others such as Pegleg (promenade, deckhand, shipyard) is their pins aren't consistent when they are pinned | 14:24 |
| ian-pittwood | Libraries should not have hard pins imo | 14:24 |
| ian-pittwood | End user apps are the hard pinned applications | 14:24 |
| ian-pittwood | And there's tools we can use to help keep packages up to date, we don't have to create our own system for it | 14:25 |
| nishantkr | alexanderhughes: yep that's right. i recently had to face a similar issue | 14:25 |
| sthussey | all of these tools were built as simple API services | 14:25 |
| sthussey | The need to use them as programmatic APIs wasn't really considered | 14:25 |
| sthussey | The correct thing is to split the code bases | 14:25 |
| nishantkr | ian-pittwood: that's a valid point. let's bring that up in the design call and decide the way forward. | 14:26 |
| sthussey | Because not hard pinning has the problem of build variance when building docker images | 14:26 |
| ian-pittwood | sthussey that's a problem that pipenv is supposed to solve, but unfortunately it's not ideal for libraries | 14:26 |
| sthussey | yeah, I don't think anyone looks to Python for how to solve dependency management | 14:27 |
| ian-pittwood | It's a difficult issue. I agree that we can probably discuss it on the design call | 14:27 |
| nishantkr | ok sounds good. | 14:27 |
| nishantkr | any other thoughts on this topic? | 14:28 |
| nishantkr | ok moving on then | 14:28 |
| nishantkr | #topic Brief intro to Airship 2.0 scope in Jira | 14:28 |
| *** openstack changes topic to "Brief intro to Airship 2.0 scope in Jira (Meeting topic: airship)" | 14:28 | |
| mattmceuen | Ah that's mine | 14:28 |
| nishantkr | please go for it | 14:28 |
| mattmceuen | I wanted to give a brief intro to some of the work jezogwza has started doing to scope out Airship 2.0 work in our Jira instance, so we see it coming and see what's most ready to be worked in terms of prioritization and being fleshed out | 14:29 |
| mattmceuen | This isn't going to be a deep dive since you can't see my screen :) | 14:29 |
| mattmceuen | https://airship.atlassian.net/secure/RapidBoard.jspa | 14:30 |
| mattmceuen | The Backlog has the general progression from top to bottom of the scope, roughly cronologically | 14:30 |
| mattmceuen | It's split into monthly sprints, with targeted delivery months in the name -- e.g. 201908 - Bootstrapping | 14:31 |
| mattmceuen | Under that are higher level chunks of functionality (e.g. Basic Config + Document Functionality) that are targeted to those sprints | 14:32 |
| mattmceuen | and under those are the granular user stories | 14:32 |
| mattmceuen | On the right, you see proposed meaningful releases for the scope -- e.g. Airship 2.0 Alpha, Airship 2.0 Beta, Airship 2.0, Post-Airship 2.0 | 14:32 |
| mattmceuen | and Airship 1.0 :) we can use this for 1.x scope as well | 14:33 |
| mattmceuen | that's really all I wanted to share here -- feel free to take a look and provide feedback to jezogwza, I know he's looking to refine this over time | 14:33 |
| nishantkr | yep that sounds a good place to start looking into. thanks for sharing it mattmceuen | 14:34 |
| mattmceuen | any questions yet? | 14:34 |
| mattmceuen | Sure thing | 14:34 |
| mattmceuen | if any - let me know | 14:34 |
| pramchan | Good summary | 14:34 |
| nishantkr | ok moving on then | 14:35 |
| nishantkr | #topic New project proposal: airshipui | 14:35 |
| *** openstack changes topic to "New project proposal: airshipui (Meeting topic: airship)" | 14:35 | |
| nishantkr | mattmceuen: this is yours. go for it ! | 14:35 |
| mattmceuen | As you guys are probably aware, there has been a design SIG around adding a user interface for airship | 14:35 |
| mattmceuen | There is a request to create a home for this work -- the airship/airshipui project | 14:36 |
| sthussey | I assume you mean a graphical interface? | 14:36 |
| sthussey | versus the existing command line interface(s) | 14:36 |
| mattmceuen | I haven't been in the SIG meetings so I don't have a whole lot of detail to share, anyone else have additional info? | 14:36 |
| mattmceuen | yep sthussey | 14:36 |
| pramchan | Ui sig meeting what's the link for meeting, I ddi not see any in therpad | 14:37 |
| jezogwza | Essentialy its a dashboard (WUI) that will provide functionality that airshipctl exposes, but moe. i.e. YAML friendly functionaity | 14:37 |
| jezogwza | The current propoal from SUSE is to use a framework called stratos they developed for cloud foundry | 14:37 |
| alexanderhughes | #link https://urldefense.proofpoint.com/v2/url?u=https-3A__etherpad.openstack.org_p_Airship-5FUI&d=DwMF-g&c=LFYZ-o9_HUMeMTSQicvjIg&r=6f0dU7tUT8UoVrYjQwg8cQ&m=MrkJ5xsBj9WB3el-Npgux8hotpubvmCc2_srNjFZwAQ&s=2VDG55yhgCj-LKLU9HXPrc3YAspYx5U1OM15iE_F47s&e= | 14:37 |
| jezogwza | We have had a ademo and some explanation about it | 14:38 |
| alexanderhughes | #link https://etherpad.openstack.org/p/Airship_UI | 14:38 |
| jezogwza | Will be mking a decision on the framework on this Fridya's sig-airship-ui call | 14:38 |
| mattmceuen | I +1 this, because a GUI is something that we've received a large volume of interest/desire for over the last year | 14:38 |
| mattmceuen | ty jezogwza & alexanderhughes! | 14:38 |
| jezogwza | eh framework has a frontend (using Angular and Node.js) and a backend written in g | 14:38 |
| jezogwza | go | 14:38 |
| jezogwza | Already has some integration with k8s and a poc for integrating with Argo was demoed last week | 14:39 |
| pramchan | where is the meeting bridge on that? no link in etherpad above? | 14:39 |
| mattmceuen | pramchan: https://wiki.openstack.org/wiki/Airship#AIRSHIP_SIG_-_Special_Interest_Meetings | 14:39 |
| pramchan | thanks | 14:40 |
| nishantkr | thanks jezogwza and alexanderhughes for sharing the details ! | 14:40 |
| mattmceuen | Although we don't yet have the full direction ironed out for the project, I think we can vote yea/nay on creation of a project in general | 14:40 |
| *** rezroo has joined #airshipit | 14:40 | |
| mattmceuen | Are we good with adding an airshipui? | 14:40 |
| alexanderhughes | +1 | 14:40 |
| jezogwza | +1 | 14:41 |
| ian-pittwood | +1 | 14:41 |
| nishantkr | +1 | 14:41 |
| dwalt | +1 | 14:41 |
| mattmceuen | \o/ | 14:41 |
| nishantkr | That sounds like all yay's :) | 14:41 |
| mattmceuen | thanks all | 14:41 |
| *** dpawlik has quit IRC | 14:41 | |
| nishantkr | #topic Updating the https://etherpad.openstack.org/p/airship-team-meeting-agenda page | 14:42 |
| *** openstack changes topic to "Updating the https://etherpad.openstack.org/p/airship-team-meeting-agenda page (Meeting topic: airship)" | 14:42 | |
| nishantkr | ian-pittwood: go for it ! | 14:42 |
| ian-pittwood | Are we still doing that page? | 14:42 |
| ian-pittwood | It's still listed on the get in touch page, but it hasn't been updated since March | 14:42 |
| mattmceuen | I guess we either need to keep it updated or remove it from the get in touch page | 14:42 |
| alexanderhughes | seems like extra effort when it's all available at #link http://eavesdrop.openstack.org/meetings/airship/ | 14:43 |
| ian-pittwood | Well that's the logs, not the agendas | 14:43 |
| alexanderhughes | I'd suggest removing from get in touch page, but, adding to IRC channel or get in touch page the link to the next week's agenda so it's easy to find | 14:43 |
| mattmceuen | That is how I'm leaning - but if anyone gets value out of it I don't mind updating either | 14:43 |
| openstackgerrit | varagini karthik proposed airship/porthole master: Chart/Dockerfile for Postgresql Utility Container https://review.opendev.org/675866 | 14:43 |
| alexanderhughes | well last week's agenda is covered in detail in the meeting log. each agenda item has its own topic and discussion | 14:44 |
| ian-pittwood | alexanderhughes that sounds fine to me | 14:44 |
| nishantkr | ok any other sugsestion? do we feel the need to update the page or just remove it? | 14:44 |
| alexanderhughes | remove page, but make upcoming agenda easy to find. and work to get it created in advance | 14:45 |
| alexanderhughes | often times it's not created until an hour before | 14:45 |
| pramchan | no keep the pages we do use it | 14:45 |
| mattmceuen | ok - if you guys use it, I can update the etherpad, just a bit of copy + paste + edit | 14:46 |
| alexanderhughes | get in touch is fine to keep, but links to previous agendas should just be replaced with link to previous meetings | 14:46 |
| nishantkr | ok if we have folks using it then it may make sense to update it for the time being. | 14:46 |
| mattmceuen | ah I see | 14:46 |
| nishantkr | #action mattmceuen to update the airship-meeting-agenda page | 14:47 |
| nishantkr | any other thoughts on this topic? | 14:47 |
| mattmceuen | one sec | 14:47 |
| nishantkr | sure | 14:48 |
| openstackgerrit | Jagan Mohan Kavva proposed airship/porthole master: Zuul gates setup for Utility Containers https://review.opendev.org/675798 | 14:48 |
| mattmceuen | It looks like the Get in Touch section only references the meeting logs (not the etherpad): https://wiki.openstack.org/wiki/Airship#Get_in_Touch | 14:48 |
| mattmceuen | I don't see the reference to the "big list of agendas" etherpad, am I missing it? | 14:48 |
| ian-pittwood | Hmm I swore it was somewhere still | 14:49 |
| mattmceuen | or did someone update it just now? :) | 14:49 |
| ian-pittwood | http://eavesdrop.openstack.org/#Airship_Team_Meeting | 14:49 |
| ian-pittwood | It's there I think | 14:49 |
| mattmceuen | ah, that's the one | 14:50 |
| mattmceuen | Cool. I'll update it, and will try to do a better job along with WC colleagues on entering the next agenda into that, ahead of time | 14:50 |
| mattmceuen | I'm good to move on nishantkr | 14:50 |
| nishantkr | thanks mattmceuen. | 14:50 |
| nishantkr | ok moving on to our last topic | 14:51 |
| nishantkr | #topic AIAB timeout (deafline) on tests is set too low by default for common hardware (i.e. desktops). Should ideally be higher by default | 14:51 |
| *** openstack changes topic to "AIAB timeout (deafline) on tests is set too low by default for common hardware (i.e. desktops). Should ideally be higher by default (Meeting topic: airship)" | 14:51 | |
| nishantkr | CobHead: was that added by you? | 14:51 |
| CobHead | Hi :) | 14:51 |
| CobHead | Yes | 14:51 |
| CobHead | So I made it past the hurdle of deploying AIAB wrt the postgresql pod entering a CrashLoopBackoff state - however, the deploy is now failing because of a short deadline, especially during the keystone test (which is set to 300s by default.) It seems to me that the behavior of the deploy is weird. If a deadline is exceeded, one would expect the script to terminate the pod immediately, not wait until the test has completed and the results has been | 14:51 |
| CobHead | returned. This also means that the default deadline should be set higher, as most people probably deploy AIAB to play around with, etc. If this deadline is always exceeded, the deploy never finishes. My runs use an average of ~370 seconds on the test, which includes the pod initialization. | 14:51 |
| mattmceuen | yeah, those timeouts have evolved over time to be more accurate, but are sensitive to the hardware they're running on, among other potential things | 14:52 |
| mattmceuen | CobHead would you be willing to bump that timeout in the manifest to a better value (if no objections from the team)? | 14:53 |
| mattmceuen | (I mean push a PS to update it in the repo) | 14:53 |
| CobHead | Yes, but if a deadline is exceeded (x) amounts of times - the script should terminate, not continue on for eternity? | 14:53 |
| mattmceuen | Couple different things going on there | 14:54 |
| mattmceuen | but agree they're separate concerns -- reasonable timeout value, vs how to handle a timeout | 14:54 |
| mattmceuen | For handling the timeout -- keep in mind that the airship in a bottle script isn't intended for prod use; it's a runner for the Shipyard API, which is intended for prod use | 14:55 |
| CobHead | I'm aware :) | 14:55 |
| mattmceuen | Shipyard retries a few times and then sets the status of the workflow to a Failed state | 14:55 |
| CobHead | I tried to increase the deadline by setting setting the variables in /treasuremap/site/aiab/deployment/site/deployment-configuration.yaml - however it seems to me that Genesis is still setting 300 as the default - and I haven't figured out where it gets this default from. | 14:55 |
| openstackgerrit | Merged airship/porthole master: Zuul gates setup for Utility Containers https://review.opendev.org/675798 | 14:56 |
| roman_g | patch it on the way while it runs :) | 14:56 |
| roman_g | Hello everyone. o/ | 14:56 |
| CobHead | I thought of that :p | 14:56 |
| mattmceuen | DId the AIAB script fail-to-fail on the first try only, or was it just waiting for all the retries to occur CobHead? | 14:56 |
| nishantkr | CobHead: armada chart deployment should fail if the timeouts have been exceeded and it would go for another retry but yeah we don't really purge the chart or remove the pods before starting another retry. | 14:56 |
| mattmceuen | o/ roman_g! | 14:56 |
| openstackgerrit | PRATEEK REDDY DODDA proposed airship/porthole master: Chart/Dockerfile for Openstack Utility Container Added Support for rbac https://review.opendev.org/674670 | 14:56 |
| mattmceuen | Armada manages that for you (given the appropriate settings) | 14:57 |
| CobHead | mattmceuen: It haven't failed, as I usually terminate it when it fails for the 4th of 5th time. | 14:57 |
| CobHead | The loop, that is. | 14:57 |
| mattmceuen | hmm ok - sounds like that could be improved on the aiab script side then | 14:58 |
| mattmceuen | thanks for bringing it up CobHead, I'll take a look | 14:58 |
| CobHead | Great - thanks :) | 14:58 |
| openstackgerrit | Jagan Mohan Kavva proposed airship/porthole master: Chart/Dockerfile for etcdctl Utility Container https://review.opendev.org/674680 | 14:58 |
| *** openstackstatus has quit IRC | 14:58 | |
| nishantkr | yep thanks CobHead | 14:58 |
| *** openstack has joined #airshipit | 14:59 | |
| *** ChanServ sets mode: +o openstack | 14:59 | |
| mattmceuen | perfect - that's the one I'll focus on | 14:59 |
| CobHead | I don't trust the GitHub one, as it haven't been updated in a while :p | 14:59 |
| nishantkr | #topic requests for review | 15:00 |
| mattmceuen | hmm - it oughta be syncing, that may be another issue | 15:00 |
| nishantkr | we have a number of PS waiting for review. | 15:00 |
| nishantkr | https://www.irccloud.com/pastebin/x8iM90bu/ | 15:00 |
| nishantkr | Please review these PS if you can. | 15:00 |
| nishantkr | I think we are out of time guys. Thanks for joining. | 15:01 |
| mattmceuen | Thanks nishantkr! | 15:01 |
| nishantkr | #endtopic | 15:01 |
| CobHead | Thanks :) | 15:01 |
| dwalt | I didn't know the Airskiff type was ready for review. Fantastic! | 15:01 |
| nishantkr | #endmeeting | 15:01 |
| roman_g | openstack: hello there | 15:02 |
| roman_g | wake up, Neo | 15:02 |
| roman_g | OK. It forgot that it run meeting. | 15:03 |
| mattmceuen | lol roman_g I just watched that movie a couple weeks ago, for the first time in maybe 15 years | 15:03 |
| roman_g | =) | 15:04 |
| jamesgu_ | dwalt: look forward to your feedback on the airskiff type patch | 15:04 |
| dwalt | jamesgu_: plan to give it a look soon :) | 15:06 |
| CobHead | mattmceuen: Since Genesis is not adhering to the deployment_configuration.yaml file, where does it get its timeouts from? | 15:06 |
| *** itxaka has joined #airshipit | 15:07 | |
| * CobHead trying to educate myself on the workings of AirShip | 15:07 | |
| dwalt | So, I think the deployment configuration file only holds configuration for Shipyard workflows. Genesis is driven by scripts generated from Promenade. | 15:08 |
| CobHead | And where does Promenade get the basis for those scripts? :) | 15:09 |
| dwalt | During that flow, it uses Armada to deploy software | 15:09 |
| dwalt | sorry, typing as fast as I can :0 | 15:09 |
| dwalt | I missed the earlier discussion a bit. Are you looking to adjust the entire genesis timeout, or the timeout for a particular software component? | 15:09 |
| CobHead | Out of curiosity - both :) | 15:10 |
| mattmceuen | dwalt: CobHead was hitting the 300s timeout on the keystone test during genesis | 15:12 |
| roman_g | mattmceuen: https://www.lightreading.com/nfv/nfv-mano/podcast-ses-networks-takes-onap-sky-high/a/d-id/753564 SES Networks Takes ONAP Sky-High | 15:13 |
| roman_g | ONAP - read ECOMP | 15:13 |
| roman_g | May be good link to fwd to Ryan's team. | 15:14 |
| openstackgerrit | Marjorie Middleton proposed airship/porthole master: Initial Commit of compute-utility container code https://review.opendev.org/674695 | 15:14 |
| *** itxaka has quit IRC | 15:16 | |
| openstackgerrit | Ian Pittwood proposed airship/pegleg master: Fix OpenSUSE image build https://review.opendev.org/676944 | 15:19 |
| dwalt | mattmceuen: ah, ty | 15:22 |
| dwalt | CobHead: The charts specified in this manifest are deployed during genesis https://opendev.org/airship/treasuremap/src/branch/master/global/software/manifests/bootstrap.yaml | 15:22 |
| openstackgerrit | Ian Pittwood proposed airship/pegleg master: Fix OpenSUSE image build https://review.opendev.org/676944 | 15:23 |
| mattmceuen | CobHead: I'm trying to track down that 300 value with dwalt, I'd have expected it to be 600 | 15:24 |
| jamesgu_ | is there a difference between chart deploy time out vs helm test timeout? The latter if exceeded will cause retries | 15:29 |
| jamesgu_ | If I remember correctly | 15:29 |
| mattmceuen | CobHead: can you share the aiab.sh logs / armada logs from around the time of the timeout? | 15:30 |
| hogepodge | alexanderhughes: mattmceuen: YouTube Airship Playlist https://www.youtube.com/playlist?list=PLKqaoAnDyfgp8YjZbzjVrmZBJR9thV27y | 15:32 |
| mattmceuen | hogepodge, that's awesome - thanks! Looks very legit | 15:35 |
| hogepodge | As long as it looks legit ;-) | 15:36 |
| mattmceuen | well I mean - looks more legit than if matt mceuen was sharing some airship playlist :D | 15:37 |
| mattmceuen | has the trappings of the illustrious OSF | 15:37 |
| seaneagan | jamesgu_: both the chart deploy timeout (`wait.timeout`) and test timeout (`test.timeout`) would cause retries if exceeded. genesis will retry forever. shipyard will retry the configured amount of times. | 15:37 |
| openstackgerrit | Ian Pittwood proposed airship/promenade master: [WIP] Pull requirements-direct into requirements https://review.opendev.org/677497 | 15:45 |
| mattmceuen | yeah.. I was getting my wait.timeout and test.timeout confused. oops. | 15:46 |
| alexanderhughes | hogepodge thanks! I'll browse youtube later and see if I can find some more airship content, like the interviews and press releases about Dell and send it to you in an email | 15:46 |
| hogepodge | Fantastic! Thank you! | 15:47 |
| mattmceuen | CobHead: following seaneagan's advice, adding a `test.timeout` with an appropriate non-300 value would be done in this yaml (would apply to both genesis and post-genesis) | 15:51 |
| mattmceuen | https://opendev.org/airship/treasuremap/src/branch/master/global/software/charts/ucp/keystone/keystone.yaml | 15:51 |
| seaneagan | likely we'd want to override the openstack keystone `test.timeout` in conjunction with that as well | 16:24 |
| *** jsuchome has quit IRC | 16:54 | |
| *** pramchan has quit IRC | 17:19 | |
| *** howell has quit IRC | 17:21 | |
| *** howell has joined #airshipit | 17:21 | |
| *** howell has quit IRC | 17:36 | |
| *** howell has joined #airshipit | 17:37 | |
| openstackgerrit | Ian Pittwood proposed airship/promenade master: Pull requirements-direct into requirements https://review.opendev.org/677497 | 17:38 |
| dwalt | jamesgu_: (and Rajeshwari) nice work on the skiff type! Just have a couple of minor requests when you have a chance to revisit it. | 17:49 |
| openstackgerrit | Rajeshwari Dharwadkar proposed airship/shipyard master: Fix issue with shipyard build image https://review.opendev.org/677032 | 17:51 |
| openstackgerrit | Rajeshwari Dharwadkar proposed airship/shipyard master: Fix issue with shipyard build image https://review.opendev.org/677032 | 17:52 |
| openstackgerrit | Matthew Carter proposed airship/drydock master: [WIP] Drydock logging enhancements https://review.opendev.org/677528 | 17:59 |
| *** bh526r has quit IRC | 18:03 | |
| openstackgerrit | Merged airship/promenade master: Fixes/updates for webhook-apiserver https://review.opendev.org/665761 | 18:09 |
| *** mbuil has left #airshipit | 18:13 | |
| openstackgerrit | Merged airship/armada master: Removing flag added to opensuse gate job to make it non-voting https://review.opendev.org/677271 | 18:22 |
| openstackgerrit | Matthew Carter proposed airship/drydock master: [fix] Update hostdeps.sh to do a non-interactive apt install https://review.opendev.org/677536 | 18:26 |
| openstackgerrit | Matthew Carter proposed airship/drydock master: [fix] Update hostdeps.sh to do a non-interactive apt install https://review.opendev.org/677536 | 18:46 |
| openstackgerrit | Matthew Carter proposed airship/drydock master: [WIP] Drydock logging enhancements https://review.opendev.org/677528 | 18:50 |
| openstackgerrit | Merged airship/treasuremap master: Stop systemd-resolved to fix DNS configuration https://review.opendev.org/674238 | 18:55 |
| openstackgerrit | Anthony Bellino proposed airship/maas master: maas-ingress and maas-ingress-errors pods with non-root user https://review.opendev.org/657671 | 19:05 |
| openstackgerrit | Roman Gorshunov proposed airship/deckhand master: Remove Python 2.x from tox https://review.opendev.org/677552 | 19:08 |
| openstackgerrit | Roman Gorshunov proposed airship/deckhand master: Remove Python 2.x from tox https://review.opendev.org/677552 | 19:12 |
| openstackgerrit | Sean Eagan proposed airship/armada master: Support builtin chart dependencies https://review.opendev.org/676831 | 19:13 |
| *** alexanderhughes has quit IRC | 19:20 | |
| openstackgerrit | Merged airship/deckhand master: Add Python 3 Train unit tests https://review.opendev.org/676041 | 19:21 |
| *** ian-pittwood has quit IRC | 19:21 | |
| openstackgerrit | Roman Gorshunov proposed airship/deckhand master: Remove Python 2.x support https://review.opendev.org/677552 | 19:42 |
| *** mcriswell has quit IRC | 20:03 | |
| openstackgerrit | James Gu proposed airship/treasuremap master: Add new airskiff type https://review.opendev.org/674963 | 20:15 |
| openstackgerrit | James Gu proposed airship/treasuremap master: Add new airskiff type https://review.opendev.org/674963 | 20:20 |
| openstackgerrit | James Gu proposed airship/treasuremap master: Add airskiff_suse site https://review.opendev.org/675440 | 20:20 |
| openstackgerrit | Scott Hussey proposed airship/promenade master: [WIP] (charts) Webhook dynamic config https://review.opendev.org/677039 | 20:23 |
| openstackgerrit | Scott Hussey proposed airship/promenade master: (charts) Webhook dynamic config https://review.opendev.org/677039 | 20:31 |
| openstackgerrit | Dmitry Ukov proposed airship/airshipctl master: Introduce container interface https://review.opendev.org/675851 | 20:35 |
| openstackgerrit | Dmitry Ukov proposed airship/airshipctl master: Add stdin support to container interface https://review.opendev.org/676105 | 20:35 |
| openstackgerrit | Dmitry Ukov proposed airship/airshipctl master: Implement imagegen cli command https://review.opendev.org/676121 | 20:35 |
| openstackgerrit | Dmitry Ukov proposed airship/airshipctl master: Implement volume mount for container https://review.opendev.org/676141 | 20:35 |
| openstackgerrit | Dmitry Ukov proposed airship/airshipctl master: Add logic to image gen command https://review.opendev.org/676179 | 20:35 |
| *** alexanderhughes has joined #airshipit | 20:40 | |
| openstackgerrit | James Gu proposed airship/treasuremap master: Add airskiff_suse site https://review.opendev.org/675440 | 20:51 |
| *** alexanderhughes has quit IRC | 20:59 | |
| openstackgerrit | James Gu proposed airship/treasuremap master: Add new airskiff type https://review.opendev.org/674963 | 21:06 |
| openstackgerrit | Trung Thai proposed airship/porthole master: The script creates and sets up of user K8S profile https://review.opendev.org/674892 | 21:18 |
| *** spilla has quit IRC | 21:19 | |
| *** kranthikirang has quit IRC | 21:38 | |
| *** kranthikirang has joined #airshipit | 21:38 | |
| openstackgerrit | Dmitry Ukov proposed airship/airshipctl master: Introduce container interface https://review.opendev.org/675851 | 21:42 |
| openstackgerrit | Dmitry Ukov proposed airship/airshipctl master: Add stdin support to container interface https://review.opendev.org/676105 | 21:42 |
| openstackgerrit | Dmitry Ukov proposed airship/airshipctl master: Implement imagegen cli command https://review.opendev.org/676121 | 21:42 |
| openstackgerrit | Dmitry Ukov proposed airship/airshipctl master: Implement volume mount for container https://review.opendev.org/676141 | 21:42 |
| openstackgerrit | Dmitry Ukov proposed airship/airshipctl master: Add logic to image gen command https://review.opendev.org/676179 | 21:42 |
| *** kranthikirang has quit IRC | 21:42 | |
| openstackgerrit | Kaspars Skels proposed airship/treasuremap master: Enable back Nagios as part osh-infra monitoring https://review.opendev.org/677571 | 21:44 |
| openstackgerrit | Jenkins Uplifter proposed airship/treasuremap master: Auto chart/image uplift to latest https://review.opendev.org/677572 | 21:46 |
| *** rezroo has quit IRC | 21:50 | |
| *** rezroo has joined #airshipit | 21:50 | |
| *** StaceyF has quit IRC | 22:04 | |
| openstackgerrit | Anthony Bellino proposed airship/treasuremap master: Secret rotation and validation https://review.opendev.org/635694 | 22:04 |
| *** mbologna has quit IRC | 22:12 | |
| *** ab2434_ has quit IRC | 22:30 | |
| openstackgerrit | James Gu proposed airship/treasuremap master: Add airskiff_suse site https://review.opendev.org/675440 | 22:40 |
| *** DanCrank has quit IRC | 22:51 | |
| *** seaneagan has quit IRC | 23:07 | |
| *** sthussey has quit IRC | 23:42 | |
| *** AlexNoskov has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!