meeting
LOGS
13:00:44 <mvollmer> #startmeeting meeting
13:00:44 <zodbot> Meeting started Mon May  8 13:00:44 2017 UTC.  The chair is mvollmer. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:44 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
13:00:44 <zodbot> The meeting name has been set to 'meeting'
13:00:49 <mvollmer> .hello mvo
13:00:50 <zodbot> mvollmer: mvo 'Marius Vollmer' <marius.vollmer@gmail.com>
13:01:14 <dperpeet> .hello dperpeet
13:01:15 <zodbot> dperpeet: dperpeet 'None' <dperpeet@redhat.com>
13:01:19 <pitti> .hello martinpitt
13:01:20 <zodbot> pitti: martinpitt 'Martin Pitt' <martin@piware.de>
13:01:43 <andreasn> .hello andreasn
13:01:44 <zodbot> andreasn: andreasn 'Andreas Nilsson' <anilsson@redhat.com>
13:02:14 <garrett> .hello garrett
13:02:16 <zodbot> garrett: garrett 'Garrett LeSage' <garrett@lesage.us>
13:02:42 <mvollmer> #topic Agenda
13:03:29 <dperpeet> * image pruning on servers / distributed storage
13:04:16 <mvollmer> * application manager, next steps
13:05:33 <mvollmer> okay, let's start
13:05:40 <mvollmer> #topic  image pruning on servers / distributed storage
13:05:52 <dperpeet> ok, so we've run out of space on fedorapeople (again)
13:06:08 <dperpeet> since we carry more than one branch now, our pruning has become a bit more complicated
13:06:21 <dperpeet> by pruning I mean the process of cleaning up unused images
13:06:27 <dperpeet> for our integration testing
13:06:56 <dperpeet> we've recently patched our vm-download process to be more careful when pruning (by considering branches other than master)
13:07:08 <dperpeet> right now I'm working on a pr to do the same for our vm-upload pruning
13:07:31 <dperpeet> I'm implementing a staggered approach, whereas right now we delete images that aren't used and are older than 30 days
13:07:51 <stefw> i've been working on distributing storage
13:07:56 <dperpeet> once we look at all pull requests and branches, we can be a lot more aggressive and don't need to wait for 30 days
13:07:57 <stefw> today i've been working on bittorrent for that reason
13:08:06 <stefw> in addition i've provisioned 300G of storage on verifymachine5
13:08:06 <dperpeet> over to stef
13:08:17 <stefw> and added an http server to the standard cockpit/test stuff
13:08:23 <stefw> so any cockpit-test instance can share its images
13:08:35 <dperpeet> is that storage persistent?
13:08:38 <stefw> so hopefully between all these things we'll solve the problem
13:08:40 <stefw> it is persistent
13:08:43 <dperpeet> nice
13:08:51 <stefw> but i think it could have an outage and be gone at some point
13:08:57 <stefw> hence my work on bittorrent
13:09:07 <stefw> where basically all the cockpit/tests containers talk to each other
13:09:10 <stefw> and just plain share images
13:09:19 <stefw> a bit longer term, but i think that's the end goal
13:09:23 <dperpeet> yes
13:09:30 <stefw> the images have been one painful part where we haven't been properly distributed and scaleable
13:09:32 <stefw> and that would change that
13:10:03 <stefw> FWIW, when you look at the effects of the tasks that are done by these automated bots
13:10:09 <dperpeet> we can iron out the technical details (tracker et cetera), but I like the direction
13:10:18 <stefw> dperpeet, ++
13:10:27 <stefw> DHT vs tracker ... and behind firewall nodes
13:10:31 <stefw> trending toward zero configuration
13:10:45 <stefw> maybe just one or two more well know addresses of nodes in order to bootstrap things
13:10:52 <stefw> lets talk about that more
13:10:53 <dperpeet> we may also want to scale logs
13:11:02 <dperpeet> at the same time
13:11:06 <dperpeet> if possible
13:11:08 <stefw> i worked a bit on that
13:11:14 <stefw> and adding a cockpit/sink container
13:11:20 <stefw> which can be run in multiple places
13:11:29 <stefw> technically we have scaling going on there
13:11:34 <stefw> it's just not in use
13:11:39 <stefw> everything gravitated towards fedorapeople.org
13:11:44 <stefw> but yes, i think we can find a solution there too
13:12:20 <stefw> on a meta level: So one of the things that's interesting here
13:12:26 <dperpeet> even if we do have backups, the one place that's a bit "static" is github
13:12:35 <stefw> yeah, but that mirrors our project
13:12:39 <dperpeet> yup
13:12:47 <stefw> we really do depend on these bots as team members
13:12:53 <stefw> and the fact that they're organic and self-organizing is important
13:13:23 <stefw> when you get away from the implementation details
13:13:27 <stefw> of how the bots work
13:13:34 <stefw> and look at the effect of what they pull off it's pretty amazing
13:13:44 <dperpeet> yes, I agree
13:13:48 <stefw> from the massive coordination of bots around testing a pull request, to the way they submit changes for review, and then other bots come in and check it out.
13:13:54 <stefw> to the way they do releases
13:14:16 <stefw> anyway ... a bit of a side track there ... but it's pretty amazing
13:14:31 <dperpeet> cockpituous is our #7 contributor on github :)
13:14:38 <stefw> yeah, i'm not surprised
13:15:25 <dperpeet> is there any part you think we should discuss now?
13:15:32 <stefw> i don't think so
13:15:43 <dperpeet> I think we can keep chipping away at this piece by piece
13:15:44 <dperpeet> as we have been
13:15:47 <stefw> i'd like to talk about this with whoever is interested
13:15:55 <stefw> but yeah, agree on the direction, and then chip away on it bit by bit
13:15:55 <dperpeet> I am obviously
13:16:15 <stefw> i also think i know why others can't just take our bots and duplicate what they do, use them elsewhere
13:16:32 <dperpeet> one thing to take away for contributors: if a PR becomes too old, it may take less time in the future for images to get deleted
13:16:34 <stefw> i think we want to make them easier, but i've figured out why that's really hard at a meta level
13:16:46 <mvollmer> do tell!
13:18:05 <stefw> who should tell?
13:18:26 <mvollmer> stefw, you said " but i've figured out why that's really hard at a meta level"
13:18:36 <stefw> ok ... but don't laugh
13:18:40 <dperpeet> kind of a cliff-hanger, I'm waiting here myself
13:18:41 <stefw> take a step away from the implementation for a second
13:18:48 <stefw> we've essentially built simple bot team members
13:18:58 <stefw> mvollmer built parts of that when doing the image creation PR+images stuff
13:19:03 <stefw> we built them a bit self-organizing
13:19:08 <stefw> and helped them to grow over time
13:19:12 <stefw> even if someone copies them as is
13:19:18 <stefw> they're sorta copying something that's alive and part of the team
13:19:25 <stefw> even if they manage to copy it as is, the two will diverge
13:19:33 <stefw> more likely they can copy parts of the components of such a bot
13:19:37 <stefw> but copying it wholesale is really hard
13:19:43 <stefw> sorta like cloning life forms is hard
13:20:01 <dperpeet> well, that's because what we call the "bot" is actually a bunch of stuff spread all over
13:20:10 <dperpeet> we've distilled a lot into cockpituous
13:20:12 <mvollmer> yeah, most of the testing framework feels like that to me
13:20:16 <dperpeet> but some core logic still remains in cockpit
13:20:28 <stefw> if you step away from the implementation ... and try and explain this even to technical people not familiar with the details
13:20:33 <stefw> how we rely on these bots, and what they do
13:20:37 <stefw> you end up coming to this conclusion
13:20:46 <stefw> without them the team stops, we rely on them to do work
13:20:47 <stefw> etc.
13:21:03 <stefw> so i am cleaning up parts of their implementation
13:21:18 <pitti> it's the distilled wisdom and best practice of many man-years, no wonder it's hard to explain in an elevator pitch :)
13:21:21 <dperpeet> cleaning up round #X :)
13:21:42 <dperpeet> I think we got a cleanup every time the team grew
13:21:52 <stefw> one of their big amazing aspects is that each of us humans on the team can modify the bots
13:21:54 <stefw> try some change out
13:22:03 <stefw> and then if it works, that change goes to them all
13:22:10 <mvollmer> i wouldn't say our bots are super sophisticated, they are just super specific
13:22:18 <stefw> but what they do is sophiscticated
13:22:20 <stefw> the implementation is not
13:22:21 <pitti> "devops"
13:22:28 <stefw> but that fundamental principle allows them to grow properly
13:22:33 <mvollmer> the Atomic guys have similar bots, with different sophistication
13:22:39 <stefw> and at the same time makes them really hard for someone else to take and copy and bring over somewhere else
13:22:41 <stefw> mvollmer, exactly
13:22:57 <stefw> and i'm coming to grips with the fact that i don't think that's a flaw necessarily
13:23:08 <mvollmer> we could say our bots are ..... proprietary
13:23:14 <stefw> so are you
13:23:18 <stefw> you are proprietory
13:23:19 <stefw> ary
13:23:35 <dperpeet> this discussion has become a bit metaphysical
13:23:35 <stefw> their componenst are not, but the bots as composed are really just a part of the project
13:23:48 <stefw> yup
13:24:50 <stefw> anyway, when people say automation is even taking over software engineering ... this is exactly what they mean ... it may look tedious from the details ... but the effect of what's happening in our project is exactly that
13:24:59 <dperpeet> essentially, we can't generalize them too much for others, because the specialization/generalization is something that didn't happen by accident
13:25:04 <stefw> yeah
13:25:19 <stefw> anyway ... that was fun :)
13:25:30 <dperpeet> cup of robots!
13:25:58 * mvollmer pokes the robot to change the topic
13:26:12 <mvollmer> #topic  application manager, next steps
13:26:38 <mvollmer> so, I think I have the basic pieces in place
13:26:44 <mvollmer> and have to decide what to work on next
13:27:01 <mvollmer> let me dig out the video link...
13:27:10 <andreasn> #link https://www.youtube.com/watch?v=Pmi6AJanRtk
13:27:15 <mvollmer> thanks
13:27:24 <mvollmer> those are the baosc pieces
13:27:40 <andreasn> really nice demo!
13:27:43 <mvollmer> i think we could release a package with metainfo to Fedora and have it all work
13:27:54 <mvollmer> but there is lots to optimize
13:28:07 <mvollmer> and the spec needs to be officially updated
13:28:11 <mvollmer> the appstream spec
13:28:29 <mvollmer> and we need to make dynamic manifests work
13:28:38 <andreasn> I have some ideas on how the design can be improved in some places, but this is a great start!
13:28:42 <mvollmer> I think I'll try to do all this in parallel
13:28:56 <pitti> mvollmer: so that "Demo application" is just a normal RPM with the extra appstream metadata?
13:29:05 <mvollmer> pitti, yes
13:29:15 <pitti> nice! (I haven't watched it to the end yet)
13:29:37 <mvollmer> i need to figure out how Debian handles appstream metadata
13:30:13 <mvollmer> and talk to richard and kalev and maybe get a server version of the fedora appstream meta data
13:30:29 <mvollmer> otherwise we'll have to install all of the GNOME and KDE meta data on the server
13:30:50 <mvollmer> but if we do that, things should just work
13:31:43 <mvollmer> one more track to work on: the actual FreeIPA application
13:31:49 <pitti> mvollmer: so there's an appstream metadata "app type" thing that says "cockpit"?
13:32:08 <pitti> (or "gnome", "kde", or other categories)
13:32:09 <mvollmer> I think once we have that in shape, we can consider actually releasing the Application Manager
13:32:30 <mvollmer> pitti, right now I am using the new "launchable" meta data
13:32:52 <mvollmer> any package that has a "systemd-unit" or "cockpit-package" launchable will be shown by Cockpit
13:33:14 <mvollmer> the system-unit launchable needs a new component type "service"
13:33:36 <mvollmer> cockpit-package launchable goes with a "addon" component
13:33:44 <mvollmer> that <extends> cockpit
13:34:05 <mvollmer> needs to be discussed and formally specced with the appstream people
13:34:21 <mvollmer> but the basic ideas have been agreed upon
13:35:43 <mvollmer> pitti, on fedora, the appstream metadata comes in a RPM itself: "appstream-data"
13:35:51 <mvollmer> pitti, do you know how that works on Debian?
13:36:07 <mvollmer> i think apt-get might download it as part of the repo metadata, actually
13:36:13 <mvollmer> anyway, I'll figure that out
13:36:33 <pitti> mvollmer: not precisely; I thought it was encoded in the Packages.gz metadata somehow
13:36:34 <mvollmer> fedora needs to be improved, likely.  Not just for us, but in general
13:36:49 <pitti> http://dep.debian.net/deps/dep11/
13:36:58 <mvollmer> pitti, they have a yaml file next to packages.gz, I think
13:37:06 <mvollmer> question is, who downloads it?
13:37:14 <mvollmer> but, I'll get there
13:37:31 <pitti> I thought e. g. gnome-software would
13:37:36 <pitti> not apt itself
13:37:40 <mvollmer> maybe
13:37:55 <mvollmer> maybe apt helps to construct the urls
13:38:05 <mvollmer> but let's not guess
13:38:38 <pitti> http://appstream.ubuntu.com/ too
13:39:24 <pitti> but these are just some pointers I know about, TBH I've never looked into the details
13:40:56 <mvollmer> Thanks!
13:41:16 <mvollmer> okay,done?
13:41:45 <andreasn> think so
13:42:30 <mvollmer> okay!
13:42:33 <mvollmer> #endmeeting