16:30:00 <bgilbert> #startmeeting fedora_coreos_meeting
16:30:00 <zodbot> Meeting started Wed Jun 12 16:30:00 2019 UTC.
16:30:00 <zodbot> This meeting is logged and archived in a public location.
16:30:00 <zodbot> The chair is bgilbert. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:00 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:05 <bgilbert> #topic roll call
16:30:07 <bgilbert> .hello2
16:30:10 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:30:10 <slowrie> .hello2
16:30:13 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:30:21 <lorbus> .hello2
16:30:21 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:30:23 <miabbott> .hello2
16:30:24 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:30:27 <dustymabe> .hello2
16:30:29 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:41 <ajeddeloh> .hello2
16:30:42 <zodbot> ajeddeloh: ajeddeloh 'Andrew Jeddeloh' <andrew.jeddeloh@redhat.com>
16:32:36 <kaeso[m]> .hello lucab
16:32:36 <zodbot> kaeso[m]: lucab 'Luca Bruno' <lucab@redhat.com>
16:33:59 <ksinny> .hello sinnykumari
16:34:00 <zodbot> ksinny: sinnykumari 'Sinny Kumari' <ksinny@gmail.com>
16:34:09 <bgilbert> #chair slowrie lorbus miabbott dustymabe ajeddeloh kaeso[m] ksinny
16:34:09 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe kaeso[m] ksinny lorbus miabbott slowrie
16:34:14 <bgilbert> #topic Action items from last meeting
16:34:17 <bgilbert> None!
16:34:35 <bgilbert> #topic Stream metadata generator
16:34:40 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/193
16:34:44 <jlebon> .hello2
16:34:45 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:34:52 <bgilbert> ksinny: go ahead
16:34:55 <bgilbert> #chair jlebon
16:34:55 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe jlebon kaeso[m] ksinny lorbus miabbott slowrie
16:35:01 <ksinny> ok
16:35:31 <ksinny> some context on stream metadata generator
16:35:57 <ksinny> stream metadata generator is a tool which gives stream metadata by taking input release index, release metadata and override file. We will get release index and release metadata from fcos artifacts location for corresponding streams.
16:36:19 <ksinny> We need to decide upon few things
16:36:30 <ksinny> 1. where should we keep override file? Should override file for all the streams be at same place or keep in separate branch?
16:37:10 <ksinny> I am thinking maybe we can make a separate project under coreos/ in github and keep it there?
16:37:32 <ksinny> something like stream-oveeride
16:37:41 <ksinny> err, stream-override
16:38:15 <bgilbert> jlebon and I discussed this at one point, and I wish I remember what we said
16:38:18 <dustymabe> ideally we'd namespace it - coreos/fedora-coreos-stream-override
16:38:25 <mnguyen_> .hello mnguyen
16:38:27 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:38:31 <bgilbert> #chair mnguyen_
16:38:31 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe jlebon kaeso[m] ksinny lorbus miabbott mnguyen_ slowrie
16:38:42 <ksinny> wfm, I am bad at naming things :)
16:38:58 <bgilbert> there's the override file as input, and there's the stream metadata as output.  both should have history.
16:39:06 <jlebon> or `fedora-coreos-stream`, and we also use it to keep history?
16:39:07 <bgilbert> in principle we could commit both to the same branch
16:39:13 <jlebon> bgilbert: :)
16:39:51 <bgilbert> or, like, master branch with all the override files, and then generated branches with the output stream metadata
16:39:52 <ksinny> I am in favor of having both stream and oveeride at same place
16:40:12 <bgilbert> ksinny, how are you envisioning that the tool would be run?  manually invoked job in CI?
16:40:24 <bgilbert> i.e., does the tool itself need to know about Git?
16:40:36 <bgilbert> or can a CI job handle the commit & push
16:41:00 <ksinny> bgilbert: I am writing as a separate tool in Go. We can run tie up later in Jenkins or any other way
16:41:27 <jlebon> at least for lockfiles, i was thinking of a separate CI job as well to handle the commit & push
16:41:42 <jlebon> could maybe share some code there
16:42:24 <bgilbert> AIUI nothing programmatic should be reading stream metadata from the Git repo
16:42:45 <bgilbert> but rather from the uploaded copy
16:43:01 <bgilbert> and similarly, nothing should be reading the override file except for the builder, which will look where it's pointed
16:43:17 <bgilbert> so I think we can do whatever's convenient, without much risk?
16:43:25 <ksinny> bgilbert: I think we should run it when release.json or releaseindex.json or override file gets updated
16:44:33 <bgilbert> ksinny: the point at which an FCOS release has occurred is exactly the moment where stream metadata is updated
16:44:38 <bgilbert> so I'd rather keep it manual at first
16:44:56 <bgilbert> can always automate
16:45:10 <ksinny> bgilbert: sure, we can keep it manual in the begining
16:45:12 <bgilbert> "manual" as in manually invoked
16:45:13 <bgilbert> +1
16:45:26 <bgilbert> (as in, probably still running in CI)
16:45:48 <ksinny> later on, by looking at timestamp change in json files should do that, isn't it?
16:45:53 <ksinny> yeah, in CI
16:46:11 <bgilbert> yeah, could do
16:47:19 <ksinny> that's all from me, I had one more question which I forgot. Might ask in Open Floor if I remember
16:47:40 <ksinny> anything to add on?
16:47:47 <bgilbert> jlebon: you haven't stated an opinion about branch structure
16:47:49 <bgilbert> any thoughts?
16:48:24 <jlebon> bgilbert: master for overrides and others for historical sound good to me to start
16:48:25 <dustymabe> i think i'd prefer to have a flat structure (single branch)
16:48:26 <bgilbert> (I'd probably bikeshed the repo name to `fedora-coreos-streams`)
16:49:05 <bgilbert> dustymabe: then there wouldn't be atomic updates, no?
16:49:25 <bgilbert> dustymabe: there'd be commit pairs: one manual one to update overrides file, then an auto one with the generated consequences
16:49:35 <bgilbert> with a push in the middle
16:50:14 <bgilbert> works technically, but feels a bit odd
16:50:22 <jlebon> yeah, it'd make git history noisier
16:50:41 <dustymabe> bgilbert: sounds like something we could massage - i.e. open PR against repo. bot detects PR and updates it with processed output, then merges
16:50:47 <bgilbert> hmm, yeah
16:50:55 <jlebon> to make things more obvious, we could rename `master` to `overrides`
16:50:59 <ksinny> separate branch for stream history and soverride sounds cleaner
16:51:04 <jlebon> and have no master branch at all
16:51:18 <bgilbert> well, thinking about dustymabe's point
16:51:31 <bgilbert> we need to be able to test override changes outside of prod
16:51:43 <bgilbert> possibly that means running the tool locally
16:51:49 <dustymabe> i honestly don't know which one is better, jlebon what I was thinking about was the structure of say https://pagure.io/fedora-comps (single branch) vs something like fedora-kickstarts (f29, f30, master)
16:51:57 <bgilbert> but it sure would be nice if we could PR a change and have CI show us the result before merging
16:53:35 <bgilbert> /shrug but yeah, ultimately I don't have a strong opinion
16:54:12 <jlebon> and the stream output from that bot would also be what we'd upload afterwards?
16:54:50 <jlebon> this essentially changes the definition of the repo from "historical" to "canonical"
16:54:55 <bgilbert> bleh
16:55:05 <ksinny> jlebon: yeah, should go to the fcos artifacts directory
16:55:23 <jlebon> (which isn't a bad thing, just calling it out)
16:55:34 <bgilbert> ksinny: his point is that the auto-updated PR _defines_ the new stream metadata
16:55:41 <bgilbert> rather than having the history just _record_ what was updated
16:55:55 <bgilbert> (as a side effect)
16:56:15 <bgilbert> yeah, good point.  proposal retracted.
16:56:18 <jlebon> i like the idea i think
16:56:22 <bgilbert> heh
16:56:25 <ksinny> ot it
16:56:39 <ksinny> got*
16:56:44 <bgilbert> jlebon: that implies that auto-generated updates would also go through PR though
16:56:53 <bgilbert> i.e. for new releases, the bot would PR and e.g. we'd manually merge
16:57:31 <bgilbert> which also seems fine, but is a different workflow
16:57:40 <bgilbert> the generator is then not responsible for stream metadata upload
16:57:45 <bgilbert> but that happens as a consequence of a PR merge
16:58:38 <bgilbert> we should probably move on soon, but, any preferences between those models?
16:59:04 <jlebon> could it be PR-based for the non-mechanical ones, and direct pushes for mechanical?
16:59:14 <bgilbert> yes, but seems confusing?
16:59:17 <jlebon> anyway, yeah we can discuss upstream
16:59:20 <bgilbert> upload happens at different times
16:59:46 <bgilbert> ksinny: want to take an action to summarize in ticket?
16:59:59 <ksinny> bgilbert: ack
17:00:13 <ksinny> I mean yes
17:00:23 <bgilbert> #action ksinny to summarize discussion in ticket
17:00:27 <bgilbert> and we can discuss further there
17:00:43 <jlebon> +1
17:00:44 <ksinny> +1
17:00:56 <bgilbert> #topic Release notes
17:00:58 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/194
17:01:10 <bgilbert> this is something we haven't really discussed at all
17:01:23 <bgilbert> presumably we will want release notes
17:01:45 <bgilbert> in CL, we never really found a way to do that properly
17:01:52 <bgilbert> the issue is that there are multiple versions in flight, and then they promote
17:02:18 <bgilbert> so if you get a new stable release, you now care what changes happened in the previous testing cycle
17:02:38 <bgilbert> but (if you're not running testing) you don't care about the current testing cycle
17:03:14 <bgilbert> any opinions on how this might work?  how did release notes work for AH?
17:04:15 <bgilbert> (also, unlike CL, we're inheriting a lot of changes.  there's a lot of content hidden inside a single line, "- Rebase onto Fedora 31"
17:04:16 <bgilbert> )
17:04:50 <bgilbert> jlebon, dustymabe, ksinny?
17:05:16 <ajeddeloh> Given that we don't share the same branching structure as CL, I wonder if release notes based on each branch would work.
17:05:20 <ksinny> In AH we have Two Week automated sent my releng to intended MLs. Also, we send a follow-up email with more details like AMI ID and important changes
17:05:49 <ajeddeloh> Like with CL you constantly hop branches, which is why our notes are basically unsuable, but that
17:05:53 <ksinny> We don't store them anywhere other than sending it to MLs
17:05:53 <ajeddeloh> s no longer the case
17:06:01 <dustymabe> bgilbert: for FAH we only send release notes for the stable releases
17:06:11 <slowrie> It might be nice to build something where we can essentially show the changes that have happened to the build as it's progressed to it's current point (using CL version terms essentially I want to see 1845.7's release notes and it shows me all notes from 1845.0 to the current point)
17:06:11 <bgilbert> ajeddeloh: isn't it?
17:06:15 <dustymabe> and the release notes are mostly automated information
17:06:31 <dustymabe> if something significant changes we'll add a paragraph about the change
17:06:31 <bgilbert> hmm, have a FAH example handy?
17:06:47 <dustymabe> bgilbert: it's not pretty, but it's something :)
17:06:49 * dustymabe finds link
17:06:58 <bgilbert> #link https://coreos.com/releases/
17:06:59 <bgilbert> on the CL side
17:07:09 <ksinny> bgilbert: https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2019-June/msg00000.html
17:07:31 <ksinny> and follow-up email https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2019-June/msg00002.html
17:07:32 <dustymabe> ksinny: right
17:07:39 <ksinny> these are from latest release
17:07:43 <kaeso[m]> does Fedora have a "web changelog" system for RPMs?
17:07:56 <dustymabe> we send two emails - planned to send them as one but we never got there
17:08:11 <jlebon> kaeso[m]: see https://github.com/coreos/fedora-coreos-tracker/issues/194#issuecomment-501368280
17:08:14 <dustymabe> kaeso[m]: i.e. given input set of rpms and output set show me the changelog ?
17:08:20 <bgilbert> okay, so always just a package version delta?
17:08:24 <bgilbert> there's never any "fixed xyz"?
17:08:26 <dustymabe> bgilbert: not always
17:08:27 <dustymabe> one sec
17:08:48 <ajeddeloh> bgilbert: unless I'm missing something we're merging changes into branches, not creating new ones constantly, right?
17:09:21 <kaeso[m]> jlebon: cool, like that, yes
17:09:21 <bgilbert> ajeddeloh: in Git, yes
17:09:34 <bgilbert> ajeddeloh: but that doesn't really affect the release note UX
17:09:49 <jlebon> kaeso[m]: here is what `rpm-ostree db diff --changelogs` gives me right now on my host: https://paste.fedoraproject.org/paste/07cZ6jArdJWHVLS7FQ0l9A
17:09:50 <bgilbert> ajeddeloh: you still have to care about the previous testing release
17:10:01 <kaeso[m]> dustymabe: I was thinking of permalinks for changelog for invidual packages, between "old" and "new" versions
17:10:20 <ksinny> jlebon: nice
17:10:21 <dustymabe> #link https://lists.fedoraproject.org/archives/list/rel-eng@lists.fedoraproject.org/message/5WBUM4XTZA7FZQZ3SVDH5GMWRH6472H2/
17:10:35 <dustymabe> ^^ example where I added notes about security update
17:10:56 <bgilbert> jlebon: that's really nice
17:11:38 <bgilbert> CL's release notes had a mix of things
17:11:45 <ajeddeloh> bgilbert: ah, I was just thinking that within a branch we can look at commits and get a set of changes that is actually what the user expects, independent of how we present that
17:11:47 <dustymabe> we used to make more effort to try to summarize relevant changes (i.e. podman version xyz with new features abc)
17:11:59 <bgilbert> ajeddeloh: hmm, maybe
17:12:05 <kaeso[m]> my initial question was for something URL-linkable, but a static "db diff" that we publish on each release would work too
17:12:12 <dustymabe> but since we're not focusing on AH any longer we've stopped that mostly
17:12:15 <bgilbert> in CL: there are package updates (for major packages; we hide minor ones) where we just link to the upstream changelogs
17:12:37 <ksinny> dustymabe: right
17:12:40 <bgilbert> if there was a change in the Gentoo packaging but not a new upstream release, that'd generally get lost
17:12:55 <bgilbert> and then there are high-level issues of which we're directly aware: CVEs and bug fixes
17:13:17 <bgilbert> so that's already sort of a broken system: issues are anywhere from surfaced to completely buried depending on where/how they were fixed
17:13:43 <bgilbert> (e.g. if we fixed a problem in Ignition upstream it was buried in the Ignition update)
17:14:27 <bgilbert> for FCOS the db diff stuff seems nice for package changes, better than CL
17:14:32 <kaeso[m]> from CL release notes, I like the quick highlighting of very few selected components (kernel, systemd, ignition)
17:14:35 <jlebon> i think it'd be really cool to make it mostly automatically generated, with a tiny header "intro" section for specific things we want to manually call out, e.g. "This release fixes zombie-melted-down-sheep-doom."
17:14:40 <bgilbert> but if we fix cosa or fcos-config...
17:15:10 <slowrie> jlebon: +1
17:15:21 <bgilbert> jlebon: +1
17:15:24 <ajeddeloh> kaeso[m]: I agree
17:15:35 <bgilbert> can we automatically pull out all CVEs?
17:15:40 <bgilbert> do we need to regex the rpm changelogs?
17:15:45 <jlebon> if it's attached to the errata, yes
17:16:07 <jlebon> it shows up as a structured entry in the updateinfo xml
17:16:11 <bgilbert> nice
17:16:31 <jlebon> well, the errata does, for CVE ids... we do specifically grep for those right now :|
17:16:47 <bgilbert> kaeso[m]: I like that too
17:17:18 <bgilbert> kaeso[m]: do you think it'd be sufficient for the automated generation to surface particular packages?
17:17:56 <bgilbert> kaeso[m]: i.e., without us manually writing notes for those pkgs
17:18:17 <kaeso[m]> I would guess so
17:18:55 <bgilbert> okay, so in summary, automatically generated CVE list, automatically generated pkg diff, section for manually listing fixes as appropriate
17:19:28 <kaeso[m]> yes. My typical usage is "quick check that table when somebody wants to use feature $foo which is only in latest version of $bar"
17:20:06 <jlebon> for a select few pkgs, wonder if we could automate linking to upstream release notes
17:20:16 <jlebon> for the github ones it shouldn't be too hard
17:20:27 <jlebon> for the kernel, not sure
17:20:28 <bgilbert> jlebon: +1
17:20:39 <bgilbert> I do think we should have a historical archive of relnotes
17:20:43 <bgilbert> is it important to mail them to a list?
17:21:05 <bgilbert> want to keep the noise level down
17:21:33 <jlebon> we could have a separate list for releases
17:21:59 <slowrie> Having something like the coreos website where you can select specific versions would be nice, if we want to also mail them to a list that's fine by me
17:22:34 <bgilbert> I was thinking RSS/Atom but those are rumored to be deceased
17:22:52 <bgilbert> ~~tooling assumptions~~
17:22:58 <bgilbert> we could tweet :-)
17:23:56 <kaeso[m]> as long as we have something that can be permalinked (ML archives, fixed URL, etc) I'm fine
17:24:05 <bgilbert> +1
17:24:20 <bgilbert> #action bgilbert to summarize discussion in ticket
17:24:26 <bgilbert> anything else on this topic?
17:24:43 <jlebon> yeah no strong opinion. whatever we decide on though, we shouldn't require an account at $WEBSITE to get notifications
17:24:48 <bgilbert> +1
17:25:27 <bgilbert> #topic Open Floor
17:26:20 <dustymabe> #info streams and stream tooling progress: https://github.com/coreos/fedora-coreos-config/issues/100
17:26:43 <jlebon> dustymabe: heh thanks, was about to bring that up :)
17:27:03 <jlebon> would appreciate input on that ticket, esp. if you think there's a cleaner way forward
17:27:11 <dustymabe> #info signing project proposal was created for fedora infra: https://github.com/coreos/fedora-coreos-tracker/issues/187#issuecomment-499999387
17:27:25 <dustymabe> ^^ looks like we have to wait on them to come back on that
17:27:30 <bgilbert> dustymabe++
17:27:41 <dustymabe> i've been told they are a bit overloaded with work right now
17:28:01 <kaeso[m]> #info updates(.stg)coreos.fedoraproject.org progress https://pagure.io/fedora-infrastructure/issue/7825#comment-574361
17:29:04 <bgilbert> okay, closing out in 30s
17:29:34 <bgilbert> #endmeeting