16:00:09 <Zlopez[m]> #startmeeting Infrastructure (2021-04-29)
16:00:09 <zodbot> Meeting started Thu Apr 29 16:00:09 2021 UTC.
16:00:09 <zodbot> This meeting is logged and archived in a public location.
16:00:09 <zodbot> The chair is Zlopez[m]. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:09 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:09 <zodbot> The meeting name has been set to 'infrastructure_(2021-04-29)'
16:00:10 <Zlopez[m]> #meetingname infrastructure
16:00:10 <zodbot> The meeting name has been set to 'infrastructure'
16:00:17 <Zlopez[m]> #chair nirik smooge siddharthvipul mobrien zlopez pingou bodanel dtometzki
16:00:17 <zodbot> Current chairs: Zlopez[m] bodanel dtometzki mobrien nirik pingou siddharthvipul smooge zlopez
16:00:18 <Zlopez[m]> #info Agenda is at: https://board.net/p/fedora-infra
16:00:26 <dtometzki> .hi
16:00:27 <zodbot> dtometzki: dtometzki 'Damian Tometzki' <linux@tometzki.de>
16:00:30 <Zlopez[m]> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:31 <mobrien> .hi
16:00:31 <dtometzki> hello
16:00:31 <zodbot> mobrien: mobrien 'Mark O'Brien' <markobri@redhat.com>
16:00:38 <Zlopez[m]> #topic aloha
16:00:45 <Zlopez[m]> .hello zlopez
16:00:45 <zodbot> Zlopez[m]: zlopez 'Michal Konečný' <michal.konecny@psmail.xyz>
16:00:56 <Zlopez[m]> Welcome everyone to today show
16:01:07 <Zlopez[m]> Sit tightly and enjoy the view
16:01:10 <darknao> .hi
16:01:11 <zodbot> darknao: darknao 'Francois Andrieu' <naolwen@gmail.com>
16:01:15 <jednorozec> .hello humaton
16:01:16 <zodbot> jednorozec: humaton 'Tomáš Hrčka' <thrcka@redhat.com>
16:01:19 <samyak-jn> .hello jnsamyak
16:01:20 <zodbot> samyak-jn: jnsamyak 'Samyak Jain' <samyak.jn11@gmail.com>
16:01:35 <dtometzki> hi
16:01:50 <Zlopez[m]> Bring your favorite snacks and drink, this will be a ride :-)
16:02:05 <Zlopez[m]> Let's start with new faces
16:02:07 <nirik> morning
16:02:14 <Zlopez[m]> #topic New folks introductions
16:02:15 <Zlopez[m]> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:02:15 <Zlopez[m]> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:02:16 * nirik has to go grab coffee, back in a sec
16:02:31 <Zlopez[m]> Any new face around?
16:02:32 <samyak-jn> Zlopez[m]: I have coffee in hand :)
16:03:14 <Zlopez[m]> freenode_jnsamyak[m]: Good for you :-)
16:03:19 <Strider83> This is my second meeting
16:03:33 <Zlopez[m]> Good to see you Strider83
16:03:42 <mobrien> Welcome Strider83
16:03:49 <Strider83> I sent an into to the mailing list, My name is Derek
16:04:17 <samyak-jn> Oh yes I read that mail, Welcome Strider83!
16:04:39 <dtometzki> welcome Strider83
16:04:54 <Strider83> I also nervously typo'd my linked in address last week in this meeting:)   https://www.linkedin.com/in/derekhanks
16:05:06 * samyak-jn wonders why I haven't sent an introductory mail, will do soon (I'm a month late :P)
16:05:31 <Zlopez[m]> Strider83: Do you have anything specific you are interested in Fedora Infra?
16:06:39 * nirik is back
16:07:04 <Zlopez[m]> nirik: welcome back to our show
16:07:16 <Strider83> I am an infrastructure guy by experience and want to learn more.  I need to work on skills like python, ansible, k8s, etc.  Fedora is my favorite distro, I have been using it since FC1, I need to run kali for an ethical hacking class right now, so it reminds me of what debian is like :P
16:07:50 <Strider83> So security is my main focus right now
16:07:54 <Zlopez[m]> We have plenty of python. ansible, k8s in our infra
16:08:10 <samyak-jn> <Strider83 "So security is my main focus rig"> Nice!
16:08:39 <Strider83> Sweet, I am also interesting to pick up other marketable skills, but I am wondering on a masters degree, CISSP, ethical hacking cert, forensics, etc...
16:08:46 <Strider83> *working*
16:09:15 <Strider83> I was the puppet guy at a big bank, but I am more of a generalist
16:09:38 <Zlopez[m]> It looks like you are really into security
16:09:45 <dtometzki> Strider83, very intresting cool
16:10:20 <mobrien> Strider83: we can always use security expertise :)
16:10:20 <Zlopez[m]> Security guys are always welcome in Fedora infra
16:10:52 <Zlopez[m]> Let's move to next topic, we can continue in this discussion in Open Floor
16:10:54 <Strider83> kk, that is the best place for me to focus at least for the next year.  Most infra seems to be migrating to the cloud.
16:11:24 <Zlopez[m]> Who will be our next host in this show?
16:11:25 * Zlopez[m] < https://matrix.org/_matrix/media/r0/download/matrix.org/lYkGwuiSFBAMlJIbRYShAANR/message.txt >
16:11:40 <Zlopez[m]> #topic Next chair
16:11:40 <Zlopez[m]> #info magic eight ball says:
16:11:46 <Zlopez[m]> #info chair 2021-04-29 - zlopez
16:11:46 <Zlopez[m]> #info chair 2021-05-06 - dtometzki
16:11:50 <Zlopez[m]> #info chair 2021-05-13 - jnsamyak
16:12:29 <Zlopez[m]> We are pretty covered, does anybody wants to take the show at 2021-05-20?
16:12:58 <Zlopez[m]> We can leave it empty for now and let's revisit this next week
16:13:06 <nirik> sounds good.
16:13:36 <Zlopez[m]> Next on our menu are announcements
16:13:43 <Zlopez[m]> And everybody wants to hear them
16:13:51 <Zlopez[m]> #topic announcements and information
16:13:57 <Zlopez[m]> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:14:03 <Zlopez[m]> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:14:11 <Zlopez[m]> #info work is being done on getting COPR proper powerPC systems
16:14:15 <Zlopez[m]> #info final freeze ended
16:14:22 <Zlopez[m]> #info Fedora 34 is out! Install/upgrade today!
16:14:44 <jednorozec> yesterday was too late for the upgrade!
16:14:44 <dtometzki> fedora 34 it works great
16:14:48 <Zlopez[m]> Congrats everybody on new Fedora release :-)
16:15:14 <Zlopez[m]> I'm on Fedora 34 Silverblue till beta and it works great
16:15:26 <dtometzki> yeah
16:15:30 <Zlopez[m]> So anything else I missed on my list?
16:17:13 <nirik> The best day to upgrade was yesterday, but the next best day is today. :)
16:17:29 <dtometzki> :-)
16:17:32 <Zlopez[m]> Every day is a best day for upgrade :-)
16:17:53 <Zlopez[m]> Let's move to the next part of our show
16:17:59 <Strider83> I started having nvidea/dell second monitor problems after my upgrade.  I will have to put that problem on hold for the next few months
16:18:14 <Zlopez[m]> Who will be the friend on phone for next weeks?
16:18:21 <Zlopez[m]> #topic Oncall
16:18:21 <Zlopez[m]> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:18:26 <Zlopez[m]> #info mobrien is on call for 2021-04-22 to 2021-04-29
16:18:26 <Zlopez[m]> #info jednorozec is on call for 2021-04-29 to 2021-05-06
16:18:32 <Zlopez[m]> #info dtometzki is on call for 2021-05-06 to 2021-05-13
16:19:07 <Zlopez[m]> #info ??? is on call for 2021-05-13 to 2021-05-20
16:19:44 <Zlopez[m]> Does anybody wants to be a friend on phone (or in our case on IRC chat) for 2021-05-13 to 2021-05-20?
16:19:56 <darknao> i can take that one
16:20:07 <Zlopez[m]> Sold!
16:20:27 <Zlopez[m]> #info Summary of last week: (from current oncall )
16:20:38 <mobrien> Quiet week
16:21:02 <Zlopez[m]> We were still in freeze, so it will probably be much less quiet this week
16:21:11 <Zlopez[m]> #topic Monitoring discussion [nirik]
16:21:11 <Zlopez[m]> #info https://nagios.fedoraproject.org/nagios
16:21:12 <mobrien> one ping, nirik took care of it as it was late my time, I think it was a case off password instead of pass+otp
16:21:16 <Zlopez[m]> #info Go over existing out items and fix
16:21:33 <nirik> so not much different here
16:21:44 <nirik> we have some machines that are down waiting on a on-site visit to sort out.
16:21:58 <Zlopez[m]> mobrien:  I always forgot about OTP
16:22:09 <nirik> the registry boxes need more space. we can do that now that we are out of freeze.
16:22:47 <mobrien> Zlopez: ya, it happens a lot. I think there is work in some places to add a separate field for it on the sign in
16:22:58 <nirik> and just some random small things that need people to fix them.
16:23:07 <nirik> thats about it, we cna move on
16:23:18 <Zlopez[m]> Damn, we still need people :-D
16:23:47 <Zlopez[m]> And today our learning topic will be presented by special guest
16:23:53 <Zlopez[m]> #topic Learning topic
16:23:53 <Zlopez[m]> #info Flatpak build chain in Fedora [otaylor] on 2021-04-29
16:24:09 <otaylor> Hi everybody
16:24:20 <dtometzki> hi otaylor
16:24:20 <nirik> hurray. welcome otaylor
16:24:24 <Zlopez[m]> otaylor:  The floor is yours
16:24:49 <otaylor> I'm Owen Taylor - I work on the desktop team at Red Hat, and mostly have been focusing on building Flatpaks out of RPMs for the last few years
16:25:34 <otaylor> Kalev Lember (kalev) is also here - he has been doing the work to maintain our Flatpak runtime images (which application Flatpaks are based on) and keep our applications up to date - so he's #1 user of the Flatpak infrastructure
16:26:31 <otaylor> I can't remember doing an IRC talk before, so this is a bit new ground to me :-) .... please feel free to interrupt and ask questions - this will be more interesting if it's interactive!
16:26:45 <otaylor> The project to add Flatpak support in Fedora started in the summer of 2017, and went live at the end of 2018, so we've had a couple of years of experience with it now.
16:27:04 <otaylor> Flatpaks are graphical applications shipped independently from the base image. Originally, Flatpaks were based on ostree, not just for local storage, but as a transport. For the Fedora (and RHEL), we implemented support upstream for storing Flatpaks in a container registry and downloading them as Docker/OCI images, so that we could handle them more uniformly with server-side containers.
16:28:03 <otaylor> Here's the packager view of how you build a Flatpak:
16:28:19 <otaylor> * create a module (fedmod rpm2flatpak)
16:28:28 <otaylor> * build the module in MBS (fedpkg module-build)
16:28:34 <otaylor> * build a container out of the package in OSBS (fedpkg flatpak-build)
16:28:40 <otaylor> * file an update in Bodhi (pushes to the testing Flatpak remote oci+https://registry.fedoraproject.org#testing)
16:28:46 <otaylor> * mark the update stable (pushes to the stable Flatpak remote oci+https://registry.fedoraproject.org)
16:29:39 <Zlopez[m]> otaylor:  Do we support beta branches for flatpaks?
16:29:43 <otaylor> Packager docs are at https://docs.fedoraproject.org/en-US/flatpak/
16:30:35 <otaylor> Zlopez[m]: You mean, like having a separate build of an application from a nightly or beta upstream release?
16:31:49 <Zlopez[m]> Like having some test branch other than stable similar to Flathub
16:31:55 <otaylor> Zlopez[m]: that's not something we support currently, but we could potentially do so - maybe as a separate Bodhi release (so, F34 Unstable Flatpaks) - the UI wouldn't be that clear though - you'd need to use the Flatpak command line to switch back and forth.
16:32:30 <Zlopez[m]> For flathub it's just different remote
16:32:37 <otaylor> Zlopez[m]: The main thing that we have is a testing remote for Flatpaks in the testing state in Bodhi
16:33:43 <Zlopez[m]> Thanks for info, I didn't knew that we have special testing remote for flatpaks in testing state
16:33:49 <Zlopez[m]> Please continue
16:33:58 <otaylor> Zlopez[m]: I think there would be some confusion if we had something separate from that, but I think first we need to get Fedora packagers actively interacting with Flatpak packaging :-)
16:34:04 <otaylor> We have about 80 applications in Flatpak form in Fedora, and that's been pretty flat over the last 18 months - new applications aren't being added very quickly.
16:34:23 <otaylor> The remotes stable and testing are a layer on top of the Fedora registry - the Flatpak client asks for, say:
16:34:30 <otaylor> https://registry.stg.fedoraproject.org/index/static?annotation:org.flatpak.ref:exists=1&os=linux&architecture=amd64&tag=latest
16:34:45 <otaylor> And that serves up a JSON index of all the Flatpaks on the registry. It looks like a database query, but it's actually just HTTP requests going to a statically generated index.
16:35:14 <otaylor> The software behind the index is flatpak-indexer running in os.fedoraproject.org (https://github.com/owtaylor/flatpak-indexer)
16:35:31 <otaylor> A big recent change behind the scenes is that the index now includes pointers to delta information, so if you are updating to the latest version of the flatpak-runtime image, the index contains a pointer to:
16:35:37 <otaylor> https://cdn.registry.fedoraproject.org/deltas/4b/3de520dcfb9d2564df4add4fabcbea7e94404a118b0c23848182e49c001887.json
16:35:51 <otaylor> And instead of downloading 657 MB from scratch, depending on what you are updating from you download 44MB, or 18MB, or 5MB. Behind the scenes this uses bsdiff, so it even can efficiently compress changes to single files.
16:36:14 <otaylor> (https://blogs.gnome.org/alexl/2020/05/13/putting-container-updates-on-a-diet/)
16:36:35 <otaylor> I'm also excited by the debuginfod work that is going on, because that addresses another problem we had with Fedora Flatpaks - there was no way to debug them or get a backtrace.
16:36:47 <otaylor> Hopefully we can eventually leverage this for ABRT support for Flatpaks as well.
16:37:12 <samyak-jn> otaylor, is their anywhere documented for newcomer to help in flatpak packaging, can you please point me to the links?
16:37:15 <Zlopez[m]> otaylor:  Is there any plan for ABRT in Silverblue?
16:37:53 <Zlopez[m]> freenode_jnsamyak[m]: https://docs.fedoraproject.org/en-US/flatpak/tutorial/
16:38:14 <samyak-jn> Thanks, Zlopez[m]!
16:38:19 <Zlopez[m]> This is a tutorial that should help you with first steps
16:38:42 <otaylor> samyak-jn:   there's also a page https://fedoraproject.org/wiki/Flatpak:Easy - that hasn't been updated for a long time, but we also haven't gotten a lot of new Flatpaks so it might still be useful
16:38:49 <nirik> Doesn't gnome-builder also have some flatpak support now? or I could be misremembering.
16:39:22 <otaylor> nirik: it does, but not useful for Flatpaks-in-Fedora - that's about building directly from source with a JSON manifest
16:39:23 <samyak-jn> Will check both links, thanks otaylor.
16:39:35 <nirik> ah ok, fair enough
16:40:28 <otaylor> nirik: Someday, being able to build a Flatpak of an application without building an RPM first could be cool to support (but maybe still being able to bundle libraries packaged as RPMs) but that's nowhere close
16:40:43 <Strider83> How does flatpatck compare to snap?
16:41:55 <otaylor> Strider83: big question - I'd say that Flatpak is laser-focused on the desktop, so has better ergonomics there. It's also fundamentally an ecosstem rather than requiring a central snap store
16:41:55 <Zlopez[m]> Strider83:  From what I read, the snap allows to package even CLI apps, but doesn't support sandboxing that great as flatpak
16:42:23 <otaylor> Strider83: so, you can have both Flathub and Fedora Flatpaks
16:42:43 <Zlopez[m]> And even snaps :-)
16:43:41 <otaylor> Strider83: It's pretty fuzzy with where snap ends - it is more about fundamentally replacing how you do packaging *everything* in the distro - servers, cli apps, etc. while with Flatpaks we want to stick with things that the user thinks of as an application
16:43:44 <nirik> so for now, is kalev basically building/updating all those flatpaks? or do some have maintainers doing that work?
16:44:48 <otaylor> nirik: for now, kalev is basically building/updating all those flatpaks. It's not clear how we get to better automation - Freshmaker was a project to do rebuilds as part Factory 2.0, but it never was deployed in Fedora and has a pretty limited role internally in Red Hat.
16:45:18 <kalev> we have a few other people, but I'm doing most of the work
16:45:48 <nirik> wonder if we could tie into anytia/new-hotness...
16:46:07 <kalev> sberg does libreoffice and mcrha does evolution and firefox people keep half an eye on firefox, but beyond that it's just me :)
16:46:22 <nirik> otaylor: so if we someday moved our registry to quay.io... everything would still work provided quay had oci container support and multiarch?
16:46:37 <Zlopez[m]> nirik: I have plans for it, but don't have time to do it
16:46:49 <otaylor> nirik: Certainly just notifying developers "your flatpak needs a rebuild" would be a start - also pulling https://fedora.fishsoup.net/flatpak-status/ into Fedora infrastructure and making it reliable.
16:47:03 <Zlopez[m]> Right now the Anitya is used at least by flathub with x-data-checker
16:47:17 <Strider83> I love flatpack for discord and Spotify.  What is the benefit of flatpack in cases where native rpms are already available?
16:47:38 <otaylor> nirik: Everything should just work if we moved our registry to quay.io - oci container support is no longer even needed.
16:47:57 <nirik> otaylor: cool. I assume that dashboard could just be a small openshift app that scrapes things? shouldn't be too hard...
16:48:09 <nirik> excellent.
16:48:24 <Zlopez[m]> Strider83:  It's great for ostree based distributions like Silverblue
16:48:59 <otaylor> nirik: the complexity of flatpak-status is largely that it's trying to do complicated checks against dist-git - it probably can be majorly simplified.
16:49:52 <otaylor> Strider83: in addition to working well on Silverblue, it also provides the ability to have reliable updates without reboots of applications, and sandboxing of applications
16:49:56 <nirik> otaylor: I wonder... if it could be merged into the packager dashboard... that would give packagers visibility into flatpaks of their packages and they might help more? just an idle thought
16:50:17 <otaylor> nirik: do people use the packager dashboard (sorry for the question...)
16:50:29 <kalev> ohh, that's a great idea
16:50:48 <kalev> otaylor: it's a new thing so I assume not a lot of people at this point
16:51:03 <kalev> but if it's useful, I guess people are going to start using it :)
16:51:20 <nirik> not sure how much use it gets. ;)
16:51:29 <otaylor> Another pain point for developers I wanted to mention is that local builds don't work well because of problems with MBS  - I have a much better working version https://copr.fedorainfracloud.org/coprs/otaylor/flatpak-module-tools-experimental/ but the MBS team hasn't had bandwidth to review and integrate those changes.
16:51:48 <nirik> https://packager-dashboard.fedoraproject.org/ for anyone interested.
16:52:25 <nirik> otaylor: FWIW, mbs folks are going to be upgrading prod to v3 soon... hopefully after that they can look at your changes?
16:52:35 <otaylor> You probably should be using that copr if you are trying this stuff out, and if you see a friendly MBS developer, tell them how much they are useful to you :-)
16:53:05 <Zlopez[m]> REMINDER: We have 8 minutes of the meeting left
16:53:39 <otaylor> I think I've covered most of what I wanted to cover here.
16:53:59 <nirik> otaylor: can do.
16:54:20 <otaylor> One small thing I'd like to accomplish is figuring out better monitoring for flatpak-indexer - to get notification if it is failing rather than just freezing the old index in place
16:54:30 <dtometzki> many thanks otaylor
16:54:32 <otaylor> Finally, should mention an intersecting effort - which is adding a filtered subset of Flathub as a 3rd-party repository to Fedora Workstation for Fedora 35. This is mostly there to support things that we *can't* easily package in Fedora - electron apps, wrappers around proprietary apps. The initial plan is to skip things that are already available as Fedora Flatpaks or that are in RPM form in Fedora and easy to make a Flatpak
16:54:34 <otaylor> of.
16:54:59 <Zlopez[m]> otaylor: Monitoring of apps is also something we want to look at
16:55:09 <otaylor> (https://pagure.io/fedora-flathub-filter/ is a repo for tracking the whitelist.)
16:55:34 <nirik> yeah, monitoring is on our list for sure.
16:55:38 <kalev> I have one more thing I'd like to mention: We may have to do the cisco openh264 dance for flatpaks as well
16:55:49 <Zlopez[m]> otaylor: I'm now looking at the subsandboxing work, really great initiative
16:56:01 <otaylor> Zlopez[m]: in this case, probably the monitoring setup could be log based, have the indexer write something every time it succesfully updates the index, and if that doesn't appear in X hours, alert.
16:56:07 <kalev> send a flatpak runtime extension that includes the openh264 rpms built in koji to cisco to hosting
16:56:38 <kalev> and then do indexing / registry hacks so that it looks like it's coming from a regular registry in fedora infra, but the binary blob is actually downloaded from cisco
16:56:38 <nirik> kalev: should be all doable...
16:56:39 <otaylor> kalev: maybe we can use extra-data for that like they do on flathub? Haven't thought about what that would mean, though :-)
16:56:56 <kalev> ah, maybe? I didn't even know that exists :)
16:56:59 <Zlopez[m]> otaylor: I think this could be done with liveliness probe on OpenShift, but I'm not sure
16:57:37 <otaylor> Zlopez[m]: the app could certainly be modified to touch a file or whatever to make a liveness probe easy to write.
16:58:22 <otaylor> But I'm not sure that's enough, since ifi it's failing, what we probably need to do is not just restart the pod, but actually inform a human
16:58:25 <nirik> many thanks for the discussion / information otaylor and kalev!
16:58:28 <Zlopez[m]> otaylor: Thanks for the talk, it was great to see some insight into flatpaks in Fedora
16:58:40 <Zlopez[m]> otaylor++
16:58:43 <kalev> oh, and one other thing from me: big thanks to all the infra folks, especially nirik who have been super responsive when dealing with various flatpak building issues
16:58:54 <samyak-jn> Thank you for the verbose talk on this otaylor!
16:58:57 <otaylor> Thanks everybody for the questions and discussion!
16:58:57 <kalev> and cverna!
16:59:03 <dtometzki> otaylor++
16:59:04 <zodbot> dtometzki: Karma for otaylor changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:59:11 <nirik> happy to help. Great work you are all doing. :)
16:59:30 * otaylor also thanks cverna and nirik and mboddu and a bunch of other people for their help! :-)
16:59:31 <Zlopez[m]> kalev: cverna is now part of the CoreOS team, but he still comes here from time to time
16:59:49 <kalev> ahh
17:00:15 <Zlopez[m]> We are now out of the time
17:00:29 <Zlopez[m]> Thanks everybody for the great show today
17:00:40 <Zlopez[m]> #endmeeting