<@salimma:fedora.im>
17:06:57
!startmeeting FESCO (2026-10-20)
<@meetbot:fedora.im>
17:06:57
Meeting started at 2026-10-20 17:06:57 UTC
<@meetbot:fedora.im>
17:06:57
The Meeting name is 'FESCO (2026-10-20)'
<@salimma:fedora.im>
17:07:02
!meetingname fesco
<@meetbot:fedora.im>
17:07:03
The Meeting Name is now fesco
<@salimma:fedora.im>
17:07:07
!group members fesco
<@zodbot:fedora.im>
17:07:09
Members of fesco: Fabio Valentini, gotmax23 (@gotmax:matrix.org, @gotmax23:fedora.im), Jef Spaleta, Kevin Fenzi, Adam Miller, ngompa (@conan_kudo:matrix.org, @ngompa:fedora.im, @pharaoh_atem:opensuse.org, @ngompa:kde.org, @ngompa:almalinux.im), Michel Lind, Timothée Ravier, Simon de Vlieger, Zbigniew Jędrzejewski-Szmek
<@salimma:fedora.im>
17:07:17
!topic Init Process
<@zbyszek:fedora.im>
17:07:45
!hi
<@zodbot:fedora.im>
17:07:47
zbyszek: Zbigniew Jędrzejewski-Szmek (zbyszek)
<@nirik:matrix.scrye.com>
17:08:04
morning
<@Lihis:matrix.org>
17:08:33
!hi
<@zodbot:fedora.im>
17:08:36
Tomi: Tomi Lähteenmäki (lihis)
<@amoloney:fedora.im>
17:08:46
!hi
<@zodbot:fedora.im>
17:08:48
Aoife Moloney: Aoife Moloney (amoloney)
<@amoloney:fedora.im>
17:08:58
hello everyone! Im here in and out :)
<@salimma:fedora.im>
17:09:02
/me trying to recover his laptop
<@decathorpe:fedora.im>
17:09:05
!hi
<@zodbot:fedora.im>
17:09:07
Fabio Valentini 🌈: Fabio Valentini (decathorpe) - he / him / his
<@salimma:fedora.im>
17:10:07
Ok wait I might need to just reboot
<@zbyszek:fedora.im>
17:10:19
Did somebody misappropriate the laptop?
<@salimma:fedora.im>
17:11:52
No, just working it too hard
<@salimma:fedora.im>
17:12:23
Once I actually have a home office it won't be so bad
<@nirik:matrix.scrye.com>
17:12:43
luckily new hardware is cheap and... oh wait... nevermind
<@salimma:fedora.im>
17:13:14
I have enough hardware. I have no physical space to turn them on without becoming safety hazard
<@salimma:fedora.im>
17:13:37
Do we have quorum? Let me count
<@decathorpe:fedora.im>
17:14:16
I count four?
<@salimma:fedora.im>
17:14:34
Yeah only four
<@salimma:fedora.im>
17:14:43
Including me (my hi disappeared)
<@nirik:matrix.scrye.com>
17:14:43
😢
<@salimma:fedora.im>
17:14:45
!hi
<@zodbot:fedora.im>
17:14:46
Michel Lind UTC+1: Michel Lind (salimma) - he / him / his
<@decathorpe:fedora.im>
17:15:05
woop woop 🥳
<@conan_kudo:matrix.org>
17:15:41
!hi
<@zodbot:fedora.im>
17:15:44
Conan Kudo 😌: Neal Gompa (ngompa) - he / him / his
<@conan_kudo:matrix.org>
17:15:46
I guess I'll make quorum
<@salimma:fedora.im>
17:16:04
thanks! let's try and get started before we lose quorum then
<@salimma:fedora.im>
17:16:36
we probably should start with the incompat update for gedit since Fabio Valentini 🌈has questions and yselkowitzmentioned he can make it?
<@yselkowitz:fedora.im>
17:16:44
!hi
<@zodbot:fedora.im>
17:16:47
yselkowitz: Yaakov Selkowitz (yselkowitz)
<@decathorpe:fedora.im>
17:16:59
I don't have questions, I gave conditional +1
<@decathorpe:fedora.im>
17:17:14
sorry if that wasn't clear
<@salimma:fedora.im>
17:17:28
ah, this could be quick then. let me just bring back my meeting script
<@salimma:fedora.im>
17:17:56
!topic #3696 Incompatible update exception for gedit and libgedit-* 51 for F45
<@salimma:fedora.im>
17:18:01
!fesco 3696
<@zodbot:fedora.im>
17:18:02
**fesco/tickets #3696** (https://forge.fedoraproject.org/fesco/tickets/issues/3696):**[FastTrack] Incompatible update exception for gedit and libgedit-* 51 for F45**
<@zodbot:fedora.im>
17:18:02
<@zodbot:fedora.im>
17:18:02
● **Opened:** a day ago by yselkowitz
<@zodbot:fedora.im>
17:18:02
● **Last Updated:** 8 hours ago
<@zodbot:fedora.im>
17:18:02
● **Assignee:** Not Assigned
<@conan_kudo:matrix.org>
17:18:03
we basically need to approve this today
<@zbyszek:fedora.im>
17:18:14
We failed at fasttracking again.
<@zbyszek:fedora.im>
17:18:23
But let's just approve this and move on.
<@salimma:fedora.im>
17:18:40
we can keep it as fast track, but given that means we need to get to +7 we can also approve it in ticket, right?
<@decathorpe:fedora.im>
17:18:49
honestly I really think fast track should need to be added when a request is *filed* otherwise it just never works
<@salimma:fedora.im>
17:19:06
otherwise, yeah, the normal fasttrack process will take too long and we almost never have 7 votes
<@conan_kudo:matrix.org>
17:19:14
requestors cannot label it as fast track
<@conan_kudo:matrix.org>
17:19:22
only we can
<@salimma:fedora.im>
17:19:26
and agreed, if we add fast track later - at that point might as well do a meeting vote
<@nirik:matrix.scrye.com>
17:19:37
sorry, was busy... I can be +1
<@nirik:matrix.scrye.com>
17:19:42
(can vote in ticket too)
<@conan_kudo:matrix.org>
17:19:56
I did add fast track pretty quickly after it was filed
<@salimma:fedora.im>
17:20:04
so with Nirik it's +5
<@nirik:matrix.scrye.com>
17:20:18
note that the next and hopefully final rc is going to be made later today, so it really needs nowish.
<@salimma:fedora.im>
17:20:20
yeah, in this case the fast track went quickly, but it's still... unnervingly close (f45 could have been released today)
<@salimma:fedora.im>
17:20:37
so we can say... if there is no objection here I will propose we approve it
<@decathorpe:fedora.im>
17:20:56
ack
<@zbyszek:fedora.im>
17:21:15
ack
<@salimma:fedora.im>
17:21:17
proposed: agreed Incompatible update exception for gedit and libgedit-* approved
<@gotmax23:fedora.im>
17:21:24
ack
<@decathorpe:fedora.im>
17:21:43
is the out of band retirement OK with releng?
<@decathorpe:fedora.im>
17:21:57
but +1
<@conan_kudo:matrix.org>
17:22:01
ack
<@salimma:fedora.im>
17:22:31
holding a sec to see if gotmax wants to vote and if nirik wants to answer the out of band retirement question
<@nirik:matrix.scrye.com>
17:22:35
I guess. I am definitely getting 'way too close to release' change fatigue... but this seems doable.
<@salimma:fedora.im>
17:22:44
if the package is not installable because it got obsoleted it... should be fine right?
<@gotmax23:fedora.im>
17:22:46
oh sure, +1
<@salimma:fedora.im>
17:22:56
!AGREED Incompatible update exception for gedit and libgedit-* approved (+6, 0, -0)
<@decathorpe:fedora.im>
17:23:08
well ... the broken Version will still be in the repos, it just won't be installable
<@salimma:fedora.im>
17:23:08
hmm apparently it does not like shouting
<@nirik:matrix.scrye.com>
17:23:13
yeah.
<@conan_kudo:matrix.org>
17:23:20
commands are case sensitive apparently
<@salimma:fedora.im>
17:23:21
!agreed Incompatible update exception for gedit and libgedit-* approved (+6, 0, -0)
<@salimma:fedora.im>
17:23:29
even though the result is shouted out
<@salimma:fedora.im>
17:23:50
sorry to drag you here for not much Yaakov
<@salimma:fedora.im>
17:23:57
please file the freeze exception asap
<@yselkowitz:fedora.im>
17:23:58
thank you, will start builds momentarily
<@gotmax23:fedora.im>
17:24:07
thanks
<@salimma:fedora.im>
17:24:11
let's move on?
<@salimma:fedora.im>
17:24:15
!topic #3693 removal of NetworkManager-{vpnc,fortisslvpn} in F43 and F44
<@salimma:fedora.im>
17:24:19
!fesco 3693
<@zodbot:fedora.im>
17:24:20
**fesco/tickets #3693** (https://forge.fedoraproject.org/fesco/tickets/issues/3693):**removal of NetworkManager-{vpnc,fortisslvpn} in F43 and F44**
<@zodbot:fedora.im>
17:24:20
<@zodbot:fedora.im>
17:24:20
● **Opened:** a week ago by zbyszek
<@zodbot:fedora.im>
17:24:20
● **Last Updated:** 19 hours ago
<@zodbot:fedora.im>
17:24:20
● **Assignee:** Not Assigned
<@salimma:fedora.im>
17:24:45
did we have a volunteer for the article last week? iirc nobody stepped up
<@zbyszek:fedora.im>
17:24:49
I think nothing has changed here during the last week.
<@gotmax23:fedora.im>
17:24:51
is there actually anything new to discuss here?
<@salimma:fedora.im>
17:25:12
yeah, nobody untagged it last week so by default it got included today
<@conan_kudo:matrix.org>
17:25:13
not really
<@salimma:fedora.im>
17:25:22
I'll untag it then
<@conan_kudo:matrix.org>
17:25:37
we're not going to approve obsoleting them in existing stable releases, and nobody wrote an article about this for F45 users
<@salimma:fedora.im>
17:25:37
proposed info nothing has moved in a week, we will wait for progress in the ticket
<@salimma:fedora.im>
17:26:02
so.. nobody cares enough to push this through I guess. sometimes one wishes for a lot (fix the stable releases too!) and then ...
<@salimma:fedora.im>
17:26:26
any ack for the proposed info? we might be out here soon (said he nervously, looking at the last meeting topic)
<@nirik:matrix.scrye.com>
17:26:51
ack
<@salimma:fedora.im>
17:27:05
!info Nothing has moved in a week, we will wait for progress in the ticket
<@decathorpe:fedora.im>
17:27:07
ack
<@salimma:fedora.im>
17:27:28
let's just do this quickly, it's more an FYI
<@salimma:fedora.im>
17:27:32
!topic fesco/docs#156 Add a policy for the global CI tests
<@salimma:fedora.im>
17:27:51
uh gosh I forgot the forge syntax
<@salimma:fedora.im>
17:27:55
!forge fesco/docs 156
<@zodbot:fedora.im>
17:27:55
**Usage:** !forge <subcommand> [...]
<@zodbot:fedora.im>
17:27:55
<@zodbot:fedora.im>
17:27:55
● issue <org> <repo> <issue_id> - return an issue from the forge
<@zodbot:fedora.im>
17:27:55
● pr <org> <repo> <pull_id> - return a pull request from the forge
<@salimma:fedora.im>
17:28:09
!forge pr fesco docs 156
<@zodbot:fedora.im>
17:28:12
**fesco/docs #156** (https://forge.fedoraproject.org/fesco/docs/pulls/156):**Add a policy for the global CI tests**
<@zodbot:fedora.im>
17:28:12
<@zodbot:fedora.im>
17:28:12
● **Opened:** 5 days ago by lecris
<@zodbot:fedora.im>
17:28:12
● **Last Updated:** 5 days ago
<@zodbot:fedora.im>
17:28:12
● **Assignee:** Not Assigned
<@salimma:fedora.im>
17:28:18
aha
<@nirik:matrix.scrye.com>
17:28:24
you can do just '!fesco' as a short hand.
<@conan_kudo:matrix.org>
17:28:29
magic!
<@nirik:matrix.scrye.com>
17:28:31
oh, not for pr's tho
<@salimma:fedora.im>
17:28:49
zbyszek has left a review and lecris has updated it, if anyone else wants to eyeball this, please do
<@conan_kudo:matrix.org>
17:29:14
I want to do a read through of that, hopefully I'll get a chance this week
<@decathorpe:fedora.im>
17:29:30
this is not entirely what kind of docs I was thinking of, as right now they don't say a lot of anything (at least in my first reading)
<@nirik:matrix.scrye.com>
17:30:10
I think they are a reasonable start... and we can add / adjust from there...
<@salimma:fedora.im>
17:30:12
yeah, iirc this came about because people asked how to get a new gating test added?
<@gotmax23:fedora.im>
17:30:19
yes
<@conan_kudo:matrix.org>
17:30:23
well... it should probably _say_ something?
<@gotmax23:fedora.im>
17:30:31
I proposed this and then got distracted by the five other policy proposal discussions we've been having 😅
<@salimma:fedora.im>
17:30:37
but I agree with nirik, if we don't even have a basic policy, better have something that is reasonable and incomplete then add the missing part
<@conan_kudo:matrix.org>
17:30:38
my first read seems to be it's more philosophy than policy
<@gotmax23:fedora.im>
17:30:42
maybe let's !action everyone to review it outside of the meeting and move on?
<@salimma:fedora.im>
17:30:48
yes, ack
<@salimma:fedora.im>
17:31:03
!action FESCo members will review this PR after this meeting
<@decathorpe:fedora.im>
17:31:07
ack
<@salimma:fedora.im>
17:31:25
!topic #3688 Imposing performance requirements for architectures hosted in the
<@salimma:fedora.im>
17:31:25
main Koji instance
<@salimma:fedora.im>
17:31:30
argh
<@salimma:fedora.im>
17:31:36
!topic #3688 Imposing performance requirements for architectures hosted in the main Koji instance
<@salimma:fedora.im>
17:31:40
!fesco 3688
<@zodbot:fedora.im>
17:31:41
**fesco/tickets #3688** (https://forge.fedoraproject.org/fesco/tickets/issues/3688):**Imposing performance requirements for architectures hosted in the main Koji instance**
<@zodbot:fedora.im>
17:31:41
<@zodbot:fedora.im>
17:31:41
● **Opened:** a month ago by ngompa
<@zodbot:fedora.im>
17:31:41
● **Last Updated:** 5 hours ago
<@zodbot:fedora.im>
17:31:41
● **Assignee:** Not Assigned
<@salimma:fedora.im>
17:31:58
this has been going in circles for a while, so I figured let's actually discuss it in a meeting for awareness / to hopefully unblock
<@zbyszek:fedora.im>
17:32:05
nirik had some updates a while back, is there more news?
<@nirik:matrix.scrye.com>
17:32:06
so, I was hoping to ask @Michel Lind UTC+1 to re-run his stats for s390x from the last week...
<@nirik:matrix.scrye.com>
17:32:18
yes, it should be much better, but I would like feedback about it.
<@nirik:matrix.scrye.com>
17:32:26
we are using the existing resources much better now
<@salimma:fedora.im>
17:32:34
yeah, I am in the process of running an update, I might have some data by tomorrow
<@salimma:fedora.im>
17:32:57
let me know what I should focus on, but I'd note that normally we don't know for sure until a mass rebuild if the conditions are good enough or not
<@nirik:matrix.scrye.com>
17:32:59
it's not been too busy tho... it's never gotten backloged for example
<@nirik:matrix.scrye.com>
17:33:30
well, in the past we have definitely had backlogs that had nothing to do with mass rebuild
<@salimma:fedora.im>
17:33:32
what date did this change happen? I can try comparing specifically performance before and after that
<@salimma:fedora.im>
17:33:55
right, it basically just requires a few heavy duty tasks
<@salimma:fedora.im>
17:34:34
before it crashed my laptop the one new finding I had is that s390x seems faster than ppc64le for small tasks, but for compute heavy builds > 5 minutes it starts becoming... much slower. so once you have gcc, llvm, etc building...
<@nirik:matrix.scrye.com>
17:35:05
2026-10-08 and after. That was when they changed cpu weights and we moved to newer kernel... but after that we changed some cpu stuff on builders (I would need to look at when that was)
<@salimma:fedora.im>
17:35:27
!info CPU weights on s390x builders changed on 2026-10-08
<@nirik:matrix.scrye.com>
17:35:53
due to increased cpus on builders large tasks are about 2/3 the time (2 cpus to 3).
<@nirik:matrix.scrye.com>
17:36:08
Anyhow, we still need more resources, but hopefully this helps short term.
<@salimma:fedora.im>
17:36:11
lmk when that is and I can try to measure (and zbyszek too) but meanwhile I'll focus on that date
<@zbyszek:fedora.im>
17:36:30
731 builds shown, 269 without x86_64 build skipped
<@zbyszek:fedora.im>
17:36:30
Build Start (UTC) x86_64 i686 aarch64 ppc64le s390x
<@zbyszek:fedora.im>
17:36:30
------------------------------------------------------------------------------------------------------------------
<@zbyszek:fedora.im>
17:36:30
libgedit-gfls-0.5.0-1.fc45 2026-10-20 17:28 0:01:24 134% 96% 215% 86%
<@zbyszek:fedora.im>
17:36:30
libgedit-amtk-5.11.0-1.fc45 2026-10-20 17:28 0:01:21 107% 103% 192% 107%
<@zbyszek:fedora.im>
17:36:30
opam-2.6.1-1.fc45 2026-10-20 16:32 0:04:48 - 149% 322% 124%
<@zbyszek:fedora.im>
17:36:30
xorg-x11-xinit-1.4.3-6.eln159 2026-10-20 16:23 0:01:20 - 84% 76% 68%
<@zbyszek:fedora.im>
17:36:30
ocaml-tar-3.5.0-1.fc45 2026-10-20 16:21 0:01:05 - 103% 135% 96%
<@zbyszek:fedora.im>
17:36:30
python-pycares-5.2.1-1.fc45 2026-10-20 16:21 0:01:14 130% 108% 117% 85%
<@zbyszek:fedora.im>
17:36:30
python-pycares-5.2.1-1.fc44 2026-10-20 16:21 0:01:04 158% 124% 151% 103%
<@zbyszek:fedora.im>
17:36:30
python-pycares-5.2.1-1.fc46 2026-10-20 16:17 0:01:11 102% 112% 130% 98%
<@zbyszek:fedora.im>
17:36:30
ocaml-decompress-1.6.1-1.fc45 2026-10-20 16:07 0:01:13 - 96% 225% 88%
<@zbyszek:fedora.im>
17:36:30
rust-cargo-semver-checks-0.51.0-5.fc44 2026-10-20 16:02 0:28:02 - 137% 100% 94%
<@zbyszek:fedora.im>
17:36:30
...
<@zbyszek:fedora.im>
17:36:30
nushell-0.99.1-9.fc46 2026-10-17 12:00 0:14:56 - 122% 148% 161%
<@zbyszek:fedora.im>
17:36:30
rust-zip-8.6.0-9.fc45 2026-10-17 11:58 0:03:24 109% 115% 158% 93%
<@zbyszek:fedora.im>
17:36:30
rust-openssl-0.10.81-3.fc46 2026-10-17 11:58 0:02:01 153% 140% 130% 103%
<@zbyszek:fedora.im>
17:36:30
rust-nu-plugin-core-0.99.1-5.fc43 2026-10-17 11:58 0:07:27 55% 171% 141% 102%
<@zbyszek:fedora.im>
17:36:30
rust-zip-8.6.0-9.fc43 2026-10-17 11:57 0:03:27 144% 140% 143% 88%
<@zbyszek:fedora.im>
17:36:30
rust-zip-8.6.0-9.fc44 2026-10-17 11:56 0:03:18 111% 142% 154% 95%
<@zbyszek:fedora.im>
17:36:30
rust-zip-8.6.0-9.fc46 2026-10-17 11:54 0:03:22 103% 114% 154% 96%
<@zbyszek:fedora.im>
17:36:30
rust-lzma-rust2-0.22.0-1.fc45 2026-10-17 11:47 0:01:25 103% 109% 124% 94%
<@zbyszek:fedora.im>
17:36:30
rust-lzma-rust2-0.22.0-1.fc43 2026-10-17 11:46 0:01:22 84% 99% 119% 89%
<@zbyszek:fedora.im>
17:36:30
rust-lzma-rust2-0.22.0-1.fc44 2026-10-17 11:45 0:01:18 108% 119% 142% 111%
<@zbyszek:fedora.im>
17:36:30
rust-nu-plugin-core-0.99.1-5.fc44 2026-10-17 11:43 0:07:40 60% 108% 118% 98%
<@zbyszek:fedora.im>
17:36:30
poster-20060221-49.fc46 2026-10-17 11:43 0:01:06 83% 93% 125% 88%
<@zbyszek:fedora.im>
17:36:30
------------------------------------------------------------------------------------------------------------------
<@zbyszek:fedora.im>
17:36:30
Total 92:05:09 117% 122% 138% 126%
<@zbyszek:fedora.im>
17:36:30
Number of builds 731 319 723 719 628
<@salimma:fedora.im>
17:36:33
good to know, so ... from 2026-10-08 larger tasks should no longer be that slow. that'd be nice
<@salimma:fedora.im>
17:36:44
all in bold :)
<@gotmax23:fedora.im>
17:36:52
hmm, Element markdown might not support tables
<@zbyszek:fedora.im>
17:36:56
```
<@zbyszek:fedora.im>
17:36:56
## 731 builds shown, 269 without x86\_64 build skipped
<@zbyszek:fedora.im>
17:36:56
Build Start (UTC) x86\_64 i686 aarch64 ppc64le s390x
<@zbyszek:fedora.im>
17:36:56
<@zbyszek:fedora.im>
17:36:56
## libgedit-gfls-0.5.0-1.fc45 2026-10-20 17:28 0:01:24 134% 96% 215% 86%
<@zbyszek:fedora.im>
17:36:56
libgedit-amtk-5.11.0-1.fc45 2026-10-20 17:28 0:01:21 107% 103% 192% 107%
<@zbyszek:fedora.im>
17:36:56
opam-2.6.1-1.fc45 2026-10-20 16:32 0:04:48 - 149% 322% 124%
<@zbyszek:fedora.im>
17:36:56
xorg-x11-xinit-1.4.3-6.eln159 2026-10-20 16:23 0:01:20 - 84% 76% 68%
<@zbyszek:fedora.im>
17:36:56
ocaml-tar-3.5.0-1.fc45 2026-10-20 16:21 0:01:05 - 103% 135% 96%
<@zbyszek:fedora.im>
17:36:56
python-pycares-5.2.1-1.fc45 2026-10-20 16:21 0:01:14 130% 108% 117% 85%
<@zbyszek:fedora.im>
17:36:56
python-pycares-5.2.1-1.fc44 2026-10-20 16:21 0:01:04 158% 124% 151% 103%
<@zbyszek:fedora.im>
17:36:56
python-pycares-5.2.1-1.fc46 2026-10-20 16:17 0:01:11 102% 112% 130% 98%
<@zbyszek:fedora.im>
17:36:56
ocaml-decompress-1.6.1-1.fc45 2026-10-20 16:07 0:01:13 - 96% 225% 88%
<@zbyszek:fedora.im>
17:36:56
rust-cargo-semver-checks-0.51.0-5.fc44 2026-10-20 16:02 0:28:02 - 137% 100% 94%
<@zbyszek:fedora.im>
17:36:56
...
<@zbyszek:fedora.im>
17:36:56
nushell-0.99.1-9.fc46 2026-10-17 12:00 0:14:56 - 122% 148% 161%
<@zbyszek:fedora.im>
17:36:56
rust-zip-8.6.0-9.fc45 2026-10-17 11:58 0:03:24 109% 115% 158% 93%
<@zbyszek:fedora.im>
17:36:56
rust-openssl-0.10.81-3.fc46 2026-10-17 11:58 0:02:01 153% 140% 130% 103%
<@zbyszek:fedora.im>
17:36:56
rust-nu-plugin-core-0.99.1-5.fc43 2026-10-17 11:58 0:07:27 55% 171% 141% 102%
<@zbyszek:fedora.im>
17:36:56
rust-zip-8.6.0-9.fc43 2026-10-17 11:57 0:03:27 144% 140% 143% 88%
<@zbyszek:fedora.im>
17:36:56
rust-zip-8.6.0-9.fc44 2026-10-17 11:56 0:03:18 111% 142% 154% 95%
<@zbyszek:fedora.im>
17:36:56
rust-zip-8.6.0-9.fc46 2026-10-17 11:54 0:03:22 103% 114% 154% 96%
<@zbyszek:fedora.im>
17:36:56
rust-lzma-rust2-0.22.0-1.fc45 2026-10-17 11:47 0:01:25 103% 109% 124% 94%
<@zbyszek:fedora.im>
17:36:56
rust-lzma-rust2-0.22.0-1.fc43 2026-10-17 11:46 0:01:22 84% 99% 119% 89%
<@zbyszek:fedora.im>
17:36:56
rust-lzma-rust2-0.22.0-1.fc44 2026-10-17 11:45 0:01:18 108% 119% 142% 111%
<@zbyszek:fedora.im>
17:36:56
rust-nu-plugin-core-0.99.1-5.fc44 2026-10-17 11:43 0:07:40 60% 108% 118% 98%
<@zbyszek:fedora.im>
17:36:56
poster-20060221-49.fc46 2026-10-17 11:43 0:01:06 83% 93% 125% 88%
<@zbyszek:fedora.im>
17:36:56
<@zbyszek:fedora.im>
17:36:56
Total 92:05:09 117% 122% 138% 126%
<@zbyszek:fedora.im>
17:36:56
Number of builds 731 319 723 719 628
<@zbyszek:fedora.im>
17:36:56
```
<@nirik:matrix.scrye.com>
17:36:57
wall of bold text
<@salimma:fedora.im>
17:37:24
unfortunately fractal is horrible at showing anything requiring scrolling..
<@zbyszek:fedora.im>
17:37:43
> 92:05:09 117% 122% 138% 126%
<@salimma:fedora.im>
17:38:06
so right now s390x is not much slower - as of October?
<@zbyszek:fedora.im>
17:38:10
This is the important part, basically s390x builds took in total 126% time of x86_64 builds.
<@nirik:matrix.scrye.com>
17:38:11
Number of cpus was changed on the 12th
<@zbyszek:fedora.im>
17:38:18
This is totally OK.
<@salimma:fedora.im>
17:38:22
a bit slower than aarch and a bit faster than ppc64le
<@nirik:matrix.scrye.com>
17:38:48
but there really hasn't been a large pile of builds since the changes as far as I see.
<@zbyszek:fedora.im>
17:38:50
(time == latency from when the first of the builds started to when the build for a given arch finished)
<@salimma:fedora.im>
17:39:07
one finding I have not confirmed is that we are building much less packages for s390x since 2025-09 or so, but I have not narrowed it down yet
<@conan_kudo:matrix.org>
17:39:25
afaik s390x has been removed from CI channels
<@conan_kudo:matrix.org>
17:39:32
so that might be why?
<@salimma:fedora.im>
17:39:35
it can't just be us not doing SRPM and noarch builds, I think. if anyone knows what happens roughly in that time I'm all ears
<@nirik:matrix.scrye.com>
17:39:42
it's not. it's just restricted to 2-3 builders only
<@zbyszek:fedora.im>
17:39:52
Yeah, in this sample, 628 for s390x, vs. ~720 for other architectures.
<@salimma:fedora.im>
17:40:00
oh... could be. that would definitely help a lot (but we still have the slowdown in the f45 mass rebuild, and that was not too long ago, after 2025-09)
<@salimma:fedora.im>
17:40:13
aha
<@salimma:fedora.im>
17:40:29
zbyszekyou're just pulling official builds, not scratch / CI right?
<@nirik:matrix.scrye.com>
17:40:34
It's 4 currently ( koji list-hosts --channel ci --arch s390x )
<@salimma:fedora.im>
17:40:38
but anyway, let's not get pigeonholed here
<@salimma:fedora.im>
17:40:56
feel free to add any info on the issue, and I'll update it with the analysis tomorrow before I go on vacation
<@zbyszek:fedora.im>
17:41:23
It's using session.listBuilds, so I _think_ this doesn't return scratch builds.
<@salimma:fedora.im>
17:42:46
for scratch builds my ideal situation is if we can start doing rpminspect / installability without waiting for the slowest arch to finish. and being able to stop a no longer needed CI build :)
<@salimma:fedora.im>
17:43:20
ok .. let's see if the performance has improved reasonably since the last tweaks, as it seems they might have - anyone else on this topic?
<@decathorpe:fedora.im>
17:43:44
(also note that s390x is the only full arch not covered by koschei, so it is much less utilized by scratch build count)
<@salimma:fedora.im>
17:44:13
and historically, despite that it still takes forever on my PR CI :)
<@decathorpe:fedora.im>
17:44:31
different problem
<@salimma:fedora.im>
17:44:35
yep
<@conan_kudo:matrix.org>
17:45:00
so I want to remind people that this ticket isn't _actually_ about s390x
<@salimma:fedora.im>
17:45:05
I agree we should prioritize production builds, but that's also why I floated the "maybe we should decouple the architectures more, especially for scratch builds"
<@conan_kudo:matrix.org>
17:45:15
s390x triggered it, but this is also about future arch evaluations
<@salimma:fedora.im>
17:45:41
(and even for existing arches, ppc64le is also running slow)
<@conan_kudo:matrix.org>
17:45:46
the reason we need to get a grip on this is because we need a metric to characterize riscv64 with
<@conan_kudo:matrix.org>
17:45:55
the reason we need to get a grip on this is because we need a metric to characterize riscv64 with
<@conan_kudo:matrix.org>
17:46:28
I don't want us to repeat the downward spiral that happened when we integrated arm after fedora 20
<@zbyszek:fedora.im>
17:47:35
I don't think we should try to decide this in the abstract. If and when riscv64 is proposed as an official arch, then we can evaluate. In particular, I expect that the state will be "the current build time is …, but two months from now we plan to have …, and then next year …" and even if we came up with a number now, it wouldn't be nuanced enough to help us decide then.
<@salimma:fedora.im>
17:48:15
I think what Conan Kudo 😌wants is that architectures are not just evaluated for performance at intake time
<@salimma:fedora.im>
17:48:26
but there is ongoing reevaluation with the possibility of being downgraded. like Debian does
<@conan_kudo:matrix.org>
17:48:32
yup
<@conan_kudo:matrix.org>
17:48:40
we don't have this, and we need it
<@salimma:fedora.im>
17:49:03
I also would like us to not be tightly coupled as we are right now, so the slowest architecture does not end up bottlenecking other tasks (hence the CI example)
<@conan_kudo:matrix.org>
17:49:10
and s390x and ppc64le are great proving grounds for a strategy that we can apply later to riscv64
<@salimma:fedora.im>
17:49:53
s390x and ppc64le are also ... politically charged since upstream/downstream care about these differently though. for riscv it seems the enthusiasm exists on both ends
<@nirik:matrix.scrye.com>
17:50:18
I think adding and demotion should be a really big deal, and not 'oh, you went over the threshold, you're out'. But having some guidence could help riscv to know what to shoot for... and others to know what to improve to... but I am not sure I like an absolete thing.
<@zbyszek:fedora.im>
17:50:30
Decoupling of CI would be great. But that's something of a separate ticket.
<@conan_kudo:matrix.org>
17:50:48
yeah, but that just makes my point... we need a way that fedora itself agrees upon to judge architectures so that ports can be held to that standard
<@conan_kudo:matrix.org>
17:50:57
not just at intake, but continuously over time
<@conan_kudo:matrix.org>
17:51:24
we have been underwater on every architecture at one point or another simply because we don't have this information to characterize and plan accordingly
<@decathorpe:fedora.im>
17:51:39
so sort of like a checkin like the one that's supposed to happen for spins?
<@salimma:fedora.im>
17:51:42
right. you might have a roadmap that everyone has good intentions about, nobody is lying, and through circumstances suddenly, "oh the new CPU is delayed" or "oh it is available but costs 2x our initial estimate"
<@conan_kudo:matrix.org>
17:51:54
yup exactly
<@conan_kudo:matrix.org>
17:52:22
it also has the advantage of being a communicable metric for IHVs that we work with
<@conan_kudo:matrix.org>
17:52:36
and that is tremendously useful for all kinds of reasons
<@salimma:fedora.im>
17:53:10
nods. and we have to run this against our existing arches, since we already have them. obviously we don't want to, say, kick out aarch64
<@conan_kudo:matrix.org>
17:53:42
when I ran buildsystem infrastructure in an old job, we had to have this information to prove we were meeting the quality expectations of the users... it also gave me the room to push for upgrades and hardware and other things
<@nirik:matrix.scrye.com>
17:54:01
I don't disagree with any of this, I just not sure what the metic(s) should be. I don't think I like the single one proposed in the ticket.
<@conan_kudo:matrix.org>
17:54:21
in the future theoryland where other people can offer hardware to run Fedora infrastructure, this can be super valuable for suitability of adding their infra to our systems
<@salimma:fedora.im>
17:54:48
should we action people who care about this to propose alternate metrics?
<@conan_kudo:matrix.org>
17:56:14
there is supposed to be an effort from CLE on this already, I just haven't heard anything from them about it
<@conan_kudo:matrix.org>
17:56:30
during the FRCL discussions that sparked this, a number of things were proposed and they were going to investigate it
<@zbyszek:fedora.im>
17:56:33
If we "downgraded" an arch for a specific release, e.g. by dropping it from the list of primary architectures, what would that mean? We'd completely stop building packages, right? Because there is no secondary koji set up. Could that arch then come back a release later?
<@conan_kudo:matrix.org>
17:56:34
no idea what happened since
<@conan_kudo:matrix.org>
17:56:58
!link https://redhat.atlassian.net/browse/CLE-6650
<@nirik:matrix.scrye.com>
17:57:05
It's super PTO season. Lots of people out (like me later this week)
<@salimma:fedora.im>
17:57:14
Ideally secondary Koji is more of a thing and downgrading to something like the current risc v koji is not too difficult
<@conan_kudo:matrix.org>
17:57:31
(well, this ticket has been open for 5 months...)
<@nirik:matrix.scrye.com>
17:57:55
onboarding/offboarding a arch is... not simple.
<@nirik:matrix.scrye.com>
17:58:27
well, might ping on it. I think that was @jednorozec ? ¯\_(ツ)_/¯
<@salimma:fedora.im>
17:58:35
ok - let's carry on in the ticket then? looks like we have a few questions we need to figure out - a) what is the acceptance criteria, b) how often to reevaluate, and c) what does promotion/demotion look like and how we can make it less painful
<@zbyszek:fedora.im>
17:58:48
Sounds good.
<@salimma:fedora.im>
17:59:17
let's carry on then
<@zbyszek:fedora.im>
17:59:21
FWIW, the current stats seem totally OK. So we might need to wait until after the next mass rebuild or something else that puts pressure on koji to have a good picture.
<@salimma:fedora.im>
17:59:28
!topic Next week's chair
<@nirik:matrix.scrye.com>
17:59:42
I'm sure there will be another flurry of big builds soon, there always are.
<@salimma:fedora.im>
17:59:47
well, I guess we can use the scratch build data to approximate the next mass rebuild, since those are more constrained
<@salimma:fedora.im>
18:00:11
who wants to take this next week?
<@salimma:fedora.im>
18:01:44
we might also cancel next week's meeting ahead of time if nobody is confident enough they would be around :)
<@zbyszek:fedora.im>
18:01:55
I can do it.
<@salimma:fedora.im>
18:02:08
!action zbyszek will chair next meeting
<@salimma:fedora.im>
18:02:16
thank you
<@salimma:fedora.im>
18:02:22
!topic Council happenings
<@salimma:fedora.im>
18:02:43
anything happened in the past week? (I should note that next week FRCL will discuss the future of Bugzilla again)
<@zbyszek:fedora.im>
18:03:05
I think p90 would be useful, but then probably it needs to be in the wallclock minutes, not percentages.
<@conan_kudo:matrix.org>
18:03:52
I will not be here next week
<@conan_kudo:matrix.org>
18:03:56
as I will be on a plane
<@salimma:fedora.im>
18:03:58
yes, that's what my previous report measured - wallclock minutes
<@amoloney:fedora.im>
18:04:04
Re:council happenings, I just posted the agenda for tomorrows meeting (Fabio would not have had time to see it yet....unless they are omnipresent....likely 👀 😄) https://discussion.fedoraproject.org/t/fedora-council-bi-weekly-agenda-2026-10-21/205540
<@amoloney:fedora.im>
18:05:43
oh and one more thing for council stuff - please check your email inbox for a google form from Shaun McCance about possible dates for a Fedora leadership meetup
<@amoloney:fedora.im>
18:06:05
I have 2 open floors as well then - will be quick, I promise!
<@salimma:fedora.im>
18:06:11
!topic Open floor
<@yselkowitz:fedora.im>
18:08:16
gedit 51.0 built for F45, FinalFE request: https://forge.fedoraproject.org/quality/blocker-review/issues/2235
<@amoloney:fedora.im>
18:08:27
1. As Michel Lind UTC+1 mentioned, the next FRCL call is next wednesday, 28th oct and its bugzilla, again! I left a comment yesterday on the requirements tracking ticket. I know this is literally the worst possible week(s) to ask for more work from folks, but if you could find some time to investigate the staging instance on forge ahead of the call, that really would be very appreicated. Tomas also has a fresher report done too. I linked v 1.1, he may also have 1.2 in the next few dats
<@amoloney:fedora.im>
18:08:36
*days
<@gotmax23:fedora.im>
18:09:18
It seems a lot of this is AI-generated without following the Fedora policy about disclosing AI usage
<@nirik:matrix.scrye.com>
18:09:43
which this do you mean?
<@amoloney:fedora.im>
18:09:46
I can assure you I am not AI, so I assume you mean the report in the comment :P
<@gotmax23:fedora.im>
18:09:50
yes
<@gotmax23:fedora.im>
18:10:01
and the Forgejo Actions workflows intended to replicate certain built-in bugzilla features
<@amoloney:fedora.im>
18:12:05
Do you think we should have the bugzilla topic for next week? It might be better to punt it to give ppl more time
<@gotmax23:fedora.im>
18:12:45
I looked closer and it seems Forgejo has some limited support for cross-repo bug relationships and has query features that are slightly less limited than I thought, and it seems the plan for everything else is to try to replace it with Forgejo Actions
<@salimma:fedora.im>
18:12:47
don't wait on my account, I basically won't be able to make meetings for the rest of the year
<@gotmax23:fedora.im>
18:12:56
yes, probably
<@nirik:matrix.scrye.com>
18:13:45
I think quality folks won't be able to look much at it until after f45 is out...
<@gotmax23:fedora.im>
18:13:45
what are we planning to discuss in the FRCL meeting, precisely?
<@amoloney:fedora.im>
18:14:02
I have a spidey sense that this is going to be, not exactly a _waste_ of time, but less valuable for people
<@gotmax23:fedora.im>
18:14:16
it sounds like if we don't have a foundation to host Bugzilla, we don't really have anything else to discuss in FRCL
<@salimma:fedora.im>
18:14:22
fwiw the last bugzilla meeting felt useless and confrontational because there was no pre-read
<@amoloney:fedora.im>
18:14:25
probably whether FESCo has had the chance to investigate the stg instance
<@salimma:fedora.im>
18:14:39
and I feel it will be more useful to let other people present alternatives to rhbz
<@gotmax23:fedora.im>
18:14:49
my account doesn't actually have access to anything on staging, i don't think
<@amoloney:fedora.im>
18:14:58
Consider this a pre-read (I learn from mistakes), but I do believe next week is still not enough time
<@salimma:fedora.im>
18:15:02
e.g. a side that wants to propose community hosted bugzilla, a side that wants to... gasp... propose using JIRA
<@gotmax23:fedora.im>
18:15:15
oh you're proposing to push it back later than next week Aoife?
<@nirik:matrix.scrye.com>
18:16:14
I think jira is a non starter all around (at least I hope so)
<@amoloney:fedora.im>
18:16:58
I am. Im flagging that theres some progress made on the Forge side that your (FESCo) input is needed on, but I dont think next week will give us any value. I think we should consider postponing at least 2 weeks (or to the next week *after* the release goes out)
<@salimma:fedora.im>
18:17:10
yeah, I hope so too, just trying to make the point it's not just bugzilla vs forgejo, surely there might be other options too
<@amoloney:fedora.im>
18:17:23
and no one is suggesting or pushing for Jira. Not from Council anyway - categorically not.
<@amoloney:fedora.im>
18:18:01
Jira was, and is an option. But no one wants that, so its a no from Council, and a no from FESCo so its a no
<@gotmax23:fedora.im>
18:18:15
I can't stop anyone, but seriously considering Jira kinda seems like a thought exercise that's not a good use of our time
<@nirik:matrix.scrye.com>
18:18:45
anyhow, I think moving at least one more week out would be good.
<@salimma:fedora.im>
18:18:55
even a quick survey of what other distributions do might be useful (does not have to be a presentation, that can be an email)
<@amoloney:fedora.im>
18:19:15
thats a very good idea
<@amoloney:fedora.im>
18:20:15
I do want to ask though about the community bugzilla instance idea. Is there anyone from FESCo, or the wider communuty, actually working on that?
<@amoloney:fedora.im>
18:20:21
I do want to ask though about the community bugzilla instance idea. Is there anyone from FESCo, or the wider community, actually working on that?
<@salimma:fedora.im>
18:21:12
Conan Kudo 😌do you remember what the blockers are for that?
<@gotmax23:fedora.im>
18:21:16
I actually wanted to ask, if Council would be willing to help us explore that
<@amoloney:fedora.im>
18:21:23
CLE have said they will look at Forge, and theres been no objection to that idea. Im just not clear if theres anyone actually looking into it in earnst?
<@amoloney:fedora.im>
18:21:30
CLE have said they will look at Forge, and theres been no objection to that idea. Im just not clear if theres anyone actually looking into it in earnest?
<@gotmax23:fedora.im>
18:21:43
if we wanted external funding of BZ to happen, we'd need to some sort of legal structure around it afaiu
<@conan_kudo:matrix.org>
18:21:50
We need somewhere to run it, and we probably will want someone from upstream with us for gap analysis
<@salimma:fedora.im>
18:22:00
there was someone in the afterparty, so I would apply Chatham House rules and not name them, that signals they would not block such an initiative if proposed
<@salimma:fedora.im>
18:22:22
so given that was very recent I am not sure anyone would have done much research into it yet
<@amoloney:fedora.im>
18:22:27
I agree with that signal too Michel Lind UTC+1 , thats the sense I got
<@gotmax23:fedora.im>
18:22:50
if we wanted external funding of BZ to happen, we'd need some sort of legal structure around it afaiu
<@conan_kudo:matrix.org>
18:22:55
I haven't been to any after-party things, so I don't know
<@nirik:matrix.scrye.com>
18:23:02
I would think it would be good to run such a thing in infra openshift... like we do many other applications managed by other groups (coreos, etc) but there would need to be a group creating a proposal/testing it in a dev env, etc.
<@amoloney:fedora.im>
18:23:07
A chance to work together 🫂 😆
<@amoloney:fedora.im>
18:25:04
So, a suggestion - I think we punt the bugzilla pt.4 until Nov 4 or 11, and in the meantime work on exploring the stg instance on Forge for the bug use cases, and I would ask someone from FESCo to be a point of contact or lead on the community bugzilla instance proposal and have some kind of idea what that effort would need to get off the ground (basically what nirik said)
<@amoloney:fedora.im>
18:25:25
Im willing to help out as well
<@nirik:matrix.scrye.com>
18:25:45
(well, dev, not staging... communishift could be used if needed)
<@salimma:fedora.im>
18:26:49
I will probably be able to make nov 11
<@salimma:fedora.im>
18:27:04
nov 4 is unlikely, anything after nov 23 also no go
<@gotmax23:fedora.im>
18:27:23
11 is fine for me, I think
<@nirik:matrix.scrye.com>
18:27:38
11 sounds best yeah
<@nirik:matrix.scrye.com>
18:27:47
(This meeting will go to 11!)
<@gotmax23:fedora.im>
18:28:40
I think Nov 11 is the earlier time so not sure if that's great for you
<@gotmax23:fedora.im>
18:28:52
but so is when it was originally scheduled for next week :)
<@salimma:fedora.im>
18:28:59
Nov 18 works too
<@salimma:fedora.im>
18:29:25
And we should close this meeting after we help Aoife settle a date
<@amoloney:fedora.im>
18:29:39
well lets aim for 11, and if more flexibility is needed to make the meeting actually useful, we can propose to move to 18
<@nirik:matrix.scrye.com>
18:30:26
sounds good to me
<@amoloney:fedora.im>
18:31:03
sorry I have one more thing but this one would really be quick. Its about the article for fedora magazine on the networkmanager thing. Im wondering if I can help wit that? Im prepping the release article and i can include a note on that, and/or reach out to the ppl working on networkmanager to see can I help them with creating their own article that I can link to from the main one as well
<@amoloney:fedora.im>
18:31:45
(I missed the part in the meeting earlier and you all had moved to a diff topic so didnt want to be rude and derail the meeting back to that one)
<@gotmax23:fedora.im>
18:32:01
Yes, I think that would be great
<@nirik:matrix.scrye.com>
18:32:17
That would be great. I asked in the ticket if the orig reporter would be interested...
<@decathorpe:fedora.im>
18:32:30
sorry, got distracted by my parents
<@amoloney:fedora.im>
18:32:40
pesky 'rents :P
<@amoloney:fedora.im>
18:33:44
ack, I will see can I help with that article then. it has tie-ins with the main release one with end users so Ill pick it up
<@zodbot:fedora.im>
18:33:56
gotmax23 gave a cookie to amoloney. They now have 118 cookies, 13 of which were obtained in the Fedora 44 release cycle
<@amoloney:fedora.im>
18:34:47
I can do the proposing for the new bz meeting too. Nov 11th provisionally? With a possible postponement, or even follow on, to Nov 18th?
<@amoloney:fedora.im>
18:35:10
I can do the proposing for the new bz meeting too. Nov 11th provisionally? With a possible postponement, or even follow on if we actually make some workable progress, to Nov 18th?
<@gotmax23:fedora.im>
18:36:18
proposed !info FRCL Bugzilla Replacement meeting is rescheduled to Nov 11th with a possible postponement, or even follow up, to Nov 18th
<@nirik:matrix.scrye.com>
18:36:37
ack
<@decathorpe:fedora.im>
18:36:41
ack
<@gotmax23:fedora.im>
18:37:00
!info FRCL Bugzilla Replacement meeting is rescheduled to Nov 11th with a possible postponement, or even follow up, to Nov 18th
<@gotmax23:fedora.im>
18:37:08
endmeeting?
<@salimma:fedora.im>
18:37:46
!endmeeting