Meeting started at 2026-07-13 15:03:05 UTC

The Meeting name is 'RELENG (2026-07-13)'

!chair nirik jnsamyak patrikp

The Meeting Name is now releng

Hello and welcome to one of many Monday meetings.

Long time no see

0/ I have screen time of more than half day, I'm hoping this days end soon :)

0/ I have screen time of more than half day, I'm hoping this day end soon :)

Anything for init from anyone? I personally don't have anything, but I have one thing for scheduled actions.

**https://forge.fedoraproject.org/releng/tickets/issues/13370#issuecomment-1061601**

Oh jednorozec had a question from migration, can we address it here?

would help me if you looked into it

In terms of points or?

leave scrum, how much effort are we talking about? what might things to look at?

I see you have written, "Forgejo doesn't publish Pagure-style AMQP messages to the messaging bus, we'd need a new trigger mechanism (such as a webhook)." Do you have a logic connection in mind for this approach?

Certainly less than with the Actions approach. The business logic already exists so the changes (probably?) wouldn't need to be super extensive, but I'm hesitant to say more. Very open ended question.

Were you able to look at it the time of investigation?

It sure can. It's just not enabled by default (which I think we should change). See: https://docs.fedoraproject.org/en-US/forge-documentation/webhook_to_fedora_messaging/

```

definitely IMHO releng org should enable this at the org level.

That's what I thought at the time but look at Kevin's comment that's right below mine...

```

That definitely makes things easier.

Yes, not a bad idea.

I can post a comment later. Anything else for init?

I'll give some more context in the comment.

If you could do it urgently would help me to look at it tomorrow. Thanks

one quick thing I want to mention: https://forge.fedoraproject.org/releng/tooling/pulls/13029 review needed here!

as mentioned I had some conversations internally with the team, and felt this a new approach would be a better way to handle it, (they do it via legacy way but it is what it is) :)

That's all from me

Nice. Do you have anything for init Kevin or can we move to the next section?

I don't have anything off hand

Any volunteers for next week?

since you took mine today (thank you) I can do next one.

Wrong copy, sorry.

So, mass rebuild week is here.

always fun

First one for me.. lol

oh nirik this reminds me of the conversation had with arrfab in the morning standup

why dont we use aws resources, and only do on prem builder usage?

maybe using both builders altogether or aws resources (I use we have sponsered instances might make it faster and less load?)

https://forge.fedoraproject.org/releng/tickets/issues/13406#issuecomment-1060740

As an aside, I'm reworking the SOP as I do the steps (and thus modifying the process), so there is a new format that I'm trying out. Draft PR with the reworked SOP is coming soon so that it can be put side by side with the comment.

The prep work should be done. I consolidated it in the first comment of the issue [1] and I keep updating it as I go along. It contains footnotes that should link to all of the relevant things. You are more than welcome to double check it and there are two PRs that are waiting for review.

In the past we did not want to build on hardware we didn't control... and koji cannot handle dynamic builders very well, so we would either have them always or have to do odd things. But also, aarch64 and x86_64 are not any kind of bottleneck. It's s390x and ppc64le that aren't available in any cloud that are.

so, it would be somewhat pointless...

Oh yeah and we need them all at one place right?

ok I might have a follow up question on this but i need to dig deeper for it, will shoot on releng channel

network does make things slower of course... you can have remote builders in some cases tho (our s390x builders are not in the same dc as the rest)

Does anything else need to be said about the rebuild at this time?

I feel like we're in a reasonably good spot.

nothing much from me

are the tools folks ready?

(gcc, glibc, bitutils)

Not sure, as the current version of the SOP says nothing about verifying that and I only got to the end of the prep section in my rework. I intend to continue, so now would probably be a good time to ask... How does one verify that? The announcement email has a link to the ticket, so is it enough to ask in the ticket? Do specific people need to be tagged?

https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/GRFRJG3X6VHX4R5QJUTKPBGJRO5AO22L/

Not sure, as the current version of the SOP says nothing about verifying that and I only got to the end of the prep section in my rework. I intend to continue, so now would probably be a good time to ask... How does one verify that? The announcement/notification email has a link to the ticket, so is it enough to ask in the ticket? Do specific people need to be tagged?

hum, they typically have a change and you can ask them in the releng ticket or the change bug... but I don't see one for f45?

gcc isn't in the driving changes for this cycle I believe.

https://fedoraproject.org/wiki/Fedora_45_Mass_Rebuild#Driving_Changes

nirik: there was no change filed for tooling folks this time.

yeah.

ok

well, they should react to the announcement if they have anything i guess

We discussed this in our call earlier today for all driving changes we need to make sure they are ok before we initiate the rebuild process.

So, how? The SOP says "do the prep changes" and then "start the rebuild", so some more context would be useful.

yeah, the changes that said they need a mass rebuild

yeah, or I can just ping them once externally to see what's happening, this is first i dont see a change tho

as explained before, ping them that's all!!

Don't double exclamation point at me, please. This should be in writing so I'm politely asking.

just make sure we have a +1 from them, for example at releng channel fabio mentioned about a blocker, I mentioned it on the ticket, they are looking into it, so make sure before rebuild you check again once, if it is ok now

:)

So the SOP should say to ping each individual owner of each driving change?

Or ask whom, exactly? And through what communication channel? This will make things hopefully better for the future too.

I'd say perhaps in their change bugzilla bug? Every approved change should have one of those...

or we could try and @mention them in the mass rebuild ticket...

We are not accountable to ping folks individually unless they mention of a blocker, we should keep check of changes that are blocking the process.

There is bugzilla but there are fedora matrix rooms too wherever you can reach out easily and get a quick answer, I dont think it is set as hard stone to do one thing, just make sure to tag/reach out to them

I am +1 on having it all in one place, in the ticket. We can try and mention them there and if they don't reply then too bad. The ticket was directly linked to in the notification email. Checking 5+ places for information at all times (mailing lists, Bugzillas, Forgejo tickets, Matrix rooms, Slack...) is no way to live.

yeah, if there's lots of them can be anoying. The tools team is usually important tho... more so than most other changes that ask for mass rebuild...

You dont have to check 5 places you can see where the conversation is going on for driving change, or where you can reach out to dedicated folks, some people might not be aware even after sending an announcement about the tracker ticket, you can point them to it. Sometime it is looking like too much of work, but once you get in habit of doing it you will realise where to look, what are important things, what are obvious blockersm and things boil down there.

So sometimes we could document the process, and put that we need to verify we are ok and unblocked doing these rebuild in documentation, specifically mentioning the places in documentation might just bring more confusion, anwyays mass-rebuild is something releng people do they are not first timers who doesnt know about community tracker change process at all, so you dont have to make it very begineer friendly

but need to make sure there is enough information to follow.\

but need to make sure there is enough information to follow.

Currently there is exactly 0, so...

The SOP just says to do the prep and the next step is starting the rebuild, so forgive me for asking for more specifics, but this is my first time actually running it.

yeah, we should modify the sop to talk about that...

It doesn't even have a section about checking in with anyone at all, there is nothing to go off of.

so just add, we need to verify if there are any blocking changes for mass rebuild, if there are we should make sure to get unblocked, and if we aren't we should be ok, this is done through communicating to folks who are driving this change.

just add a note, it doesn't require a section.

I think in the past the thought was just to have them contact releng if they had something, but more communication is good.

no worries about questions, keep them coming, I just think I have addressed these before, that's all

and I think once in the past they did have some issue and it wasn't communicated... resulting in having to rebuild things again.

https://docs.fedoraproject.org/en-US/infra/release_guide/sop_mass_rebuild/#_considerations just extend the first point here, it is already written "communicate clearly what actions are being performed" which implies the changes it is happenign for, but yes I agree we can write it better.

I'm still unsure about who to check in with. Tooling is most important? So if there are no tooling changes, I don't need to proactively check in with anyone? Just confirm that issues people actually raised (such as Fabio) are resolved before commencing?

yeah, I would say: if there's a tools change, check in with tools folks. If not, resolve any issues brought up by the change owners who needed a mass rebuild before starting.

or something like that

For this one, yes. And look out for tracker ticket, your announcement email replies and releng channel if there are any thing people addressed as blocker before rebuild

Cool, that makes things more clear for me. Thank you both for the explanation.

I will mention Fabio directly in the ticket and politely ask him to please let me know once they're good to go.

already did that, you can follow up before the mass rebuild day

Just to double check, we are to monitor releng Matrix room, reply to my notification email, and the ticket itself. These 3 places. And if nobody raises any issues there then we can assume we're good to go? Since there are no tooling changes, that is the one exception where we check in proactively.

So we don't have to do it this time but usually we would. But only tooling folks.

Good. Thank you both for sharing your experience and wisdom, we are at time here.