Meeting started at 2026-08-27 16:30:34 UTC

The Meeting name is 'fedora-ai-ml-sig'

!hi

Tom Rix: Tom Rix (trix)

!hi

!hi

Carol Chen: Carol Chen (cybette) - she / her / hers

tflink: Tim Flink (tflink)

!hi

Mohammadreza Hendiani: Mohammadreza Hendiani (man2dev)

ok, let's get this party started

!hi

Onuralp S.: Onuralp SEZER (thunderbirdtr) - he / him / his

Tom Rix: this one is yours

working on the 2.13 has stalled because of libstdc++ update conflicting with rocm , breaking this and rocfft.

in f45 and rawhide, yes?

yes that happened today in f45

see my ranting on ai/ml channel.

is there more to talk about or shall we move on?

we can move on, as the tides comes, so do last minute toolchain updates come to fedora releases.

next bit is ROCm 10 has landed in the public.

this may or may not be a big deal, i am treating it as a big deal until i know otherwise.

https://fedoraproject.org/wiki/Changes/ROCm_8.x

Is there huge change?

for some reason, they changed rocm 8 to rocm 10

should be renamed to 10. crazy amd guys with their naming :)

so whatever was said about rocm 8 in the past is now rocm 10

yes.

The only news I heard about it was that some new target gpu got added.

from an ABI POV, I don't think there is a huge change when compared to 7.2. there shouldn't be much change from 7.14. the biggest changes are release cadence and support model AFAIK

it may just be maketing, but major version changes are a license to make breaking abi changes. we need to manage abi breakage

note the "should" in what I said. we haven't confirmed the size of changes yet

that's what i will be doing asap. 10 is going into this copr

the fact that libabigail seems to be crashing on the 7.14 debuginfo packages isn't helping that check go any faster :-/

https://copr.fedorainfracloud.org/coprs/g/rocm-packagers-sig/preview/packages/

yes libabigail is crashing on debug info.

have you filed a bug against libabigail? I was about to

i was working around it.

claude seems to think it's due to some change in DWARF in new GCC or LLVM

i did file a bug against fedora-review not working :(

anyhow, I'm bringing the meeting into a rathole. we can continue that conversation offline

https://fedoraproject.org/wiki/SIGs/HC#Testing

has the 'pkgdiff' tests. i used those on 7.14, so we have got that covered, if not completely documented.

it will be a matter of using them once 10 is in the copr. it will compare that copr package to rawhide package.

i sorta kitchen sinked the pkgdiff test. if there is some other pkg tests i missed, send me a pr and i'll get the changes into the set of dockers. the sooner the better.

ok, I'll follow up with you offline about making progress there. I just started poking at abi checks for rocm packages the other day

cool.

anything else on this?

maybe we can sic some ai on the analysis

other than the new need to re-name that rocm8 feature to rocm10?

yeah.

but that won't be until F46 so we have a bit of time

yes. and we have the compat set already in place if things go badly.

cool, any other topics for today? i have one short one that can wait until the end

all i got, thanks

ok, I'll do my quick topic, then

this is a quick one and mainly a "my bad" kind of thing

I didn't realize how many open PRs were out there and some tickets waiting for feedback

apologies for the delays there

that's all, feel free to participate on those tickets/PRs if you'd like it wasn't a "tflink is the only one who can comment" kind of thing

that's all, feel free to participate on those tickets/PRs if you'd like. it wasn't a "tflink is the only one who can comment" kind of thing

with that, we can move on to ...

any other items to discuss today?

Has anybody looked ito the licensing of Nemotron 3 family of models, evidently the new NVIDIA models are completely end-to-end open source. I'm not talking about just the weights on the model, I'm talking about even the data set or how to reproduce it. So theoretically it can be remade or it can be shipped as well, but I still haven't had the chance to look into all the licensing to confirm it, and make sure it's actually open source, but at least at least from the outside, it looks like Apache 2.

I have not

If true, theoretically this could be the first model that we could ship as a package in Fedora.

Which I expect would be very interesting to see how rpm would react to

AMD released a more open model a while back, it had training data and everything needed to build from scratch IIRC but it's been a while. The biggest problem I ran into was that the required HW to train it from scratch was very much non-trivial. I suspect that any open model might run into the same HW requirement problems

I don't think that any part of Fedora has addressed the question of whether shipped models would have to be trained in-house and whether that could be done outside of koji

Mohammadreza Hendiani: sorry if it is stupid question if license okay, shipping how exactly ?

Yeah, the prices are insane. The closest thing that I can think of that might be able to build it and is the cheapest option, is the Mac Mini, I guess.

the HW requirements for the AMD model was something along the lines of 8x MI210 at the low end, IIRC

I think the question comes down to shipping binaries and whether open weights are acceptable. I don't think that any distro has really looked at shipping open weights yet but I could be wrong

unless I'm misunderstanding your question

Well, theoretically, we would have to put it in RPM, or we could just make a DNF extension to call hugging face package clint but then we would need to have a hooking face compatible interface running on Fedora

for training models, my opinion is models should be trained in house, so we don't get stuck with a blob that may or may not work release to release.

My 2 cents opinion would be I wouldn't ? Most people use quantized and choose smarter via cli or tool better also agreed with Tom not every model work out of box. Support takes time or deals needs to be made so ship in day zero

But thank you for answering

No, you got my question correct. I don't want to, ship something that hasn't been reviewed and remade by us also, but I don't see us having the infrastructure for it unless some company sponsors us to do it

may need to up the 2 day timeout on koji 😅

I would say more than 2days. :)

yeah, there is a real question about whether it would be worth it and whether it would be approved. anything built and shipped like that would probably not be at the top of any leader boards and would be difficult to update

I don't think that there is any realistic way to have koji train models from scratch

A way of provide I would say (example install nvidia driver just meta package but download or tools that download blob/data but core package is there)

Also train %100 nope

I'm not saying that it couldn't be done, I just don't think that the kind of surgery/grafting into koji that would be required to have the HW needed to train models isn't realistic

I think the real value proposition comes after you build it once, because then fine-tuning it, to my understanding, is not that expensive as rebuilding it.

I've heard of Nemotron but haven't looked into how "open" it is. The license is this: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/ which is not a recognized open source license like Apache (which I believe the AMD models were released under)

That's why I want to know, can we get access and use external hardware for such thing as packaging or in this case, I guess, model creation? The only other external hardware that we have access to, to my knowledge, is the AWS servers.

if there is interest from Fedora as a whole, we could start looking into this but there are a lot of questions that would need to be addressed and proposals that would have to happen

and we'd have to find a sponsor with deep pockets to get access to the HW needed to create some kind of "fedora model"

which would almost definitely be borrowed stuff in the cloud

tflink: well I would ask/say we need datasets first (if not ready )

lemme look for the AMD released model I looked at a while back. I think that included datasets and scripts to train

The dataset of nemotron 3 is on hogging face.

Side note I have 2dgx in house can do try depends on model but for tooling of course not gonna be under fedora system but I can offer help at least

https://www.amd.com/en/developer/resources/technical-articles/introducing-amd-first-slm-135m-model-fuels-ai-advancements.html

the link to sources etc. is at the bottom of the page

there is a list of models here that may be a similar kind of open but I haven't looked into them in detail: https://www.amd.com/en/developer/resources/open-source-models.html

Just a quick look and I'm already finding a 404 page for one of the databases it uses.

the newest model listed "was trained on 64x MI300 GPUs" yikes

I felt poor in my gpu :)

there may be updated links, it's been a while and I doubt that the datasets disappeared entirely. it was released 2 years ago

Well, that's cool. I'll take a look at it.

anything else on this or other topics for the remaining 15 minutes?

Searching around for licenses, I just saw this: https://openmdw.ai/

Looks like the latest Nemotron 3 Ultra is released under the openMDW (previous ones under Nvidia's nemotron license)

I wondered if anything was using it yet but you already answered that question :)

quite a few, apparently: https://huggingface.co/models?license=license:openmdw-1.1&sort=trending

heh I just copied the same link

yep

anyway, just commenting, nothing much to contribute to the meeting otherwise 😆

thanks for the input

it seems like there are no other topics, so I'll close out the meeting

thanks for coming, everyone