Meeting started at 2025-01-30 17:30:39 UTC

The Meeting name is 'fedora-ai-ml-sig'

!hello

Tim Flink (tflink)

!hi

Mohammadreza Hendiani (man2dev)

!hi

None (mystro256)

!hi

Tom Rix (trix)

!hi

Alexander Lent (xanderlent) - he / him / his

welcome everyone, let's get this party started

@trix take it away

yes, general discussion about npu this last week. it would be good to get into f43 and be a feature

not somethign i can do myself but maybe Alexander Lent can push forward

are there specific npus that you're talking about or is this a general thing?

also, is an npu different from onboard gpus? I think they are

there is intel and amd's that i know of. maybe arm.

I thought that amd's npu didn't work on linux, did that change?

part of microsoft's directML

6.14 i think will have support.

cool, glad to be wrong on that one :)

has the code been accepted yet?

it was pending when I last checked

i _think_ david a. pushed it as part of drm-next, but i may be wrong.

F43 is 6 months away.

we are all about enabling hw that hasn't hit the streets!

The ones I know of are Intel's AI Boost NPU, AMD's XDNA NPU, the Qualcomm Hexagon NPU, and a few assorted ones found in ARM SoCs.

tflink: These are discrete accelerators designed for accelerating ML/Tensor/Matrix ops generally.

I assume that qualcomm hasn't open sourced any support for their NPU, is that an option for Fedora?

The goal seems to be efficient offloading of various ML math tasks; for the first-gen Intel NPU in Meteor Lake, Intel notes that the GPU is strictly faster but the NPU consumes less power.

Last I heard, it was much like the rest of the Qcom Snapdragon X platform: Linux support is coming soon, when they get around to it

is intel's ML stack in a shape where it can use NPUs in Fedora?

ie, is it reasonable to plan for anything other than AMD's XDNA for f43?

I think so, at least if we package the Intel NPU Level Zero driver and OpenVINO, people will be able to use models through OpenVINO.

that might be a tall order, though. unless the stack has improved since I last talked to folks about it

i believe david said npu != oneapi

same a xdna != rocm

ah, ok. if it doesn't have the same kernel module and llvm problems that oneapi has, my concern may be incorrect

On the Intel side OpenVINO uses the Level Zero API to talk to the NPU driver, but other than the loader it's an entirely different driver implementation than Intel's GPU Level Zero, and it doesn't use SYCL at all.

Correction: involve SYCL at all

From my perspective it kind of seems like they shoehorned the npu operations into level zero since that was the hot new API at Intel at the time. They wrote a whole set of custom extensions to the API that do most of the neural related ops.

reaching for the antacids :|

but it sounds like the software and kernel interfaces needed to make intel's NPU work already exist upstream?

Yes, kernel module has been upstream since I think 6.6?

cool

is the proposal to add NPU support to pytorch? or something like vllm/ollama/something else I'm not thinking of?

onnx ?

I think that has npu support too

if onnx is possible, that would good, it is a more vendor neutral

I haven't yet looked into ONNX and the Intel NPU.

I'm not sure I can get the Intel NPU pytorch support ready for F43, it will require some Python and Rust deps to be packaged.

As for pytorch, support is through a custom set of Python libraries that go on top of/alongside pytorch. Namely intel-npu-acceleration-library and IPEX or ipex-llm.

ipex ? intel pytorch extension ?

so we're looking at pytorch and/or onnx for intel but that might need to wait. did I understand correctly?

yeahhh, we don't have intel gpu in basic pytorch yet.

what about for AMD? is there a new non-rocm stack?

is this it https://github.com/intel/ipex-llm

yes, i look at this when i was a hatter.

I might be more dense than usual today but I'm still having trouble figuring out exactly what the proposal is for or if it's reasonable to expect it to happen for F43.

I wonder if it would be best to save more discussion until the next meeting and start a discourse thread or wiki page to start hashing out the details before the next meeting

yes. i need to drop sooo..

if that works, do folks want to do wiki or discourse?

or something else, I don't have strong feelings. if nobody has preferences, I'll start a discource thread after the meeting

I tried updating maturin to 1.8.1 for python-safetensors was not very successful though

oh small note:

tflink: I think part of the problem with these NPU stacks, at least on Linux, is that they are all pretty custom at this point.

I'm not sure there is currently a single high-level API we could ship that covers every npu device. Microsoft is working on DirectML on their side, but support for that is still experimental on a per vendor basis, even on current Windows.

you could say the same thing about all the scientific stacks, honestly

not quite to the same extent but it's definitely a thing :)

anyhow, if there are no objections, I'll take the action item to create a discourse thread to hash out the "NPU for F43+" details

Mohammadreza Hendiani: Yeah, maturin is a large-ish project and usually requires some work on its dependencies for a new version. But - why 1.8.1? I only see `requires = ["maturin>=1.0,<2.0"]` glancing at https://github.com/huggingface/safetensors/blob/f05a1ec483c470202a3413772eef8eab6c3a8ba0/bindings/python/pyproject.toml#L92.

anything else on the NPU topic? it seems like some folks have moved on to a new topic

yeah its a really hard package to build

is there more on the maturin topic?

if i recall it was the minimal dependency that python-safetensors needed

... is that a yes?

no

im done

!undo

there are no other topics on the agenda so it's time for

anything else that folks would like to bring up?

music: Oh i remembered why it was 1.8 it just happened to be the last stable release

ok, thanks for coming, everyone