15:02:20 <pwhalen> #startmeeting Fedora ARM and AArch64 Status Meeting
15:02:20 <zodbot> Meeting started Tue Mar  2 15:02:20 2021 UTC.
15:02:20 <zodbot> This meeting is logged and archived in a public location.
15:02:20 <zodbot> The chair is pwhalen. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:20 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:02:20 <zodbot> The meeting name has been set to 'fedora_arm_and_aarch64_status_meeting'
15:02:20 <pwhalen> #chair pwhalen pbrobinson jlinton coremodule
15:02:20 <pwhalen> #topic roll call
15:02:20 <zodbot> Current chairs: coremodule jlinton pbrobinson pwhalen
15:02:26 * jlinton waves
15:02:59 <pwhalen> Good morning folks, sorry for the late start. Who's here today?
15:03:03 <pwhalen> howdy jlinton
15:03:10 <jlinton> Good morning
15:04:26 * pwhalen waits to see if we have a quorum today
15:04:41 <jsmith> Morning all
15:05:05 <pwhalen> Good morning  jsmith
15:05:08 <copperi> pwhalen: how many is that ?
15:05:33 <pwhalen> copperi: if you're here too, thats enough :)
15:05:49 <pwhalen> #topic 1) ==== Userspace Status  ====
15:06:12 <pwhalen> Any new user space issues?
15:07:13 <pwhalen> #info No issues reported.
15:07:22 <pwhalen> #topic 2) ==== Kernel Status ====
15:07:32 <pwhalen> #info kernel-5.11.2-300.fc34
15:07:32 <pwhalen> #link https://koji.fedoraproject.org/koji/buildinfo?buildID=1715703
15:08:39 <pwhalen> I've started to test this all around, not seen any new issues with it
15:08:44 <jsmith> I'll do some testing on the latest image over the next couple of days, as I have a couple of new installs to test.
15:08:48 <jlinton> I saw the stuck shutdown thing with an early 4.12 build, if it keeps doing it I will investigate open a bz.
15:09:23 <jlinton> But really there probably should be some kind of systemd bz too, because it seems the timeouts just keep getting extended past their deadlines.
15:09:53 <pwhalen> jlinton: I've seen some others report similar. Not really looking at 5.12 much myself. This is in rawhide?
15:10:06 <pwhalen> the systemd issue, is that rawhide?
15:10:10 <jlinton> Yes, local built/fedora kernel
15:10:34 <jlinton> on rawhide
15:10:45 <pwhalen> ok. I'll take a look at the rawhide composes in openqa too.
15:11:07 <pwhalen> jsmith: which hardware do you do testing on?
15:13:24 <pwhalen> #info Please test and report any issues to the list or #fedora-arm.
15:13:33 <pwhalen> #topic 3) ==== Bootloader Status ====
15:13:41 <pwhalen> #info uboot-tools-2021.04-0.3.rc2.fc34
15:13:41 <pwhalen> #link https://koji.fedoraproject.org/koji/buildinfo?buildID=1711269
15:14:00 * nirik arrives late.
15:14:04 <pwhalen> Anyone have issus with uboot
15:14:17 * pwhalen hides from nirik
15:15:24 <pwhalen> For uboot I've seen issues pxe booting on the Wandboard. Usb not working.
15:15:56 <pwhalen> usb not working I think was on the rpi4
15:16:47 <pwhalen> #info Please test and report any issues to the list or #fedora-arm.
15:17:37 * pbrobinson is late
15:18:08 <pbrobinson> new u-boot RC landed this morning
15:18:08 <pwhalen> #topic 4) ==== Fedora 34 ====
15:18:09 <jlinton> There is a new pftf build, the usb should be working cross soc/etc, this is the first version that starts to support the rpi400/etc
15:18:36 <pwhalen> oh nice, will check it out. Any progress on the pxe boot stuff?
15:18:59 <jlinton> I duplicated your problem a few weeks ago, bit of a mystery, and I left it to do "real" work.
15:19:00 <pbrobinson> use HTTP boot, PXE is insecure?
15:19:10 <jlinton> I think the http boot has the same problem.
15:19:19 <pbrobinson> shitty UEFI driver?
15:19:20 <jlinton> its dropping the host IP at some point
15:19:25 <jlinton> maybe
15:19:47 <jlinton> s/driver//g
15:19:50 <jlinton> lol
15:19:59 <pwhalen> hey!
15:20:03 <pbrobinson> seen that on a few vendor drivers, was even on one vendor bug when they updated the uefi vendor driver and the problem went away
15:20:31 <pwhalen> #info Latest nominated compose
15:20:31 <pwhalen> #link https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test
15:20:42 <pwhalen> #info OpenQA testing:
15:20:42 <pwhalen> #link https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=34&build=Fedora-34-20210227.n.0&groupid=5&groupid=1
15:20:51 <pbrobinson> on f34 kernel I have some patches I need to push, should get them done tomorrow
15:21:02 <pbrobinson> fixes across a few devices, rpi, jetson, etc
15:21:03 <pwhalen> included x86 there to see a comparison
15:21:06 <jlinton> I still need to land my config changes too
15:21:19 <pbrobinson> jlinton: can you co-ordinate them with me please
15:21:20 <pwhalen> pbrobinson: music to my ears
15:22:26 <pwhalen> #info systemd-oomd.service: Failed with result 'core-dump'
15:22:26 <pwhalen> #link https://bugzilla.redhat.com/show_bug.cgi?id=1930875
15:23:14 <pbrobinson> and for f34 uboot there's a new RC out so I'll do a build soon, but that likely won't be until the weekend
15:23:35 <pbrobinson> I have a sniff of issues around the PBP but that's going to need a deep dive
15:23:40 <jlinton> Yah, I waited a week and everything changed, so I need to get the right people on CC
15:23:47 <pwhalen> #info Resolved with upgrade to systemd-248~rc2-1.fc34, but proper fix posted.
15:24:20 <pwhalen> #info Raspberry Pi 3B+ boots incredibly slow after CPU failed to come online errors
15:24:20 <pwhalen> #link https://bugzilla.redhat.com/show_bug.cgi?id=1921924
15:25:17 <pwhalen> That rpi3 bug aarch64 only, armv7 worked with all cpus
15:25:30 <pbrobinson> interesting on my rpi3+ which was a system-upgraed from f33 doesn't show the issue with aarch64
15:26:04 <pbrobinson> I need to upgrade my original rpi3 aarch64 system to further test
15:26:26 <pbrobinson> and I'll do a clean deployment of them too on a different card when I get a sec
15:26:37 <pwhalen> huh.. maybe fixed? I'll test today as wlel
15:26:51 <pwhalen> #info [abrt] gnome-shell: nouveau_fence_signalled(): gnome-shell killed by SIGSEGV
15:26:51 <pwhalen> #link https://bugzilla.redhat.com/show_bug.cgi?id=1930977
15:26:51 <pwhalen> #info Accepted F34 Beta Blocker
15:27:27 <pwhalen> #info [abrt] xorg-x11-server-Xorg: System(): Xorg killed by SIGABRT
15:27:27 <pwhalen> #link https://bugzilla.redhat.com/show_bug.cgi?id=1930978
15:27:29 <pbrobinson> is there someone assigned to work on it?
15:28:11 <pwhalen> there is, but not seen any movement
15:28:47 <pwhalen> rather large update went out too, I'll test again
15:29:01 <jlinton> The second one seems to be pretty strongly related to the first, since nouveau continues to work on my gt710
15:29:21 <pwhalen> jlinton: agreed, likely a dupe and should be closed
15:30:50 <pwhalen> Others issues I've seen  - pine64 no usb in uboot (will check the next RC), Wandboard pxe not working, hangs after attempting
15:31:13 <pwhalen> Latest rpi fw (20210225-1.5985247.fc34) broke boot on rpi4-4GB, rainbow then nothing.
15:31:35 <pwhalen> Not sure if one of the config options needs adjusting
15:31:36 <pbrobinson> I will revisit that, I suspect I know why, won't likely be until the weekend
15:33:00 <pwhalen> ah, and this morning I tested armhfp xfce on the rpi3, tosses errors about CMA has graphical issues
15:33:13 <pwhalen> [drm:vc4_bo_create [vc4]] *ERROR* Failed to allocate from CMA:
15:34:20 <pwhalen> Need to look at that more after the meeting.
15:34:45 <pwhalen> Any other issues not mentioned?
15:35:15 <jsmith> I'll retest this week to see if https://bugzilla.redhat.com/show_bug.cgi?id=1892329 is still an issue
15:35:18 <pwhalen> I dont have a Rockpro64, if someone does it would be great to get that tested too
15:35:18 * nirik wants to bring up his fav bug quickly in open floor. :) (sorry pwhalen :)
15:36:09 <pbrobinson> I removed the CMA allocation in the kickstarts because it had another problem, and I thought that the rpi was allocating CMA based on the firmware DT now. We might need to assign a CMA within the config.txt
15:36:12 <pwhalen> no worries nirik, I dont really have an update on it, tried to get on to the builder but authentication was failing, need to try again
15:36:27 <pbrobinson> nirik: the builders?
15:36:35 <nirik> yeah, auth in stg has been... in flux due to auth system testing.
15:36:46 <pwhalen> pbrobinson: right, I noticed that, but it does allocate 64M
15:37:04 <pwhalen> (noticed it was removed from the cmdline)
15:37:06 <pbrobinson> pwhalen: 64Mb is the kernel default allocation
15:37:10 <nirik> yeah, the armv7 oom killing things too much with a lpae kernel... ( https://bugzilla.redhat.com/show_bug.cgi?id=1920183 )
15:38:28 <pbrobinson> nirik: so we dropped it back to the original 24G right, but we couldn't really debug to work out where it was introduced because of the other issue where we lost a change with the ARK kernel change?
15:38:35 <pbrobinson> what did I miss?
15:39:14 <nirik> yeah, so I setup buildvm-a32-01.stg to try and isolate it, but auth has been wonky there, so pwhalen hasn't been able to get in...
15:39:15 <pwhalen> pbrobinson: ok, so it might need to be readded
15:40:00 <nirik> pwhalen: I can work with you sometime out of meeting to get access one way or another to that vm. ;) Also, would it be worth trying 5.11/5.12 now?
15:40:03 <pwhalen> nirik: I will try again today
15:40:54 <pbrobinson> nirik: probably worth trying 5.11, there was some changes there which may assist
15:41:05 <pwhalen> I was thinking about 5.11 too, wouldnt hurt. 5.12 I havent tested too much
15:42:08 <nirik> ok.
15:42:10 <nirik> thanks
15:43:39 <pbrobinson> let's start with 5.11 and iterate from there
15:44:46 * smooge catches up
15:45:05 <pwhalen> I cant reproduce outside of koji, not sure if it was reproduced in staging yet
15:45:11 <pwhalen> staging seems down
15:46:02 <pwhalen> #topic 5)  == Open Floor ==
15:46:19 <nirik> it has reproduced there in the past, will check it again today/soon
15:46:43 <jlinton> Anyone else notice/bothered we seem to have significantly increased our kernel build times over the last couple years?
15:46:56 <jlinton> I'm guessing its something like 3x+
15:47:08 <pwhalen> ouch, arm specifically?
15:47:14 <jlinton> Yah, aarch64
15:47:32 <jlinton> I'm wondering if we should consider some kind of "diet"
15:47:59 <pbrobinson> jlinton: it varies a lot depending on the builder you land on, some I see is slow, other times they're quicker than x86
15:48:24 <pbrobinson> jlinton: I vote we put it on a diet by not building any of the rpi support
15:48:29 <jlinton> This is mostly based on my 'local' TX2 build times, not so much koji.
15:49:52 <jlinton> lol (re rpi), yah i'm the guy who wants to turn everything on, but i'm wondering if there is a way to know how many of these modules are really being used.
15:50:17 <jlinton> Vs how many we get from fedora common.
15:51:10 <pbrobinson> I've been turning, or trying to turn, off a bunch of stuff in ARK overall, like legacy crap we don't care about such as the PS2 stack
15:51:23 <jlinton> (and some of the slowness, has been things like enabling compressed modules/etc)
15:51:25 <pbrobinson> it takes time because people don't review/ACK stuff
15:51:36 <jlinton> put me on CC?
15:52:01 <pbrobinson> I can do, Mark and Al and others are on there, the more the merrier ;-)
15:52:17 <pbrobinson> I need to rebase that one
15:52:21 <jlinton> although my gitlab notifications seem to be getting eaten at least part of the time.
15:52:41 <pbrobinson> the ark CI/email bridge thing is a trash fire
15:52:59 <pbrobinson> I'll just ping ones to you directly
15:54:15 <pwhalen> Anything else for the meeting?
15:54:47 <pwhalen> #endmeeting