17:00:41 <adamw> #startmeeting F39 Final Go/No-Go meeting
17:00:41 <zodbot> Meeting started Thu Oct 26 17:00:41 2023 UTC.
17:00:41 <zodbot> This meeting is logged and archived in a public location.
17:00:41 <zodbot> The chair is adamw. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
17:00:41 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:00:41 <zodbot> The meeting name has been set to 'f39_final_go/no-go_meeting'
17:00:41 <adamw> 
17:00:42 <adamw> #meetingname F39-Final-Go_No_Go-meeting
17:00:42 <zodbot> The meeting name has been set to 'f39-final-go_no_go-meeting'
17:00:47 <nirik> morning
17:00:51 <thebeanogamer> Evening
17:00:52 <adamw> #topic Roll Call
17:00:57 <thebeanogamer> .hello thebeanogamer
17:00:59 <zodbot> thebeanogamer: thebeanogamer 'Daniel Milnes' <daniel@daniel-milnes.uk>
17:01:18 <sgallagh_> .hello sgallagh
17:01:19 <zodbot> sgallagh_: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
17:01:22 <nirik> Apparently RC-1.3 will not be joining us today. ;)
17:01:30 <pboy> .hi
17:01:31 <zodbot> pboy: pboy 'Peter Boy' <pboy@uni-bremen.de>
17:01:52 <frantisekz> .hello2
17:01:53 <zodbot> frantisekz: frantisekz 'František Zatloukal' <fzatlouk@redhat.com>
17:03:09 <coremodule> .hello2
17:03:11 <zodbot> coremodule: coremodule 'Geoffrey Marr' <gmarr@redhat.com>
17:03:12 <adamw> #info nirik will represent release engineering
17:03:27 <adamw> #info sgallagh_ will represent fesco
17:03:37 * kparal is here as part of Quality
17:03:43 <adamw> #info kparal will represent QA
17:03:45 <decathorpe> .hi
17:03:46 <zodbot> decathorpe: decathorpe 'Fabio Valentini' <decathorpe@gmail.com>
17:03:51 <adamw> quality...q
17:04:09 * sumantrom is here
17:04:20 <neil> .hi
17:04:21 <zodbot> neil: neil 'Neil Hanlon' <neil@shrug.pw>
17:04:26 <jednorozec> .hello humaton
17:04:32 <zodbot> jednorozec: humaton 'Tomáš Hrčka' <thrcka@redhat.com>
17:04:46 <geraldosimiao> .hello geraldosimiao
17:04:47 <zodbot> geraldosimiao: geraldosimiao 'Geraldo S. Simião Kutz' <geraldo.simiao.kutz@gmail.com>
17:06:32 <Son_Goku> .hello ngompa
17:06:33 <zodbot> Son_Goku: ngompa 'Neal Gompa' <ngompa13@gmail.com>
17:08:41 <adamw_irc> sigh, sorry folks
17:08:58 <frantisekz> now, there are two of them!
17:09:04 <thebeanogamer> This is getting out of hand
17:09:12 <frantisekz> :D
17:09:29 <adamw> #info amoloney is busy ATM so I will run the meeting for now, we may hand off when she's back
17:10:04 <adamw> in case anyone didn't know, amoloney is the new Operations Architect (that is *absolutely* a different thing from Program Manager) and will be taking this kinda thing over going forward
17:10:23 <adamw> #chair nirik
17:10:23 <zodbot> Current chairs: adamw nirik
17:10:26 <adamw> just in case i drop off again
17:10:31 <frantisekz> new bcotton u say?
17:10:52 <adamw> new and, because of irish accent, clearly improved
17:11:01 <sgallagh_> adamw: No notes.
17:11:18 <adamw> okay, let's get rolling with some exciting boilerplate
17:11:20 <adamw> #topic Purpose of this meeting
17:11:20 <adamw> #info Purpose of this meeting is to check whether or not F39 Final is ready for shipment, according to the release criteria.
17:11:20 <adamw> #info This is determined in a few ways:
17:11:20 <adamw> #info 1. Release candidate compose is available
17:11:21 <adamw> #info 2. No remaining blocker bugs
17:11:23 <adamw> #info 3. Test matrices are fully completed
17:11:25 <adamw> #info 4. Fedora CoreOS and IoT are ready
17:11:30 <adamw> #topic Current status - RC
17:11:49 <adamw> #info we have, not just one, but *two* exciting RCs!
17:12:14 <adamw> #info RC-1.1 had no attempt to address the ARM blockers, RC-1.2 has a rollback of uboot-tools to an earlier version
17:12:28 <adamw> so, since we do have RCs, let's move on.
17:12:29 <adamw> #topic Current status - blockers
17:12:29 <adamw> #link https://qa.fedoraproject.org/blockerbugs/milestone/39/final/buglist
17:12:31 <nirik> RC-1.3 sank into the swamp. ;(
17:12:37 <adamw> here's the fun part...
17:12:50 <adamw> we have some stuff to discuss, unfortunately, so let's do mini blocker review.
17:13:21 * adamw waits for his pet blockerbugs instance to sync
17:14:13 <adamw> #topic (2246385) F39 RC1.2 images have sometimes "RC" in their name
17:14:13 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=2246385
17:14:13 <adamw> #info Proposed Blocker, distribution, NEW
17:14:27 <adamw> so, we had a bit of a booboo with the pungi config and we got bad filenames.
17:14:43 <adamw> RC-1.3 was supposed to fix this, but as nirik noted, it burned down and fell in the swamp.
17:14:44 <nirik> I think this is a blocker...
17:15:13 <nirik> it would cause a lot of confusion...
17:15:53 <geraldosimiao> all of these?
17:15:55 <geraldosimiao> Fedora-Everything-netinst-x86_64-39_RC-1.2.isoFedora-Kinoite-ostree-x86_64-39_RC-1.2.isoFedora-Onyx-ostree-x86_64-39_RC-1.2.isoFedora-Sericea-ostree-x86_64-39_RC-1.2.isoFedora-Server-dvd-x86_64-39_RC-1.2.isoFedora-Server-netinst-x86_64-39_RC-1.2.isoFedora-Silverblue-ostree-x86_64-39_RC-1.2.iso
17:16:04 <geraldosimiao> oh my...
17:16:05 <nirik> and all the checksum files
17:17:07 <thebeanogamer> I'd probably agree with blocker, and if all we need to do is fix some Pungi config then it shouldn't really slow us down
17:17:11 <adamw> we *could* manually rename them and redo the checksum files, but it seems like a lot of work that someone might get wrong.
17:17:12 <nirik> As adamw points out we could rename and resign all the checksum files. I don't think there's RC in any of the actual content... just the media filenames. That would be anoying, but we could do it.
17:17:20 <nirik> or we could do a rc-1.4
17:17:40 * dustymabe waves
17:17:47 <geraldosimiao> do we have a criterion on this?
17:17:53 <kparal> what about some composeinfo files or something like that? won't those be broken?
17:17:54 <sgallagh_> If it was some of the less-used Spins, I might handwaive it
17:17:58 <jednorozec> the config change is trivial compared to renaming and resigning
17:18:14 <sgallagh_> But Everything, Server and Silverblue are too important
17:18:29 <thebeanogamer> If we're not sure where it's in use, my preference would be for a new RC
17:19:08 <thebeanogamer> Seems safer than a best guess at renaming
17:19:16 <sgallagh_> A new RC would mean slipping, no?
17:19:52 <nirik> quite possibly yeah...
17:20:10 <geraldosimiao> how many hours? to respin?
17:20:41 <nirik> 4ish
17:20:48 <geraldosimiao> we have already blocker reviews where we suspend the meeting to wait for new iso...
17:20:56 <pboy> Last time we made a break of 4 hours or so and decide after some smoke tests.
17:20:58 <geraldosimiao> oh.. 4 hours...
17:21:16 <geraldosimiao> for me here is ok.
17:21:23 <nirik> we could try that.
17:21:31 <sgallagh_> Maybe we should figure out first if we think the fixes from RC1.2 are valuable enough not to ship 1.1?
17:21:42 <nirik> But might be good to settle on what blockers there are first. ;)
17:21:44 <adamw> well
17:21:56 <adamw> we should probably just vote on the other blocker first
17:22:03 <adamw> i should've put that one up before this one, really. let's do that
17:22:13 <Southern_Gentlem> .meetings
17:22:13 <adamw> #info we're gonna consider the other blocker first as it's a clearer case, and come back to this
17:22:23 <frantisekz> (nirik: can we have rc 1.4 spinning in the background just in case?)
17:22:25 <adamw> #topic (2246410) Failed media check immediately disappears on bare metal, shows a black screen instead
17:22:25 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=2246410
17:22:26 <adamw> #info Proposed Blocker, dracut, NEW
17:22:33 <adamw> frantisekz: let's not, because reasons
17:22:37 <frantisekz> okey
17:22:43 <adamw> (the soas thing, and this)
17:23:00 <adamw> okay, so, yeah, this one has a criterion. and seems to be reproducible (I can reproduce it too).
17:23:04 <nirik> we could, but lets wait and do this deliberately. ;)
17:23:29 <adamw> i can't really see any way out of a +1 for this one, honestly.
17:23:42 <thebeanogamer> Agreed
17:23:47 <nielsenb> FinalBlocker +1
17:24:15 <nirik> yeah... and it's not clear whats causing it.
17:24:18 <nirik> did beta have it?
17:24:22 <geraldosimiao> I gues this can go with the combo "last minute/difficult to fix" waive formula...
17:24:38 <kparal> I can test Beta, if you give me 5 minutes
17:24:49 <kparal> FinalBlocker +1
17:24:50 <nirik> I wonder if it's a kernel / simpledrm issue?
17:24:59 <nirik> but yeah, FinalBlocker +1 I fear.
17:25:38 * thebeanogamer quietly hopes the bitflip we're testing this with is just in a very unfortunate place in the ISO, but doubts it
17:25:59 <kparal> thebeanogamer: no, I actually discovered it when it flipped elsewhere
17:26:17 <jforbes> nirik: aren't we still in grub at that point?
17:26:47 <adamw> i don't think it's really a 'graphics' issue
17:26:52 <adamw> jforbes: no
17:26:57 <adamw> this is implemented in dracut
17:27:12 * Southern_Gentlem wonders whose turn was it to put kparal in the closet
17:27:34 <adamw> https://github.com/dracutdevs/dracut/blob/master/modules.d/90dmsquash-live/dmsquash-live-root.sh#L69
17:28:30 <nirik> there were some plymouth changes in late sept...
17:28:40 <nirik> oh, of last year
17:28:46 * nirik cleans his glasses
17:28:47 <kparal> nirik: Beta is also affected
17:28:54 <geraldosimiao> SouThern_Gentlem good point
17:28:57 <nielsenb> Whoops
17:30:13 <kparal> ah, when I add nomodeset, the bug is not triggered
17:30:15 <adamw> (despite the 'live' bit in the code i think it's actually used on installer images too)
17:30:17 <nirik> soooo... do we want to wait a bit while people frantically investigate and see if we can get a fix?
17:30:19 <Southern_Gentlem> kparal, is this happening on certain equipment ? i have nt seen it and no one has complained
17:30:20 <adamw> oh, that's interesting.
17:30:33 <adamw> Southern_Gentlem: it happens on every system we've checked so far...
17:30:42 <kparal> Southern_Gentlem: so far, everybody reproduced it on their home machines
17:30:42 <sgallagh_> But only on bad media?
17:30:50 <adamw> two of kparal's systems, two of cmurf's, one of mine
17:31:05 <adamw> sgallagh_: yes. well, i mean, you only hit this path where it should stop and show a message if the medium is bad.
17:31:10 <adamw> if the medium is good it just...keeps booting.
17:31:34 <adamw> Southern_Gentlem: have you actually tried booting a medium which fails the check?
17:31:56 <kparal> fwiw, the percentage counting seems to be always displayed correctly. Only when the counting stops, the display goes  black instead of staying on.
17:32:22 <Southern_Gentlem> since i have already have cheched the checksum i skip most of the time
17:33:07 <sgallagh_> Right, so it defeats the purpose of the check... but honestly how likely is it to actually be broken?
17:33:20 <adamw> we literally have a criterion that specifically says this has to work
17:33:31 <Southern_Gentlem> so its saying the media is broken even if it is not ?
17:33:38 <adamw> https://bugzilla.redhat.com/show_bug.cgi?id=2246410#c3
17:33:46 <sgallagh_> Southern_Gentlem: No, if it's broken it fails to report that.
17:33:51 <sgallagh_> And just goes black
17:33:52 <adamw> Southern_Gentlem: no. when the media is actually broken, instead of a message telling you so, you see a black screen.
17:34:23 <Southern_Gentlem> truefully i think we can waive this and put in common bugs
17:34:43 <Southern_Gentlem> its not that major
17:35:02 <adamw> there. is. literally. a. release. criterion. that. says. this. is. a. blocker.
17:35:13 <adamw> we have release criteria for a reason: so we don't just make stuff up as we go along
17:35:15 <Southern_Gentlem> too late to fix
17:35:21 <adamw> ah, okay
17:35:22 <Southern_Gentlem> :0
17:35:25 <adamw> yes, we could potentially do that.
17:35:28 <kparal> It's definitely a blocker, but I agree we could waive it under the late blocker exception
17:35:31 <adamw> but that's a separate point in the meeting
17:35:35 <adamw> right now, we are voting on whether it's a blocker
17:35:38 <sgallagh_> Right, I don't think anyone's disagreeing on blocker status.
17:35:43 <sgallagh_> +1 for the record
17:35:43 <Southern_Gentlem> +1 fb
17:35:44 <adamw> okay
17:35:47 <Son_Goku> yeah, this is definitely blocker +1 FB
17:35:55 <sumantrom> +1 fb
17:35:55 <thebeanogamer> FinalBlocker +1
17:36:04 <geraldosimiao> So, first accept as a blocker, then waive it as too late and difficult to fix...
17:36:10 <jednorozec> FB +1
17:36:11 <geraldosimiao> FB +
17:36:21 <geraldosimiao> FB +1
17:36:25 <adamw> proposed #agreed 2246410 - AcceptedBlocker (Final) - this is accepted as a clear violation of "Validation of install media must work correctly for all release-blocking images. ... the installer's mechanism for verifying that the install medium is intact must complete successfully if the medium is correctly written and return a legible failure message if it is not."
17:36:29 <farribeiro> Fb +1
17:36:31 <Son_Goku> we can shunt it to F40 though once we waive it?
17:36:36 <nielsenb> ack
17:36:38 <adamw> if we waive it. that would be the result, yes.
17:36:40 <Son_Goku> adamw: ack
17:36:45 <geraldosimiao> ack
17:36:47 <jednorozec> ack
17:36:51 <farribeiro> Ack
17:36:53 <nirik> ack
17:36:55 <Southern_Gentlem> Son_Goku,  no we need it fixed for the respins
17:36:56 <kparal> ack
17:36:57 <Southern_Gentlem> ack
17:37:01 <sgallagh_> ack
17:37:08 <Son_Goku> Southern_Gentlem: oh right yes...
17:37:09 <frantisekz> ack
17:37:15 <adamw> okay, let's go back to the other proposed blocker, then.
17:37:24 <geraldosimiao> yeah, fized for respins woukd be great
17:37:26 <adamw> #topic (2246385) F39 RC1.2 images have sometimes "RC" in their name
17:37:26 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=2246385
17:37:26 <adamw> #info Proposed Blocker, distribution, NEW
17:37:28 <geraldosimiao> fixed
17:37:29 <Southern_Gentlem> but if its in Gold then if the respins have so be it
17:37:42 <nirik> FinalBlocker +1 from me.
17:37:46 <sgallagh_> Point of clarification: why is this a Final blocker and not a Beta blocker?
17:37:48 <Son_Goku> FinalBlocker +1
17:37:55 <adamw> sgallagh_: because we're at final.
17:37:57 <sgallagh_> (The criterion)
17:38:02 <Son_Goku> sgallagh_: because RC isn't supposed to be in any compose we choose to make "Final"
17:38:06 <adamw> oh, the media check criterion? dunno.
17:38:06 <Southern_Gentlem> so are we on 1.2 or 1.3
17:38:11 <adamw> or are you talking about this one?
17:38:18 <sgallagh_> Sorry, I was still talking about the checksum one
17:38:35 <sgallagh_> I went off to find it and we changed topics in the meantime
17:38:36 <adamw> sgallagh_: i dunno off the top of my head.
17:39:05 <adamw> so, do we have any blocker rationale for this?
17:39:14 <Southern_Gentlem> so are we on 1.2 or 1.3
17:39:20 <sgallagh_> I was about to ask: is there actually a criterion for this?
17:39:50 <adamw> i don't *think* there is
17:39:52 * nirik looking
17:40:03 <adamw> well
17:40:04 <geraldosimiao> didn't found criterion on this
17:40:08 <adamw> we...could crowbar https://fedoraproject.org/wiki/Basic_Release_Criteria#Self-identification to fit
17:40:14 <adamw> "Any component which prominently identifies a Fedora release version number, code name, milestone (Beta, Final), or Edition (Workstation, Server, CoreOS) must do so correctly."
17:40:26 <adamw> that's...not...really what we meant it for, but...it's sorta...
17:40:29 <Southern_Gentlem> if we release will it be 1.2 or the 1.3 isos
17:40:37 <adamw> Southern_Gentlem: there is no 1.3. it exploded.
17:40:38 <thebeanogamer> What does the /etc/os-release look like on these images?
17:40:42 <Southern_Gentlem> ty
17:40:49 <sgallagh_> thebeanogamer: unrelated
17:40:53 <adamw> thebeanogamer: it's correct on the images openqa tests.
17:40:54 <kparal> I thought we had some requirement that the infra artifacts are in proper state, or something?
17:41:07 <adamw> sgallagh_: well, i know where he's going with it - we *do* have criteria for os-release, so if that was wrong, it'd let us block
17:41:09 <adamw> but i think it's right
17:41:10 <thebeanogamer> sgallagh_: I ask because it's mentioned in the wording of that blocker criterion
17:41:38 <thebeanogamer> Though it's under "Also", so we don't technically have to
17:41:38 <sgallagh_> Sorry, bad choice of word.
17:41:41 <Southern_Gentlem> what isos do not have RC
17:41:45 <adamw> kparal: i can't find anything to that effect
17:41:49 <thebeanogamer> Arguing semantics this close to release scares me though
17:41:52 <nirik> "A correct checksum must be published for each official release image. "
17:42:07 <nirik> they aren't correct, they have RC in the names? ;)
17:42:13 <jednorozec> heh
17:42:17 <adamw> thebeanogamer: it's awkward, but necessary because we are fallible humans and turns out it's hard to write everything down the way you mean it. :P
17:42:17 <sgallagh_> I said "unrelated" and should have been more clear: "the thing that names the images doesn't get it from os-release"
17:42:18 <Southern_Gentlem> ???
17:42:24 <jednorozec> but the checksum is correct
17:42:33 <adamw> yeah, i don't think that one stretches
17:42:38 <Southern_Gentlem> -1
17:43:23 <Southern_Gentlem> does Workstation,KDE, and Server have the RC
17:43:31 <sgallagh_> Server has the RC at least
17:43:39 <nirik> yeah, the only other thing I see is the self identification thing...
17:43:42 <sgallagh_> So does Everything netinst and Silverblue
17:44:05 <Southern_Gentlem> so Workstation and KDE?
17:44:16 <geraldosimiao> work and KDE are fine
17:44:23 <nirik> but if we release these, we will get a billion 'But I got the RC, where's the final' comments.
17:44:29 <Southern_Gentlem> so those are the blocking DEs -1
17:44:46 <geraldosimiao> server are not?
17:44:54 <geraldosimiao> or everything?
17:44:57 <thebeanogamer> Is "We'll take 1.2, but spin a 1.4 where the name fix is the only thing changed" an option?
17:44:58 <adamw> one option open to us here is to agree that there *should* be a release criterion
17:45:01 <adamw> this is something we've done before
17:45:18 <adamw> we can basically agree to add a new criterion in principle, then do all the wordsmithing after the meeting
17:45:21 <nirik> yeah. I think there should be.
17:45:26 <nirik> workstation is affected.
17:45:27 <Southern_Gentlem> +1 new Crit
17:45:28 <sgallagh_> Actually, is it just the server Netinst, or both the netinst and DVD?
17:45:32 <nirik> basically all the images and their checksums
17:45:44 <adamw> so, i think i'm gonna propose that
17:45:47 <thebeanogamer> +1 to making this a new blocker
17:46:02 <nirik> well, I mean workstation iso is fine, but its got a wrong checksum file
17:46:09 <adamw> proposal: we agree in principle that there should be a criterion requiring that the final release image names do not indicate in any way that they are *not* final images
17:46:10 <thebeanogamer> *release criteria
17:46:19 <sgallagh_> adamw: +1
17:46:20 <nirik> +1
17:46:22 <thebeanogamer> +1
17:46:23 <jednorozec> +1
17:46:26 <Southern_Gentlem> +1
17:46:31 <geraldosimiao> +1
17:46:42 <nielsenb> +1
17:46:45 <Son_Goku> +1
17:46:53 <sgallagh_> The more I consider it, the more I'd be okay hanging this off the "self-identification" criterion too
17:46:55 <adamw> #info there is strong support for a criterion requiring that the final release image names do not indicate in any way that they are *not* final images
17:47:09 <adamw> sgallagh_: i think i kinda prefer the 'new criterion' track personally...
17:47:40 <sgallagh_> Sorry, I meant hanging the new criterion off that existing one as a clarifying statement
17:47:43 <adamw> so, based on that: blocker votes? i'm +1 blocker now
17:47:48 <sgallagh_> +1 blocker
17:47:52 <nirik> +1 blocker
17:47:54 <adamw> sgallagh_: oh, i see. we can think about that after the meeting when we have time :)
17:47:57 <dustymabe> +1
17:47:57 <nielsenb> FinalBlocker +1
17:47:59 <frantisekz> +1 blocker
17:48:05 <geraldosimiao> +1 FB
17:48:10 <Southern_Gentlem> -1FB
17:48:22 <adamw> wow, that's not suspicious at all
17:48:25 <mattdm_really> heh
17:48:32 <adamw> get out of here, narc
17:48:32 <geraldosimiao> ;D
17:48:37 <Son_Goku> this feels sussy
17:49:20 <geraldosimiao> I feel I'm on a rollercoaster
17:49:31 <mattdm_really> It will be really suspicious if I say "let's not bother shipping this release anytime soon"
17:49:37 <adamw> mattdm_really: the story so far: we accepted 2246410 as a blocker, there is some sentiment to waive it
17:50:05 <adamw> we are trending in the direction of accepting 2246385 as a blocker on the basis of a new criterion that we agreed ought to exist
17:50:22 <mattdm_really> thank you for the summary
17:50:39 <adamw> would you like to tell us we're doing it all wrong? :D
17:50:41 * nirik feels bad, it's my fault for not catching the config that caused the RC in image names thing. :(
17:50:41 <Southern_Gentlem> my vote for new crit was for the future not this release
17:51:00 <mattdm_really> I'm tentatively ok with that, but I will vote to waive if so
17:51:15 <sgallagh_> mattdm_really: On which?
17:51:38 <sgallagh_> Some of this is also predicated on whether we think that the fixes in RC 1.2 are important enough not to just go with 1.1
17:51:47 <mattdm_really> on the grounds that, hey, if the media _is_ corrupted, "suddenly the screen goes blank" is a non-surprising possibiltiy
17:52:03 <sgallagh_> Oh, that one.
17:52:11 <sgallagh_> Yeah, I made much the same comment above :)
17:52:16 <adamw> proposed #agreed 2246385 - AcceptedBlocker (Final) - this is accepted as a blocker as a violation of the criterion we agreed in principle ought to exist, requiring Final image names not to indicate that they are not Final in any way
17:52:21 <adamw> sgallagh_: i think RC 1.1 has both bugs.
17:52:26 <nirik> yes. it does
17:52:28 <sgallagh_> yes
17:52:39 <sgallagh_> Wait, it has the RC issue too?
17:52:41 <sgallagh_> Dammit
17:52:43 <nielsenb> ack
17:52:46 <frantisekz> ack
17:52:47 <Southern_Gentlem> ack
17:52:49 <thebeanogamer> ack
17:52:55 <adamw> #agreed 2246385 - AcceptedBlocker (Final) - this is accepted as a blocker as a violation of the criterion we agreed in principle ought to exist, requiring Final image names not to indicate that they are not Final in any way
17:53:11 <geraldosimiao> ack
17:53:12 <mattdm_really> +1 ftr
17:53:15 <jednorozec> ack
17:53:28 <nirik> so... I guess slip or hero sprint?
17:53:44 <geraldosimiao> here... Fedora-Server-dvd-x86_64-39_RC-1.1.iso
17:53:45 <sgallagh_> We *did* just basically demand a hero sprint last night too
17:53:53 <geraldosimiao> same problem with 1.1
17:54:10 <adamw> hey, we're not done with blockers yet!
17:54:11 <farribeiro> .hi
17:54:12 <sgallagh_> It feels uncomfortably similar to the Bad Old Days if we push for another today
17:54:15 <zodbot> farribeiro: farribeiro 'Fábio Ribeiro' <farribeiro@gmail.com>
17:54:26 <nirik> oh yeah, lets do all the blockers... sorry
17:54:26 <adamw> #topic Accepted blocker review
17:54:35 <Son_Goku> sgallagh_: tbf, our composes failed today, so...
17:54:38 * neil slaps nirik around a bit
17:54:39 <adamw> #info 2241632 - haven't had time to check yet, but pretty sure this ought to be fixed in 1.2
17:54:43 * mattdm_really votes to just rename the files and edit everything referencing them! YOLO!
17:54:46 <neil> (object unconfirmed)
17:54:48 <mattdm_really> (not really.)
17:54:56 <adamw> #topic (2113005) Live image made with BOOTX64.EFI from latest shim-x64-15.6-2 fails to boot on some boards
17:54:56 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=2113005
17:54:56 <adamw> #info Accepted Blocker, shim, NEW
17:54:59 <sgallagh_> mattdm_really: That was discussed as a possibility
17:55:03 <adamw> mattdm_really: we did consider that, but it's a bit...messy.
17:55:11 <adamw> so, it's time: let's just waive this again
17:55:19 <nirik> +1 waive. ;(
17:55:28 <mattdm_really> I feel like our chances of screwing that up are worse than the risk of someone being really confused by it and not reassured by docs
17:55:35 <Southern_Gentlem> +1 Waive
17:55:40 <pjones> hopefully fixable as an erratum this cycle, but not quite yet.
17:55:40 <adamw> for newcomers - the actual bug here was fixed long ago, but we cannot do a new shim build and get it signed by microsoft without some changes in the kernel that have been taking a long time
17:55:42 <mattdm_really> anyway I will stay on topic
17:55:42 <geraldosimiao> +1 Waive
17:55:44 <sgallagh_> +1 waive
17:55:46 <mattdm_really> +1 waive
17:55:48 <nielsenb> +1 waive
17:55:56 <adamw> nothing much has changed since the last five times we waived this, so...
17:56:00 <frantisekz> +1 aive
17:56:05 <frantisekz> * +1 waive
17:56:16 <jforbes> +1 wave too I know why we are waiting for the kernel and it is still premature to pull them in
17:56:20 <sumantro> +1 waive
17:56:27 <jednorozec> +1 waive
17:56:28 <Son_Goku> +1 waive
17:56:29 * Son_Goku sobs
17:56:30 <adamw> proposed #agreed this is waived to Fedora 40 Beta under the "Difficult to fix blocker bugs" category - https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Exceptional_cases
17:56:33 <neil> +1 waive
17:56:38 <nielsenb> ack
17:56:39 <geraldosimiao> ack
17:56:42 <Southern_Gentlem> ack
17:56:46 <sgallagh_> ack
17:56:50 <sumantro> ack
17:56:52 <thebeanogamer> ack
17:56:53 <farribeiro> ack
17:56:57 <Son_Goku> ack
17:56:59 <neil> axk
17:57:04 <neil> ack*
17:57:11 <adamw> #agreed 2113005 is waived to Fedora 40 Beta under the "Difficult to fix blocker bugs" category - https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Exceptional_cases
17:57:36 <Son_Goku> umm
17:57:37 <Son_Goku> wait
17:57:40 <adamw> waiting
17:57:43 <Son_Goku> oh nvm
17:57:45 <Son_Goku> I can't read
17:57:51 <Son_Goku> I mentally replaced 40 with 39 and confused myself
17:58:05 <Son_Goku> we're good
17:58:26 * thebeanogamer needs to bail for 20 minutes, London Underground WiFi is not friendly to IRC
17:58:27 * Son_Goku has lost hope on this bug ever getting fixed though
17:58:31 <adamw> #topic (2241252) U-Boot fails to load kernel provided DTs from /boot (RPi4 boots to blank screen)
17:58:31 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=2241252
17:58:31 <adamw> #info Accepted Blocker, uboot-tools, MODIFIED
17:58:52 <adamw> so, this one is being fun! we have various people testing boot on various pis with various images
17:58:55 <frantisekz> I'd confirmed the issue is fixed, for the 1.2 rpi4 at least
17:58:56 <adamw> and getting different results
17:59:10 <adamw> we *think* most of the variation is due to some weirdness with the server image and a bug in the image writer script
17:59:11 <Son_Goku> so marcan tried to repro this and couldn't at all
17:59:11 <neil> oh, uboot.
17:59:43 <nielsenb> For anyone wondering why everyone ships bespoke one off versions of U-boot that never get updated, this is why...
17:59:45 <Son_Goku> alas my RPi4s are all burned out, so I couldn't join in the fun
18:00:06 <Son_Goku> nielsenb: I do not have nice things to say about arm and uboot
18:00:07 * nirik has no pi's....humm... pie!
18:00:24 <nielsenb> Me either...
18:00:27 <nirik> so, we need some more testing here for clarity? or ?
18:00:33 <coremodule> yes, I can confirm both RPi4 and RPi400 work with RC1.2 (not that theres a difference, but I've tried on two different RPi4 variants)
18:00:35 <sgallagh_> I've got an RPi 400 if we need more data points, but do we?
18:00:48 <pwhalen> I reproduced the black screen on the RPi4 with server image, not using the script. Its a bug in the FW provided dtb it seems. But it also seems to only affect some revisions
18:01:32 * Southern_Gentlem hopefully it works on the 5s
18:01:37 <Son_Goku> marcan tried workstation and couldn't reproduce at all
18:01:38 <pwhalen> The display works by adding 'nomodeset'
18:01:59 <pwhalen> Son_Goku: the workstation and minimal use the kernel dtb, so they're not affected
18:02:02 <Southern_Gentlem> -1 fb _commonbug
18:02:02 <Son_Goku> ahhh
18:02:08 <sgallagh_> Oh, hang on.
18:02:14 <Son_Goku> wait, so what *doesn't* use the kernel dtb?!
18:02:22 <pwhalen> server
18:02:29 <Southern_Gentlem> -1 fb +1commonbug
18:02:36 <pwhalen> uboot can't read xfs
18:02:39 <Son_Goku> oh
18:02:42 <adamw> that's not the only issue we have here...
18:02:52 <Son_Goku> that's... good to know
18:02:52 <adamw> nielsenb also reports the workstation image crashing back to gdm
18:02:57 <dustymabe> coreos too
18:02:59 <Son_Goku> wait
18:03:02 <adamw> coremodule: frantisekz did you see that?
18:03:02 <Son_Goku> huh
18:03:13 <frantisekz> adamw: nope
18:03:20 <Son_Goku> does server not do uboot->grub?
18:03:21 <coremodule> adamw, no, on either variant
18:03:25 <frantisekz> worked amazingly fine with the 1.2 RC
18:03:48 <nielsenb> adamw: trying that again with nomodeset right now to see if it fixes _that_ issue too
18:04:03 <adamw> nielsenb: what variant do you have?
18:04:04 <sgallagh_> This would probably explain the issues I had with Silverblue on aarch64 VMs under UTM too...
18:04:07 <frantisekz> nielsenb: how much ram does your rpi4 has?
18:04:09 <frantisekz> *have
18:04:39 <coremodule> I have tried 4GB RPi400, 4GB RPi4, and 2GB RPi4, all with no issues.
18:04:42 <nielsenb> I believe it's the 8 GB flavor
18:04:54 <frantisekz> hmm, afaik, the 8 gig one is a bit different
18:04:55 <dustymabe> mine is 8GB FWIW
18:05:00 <nielsenb> adamw: I'm testing workstation again to see if nomodeset works around the crash to GDM issue
18:05:06 <adamw> dustymabe: can you try the workstation image maybe?
18:05:29 <frantisekz> (I have 4 GB, that worked fine)
18:05:32 <adamw> basically, my impression here is...we've got a soup of different experiences going on and i'm just worried about signing off the release in this state
18:05:55 <dustymabe> adamw: would prefer it to be someone else if anyone else has an 8G
18:06:09 <dustymabe> I'm set up to pretty easily test FCOS, but not anything else
18:06:11 <coremodule> Give a moment and I could test 8GB.
18:06:20 <nirik> yeah...
18:06:22 <dustymabe> so if someone else has a good workflow for it already set up, that would be useful
18:06:24 <coremodule> Need two minutes
18:06:29 <nirik> this all is messy
18:06:36 <nielsenb> If only I had a faster card writer and / or card...
18:06:43 <nielsenb> Or more cards
18:06:48 <nielsenb> Or more Pis
18:06:50 <adamw> coremodule: okay, go for it
18:06:54 <adamw> if only I had more pie
18:06:56 <dustymabe> coremodule++
18:06:57 <zodbot> dustymabe: Karma for coremodule changed to 2 (for the release cycle f39):  https://badges.fedoraproject.org/tags/cookie/any
18:07:04 * sgallagh_ bakes furiously
18:07:38 <geraldosimiao> coremodule++
18:07:39 <zodbot> geraldosimiao: Karma for coremodule changed to 3 (for the release cycle f39):  https://badges.fedoraproject.org/tags/cookie/any
18:08:09 <nielsenb> At 300s of this card write...
18:08:16 * adamw watches a dvd of a concert from 1993 where the performer stops playing and yells at someone to put their camera away. wow, different time
18:08:37 <nirik> "To make an apple pir from scratch, you must first create the universe"
18:08:45 <nirik> pie. sheesh.
18:09:13 <coremodule> 8GB RPi4 boots and shows GUI fine on Workstation RC1.2
18:09:28 <nielsenb> Can you create an account and actually sign in?
18:09:32 <coremodule> Yes,
18:09:37 <nielsenb> Must be nice :D
18:09:50 <sgallagh_> coremodule: and failed on RC 1.1?
18:10:05 <coremodule> I didn't test RC1.1
18:10:29 <coremodule> this is raspberrypi,4-model-b Raspberry Pi 4 Model B Rev 1.4
18:10:45 <frantisekz> there is 1.5 out in the wild too, not sure about the diffs though
18:10:54 <coremodule> So I haven't seen issues on 4 Pi variants... I can get the rev for each one.
18:10:55 <nielsenb> Mine is a 1.1
18:11:23 <frantisekz> nielsenb that's weird, I was almost sure 8 gigs were always 1.4+
18:11:39 <nielsenb> It is possible mine is a 4, I'm waiting to boot into it to be sure
18:11:58 <adamw> y'know, as a sidebar, i have to admit that i am really impressed that aarch64, on being invented after we were deeply and painfully aware of all the issues with BIOS and even UEFI, took one look at both of them and said "hold my beer"
18:12:14 <pwhalen> heh
18:12:18 <adamw> it takes real engineering ingenuity to somehow make your architecture's initialization work *worse* than either
18:12:26 <frantisekz> :D :D , you should've been there in the Brno office when lbrabec was testing PineBook...
18:12:55 <nielsenb> 580s of writing....
18:13:34 <Southern_Gentlem> nielsenb, you need a new SD card
18:13:38 <sgallagh_> frantisekz: I think I heard the swearing from the US
18:13:52 <frantisekz> yep, that was us :D
18:13:54 <nielsenb> 20.3 MB/s...
18:14:09 <nielsenb> Not really sure how coremodule wrote one much faster
18:14:34 <Son_Goku> and now that I know xfs + uboot = bad, I just killed that from Fedora Asahi Server
18:14:36 <frantisekz> (and I thought my Samsung Evo Pro with 80 MB/s avg was bad...)
18:15:13 <Son_Goku> but I am seriously impressed and depressed at how nuts aarch64 is
18:15:17 <nielsenb> This started out a last faster than that
18:15:28 <nielsenb> I've never seen a uSD sustain anywhere near 80 MB/s
18:15:52 <coremodule> I didn't write it any faster, it was already written...
18:15:53 <adamw> hi, amoloney
18:15:57 <adamw> it's all chaos over here
18:16:06 <amoloney> hi all
18:16:14 <amoloney> what'd I miss?
18:16:18 <nirik> all chaos all the time. ;) Welcome amoloney
18:16:24 <adamw> nielsenb: when it starts out it's all cached somewhere
18:16:33 <adamw> then it drops off drastically as it actually has to start writing stuff to the medium
18:16:35 <nielsenb> Would probably be faster for me to try to find the box for this thing....
18:16:38 <Southern_Gentlem> so can we skip this and come back
18:16:55 <adamw> Southern_Gentlem: we *could*, but i figured we might want to try and straighten things out more
18:17:04 <adamw> personally, the more chaotic the Pi situation is, the more slip-y I feel
18:17:11 * nirik goes to get more coffee.
18:17:18 <nielsenb> It's done!
18:17:33 <adamw> amoloney: we accepted both proposed bugs as blockers (but haven't decided whether to waive them yet). we're going over all the accepted blockers now. we're on "Raspberry Pi stuff" right now.
18:17:39 * Southern_Gentlem waits on the taxi (early release for football game)
18:17:47 <decathorpe> isn't the Pi 4 the one model that has basically been sold out ever since it was introduced? ...
18:18:03 <adamw> pretty much, yeah.
18:18:05 <Southern_Gentlem> until recently yes
18:18:16 <sgallagh_> decathorpe: I think it's no longer the only one. Now the Pi5 is ALSO sold out :)
18:18:17 <adamw> but that doesn't mean there aren't many of them - they made a ton, and sold a ton. there's a lot of em.
18:18:20 <smooge> when they said 'hey we got a Pi 5 coming sreal soon'
18:18:32 <Southern_Gentlem> roughly 5K hit the market a month ago and was instantly sold out
18:18:38 * decathorpe wonders how many people would actually be impacted by Fedora 39 problems on RPi4
18:18:49 <amoloney> ok thank you for the overview :)
18:18:51 <nielsenb> Which is why I'm beginning to think this might be a 4 GB one, my 8 GB went somewhere else for a project IIRC
18:18:53 <adamw> quite a lot, i think? it's gotta be the most popular aarch64 platform.
18:18:58 <nielsenb> Really wish it was silkscreened on the board
18:18:58 <sgallagh_> decathorpe: Unfortunately, MANY
18:19:12 <nielsenb> Yeah, if you're going to say you support aarch64, the Pi kinda needs to work
18:19:12 <mattdm_really> decathorpe: not a huge fraction, but a big enough absolute number, I think
18:19:23 <mattdm_really> that and gravaton
18:19:27 <nielsenb> Yeah
18:19:37 <adamw> nielsenb: so...do we know for sure exactly what rev you're having problems on?
18:19:44 <Son_Goku> the problem is that the RPi4 is mostly a mess of downstream changes in the RPiOS packages :(
18:19:50 <nielsenb> We will in a few seconds
18:19:52 <adamw> okay
18:19:58 <coremodule> nielsenb, Did you try a different microSD card than the one you originally used?
18:20:17 <farribeiro> Sometimes I think it's not worth buying an arm-based device
18:20:20 <nielsenb> It's a 4 GB, so in that regard, I'm an idiot
18:20:25 <adamw> so as best as i can boil it down, we have nielsenb and hristo marinov seeing problems we can't account for
18:20:28 <frantisekz> nah, can happen
18:20:38 <adamw> hristo says his display blanks out soon after it starts showing messages
18:20:48 <coremodule> Yeah, no worries there.
18:20:57 <frantisekz> which looks like the issue with RC < 1.2
18:20:59 <adamw> farribeiro: only sometimes?
18:21:01 <coremodule> Were their displays plugged into the correct port?
18:21:12 <adamw> i have a pile of them i got for free and they weren't worth it either. ;)
18:21:12 <dustymabe> FWIW with the previous uboot my CoreOS system at least booted, just didn't get display
18:21:21 <coremodule> There are two micro-HDMI ports on the RPi4, only one of them outputs display
18:21:24 <dustymabe> which I don't need anyway
18:21:32 <dustymabe> but with the new one for some reason it won't even boot
18:21:38 <adamw> well that's not good either
18:21:42 <pwhalen> adamw: hristo is the FW dtb. I reproduced and have a workaround (nomodeset)
18:21:42 <nielsenb> In my case, it starts with display, and eventually goes away
18:21:44 <adamw> dustymabe: so RC 1.1 is ok but RC 1.2 is not?
18:21:49 <Southern_Gentlem> bad sdcaed
18:21:57 <farribeiro> I have an arm32 and had the misfortune that it was discontinued on F36
18:22:00 <nielsenb> Doesn't crash back to GDM with nomodeset
18:22:03 <dustymabe> adamw: well it's a bit hard for me to answer that question
18:22:03 <coremodule> Southern_Gentlem, yeah, agreed. Try a new microSD card nielsenb
18:22:05 <nielsenb> Though it does look like crap
18:22:06 <adamw> pwhalen: okay, so...is that going to affect *all* server boots on all pis?
18:22:13 <dustymabe> but when we were testing with beta content that was the case
18:22:25 <dustymabe> and I think all the uboot stuff hasn't changed since beta until the last day
18:22:25 <Southern_Gentlem> coremodule, that what i said 10 min ago LOL
18:22:32 <nielsenb> Behavior doesn't change with SD card
18:22:36 <adamw> dustymabe: right, rc 1.1 should be same as beta, i think
18:22:43 <farribeiro> btw... as well as problems with the connectors
18:22:50 <nielsenb> I have needed nomodset all through F39
18:22:50 <coremodule> Southern_Gentlem, yeah, I buy new microSD cards and chuck the old ones each release for that reason.
18:22:55 <dustymabe> but keep in mind for FCOS we don't include the uboot stuff in the image - which is why I don't consider this a real blocker for F39
18:23:12 <dustymabe> but just in case what I'm seeing may somehow affect another variant, I'm putting it out there
18:23:13 <adamw> okay. well. it's a mess.
18:23:40 <dustymabe> "this a real blocker for F39" -> "this" being my most recent experience with the new uboot + FCOS
18:23:41 <pwhalen> adamw: it will.
18:23:51 <farribeiro> I see a lot of horizons for ARM
18:24:00 <adamw> pwhalen: and is that a new problem? or did f38 have the same problem?
18:24:02 <pwhalen> Booting with serial console works, or headless. With display you'll need to add it
18:24:21 <pwhalen> I *think* F38 had another version of uboot
18:24:33 <nielsenb> F38 works fine for me
18:25:22 <nirik> this is quite a tangle... because in addition to all the rev's of hardware, there's the installer script issues
18:25:42 <Southern_Gentlem> so far we have working, not working with old media, and we suspect PEBCAK
18:25:54 <pwhalen> nirik: I'll fix that today
18:26:02 <nielsenb> Workstation 39 RC1.2 is old media?
18:26:11 <Southern_Gentlem> SD CARD
18:26:17 <nielsenb> I said SD card doesn't matter
18:26:20 <nielsenb> I have several
18:26:25 <nielsenb> I see this with all of them
18:26:42 <Southern_Gentlem> and not seen on new media
18:27:00 <nielsenb> You'll only accept my report if I go buy a new SD card...?
18:27:21 <smooge> nielsenb: coremodule do you know what the firmware installed on each system is?
18:27:32 <nielsenb> BIOS 2023.07 07/01/2023
18:28:22 <nirik> We kinda need a table here... rev, image version, image type, result...
18:28:26 <adamw> so. um. we have nielsenb, who has a crash in g-i-s that's reproducible for him but nobody else has seen yet (right?)
18:28:26 <smooge> yeah
18:28:30 <nirik> or at least I am getting nicely confused. ;)
18:28:42 <smooge> soryr the yeah was for nirik
18:28:55 <pwhalen> I have not
18:29:00 <nielsenb> Right, but we know aarch64 test coverage is a bit of an issue don't we?
18:29:03 <farribeiro> afk
18:29:03 <adamw> and we have hristo's case, which pwhalen says will be general: boot of the server image will not display on screen correctly without nomodeset
18:29:07 <coremodule> nielsenb, what rev is your 4gb rpi? did you confirm it was 1.1?
18:29:10 <nielsenb> And also, as mentioned, so many revisions
18:29:14 <nielsenb> Confirmed 1.1
18:29:17 <coremodule> cool
18:29:22 <pwhalen> nirik: its very hard to keep straight, this is *just* the pi too
18:29:35 <nielsenb> It could also just be something stupid like it playing badly with my monitor
18:29:35 <adamw> i think it boils down to the two things i just wrote
18:29:52 <adamw> nielsenb: do you have any journal messages or anything from the crash?
18:30:04 <nielsenb> Nope, only what I mention on that ticket, which I really don't think is related
18:30:14 <nielsenb> [  369.533317] hdmi-audio-codec hdmi-audio-codec.1.auto: ASoC: error at snd_soc_dai_startup on i2s-hifi: -19
18:30:16 <nielsenb> [  369.533351]  MAI: ASoC: error at __soc_pcm_open on MAI: -19
18:30:25 <nielsenb> Happens every time before dumping back to GDM
18:30:57 <nielsenb> I will happily accept this Pi is just "special" somehow
18:30:58 <mattdm_really> I think the libera webclient is dropping things? wheee
18:31:00 <pwhalen> adamw: the nomodeset may not be needed on all revisions.
18:32:10 <nirik> https://github.com/raspberrypi/linux/issues/4575 perhaps?
18:33:00 <nirik> (which boils down to: harmless apparently)
18:33:05 <adamw> pwhalen: siiiigh
18:33:44 <nielsenb> Right, so perhaps this is all just something stupid with my monitor?
18:33:47 <adamw> should we file a new bug for hristo's case, for clarity?
18:33:56 <adamw> nielsenb: can you try another display?
18:34:05 <nielsenb> Only another one of the same
18:34:11 <pwhalen> nielsenb: 4k?
18:34:12 <nielsenb> Well
18:34:15 <nielsenb> Actually
18:34:22 <nielsenb> 1440p
18:34:30 <geraldosimiao> matt_really: yeah, the libera webclient doesn't have sturdy hands
18:34:40 <dustymabe> adamw: hristo's case being: boot of the server image will not display on screen correctly without nomodeset ?
18:34:49 <adamw> yes.
18:34:57 <nielsenb> Isn't that the same as my case?
18:34:58 <dustymabe> i think that's what this bug is right?
18:34:58 <thebeanogamer> I can heartily recommend the poor man's IRC bouncer (tmux session with irssi on a box you SSH to)
18:35:08 <nielsenb> If anything, there should be a new bug for "crash back to GDK"
18:35:23 <dustymabe> yes, crash back to GDM is a new bug I think
18:35:33 <nielsenb> Yeah, GDM, sorry
18:35:39 <nielsenb> I can test with  a different monitor in a few minutes here
18:35:46 <nielsenb> It's currently in use
18:36:06 <nirik> nielsenb: and you are testing 1.2 images? both server and workstation? or just server?
18:36:18 <nielsenb> Workstation right now, there's no GDM in server
18:36:20 <nielsenb> 1.2
18:36:27 <nirik> k
18:36:29 <nielsenb> Server also needs nomodeset for my pi to boot with a display connected
18:36:37 <nielsenb> Which I believe matches hristo's case
18:36:48 <adamw> nielsenb: no, we don't think your cases are the same.
18:36:59 <sgallagh_> I've lost track in the confusion: what are we trying to prove, right now?
18:37:05 <adamw> pwhalen knows what's going on with hristo's case, and reproduced it, and it's specific to a property of the server image.
18:37:22 <adamw> sgallagh_: i think we're trying to make nielsenb's bug go away, but we *probably* can live with not waiting on that
18:37:24 <pwhalen> nielsenb: it boots without it, either, just no display. If you have a serial console you'll see initial-setup
18:37:43 <nielsenb> Ah, okay, mine will not boot without it
18:37:51 <nielsenb> So I guess it is different
18:37:55 <adamw> if we're willing to make the assumption nielsenb's problem is somehow something quirky about his pi or his display or...something. given that we do have several other folks who've tested workstation and found it worked okay.
18:38:13 <pwhalen> nielsenb: what power supply do you use?
18:38:35 <nielsenb> Uh, whatever comes in a Canakit
18:38:45 <adamw> okay, let me attempt an info here
18:38:47 <pwhalen> ok, sometimes power has been an issue
18:39:08 <nirik> nielsenb: BTW, many thanks for testing here. It's definitely appreciated... :)
18:39:47 <adamw> #info we have had several folks test 1.2 on different Pis. On the *whole* this issue seems to be addressed. However, pwhalen says there does seem to be a reproducible and explainable problem which means Server images will not show anything on a monitor unless booted with nomodeset (other editions are not affected by this)
18:39:55 <coremodule> nielsenb, can you run $cat /proc/cpuinfo | grep Model
18:39:59 <coremodule> and post the result?
18:40:23 <adamw> #info nielsenb seems to have a problem on Workstation where his display goes blank shortly after gnome-initial-setup appears, but other testers have not been able to reproduce this
18:40:23 <nielsenb> Uh, you would think, but the thing just died
18:40:27 <nielsenb> Hold plz
18:40:29 <pwhalen> adamw: might mention some revisions of the rpi4
18:40:40 <adamw> pwhalen: sigh, okay
18:40:42 <adamw> #undo
18:40:42 <zodbot> Removing item from minutes: INFO by adamw at 18:40:23 : nielsenb seems to have a problem on Workstation where his display goes blank shortly after gnome-initial-setup appears, but other testers have not been able to reproduce this
18:40:43 <adamw> #undo
18:40:43 <zodbot> Removing item from minutes: INFO by adamw at 18:39:47 : we have had several folks test 1.2 on different Pis. On the *whole* this issue seems to be addressed. However, pwhalen says there does seem to be a reproducible and explainable problem which means Server images will not show anything on a monitor unless booted with nomodeset (other editions are not affected by this)
18:40:47 <sgallagh_> coremodule: That grep won't work?
18:40:55 <adamw> #info we have had several folks test 1.2 on different Pis. On the *whole* this issue seems to be addressed. However, pwhalen says there does seem to be a reproducible and explainable problem which means Server images will not show anything on a monitor on some Pi revisions unless booted with nomodeset (other editions are not affected by this)
18:41:05 <adamw> oh bah. i don't know if that works with a leading space.
18:41:07 <adamw> anyone know?
18:41:26 <sgallagh_> coremodule: "Model" doesn't appear in
18:41:26 <sgallagh_> ```
18:41:26 <sgallagh_> [sgallagh@mmmpi ~]$ cat /proc/cpuinfo
18:41:26 <sgallagh_> processor	: 0
18:41:26 <sgallagh_> BogoMIPS	: 108.00
18:41:27 <sgallagh_> Features	: fp asimd evtstrm crc32 cpuid
18:41:27 <sgallagh_> CPU implementer	: 0x41
18:41:28 <sgallagh_> CPU architecture: 8
18:41:28 <sgallagh_> CPU variant	: 0x0
18:41:29 <sgallagh_> CPU part	: 0xd08
18:41:29 <sgallagh_> CPU revision	: 3
18:41:30 <sgallagh_> ...
18:41:30 <sgallagh_> ```
18:41:33 <adamw> #info nielsenb seems to have a problem on Workstation where his display goes blank shortly after gnome-initial-setup appears, but other testers have not been able to reproduce this
18:41:36 <nielsenb> I also need nomodeset for server to boot with a display connected, which I believe is the original description of 2241252
18:42:03 <adamw> nielsenb: so you don't see anything on a serial console in that case?
18:42:06 <nielsenb> dmesg|grep DMI gives [    0.038253] DMI: raspberrypi,4-model-b Raspberry Pi 4 Model B Rev 1.1/Raspberry Pi 4 Model B Rev 1.1, BIOS 2023.07 07/01/2023
18:42:15 <kparal> adamw: I think a leading space makes it a non-command
18:42:33 <nielsenb> adamw: never checked actually, but it never makes it to ssh being up
18:43:02 * kparal29 loves irc so much
18:43:16 <nielsenb> Yep, that grep doesn't work
18:43:33 <pwhalen> nielsenb: did you use arm-image-installer to write the image? If so, try with dd and it should work. I have to fix a bug in the script
18:43:46 <adamw> okay, so...i think we should probably move on at this point
18:43:49 <nielsenb> I have always used arm-image-installer, yes
18:44:17 <adamw> oh, yeah.
18:44:21 <nielsenb> Okay, so check server with dd, check workstation with different monitor
18:44:26 <coremodule> nielsenb, interesting, must be raspbian only
18:44:37 <pwhalen> nielsenb: apologies, there is a bug we noticed recently there, I'll fix it shortly.
18:44:38 <adamw> #info we also discovered a bug in the fedora-arm-image-installer script which means it will not write Server images correctly. this can be corrected outside of the compose process
18:44:54 <adamw> #topic (2244305) Fedora Server 39 does not boot on Raspberry Pi 4 (RPi4) from microSD card slot
18:44:54 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=2244305
18:44:54 <adamw> #info Accepted Blocker, uboot-tools, MODIFIED
18:45:06 <adamw> so, this one seems rather less important. but we *do* hope the uboot-tools downgrade fixes it.
18:45:13 <pwhalen> nielsenb: xzcat Fedora-Server-39-1.2.aarch64.raw.xz | sudo dd of=/dev/sdX bs=32M oflag=direct bs=4M status=progress
18:45:15 <adamw> i dunno if it's possible to confirm, since it seems hard to reproduce...
18:45:26 <adamw> pwhalen: nielsenb can you do further debugging somewhere else?
18:45:35 <adamw> maybe https://matrix.to/#/#arm:fedoraproject.org ?
18:45:35 <nielsenb> Where would you like me to do it?
18:45:44 <adamw> if you get significant results, please report back later...
18:45:51 <coremodule> #fedora-arm or private message
18:45:53 <nielsenb> I was going to report the results on 2241252
18:46:06 <nielsenb> Since I need to go back to $DAYJOB here shortly
18:46:14 <nielsenb> And this monitor isn't freeing up
18:46:40 <adamw> sure
18:47:58 <adamw> #info the original reporter's https://bugzilla.redhat.com/show_bug.cgi?id=2244305#c20 indicates that this is fixed
18:48:09 <nirik> yeah, hopefully fixed.
18:49:11 * thebeanogamer offers some emotional support to kparal's poor internet connection
18:49:39 <adamw> okay, so. um.
18:50:23 <adamw> i am trying to figure out where to go from here. pwhalen, do we want to consider the server case as a blocker? is it something that's fixable>
18:51:22 <adamw> aside from that, we have the two unaddressed blockers we accepted earlier
18:51:30 * nirik nods sadly.
18:51:53 <adamw> so we can decide what we want to do with those, i guess
18:52:02 <adamw> #topic (2246385) F39 RC1.2 images have sometimes "RC" in their name (redux)
18:52:09 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=2246385
18:52:09 <adamw> #info Proposed Blocker, distribution, NEW
18:52:21 <adamw> so, this is accepted. does anyone think we should waive it?
18:52:25 <adamw> personally i'd say no
18:52:31 <nirik> I do not.
18:52:38 <coremodule> No.
18:52:46 * kparal is neutral here
18:53:16 <sgallagh_> I'm on the fence. Are we considering this to be the "Last Blocker"?
18:53:36 <nirik> I think we should always do that?
18:53:43 <Son_Goku> we should not waive it
18:53:46 <sgallagh_> That's kind of my point
18:53:51 <geraldosimiao> I'm neutral too. If this was the last one, then waive it. But it seems its not.
18:54:00 <nirik> if this was the last blocker... I would vote we should block on it. ;)
18:54:00 <Son_Goku> but it's also easily fixable by running a compose again
18:54:17 <sgallagh_> My policy is "if we wouldn't hold the release just for this... it's not a blocker"
18:54:57 <nirik> Me too. But in this case... I would. ;)
18:54:58 <pwhalen> adamw: for server, I think we can document adding nomodeset,  headless or serial console.
18:55:23 <dustymabe> I can understand the "we respin the compose with all the same content that we've already verified" argument
18:55:59 <sgallagh_> dustymabe: there's just enough fuzz in the compose process that it's not that easy, sadly.
18:56:06 <sgallagh_> We at least need to re-test that everything boots
18:56:10 <kparal> sgallagh_: since everybody agreed to create a new release criterion exactly for this, it wouldn't make sense to waive it right afterwards
18:56:18 <nirik> we do have bots for lots of testing now
18:56:27 <sgallagh_> Yeah, I agree. This is a blocker.
18:56:38 <frantisekz> if it's just about the confusion? I would waive it, if it's about potential issues somewhere something not expecting RC in the filename?
18:56:41 <sgallagh_> And I guess I'm not in favor of waiving it. But it's a hard choice.
18:56:54 <frantisekz> I mean, users would use fmw, and won't see the image name...
18:57:07 <sgallagh_> frantisekz: It's still visible for installer images
18:57:16 <sgallagh_> Like Server Edition
18:57:28 <Son_Goku> and it's a silly thing to let through
18:57:34 <nirik> it's worse even... since some images are the right name, but checksums have RC
18:57:38 <Son_Goku> ugh
18:57:40 <frantisekz> yeah, but I'd say those users would understand it doesn't matter... imo
18:57:55 <neil> simply for the inconsistency of it, I think we cannot waive
18:57:59 <Son_Goku> yes, but it doesn't make us look good if basic things like that are an issue
18:58:00 <nirik> so people will say "Oh, I downloaded the release, but I can only find the RC checksum! Where is the final checksum!
18:58:08 <Son_Goku> it's a polish thing too
18:58:15 <kparal> all I'm saying, if we waive this, the criterion also shouldn't exist. Otherwise it doesn't make sense.
18:58:18 <neil> Son_Goku++ yea. polish + professionalism, or something
18:58:22 <nirik> kparal: agreed
18:58:33 <Son_Goku> my thing is, the fact that it's easy to fix makes me even more willing to block on it
18:59:22 <mattdm_really> easy fix = rebuild, or easy fix = hacky rename?
18:59:31 <frantisekz> I'd say rebuild...
18:59:32 <sgallagh_> So I guess we go back to our roots and release on Halloween this year?
18:59:35 <Son_Goku> :D
18:59:42 <mattdm_really> I am all for that
18:59:43 <neil> spooky!
18:59:53 <Son_Goku> shame we can't put VERSION_CODENAME="Halloween"
19:00:15 <adamw> okay, so in general, we are not willing to waive this. we can have the What To Do discussion a bit later.
19:00:25 <sgallagh_> Yeah :(
19:00:34 <dustymabe> sgallagh_: halloween release would be if we were "go" today, right?
19:00:38 <nirik> yep
19:00:39 <adamw> #agreed there is no support for waiving this, so it is considered an unaddressed blocker
19:00:53 <adamw> #topic (2246410) Failed media check immediately disappears on bare metal, shows a black screen instead (redux)
19:00:59 <sgallagh_> Cripes, this month keeps getting away from me.
19:01:01 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=2246410
19:01:02 <adamw> #info Proposed Blocker, dracut, NEW
19:01:06 <adamw> so, same question herte
19:01:10 <adamw> do we want to waive this one?
19:01:20 <Son_Goku> ugh
19:01:20 <sgallagh_> This one I'm in favor of waiving.
19:01:22 <adamw> the justification would be late discovery
19:01:23 <Son_Goku> yeah
19:01:36 <Son_Goku> we literally just found it, and it's not like it lets you install a corrupted system
19:01:43 <adamw> "Last minute blocker bugs - bugs proposed as blockers 5 days or fewer before the scheduled Go_No_Go_Meeting for a milestone release (Beta or Final) can be considered under this policy, as there are some circumstances in which we believe it is not sensible to delay an otherwise-impending release to fix a bug which would usually be accepted as a blocker if discovered earlier. In these circumstances, the bug can instead be accepted as a blocker for the
19:01:43 <adamw> next milestone release."
19:01:44 <Son_Goku> but commonbugs and block beta on it
19:01:50 <sgallagh_> Late discovery AND as I said and mattdm agreed: bad media causing a black screen is not an unreasonable outcome
19:02:09 <nirik> yeah, I guess I could support waiving this one.
19:02:10 <frantisekz> what the media check should assure - you don't use corrupted media, it assures that
19:02:11 <adamw> it's better than just proceeding to install, i guess.
19:02:11 <sgallagh_> Bad media leading to a bad install: terrible
19:02:19 <kparal> adamw: does it make sense to waive a bug with the late blocker exception, when we already have a non-waived blocker now (RC)?
19:02:27 <Son_Goku> yes
19:02:36 <frantisekz> yes
19:02:38 <adamw> kparal: i'm getting the sense people might want to hero an rc4 and ship it if it passes smoke testing
19:02:44 <Son_Goku> yes plz
19:02:52 <adamw> which i don't love, but hey\
19:02:57 <kparal> ok. but if we slip a week, that late blocker exception will get invalid, right?
19:03:01 <adamw> we *could* move this decision to later, i guess
19:03:09 <adamw> yeah, that's true, there's kind of an ordering problem
19:03:17 <neil> the ticket says it is also avoidable with nomodeset, so, it can be documented around IMO
19:03:21 <adamw> it's not much fun trying to figure out what order to do things in this meeting :)
19:03:40 <nirik> indeed
19:03:44 <sgallagh_> It's like trying to identify build dependencies automatically in Fedora :)
19:03:53 <adamw> we can make a kind of conditional decision, i guess
19:04:02 <nirik> I would think the waive would only apply to this go/nogo... next one all would be back on the table.
19:04:26 <adamw> nirik: that's not how we drew the process up, but...we can tweak it
19:04:47 <Son_Goku> sgallagh_: you just want to trigger me don't you? :P
19:04:48 <nirik> ok.
19:04:58 <adamw> we *could* leave this undetermined and vote later
19:05:06 <adamw> as in, after we have a new rc, or something
19:05:18 <adamw> let's leave it open for now and move a bit further
19:05:22 <dustymabe> ehh
19:05:29 <mattdm_really> I do not want anyone to feel like hero work is _required_.
19:05:33 <dustymabe> if no need to hero, then would prefer not to
19:05:42 * neil nods
19:05:44 <adamw> #info there is some support for waiving this bug if it will allow us to ship this week, but we kinda need to work through the rest of the meeting first. let's table it
19:05:51 <dustymabe> which would make the result of this decision meaningful
19:05:55 <kparal> if we want to do smoke testing on RC4, I guess we should decide the waiver now
19:06:09 <kparal> not exactly now, just today
19:06:14 <adamw> so
19:06:20 <adamw> in the meantime, i did this
19:06:45 * sgallagh_ can't help but feel like if we'd started the RC4 build at the beginning of this meeting, it would be almost done by now...
19:06:55 * kparal has a suspicion this go/no-go will be of a record length
19:06:58 <adamw> #topic new proposed blocker
19:07:00 <adamw> #topic (2246428) No display on monitor on Server images on some Raspberry Pi variants due to bug in firmware DTB
19:07:00 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=2246428
19:07:00 <adamw> #link https://pagure.io/fedora-qa/blocker-review/issue/1424
19:07:00 <adamw> #info Proposed Blocker, uboot-tools, NEW
19:07:07 <adamw> sgallagh_: it probably would, yes.
19:07:18 <sgallagh_> kparal: Unlikely. I think we had one in the 20s that we technically kept open for three weeks.
19:07:27 <kparal> ;-D
19:07:29 <adamw> so i split the apparently-pinnable-down arm issue into a new bug
19:07:34 <adamw> so we can consider it on its merits
19:07:37 <Son_Goku> nice
19:07:39 <adamw> pwhalen: did i get the description right?
19:07:42 <mattdm_really> It is aesthetically pleasing to ship in October still. But I don't think it's a crisis if we don't. I don't want a Thanksgiving release, though :)
19:07:52 <pwhalen> adamw: heh, wfm
19:08:27 * neil stares at mattdm_really
19:08:34 <neil> no.jpg
19:08:38 * Son_Goku flinches
19:08:40 * mattdm_really is all about blaming this on lack of a funded program manager if anyone complains :-/
19:08:41 <adamw> so. do we want to block on this issue?
19:08:51 <Son_Goku> mattdm_really: I'm definitely game for that
19:08:53 <adamw> i'm all about blaming it on pi
19:09:02 <Son_Goku> no Ben and bad Pi
19:09:10 * neil too, as long as that doesn't come across as too harsh :\
19:09:19 <neil> I made a _fantastic_ pie this week.
19:09:31 <neil> (apple, of course)
19:10:10 <dustymabe> ok so this issue is different from the other one because it is for server and not workstation?
19:10:12 <Son_Goku> adamw: this can be worked around with nomodeset, right?
19:10:27 <nirik> I've gotten turned around again... is this one that could be 'fixed' in image-installer?
19:10:32 <pwhalen> Son_Goku: yes. And only needed if using a display
19:10:42 <Son_Goku> then I think we could squeak by
19:10:50 <neil> me too
19:10:50 <pwhalen> Many people use it headless or with serial console
19:11:00 <dustymabe> so rc 1.1 for workstation and server didn't' work without nomodeset, but rc 1.2 works with workstation, but still requires server to have nomodeset ?
19:11:00 <adamw> nirik: no, it's not
19:11:02 <Son_Goku> we could make the arm-image-installer do the needful
19:11:14 <adamw> dustymabe: if i got everything straight, yes.
19:11:23 <dustymabe> adamw: ok, thanks for clarifying
19:11:27 <pwhalen> dustymabe: sounds about right
19:11:30 <adamw> but only on...some variants of pi, according to pwhalen. do you know which?
19:11:30 <nielsenb> Wait, so everything I saw with Sever was expected?
19:11:37 <nielsenb> I thought I had discovered something
19:11:55 <pwhalen> adamw: no, only initial reports it worked for some
19:12:01 <adamw> nielsenb: we *think* so. if you try it without using the installer script (just directly dd the image), and with nomodeset, and you *still* have trouble, that'd be new.
19:12:12 <nielsenb> I just did it with dd
19:12:16 <nielsenb> And it works, but boots to black
19:12:21 <Son_Goku> with nomodeset?
19:12:27 <adamw> yeah, that's *probably* this bug.
19:12:29 <nielsenb> Without nomodeset, which is not what I was seeing early
19:12:34 <Son_Goku> okay
19:12:36 <Son_Goku> that's this bug then
19:12:37 <nielsenb> I bet nomodeset makes this book to display
19:12:45 <nielsenb> Or at least it did earlier
19:13:42 <nirik> so, we could say then that there is a workaround, display attached isn't common?
19:13:43 <adamw> i agree this seems less serious for server than ws
19:13:49 <Son_Goku> I would be fine with waiving this, commonbugs with workaround
19:13:53 <adamw> not sure if it's *enough* less serious to not be a blocker
19:13:56 <mattdm_really> yep
19:14:07 <adamw> but i'll defer to the arm experts
19:14:16 <Son_Goku> stuff like this sucks, but realistically I don't know how we'd get a fix
19:15:08 <Son_Goku> there's a pile of improvements sitting in the asahi uboot tree that we'll look to prioritize upstreaming and pulling into mainline fedora, but not in the next 24-72 hours
19:15:13 <nirik> I'm ok rejecting it as a blocker on that basis... would FE tho if we slip and get a fix.
19:15:45 <adamw> so, votes? you know, with numbers? i'm a simple vote counting algorithm, i like numbers
19:15:52 <pwhalen> -1 blocker, we have a workaround for server+display
19:15:56 <Son_Goku> -1 Blocker
19:16:03 <Son_Goku> +1 FE
19:16:23 <dustymabe> what is the workaround? like the mechanics? how does the user set nomodeset?
19:16:29 <nirik> -1 blocker, +1 FE, ); DROP TABLES;
19:16:35 <frantisekz> -_- :D
19:16:43 <coremodule> -1 blocker
19:16:46 <frantisekz> -1 Blocker
19:16:54 <sgallagh_> -1 blocker, +1 FE
19:17:03 <nirik> dustymabe: you can use arm-image-installer, or move the usd to another box and change it there?
19:17:06 <pwhalen> dustymabe: as it boots or with the fixed arm-image-installer
19:17:13 <nirik> or if you can ssh in you can change it.
19:17:31 <dustymabe> ok so you can set kernel arguments with the arm-image-installer? that works then
19:17:36 <Son_Goku> yup
19:17:38 <adamw> okay, well, now we have no blockers, thanks nirik
19:17:43 <adamw> i guess we can release
19:17:46 <Son_Goku> well
19:17:54 <Son_Goku> recompose blocker is still there
19:18:07 <nirik> :)
19:18:08 <Son_Goku> it avoided the drop tables :P
19:18:10 <adamw> i was joking about nirik's joke
19:18:28 <Son_Goku> adamw: and I piled on :P
19:18:56 <nirik> so, where are we?
19:19:15 <geraldosimiao> *geraldosimiao is lost
19:19:27 <adamw> proposed #agreed 2246428 - RejectedBlocker (Final) - this is rejected on the basis that display is less important for the Server case, it doesn't affect all variants, and there is a workaround (nomodeset)
19:19:33 <Son_Goku> adamw: +1
19:19:37 <adamw> we have just about slogged our way through the blockers
19:20:05 <adamw> so we're gonna do the matrix status, then we'll have a "what to do" discussion which can involve (again) considering waivers
19:20:24 <adamw> i will resummarize the unaddressed blockers when we get there so nobody is lost (hopefully)
19:20:27 <geraldosimiao> where are the acks?
19:20:33 <adamw> yeah, need a few acks
19:20:38 <geraldosimiao> ack
19:20:41 <coremodule> ack
19:20:43 <pwhalen> ack
19:20:49 <Son_Goku> ack
19:21:06 <adamw> #agreed 2246428 - RejectedBlocker (Final) - this is rejected on the basis that display is less important for the Server case, it doesn't affect all variants, and there is a workaround (nomodeset)
19:22:02 <adamw> okay
19:22:12 <adamw> #topic Current status - test matrices
19:22:12 <adamw> #link https://fedoraproject.org/wiki/Category:Fedora_39_Test_Results
19:22:20 <adamw> so, uh, i did not even get around to checking this yet. let's see
19:22:48 <adamw> we're missing real-world cloud tests
19:22:58 <adamw> if anyone can do some ec2 tests that'd be lovely, i usually do it but have not had time yet
19:23:07 <adamw> i suppose if we're doing a new RC we can do them on that...
19:23:14 <dustymabe> davdunc: ^^
19:23:20 <mattdm_really> out of superstition because of the pi being broken, I'd love to see some ec2 arm test results
19:23:47 <dustymabe> mattdm_really: FWIW coreos tests every build on EC2
19:23:58 <adamw> mac dual boot was last tested on 20231014.n.0 nightly, that's probably okay
19:24:04 <neil> I can do some ec2 tests
19:24:05 <dustymabe> EC2 uses UEFI that doesn't require uboot
19:24:14 <adamw> dustymabe: yeah, i want to automate it for regular cloud images too, just haven't got to it yet
19:24:21 <mattdm_really> dustymabe yeah, that's jon masters' work paying off
19:25:05 <dustymabe> so ARM images in our next stream have been passing tests on GCP AWS and OpenStack (Vexxhost)
19:25:33 <dustymabe> we haven't got to testing ARM images on Azure yet (but we do create them)
19:25:42 <adamw> desktop app basic - system settings hasn't been run on workstation since the 20231014.n.0 nightly either, again i *think* that's probably okay, but would be nice if someone can give it a quick run through
19:26:11 <adamw> and yes, we don't have any azure tests yet because davdunc hadn't got around to publishing the images and providing launch instructions yet
19:27:09 <adamw> #info test coverage is mostly good but with a few gaps: missing real-world cloud testing, and macOS and Workstation system settings tests haven't been run since 20231014.n.0 nightly
19:27:30 <adamw> #topic Fedora CoreOS / IoT check-in
19:27:37 <adamw> dustymabe: pbrobinson: coremodule: are CoreOS and IoT prepared for release?
19:28:52 <geraldosimiao> adamw: I'm doing system settings on workstation 1.2 tight now
19:28:58 <geraldosimiao> *right now
19:29:14 <coremodule> IoT works like it should, I haven't tried the Oct 26 release, but we could go with Oct 25. pwhalen, thoughts?
19:29:45 <coremodule> Fedora-IoT-39-20231025.1
19:29:47 <pwhalen> adamw: IoT should be good, we had a failure today with zezere in openqa, but nothing changed there.
19:30:24 <adamw> probably just a blip, that test seems to sometimes have some kinda issue with timing or the client/server mutexes or something.
19:30:36 <adamw> geraldosimiao: awesome, thanks
19:31:13 <pwhalen> adamw: I've done recent manual testing on hardware and had no issues with zezere
19:31:20 * mattdm_really thinks "oh, there was a problem with the client/server mutexes" is a good phrase to provide your manager to explain any technical problem....
19:31:27 <nirik> There was a big proxy/network issue last night for about an hour.
19:31:29 <adamw> coremodule: i think we'll need to wait till uboot-tools is pushed and then take a build from after that, won't we?
19:31:38 <adamw> mattdm_really: it *does* work great for that
19:31:43 <coremodule> adamw, yes
19:31:49 <coremodule> good point.
19:31:51 <dustymabe> adamw: I don't know of any good reasons not to. after "Go" decisions is made there is still some work to do on our part, but we don't have any blockers at the moment
19:32:00 <adamw> thanks
19:32:31 <adamw> #info coreos and iot report that they are in good shape for release, we will pick an iot compose to ship once we've decided what to ship for the fedora compose and pushed things stable
19:32:48 <adamw> #topic Go/No-Go decision
19:32:49 <adamw> okay, so
19:32:53 <adamw> normally we would just vote here
19:32:59 <adamw> but let's use this space to decide what the heck we're gonna do
19:33:11 <adamw> the story so far:
19:34:01 <adamw> #info we have two unaddressed accepted blockers, "(2246385) F39 RC1.2 images have sometimes "RC" in their name" and "(2246410) Failed media check immediately disappears on bare metal, shows a black screen instead"
19:34:29 <adamw> i am treating 2241252 as "addressed" (though I gotta admit i'm a little concerned about something showing up to bite us there if we ship quickly)
19:34:58 <adamw> we have three choices, I guess:
19:35:07 <adamw> 1. waive both unaddressed blockers and ship RC-1.2
19:35:40 <adamw> 2. waive the media check one, do an RC-1.4 to fix the image names, smoke test RC-1.4 and hope everything else works the same as RC-1.2, ship it
19:35:49 <adamw> 3. take a breather and come back next week
19:36:23 <amoloney> 
19:36:23 <amoloney> 
19:36:24 <amoloney> 
19:36:29 <amoloney> 
19:36:31 <mattdm_really> 2 makes me a little nervous, both in last-minute work and potential for new problems
19:36:34 <amoloney> 
19:36:34 <nirik> I would be willing to do 2 or 3...
19:36:39 <amoloney> 
19:36:42 <mattdm_really> hi amoloney
19:36:46 <amoloney> 
19:36:46 <nirik> amoloney: cat on your keyboard? ;)
19:36:51 <amoloney> 
19:36:56 <amoloney> 
19:36:59 <mattdm_really> I don't know if it is just what the web client is showing, but I'm getting nothing
19:37:01 <amoloney> 
19:37:06 <amoloney> 
19:37:12 <frantisekz> yeah, me in hexchat too
19:37:13 <amoloney> 
19:37:17 <thebeanogamer> amoloney's messages are invisible in irssi
19:37:18 <amoloney> 
19:37:22 <sgallagh_> I think poor amoloney fell asleep on her keyboard
19:37:23 <amoloney> 
19:37:28 <amoloney> on nothing but knee-jerk
19:37:28 <adamw> can anyone op here?
19:37:33 <amoloney> reaction, no to option 1
19:37:36 <amoloney> no plate :(
19:37:38 <amoloney> i had a zillion spaces typed
19:37:40 <amoloney> sorry
19:37:42 <adamw> haha
19:37:42 <mattdm_really> lol
19:37:43 <nirik> ha.
19:37:54 <nirik> yeah, I am also -1 / no on 1
19:37:59 <amoloney> here for comic relief i suppose
19:38:07 <adamw> is *anyone* in favor of 1?
19:38:16 <nirik> if others are willing to do 2, I am happy to do compose, etc...
19:38:19 <neil> we needed it, amoloney :)
19:38:29 <geraldosimiao> 2 or 3 maybe
19:38:30 <neil> I like 2, personally..
19:38:33 <neil> not 1
19:38:36 <sgallagh_> I'm in the "wouldn't fight hard to stop 1, but not in favor" camp
19:38:46 <adamw> oh, btw, can someone FE vote on https://pagure.io/fedora-qa/blocker-review/issue/1422 too? just so we can pull that into RC-1.4 if we do it
19:38:57 <adamw> okay, let's kick 1 off the table
19:38:58 <nirik> also...
19:38:59 <adamw> we're down to 2 or 3
19:39:02 <frantisekz> yeah, I don't have problem with 1, but seems that's off
19:39:04 <mattdm_really> How confidant are y'all _really_ in pulling off option 2?
19:39:11 <nirik> there's a further fix needed for the toolbox hardlinks thing.
19:39:23 <frantisekz> infra side nirik? or in compose too?
19:39:23 <sgallagh_> adamw: If we do a 1.4 for immediate turnaround, I'm against pulling in any FEs
19:39:29 <nirik> if we do a 1.4 it would be nice to get it in.
19:39:34 <mattdm_really> sgallagh +1
19:39:42 <nirik> infra side, need to update imagefactory-plugins and restart kojid
19:40:40 <mattdm_really> well if kojid doesn't restart that puts us automatically to option 3 :)
19:40:51 <sgallagh_> Seriously, IF we take option 2), I want it to be exactly the same contents. Nothing changes but the artifact names.
19:41:07 <neil> it's only koji, wcgw! (I actually mean that seriously, as I've not actually had any issues with koji once it's .. you know.. running)
19:41:07 <mattdm_really> yeah I am with sgallagh_ on that
19:41:09 <geraldosimiao> the FE is for SOAS only, and it fixes some terrible UX
19:41:15 <nirik> sgallagh_: no imagefactory fix?
19:41:34 <sgallagh_> I may have missed that; what was it?
19:41:46 <Son_Goku> the toolbox image is 5.5GB right now
19:41:46 <adamw> yeah, i kinda want the FEs to fix SoaS and toolbx, honestly. they can't possibly break anything else. we swear...
19:41:47 <nirik> toolbox container is much larger than before.
19:41:55 <Son_Goku> because all the hardlinks are real files instead
19:42:25 <sgallagh_> Honestly, if we're going to add in *anything* else, I'm opposed to trying to hero this.
19:42:28 <nirik> On the other hand, if we slip, we could possibly fix the verify check thing, sort out the 🥧 situation more fully, etc...
19:42:56 <Son_Goku> the toolbx fix isn't in the compose repos anyway
19:42:59 <Son_Goku> it's only on the infra side
19:43:00 <mattdm_really> I am currently +0.5 for 1 (still), 0 for 2 (have bad feeling about this), and reluctantly +0.75 for 3. I just can't bring myself to make it a whole 1.
19:43:52 <thebeanogamer> Much though I'd really like to ship 39 (2), if we're still finding bugs in the middle of the go/no-go meeting then I think 3 is the answer
19:43:55 <Son_Goku> I'm +1 to option 2
19:44:07 <nirik> I'd defer for 2 to QE folks... they would be most heavily affected by having to do a bunch of testing.
19:44:08 <adamw> i'd be willing to do 2 but i think i kinda prefer 3
19:44:16 <smooge> from the peanut gallery: I would say 3. Too much effort to make an actual Halloween release will end up with tricks and few treats.
19:44:24 <geraldosimiao> I think one more week... must hold kparal on chains for that period...:D
19:44:33 <adamw> nirik: it wouldn't be a 'bunch' of testing, honestly, we'd literally just make sure each release blocking image booted then rely on rc1.2 testing for everything else
19:44:45 <Son_Goku> and that's already done by openqa bots
19:44:52 <nirik> and there are the bots. ;)
19:44:54 <nirik> yeah
19:44:55 <adamw> no, i'd do that humanly
19:44:56 <thebeanogamer> smooge: The Phoronix and Register headlines write themselves
19:44:57 <sgallagh_> adamw: I could be okay with that, so long as we're not making any compose-side changes besides config
19:45:06 <adamw> need to check they boot on actual hardware with actual nmedia
19:45:06 <amoloney> is a week really the worst thing if we can ship a polished, well tested release?
19:45:07 <pboy> I'm back from a Docs session: From Server WG POV I really don't like 2.  This is a surprise package
19:45:14 <adamw> but yes, i *prefer* 3
19:45:17 <sgallagh_> An ImageFactory upgrade pushes me towards option 3)
19:45:26 <Son_Goku> realistically, we aren't going to get uboot fixes in a week
19:45:37 <sgallagh_> amoloney: Yep, definitely your first Go/No-Go :-D
19:45:38 <Son_Goku> what we're left with is what we're going to get then
19:45:50 <amoloney> hehe
19:45:54 <adamw> well, i would like to have the time to at least nail down exactly where we are
19:46:14 <nirik> if we did do 2, we would have to meet again to be 'go' to it right?
19:46:14 <mattdm_really> okay, I'll change +0.75 to +1.
19:46:19 <adamw> and it may be possible to fix specific bugs, i think. sure we're not going to bump back to 2023.10 and then patch stuff on top of it.
19:46:22 * mattdm_really looks sadly at halloween party
19:46:25 <Son_Goku> if we *do* punt for a week, I'd rather have the epoch bumped uboot-tools yoinked
19:46:34 <adamw> nirik: yeah, we'd have to leave this meeting open or reconvene, i think
19:46:43 <adamw> Son_Goku: eh? then how do we fix the blockers?
19:46:57 <nirik> right.
19:47:03 <Son_Goku> at least one of them, there may be fixes in the asahi uboot tree that fixes them
19:47:04 <geraldosimiao> I'm more for a 3
19:47:21 <nirik> so, I guess I lean toward 3 also... I must be old.
19:47:22 <Son_Goku> the other one seems like it's not fixed either way
19:47:30 <adamw> Son_Goku: huh? what other one?
19:47:37 <Son_Goku> the sd card/usb one
19:47:48 <Son_Goku> is the first one I refer to
19:47:49 <mattdm_really> amoloney let's make sure we ship F40 on time to make up for this :)
19:47:55 <adamw> no, it seems like it *is* fixed with the rollback
19:47:57 <Son_Goku> the other one is the black screen/dtb issue
19:48:17 <adamw> Son_Goku: afaict both are substantially addressed with the rollback.
19:48:17 <amoloney> @mattdm_really agreed
19:48:24 <adamw> no pressure!
19:48:30 <Son_Goku> mattdm_really: you are optimistic aren't you
19:48:31 <amoloney> eeep
19:48:59 <amoloney> remind me, for fun, what we gain from delaying a week?
19:49:08 <Son_Goku> I don't know if we do gain anything
19:49:16 <amoloney> is it a no heroics? (which I agree with)
19:49:24 <sgallagh_> That's the minimum we gain, yes
19:49:24 <adamw> 1. no heroics
19:49:31 <adamw> 2. safer to pull in SoaS and toolbx fixes
19:49:32 <geraldosimiao> a week form people asking us when it will be releades
19:49:37 <geraldosimiao> *from
19:49:45 <geraldosimiao> *released
19:49:46 <adamw> 3. we get time to get more testing on exactly where we stand on ARM
19:49:54 <adamw> 4. we get time to work on the media check blocker
19:50:02 <nirik> the third one is the most compelling to me...
19:50:17 <neil> agreed
19:50:18 <adamw> 5. we get time to do more general validation testing. rc1.2 was *already* late
19:50:23 <amoloney> then its starting to become clearer...
19:50:31 * Son_Goku shrugs
19:50:32 <adamw> 1.4 would be hilariously, psychopathically late
19:50:42 <nielsenb> Am I the last person with an ARM issue?
19:50:52 <dustymabe> yeah. it would be nice to figure out what "new" problem I'm hitting in https://bugzilla.redhat.com/show_bug.cgi?id=2241252#c22
19:51:02 <dustymabe> nielsenb: no :)
19:51:06 <adamw> nielsenb: you're so far the only person who seems to have your *specific* ARM issue. but i think only four or five people have tested.
19:51:18 <sgallagh_> I could tolerate option 2), but I think I prefer slipping a week,.
19:51:20 <nielsenb> Well that's good
19:51:20 <adamw> one benefit of slipping would be to try and find out if more people have your issue.
19:51:45 <adamw> okay, so...i'm kinda hearing a general preference for option 3
19:51:47 <Son_Goku> another I guess would be marcan can try his hand at fixing things on top of uboot 2023.10
19:51:48 <adamw> does anybody disagree
19:51:57 <nielsenb> I am a few minutes away from having a better sense of what condition my condition is in
19:52:00 <sgallagh_> But I'd like us to be really paranoid about what fixes we accept, lest we (re-)introduce bugs
19:52:04 <pboy> nielsenb no
19:52:26 <nielsenb> So many SD cards....
19:52:27 <geraldosimiao> ask register and phoronix readers to help test arm...;)
19:52:38 <adamw> .fire geraldosimiao
19:52:38 <zodbot> adamw fires geraldosimiao
19:52:40 <Son_Goku> adamw: I still prefer 2, but...
19:52:41 <mattdm_really> geraldosimiao seriously, though
19:52:47 <neil> I can definitely get a handful of more rpi/arm testers
19:52:48 <geraldosimiao> :D
19:52:51 <nirik> sgallagh_: sure, but we should get a few things in... the imagefactory and soas things seem pretty safe
19:53:00 <mattdm_really> "hilariously, psychopathically late" doesn't sound like the right choice.
19:53:01 <dustymabe> +1 for option 3
19:53:28 <adamw> okay
19:53:38 <Son_Goku> mattdm_really: blame losing our fpgm
19:53:45 <Son_Goku> everything has gone kinda wrong this cycle
19:53:50 <adamw> so, on that basis: we don't need to talk about waiving anything
19:53:55 <adamw> and we can just proceed to a vote
19:53:58 <sgallagh_> nirik: Can we land those and get a new RC compose today/tomorrow? I'd love for us to have 5+ days of testing for a change :)
19:54:07 <adamw> FESCo?
19:54:11 <adamw> sgallagh_: yes, we can certainly do that
19:54:13 <Son_Goku> 0
19:54:13 <sgallagh_> No-Go
19:54:16 <nirik> sure, if it's requested. All thats lined up.
19:54:16 <Son_Goku> no-go
19:54:20 <adamw> Releng?
19:54:23 <nirik> no-go
19:54:30 <adamw> QA?
19:54:32 <mattdm_really> Son_Goku that blame ain't wrong
19:54:39 <geraldosimiao> no-go
19:54:40 <Son_Goku> mattdm_really: I know
19:54:52 <adamw> (per the QA team policy we must be no-go as there are outstanding unaddressed blockers)
19:55:02 <adamw> #agreed Fedora Linux 39 Final is NO-GO
19:55:03 <geraldosimiao> yeah, no-go
19:55:21 <adamw> #info The next F39 Final Go/No-Go meeting will be Thursday, 2023-11-02 at 1700 UTC
19:55:25 <nirik> still no sad trombone emoji
19:55:32 * neil sends hugs over the wire
19:55:40 <amoloney> we heard it in our heads
19:55:42 <decathorpe> 🥀️
19:55:47 <adamw> #info F39 Final shifts to target date #3: Tuesday 2023-11-07
19:55:48 <Son_Goku> well, at least marcan now has a chance to try to see if he can fix uboot-tools 2023.10 to work
19:55:57 * mattdm_really makes note of mini trombones as potential swag item
19:55:58 <adamw> and let's please make this date, cos i'm going on vacation on the 5th. :P
19:56:06 <smooge> mattdm_really: I wouldn't put a release at hilariously late unless it was after Christmas. Missing Halloween by one week isn't too bad
19:56:19 <adamw> smooge: i said 1.4 was hilariously late in the context of the release process
19:56:19 <nirik> and thanksgiving is right out.
19:56:23 <Son_Goku> I really don't want a repeat of F37
19:56:32 <Son_Goku> that sucked
19:56:35 <nirik> F18 is the one to avoid.
19:56:39 <amoloney> its a Hocus Pocus release?
19:56:40 <smooge> I was thinking that one
19:56:53 <smooge> amoloney: it is the Amuck release
19:56:55 <amoloney> appropriate anytime around halloween
19:57:15 <nirik> it released 2013-01-15
19:57:26 <smooge> F18 was what I guage lateness by
19:57:32 <mattdm_really> thanks everyone. just in time for my 16:00 meeting!
19:57:50 <adamw> the life of an FPL
19:57:51 <mattdm_really> smooge meee tooo. But in this case it was "late in the week to actually test"
19:57:58 <smooge> or maybe RHL8
19:58:04 <adamw> thanks for coming everyone, sorry for the painful meeting
19:58:11 <nirik> was that really mattdm? :)
19:58:13 * nirik runs
19:58:13 <adamw> that was the best i could do to order things :/
19:58:17 <adamw> is that really nirik?
19:58:18 <amoloney> well I have learned a_lot
19:58:21 <neil> thank you adamw 🫂
19:58:27 <smooge> adamw++
19:58:31 <neil> adamw++
19:58:35 <nirik> yeah, it's not at all easy to figure an order here. :(
19:58:36 <thebeanogamer> adamwill++
19:58:37 <geraldosimiao> adam++
19:58:37 <sgallagh_> adamw: You did just fine. Thanks for keeping all these kittens in their corral... mostly
19:58:37 <amoloney> adam++
19:58:38 <zodbot> amoloney: Karma for adam changed to 2 (for the release cycle f39):  https://badges.fedoraproject.org/tags/cookie/any
19:58:42 <smooge> ah man no cookies
19:58:53 <sgallagh_> adamwill++
19:58:55 <geraldosimiao> nilsenb++
19:59:06 <amoloney> bye all!
19:59:06 <adamw> you can send me *actual* cookies
19:59:11 <adamw> bye amoloney!
19:59:12 <pboy> adamwill++
19:59:12 <geraldosimiao> nielsenb++
19:59:14 <zodbot> pboy: Karma for adamwill changed to 7 (for the release cycle f39):  https://badges.fedoraproject.org/tags/cookie/any
19:59:17 <zodbot> geraldosimiao: Karma for nielsenb changed to 1 (for the release cycle f39):  https://badges.fedoraproject.org/tags/cookie/any
19:59:19 <decathorpe> adamwill++
19:59:19 <adamw> #endmeeting