17:04:39 <amoloney> #startmeeting F39 Final Go/No-Go meeting
17:04:39 <zodbot> Meeting started Thu Nov  2 17:04:39 2023 UTC.
17:04:39 <zodbot> This meeting is logged and archived in a public location.
17:04:39 <zodbot> The chair is amoloney. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
17:04:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:04:39 <zodbot> The meeting name has been set to 'f39_final_go/no-go_meeting'
17:04:39 <saluki> .hello saluki
17:04:41 <zodbot> saluki: saluki 'Seth Maurice-Brant' <fedoraproject.kwudj@passinbox.com>
17:04:50 <Son_Goku> .hello ngompa
17:04:51 <zodbot> Son_Goku: ngompa 'Neal Gompa' <ngompa13@gmail.com>
17:04:51 <thebeanogamer> .hello thebeanogamer
17:04:55 <zodbot> thebeanogamer: thebeanogamer 'Daniel Milnes' <daniel@daniel-milnes.uk>
17:05:03 <amoloney> #meetingname F39-Final-Go_No_Go-meeting
17:05:03 <zodbot> The meeting name has been set to 'f39-final-go_no_go-meeting'
17:05:07 <geraldo_simiao> .hello geraldosimiao
17:05:08 <zodbot> geraldo_simiao: geraldosimiao 'Geraldo S. Simião Kutz' <geraldo.simiao.kutz@gmail.com>
17:05:25 <amoloney> #topic Roll Call
17:05:30 <coremodule> .hello2
17:05:31 <zodbot> coremodule: coremodule 'Geoffrey Marr' <gmarr@redhat.com>
17:05:58 <nirik> morning again
17:06:28 <amoloney> evening :)
17:06:30 <decathorpe> .hi
17:06:31 <zodbot> decathorpe: decathorpe 'Fabio Valentini' <decathorpe@gmail.com>
17:06:38 <adamw> .hello adamwill
17:06:39 <zodbot> adamw: adamwill 'Adam Williamson' <awilliam@redhat.com>
17:06:43 <neil> o/
17:06:45 <neil> hi
17:06:46 <neil> .hi
17:06:47 <zodbot> neil: neil 'Neil Hanlon' <neil@shrug.pw>
17:07:08 <Son_Goku> I guess I should say it again for the "roll call"
17:07:12 <Son_Goku> .hello ngompa
17:07:13 <zodbot> Son_Goku: ngompa 'Neal Gompa' <ngompa13@gmail.com>
17:08:07 <adamw> davdunc: pre-emptive ping for later
17:08:19 <amoloney> ok 3 mins is enough for roll call I think
17:08:29 <amoloney> we have important business to be about!
17:08:32 <neil> :D
17:08:41 <amoloney> #topic Purpose of this meeting
17:08:48 <adamw> amoloney: i'd chair a couple of people at this point, just in case you lose your connection or whatever
17:08:55 <adamw> always good to have backup chairs
17:09:10 * nirik can override and add chairs if needed too. ;)
17:09:44 <amoloney> crap now I have to admit I can never remember the way to do that...I think its #chair<someone>?
17:10:01 <adamw> yeah
17:10:04 <adamw> well, with a space
17:10:05 <amoloney> excellent
17:10:08 <amoloney> tag youre it
17:10:14 <amoloney> #chair adamw
17:10:14 <zodbot> Current chairs: adamw amoloney
17:10:17 * mattdm_not_sus picks up a paving stone to place on the accelerator pedal
17:10:20 <amoloney> #chair nirik
17:10:20 <zodbot> Current chairs: adamw amoloney nirik
17:10:37 * Son_Goku stares at mattdm_not_sus
17:10:41 <amoloney> anyone else want to be added as chair?
17:11:42 <amoloney> Ok ping if you change your mind. We shall continue then...
17:11:45 <adamw> Son_Goku: seems like very mattdm-like behaviour
17:11:55 <amoloney> #info Purpose of this meeting is to check whether or not F39 Final is ready for shipment, according to the release criteria.
17:12:21 <amoloney> #info This is determined in a few ways:
17:12:58 <amoloney> #info 1. Release candidate compose is available
17:13:08 <amoloney> #info 2. No remaining blocker bugs
17:13:19 <amoloney> #info 3. Test matrices are fully completed
17:13:34 <amoloney> #info 4. Fedora CoreOS and IoT are ready
17:13:59 <amoloney> and with that...
17:14:05 <amoloney> #topic Current status - RC
17:14:33 <adamw> well, we've got one!
17:14:45 <adamw> https://dl.fedoraproject.org/pub/alt/stage/39_RC-1.5/ is the current RC
17:15:03 <adamw> it addresses all current *accepted* blockers except the shim one (2113005)
17:15:26 <amoloney> which has been a known blocker for several releases, correct?
17:15:34 <pjones> correct
17:16:05 <nirik> sadly so
17:16:24 <adamw> right
17:16:48 <amoloney> silver lining would be this is not a blocker for this one then either though! Glass half full and all that
17:17:29 <adamw> yes, basically. we can vote to waive it later
17:17:30 <nirik> yeah, we usually accepted it as a blocker then waived it as 'too hard/long to fix'
17:17:41 <amoloney> ack
17:18:02 <nirik> note that we are missing some artifacts in RC-1.5... but we can ignore that unless we are go and discuss later.
17:19:23 <amoloney> If theres nothing we need to talk about on the RC Im going to info it and move to the next item
17:19:25 * dustymabe 👋
17:20:02 <amoloney> #info RC-1.5 is the current release candidate
17:20:21 <nirik> +1 move on
17:20:37 <amoloney> #topic Current status - blockers
17:21:11 <amoloney> #Link https://qa.fedoraproject.org/blockerbugs/milestone/39/final/buglist
17:21:18 <amoloney> #chair adamw
17:21:18 <zodbot> Current chairs: adamw amoloney nirik
17:22:24 <adamw> ahoy
17:22:29 <adamw> time for mini blocker review!
17:22:39 <dustymabe> 🎉
17:22:42 <adamw> #topic (2246871) F39 Server Edition for aarch64 SBC disk image is not bootable
17:22:42 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=2246871
17:22:42 <adamw> #link https://pagure.io/fedora-qa/blocker-review/issue/1429
17:22:42 <adamw> #info Proposed Blocker, arm-image-installer, NEW
17:23:28 <pboy> Well, most of the issue could be cleared.
17:23:31 <nirik> So, it seems this may have been something particualr to pboy's system? but we don't know what?
17:23:40 <nirik> hey pboy!
17:23:52 <pboy> nirik no it is not articular to my system, but to server.
17:24:11 <pwhalen> Seems so, I will try to work it out and fix it, but its fine for me on multiple installations.
17:24:11 <pboy> But we know now the reason
17:24:11 <nirik> so any server install? thats... odd. There's not many differences...
17:24:22 <adamw> so...this one is a bit messy. i have attempted to summarize the current status, to the best of my knowledge, in https://bugzilla.redhat.com/show_bug.cgi?id=2246871#c35 .
17:24:48 <adamw> nirik: the significant thing, I think, is that Server still uses LVM by default. no other edition does.
17:25:09 <pboy> nirik: I you use Server to cfeate the disk image, it is always non-bootlbe for non-RPi devides, ´. g you use workstation yuou always get bootable image.
17:25:16 <pwhalen> adamw: summary is great. I have one system that uses lvm and fedora as the vg to test this.
17:25:17 <adamw> having a VG (or LV, I forget which) called 'fedora' makes things tricky for the image installer script because the image also contains a VG called 'fedora'.
17:25:28 <nirik> yeah, lv/xfs... is one of the few differences
17:25:46 <adamw> so the script has to go down a different codepath which tries to handle that, which explains why you can get a different result in this case.
17:26:04 <adamw> pwhalen, have you tried writing an f39 server image on that system yet? can you reproduce pboy's issue that way?
17:26:13 <nirik> so, in theory we could fix this in a 0 day update of arm-image-installer right?
17:26:14 <pwhalen> yes
17:26:14 <adamw> i was going to try it this morning but sort of ran out of time trying to juggle work on all three blockers
17:26:32 <pwhalen> adamw: yes, on both sysytems. One with lvm and vg fedora, the other btrfs.
17:26:48 <pwhalen> nirik: yes.
17:26:50 <adamw> pwhalen: and did you see the problem trying to write the image on the lvm system?
17:27:05 <pwhalen> adamw: no.
17:27:14 <adamw> yayyyy
17:27:24 <pboy> Well, for me that is no longer a blocker. We know, just use workstation live media on your server and it works.
17:27:27 <nirik> Love those heisenbugs
17:28:00 <nirik> I think I am ok saying -1 blocker but we should still try and get to the bottom of it and fix it if we can...
17:28:16 <pboy> nirik +1
17:28:56 <nirik> perhaps it's worth making a debug version of the installer and having pboy run that and see more clearly what goes wrong... just a thought.
17:29:01 <pwhalen> this bug also conflates multiple issues. There is another issue with lvm that has nothing to do with the script .
17:29:23 <adamw> yeah, my overall sense is that nothing here needs to block the release. we can commonbugs that you should remove the /etc/lvm/devices file if necessary, and try to fix whatever bugs may remain in the script as we go along.
17:29:44 <adamw> nirik: the 'debug version' is just 'run it with sh -x'. :P
17:29:48 <nirik> yeah, that sounds like a installer issue... it should wipe that
17:29:50 <geraldo_simiao> Common bugs +1
17:30:02 <nirik> sure
17:30:06 <pboy> The think is now, how serious we take the ARM SBC.  The current image is inferior according to Server installation media standards.
17:30:33 <pboy> But it is OK as a Nerd toy,##
17:31:07 <Son_Goku> Common bugs +1
17:31:11 <Son_Goku> and just let it go at this point
17:31:32 <nirik> ideally we would fix things like this, but in order to do that we probibly need to add more test cases and test eariler next time...
17:32:41 <adamw> pboy: i agree with peter and disagree with you that a relatively minor bug like a leftover file that affects operations you don't need to do on first boot makes the image a "toy".
17:32:44 <pboy> nirik yes, probably it is not worth it in the current time frame.
17:32:54 <lruzicka_> nirik, if only people who use it would report their findings
17:33:11 <adamw> nirik: i think it probably would make more sense to wipe it at image build time, but we could potentially see if the script can remove it as a workaround
17:33:26 * nirik nods
17:33:27 <pboy> adamw We would never accept this in our x86 DVD installation media!!
17:33:33 <adamw> i think we probably would
17:33:38 <pwhalen> right
17:33:39 <Son_Goku> yeah, we have before
17:33:52 <Son_Goku> technically we still are with the shim thing
17:34:12 <adamw> well, that's a waived blocker. but anyhow
17:34:18 <nirik> anyhow, more discussion? voting?
17:34:21 <adamw> it seems like there's mostly a consensus here
17:34:24 * Son_Goku mumbles deprecations about the shim thing
17:34:25 <adamw> -1 blocker
17:34:32 <Son_Goku> -1 FB
17:34:37 <thebeanogamer> -1 blocker
17:34:40 <nirik> -1 blocker
17:34:47 <lruzicka> -1 blocker
17:34:56 <mattdm_not_sus> -1 blocker
17:37:28 <adamw> proposed #agreed 2246871 - RejectedBlocker (Final) - this bug kinda rolls up two issues: possible problems writing the Server image from a system with a VG called 'fedora' or writing the image multiple times to the same media, and problems caused by a file on the image that shouldn't be there. as best we understand this right now, nothing here is bad enough to block the release on
17:37:33 <adamw> sorry, that's a bit hard to write concisely :)O
17:38:43 <lruzicka> ack
17:38:50 <geraldo_simiao> ack
17:39:07 <nirik> ack
17:39:31 <coremodule> ack
17:39:41 <thebeanogamer> ack
17:39:54 <adamw> #agreed 2246871 - RejectedBlocker (Final) - this bug kinda rolls up two issues: possible problems writing the Server image from a system with a VG called 'fedora' or writing the image multiple times to the same media, and problems caused by a file on the image that shouldn't be there. as best we understand this right now, nothing here is bad enough to block the release on
17:41:43 <adamw> #topic (2247611) [aarch64] F39 RC1.5 unxzed images too large
17:41:44 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=2247611
17:41:44 <adamw> #link https://pagure.io/fedora-qa/blocker-review/issue/1432
17:41:44 <adamw> #info Proposed Blocker, distribution, NEW
17:41:44 <adamw> #info Ticket vote: FinalBlocker (+0,0,-4) (-adamwill, -kevin, -lruzicka, -frantisekz)
17:42:05 <adamw> so, well, this one's kind of a bit murky (I did a bit more research since my ticket vote)
17:42:11 <adamw> see https://pagure.io/fedora-qa/relval/issue/27
17:42:42 <adamw> basically, it seems like we were sort of relying on the `disk_size` limit in fedora pungi config to enforce these size restrictions...but at some point we institutionally forgot about that, and have just been bumping those sizes when the images fail to compose
17:43:41 <nirik> oof. Someday we will get good at this releasing things process.
17:43:45 <adamw> the disk_size for workstation got bumped from 14 to 15 and then from 15 to 16 in the last few years. also, those sizes are specified in power-of-two gigabytes. (I think perhaps we picked 14 because 14 power-of-two GiB is under 16 power-of-ten GB)
17:44:40 <lruzicka> adamw, Did not we want them to be copyable to certain flash sizes?
17:44:44 <adamw> so...i think i might actually want to accept this as a blocker, but waive it on the basis we've kinda messed this up lately and it's way too late to try and slice the workstation image back down to 14 for f39
17:44:57 <mattdm_not_sus> yes to that
17:44:58 <adamw> lruzicka: yes, that's why I said "(I think perhaps we picked 14 because 14 power-of-two GiB is under 16 power-of-ten GB)"
17:45:07 <nirik> yeah, I was just thinking that... I think we need to sort this out better, but it shoudl not be right now.
17:46:02 <Son_Goku> +1 FB 1+ waive as too-late-and-too-hard
17:46:04 <nirik> also note that if some release blocking image goes over size in rawhide, it's breaking all composes until addressed and sometimes it's hard to do process to increase the size.
17:47:04 <mattdm_not_sus> 32GB flash drives are down to $2.99 in single quantities at microcenter
17:47:12 <Son_Goku> wow... that's cheap
17:47:20 <mattdm_not_sus> $3.99 for 64GB
17:47:24 <adamw> heh
17:47:41 <Son_Goku> I bought a pile of 16GB sticks (~4 of them) for $15 a couple years ago
17:47:49 <Son_Goku> I think I got ripped off
17:47:53 <lruzicka> mattdm_not_sus, yeah, we just need to decide that we do not want to support smaller ones and we can kick the limits
17:47:56 <mattdm_not_sus> $1.99 for 16GB but they only have 2 in stock and only one brand
17:49:05 <nirik> ok, so where are we ? revoting?
17:49:05 <mattdm_not_sus> anyway. not the point right now, but I feel like it's safe to increase the limit
17:49:28 <nirik> yeah which should be workstation working group for workstation image...
17:51:15 <adamw> we're not revoting, just voting. :P
17:51:22 <adamw> well, i guess if you voted on the ticket you can revote here.
17:51:35 <adamw> i'm +1 on the current criteria (but will also vote to waive)
17:52:03 <nirik> Ditto. Change vote to +1, but will vote to waive
17:52:12 <geraldo_simiao> +1 FB and +1 waive
17:52:54 <adamw> we'll vote on waivers a bit later, just...process
17:54:05 <adamw> proposed #agreed 2247611 - AcceptedBlocker (Final) - this is accepted as a violation of the image size criterion, with the test case wording supporting the interpretation that size limits for disk images should be read as applying to the uncompressed size
17:54:11 <Son_Goku> ack
17:54:13 <adamw> (we can clarify this in the criteria later)
17:54:17 <geraldo_simiao> ack
17:54:19 <lruzicka> ack
17:54:55 <nirik> ack
17:55:04 <thebeanogamer> ack
17:55:57 <adamw> #agreed 2247611 - AcceptedBlocker (Final) - this is accepted as a violation of the image size criterion, with the test case wording supporting the interpretation that size limits for disk images should be read as applying to the uncompressed size
17:56:08 <adamw> #topic (2247659) User switching causes gnome-shell to crash occassionally.
17:56:08 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=2247659
17:56:08 <adamw> #link https://pagure.io/fedora-qa/blocker-review/issue/1433
17:56:08 <adamw> #info Proposed Blocker, gnome-shell, NEW
17:56:15 <adamw> so, openQA test history here:
17:56:28 <adamw> https://openqa.fedoraproject.org/tests/2243821#next_previous
17:56:32 <adamw> (set the dropdown to 100)
17:56:42 <adamw> the test's failed two times in the last month
17:56:49 <Son_Goku> our old friend fast user switching
17:56:59 <nirik> -1 FinalBlocker... I think this is too rare to be blocking
17:57:05 <Son_Goku> -1 FinalBlocker
17:57:10 <adamw> rawhide is pretty similar - https://openqa.fedoraproject.org/tests/2243208#next_previous
17:57:21 <adamw> the bug is pretty bad if you hit it, because you lose whatever you had in the session
17:57:32 <adamw> but on the whole...it's probably okay to fix it as a post-release update
17:57:37 <thebeanogamer> -1 FinalBlocker, 2/100 is an acceptable level of intermittency
17:57:46 <nirik> you should always be ready to loose your session. ;) Be prepared!
17:57:49 <lruzicka> Son_Goku, no, none of a fast switching. Regularly switched to another user. Happened on the first attempt.
17:58:15 <Son_Goku> lruzicka: two users logged in at once is "fast user switching"
17:58:31 <Son_Goku> (yes I know it makes no sense as a name, but that's what it's called)
17:58:33 <lruzicka> Son_Goku, Linux is a multi-user system. Please, do not forget about this.
17:58:51 <Son_Goku> I am aware, but GUI user switching has been off and on broken for almost six Fedora releases
17:58:59 <adamw> note it's not 2/100 (I can't give an exact number out of 100 as some of the failures are not this crash)
17:59:15 <adamw> Son_Goku: well, it's "fast" because you don't have to log out first.
17:59:24 <lruzicka> For me, this happened like 4/5 times
17:59:31 <lruzicka> I tried 5 times :D
17:59:33 <Son_Goku> adamw: yes I figured that
17:59:37 <adamw> unfortunately we don't run the openqa test on aarch64
17:59:44 <Son_Goku> ... yet
17:59:57 <adamw> nah, the GUI tests just aren't terribly reliable on aarch64
18:00:00 <adamw> and this is a *long* test
18:00:03 <Son_Goku> ugh
18:00:04 <adamw> so it'd probably fail a whole lot
18:00:23 <adamw> i keep meaning to try and figure out why it's still pretty unreliable but never get the roundtuits. the hardware we have in prod now ought to be fairly powerful :(
18:01:22 <nirik> never enough time...
18:01:33 <adamw> #10 0x0000ffffa2d475d8 in g_assertion_message_expr (domain=domain@entry=0xffffa2770018 "libmutter", file=file@entry=0xffffa279fb18 "../src/backends/native/meta-onscreen-native.c", line=line@entry=196, func=func@entry=0xffffa27a1868 <__func__.14> "meta_onscreen_native_notify_frame_complete", expr=expr@entry=0xffffa279fae8 "!cogl_onscreen_peek_head_frame_info (onscreen)") at ../glib/gtestutils.c:3523
18:01:33 <adamw> s = 0xaaaaefc5fd50 "assertion failed: (!cogl_onscreen_peek_head_frame_info (onscreen))"
18:01:35 <neil> esp. not with daylight savings ...
18:01:37 <adamw> seems to be the key problem, btw.
18:02:38 <nirik> well, sure, it seems obvious now...
18:02:40 <nirik> (I kid)
18:02:53 <adamw> hehe
18:03:03 <adamw> jadahl can probably figure it out. but we're not gonna fix it now
18:03:17 <adamw> it doesn't seem to be new code, so the problem must be in something leading up to it, i guess.
18:04:01 <adamw> so, we have -3 atm
18:04:03 <adamw> anyone want to argue +1?
18:04:18 <lruzicka> nah, I am fine
18:04:37 <geraldo_simiao> -1 FB
18:04:44 <lruzicka> Or accept and waive
18:06:06 <adamw> proposed #agreed 2247659 - RejectedBlocker (Final) - this is a bad bug if you hit it, but openQA test results indicate it doesn't happen terribly often (at least on x86_64), so we think it's acceptable to try and fix this post-release
18:06:12 <Son_Goku> ack
18:06:21 <nirik> ack
18:06:36 <coremodule> ack
18:06:50 <adamw> #agreed 2247659 - RejectedBlocker (Final) - this is a bad bug if you hit it, but openQA test results indicate it doesn't happen terribly often (at least on x86_64), so we think it's acceptable to try and fix this post-release
18:07:05 <adamw> okay
18:07:31 <adamw> let's move on to accepted blocker review
18:07:38 <adamw> #topic (2247311) Update Firefox in Fedora 39 Final to fix CVE-2023-5721
18:07:38 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=2247311
18:07:38 <adamw> #link https://pagure.io/fedora-qa/blocker-review/issue/1431
18:07:38 <adamw> #info Accepted Blocker, firefox, ON_QA
18:07:44 <adamw> #info this is addressed in RC-1.5, so all good
18:07:50 <adamw> #topic (2113005) Live image made with BOOTX64.EFI from latest shim-x64-15.6-2 fails to boot on some boards
18:07:50 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=2113005
18:07:50 <adamw> #link https://pagure.io/fedora-qa/blocker-review/issue/1154
18:07:50 <adamw> #info Accepted Blocker, shim, NEW
18:07:57 <adamw> i think now is an appropriate time to do waiver votes!
18:08:14 <adamw> proposal: as for previous releases, let's waive this as being impossible to fix in a reasonable timeframe. we are still waiting on upstream merges
18:08:39 <nirik> +1
18:08:51 <thebeanogamer> +1 waive, +1 cursing Microsoft's name
18:09:00 <geraldo_simiao> +1 waive
18:09:13 <Son_Goku> +1 waive +1 curse everyone involved in it
18:10:21 <mattdm_not_sus> +1 waive, -1 cursing, +1 back to making it a prioritized bug (which was rejected 'cause it was already a blocker)
18:11:19 * Son_Goku stares at mattdm_not_sus again
18:11:21 <lruzicka> +1 waive
18:11:28 <adamw> #agreed 2113005 is waived to Fedora 40 Beta under the "Difficult to fix blocker bugs" provision: https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Exceptional_cases
18:11:32 <lruzicka> ack
18:11:35 <Son_Goku> ack
18:11:45 <adamw> i don't think making this prioritized will accomplish anything, everyone involved very much wants to fix it, as soon as it's practical to do so it's gonna happen
18:11:56 <adamw> pjones: is that accurate?
18:12:14 <nirik> ack
18:12:24 <pjones> yeah, that's a fair summary
18:12:26 <adamw> oh, sorry, i didn't make that a proposal. ah well, retroactive acks?
18:12:49 <lruzicka> as if it happened
18:13:03 <Son_Goku> adamw: timetraveled ack
18:13:28 <pjones> I also don't think it's worth *cursing* people about it.  Have some respect for work other people are putting in.
18:13:36 <adamw> yeah, i think that was poorly put.
18:14:01 <thebeanogamer> My apologies, that's not how I meant it
18:14:35 <adamw> alrighty, so moving on
18:14:55 <adamw> #topic (2247611) [aarch64] F39 RC1.5 unxzed images too large
18:14:55 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=2247611
18:14:55 <adamw> #link https://pagure.io/fedora-qa/blocker-review/issue/1432
18:14:55 <adamw> #info Accepted Blocker, distribution, NEW
18:15:01 <adamw> so, back to this one, as an accepted blocker
18:15:14 <mattdm_not_sus> (prioritized doesn't mean people don't want to fix it; it's just another way of tracking critical things. but okay to not bother in this case)
18:15:17 <nirik> +1 waive as being too late/hard to fix now
18:15:21 <adamw> proposal: waive this as both "reported too late" and "difficult to fix" (we can't just magically hack a gig off workstation at this point)
18:15:29 <Son_Goku> ack
18:15:30 <nirik> +1
18:15:33 <mattdm_not_sus> +1
18:15:34 <Son_Goku> +1
18:15:35 <lruzicka> +1
18:15:50 <adamw> mattdm_not_sus: i just meant, the additional tracking isn't really going to achieve anything except bugging people unnecessarily. everyone is already aware this needs fixing as soon as we can.
18:16:51 <adamw> proposed #agreed 2247611 is waived to Fedora 40 Beta under both "Last minute blocker bugs" and "Difficult to fix blocker bugs" provisions (https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Exceptional_cases ). ahead of F40 we will look at tightening up the process here and probably bumping the size requirement
18:17:20 <nirik> ack
18:17:35 <Son_Goku> ack
18:19:05 <lruzicka> ack
18:19:25 <adamw> #agreed 2247611 is waived to Fedora 40 Beta under both "Last minute blocker bugs" and "Difficult to fix blocker bugs" provisions (https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Exceptional_cases ). ahead of F40 we will look at tightening up the process here and probably bumping the size requirement
18:19:59 <adamw> okay. I believe that's everything for blockers. so as things stand we have no accepted, unaddressed, unwaived blockers
18:20:29 <adamw> #info current blocker state: we have three accepted blockers, one is addressed, two are waived. we have no 'active' blockers
18:20:40 <nirik> \o/
18:20:57 <mattdm_not_sus> Good enough!
18:21:00 <amoloney> hurray!
18:21:08 <geraldo_simiao47> Great
18:21:20 <lruzicka> Wait, my test machine now crashed terribly.
18:21:31 <dustymabe> lruzicka: mine too.. wait a second let me paste some logs
18:21:41 <lruzicka> :D
18:22:09 <neil> I was just about to 🥳 but I guess I will wait
18:22:17 <dustymabe> :D - /me jokes
18:22:26 <amoloney> is this some dark humour that Im not getting, or is this real?
18:22:34 <dustymabe> .fire dubiousness
18:22:34 <zodbot> adamw fires dubiousness
18:22:36 <dustymabe> oops
18:22:39 <dustymabe> .fire dustymabe
18:22:39 <zodbot> adamw fires dustymabe
18:22:45 <Son_Goku> .fire lruzicka
18:22:45 <zodbot> adamw fires lruzicka
18:22:57 <Son_Goku> amoloney: I seriously hope it's dark humor
18:23:07 <adamw> ./kickban lruzicka
18:23:11 <adamw> did anyone hear anything? i didn't
18:23:12 <Son_Goku> otherwise we're in for another cycle where a release party happens before the release _again_
18:23:19 <amoloney> small favour for the newbie? please wait a few releases that Im involved in before making 'jokes' :-D
18:23:35 <adamw> haha, sorry amoloney
18:23:37 <amoloney> my heart dropped ha
18:23:47 <Son_Goku> I was a little scared too :)
18:23:47 <adamw> i think the first time someone did `.fire lruzicka` he was terrified
18:23:56 <Son_Goku> lol wut
18:24:07 <Son_Goku> it's not like anyone here is his boss
18:24:07 <adamw> it's a bit disconcerting if you don't know it's a meme :D
18:24:12 <Son_Goku> lol
18:24:19 * decathorpe has always wondered why it's adamw's responsibility to fire people
18:24:31 <lruzicka> adamw, I am still terrified :D
18:24:41 <amoloney> its because we just accept his will :-D
18:24:45 <geraldo_simiao47> For me the first time I was fired, was an honour...
18:24:52 <michel-slm> amoloney++
18:24:52 <zodbot> michel-slm: Karma for amoloney changed to 1 (for the release cycle f39):  https://badges.fedoraproject.org/tags/cookie/any
18:24:53 <tflink> decathorpe: it started as a joke and then someone made it into a zodbot command and the joke has lived on
18:24:55 <geraldo_simiao47> :)
18:25:04 <amoloney> right lets 'fire' up the test matrices!
18:25:22 <amoloney> #topic Current status - test matrices
18:25:51 <adamw> er, let me...take a look
18:26:07 <thebeanogamer> amoloney++
18:26:07 <zodbot> thebeanogamer: Karma for amoloney changed to 2 (for the release cycle f39):  https://badges.fedoraproject.org/tags/cookie/any
18:26:33 * nirik gets more coffee while adamw looks
18:27:02 <amoloney> #link https://fedoraproject.org/wiki/Category:Fedora_39_Test_Results
18:27:38 <adamw> we're missing real-world cloud testing
18:27:43 <adamw> anyone feel like firing up ec2 quick
18:28:24 <adamw> we don't have every single test for aarch64 workstation, but i think we have enough to be fine (given we covered all the missing ones on x86_64)
18:28:52 * nirik can try a ec2
18:28:53 <Son_Goku> I don't exactly have monies
18:28:57 <dcavalca> adamw: if you have an AMI ID to test I can fire an instance up
18:29:26 <adamw> dcavalca: they're on the page
18:29:36 <adamw> https://fedoraproject.org/wiki/Test_Results:Fedora_39_RC_1.5_Cloud
18:29:47 <adamw> there are tables of AMIs for x86_64 and aarch64
18:29:54 <adamw> sorry, i usually do this but was just spread too thin...
18:31:28 <adamw> we're missing the USB UEFI tests for the aarch64 ISO images, too, if anyone has hardware that can run those
18:31:48 <dcavalca> ok spinning up ec2 instances for x86_64 and aarch64 now
18:31:56 <adamw> other than that we're looking good
18:32:05 <adamw> thanks
18:32:41 <adamw> you should be able to blow through startup , logging , identification , services_start and selinux easily right after boot
18:32:57 <adamw> then you can do package_install_remove and update_cli and reboot_umount quite easily after that
18:33:15 <adamw> service_manipulation takes forever but we can live without that since it works in openqa and everywhere else, no reason it'd be busted on ec2
18:33:30 <nirik> [fedora@ip-172-16-0-212 ~]$ uname -r
18:33:31 <nirik> 6.5.6-300.fc39.x86_64
18:33:56 <adamw> lruzicka: pwhalen do you have UEFI-capable arm64 hardware by any chance? i don't
18:34:30 <lruzicka> We have fried the PINEBOOK today and the only thing we have is a Raspberry 400.
18:34:49 <Southern_Gentlem> m2 mac?
18:35:15 <neil> I might be able to help with uefi arm
18:35:26 <Son_Goku> Southern_Gentlem: I think adamw would kill me if we used that for testing right now :)
18:35:58 <adamw> .fire everyone whose nick starts with 's'
18:35:58 <zodbot> adamw fires everyone whose nick starts with 's'
18:36:18 <neil> *technically* they start with 'S'
18:36:19 <Son_Goku> also we don't yet have a uefi iso boot mode in AS Macs
18:36:40 <adamw> neil: if you're in a position to test that quickly it'd be great
18:36:49 <neil> adamw: pgreco tells me he can test, but doesn't have a graphical interface, only serial..
18:36:54 <dcavalca> adamw: everything looks good on ec2
18:37:15 <dcavalca> on aarch64 I have
18:37:15 <adamw> neil: the test is just that the installer image boots and you can run an install...
18:37:17 <dcavalca> Nov 02 18:36:17 localhost kernel: CPU features: detected: Hardware dirty bit management
18:37:26 <adamw> dcavalca: this is more about the firmware, AIUI.
18:37:33 <dcavalca> in the log but that's totally fine, it's just the regexp on the testcase that's greedy
18:37:40 <dcavalca> yup
18:37:46 <nirik> yeah, confirm ec2 ok here.
18:37:50 <dcavalca> I also have access to GCP if you need me to test something there
18:37:54 <adamw> oh, yeah, the test case mentions the grep is a bit greedy
18:38:18 <neil> adamw: have a link to the test case I can share?
18:38:25 <adamw> neil: https://fedoraproject.org/wiki/QA:Testcase_Boot_default_install
18:38:27 <neil> ty!
18:38:56 <adamw> but it's basically 'grab https://kojipkgs.fedoraproject.org/compose/39/Fedora-39-20231031.1/compose/Server/aarch64/iso/Fedora-Server-netinst-aarch64-39-1.5.iso , dd it to a USB stick, boot from that USB stick, install"
18:39:11 <adamw> needs to be on hardware that supports this kind of PC-style 'UEFI boot from generic medium', though. that's the tricky part.
18:40:31 <pwhalen> adamw: I can if someone else doesn't already have it
18:41:00 <adamw> if you don't mind firing it up in parallel, sure - the more the merrier :P
18:41:19 <pwhalen> sure thing :)
18:41:35 <nirik> nothing like last minute testing with time pressure. :)
18:42:52 <Son_Goku> I'm sure it'll work fine :D
18:43:14 <Son_Goku> if we broke arm uefi, people would be screaming, right? ... right?
18:43:21 <adamw> heh
18:44:27 <adamw> oh, shoot. you know what. we kinda missed a blocker.
18:44:36 <adamw> let's circle back while pwhalen tests
18:44:36 <neil> too late. no going back
18:44:38 <neil> #yolo
18:44:42 <adamw> #topic Oops, we missed a blocker
18:44:53 <adamw> #topic (2242759) dnf system-upgrade fails on some RPi4 due to system boot date that pre-dates gpg key
18:44:53 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=2242759
18:44:53 <adamw> #link https://pagure.io/fedora-qa/blocker-review/issue/1392
18:44:54 <adamw> #info Accepted Previous Release Blocker, distribution, NEW
18:44:56 <neil> pgreco is on the arm test too. so we'll see who finishes the race D:
18:45:02 <adamw> so, yeah, this one's hanging around like a bad smell
18:45:03 <adamw> woohoo, races!
18:45:15 <nirik> oh yeah, this ol one.
18:45:16 <neil> ouch. yeah that one does smell :(
18:45:17 <amoloney> cash prize for the winner!
18:45:20 <amoloney> :-D
18:45:21 <Son_Goku> bleeeeech
18:45:31 <adamw> so. ultimately i don't think we should block the release on this
18:45:34 <geraldo_simiao> Race condition? ;)
18:45:36 <neil> Son_Goku: s/ec/eac/
18:45:48 <Son_Goku> lol
18:45:52 <Son_Goku> adamw: I agree
18:45:59 <nirik> adamw: I agree... it's not even clear the right answer here. ;(
18:46:01 <neil> agreed there
18:46:03 <Son_Goku> also isn't there a systemd update that will make this mostly go away?
18:46:11 <adamw> Son_Goku: not afaik, no.
18:46:23 <Son_Goku> meh
18:46:29 <Son_Goku> it's also mostly unfixable
18:46:31 <adamw> it turns out the 'just update systemd and the problem goes away' plan doesn't work if *anything at all* gets built after systemd
18:46:37 <Son_Goku> ugh
18:46:49 <mattdm_not_sus> 🚢
18:46:57 <nirik> depending on network here also seems wrong to me.
18:47:02 <adamw> i mean, i think it's not unfixable in theory, just any practical way to fix it is too radical to do in f39 at least
18:47:29 <pgreco> neil: my sata-usb adapter seems to have left the realm of the living, so it's gonna take me a few minutes to clear up a disk ;)
18:47:53 <adamw> we might be able to come up with a plan to avoid this for upgrades from f40 onwards, or something. but unless someone has a genius plan, i don't see really a safe way to hack up our existing releases and f39 to make it work :/
18:48:20 <nirik> so the critera here is upgrades have to work right?
18:48:22 <adamw> yeah
18:48:28 <adamw> this is a conditional violation for systems with no RTC
18:48:44 <nirik> which is all the pis?
18:49:00 <dustymabe> i think pi5 (brand new) has an RTC
18:49:10 <dustymabe> older ones don't
18:49:22 <nirik> so, I think this is probibly a blocker, but we should waive it also under the 'sorry, too late/hard to fix' thing.
18:49:25 <adamw> maybe other cheap SBCs too? i dunno
18:49:31 <geraldo_simiao> What's an RTC?
18:49:33 <adamw> the problem with waiving it, i think, is we get in a shim situation
18:49:36 <adamw> real time clock
18:49:37 <nirik> realtime clock
18:49:39 <Son_Goku> geraldo_simiao: real-time clock
18:49:48 <geraldo_simiao> Thanks :)
18:49:52 <adamw> hardware thingy that keeps track of what time it is, so you don't always boot in 1973
18:50:01 <adamw> (and rely on the OS to find the real time)
18:50:09 <nirik> well, we could also adjust critera later and say 'if you have a no rtc system... '
18:50:17 <Son_Goku> we probably should
18:50:25 <adamw> so for me the problem with waiving it is, we'd wind up in a kinda shim situation
18:50:27 <Son_Goku> unless we're going to implement some kind of fallback fake rtc
18:50:47 <adamw> we'd have to waive it for, like, a *long* time. at least until two releases after whatever fix we came up with, unless the fix was sufficiently 'safe' to be applied to stable releases (which i doubt)
18:50:50 <nirik> well, if we don't accept it as a blocker, we are still in the same situation no?
18:51:12 <adamw> we're in the same situation in *practical* terms, i mean in "blocker process" terms
18:51:39 <adamw> it would just kinda suck having to keep waiving this every beta and final for several cycles
18:51:43 <nirik> I would hope we could come up with a fix that would work upgrading to 40+
18:51:45 <Southern_Gentlem> rock and a hard place
18:51:57 <adamw> nirik: i'm not sure we can? all the ideas seem too radical to change in a stable release
18:52:03 <adamw> and i don't think you can fix it from the target release
18:52:07 <adamw> but maybe?
18:52:27 <adamw> maybe we can waive it at least once and see how it goes
18:52:47 <nirik> yeah... and if it's going to be a longer term thing, we change critera?
18:53:34 <nirik> oh, I think I misread... systemd-timesyncd doesn't need network to fix this?
18:53:49 <dustymabe> I kind of feel like the behavior that systemd-timesyncd has where it writes out a file on shutdown and then sets the time to that value on boot should be default in systemd and not only if systemd-timesyncd is enabled
18:53:59 <adamw> right. but to me, it would be rather a big change to just rip out chrony and replace it with timesyncd on existing stable releases
18:54:14 <dustymabe> adamw: correct
18:54:20 <thebeanogamer> Is adding chrony to the required services during an update a more acceptable change?
18:54:21 <dustymabe> it can be documented as a workaround for this
18:54:31 <adamw> i suppose we could try and do some hinky thing of enabling timesyncd in the upgrade environment only, but...i'm not sure if that would even work? maybe it needs to be running *before* the reboot as well as *after* to wrok correctly?
18:54:34 <dustymabe> thebeanogamer: I don't think so because we don't have network?
18:54:37 <nirik> or perhaps chrony could do the same thing ?
18:54:43 <adamw> thebeanogamer: *that* would rely on network, i think
18:54:44 <Son_Goku> chrony could be enhanced to do the same thing
18:54:45 <thebeanogamer> dustymabe: Urgh good point
18:54:51 <adamw> that was my suggestion, but it didn't go down terribly well
18:54:52 <dustymabe> adamw: yeah I don't think that would work
18:55:04 <adamw> anyhow
18:55:07 <dustymabe> I think timesyncd has to be running before so on the previous shutdown it writes out the file
18:55:08 <adamw> we don't have to fix it in this meeting :D
18:55:13 <Son_Goku> we could ask the chrony folks about thie problem and see if they can make a fix for f40
18:55:27 <Son_Goku> because we do ship chrony everywhere right now
18:55:28 <nirik> or even f38/39
18:55:56 <adamw> so...shall we go with waiving it?
18:55:59 <pjones> it seems like the better fix is to teach rpmlib to be able to tell if there's an rtc and disable the validity window check if there isn't.
18:56:25 <geraldo_simiao> +1 waive
18:56:34 <Southern_Gentlem> +1 waive
18:56:41 <pjones> (or maybe if there isn't /and/ there's no time synchronization happening... but that's complicated)
18:56:52 <thebeanogamer> +1 waive
18:56:54 <nirik> yeah, I think we should accept it, waive and try and deal with it somehow before f40 beta. ;)
18:57:07 <dustymabe> pjones: yeah, I mean this is kind of a generic problem with systems with no RTC
18:57:25 <nirik> if you have no rtc, just do fresh installs. problem solved. ;)
18:57:27 <dustymabe> this happens to be one symptom, but it's not the only problem systems with no RTC face
18:57:34 <adamw> proposed #agreed 2242759 is waived to Fedora 40 Beta as "Difficult to fix blocker bugs" - it turns out to be rather harder to fix than we expected, and we do not want to try and rush something in for the release window. we plan to document the possible workarounds for this and keep trying to come up with a good fix post-release
18:57:47 <nirik> +1
18:58:02 <lruzicka> +1
18:58:04 <lruzicka> ack
18:58:05 <thebeanogamer> ack
18:58:13 <neil> ack
18:58:15 <geraldo_simiao> ack
18:58:29 <Son_Goku> ack
18:58:35 <Son_Goku> +1
18:58:37 <adamw> #agreed 2242759 is waived to Fedora 40 Beta as "Difficult to fix blocker bugs" - it turns out to be rather harder to fix than we expected, and we do not want to try and rush something in for the release window. we plan to document the possible workarounds for this and keep trying to come up with a good fix post-release
18:58:41 <Son_Goku> ack
18:59:40 <nirik> ack
19:00:00 <adamw> okay, so, back to
19:00:01 <adamw> #topic Current status - test matrices
19:00:06 <adamw> sorry for the interruption, amoloney
19:00:26 <adamw> how's the testing going, pwhalen/pgreco ?
19:00:27 <amoloney> not a problem, it was a fairly important blocker to discuss
19:00:53 <pwhalen> adamw: 43%
19:01:38 <pgreco> adamw: found a sata-usb that works, erasing a disk to start from scratch
19:02:20 <neil> pwhalen is off to a convincing lead, but, anything can happen!
19:02:22 <adamw> pwhalen: hey, if it's got all the way there it's probably fine!
19:02:28 <adamw> ...nah, we should probably wait for bootloader. :P
19:03:20 <geraldo_simiao> pgreco++
19:03:43 <geraldo_simiao> pwhalen++
19:04:15 * nirik has a mustang around somewhere on a shelf. I should set that up someday for testing.
19:04:48 <geraldo_simiao> Humm doesn't work on web, or this username
19:04:57 <pwhalen> 508/681
19:05:08 <pwhalen> almost there.
19:05:23 <adamw> excitement!
19:06:04 <nirik> 🌭 the mustard indicates progress.
19:07:41 <geraldo_simiao> 😊
19:08:01 <adamw> so, what did everyone dress as for halloween
19:08:27 <Southern_Gentlem> grumpy old horn dog
19:08:30 <Son_Goku> the scariest thing I could think of: myself :)
19:08:31 <lruzicka> 🍌
19:08:39 <lruzicka> The banana peel indicates it, too
19:09:05 <lruzicka> 🕷️
19:09:09 <neil> i dressed up as a human
19:09:17 <neil> scariest monster tbh
19:10:51 <amoloney> you people really do have dark humour :-D
19:10:58 <amoloney> I feel at home ha
19:11:37 <neil> :)
19:11:39 <neil> amoloney++
19:11:39 <zodbot> neil: Karma for amoloney changed to 3 (for the release cycle f39):  https://badges.fedoraproject.org/tags/cookie/any
19:13:36 <amoloney> its been way too long since I was at a fancy dress party for halloween (which is a costume party for those who are in North America as apparently that description is not a thing?)
19:16:32 <Son_Goku> fancy dress party in north america is quite different
19:16:36 * nirik goes to a halloween thing every year. I always go as the devil... it's tradition. ;)
19:16:40 * thebeanogamer has decided to celebrate the potential for an approved RC by smashing their head into the server rack they're working on and needs to lie down for a few
19:16:45 <Son_Goku> that usually means... literally fancy dresses rather than costumes
19:17:15 <amoloney> yeah its like tuxedos and stuff!
19:17:22 <amoloney> which would also be funny
19:17:33 <Southern_Gentlem> or a tux tshirt
19:17:35 * sgallagh_ has actually shown up at a European-style fancy-dress party in a suit and tie.
19:17:47 <adamw> a tuxedo works for both!
19:17:55 <lruzicka> one guy once went dressed up as a "wounded guy" with artificial blood all over, but then got stoned, fell asleep on the street and some nice lady called an ambulance and the police.
19:17:55 <adamw> if it turns out to be a costume party just say you came as james bond
19:18:01 <nirik> In the US at least we confusingly have also "white tie" and "black tie"
19:18:02 <sgallagh_> I covered my mistake with a pair of sunglasses, becoming the MiB
19:18:05 <amoloney> thebeanogamer are you still ok?
19:18:13 <adamw> lruzicka: ...it was you, wasn't it
19:18:24 <lruzicka> adamw, I do not drink that much :D
19:18:33 <geraldo_simiao> Go dressed as Tux...
19:18:34 <adamw> nirik: pfft, that's not confusing at all. (and those are british terms originally)
19:18:35 <lruzicka> adamw, I ask for a friend
19:18:43 * thebeano1amer is ok, other than suddenly talking in the third person
19:19:14 <amoloney> head trauma will do that to you...
19:19:19 <nirik> it's always the british!
19:19:24 <adamw> white tie is a stiff collar, white bow tie and evening tailcoat. black tie is a soft collar, black bow tie and a dinner jacket. everyone knows THAT. pfft.
19:19:33 <adamw> (....everyone knows that, right?)
19:19:37 <amoloney> we do now
19:20:10 <amoloney> Adam always learns us
19:20:35 <Southern_Gentlem> https://m.media-amazon.com/images/I/61ItOgu-CiL._AC_SX466_.jpg
19:20:51 <adamw> that works for all occasions
19:21:28 * nirik notes adamw didn't say anything about pants.
19:21:30 <adamw> https://genius.com/The-hold-steady-t-shirt-tux-lyrics
19:21:35 <adamw> nirik: always optional
19:21:35 <sgallagh_> All outfits work for all occasions if you just don't give a damn
19:21:40 <adamw> (no, actually, the pants are the same for both.)
19:22:03 <nirik> so, hows that image looking? ;)
19:22:21 <amoloney> I hope youre referring to the tests ...
19:22:31 <pgreco> downloading  rpms, 21%
19:22:34 <pwhalen> nirik: like it heard you, rebooting
19:22:50 <neil> Southern_Gentlem: hey.. I have one of those.. but it says "SELF" on it ;)
19:23:01 * nirik goes to look for any progress on the framework suspend thing
19:23:39 <pwhalen> success!
19:23:54 <adamw> woohoo!
19:24:01 <adamw> ...and with that i guess we can say the matrix coverage is good
19:24:06 <nirik> excellent!
19:24:14 <pwhalen> updated a couple tests on the matrix
19:24:16 <pgreco> adamw: intslling 275/432
19:24:20 <adamw> we were hoping to have azure testing done for final, but davdunc isn't around so i've no idea how to do that
19:24:40 <adamw> #info matrix coverage is acceptable
19:24:57 <neil> oh. i can do azure..
19:24:58 <dustymabe> FWIW FCOS `next-devel` and `next` boots and passes tests on Azure
19:25:11 <geraldo_simiao> Acceptable is great
19:25:17 <neil> will do that asap..
19:25:28 <adamw> neil: do you know where there are images?
19:25:36 <adamw> davdunc was supposed to provide a list, but i don't think it happened
19:25:43 <lruzicka> Does it take long? I guess I should be going ...
19:25:44 <neil> ah.. i was hoping that had happened
19:25:45 <adamw> you know, whatever the azure equivalent of AMIs is.
19:25:59 <adamw> dustymabe: great, that's a good indicator
19:26:03 <neil> I do know how to hack and slash a qcow/vhd into azure, though
19:26:11 <adamw> eh, we can probably live without it
19:26:22 <neil> ack
19:26:23 <adamw> would be great to do it async from this meeting and let us know if there's any trouble though
19:26:26 <neil> can do
19:26:32 <adamw> amoloney: wanna move on to next topic?
19:26:39 <amoloney> indeed
19:26:54 <amoloney> #topic Fedora CoreOS / IoT check-in
19:27:03 <amoloney> dustymabe: pbrobinson: coremodule: are CoreOS and IoT prepared for release?
19:27:36 <coremodule> aye, IoT looks good from my most recent testing, we can figure out which release will be Gold later this week/early next
19:27:48 <adamw> note, i've sent the stable push request for the one thing that wasn't stable already (firefox and nss)
19:27:58 <pwhalen> IoT is ready for the final compose. We'll do it tomorrow so its ready for next week
19:28:00 <adamw> so that should be pushed soon and nightlies after today should match RC-1.5
19:28:05 <Southern_Gentlem> neil yep with tux coming out of the pocket
19:28:06 * nirik can do that here after this meeting, if it ever ends
19:28:30 <adamw> note we may want to push linux-firmware stable for f39 very soon to fix https://bodhi.fedoraproject.org/updates/FEDORA-2023-1069778c50#comment-3266891 ...
19:29:16 <nirik> hurm
19:29:58 <nirik> I don't think we want to push that now... just make sure it's pushed as soon as we stage the release (ie, in the first 0 day batch)
19:30:37 <dustymabe> amoloney: yep. I think we're good
19:31:06 <adamw> nirik: yes, but my point is, we need to do that *ASAP* or else all f38->f39 upgrades will fail (including openqa tests, which is a problem)
19:31:09 <Southern_Gentlem> yeah i hit that today updating the respin builder to f39
19:31:20 <adamw> i guess i can add the update to the openqa workarounds to fix update tests at least
19:31:29 <pgreco> amoloney: sorry for the noise ;). adamw: installed and rebooted without issues. absolute success
19:31:33 <amoloney> Then once I info that both CoreOS and IoT are good I think we might be ready to poll?
19:31:40 <nirik> well, we can't until after we stage the release... so tomorrow. ;(
19:31:44 <adamw> we can't push it before we freeze the 'final' repo, but i'm saying maybe we need to do that quicker than usual
19:31:54 <adamw> amoloney: i believe so
19:31:58 <amoloney> exciting!
19:32:17 <amoloney> ok any more 'oops we should check xx'?
19:32:35 <adamw> i can't think of any...
19:33:24 <amoloney> righteo Ill info
19:34:04 <amoloney> #info Both Fedora CoreOS and Fedora IoT are in good condition and ready
19:34:22 <amoloney> #topic Go/No-Go decision
19:34:28 <amoloney> Tis time
19:34:32 <adamw> yaaay
19:34:36 <neil> <3
19:34:40 <neil> 🥳
19:34:49 <amoloney> I will poll each team. Please reply 'go' or 'no-go'
19:34:54 <amoloney> FESCo?
19:35:07 <nirik> go
19:35:13 <amoloney> Releng?
19:35:18 * nirik puts on another hat
19:35:19 <nirik> go
19:35:28 <amoloney> QA?
19:35:32 <lruzicka> go
19:35:59 <amoloney> #agreed Fedora Linux 39 Final is GO
19:36:09 <adamw> yaaay
19:36:12 <geraldo_simiao> 😍
19:36:17 <thebeanogamer> wooooo
19:36:21 <alciregi> \o/
19:36:23 <neil> WOOOO! 😍
19:36:23 <lruzicka> Jubilation, she loves me again ....
19:36:25 <adamw> good job, everyone
19:36:35 <nirik> great!
19:36:37 <amoloney> whats our release date?
19:36:44 <adamw> 7th
19:36:46 <amoloney> I am thinking its 7th?
19:36:47 <lruzicka> Happy Release, I need to rush. Take care.
19:36:47 <nirik> 2023-11-07
19:36:59 <amoloney> excellent. Wouldnt be the first time I recorded it wrong
19:37:01 <neil> thank you lruzicka :) you take care too
19:37:12 * nirik sadly has another topic before we close tho... or I guess we can discuss out of meeting if desired.
19:37:14 <decathorpe> so we can announce release is GO and the release party together? ;)
19:37:16 <amoloney> #info Fedora Linux 39 Final will release on the early target date (2023-11-07)
19:37:22 <adamw> we can do an open floor
19:37:25 <amoloney> thank you all!
19:37:26 <adamw> er
19:37:27 <decathorpe> early? :)
19:37:30 <adamw> that's not the early target date :D
19:37:39 <nirik> it's early for xmas!
19:37:39 <amoloney> aw sweet baby
19:37:40 <adamw> i think we're at target date #3
19:37:42 <neil> hehe
19:37:46 <amoloney> #undo
19:37:46 <zodbot> Removing item from minutes: INFO by amoloney at 19:37:16 : Fedora Linux 39 Final will release on the early target date (2023-11-07)
19:37:55 <geraldo_simiao> Yes target #3
19:38:09 <neil> if we've learned nothing from the rpi rtc issue, is it not that time is relative, adamw?
19:38:15 <amoloney> info Fedora Linux 39 Final will release on target date #3 (2023-10-17)
19:38:29 <amoloney> one more try
19:38:44 <adamw> bingo
19:38:49 <adamw> neil: hehe
19:38:52 <nirik> next new release date: 1970-01-01!
19:38:56 <Son_Goku> lol no
19:38:59 <neil> err -- 10-07 , not 17
19:39:02 <amoloney> #info Fedora Linux 39 Final will release on target date #3 (2023-11-07)
19:39:07 <adamw> nirik: dangit, you beat me to the joke
19:39:08 <neil> woo!
19:39:09 <neil> nice
19:39:12 <Son_Goku> woot
19:39:28 <amoloney> nearly failed at the last info hahaha
19:39:30 <adamw> amoloney: can you do an open floor for nirik's thing?
19:39:34 <amoloney> I can indeed
19:39:47 <amoloney> #action amoloney to announce decision
19:39:53 <amoloney> #topic Open Floor
19:39:56 <nirik> ok, so, RC-1.5, which is "go" for release... is missing some images. These failed to compose for basically 3 reasons:
19:40:12 <nirik> 1. Some weird transient 502 errors when making ostree installers that we haven't been able to track down
19:40:27 <nirik> 2. Some weird aarch64 python SIGILL's on live media composes
19:40:35 <nirik> 3. Dep issues on Astronomy
19:40:50 <nirik> For the record, here's the missing ones:
19:40:56 <adamw> the problem with 1 and 2 is, well, we haven't fixed them. so any compose we do is likely to be missing *something*
19:41:01 <nirik> (pardon the long paste)
19:41:04 <nirik> missing ostree install images:
19:41:04 <nirik> Onyx - x86_64
19:41:05 <nirik> Sericea - x86_64
19:41:05 <nirik> Silverblue - aarch64
19:41:05 <nirik> Kinoite - ppc64le
19:41:05 <nirik> missing Live images:
19:41:07 <nirik> Security - x86_64
19:41:11 <nirik> Astronomy_KDE - x86_64
19:41:13 <nirik> LXQt - aarch64
19:41:15 <nirik> KDE - aarch64
19:41:17 <nirik> Workstation - aarch64
19:41:21 <Son_Goku> holy crap
19:41:29 <nirik> So, we could do a few things about this... we could:
19:41:35 <Son_Goku> that's a lot of missing stuff
19:41:45 <nirik> 1. nothing
19:41:45 <nirik> 2. try and rebuild with another name (1.6?)
19:41:45 <nirik> 3. ship the nightly versions from tomorrow (if available)
19:41:59 <Son_Goku> I'd really want to try for 2
19:42:18 <Son_Goku> we have a content lock for GA now anyway
19:42:25 <nirik> 2 and 3 are more work for sure...
19:42:45 <decathorpe> does option 1) mean that those images will just not be available for Fedora 39?
19:42:46 <nirik> the name will be different from all the rest in both cases and websites will need changes, we will have to distribute them from some other place, etc.
19:42:53 <nirik> decathorpe: yep.
19:42:54 <sgallagh_> Worth noting: Silverblue aarch64 was also missing from F38, so there would be no official media from which to install AT ALL
19:43:18 <sgallagh_> For F38, people have been given the F37 media and told to update from there
19:43:26 <decathorpe> that's not good
19:43:40 <decathorpe> wasn't Onyx just added in F39? so there would be no way to install it at all?
19:43:43 <sgallagh_> So I would also recommend a no-changes rebuild
19:43:49 <nirik> The live ones we likely cannot fix right now... we don't have any fix in hand anyhow
19:44:17 <adamw> we cannot, by policy, rebuild and ship a different compose
19:44:24 <nirik> right.
19:44:24 <adamw> we signed off 1.5. not a 1.6 that doesn't exist yet.
19:44:26 * dustymabe kind of feels like we should just deliver one install media and make it netinstall (pull from remote OSTree repo)
19:44:59 <nirik> if we do 3... the name will be YYYYMMDD.n.0 instead of 1.5... but it should be the same content.
19:45:12 <adamw> i think 3 is probably the best option
19:45:14 <adamw> we've done that before
19:45:14 <nirik> (and if they compose and don't hit the sporadic issue)
19:45:40 <adamw> i know it's work for releng to decide where to put the bits and websites team to link to them :/
19:46:08 <nirik> yeah, but I think we should try and do something (aside from fixing the issues...)
19:46:08 <neil> yeah i think 3 makes the most sense (despite it being a pain logistically)
19:47:18 <nirik> One advantage of 2 is that we can just keep trying those. ;)
19:47:35 <nirik> but I agree 1.6 is bad on naming because it shouldn't exist. I guess we could call it something else?
19:48:00 <adamw> i don't really like the idea of inventing some new label name
19:48:16 <adamw> we have conventions about those, i think they're even enforced in productmd to some extent
19:48:29 <nirik> oh, how about... 20231102.n.1 ?
19:48:40 <nirik> well, no, 20231103.n.1
19:48:46 <nirik> (after the final compose)
19:48:50 <adamw> well, no, it wouldn't be a nightly.
19:48:57 <adamw> and i don't think that's a valid label name...
19:49:06 <nirik> to be clear.. I am not talking about a entire new compose.
19:49:28 <adamw> oh
19:49:31 <nirik> I'm talking about specifically doing a compsoe of image X named whatever we want with the final repos
19:49:31 <adamw> what are you talking about?
19:49:44 <adamw> oh, okay. like a scratch build?
19:49:45 <nirik> just each image as a one off.
19:49:57 <adamw> well, i guess i'd be fine with that, if we can do it
19:49:57 <nirik> yes, but I can make them official builds so they stay around
19:50:20 <nirik> it would have to be after the final nightly tho
19:50:27 <nirik> or else it would be missing firefox/nss
19:51:24 <nirik> so, how about we try 3... and if something fails again we fall back to doing those specific ones as method 3 with .n.1
19:51:48 <Son_Goku> I don't see what's wrong with 1.6?
19:51:51 <adamw> if we're doing non-pungi image builds i'd kinda like the name *not* to look like it came out of a nightly compose, but meh, it's not too important
19:51:58 <adamw> i prefer names that don't lie
19:52:09 <adamw> '1.6' looks like it came out of a candidate compose with label 1.6
19:52:10 <nirik> I'm open to whatever name...
19:52:21 <adamw> which never actually happened
19:52:31 <nirik> 1.5-respin ?
19:52:32 <adamw> we can bikeshed the name tomorrow, i guess :D
19:52:34 <Son_Goku> why couldn't we do a 1.6 based on the same content set?
19:52:48 <Son_Goku> that's just kevin running the process no?
19:52:49 <nirik> because by policy there's no cause to do another compose after we are go.
19:52:59 <adamw> Son_Goku: because that'd be needlessly confusing. we couldn't ship it. so it would exist but not actually be the released compose but we'd release some images from it. wat. i don't like that
19:53:03 <adamw> (and yes, that)
19:53:47 <nirik> well, we can talk about it more, I just wanted to bring it up...
19:54:17 <adamw> yeah, i think we have a viable plan anyhow
19:54:20 <adamw> we can work on the details later
19:54:53 <nirik> hum so nightly will still have the 'prerelease' thing?
19:55:21 <Son_Goku> it would :(
19:55:39 <nirik> oh no it won't...
19:55:46 <Son_Goku> no?
19:55:48 <Son_Goku> why not?
19:55:52 <nirik> it's keying off fedora-release
19:56:00 <adamw> um
19:56:00 <dustymabe> #proposed let nirik use his best judgement and everyone be happy with that
19:56:04 <adamw> it depends on the image
19:56:20 <adamw> i think lives key off fedora-release but traditional installers depend on an argument passed at build time? but i have to relook this up every time
19:56:32 <nirik> yeah, I can never recall
19:56:33 <adamw> anyhow, maybe we can close out the meeting now?
19:56:38 <adamw> and work this all out over on matrix
19:57:16 <nirik> yep. +1
19:57:29 <thebeanogamer> Sounds good
19:57:33 <adamw> amoloney: still awake? :D
19:57:42 * thebeanogamer is looking forward to the change proposal for moving all this to Matrix
19:57:49 <mattdm_not_sus> where on matrix?
19:58:14 <amoloney> I am :) the benefit of DST is its only 8pm now, rather than 9pm so yay I guess!
19:58:31 <amoloney> does this need to be info-ed or is it ok to wrap as is?
19:59:21 <adamw> #info several images are missing from RC-1.5 due to known bugs in the compose process, releng will work on a plan to try and provide builds of those images from nightly composes or special build tasks
19:59:27 <adamw> there
19:59:34 <nirik> mattdm_not_sus: releng channel?
19:59:36 <adamw> mattdm_not_sus: releng channel i guess
19:59:42 <mattdm_not_sus> wfm!
19:59:49 <amoloney> thanks Adam!
19:59:57 <amoloney> fin
20:00:01 <amoloney> #endmeeting