f37-final-go_no_go-meeting
LOGS
17:01:16 <bcotton> #startmeeting F37 Final Go/No-Go meeting
17:01:16 <zodbot> Meeting started Thu Oct 27 17:01:16 2022 UTC.
17:01:16 <zodbot> This meeting is logged and archived in a public location.
17:01:16 <zodbot> The chair is bcotton. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
17:01:16 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:01:16 <zodbot> The meeting name has been set to 'f37_final_go/no-go_meeting'
17:01:16 <bcotton> #meetingname F37-Final-Go_No_Go-meeting
17:01:16 <zodbot> The meeting name has been set to 'f37-final-go_no_go-meeting'
17:01:21 <bcotton> #topic Roll Call
17:01:31 <bytehackr> \o
17:01:34 <nirik> morning
17:01:37 <bcotton> looks like I picked the wrong week to quit sniffing glue
17:02:07 <bytehackr> :)
17:02:16 <nirik> there's no good week.
17:02:27 <zodbot> pong
17:02:56 <sgallagh> .hi
17:02:57 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
17:02:59 <travier> .hello siosm
17:03:00 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
17:03:41 <bytehackr> .hi
17:03:42 <zodbot> bytehackr: bytehackr 'Sandipan Roy' <saroy@redhat.com>
17:03:47 <coremodule> .hello2
17:03:47 <zodbot> coremodule: coremodule 'Geoffrey Marr' <gmarr@redhat.com>
17:04:32 * nirik wonders if anyone put out the adamw signal.
17:04:55 <bcotton> he
17:05:04 <bcotton> he's explaining why i am the way that i am to his cat
17:05:04 <cmurf> get out your magic 8 balls folks (i said it earlier, but want it in the minutes)
17:05:24 <travier> :D
17:05:37 <bcotton> just used my SeaGL magic 8 ball and it said "review logs". i'm not sure how to interpret that
17:05:48 <Penguinpee> .hello gui1ty
17:05:49 <zodbot> Penguinpee: gui1ty 'Sandro .' <gui1ty@penguinpee.nl>
17:05:50 <bcotton> but! let's start the process
17:05:57 <cmurf> .hello2
17:05:58 <zodbot> cmurf: cmurf 'Chris Murphy' <chris@cmurf.com>
17:06:05 <bcotton> #topic Purpose of this meeting
17:06:05 <bcotton> #info Purpose of this meeting is to check whether or not F37 is ready for shipment, according to the release criteria.
17:06:05 <bcotton> #info This is determined in a few ways:
17:06:12 <bcotton> #info 1. No remaining blocker bugs
17:06:12 <bcotton> #info 2. Release candidate compose is available
17:06:12 <bcotton> #info 3. Test matrices for Final are fully completed
17:06:19 <bcotton> #topic Current status - blockers
17:06:19 <bcotton> #link https://qa.fedoraproject.org/blockerbugs/milestone/37/final/buglist
17:06:33 <bcotton> #info 1 Proposed Blockers
17:06:33 <bcotton> #info 2 Accepted Blockers
17:06:35 <bcotton> SO!
17:06:45 <bcotton> #topic (2137661) upcoming critical openssl vulnerability
17:06:45 <bcotton> #link https://bugzilla.redhat.com/show_bug.cgi?id=2137661
17:06:45 <bcotton> #link https://pagure.io/fedora-qa/blocker-review/issue/987
17:06:51 <dustymabe> .hi
17:06:52 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
17:06:52 <bcotton> #info Proposed Blocker, openssl, NEW
17:06:52 <bcotton> #info Ticket vote: FinalBlocker (+11,0,-3) (+bytehackr, +mattdm, +bcotton, +ngompa, +geraldosimiao, +chrismurphy, +nixuser, +copperi, +sumantrom, +frantisekz, +augenauf, -sgallagh, -catanzaro, -gui1ty)
17:07:08 <nirik> quite a lot of +1s
17:07:42 <frantisekz> .hello2
17:07:45 <zodbot> frantisekz: frantisekz 'František Zatloukal' <fzatlouk@redhat.com>
17:07:48 <bcotton> indeed
17:07:48 <Penguinpee> most of them chosen ones
17:08:15 <bcotton> the main hesitation, which i understand, is that we don't know what the vulnerability actually is
17:08:17 <nirik> I'm not sure what critera is being used here...I don't think we have much that fits this
17:08:28 <copperi[m]> .hello copperi
17:08:29 <zodbot> copperi[m]: copperi 'Jan Kuparinen' <copper_fin@hotmail.com>
17:08:46 <sgallagh> ""The release must contain no known security bugs of 'important' or higher impact according to the Red Hat severity classification scale which cannot be satisfactorily resolved by a package update (e.g. issues during installation).""
17:08:52 <bcotton> #info Criterion: The release must contain no known security bugs of 'important' or higher impact according to the Red Hat severity classification scale which cannot be satisfactorily resolved by a package update (e.g. issues during installation).
17:08:52 <Eighth_Doctor> .hello ngompa
17:08:53 <zodbot> Eighth_Doctor: ngompa 'Neal Gompa' <ngompa13@gmail.com>
17:08:57 <bcotton> #link https://fedoraproject.org/wiki/Fedora_37_Final_Release_Criteria#Security_bugs
17:09:11 <nirik> right, but we don't know if it could be resolved by an update or not...
17:09:16 <Eighth_Doctor> fwiw, I'm not a "chosen one" for being able to review the bug
17:09:27 <Eighth_Doctor> I have no idea and I'm going off what the "chosen ones" say
17:09:37 <bcotton> yeah, i don't have access to the actual bug either
17:09:41 <sgallagh> My argument is that there currently isn't a known bug that fits that description
17:09:46 <sgallagh> All we have is supposition.
17:09:49 <bytehackr> riight and as per pre-announce security bulletin of red hat its a critical: https://access.redhat.com/security/vulnerabilities/RHSB-2022-004
17:09:56 <Penguinpee> could someone wearing a red head dress at least give some indication s to the severity and scope.
17:10:11 <adamw> sorry, lost track of time
17:10:12 <sgallagh> Penguinpee: We don't have access either. Only the Product Security Team
17:10:13 <adamw> .hello adamwill
17:10:14 <zodbot> adamw: adamwill 'Adam Williamson' <awilliam@redhat.com>
17:10:27 <nirik> and I am not sure they can really say much of anything...
17:10:33 <Penguinpee> ah, I see.
17:10:33 <frantisekz> (and some of the QA people have access too...)
17:10:38 <adamw> Penguinpee: anyone who know details of the bug really is not allowed to say anything beyond what's publicly disclosed already
17:10:41 <Eighth_Doctor> my preference would actually be to not block on this, but I'm super wary of ignoring RH security
17:10:41 <sgallagh> So we have to work from the public information
17:10:52 <jforbes> There is nothing in the bug of any substance
17:11:15 * Penguinpee divides time between cooking dinner and IRC
17:11:31 <cmurf> well unfortunately RH security isn't saying whether Fedora 37 as it is, should be released or delay
17:11:41 <cmurf> even if it's just informational advise
17:11:54 <nirik> so I guess the big thing is if it affects common things like curl / web browsers... as people often use live media to run those things for testing/whateevr.
17:11:54 <Eighth_Doctor> they recommended to mattdm we delay
17:11:54 <adamw> cmurf: yes, they did.
17:12:20 <Eighth_Doctor> nirik: it doesn't affect Firefox, since that uses NSS
17:12:23 <cmurf> Eighth_Doctor: they did? i didn't see that, ok that's sorta changes the calculation I suppose
17:12:24 <adamw> https://pagure.io/fedora-qa/blocker-review/issue/987#comment-822932
17:12:27 <Eighth_Doctor> but it does affect everything else
17:12:32 <adamw> "RH product security tells me that they advise making sure our installation media and various images are updated."
17:12:51 <cmurf> yeah interesting
17:12:51 <cmurf> got it
17:12:52 <frantisekz> it could affect netinst too I guess?
17:12:56 <Eighth_Doctor> technically we could satisfy this if we could do regular image composes, but we can't
17:13:04 <dustymabe> FWIW in Fedora CoreOS we have a bit more control over our users update schedule (automated updates) so we're considering delaying our rebase to F37 for our `testing` stream until after the SSL CVE fix has been rolled out.
17:13:07 <Penguinpee> adamw: but that begs the question sgallagh posed in the ticket
17:13:23 <nirik> Eighth_Doctor: right.
17:13:31 <adamw> yes, that is a reasonable concern
17:13:37 <nirik> anyhow, I guess I'm +1 also to blocking on it. ;(
17:13:40 <Eighth_Doctor> the underlying problem is we have no process to re-compose media and roll that out
17:13:47 <sgallagh> adamw: Which is?
17:13:49 <Eighth_Doctor> we have respins sig for x86 media, but that's it
17:13:55 <cmurf> Eighth_Doctor: agreed
17:13:58 <bcotton> the fact that we're largely stuck with this for 6 months (and entirely stuck on anythign that the Respins SIG doesn't Respin) makes the thought of waiting a week a lot more palatable
17:14:02 <adamw> knowing we have that time constraint, would it be feasible to push go/no-go next week to friday? or can we not do that?
17:14:04 <sgallagh> Conan Kudo: Other than the Respins?
17:14:18 <cmurf> example 92 why we need easier re-compose ability
17:14:25 <nirik> adamw: which time constraint?
17:14:25 <Eighth_Doctor> Stephen Gallagher: respins is technically not a re-compose process for all deliverables
17:14:29 <adamw> Conan Kudo: we have no process because we don't want the burden of doing it. the process isn't the hard part.
17:14:43 <adamw> finding time for releng and qa to go through a whole spin-and-test cycle multiple times during a release is the hard part.
17:14:51 <bcotton> i've been told in the past that we cannot rely on CPE to do any of the releng work over the weekend, so a friday go/no-go seems unpossible
17:15:00 <adamw> nirik: the constraint that the openssl disclosure/release is planned for next tuesday
17:15:12 <adamw> which puts us on a tight timeframe to try and build, test and sign off next thursday
17:15:15 <bcotton> now, could we do a go/no-go the following monday or tuesday and then release on thursday/friday? maybe
17:15:21 <adamw> as sgallagh pointed out at https://pagure.io/fedora-qa/blocker-review/issue/987#comment-823010
17:15:33 <nirik> it could be tight, but I think it's possible...
17:15:45 <bcotton> afaik tuesday releases are mostly 1. tradition and 2. press timing
17:16:11 <nirik> and 0. timing for all the things that need to be done for a release
17:16:15 <Eighth_Doctor> we could release on Thursday as a special case if we want to push it out next week
17:16:51 <adamw> i feel like changing the release date is a bigger and more difficult change than squeezing the go/no-go, but i'm not the one who does the work between thursday and tuesday, i guess
17:16:54 <adamw> i'd have to defer to nirik on that
17:17:09 <adamw> and yes, getting the fix on tuesday and signing off on thursday is possible, it's just tight
17:17:23 <Eighth_Doctor> the openssl update will trigger OpenQA runs at least, so we can get some reasonable data on how much it breaks
17:17:23 <adamw> we need the disclosure to happen, then we need the openssl build to run, then we need the compose to run
17:17:28 <adamw> then we have whatever time is left to validate
17:17:36 <sgallagh> adamw: I think it's too tight, since openssl drives so much of the distro
17:17:39 <cmurf> i'm not sure we can do sufficient regression testing between tuesday and thursday
17:17:45 <bcotton> we're getting a little ahead of ourselves at this point
17:17:46 <nirik> I don't think it's practical to release on thursday... say thing comes out tuesday, we make a rc, we need to test it, get sign off, sync it to mirrors, etc.
17:17:50 <sgallagh> I wouldn't be comfortable carrying over the results of earlier composes
17:17:53 <cmurf> we don't know when tuesday the fix will appear, or how much longer it'll take to get a compose
17:17:57 <bcotton> first we have to decide if this is a blocker
17:18:11 <bcotton> then we can figure out what to do about it
17:18:15 <bytehackr> blocker +1
17:18:26 <Eighth_Doctor> my preference would to be to not block on it, but everyone else suggests we should so...
17:18:34 <Eighth_Doctor> because F36 is equally impacted, as is RHEL 9
17:18:36 <Eighth_Doctor> everyone is toast
17:18:54 <nirik> +1 blocker here.
17:18:56 <adamw> given that i'm not a security expert, the details aren't public, and the advice from rh's security experts is to patch it, i'm still +1
17:19:04 <sgallagh> My point is that if we had shipped on the original schedule, we wouldn't have ground the gears to a halt after Go/No-Go with so little info to go on
17:19:16 <nb> sgallagh good point
17:19:38 <adamw> Conan Kudo: that's outside our area of competency. but if rh prodsec is saying that to us, i would assume they'll also be telling RH that a new RHEL 9.x is needed asap
17:19:40 <cmurf> sgallagh: true but knowledge and our position in the timeline changes things
17:19:45 <Eighth_Doctor> I'd actually be more confident about not blocking if we could update the images :(
17:19:46 <adamw> RHEL does do 'respins', in that sense, of course.
17:20:06 <nb> If I remember correctly, we did an official re-compose at least once in the past
17:20:09 <Eighth_Doctor> adamw: it's probably going to slip right into RHEL 9.1 releasing next month, but who knows
17:20:21 * mattdm is here. sorry, family thing
17:20:25 <sgallagh> nb: Followed immediately by a general agreement never to do so again
17:20:28 <adamw> nb: we did it forever ago, when the compose process was different and we had a lot fewer things, iirc.
17:20:41 <nirik> I am not in favor of doing a release, then doing another release right after when we could wait and do it once. ;)
17:20:44 <adamw> since then we've done kinda unofficial single image fixups, but never a full rebuild
17:20:45 <Eighth_Doctor> we also don't generally have a "make images" button
17:21:19 <cmurf> have we heard from anaconda folks?
17:21:34 <mhroncok> .hello churchyard
17:21:35 <cmurf> that may not be the only thing impacted on the media but seems like the most important thing potentially impacted
17:21:35 <zodbot> mhroncok: churchyard 'Miro Hrončok' <mhroncok@redhat.com>
17:21:36 <nb> I think I am a weak +1 to block.  If Product Security Team says they recommend it, then we might should listen to them
17:21:48 <nb> as much as I want to not block
17:21:49 <bcotton> so i'm seeing roughly +12/-4, which is a pretty strong outcome
17:22:02 <Penguinpee> So RH is mostly concerned about vulnerable images getting out into the wild? But that's still true for all current images.
17:22:02 <Eighth_Doctor> I'm maintaining my +1 even though I really want to do -1
17:22:18 <adamw> i keep saying, don't focus overly on the installer
17:22:27 <adamw> we also have to consider anything people might plausibly do on live imaegs
17:22:29 <Eighth_Doctor> adamw: I'm more worried about arm images
17:22:41 <Eighth_Doctor> live images don't bother me at all
17:22:46 <travier> Penguinpee: yes, all existing images are vulnerable, but if we don't wait, we don't have one that is not
17:22:49 <adamw> Penguinpee: yes, but if we slip a week and include the fix in f37, we have non-vulnerable images out for the public in a week and a half
17:22:52 <sgallagh> adamw: Which is bullcrap, IMHO. Since all of our existing images are riddled with holes
17:22:57 <Eighth_Doctor> I'm not factoring them into my decision, since those are respun regularly
17:22:57 <adamw> if we don't slip, we do not have non-vulnerable images out for another six months
17:23:13 <bcotton> Conan Kudo: not all of them and not on non-x86_64
17:23:13 <nb> I'm not too worried about live images - respins sig makes them every 2 weeks
17:23:21 <nb> but ARM and installer and stuff don't get respun
17:23:34 <adamw> Conan Kudo: since the respins are not official, and not that widely publicized, that's not really a safe assumption.
17:23:38 <Penguinpee> adamw: why is that. there are respins of almost everything.
17:23:43 <Eighth_Doctor> Ben Cotton (he/him): yes, my point is x86 live images are the only things I am not using as a factor for my decision
17:23:45 <nirik> There's also things like server kvm image
17:23:56 <Eighth_Doctor> adamw: which is a mistake in itself that we should seriously fix
17:23:58 <bcotton> none of the labs get respun, either, as far as i can tell
17:24:02 <mattdm> I wish I had data on cloud images -- how many people actually update, or ever use later versions than the first.
17:24:05 <mattdm> But it's guesswork.
17:24:09 <adamw> Penguinpee: there's no validation. we don't really know if the respins...work. and i'm not having QA go through a full validation cycle every two weeks.
17:24:12 <bcotton> so respins are absolutely not the answer here
17:24:12 <cmurf> quite a lot of the +12 is predicated on a presumption that there's something on the media affected by a zero day exploit, if that's not true or has more limited impact, the vote might swing in the exact opposite direction
17:24:13 <sgallagh> If someone is installing the KVM image and not immediately doing a DNF update, they deserve what they get 👍️
17:24:13 <mattdm> I wish I had more data period
17:24:21 <cmurf> but all we have is public information, which is almost nothing
17:24:22 <travier> The problem is not so much about what folks might do with live image but more how bad it can impact netinstalls. If I can compromise a netinstall then it's really bad
17:24:24 <Eighth_Doctor> mattdm: I know people _try_ to use updated cloud images, but that build process breaks so often lately...
17:24:29 <Eighth_Doctor> and the discoverability is super-low
17:24:31 <cmurf> i think anaconda uses its own ssl unless passing a flag to use openssl?
17:24:34 <nb> adamw the respins get tested, maybe not to the extent of what QA normally does, but respins does have their own testnig process
17:24:37 <Eighth_Doctor> we get questions about it in the cloud-sig all the time
17:24:38 <dustymabe> what if the process of updating itself (TLS is everywhere) could compromise the system
17:24:58 <dustymabe> then all those other distros are going to be looking to re-spin their install media too
17:25:12 <adamw> cmurf: we have to consider that possibility based on the public information (a critical vuln in openssl). it's a much safer thing to do to assume that might be the case, than assume it *isn't* the case.
17:25:14 <travier> We have the same issue for FCOS, even though we don't use that much openssl: if you can compromise the first boot of FCOS then that's really bad
17:25:14 <sgallagh> dustymabe: What if openssl turns out it's been a trojan horse all along?
17:25:21 <sgallagh> We can make up whatever boogeymen we want.
17:25:26 <Penguinpee> it feels a bit like meeting a secret agent: trust me (and then all hell breaks loose)
17:25:36 * Eighth_Doctor shrugs
17:25:37 <Southern_Gentlem> better safe now than sorry later
17:25:39 <dustymabe> sgallagh: or maybe they are trying to zip through a trojan horse with this update - rememember remember the 1st of november
17:25:49 <Eighth_Doctor> this is the downside of centralizing your crypto, even if it doesn't happen too often, it's painful when it does
17:25:51 <sgallagh> Ha
17:26:01 <mattdm> sgallagh: (I am not quite sure I'm comfortable with that as a statement. But that's probably another discussion.)
17:26:28 <bcotton> okay, well we can argue all day, but i don't see any minds changing one way or another so
17:26:30 <bcotton> proposed #agreed 2137661 - AcceptedBlocker(Final) - Given the limited public information, we determine this violates"The release must contain no known security bugs of 'important' or higher impact according to the Red Hat severity classification scale which cannot be satisfactorily resolved by a package update"
17:26:37 <nb> ack
17:26:41 <nirik> ack
17:26:42 <sgallagh> That was overstated, but honestly: if you don't patch your systems, we can't do much for you
17:26:46 <Eighth_Doctor> Ben Cotton (he/him): ack 😭
17:26:48 <bcotton> as usual, Stephen Gallagher retains his right to "i told you so"
17:26:51 <Southern_Gentlem> ack
17:26:53 <adamw> Stephen Gallagher: there's also the publicity angle to consider, of course. i do agree to some extent with a lot of your points, but it kinda looks pretty bad if we ship a release on the day this openssl news comes out and people say "so that's fixed, right?" and we say "no"...
17:27:03 * Penguinpee is not changing his mind (based on too little available info)
17:27:05 <adamw> ack
17:27:06 <copperi[m]> ack
17:27:12 <bcotton> i'll also point out that we could still choose to waive this blocker ;-)
17:27:20 <frantisekz> ack
17:27:24 * sgallagh slugs Ben Cotton (he/him)
17:27:24 <adamw> i would probably word it to be less definitive about "this violates", but whatever
17:27:25 <bytehackr> ack
17:27:36 <sgallagh> patch
17:27:38 <Eighth_Doctor> well, it's difficult-to-fix given the code doesn't exist
17:27:38 <adamw> and yes, we can continue arguing about it if someone wants to propose waiving it. ;)
17:27:45 <Penguinpee> nack
17:27:48 <bcotton> Stephen Gallagher: patch away
17:27:49 <mattdm> Penguinpee:  Let me clarify that a little bit. I wish I could give even more information but I don't have much to go on. Since we have so little, I asked some people with more access for their opinions, and they shared it.
17:27:50 <Eighth_Doctor> so you could waive under that condition :P
17:27:50 <adamw> Conan Kudo: sure it exists. it's just not public yet.
17:28:01 <Eighth_Doctor> adamw: doesn't exist to me
17:28:04 <Eighth_Doctor> or anyone else
17:28:10 <Eighth_Doctor> we actually don't even know if a patch is ready either
17:28:15 <sgallagh> proposed #agreed 2137661 - AcceptedBlocker(Final) - Given the limited public information, we are unable to definitively determine whether this violates"The release must contain no known security bugs of 'important' or higher impact according to the Red Hat severity classification scale which cannot be satisfactorily resolved by a package update". We therefore are blocking out of an abundance of caution.
17:28:28 <Eighth_Doctor> we know a disclosure is happening on Nov 1 with a release, we don't know if they're working right up to the deadline
17:28:29 <bcotton> ack
17:28:32 <Southern_Gentlem> ack
17:28:33 <nb> ack
17:28:37 <nirik> sure, ack
17:28:44 <mhroncok> ack
17:28:44 <frantisekz> ack
17:28:46 <dustymabe> +1
17:28:47 <Eighth_Doctor> Stephen Gallagher: ack 😭
17:28:50 <copperi[m]> ack
17:28:50 <travier> +1
17:28:57 <adamw> yeah, i like that more, ack
17:28:59 <jforbes> ick err ack
17:29:00 <sgallagh> Conan Kudo: Generally speaking, we can actually make that assumption.
17:29:07 <Penguinpee> mattdm: thanks. yet I'm not comfortable with making a decision on little to no info. So, maybe I should vote FinalBlocker 0
17:29:21 <bcotton> #agreed 2137661 - AcceptedBlocker(Final) - Given the limited public information, we are unable to definitively determine whether this violates"The release must contain no known security bugs of 'important' or higher impact according to the Red Hat severity classification scale which cannot be satisfactorily resolved by a package update". We therefore are blocking out of an abundance of caution.
17:29:31 <Eighth_Doctor> Stephen Gallagher: having been involved in a few of these before, I don't make that assumption anymore
17:29:31 <sgallagh> Historically, an embargo lift date is never given until a fix is identified (or the embargo has been lifted by other means, such as an accidental disclosure)
17:29:40 <adamw> Conan Kudo: sure we do. they've declared a release date.
17:29:51 <jforbes> sgallagh: that is a rather bold statement
17:29:52 <geraldosimiao> .hello geraldosimiao
17:29:53 <zodbot> geraldosimiao: geraldosimiao 'Geraldo S. Simião Kutz' <geraldo.simiao.kutz@gmail.com>
17:29:54 <dustymabe> Penguinpee: the way I look at is it if my good friend told me to not go outside today, and to trust them. I'd stay inside
17:29:54 <bcotton> okay, moving on
17:29:56 <adamw> Conan Kudo: they didn't announce that they're (only) disclosing an issue, they announced the release of 3.0.7, containing a fix for a CRITICAL issue.
17:30:05 <bcotton> let's review the already-accepted blockers
17:30:06 <bcotton> #topic (2135772) Editing the recurring event freezes Calendar.
17:30:06 <bcotton> #link https://bugzilla.redhat.com/show_bug.cgi?id=2135772
17:30:06 <bcotton> #link https://pagure.io/fedora-qa/blocker-review/issue/977
17:30:06 <bcotton> #info Accepted Blocker, gnome-calendar, NEW
17:30:07 <sgallagh> s/never/rarely/ then?
17:30:27 <Eighth_Doctor> Stephen Gallagher: we'll go with that
17:30:27 <adamw> so, if we didn't have anything else, i'd be in favor of waiving this
17:30:36 <adamw> but, now we have something else.
17:30:36 <sgallagh> dustymabe: https://xkcd.com/1170/
17:30:36 <jforbes> sgallagh: frequently embargo lift dates are set before work has even begun on a fix
17:30:41 <bcotton> So this is the one that upstream said "wow, this is totally bad"
17:30:59 <nb> I would propose to waive 2135772
17:31:16 <sgallagh> That tends to happen more with cross-project flaws than ones in a specific library.
17:31:19 <adamw> i don't see any point in waiving it now if we're gonna slip for openssl
17:31:34 <adamw> in fact we kinda only intended the waiver process to be used in order to sign off a release
17:31:38 <sgallagh> Like if an algorithm is discovered as vulnerable in multiple implementations.
17:31:39 <Penguinpee> dustymabe: i'd be with you if this were about a good friend. i don't feel like that regarding RH (no offense)
17:31:44 <bcotton> #info Upstream says "I'm pretty positive nothing about recurrent events has ever worked properly. There's a lot more that needs to be done to make it work reliably, and I'm considering how to do that, but yeah, it's just stupidly broken."
17:31:48 <nb> adamw well, true
17:31:51 <adamw> if we agree to waive this but we're still no-go, it puts it in a kind of weird twilight zone
17:32:06 <cmurf> waive them both
17:32:11 <adamw> so, is anyone going to propose that we waive the openssl bug?
17:32:14 <bcotton> waivers are only valid until the next go/no-go, imo, whether that's the same release or the next milestone
17:32:19 <cmurf> difficult to fix and late proposed blockers
17:32:43 <bcotton> let's get through reviewing all of the blockers and then we'll discuss waiving
17:32:57 <cmurf> fair
17:33:06 <bcotton> anything else to add on 2135772?
17:33:06 <nirik> is this one likely to get a fix?
17:33:22 <cmurf> sounds like it's ambiguous whether it gets a fix
17:33:40 <bcotton> nirik: i wouldn't bet on it getting a fix for f37 at this point
17:33:45 <Eighth_Doctor> I don't think this is getting a fix
17:33:46 * nirik wonders if we could disable recurrent events until they are fixed
17:33:55 <cmurf> it's not a data loss or corrupting bug, it just doesn't change the recurring event and the program crashes?
17:33:58 <Eighth_Doctor> unless the fix is hiding editing recurring events
17:34:20 <sgallagh> cmurf: Can't be deleted, either
17:34:27 <cmurf> it's a bad bug but i'd say it's going to get fixed on its own time scheduled regardless of what we have to say about it
17:34:33 <bcotton> that sounds like a question for the Workstation WG to decide
17:34:40 <cmurf> if it's made a blocker, Workstation WG will just remove it from the media
17:34:50 <bcotton> so let's look at our last accepted blocker
17:34:50 <cmurf> we've discussed it and that's what would happen
17:34:56 <bcotton> #topic (2137600) Plasma on Wayland hangs at startup when booting with nomodeset ("basic graphics" mode) on UEFI
17:34:57 <bcotton> #link https://bugzilla.redhat.com/show_bug.cgi?id=2137600
17:34:57 <bcotton> #link https://pagure.io/fedora-qa/blocker-review/issue/986
17:34:57 <bcotton> #info Accepted Blocker, kwin, NEW
17:35:04 <sgallagh> cmurf: It's already declared a blockr
17:35:15 <geraldosimiao> Ohh, that
17:35:16 <sgallagh> *blocker
17:35:36 <sgallagh> (Blockr is the dating app for blocker bug meeting participants...)
17:35:37 <Eighth_Doctor> ...
17:35:42 <adamw> Stephen Gallagher: wow, worst dating app *ever*
17:36:04 <adamw> so, yeah, we basically know what's going on here and the proposed kernel patches would probably fix it
17:36:07 <Eighth_Doctor> 🤦‍♂️
17:36:25 <Eighth_Doctor> yeah, it also looks like we're going to add basic graphics mode testing too
17:36:42 <Eighth_Doctor> hopefully we can stop this trend of discovering bugs with basic graphics mode at the very last second
17:36:43 <adamw> well, we already had it, but we've extended it to make it clearer what combinations need testing
17:36:56 <adamw> which is a sad chart of sadness, but hey.
17:37:02 <Eighth_Doctor> yeeeeeah
17:37:11 <travier> Reading the bug I was wondering who would be doing that but now I get it 🍎
17:37:25 <Eighth_Doctor> yuuuup
17:37:40 <Eighth_Doctor> blame fancy computers
17:37:51 <travier> Will we officially support M1's with F37?
17:37:57 <sgallagh> NO
17:38:12 <travier> so we should probably not block then
17:38:52 <adamw> travier: the bug we have does not affect M1s
17:38:56 <adamw> in fact they're likely the only thing that works
17:39:01 <travier> oh
17:39:06 <adamw> (well, them and any other platform that uses a native 10-bit framebuffer)
17:39:11 <mattdm> M1s for everyone!
17:39:12 <Eighth_Doctor> no, it broke non-M1
17:39:19 <Eighth_Doctor> that's why the bug
17:39:28 <Eighth_Doctor> M1 is the only type that works right now 🤣
17:39:38 <travier> Oh, so it's for NVIDIA setups then?
17:39:45 <Eighth_Doctor> yup
17:39:48 <adamw> earlier on i was suggesting we could go with a patch that fixes other things but breaks M1s, but that's not a concern any more, the proposed kernel patches ought to make things work for everyone.
17:40:00 <bcotton> so it sounds like there are at least short-term patches in the works that we could incorporate relatively quickly
17:40:01 <travier> ok, so it makes more sense to block on it then
17:40:10 <adamw> travier: it's already a blocker
17:40:11 <bcotton> perhaps even Actual Fixes™?
17:40:20 <adamw> we're at the point in the meeting where we discuss existing blockers and see what's going on with them
17:40:24 <travier> .👍
17:40:24 <Eighth_Doctor> yeah, we can probably ship it into the kernel nowish if drm folks are tentatively okay with it
17:40:31 <adamw> so, assuming we're not still trying to ship this week, this should be fine.
17:41:03 <jforbes> wait, what? I thought this was a kwin bug, not kernel
17:41:05 <adamw> we'll let the drm folks fight about it a bit then jforbes can include...whatever they decide on. or something we decide on if they can't agree.
17:41:12 <Eighth_Doctor> jforbes: it's both
17:41:19 <adamw> jforbes: it's...sort of both, but we have proposed kernel patches to fix it
17:41:28 <jforbes> lovely
17:41:44 <adamw> jforbes: see https://bugzilla.redhat.com/show_bug.cgi?id=2137600#c37
17:41:44 <bcotton> should we re-assign this bz to the kernel component?
17:41:51 <Eighth_Doctor> kwin is working on a fix too, but it might not be backportable to 5.26
17:42:00 <adamw> apparently v1 of that patch incited a hot controversy on "IRC", i've no idea what IRC channel or what the controversy was.
17:42:03 <Eighth_Doctor> definitely don't expect a 5.25 backport, since the 5.25 series just ended
17:42:09 <adamw> Ben Cotton (he/him): i think it's okay, everyone relevant is on it.
17:42:15 <bcotton> roger
17:42:28 <bcotton> #info kernel patches exist to address this issue
17:42:28 * jforbes notes that 6.0.5 kernels are in updates-testing now.
17:42:53 <jforbes> Which, given that 5.19 is EOL, and 5.19.17 is known broken, isn't such a bad thing, but...
17:42:56 * Eighth_Doctor notes plasma 5.26 is in updates-testing now
17:43:20 <adamw> yeah, i don't know if we're having that debate in this meeting...
17:43:44 <Eighth_Doctor> kernel and plasma are in the same boat at the same time :/
17:44:20 <bcotton> okay, anything else on this one?
17:44:44 <jforbes> adamw: I know, and given the workflow now, I can always go back, but F35 and 36 will be moving forward either way, not going to hold them back
17:44:47 <Eighth_Doctor> adamw: could we add basic graphics mode testing to OpenQA?
17:45:07 <Eighth_Doctor> even to just suss out super-basic stuff like this would be a huge win
17:45:15 <sgallagh> I could maybe be talked into a one-week slip if we have small, targeted fixes for the blockers. But if we're bumping major components, I'm going to push for a two-week slip.
17:45:23 <frantisekz> it doesn't encounter this issue in vms
17:45:36 <Eighth_Doctor> it's reproducible in UEFI VMs
17:45:54 <bcotton> okay, sounds like we've said all that we need to say on this particular bug for now. so
17:45:57 <geraldosimiao> Eighth_Doctor: Yes
17:46:11 <bcotton> #topic Current status - blockers
17:46:11 <bcotton> #info 3 Accepted Blockers
17:46:34 <bcotton> does anyone want to propose that we waive all three accepted blockers?
17:47:32 <Eighth_Doctor> two of the three blockers have fixes we could release nowish
17:47:37 <adamw> Conan Kudo: we could, yeah. i didn't do it in the past because it's not considered 'valid' for release validation, but yeah, might be worth it to catch some problems. the concern might be that it would run into some bug that nobody wants to fix because it only affects VMs and there's no reason to use nomodeset in a VM
17:47:57 <bcotton> Conan Kudo: if we waive them, they don't get released
17:47:57 <adamw> Conan Kudo: er...two?
17:48:12 <Eighth_Doctor> there's a patch for calendar to hide editing recurring events, afaik
17:48:27 <Eighth_Doctor> and see aforementioned dri-devel posts for kernel fixes
17:49:06 <adamw> oh, okay. not sure if hiding recurring events is really the answer, but meh. i would say, one way or another, i don't want to slip for the calendar bug alone. whether that means waiving it or removing calendar or hiding recurring events i don't care
17:49:21 <adamw> the KDE nomodeset one i was willing to waive for being discovered late and having a workaround (use the X11 session)
17:49:33 <Eighth_Doctor> sure
17:49:33 <adamw> but the openssl one feels like...we really should block on it
17:49:42 <nb> -1 to waiving openssl
17:49:47 <adamw> it doesn't make much sense to say "we're accepting it as a blocker out of caution" and then say "...but we're not actually blocking on it"
17:49:56 <mattdm> Right. :)
17:50:04 <sgallagh> I agree. We need to block for openssl at this point.
17:50:20 <sgallagh> Decision was made, let's stick to it.
17:50:24 <Eighth_Doctor> I'm not saying we should waive anything
17:50:32 <Eighth_Doctor> we already put ourselves into this state, we should stick to it
17:50:43 <Eighth_Doctor> I'm just pointing out that the non-openssl blockers have potential fixes
17:50:44 <bcotton> i'm not seeing any proposals to waive, so let's move on
17:50:57 <sgallagh> So as long as we're not waiving at least one, waiving the others is irrelevant
17:51:04 <mattdm> For the record, I am _not_ 100% sure that come Tuesday, we won't all be like "what, really, that's what that was all about?"
17:51:05 <bcotton> exactly! it's all or nothing
17:51:15 <nirik> catch the waive
17:51:25 <mattdm> But, I'm, like, 70% sure? Which is pretty high in terms of confidence about the future in Fedora in general :)
17:51:26 <sgallagh> 🏄‍♂️
17:51:35 <adamw> mattdm: sure, me either. but we've gotta be responsible, i guess
17:51:46 <Eighth_Doctor> 🏄‍♂️
17:51:55 <adamw> we will accept only a moderate amount of 'told ya so' from sgallagh
17:52:16 <cmurf> i dunno i think i'd waive all security bugs until we have a secure initrd :\
17:52:16 <bcotton> #topic Current status - test matrices
17:52:16 <bcotton> #link https://fedoraproject.org/wiki/Category:Fedora_37_Test_Results
17:52:38 <sgallagh> I'm not going to play that card. I respect the decision, I just think it would have been a different story a week ago.
17:52:40 <bcotton> with the understanding that adam is considering invalidating all of our current tests ;-) how are things looking?
17:53:06 <adamw> seems a bit pointless to go over this now
17:53:18 <adamw> but briefly, if we were going to sign off on this candidate, coverage is great. :P
17:53:43 <cmurf> preemptive invalidation 😛
17:54:24 <bcotton> #info Coverage is great
17:54:25 <bcotton> #topic Current status - Release Candidate
17:54:25 <bcotton> #info We have one. It has blockers
17:54:30 <bcotton> #topic Go/No-Go decision
17:54:44 <nirik> pretty clearly no go today.
17:54:53 <sgallagh> Nyet
17:54:58 <bcotton> #info By policy, F37 Final is NO-GO due to outstanding blockers
17:55:15 <adamw> no-go, indeed.
17:55:21 <bcotton> #info The next F37 Go/No-Go meeting will be Thursday, 2022-11-03 at 1700 UTC
17:55:21 <Eighth_Doctor> 😭
17:55:21 <bcotton> #info F37 shifts to target date #3: Tuesday 2022-11-08
17:55:29 <sgallagh> I think we need to discuss the slip, though
17:55:32 <bcotton> #info The Release Party will happen prior to the release
17:55:53 <sgallagh> Or rather, whether we're going to accept big changes
17:56:00 <Eighth_Doctor> oh gosh
17:56:01 <bcotton> #link https://fedoramagazine.org/youre-invited-to-the-fedora-linux-37-release-party/
17:56:01 <bcotton> #topic Now what?
17:56:09 <Eighth_Doctor> well, we could always turn it into an upgrade party?
17:56:20 <Eighth_Doctor> people can technically upgrade to 37 ahead of time now
17:56:22 <sgallagh> As I said above, I think if we are going to pull in a new kernel, new KDE and new Firefox... we should slip two weeks.
17:56:34 <mattdm> sgallagh: Aka "I hope Adam was joking with that comment in the blocker vote ticket?"
17:56:41 <adamw> which kinda ties in with the 'we might need two weeks anyway given the openssl schedule' thing
17:56:43 <sgallagh> Because that's an unreasonable amount of burden to put on the QA folks.
17:56:44 <bcotton> #info There's a rumination on the table to do additional updates
17:56:46 <bcotton> #link https://pagure.io/fedora-qa/blocker-review/issue/987#comment-823063
17:56:56 <bcotton> i didn't call it a proposal on purpose, but
17:56:56 <adamw> i wasn't really joking. i do think there's a case for it
17:57:01 <adamw> as jforbes says, kernel 5.19 is EOL now
17:57:03 <nirik> I don't think we need to slip extra... if there's bugs it won't get a go
17:57:08 <Eighth_Doctor> and so is Plasma 5.25
17:57:10 <jforbes> Well, No one has agreed to pull in a new kernel, but if we were to do so, we did have a full 6.0 kernel test week just last week.
17:57:21 <jforbes> And F36 and F35 will be on 6.0 kernels
17:57:27 <mattdm> I mean, there's always new stuff coming out just around the corner.
17:57:41 <Eighth_Doctor> mattdm: yes, but not literally sitting in updates-testing
17:57:53 <bcotton> that's a lot of churn in a small window
17:57:56 <cmurf> if jforbes is on board with 6.0 in f37, I will quickly get on board with that
17:57:57 <adamw> it just feels like, if we're going to be pretty late with the release, it's a better story to say "look, we baked you a nice shiny release with kernel 6.0 and GNOME 43.1 and Plasma 5.26" than "look, we baked you all this stuff that's old now"
17:57:59 <mattdm> Conan Kudo: Sometimes! "Zero day"
17:58:16 <cmurf> it's a hilarious set of tradeoffs, sure
17:58:19 <cmurf> just pick one
17:58:19 <Eighth_Doctor> and Plasma 5.26 has been sitting in testing for a while now
17:58:23 <jforbes> And from a press standpoint, shipping with a 5.19 kernel when 6.0 is out, is one thing, but after 5.19 is EOL kinda sucks.  Plus we get a good bit of hardware support
17:58:25 <bcotton> if we pulled in gnome, kernel, and plasma today and did an RC to test so that we're just waiting for openssl, i'd be okay with that, i think
17:58:27 <adamw> it certainly is more risky from a QA perspective, but this is me with my Fedora on, not my QA hat. :D
17:58:42 <nirik> bcotton: +1
17:58:48 <sgallagh> Ben Cotton (he/him): That's a reasonable approach.
17:58:49 <cmurf> adamw: not confidence inspiring!
17:58:50 <adamw> the GNOME update isn't built yet, i'd have to talk to workstation folks about the schedule for that
17:58:57 <adamw> Plasma  and kernel are in u-t already though
17:58:59 <jforbes> But I did not force 6.0 for a reason, so I think it is entirely QAs call as to whether we go with 6.0 or not
17:59:01 <adamw> and firefox
17:59:03 <bcotton> i wouldn't want to wait and then try to do the whole matrix in ~36 hours
17:59:27 <adamw> with my QA hat on, i am more relaxed about stuff like this than i would have been in years past, since we do have openqa now, and good communication with upstreams, and so on
17:59:38 <Eighth_Doctor> Plasma updates run through OpenQA now
17:59:47 <Eighth_Doctor> so they actually do get significant automated testing
17:59:52 <adamw> pulling in new stuff isn't a complete leap into the unknown like it used to be
17:59:55 <mattdm> Can we validate an RC with just the minimal openssl fix and also the other thing in parallel or close to parallel?
18:00:05 <nirik> even if we don't do another rc, we can test nightlys...
18:00:14 <adamw> Ben Cotton (he/him): oh, i wouldn't want that either. this is kinda presuming we slip *two* weeks
18:00:40 <bcotton> the gnome update shouldn't be too disruptive and we just did a kernel 6.0 test week as jforbes noted
18:00:40 <adamw> mattdm: we could, yes. the tricky thing there might be the KDE nomodeset fix
18:00:48 <cmurf> since the whole internet runs on openssl, i'm pretty sure cloud, server, and iot folks will want to see thorough regression testing lest we make our lives more miserable than shipping with a 0day
18:00:49 <bcotton> nirik: i suspect we get more testing if we call it an RC, but that might be wrong
18:00:52 <adamw> we are gonna need a kernel and/or kwin update to fix that
18:01:00 <sgallagh> cmurf++
18:01:00 <zodbot> sgallagh: Karma for chrismurphy changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
18:01:08 <cmurf> Workstation may not be as susceptible to openssl regressions but that's just a guest
18:01:09 <cmurf> guess
18:01:20 <nirik> bcotton: well, if we just push those things stable, nightly's will be testable... but of course we can't go back
18:01:23 <Eighth_Doctor> we can talk to kde upstream about a 5.26 kwin fix, but I'm much more confident on jforbes pulling the simpledrm patch
18:01:32 <mattdm> Two weeks is based on the assumption that we won't have a trustable fix in time on Tuesday to have a compose we feel good about on Thursday?
18:01:36 <Eighth_Doctor> Hugl hasn't started working on a fix yet for kwin
18:01:50 <adamw> i suppose we could have one RC with only the openssl fix, and releasing that would require waiving the nomodeset bug; and another RC with all the new stuff, including a fix for the nomodeset bug...
18:02:06 <adamw> mattdm: yes.
18:02:10 <sgallagh> mattdm: It's based on the fact that OpenSSL is *everywhere* and I at least wouldn't be comfortable with a 36-hour testing marathon before Go/No-Go
18:02:18 <nirik> thats a lot more work sadly.
18:02:34 <sgallagh> Because I'd want to see a thorough retest with that change, particularly if it happens to be some core functionality.
18:02:38 <mattdm> nirik: I would definitely feel better with a "go back" option.
18:02:51 <adamw> even if the fix appears at 00:01 tuesday in europe, or something, we still need the openssl package build to happen and someone to file a compose request (could be done earlier if it's not me, i guess) and the compose to happen before we can start testing it
18:02:55 <Eighth_Doctor> we could also create nightlies with u-t content, no?
18:03:01 <cmurf> sgallagh: and we won't know until late tuesday
18:03:04 <Eighth_Doctor> like that's a thing that should be technically possible
18:03:09 <cmurf> afternoon EDT
18:03:15 <jforbes> Let's just drop openssl...
18:03:35 <Eighth_Doctor> too late for that, we just made systemd depend on it :(
18:03:44 <sgallagh> So realistically, if we're blocked for openssl, I think we really should give ourselves two weeks, possibly lifting the Freeze for the remainder of this week?
18:04:06 <cmurf> jforbes: more practical to drop initrd's 😆
18:04:08 <Eighth_Doctor> Stephen Gallagher: I think that's sensible
18:04:11 <jforbes> heh, was a joke, a whole lot of things depend on it, and it would take us months to unwind
18:04:23 <nirik> doing 2 rc's would allow for a 'go back' but I think would also result in less testing... only those who specially tested that rc would have those things. If we landed them in nightlys everyone running f37 would get them
18:04:31 <jforbes> cmurf: there is a proposal for that of sorts.
18:04:32 <sgallagh> Nah, we can just slot in libressl, right? /me runs
18:04:43 <Eighth_Doctor> Stephen Gallagher: ugh, sigh
18:04:44 <nirik> -1 on lifting freeze
18:04:51 <nirik> or perhaps -100
18:04:52 <adamw> oh, the announcement has a window
18:04:53 <adamw> "This release will be made available on Tuesday 1st November 2022 between
18:04:53 <adamw> 1300-1700 UTC."
18:04:54 <cmurf> sgallagh: i think 2 weeks is reasonable to just assume at this point, but i don't know about everything being released from freeze...
18:05:03 <cmurf> that could be a big can of worms
18:05:15 <dustymabe> yeah I'm not a big fan of opening the flood gates again
18:05:17 <nb> I donj't think we should lift the freeze.  Allowing a few shiny updates through, maybe, but not wide open
18:05:26 <bcotton> practically speaking, FESCo would have to approve lifting the freeze and by the time we got enough votes to do it, it'd close again anyway
18:05:28 <adamw> openssl builds take ~40 minutes
18:05:38 <mhroncok> Maybe we can get Python 3.11 final in
18:05:40 <adamw> so at absolute best, really, we're gonna be kicking off the compose at 1400 UTC
18:05:49 <mattdm> I'm definitely envisioning an impending disaster with lifting the freeze.
18:05:50 <sgallagh> OK, I withdraw the Freeze-lift suggestion
18:05:57 <adamw> which means it'd be finished about...2000 UTC?
18:06:00 <cmurf> > <@adamwill:fedora.im> "This release will be made available on Tuesday 1st November 2022 between
18:06:00 <cmurf> > 1300-1700 UTC."
18:06:00 <cmurf> alright so roughly by noon on tuesday, EDT
18:06:02 <travier> -1 to lifting freeze, +1 to cherry-picking some updates
18:06:18 <Eighth_Doctor> mhroncok: is py3.11 final in u-t yet?
18:06:28 <adamw> yeah, i wouldn't propose lifting the freeze, we're not that late. i'd rather want to target shiny things, as i said - kernel, gnome, plasma, firefox
18:06:41 <mhroncok> Eighth_Doctor: Bodhi says so https://bodhi.fedoraproject.org/updates/FEDORA-2022-a9a4c48d06 -- may not be on the mirros yet
18:06:43 <jforbes> So, I do want a little bit of time for feedback to pass on the kernel fix, but the plan is to put that fix into 6.0.5 and do another build?
18:07:10 <adamw> jforbes: basically, yeah. we'd defer to your sense for the upstream process there, but of course the earlier we can get a build to test the better
18:07:15 <mhroncok> s/mirros/mirrors/
18:07:15 <mattdm> Even with that, I'm a little worried. The only thing that balances that is ... those things are coming to people anyway and this gives them more testing...
18:07:27 <jforbes> Or 6.0.6 will release on Saturday
18:07:58 <adamw> mattdm: yeah, if we don't pull these things in, people get them as a 0-day update anyway
18:08:08 <sgallagh> Additional question: who decides which things get cherry-picked in?
18:08:17 <adamw> which, i dunno. who likes giant 0-days? i feel like it makes people think 'why isn't this stuff in the release?'
18:08:25 <adamw> Stephen Gallagher: we use FEs, i guess. that's what they're for.
18:08:36 <sgallagh> Ah, right. That makes sense.
18:08:47 <adamw> if there seems to be general support for the concept, i will file FE bugs after this meeting, and we can vote on 'em.
18:08:54 <nirik> in the end QE decides which FE's are pulled in
18:08:58 <sgallagh> (Suffering from a head-cold today, so not braining entirely well)
18:09:12 <adamw> jforbes: saturday would be fine, i guess, if we're assuming a two week slip.
18:09:18 <bcotton> okay, so how's this for an omnibus summary of The Plan?
18:09:31 <mattdm> Is there any way we could do an openssl-minimal-fix compose and snapshot that state if everything turns out to be ... pear-shaped...  two weeks from now?
18:09:40 <bcotton> proposed #agreed Target date #3 shifts to 15 November in order to allow time for thorough verification of Tuesday's openssl update. Kernel 6.0 will be allowed to fix RHBZ 2137600. We will vote for FEs for GNOME and Plasma updates through the usual process
18:09:50 <sgallagh> Proposal: Slip two weeks for a new target date of Nov. 8, allow certain important packages to request FEs
18:10:05 <sgallagh> Sorry, Nov 15
18:10:05 <bcotton> err
18:10:08 <adamw> mattdm: we *could*. that would more or less require us to wait until tuesday to start including anything *else*, though
18:10:22 <adamw> or, well. hmm. at least, we couldn't push anything else stable till we get that minimal compose done.
18:10:32 <sgallagh> s/8/15/
18:10:33 <adamw> nirik: wdyt?
18:10:34 <bcotton> my proposal is probably wrong, but we can fix it!
18:10:37 <nirik> and like I said, the stuff would get less testing
18:10:57 <adamw> Ben Cotton (he/him): patch; i'd rather just send kernel through the FE process too
18:11:16 <adamw> i'm anticipating people will vote for it, but if they don't, we can do a sort of rollback and do a 5.19 kernel with the drm fix. it's just kind of a pain for jforbes.
18:11:16 <sgallagh> ^Same
18:11:35 <bcotton> proposed #agreed Target date #3 shifts to 15 November in order to allow time for thorough verification of Tuesday's openssl update. We will vote for FEs for kernel, GNOME, and Plasma updates through the usual process
18:11:41 <bcotton> dang. that's what i had originally :-)
18:11:41 <nirik> if we made a rc1.5 with kernel/gnome/plasma/firefox/whatever it would only get testing from openqa + users specifically testing it. If we push them stable, everyone running f37 will get them
18:11:48 <sgallagh> patch
18:11:48 <nb> ack
18:12:06 * bcotton awaits another patch
18:12:22 * nirik wonders if we shouldn't leave open the chance of one week...
18:12:24 <sgallagh> proposed #agreed Target date #3 shifts to 15 November in order to allow time for thorough verification of Tuesday's openssl update. We will vote for FEs through the usual process to allow some important late features to land.
18:12:26 <nirik> but otherwise ok
18:12:47 <bcotton> i can live with that. i specifically called out those three to show intent, but the end result is the same
18:13:12 <adamw> nirik: it's reaaaaaally borderline. best case we have a compose for testing at 2000 UTC on tuesday, before go/no-go at 1700 UTC (right?) thursday
18:13:13 <sgallagh> Right, but since we've already had at least a fourth (python 3.11) proposed, I figured it was better to be generic
18:13:25 <adamw> 2000 UTC tuesday is pretty much after brno is done testing for the night, so realistically they get one day on it
18:13:27 <mattdm> nirik: The difference would be tomorrow, the weekend, monday, and probably tuesday, right?
18:13:32 <adamw> me and the other NA folks get a day and a half
18:13:43 <adamw> Stephen Gallagher: yeah, also firefox
18:14:03 <frantisekz> I am planning to propose FE for mozjs102/gjs combo too
18:14:13 <nirik> mattdm: yes...
18:14:20 <adamw> frantisekz: would that not be part of gnome 43.1?
18:14:27 <sgallagh> Propose all you want; doesn't guarantee acceptance ;-)
18:14:53 <frantisekz> heh, should be, I shall ping somebody from Workstation WG to just edit my patch or pull those builds from it to GNOME mega
18:14:53 <mattdm> I'm -1 to this proposal without having a minimal just-the-openssl (waive the other current blockers) fallback
18:15:35 <frantisekz> I guess we could do that as RC 1.5?
18:15:43 <frantisekz> and RC 1.6 mega thingy?
18:16:00 <nirik> other way?
18:16:03 <sgallagh> mattdm: With respect, those other issues *are* blockers. We were going to waive them via the Late Blocker exception, but now that time is available they should really get included if at all possible
18:16:22 <mattdm> I am open to bringing back release names for "Fedora Linux 37 (Mega Thingy)"
18:16:33 <frantisekz> heh :D
18:16:39 <sgallagh> That... might not go so well in newsfeeds.
18:16:46 <bcotton> "Fedora Linux 37 (In a Row?)"
18:16:52 <sgallagh> "Fedora Linux 37 Creepy Spam"
18:17:18 * nb proposes "Fedora Linux 37 (The Return of Beefy Miracle)"!!
18:17:25 <jforbes> mattdm: only if you can get a logo designed to go with it
18:17:37 <sgallagh> Isn't that just a synonym for mattdm's?
18:17:45 <nb> lol
18:17:50 <mattdm> Stephen Gallagher: Yes, that's the same exception I'd suggest waiving them under. That compose would be as if we are trying to get a release out as fast as possible.
18:18:30 <sgallagh> mattdm: As noted above, that still limits us severely, since we can't (realistically) start on the testing of those fixes prior to getting the OpenSSL patch.
18:19:08 <sgallagh> And if we start work on them, we need to finish it. Backing them out will be harder than fixing them..
18:19:55 <adamw> > * <@nb:libera.chat> proposes "Fedora Linux 37 (The Return of Beefy Miracle)"!!
18:19:55 <adamw> needs more unicode
18:19:59 <nirik> we can be somewhat cautious landing things... I think we could do some test compose and run it thru openqa at least? and//or tell users 'install these from updates-testing' and test?
18:20:06 <mattdm> I mean... the openssl stuff needs to be tested anyway
18:20:32 <sgallagh> mattdm: It needs to be regression tested.
18:20:32 <cmurf> adamw: Variation in needs more cowbell?
18:20:33 <nirik> but it would be good to test the rest now while we have time instead of waiting until we have openssl in hand...
18:20:37 <sgallagh> Bug verification is different
18:20:43 <bcotton> alright, so the current proposal on the table is
18:20:43 <bcotton> proposed #agreed Target date #3 shifts to 15 November in order to allow time for thorough verification of Tuesday's openssl update. We will vote for FEs through the usual process to allow some important late features to land.
18:20:45 <bcotton> we know what mattdm thinks of it. where does everyone else stand?
18:20:45 <cmurf> * Variation on needs more cowbell?
18:20:45 <adamw> yes, unless i'm missing something, we could do a candidate compose with other stuff in it before the openssl-only compose, if we wanted
18:20:54 <adamw> as we can add things to and remove things from the side tag as we like
18:20:57 <sgallagh> ack
18:21:09 <nb> ack
18:21:13 <mattdm> Yes: reminder... I am the FPL, not the Fedora Pope :)
18:21:21 <copperi[m]> ack
18:21:28 <adamw> i am on board with the plan, i'm willing to work with releng to accommodate matt's "get a minimal openssl-fix-only compose on tuesday" plan if it's generally desired, we can work out the details as we go
18:21:36 <Eighth_Doctor> let's land all the stuff ahead of openssl
18:22:03 <adamw> we can't "land" it, in the sense of push it stable, before openssl, if we want to do the minimal compose
18:22:04 <nirik> Eighth_Doctor: that would be my preference too, then we only have openssl to deal with
18:22:24 <adamw> but we can do a candidate compose with kernel 6.0, plasma 5.26, firefox 106, and whatever else we want. if we like. we could do that now.
18:22:27 <nirik> right, having a openssl only compose makes it harder
18:22:28 <sgallagh> nirik, Conan Kudo same here
18:22:46 <sgallagh> adamw: I like. Do it.
18:23:31 <travier> +1 for adding things early to give time to test
18:23:44 <travier> then the openssl fix will be the cherry
18:23:51 <mattdm> Just so I'm understanding: the candidate compose would pull those things in but they wouldn't be moved to stable generally?
18:23:54 <adamw> right now we don't have the simpledrm fix in the kernel, and we don't have gnome 43.1. but we could do a build with other things, i guess.
18:24:19 <nirik> mattdm: right.
18:24:22 <jforbes> adamw: Well, 6.0.5 is in update's testing, so if you are going to do one, push that in
18:24:25 <adamw> mattdm: that's an option, yes. nirik is arguing (AIUI) for pushing them stable earlier so we get more testing (from folks who don't have u-t enabled). the cost of that option is we could not do the minimal compose.
18:24:29 <nirik> we have a f37-compose tag we can put things in
18:24:43 <nirik> right
18:24:59 <jforbes> It doesn't have the particular fix, but the specific fix is easy to verify
18:25:19 <adamw> (this is, of course, assuming we accept FEs)
18:25:33 <adamw> jforbes: yeah. i can build a scratch kernel to test the fix, i guess
18:25:42 <bcotton> okay, so it sounds like we have a general consensus
18:25:47 <jforbes> adamw: I can build one with it
18:25:59 <adamw> Ben Cotton (he/him): well, we are still arguing the 'do a minimal compose or not' point, but i guess we can continue that outside the meeting
18:26:11 <bcotton> #agreed Target date #3 shifts to 15 November in order to allow time for thorough verification of Tuesday's openssl update. We will vote for FEs through the usual process to allow some important late features to land.
18:26:12 <adamw> jforbes: if you could that'd be great
18:26:14 <jforbes> Just wanted to see a little feedback when we have 2 different patches pushed out in hours with heated discussion
18:26:21 <jforbes> both today
18:26:31 <adamw> jforbes: i'll leave it up to you whether to go with v1 or v2...the drm-devel list does seem to have the discussion about what some people didn't like in v1
18:26:46 <adamw> er, dri-devel
18:27:02 <jforbes> right
18:27:05 <adamw> https://lists.freedesktop.org/archives/dri-devel/2022-October/377407.html
18:27:25 <mattdm> The "additional testing" would be from people who have updated already and might notice if we break something horribly?
18:27:37 <mattdm> (Sorry I'm slow at typing and thinking both at once today.)
18:27:57 <adamw> mattdm: yes. we've already disabled u-t by default for f37 now
18:28:03 <adamw> so a typical f37 install does not get the stuff from u-t
18:28:11 <adamw> only people who actively enable it, like for stable releases
18:28:12 <mattdm> (I noticed: I had to turn it back on!)
18:28:31 <adamw> if we push things stable, anyone who's running f37 but hasn't turned on u-t will get it, and hopefully yell if it breaks stuff
18:29:05 <mattdm> What practical benefit would that give vs. doing it 5-6 days from now? Gonna break either way.
18:29:32 <mattdm> Or rather, if we do it later but have the composes you mentioned, hopefully we'll catch it before then.
18:30:10 <nirik> less time to fix things we do find...
18:30:11 <dtometzki> what is u-t ?
18:30:17 <nirik> updates-testing repo
18:30:53 <mattdm> nirik: Assuming that they're _only_ found by people we shipped broken stuff to (on beta, to be fair)
18:31:15 <adamw> the practical benefit would be we'd find out sooner and have more time to fix whatever was borked.
18:32:05 <mattdm> Yeah, I get the more time thing -- but that's assuming that the wider push will find things we wouldn't with u-t + validation testing, right?
18:32:16 <adamw> yes.
18:32:39 <nirik> The number of users of f37 is much higher than the number who test rc's... :) at least I think so
18:33:11 <bcotton> i think we can work out this part of the plan outside of the meeting
18:33:21 <bcotton> anything else before we close out?
18:33:22 <mattdm> I guess I'm setting the "big important hard to fix thing found in that way" weight as low in my thinking.
18:33:47 <nirik> fedora-37 	0.1766 % 	66767
18:34:05 <mattdm> Ben Cotton (he/him): Okay. I've said my bit. I think the risk mitigation is worth it.
18:34:22 <bcotton> now you're speaking my language
18:34:30 <nirik> (ie, 67k mirror hits for f37 today)
18:34:47 <mattdm> nirik: one sec, looking at countme
18:36:41 <mattdm> For this week, 33220 for updates-released and 9367 for updates-testing.
18:37:52 <mattdm> And generally the former should be a superset of the latter. So about 28% of current F37 systems have the testing repo enabled
18:38:19 <nirik> right, but thats not what I am compairing. ;) That number to... say 10 qe people?
18:38:43 <adamw> https://bugzilla.redhat.com/show_bug.cgi?id=2138247 - kernel 6.0 FE proposal
18:38:50 <nirik> anyhow, I'll stop we can do the fallback compose if you want
18:39:48 <jforbes> mattdm: how many of those with testing still enabled are because packit is checking, but they are not updating at all
18:40:12 <jforbes> meaning, they aren't actually testing.  I have such a laptop here
18:40:47 <mattdm> jforbes: Definitely some! I was just trying to think of how I can quantify that
18:41:03 <bcotton> last call for meeting-relevant #content
18:41:11 <mattdm> lol sorry
18:41:23 <jforbes> right, but it could be some, it could be a majority, we have no way of knowing since the default change was in an update itself
18:42:14 <bcotton> #endmeeting