18:59:18 <jlaska> #startmeeting F-13-Alpha engineering readiness meeting
18:59:18 <zodbot> Meeting started Thu Feb 25 18:59:18 2010 UTC.  The chair is jlaska. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:59:19 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:59:40 <jlaska> #topic gathering devel, rel-eng and qa reps
19:00:13 <adamw> hello!
19:00:19 <adamw> Oxf13: ping
19:00:37 <jlaska> who do we have on the devel side ... notting around?
19:01:00 * stickster not on devel side but here
19:01:01 * Oxf13 sets down the bottle of booze
19:01:09 <jlaska> Oxf13: pick that sucker back up!
19:01:14 <adamw> oh, i wouldn't let that get too far away if i were you
19:01:15 <jlaska> and pass it around too
19:01:35 <adamw> Alcohol: Proud Partner of the Fedora Development Process
19:02:02 <jlaska> notting isn't available for the meeting ... anyone else want to represent devel
19:02:15 <jlaska> notting lurks
19:02:35 <stickster> The notting knows....
19:02:43 <Oxf13> I could dual role
19:02:53 <Oxf13> but maybe we could drag clumens in here
19:02:53 * nirik also lurks.
19:03:08 <jlaska> nirik: howdy
19:04:02 <jlaska> welcome dlehman
19:04:26 <Oxf13> sometimes I regret turning off join/part
19:04:31 <nirik> perhaps someone from desktop would like to represent devel folks? as they are the critical path?
19:04:32 <jlaska> and Mr. Lumens
19:05:52 <ajax> i guess i count as desktop
19:06:16 <jlaska> alrighty, let's get started then
19:06:24 <jlaska> #topic intro
19:06:43 <jlaska> I'll keep this brief ... but why are we here?
19:07:04 <jlaska> Going by https://fedoraproject.org/wiki/Engineering_Readiness_Meetings ..
19:07:38 <jlaska> #info The purpose is to decide whether the alpha has met the release criteria
19:07:55 <jlaska> since we'll probably reference this link quite a bit ...
19:07:56 <jlaska> https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria
19:08:12 <ajax> mclasen would be a better representative but he's offline atm.  might be on in a bit.
19:08:33 <jlaska> alright ... so we've got folks from devel (notting, ajax, clumens, dlehman), rel-eng (Oxf13) and qa (adamw, jlaska)
19:08:45 <jlaska> now the fun part ...
19:08:54 <Oxf13> desktop is here too, ajax, dcbw
19:09:02 <jlaska> #topic Go or No Go
19:10:03 <jlaska> So where do we stand with the Alpha release criteria
19:10:14 <Oxf13> so lets go over what works, what doesn't work, and what's unknown
19:10:17 <jlaska> we've got OPEN blocker bugs -- https://bugzilla.redhat.com/showdependencytree.cgi?id=538273&hide_resolved=1
19:10:27 <jlaska> Oxf13: yeah, we'll hit that
19:10:29 <adamw> on the actual 'official' build we have, rc3, we have multiple blocker bugs
19:11:02 <adamw> traditional x86-64 install is broken, live install is broken, installing updates is broken
19:11:24 <adamw> we have somewhat-tested fixes for all three issues, but we do not have an anaconda build which incorporates the first two yet
19:11:39 <jlaska> #info traditional x86-64 install is broken (bug#568235)
19:11:41 <adamw> and we certainly don't have a full set of candidate images which incorporate the fixes, never mind have run the full test matrix on such images
19:11:44 <jlaska> #info live install is broken (bug#565840)
19:12:12 <jlaska> #info installing updates is broken (bug#568193 bug#567346)
19:12:54 <notting> that would imply that the 'go/no go' decision is obvious.
19:13:00 <stickster> *nod
19:13:03 <jlaska> For those watching at home ... the current RC3 test results are located at https://fedoraproject.org/wiki/Test_Results:Fedora_13_Alpha_RC3_Install
19:13:27 <adamw> and https://fedoraproject.org/wiki/Test_Results:Fedora_13_Alpha_RC3_Desktop
19:13:37 <Oxf13> notting: it'd be more obvious if we didn't have fixes in hand for all known issues
19:13:38 <jlaska> adamw: ah, thx
19:14:47 <jlaska> Oxf13: I don't want to slip either ... but we've already delayed this meeting once, and the purpose of this meeting was to be the "Where are we now" check point
19:14:55 <jlaska> so we don't keep waiting, and waiting
19:15:16 <Oxf13> right, I'm just adding input
19:15:26 <jlaska> Oxf13: right on
19:16:00 <jlaska> Oxf13: and adamw: have been closer to verifying those fixes
19:16:01 <adamw> i'm not comfortable with just getting an anaconda build and spinning up an rc4 and saying 'oh it's probably fine' and shipping it...and i can't see any other way we could get in under the wire
19:16:10 <adamw> so unfortunately i can't see much choice but slipping at this point :/
19:16:47 <jlaska> we touched /sbin/loader ... I think we should enact "/sbin/loader rule#3 - don't do touch it again"
19:17:17 <Oxf13> tell me more about the loader touch.  Was it to fix another bug, or was it something else?
19:17:33 <jlaska> it was to address a bug filed during earlier rawhide acceptance testing
19:17:50 <jlaska> bug#563009
19:18:54 <Oxf13> ok, so that wasn't a blocker bug
19:19:08 <jlaska> no, I don't believe so
19:19:45 <stickster> .bug 563009
19:19:47 <zodbot> stickster: Bug 563009 No logging type for "loader" messages - https://bugzilla.redhat.com/show_bug.cgi?id=563009
19:20:05 <jlaska> stickster: ah, so that's the new zodbot incantation
19:20:18 <Oxf13> ok, good to know.
19:20:30 <jlaska> alright, so I think we have a no go from QA (adamw and jlaska)
19:20:52 <jlaska> Oxf13: any thoughts/concerns/comment from rel-eng?
19:20:57 <Oxf13> well.
19:21:42 <Oxf13> I'm really reluctant to say nogo, because we're so close.  And because it feels like a big fat failure after all the adjustments and work we put in for earlier feature freeze, NFR, etc..
19:22:04 <Oxf13> I agree that we don't have much data to provide confidence in the fixes at this point
19:22:47 <Oxf13> and I think what will likely happen is we'll make RC4 today, have it pass testing and sit there for 10~ days
19:23:11 <adamw> i agree that's the most likely case, but the chance of something else happening is substantially non-zero...
19:23:43 <Oxf13> right.
19:23:55 <jlaska> I'd even offer not spinning RC4 until we have confirmation on these remaining issues
19:23:57 <Oxf13> while it sits there, updates-testing and branched will move on
19:24:09 <jlaska> Oxf13: so there's something we couldn't offer in previous releases, right?
19:24:16 <Oxf13> jlaska: we've got conformation on everything but x86_64 booting
19:24:18 * stickster feels Oxf13's pain, and moreover, that the release criteria and these meetings are a great step forward in removing doubt and making it so we can move on in an objective way without blame
19:24:23 <jlaska> updates-testing and rawhide will go on
19:25:03 * stickster wants to recognize the superhuman effort of rel-eng and QA to try and get us over the line in time
19:25:04 <Oxf13> It also sucks because technically we have enough time to spin the isos, validate the wiki and post the isos for download
19:25:26 <Oxf13> but we've moved our decision meeting back earlier than the amount of time it takes to complete those tasks
19:25:59 <jlaska> Oxf13: If I understand though, after this meeting, there are dependent tasks that make it difficult to reverse the decision should testing yield alpha failures?
19:26:30 <Oxf13> jlaska: I believe that was the reasoning at least for final release.  I don't know for certain if that's the case for Alpha/Beta releases
19:26:42 <stickster> It's a decision point. Hurts when you know things could change, but that doesn't remove the need for having one.
19:27:02 <stickster> "If we only had..." :-)
19:27:06 <adamw> i think the question is whether the decision point may be too early for alpha/beta releases
19:27:07 <Oxf13> given that this is the go/nogo point, releng would have to vote nogo, even though it hurts.  badly
19:27:17 * stickster pats Oxf13 on the back
19:27:22 * adamw hands oxf13 the bottle back
19:27:27 <Oxf13> and even though we could see all green by tomorrow
19:27:31 <jlaska> adamw: hey, you broke the circle
19:27:39 <stickster> Who's bogarting the bottle
19:27:51 <adamw> jlaska: his need is greater than ours
19:27:57 <jlaska> indeed
19:28:22 <jlaska> alright ... anyone from devel care to weigh in?
19:28:39 <dcbw> it's hard to test if installation doesn't work welll
19:28:42 <jlaska> hopefully one person, but we have multiple people representing ... clumens: dlehman: ajax: notting: dcbw
19:29:09 <clumens> i have nothing good to say.
19:29:18 <jlaska> Oxf13: hand the bottle to clumens next please
19:29:18 <notting> jlaska: from the standpoint of the current data/matrix, i don't see how we can say go
19:29:28 <ajax> from a desktop point, #568193 not being especially tested yet makes me side with no go.
19:29:37 <stickster> .bug 568193
19:29:38 <zodbot> stickster: Bug 568193 updating fails with a fatal error - https://bugzilla.redhat.com/show_bug.cgi?id=568193
19:29:51 <Oxf13> dcbw: that's why you're supposed to isntall from the last known good point and yum update
19:30:04 <adamw> ajax: oxf13 and I have positive results with latest packagekit, but we don't have an official image with it yet.
19:30:07 <Oxf13> so that we remove the mental block of "installer doesn't work, abandon all hope"
19:30:24 <stickster> jlaska: Should we #info the various votes to make the minutes sensible?
19:30:24 <dcbw> Oxf13: right, that can happen in parallel with installer testing of course
19:30:31 <jlaska> stickster: you beat me too it
19:30:49 <dcbw> Oxf13: but in the end, if we can't make installation work well, we can't ship it until we have a week at least of full-on testing *after* the installer works well
19:30:55 * stickster just playing back seat driver as  a frequent Reader O' Minutes
19:31:01 <dcbw> since tons of people install with the livecd
19:31:42 <jlaska> #info QA votes 'no go' - citing additional testing against remaining F13Alpha blockers and walking the Alpha test matrix
19:32:19 <ajax> i guess i don't see why, if we have such confidence in the pending fixes, we'd wait until 2 weeks from today
19:32:28 <jlaska> #info Rel-Eng votes 'no go' - reluctant considering how close we are and that the next RC will likely be the final
19:32:36 <Oxf13> dcbw: I think we were more looking for data on userland stuff after the install
19:32:43 <Oxf13> ajax: we're tied to Tuesday releases
19:32:52 <Oxf13> ajax: and thus if we're no go, we slip a week
19:33:03 <Oxf13> which means not this tusday as originally planned, but next tuesday
19:33:19 <ajax> k
19:33:44 <adamw> ajax: if you notice we haven't done much x86-64 installer matrix testing at all
19:33:52 <adamw> ajax: since one of the bugs that's only jsut been resolved broke x86-64 install entirely
19:34:03 <adamw> so we have no verification that all those tests pass on x86-64, even with the *initial* fiox
19:34:07 <stickster> clumens: dlehman: You guys are machines. Thank you for putting so much sweat into trying to get things in working order.
19:34:13 <Oxf13> I'd ask what x86_64 testing happened prior to RC3
19:34:16 <adamw> yeah, big thanks to anaconda guys
19:34:26 <Oxf13> since it was RC2->RC3 that broke x86_64
19:34:26 <jlaska> #info Devel votes 'no go' - notting cited the current data and matrix results, dcbw cited importance of installing
19:34:47 <jlaska> Oxf13: the only way I wouldn't retest x86_64 against RC4 ... is if we shipped RC2
19:34:49 <dlehman> agreed w/ no-go
19:35:04 <ajax> i have a 64-bit machine running f13 (updated from f12); as far as post-install goes, it seems pretty solid.
19:35:18 <Oxf13> jlaska: that's not what I asked
19:35:21 <jlaska> Oxf13: but I agree that having previous test data for that architecture puts us in better shape than being in the dark
19:36:03 <dlehman> slipping sucks, but if we slip we can get the x86_64 matrix completed, no?
19:36:18 <jlaska> dlehman: correct
19:36:42 <jlaska> Oxf13: RC2 install testing - https://fedoraproject.org/wiki/Test_Results:Fedora_13_Alpha_RC2_Install#Test_Matrix
19:36:47 <dlehman> it's one week. may not even need to slip beta.
19:36:47 <Oxf13> dlehman: we should be able to get that matrix complete tonight/tomorrow
19:36:59 <jlaska> Oxf13: tomorrow
19:37:11 <jlaska> at best
19:37:26 <jlaska> Oxf13: RC1 install testing - https://fedoraproject.org/wiki/Test_Results:Fedora_13_Alpha_RC1_Install#Test_Matrix
19:37:48 <stickster> dlehman: I would think so
19:37:56 <jlaska> Oxf13: the test composes and acceptance runs are there too .. https://fedoraproject.org/wiki/Category:Fedora_13_Test_Results
19:38:11 <Oxf13> ok, looking at the RC2 red x86_64 results, those are issues which are confirmed fixed on RC3 i386
19:38:17 <Oxf13> and they weren't x86-64 specific
19:38:39 <jlaska> Oxf13: what are you suggesting?
19:38:53 <stickster> dlehman: The procedure from here is that the release slips by a week, IIRC: https://fedoraproject.org/wiki/Engineering_Readiness_Meetings#Contingency_Plan
19:38:58 <Oxf13> I'm suggesting that it's extremely unlikely we'll find an x86_64 specific issue, once we get past the booting
19:39:11 <dlehman> ah, ok
19:39:16 <jlaska> Oxf13: oh yeah, I agree with that
19:39:18 <Oxf13> stickster: that contingency plan was made before we had no frozen rawhide
19:39:26 <Oxf13> stickster: in which rawhide would stay frozen if we slipped
19:39:32 <Oxf13> stickster: which is not the case any more
19:40:22 <stickster> Oxf13: Was that discussed at the NFR conference? I don't remember
19:40:33 <Oxf13> this particular wrinkle wasn't
19:40:46 <jlaska> well ... we're slipping ... so should we move on to contigency plan?
19:40:47 <Oxf13> mostly because I forgot that this would be a benefit of NFR
19:40:57 <Oxf13> jlaska: sure
19:41:08 <jlaska> Oxf13: sounds like you might have other options/ideas there ...
19:41:15 <jlaska> #topic contingency plan
19:41:35 <jlaska> #info https://fedoraproject.org/wiki/Engineering_Readiness_Meetings#Contingency_Plan indicates a 1 week slip as a contingency plan
19:41:47 <jlaska> Oxf13: take it away
19:41:50 <Oxf13> traditionally we've struggled with the decision to slip all further milestones or not if we silp a release
19:42:28 <Oxf13> often we've sided on slipping everything, because slipping a alpha or beta increased the amount of time trees were frozen and inhibited further bugfixing and testing
19:42:33 <Oxf13> limiting developer time
19:42:44 <Oxf13> however, with NFR, that's no longer the case
19:42:50 <Oxf13> rawhide itself is still wide open
19:43:01 <Oxf13> F-13 builds can be published to updates-testing
19:43:09 <Oxf13> and tested there
19:43:11 <jlaska> but ... the milestones do offer a line in the sand that draws people out to test
19:43:23 <jlaska> where they might not come download isos before then
19:43:49 <jlaska> but NFR does seem to open the door for more test options than we had before
19:43:49 <Oxf13> so by slipping alpha, we're not actually taking away any development time from the developers
19:44:07 <adamw> aiui oxf13 is just pointing out that now there's no particular need to slip beta and final
19:44:11 <adamw> just because we're slipping alpha
19:44:33 <Oxf13> yes, I'm proposing that we do not adjust beta/final dates.
19:44:41 <stickster> Because content hasn't stopped dead while we wait to get things composed and then unfrozen?
19:44:48 <Oxf13> stickster: that is correct
19:45:02 <stickster> Thanks, "I'm just a simple caveman" to quote a friend :-)
19:45:14 <Oxf13> there are not only one, but two pressure release valves that have been left open while we get Alpha out the door
19:45:43 <stickster> Right, crazy crap -> Rawhide, controlled, careful... uh, stuff -> F-13-updates-testing
19:46:09 <jlaska> even more controlled/tested content F-13-base ?
19:46:27 <Oxf13> yes, we can and will still approve things in bodhi into the branched repo
19:46:33 <Oxf13> things not of crit-path nature will just make it in
19:46:43 <Oxf13> things of crit-path will not make it in unless releng/qa says so
19:47:03 <stickster> Which, supposedly, people will be able to grab as part of a routine first-session update run in the Alpha
19:47:12 <Oxf13> stickster: yes
19:47:18 * stickster wondering how that affects Alpha test matrix?
19:47:21 <Oxf13> stickster: or even during alpha install
19:47:29 <stickster> I guess no different than previous teset releases
19:47:31 <stickster> *test
19:47:34 <Oxf13> right
19:47:41 <Oxf13> there has always been a flood of updates for milestone installs
19:47:47 <Oxf13> only this time the flood is more controlled and tested
19:48:06 <stickster> Kinder, gentler firehose ;-)
19:48:45 <stickster> Go on, my caveman questions are answered
19:49:07 <jlaska> I feel like there is still something to the milestone announcement that draws people out to download/test where they may not have with nightly images.  I'd be worried if we had multiple slips and how that might impact the feedback we want.
19:49:25 <Oxf13> I'm not sure I follow
19:49:26 <jlaska> not that I'm arguing we slip the whole thing, since Oxf13 you point out how NFR helps us out now
19:49:29 <Oxf13> we're still going to have milestones
19:49:33 <jlaska> right right
19:49:36 <Oxf13> we're still going to have plenty of change between the milestones
19:49:50 <Oxf13> and one milestone will be a week later than originally intended
19:50:11 <jlaska> right, we reduce the test time leading up to the next milestone
19:50:28 <Oxf13> for those subsets of users who only test the milestones
19:50:30 <jlaska> only that I think people wait for the bits to go out ... before they dive in
19:50:56 <jlaska> Oxf13: right, we have a lot of folks who more closely follow things are provide feedback near realtime
19:51:21 <jlaska> so just my worry ... if we also slipped the Beta .. or had to slip the Alpha again ... I'd want to revisit this
19:51:25 <jlaska> adamw: what's your take?
19:51:39 <adamw> a week doesn't hugely worry me
19:51:49 * jlaska info's ...
19:52:06 <stickster> jlaska: So what you're saying, if I understand you...
19:52:23 <Oxf13> is it #proposal or #idea for doing a vote?
19:52:25 <stickster> is that the contingency plan should always have the option of slipping everything a week, but it's not an automatic thing
19:52:32 <jlaska> #info Oxf13 notes that with No Frozen Rawhide (NFR), we no longer have the pressure valves from previous releases.  Developing (rawhide) and testing (f-13-updates-testing) can continue
19:52:40 <jlaska> Oxf13: ah yes!
19:52:56 <stickster> Oxf13: I think it's #idea?
19:53:26 <Oxf13> #idea Do not slip Beta/Final milestone dates, due to NFR not shorting developers on development time.
19:53:31 <Oxf13> I'm +1 obviously
19:53:38 <jlaska> thanks, you type faster ;)
19:54:13 <adamw> +1
19:54:30 <jlaska> I agree w/ adamw, I'm up for seeing how that goes.  But hope we can revisit should additional slips happen
19:55:11 <jlaska> any other takers?
19:55:49 <Oxf13> Yeah I'd like to hear from the developers on this one
19:56:07 <Oxf13> notting clumens dlehman ajax dcbw  what say you?
19:56:46 <notting> Oxf13: sounds reasonable at a first glance
19:56:56 <stickster> I'd like to note that more than a week here puts pressure on docs, translation, marketing, and other groups too. It's not just about bits getting out the door
19:57:19 <stickster> So it's vital that we not consider this "It's OK that Alpha and Beta both slip up to one week"
19:57:56 <stickster> I would almost want to err on the side of, "We can absorb the first one, all others come at a cost."
19:58:03 <ajax> seems sensible, at least for this round.
19:58:06 <jlaska> +1
19:58:19 <Oxf13> stickster: yes, that's what we've usually done, with grumbling from various crowds about shorting the developers time
19:58:30 <jlaska> #agreed decided to absorb the slip and not move the F-13 release out by one week (future slips may incur a cost)
19:58:57 <jlaska> alright folks ... anything else to decide here?
19:59:14 <Oxf13> we have to assign some action items
19:59:50 <jlaska> okay, help me out ... what are the tasks?
19:59:50 <Oxf13> #action Oxf13 will announce the slip to (devel-)announce
19:59:59 <jlaska> ah, gotcha
20:00:02 <Oxf13> somebody has to update the schedule pages
20:00:07 <stickster> Oxf13: I'll take that
20:00:09 <Oxf13> which poelcat usually does
20:00:14 <mchua> (and logistics, plz, to let folks know there isn't a readiness meeting today)
20:00:17 <jlaska> I can talk to poelcat (unless stickster wants it)
20:00:18 <Oxf13> because he owns the taskjuggler generation
20:00:34 <stickster> I can deal with the wiki, but not so sure about TJ
20:00:42 <Oxf13> #action stickster will update schedule pages and poke poelcat about TJ
20:00:43 <jlaska> #action jlaska or stickster - talk to poelcat for task juggler schedule update
20:00:55 <stickster> heh, an #undo will remove one of those
20:01:02 <stickster> (the latest)
20:01:04 <Oxf13> we need somebody to re-schedule the readiness meeting
20:01:06 <jlaska> #undo
20:01:07 <jlaska> neat
20:01:07 <zodbot> Removing item from minutes: <MeetBot.items.Action object at 0x28382a10>
20:01:13 <stickster> Oxf13: I can handle that too, I told poelcat I would
20:01:25 <Oxf13> #action stickster will re-schedule release readiness meeting
20:01:25 <dlehman> +1 on not slipping beta
20:01:57 <jlaska> what else remains?
20:02:01 <Oxf13> We should also communicate to RHEL engineering, in case they were using our Alpha as a milestone of sorts
20:02:16 <Oxf13> stickster: want to take that one?  (off the record as it were)?
20:02:23 <stickster> Sure
20:02:36 <stickster> Oxf13: I'll end up relaying your announcement, so it depends on your action
20:02:40 <Oxf13> ok.  I can't think of anything else.  Just getting RC4 created and ran through
20:02:56 <jlaska> Oxf13: we'll obviously rinse and repeat on that
20:03:50 <Oxf13> jlaska: I sincerely hope there isn't a rinse/repeat for RC5
20:04:04 <jlaska> #action QA - continue testing RC4 when available and verifying remaining F13Alpha bugs
20:04:12 <jlaska> Oxf13: here here
20:04:32 <jlaska> okay, shall we call it a meeting?
20:04:53 <Oxf13> please
20:05:03 <jlaska> thanks everyone
20:05:04 <Oxf13> my liver is late for an appointment
20:05:07 <jlaska> heh
20:05:09 <jlaska> #endmeeting