18:02:25 <Oxf13> #startmeeting Fedora Release Engineering Meeting
18:02:25 <zodbot> Meeting started Mon Oct  5 18:02:25 2009 UTC.  The chair is Oxf13. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:02:25 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:02:31 <Oxf13> #topic roll call
18:02:56 <Oxf13> ping: notting jwb rdieter_work lmacken spot wwoods warren dgilmore poelcat
18:03:01 * poelcat here
18:03:04 * stickster lurks like a spider cricket
18:03:05 <adamw> i'm here for the go/no-go portion, representing QA
18:03:12 * lmacken here
18:03:14 <adamw> jlaska's busy so i'm covering for us
18:03:15 * notting is here
18:03:41 * wwoods lurks but probably won't have much in the way of useful commentary
18:03:42 <warren> pong
18:05:45 <Oxf13> ok.
18:06:13 <Oxf13> Basically we decide now if we think we can release Fedora 12 Beta at the regularly scheduled mark.
18:06:29 <Oxf13> Given that we haven't hit RC phase yet, and there are still outstanding bugs, my vote is no go.
18:06:35 <Oxf13> and thus slip a week
18:07:00 <adamw> outstanding bugs, for reference: https://bugzilla.redhat.com/showdependencytree.cgi?id=F12Beta&hide_resolved=1
18:07:08 <poelcat> +1 w/ slip to rest of the schedule, including GA date
18:07:14 <notting> silly people finding bugs. we should stop that.
18:07:24 * poelcat not trying to be negative ned, but we don't have a lot of wiggle room
18:07:50 <adamw> qa's position is that if there is a slip, the rest of the schedule should slip accordingly (as poelcat says)
18:07:52 <warren> Oxf13: I request, please tell the releng list clearly when it is time to stop tagging.
18:08:00 <warren> Oxf13: are we auto-tagging new packages right now?
18:08:00 <Oxf13> warren: can do
18:08:10 <Oxf13> warren: if they aren't in the critical path and somebody asks for it, sure.
18:08:15 <warren> ok
18:08:30 <warren> is there a permanent URL for a critical path list?
18:08:47 <Oxf13> not as of yet
18:09:00 <warren> where is the non-permanent URL?
18:09:11 <Oxf13> http://test1185.test.redhat.com/results/633-autotest/brutus.test.redhat.com/rats_sanity/results/critpath.log
18:09:20 <Oxf13> We cut cut that and paste it into a fpaste or some such
18:09:52 <wwoods> the critical path list changes every time the packages involved change
18:09:58 <wwoods> so there won't be a permanent static list ever
18:10:16 <Oxf13> there can be a perminant URL to the day's list
18:10:17 <poelcat> that sounds logical, but less than ideal
18:10:19 <wwoods> just want that to be clear
18:10:29 <Oxf13> permanent even.
18:10:36 <wwoods> poelcat: no, it's ideal, the list is derived by expanding the dependencies of a short list of required packages
18:10:41 <wwoods> obviously changing those packages will change the list
18:10:47 <wwoods> therefore the list will never be static
18:11:06 <Oxf13> it's less than ideal because people will have to check each and every day if their packages are suddenly critpath
18:11:35 <wwoods> surely we can work out some kind of notification for that, clever people that we are
18:11:40 <warren> copy it to a wiki page then?
18:11:42 <warren> brb restroom
18:12:06 <jwb> i'm here sort of
18:12:23 <Oxf13> easiest I would think would be to run whatever snippit of code that generates this list and dump it to some html space we control, every day
18:12:42 <wwoods> we're already running it every day
18:12:57 <Oxf13> wwoods: but we have no public static url to the results.
18:13:02 * stickster thinks there's a script ref'd in the "Critical Path Proposal" page
18:13:30 <wwoods> yes. there's an updated version in the autoqa test: https://fedorahosted.org/autoqa/browser/tests/rats_sanity/sanity.py
18:14:42 <wwoods> I guess I could dump the list into fpaste or the wiki or something daily
18:15:42 <wwoods> doesn't python-fedora have a paste library or something?
18:15:52 <Oxf13> I think so
18:16:00 <Oxf13> or there was a fpaste script written
18:16:07 <stickster> There's a fpaste pkg
18:16:23 <notting> we may be going slightly afield from the question at hand :)
18:16:32 <poelcat> can we conclude "go no-go" discussion and go forward plan?
18:16:36 <Oxf13> I think we already answered the question at hand.
18:16:46 <Oxf13> does anybody disagree with the no-go call?
18:17:20 <Oxf13> I think the only real pressing one is missing firstboot
18:17:47 <Oxf13> and maybe the liveinst one.
18:18:50 <Oxf13> all the rest have been built/tagged today
18:18:59 <Oxf13> but we don't have a good testing of a rawhide built with that content
18:19:13 <Oxf13> so it very well may be we'll enter RC phase tomorrow or the day after and be really nice and early
18:20:16 <Oxf13> #agreed No-go for Fedora 12 Beta.  Slip 1 week (and all future dates by 1 week)
18:20:57 * poelcat volunteers update schedules
18:21:06 <poelcat> do you want me to announce slip in email too?
18:21:16 <Oxf13> poelcat: if you'd like sure!
18:21:27 <Oxf13> #action poelcat to update schedules and send out slip announcement.
18:22:04 * stickster will let folks know in various collateral areas, e.g. Red Hat PR
18:22:32 <Oxf13> alright, I think we can move on from there.
18:22:38 * abadger1999 Just let mmcgrath know
18:24:53 <mmcgrath> there's no way to not slip?
18:26:34 <abadger1999> Oxf13: ^
18:26:40 <Oxf13> mmcgrath: we'd have to align the stars perfectly and fix every bug today
18:26:46 <Oxf13> and have a perfect rawhide tomorrow to compose from
18:26:47 <abadger1999> mmcgrath: Final is the key problem right, not beta?
18:26:57 <mmcgrath> we don't have plans to revert whatever bugs are causing the slip?
18:26:58 <mmcgrath> abadger1999: yeah.
18:26:59 <abadger1999> So if we could slip beta but not slip final?
18:27:08 <Oxf13> mmcgrath: it's not as simple as just reverting a change.
18:27:16 <Oxf13> some of these bugs we don't even know what change introduced the bug
18:27:20 <poelcat> abadger1999: not if you want a predictable GA date
18:28:15 <Oxf13> mmcgrath: the scheduling issue with Final is the migration of our infrastructure to PHX2 right?
18:28:24 <mmcgrath> yes.
18:28:40 <mmcgrath> because as it stands,  I'm pretty sure we'd start shutting down systems on the 18th at the earliest (which is already later then we wanted)
18:28:53 <mmcgrath> and that's also the day _after_ the reelase, which is a bad time to be doing that stuff anyway.
18:29:33 <Oxf13> and you need a pretty big window to do the move right?
18:29:39 <mmcgrath> 48 hours.
18:29:48 <mmcgrath> I'd prefer not to do it on thanksgiving
18:30:10 <mmcgrath> but if I have to I will, I'll meet up with Eric and find out what implications this will have.
18:32:49 <Oxf13> so our choices seem to be
18:32:55 <Oxf13> A) slip everything a week and screw mmcgrath
18:33:09 <Oxf13> B) don't slip now, run the risk of slipping later and hitting A
18:33:29 <Oxf13> C) slip for 2 weeks and have the migration happen before we release
18:33:54 <notting> D) slip beta 1 week, final 2 weeks?
18:33:58 <notting> or is that what you meant by C)?
18:34:05 <mmcgrath> C) sounds like a recipe for disaster
18:34:36 <mmcgrath> nothing personal to the people involved but I do not have confidence that things will be working properly after the migration.
18:34:39 <poelcat> mmcgrath: what are consequences of moving late?
18:34:58 <Oxf13> ok, so B seems like the least risk to getting our release done
18:35:02 <mmcgrath> poelcat: Not sure, AFAIK they're flipping the switch off on the 30th.  we were already one of the last groups to move.
18:35:10 <notting> mmcgrath: on the premise that any large infrastructure change will cause *something* to break?
18:35:11 <Oxf13> we'll just have to be militant with the changes that go in post-beta
18:35:28 <notting> Oxf13: for B), you mean slip beta but *not* final?
18:35:49 <mmcgrath> notting: yeah, there's a lot of networking configs to be changed, lots of new equipment to deal with, reconfiguration of our mirroring system, etc, etc.
18:35:57 <Oxf13> notting: right
18:36:04 <Oxf13> option B would be slip beta, but retain final date
18:36:23 <notting> Oxf13: that means that beta is live for only 8 days when we start composing RCs
18:36:34 * stickster doesn't think that's a very good option.
18:36:54 <Oxf13> having the switch thrown on our hardware before we release also isn't a very good option
18:36:55 * poelcat isn't realy confident that "milantacy" as a strategy has saved us before
18:37:18 <Oxf13> I'm not so sure we can tell RHT to "stop the presses, we're not done yet"
18:37:19 <poelcat> mmcgrath: can you get confirmation that 30th is a drop dead date and can't be changed?
18:37:43 <poelcat> mmcgrath: or what the consequences (finanical or otherwise) are?
18:37:44 <mmcgrath> poelcat: I've sent that email and am asking.  I've been told that this whole time but as we get closer it's possible they haven't gotten things done either so they may have also slipped.
18:37:45 <jlaska> well, I can say we've definitely done that in the past (B) ... and it seems to always result in a slip come final anyway
18:38:20 <Oxf13> it seems that no matter what we do we slip
18:38:36 <abadger1999> Questions for C == Can we move the important services earlier; can we slip release longer than 2 weeks; if things that we move early don't work, does that impact the release date?
18:38:36 <Oxf13> we pushed out the beta dates when we slipped alpha, and that didn't really help us much
18:39:05 <mmcgrath> If there really is no possible way to avoid the slip, there really isn't a way to do it.  We'll figure something out.
18:39:21 <mmcgrath> But note if we slip again, you're basically releasing on thanksgiving.
18:39:23 <poelcat> Oxf13: we totally needed that time or Alpha wouldn't have gone out on time
18:39:58 <Oxf13> poelcat: we needed that time for alpha, but it certainly didn't prevent us from having to slip Beta again
18:40:15 <Oxf13> I'm still not convinced that moving later dates actually buys us anything other than more time for developers to fuck it up
18:41:12 <poelcat> until we fix the underlying problem i think its the best option we have
18:41:35 <Oxf13> underlying problem being that our 6 month schedule is untenable?
18:41:43 <poelcat> i agree it isn't a guarantee
18:41:48 <poelcat> churn
18:42:12 <Oxf13> the only way to prevent churn is to not allow churn
18:42:22 <notting> and then you get no butter. oh wait.
18:42:24 <Oxf13> which means harsher freezes
18:42:32 <Oxf13> but you need to allow that churn to happen somewhere else
18:42:45 <jlaska> which leads us to a key proposal
18:42:52 <poelcat> :)
18:43:22 * notting waits for the glorious autoqa/nofrozenrawhide/israwhidebroken future
18:43:23 <Oxf13> but we're still stuck with the current situation
18:44:18 <poelcat> just brainstorming.... E) skip beta and go straight to RC
18:45:09 <Oxf13> well, essentially Beta is the public "release candidate"
18:45:21 <Oxf13> the public tells us what's wrong with it, we fix it, and make the actual release
18:45:23 <notting> poelcat: we really need a public milestone
18:45:36 <Oxf13> Beta doesn't have to soak for long
18:47:17 <Oxf13> I'm leaning more and more to slip beta, retain final
18:47:33 <poelcat> mmcgrath: when will you have more data?
18:47:45 <mmcgrath> poelcat: not by the end of the meeting.
18:48:36 <Oxf13> so lets at least slip beta today, and determine what to do with the rest of the schedule later this week
18:49:14 <jlaska> my only concern about not slipping final is that the beta testing period is already only 13 days long
18:49:36 <Oxf13> #idea Slip Beta by 1 week, postpone the rest of the schedule discussion until more information is received regarding data center moves
18:49:37 <poelcat> jlaska: right... it would be 7 days less
18:49:37 <jlaska> I know it's the previous "preview" milestone, but I think that just sets us up for further pain on the final
18:49:39 <Oxf13> +1
18:49:58 <Oxf13> jlaska: so I have a little issue with that.
18:50:15 <Oxf13> jlaska: Aside from media based installs, the "beta" testing period started when we froze, not when we release
18:50:52 <Oxf13> so when we "release" doesn't matter as much, because we've been doing testing all along
18:51:05 <jlaska> on the beta bits yeah
18:51:21 <jlaska> right?  I mean not a lot of focus on the post-beta collection of changes?
18:51:23 * poelcat notes we don't have beta bits
18:51:29 <poelcat> or we wouldn't be here
18:51:38 <jlaska> poelcat: indeed
18:51:58 <poelcat> and if we don't have bits how can the testing period have begun?
18:52:01 <Oxf13> jlaska: if you're worried about post-beta changes, that's a simple fix.  Don't allow any post-beta changes.
18:52:17 <Oxf13> poelcat: wow.
18:52:22 <Oxf13> Ok, lets spell it out here.
18:52:33 <notting> Oxf13: what they may be trying to say (or not)... is that for a chunk of our userbase, they're waiting until beta release to start testing
18:52:36 <Oxf13> We froze our tree, so that we could produce a beta.  We have to test those bits to see if they're good enough to be beta
18:52:41 <Oxf13> so we've already started testing.
18:52:47 <Oxf13> some bits aren't quite good enough, and will have to be updated
18:52:57 <Oxf13> and then those few remaining bits will be "beta"
18:53:07 <Oxf13> so many things are /already/ beta.
18:53:39 <Oxf13> notting: and what I'm saying is that for a chunk of our userbase, the testing they will do doesn't really matter to us getting a release out
18:53:44 <jlaska> Oxf13: I agree in the spirit of limiting post-beta changes to a set of strict "criteria" (to be established hopefully in the foreseeable future)
18:54:05 <warren> Have we considered other effects of changing to dracut that we haven't been testing since Alpha?
18:54:08 <Oxf13> jlaska: we can define the criteria as we go.  if it's crit-path, it needs to be fixing a blocker
18:54:15 <notting> Oxf13: they're testing the still-refined storage code  on a pile of new hardware :/
18:54:21 <warren> Does the anaconda install media use a dracut initrd?
18:54:34 <Oxf13> warren: no
18:54:38 <notting> the anaconda stuff is special. always has been.
18:54:46 <warren> ok
18:56:07 <Oxf13> I think we've pretty much established that if the package isn't in the critical path, we don't really care about it, as it's not going crash our tree
18:56:07 <jlaska> to be clear, the only thing in our way of releasing is 3 unMODIFIED F12 blocker bugs, right?
18:56:24 <Oxf13> jlaska: F12Beta bugs right?
18:56:28 <jlaska> yeah
18:56:35 <Oxf13> 2 really
18:56:55 <Oxf13> liveinst partitioning, and missing firstboot (which is actually a vt switch crash on some hardware)
18:57:01 <jlaska> liveinst, firstboot and the ppc64.img
18:57:09 <jlaska> oh they're the same core issue?
18:57:32 <Oxf13> jlaska: I think I have ppc64.img solved, I just need you to test an image
18:57:42 <jlaska> oh cool
18:58:03 <jlaska> well, for the purposes of this meeting, we probably don't need to detail the list (sorry)
18:59:33 <jlaska> so where are we?  reassess schedule impact after hearing back from mmcgrath?
18:59:33 <Oxf13> yeah, we could very well go RC for Beta tomorrow
18:59:42 <Oxf13> jlaska: that was my proposal some 50 lines ago
19:00:49 <poelcat> Oxf13: at the top the meeting you proposed "no-go" ?
19:01:16 <Oxf13> 11:49 <Oxf13> #idea Slip Beta by 1 week, postpone the rest of the schedule discussion until more information is received regarding data  center moves
19:01:35 <Oxf13> that was 10 minutes ago
19:03:46 <Oxf13> so far, I'm the only one who has expressed favor of this proposal
19:04:22 <jwb> i like it
19:05:00 * poelcat willing to be in favor of proposal if we set a specific date and time this week to rediscuss GA date
19:05:01 * notting strongly feels that we'll end up slipping final anyway
19:05:42 <jlaska> I don't see us reaching a decision in this meeting on the 2nd date ... so no objections here ... let's just set a meeting time as poelcat suggested?
19:06:30 * poelcat proposes Thursday @ 18:00 UTC
19:06:41 <poelcat> IOW same start time as today
19:06:49 <Oxf13> that meeting time depends on when mmcgrath gets data about the DC move hard dates
19:06:49 <Oxf13> We have the datacenter move and thanksgiving to deal with
19:06:56 <Oxf13> not to mention threatening Fedora 13 schedules
19:07:29 <poelcat> then we'd better figure it out soon :)
19:07:32 <jlaska> poelcat: that date works for me
19:07:47 <notting> mmcgrath: you mentioned 'not by the end of this meeting'... can you have more concrete datacenter schedule info by thursday?
19:09:16 <mmcgrath> notting: I'm not sure, it's not my totally in my hands.
19:09:34 <mmcgrath> but I'll schedule another meeting and find out what I can.
19:09:52 <notting> even if it's just confirmation of the "yes, the 30th is a hard date" sort of thing
19:10:08 <mmcgrath> <nod>
19:11:32 <Oxf13> poelcat: provided I'm not in Jury Duty, Thursday seems fine to me
19:11:55 <poelcat> if you are, i'll speak for you ;-) :)
19:13:31 <Oxf13> anybody not in favor of meeting on Thursday 1800 UTC?
19:14:12 <notting> i doubt the board meeting will run into it, so sure.
19:16:39 <Oxf13> alright
19:16:58 <Oxf13> #agreed Meeting for deciding what to do with the rest of the F12 schedule set for this Thursday at 1800 UTC
19:17:15 <Oxf13> anything else before we stick a fork in this meeting?
19:19:14 <Oxf13> alright done.
19:19:16 <Oxf13> #endmeeting