18:25:23 <f13> #startmeeting Fedora Release Engineering
18:26:09 <f13> ping: notting wwoods lmacken jwb spot warren rdieter poelcat dgilmore
18:26:18 * poelcat here
18:26:26 <f13> no bill, he's on a plane I think.
18:26:33 <f13> no rex, he doesn't seem to be on IRC
18:26:48 <f13> no luke, he doesn't appear to be on IRC either
18:27:03 <jwb> i'm here
18:27:04 <jwb> yay
18:27:20 <poelcat> f13: http://poelstra.fedorapeople.org/schedules/f-12/f12-releng-updated-version-2009-07-13.txt
18:27:34 <f13> thanks
18:28:40 * wwoods hereish
18:29:57 <f13> alright
18:30:06 <f13> #topic F12 Schedule
18:30:53 <f13> A few of us just had a Fedora Talk and Gobby session to fine tune the F12 schedule.  We've created a test compose prior to freezes, and tuned the blocker review days as well as the tail end of the schedule with go/nogo decisions and such
18:31:11 <f13> a /ton/ of work has gone into this schedule so this is last call for anything you find wrong
18:31:26 <f13> http://poelstra.fedorapeople.org/schedules/f-12/f12-releng-updated-version-2009-07-13.txt is the current iteration
18:31:29 <jwb> there's no entry for "enable F12 updates"
18:31:43 <f13> that's a good point.
18:31:50 <f13> jwb: when would you like to start pushing F12 updates?
18:32:00 <jwb> so i have a question first
18:32:14 <jwb> does this account for "no frozen rawhide"?
18:32:31 <f13> no
18:32:44 <f13> that proposal has yet to be accepted, so this is the same business as usual
18:33:33 <jwb> so rawhide is frozen from 9-22 until GA?
18:33:42 <Ab3L> question: do you know how to see which video driver is running?
18:33:54 <jwb> Ab3L, ask in #fedora
18:34:19 <f13> jwb: yeah, that should match up with previous releases, freeze just prior to "Preview" and remain frozen
18:34:51 <jwb> f13, have we decided if/how we're going to do inheritance for updates before GA?
18:35:08 <jwb> e.g. how are we going to avoid inadvertently breaking freeze for both the compose and buildroot
18:35:37 <f13> more koji tags
18:36:21 <jwb> so we can safely enable updates at, say, Beta Release
18:36:45 <f13> yes
18:36:54 <f13> for some value of "safely"
18:36:59 <jwb> hm.  that might be too early
18:37:23 <f13> I think we're also missing the mass branch date here
18:37:30 <jwb> yes
18:37:36 <jwb> i was just looking for that
18:37:43 <f13> we had dropped af ew things to aid in concentration on the important dates we were discussion
18:37:47 <f13> this isn't a full schedule list
18:37:47 <jwb> the two might want to go hand-in-hand
18:37:52 <poelcat> pre-branch is 2009-08-18
18:38:14 <jwb> poelcat, that's allowable.  we're talking about branching everything regardless
18:38:33 <f13> in the old style, we had pre-branching and then forced mass branching
18:38:36 <poelcat> jwb: okay, afraid we'd left soemthing out
18:38:54 <f13> I think we previously did forced mass branching just before PR
18:39:30 <jwb> which is what now?  beta?
18:39:38 <f13> beta
18:39:43 <f13> trying to find my announcement
18:40:56 <jwb> so my concern with doing updates for f12 starting with beta is running into more XFCE situations
18:41:00 <f13> We did it at 4/15 for F11
18:41:09 <f13> jwb: yes, that is a concern.
18:41:23 * nirik looks up. wonders what he did.
18:41:28 <jwb> nirik, nothing
18:41:39 <f13> nirik: we had packages in updates, that we also wanted in the final release
18:41:44 <nirik> ah, right.
18:41:58 <f13> so we mass branched just before the "final freeze"
18:42:12 <jwb> perhaps we disallow stable updates until 10-29 (or whenever we sync to mirrors)
18:42:19 <jwb> and just allow updates-testing
18:42:33 <f13> so we'd be mass branching around 9/21
18:42:57 <f13> jwb: not sure I like that, leaves no way for 0-day updates for things
18:43:04 <f13> or pre-staged 0-day updates
18:43:11 <jwb> sure it does... GA is 11-3
18:43:19 <jwb> so we have 5 days
18:43:20 <f13> oh, sorry, my mistake
18:43:59 <jwb> i was going with "RC is final"
18:46:42 <f13> but you're talking about 2 things though.  A) when bodhi will accept requests, and B) when you'll push portions of those requests
18:47:47 <jwb> yes.  though A is mostly pointless without b...
18:47:56 <f13> yeah
18:48:25 <f13> well, if we think about it in terms of no-frozen-rawhide, at what point would we flip the switch to push things to 0-day updates as opposed to freeze breaks?
18:48:32 <f13> that'd be around the RC phase right?
18:48:46 <jwb> think so
18:50:05 <f13> could just shoot for the same date now
18:50:38 <jwb> f13, yeah.  and in the interest of progress, we can just go with that now and revisit if needed later.  it's somewhat orthogonal to the fundamental schedule
18:50:45 <f13> k
18:51:29 <f13> #agreed Mass branch will happen just prior (day before) to final freeze
18:52:23 <f13> #agreed bodhi will accept/push updates as of 2009-10-21 which is the RC phase for the final release.
18:52:43 <f13> #action poelcat will add such dates to the schedule
18:53:02 <f13> Was there anything else on the schedule?
18:54:04 <f13> guess not
18:54:17 <f13> #topic Orphan round up
18:54:31 <f13> We're supposed to do this near Feature freeze, which is coming up fast
18:54:37 <f13> warren did it in the past for us
18:54:48 <jwb> is he not around?
18:55:21 <f13> I think I did it last time around
18:55:42 <f13> I can do it this time too
18:55:57 <f13> I'll need to send a list of the orphans a week prior to feature freeze
18:56:27 <f13> which would be 7/21, a week from tomorrow
18:56:51 <f13> #action jkeating will file a releng ticket to purge the orphans by feature freeze
18:58:11 <f13> looks like the script needs some work too, pkgdb changes likely
18:58:52 <f13> #topic FAD follow up
18:59:08 <f13> I really ahve to get no frozen rawhide pushed through, so I will be spending some time on that this week
18:59:30 <f13> I may have to call some more meetings to fine tune some of the missing data points
18:59:37 <jwb> ok
19:00:40 <f13> I have nothing else for this
19:01:37 <f13> #action f13 will spend more time on FAD follow up this week
19:01:47 <f13> #action f13 will file a releng ticket to purge the orphans by feature freeze
19:01:52 <f13> #topic open floor
19:02:06 <jwb> last update push took about a week
19:02:07 <f13> As you can see, we're using the new meet bot, this will help us with our meeting logging and reporting
19:02:12 <jwb> for one reason or another
19:02:12 <f13> jwb: A WEEK?!
19:02:16 <jwb> yeah
19:02:18 <poelcat> clarification:  bodhi will accept/push updates as of 2009-10-21 == enable F12 updates ?
19:02:20 <f13> oh, just various issues
19:02:23 <jwb> yeah
19:02:27 <f13> not a week to mash?
19:02:42 <f13> poelcat: yeah, that sounds right
19:02:45 <jwb> correct, not a week to mash.  longest repo mash time was 24 hours i think for f11-updates.  i have one running rightnow
19:03:14 <jwb> somewhat annoyingly, we can have a mash run for hours and then fail because koji pruned a signed RPM
19:03:21 <jwb> which causes big delays
19:03:38 <f13> yeah, I thought we were going to back down the pruning a bit
19:03:43 <f13> leaving more than just one copy
19:03:49 <f13> er more than just the latest copy
19:03:57 <jwb> guess that hasn't happened yet
19:03:59 <f13> #topic updates pushes
19:04:03 <skvidal> open floor question - where does critical-path controls for packages come into play?
19:04:06 <f13> ok, guess we need a ticket for that.
19:04:07 <skvidal> ah, sorry
19:04:23 <f13> #action f13 will file a ticket to remember to back down koji signed package pruning
19:04:35 <jwb> f13, i'd almost rather have bodhi call write-signed-rpm on all RPMs before mash, or have mash do it once on error case before puking
19:05:06 <f13> jwb: if you'd like to work on that code, that's fine, however it would require that mash/bodhi/whatever run as a user with admin rights
19:05:09 <f13> which is a tad...s cary
19:05:12 <f13> plus cert management
19:05:16 <jwb> bodhi already does?
19:05:20 <jwb> considering it moves tags...
19:05:22 <f13> yeah
19:05:48 <jwb> so really should be fairly simple.  except bodhi doesn't do anything with signing at the moment
19:05:54 <jwb> dunno if lmacken will like that or not
19:05:56 <dgilmore> jwb: it shouldnt be pruning the latest ones
19:06:08 <jwb> dgilmore, i don't believe it is
19:06:20 <jwb> it's pruning older ones
19:06:21 <f13> dgilmore: it's not the latest
19:06:24 <f13> which is what is so frustrating
19:06:32 <f13> because I think it's a bug in bodhi that is making this harsh
19:06:50 <dgilmore> jwb: then why is it being mashed?
19:06:50 <f13> bodhi is trying to re-push obsoleted updates into -testing I think
19:06:58 <dgilmore> ok
19:07:14 <f13> or they're not getting properly obsoleted
19:07:15 <jwb> f13, i hit it for more than just updates-testing
19:07:18 <f13> but I'd need more data
19:07:38 <f13> jwb: then something is going very wrong, and we shouldn't paper over it with kludges in bodhi/mash
19:07:46 <f13> do you have any examples we can look at?
19:07:50 <jwb> sec
19:08:01 <jwb> koji write-signed-rpm 4ebfc273 python-EnthoughtBase-3.0.2-1.fc10
19:08:12 <jwb> that's the last one i ran.  for f10-updates i believe
19:08:40 <f13> Thu Jul  2 08:07:25 2009: Tagged python-EnthoughtBase-3.0.2-1.fc10 with dist-f10-updates
19:08:40 <f13> Fri Jul 10 07:13:31 2009: Untagged python-EnthoughtBase-3.0.2-1.fc10 from dist-f10-updates
19:08:40 <f13> Fri Jul 10 07:13:42 2009: Tagged python-EnthoughtBase-3.0.2-1.fc10 with dist-f10-updates [still active]
19:09:05 <dgilmore> jwb: that sounds like sign_unsigned did not write the signed rpm
19:09:32 <f13> odd that it was tagged on the second, then untagged/retagged today
19:09:39 <f13> er not today, the 10th
19:10:02 <jwb> f13, and only 11 seconds apart...
19:10:29 <jwb> perhaps that untag tells koji to prune it
19:10:29 <dgilmore> jwb: that sounds like bodhi's doing
19:10:47 <f13> bodhi - 2009-07-03 19:50:31
19:10:47 <f13> This update has been pushed to stable
19:10:47 <jwb> could be part of a bundled update?
19:10:50 <dgilmore> afik there is nothing in koji that prunes sigs
19:10:56 <f13> bodhi seems to think it pushed to stable on the 3rd
19:11:00 <f13> dgilmore: yes there is
19:11:03 <jwb> dgilmore, it's not the sigs.  it's the signed RPMs
19:11:03 <dgilmore> its all managed outside of koji code
19:11:09 <f13> dgilmore: the koji-gc scripts do it
19:11:18 <f13> well sure, pedantic, nothing "in koji" does it
19:11:26 <dgilmore> f13: right but thats outsidenof koji
19:11:37 <f13> it's outside of the koji code base, but it's part of our koji deployment
19:12:02 <dgilmore> f13: what i meant was nothing in koji-hub  that goes and deletes signed rpms
19:12:17 <f13> Yeah, I don't think anybody claimed that there was
19:12:52 <dgilmore> f13: 14:10 < jwb> perhaps that untag tells koji to prune it
19:13:06 <f13> ah
19:13:07 <dgilmore> f13: i was just saying it doesnt
19:13:08 <f13> missed that
19:13:36 <f13> so we could use some insight from luke as to what happened with this update
19:13:57 <f13> jwb: any more that you find where we are missing sigs, we need to investigate them on a case by case basis.
19:14:08 <f13> could be bodhi bugs, could be sign_unsigned bugs
19:14:23 <jwb> well, considering it's happening to at least one repo per push, that's going to take a while
19:14:32 <jwb> fortunately, F9 updates are dead
19:16:38 <f13> right, but the point is we shoudln't be hitting these scenarios.  We need to fix the causes, not cure the symptom
19:17:06 <dgilmore> right
19:17:57 <dgilmore> i had one epel package that did not have a written out signed rpm that broke a push so far
19:18:36 <jwb> that's actually a normal error path
19:18:45 <jwb> well, assuming it wasn't signed at all
19:19:53 <dgilmore> it was signed
19:19:58 <dgilmore> just not written to disk
19:20:06 <jwb> ok, then sounds like what i'm seeing
19:20:20 <f13> probably falls under "signed_unsigned is a bloody piece of crap"
19:20:45 <dgilmore> its either there or koji failing to write it
19:20:51 <f13> anyway, is there anything else regarding updates we need to talk about at this meeting?
19:21:01 <dgilmore> i have nothing
19:21:12 <f13> was there any other action item we wanted jwb?
19:21:24 <wwoods> did I miss the critpath discussion (if any)?
19:21:32 <f13> not yet
19:21:39 <jwb> f13, nope.  just wanted to give a quick update
19:21:50 <f13> I think we're out of time this week, but can have a specific meeting for critical path if we want it.
19:22:18 * wwoods defers to skvidal here
19:22:19 <f13> but for the sake of...
19:22:30 <f13> #topic Critical Path
19:22:38 <f13> skvidal: you had mentioned something about it?
19:22:39 <skvidal> we can do it next week
19:22:43 <f13> ok
19:22:51 <f13> #agreed Critical Path will be talked about next week
19:22:57 <f13> #topic Open Floor
19:23:05 <f13> anything else?  Will close meeting in 2 minutes
19:23:12 <mitr> signing server?
19:23:21 <f13> mitr: on my to-do list this week
19:23:47 <mitr> f13: I'll be happy to help as much as I can.
19:29:07 <f13> ok, thanks all!
19:29:17 <f13> #endmeeting