18:03:29 <f13> #startmeeting Fedora Release Engineering Meeting
18:03:33 <f13> #topic roll call
18:03:56 <f13> ping: notting, jeremy, jwb, wwoods, lmacken, spot, warren, rdieter, dgilmore, poelcat
18:04:00 <warren> pong
18:04:02 * notting is here
18:06:32 * dgilmore is here
18:06:47 <rdieter> hi
18:06:49 * wwoods 
18:07:51 <f13> well that's a good set.  Lets get going
18:07:57 <f13> #topic old business
18:08:31 <f13> #topic orphans (old business)
18:08:49 <f13> I sent out an updated list of orphans and write a script to depsolve those orphans if they were removed.
18:09:00 <f13> I need to send it out again with current status.
18:09:22 <f13> #topic FAD (old business)
18:09:39 <f13> I spent some time last week polishing up the no frozen rawhide proposal for FESCo submission, and it passed!
18:09:50 <f13> We'll talk more about what that means to us later in the meeting.
18:10:08 <f13> #topic gpg purging in koji (old business)
18:10:27 * poelcat logging
18:10:30 <f13> I did not get this ticket filed, however I did  look at the script and noticed that it doesn't even have our current keys listed, so this script needs even more work.
18:10:48 <f13> so that's still an open action item.
18:11:09 <f13> #action f13 will file a ticket about koji-gc signature script needing work to be current.
18:11:28 <f13> that was the left over stuff from last meeting's old business.
18:11:32 <f13> lets review last week's topics.
18:11:41 <f13> #topic Critical Path (old business)
18:11:52 <f13> There were 2 action items here.
18:12:03 <f13> one was lmacken reporting on bodhi changes for critical path.
18:12:09 <f13> I haven't seen lmacken surface yet this meeting.
18:12:14 <lmacken> I'm kind of here
18:12:21 <f13> ah
18:12:25 <f13> do you have anything to report?
18:12:27 <lmacken> but I haven't looked into the critical path + bodhi stuff yet :(
18:12:34 <f13> ok.
18:12:41 <lmacken> it "shouldn't be too hard"
18:12:45 <f13> #action lmacken to report next week on bodhi changes for Critical Path Packages
18:12:50 <lmacken> will do :)
18:13:17 <f13> the next action item was for me to get offtrack packaged so that 'make tag-request' would work in case we don't get no frozen rawhide done in time and would be using bodhi for update submissions
18:13:26 <f13> I did not get this done, so it continues to be an action item.
18:13:50 <f13> #action f13 will get offtrack packaged and make tag-request functional.
18:13:57 <wwoods> is there a link for status / proposed process changes for bodhi/critpath stuff?
18:14:12 <f13> that's a good question.
18:14:24 <f13> one would hope that it was linked from the proposal page
18:15:26 <f13> lmacken: is there a ticket filed already?
18:15:37 <lmacken> I don't believe so
18:17:22 <f13> as the people running with the proposal, wwoods skvidal can one of you file a ticket for what we want to see changed?
18:17:29 <wwoods> Here's the thing: there's two parts to the proposal, right? 1) here's the package list, and 2) here's some new rules regarding updates to packages on the list
18:17:33 * skvidal reads back
18:17:36 <skvidal> yes
18:17:45 <skvidal> and what changes we want to bodhi to make 2 happen
18:17:48 <wwoods> from the QA POV, I only care about #1
18:17:56 <wwoods> because I'm doing automated testing on packages in that list
18:18:07 <skvidal> okay
18:18:14 <f13> wwoods: you care about 2, because you'll have to interact with bodhi somehow to record your results.
18:18:16 <skvidal> f13: open a releng or a bodhi ticket?
18:18:19 <f13> bodhi
18:18:23 <skvidal> f13: okay
18:19:01 <skvidal> I'll do that
18:19:07 <f13> #action skvidal will file a bodhi ticket to define the changes we want to see for critical path packages
18:19:07 <wwoods> f13: right, but I've not been further involved in drafting that part of the proposal
18:19:26 <wwoods> which is why I'm asking - I don't know what's reasonable to ask of the maintainers
18:19:26 <f13> anything else on critical path?
18:19:29 <skvidal> wwoods: if you'd like to be - I'm all ears - but I can file a ticket
18:19:42 <wwoods> I'd like to stay informed at least, but yeah, do that
18:19:45 <skvidal> wwoods: and I'll pass it to you for your critique if that's helpful
18:19:48 <f13> wwoods: maintaines shouldn't see much different
18:19:57 <f13> probably modeled after the security team clearance for security updates
18:21:00 <f13> Ok, if there is nothing else, we'll move on
18:24:04 <f13> #topic mass rebuild
18:24:09 <f13> there we go
18:24:28 <f13> Ok, bill completed his task of creating the f12 rebuild page
18:24:34 <f13> and announcing the rebuild.
18:24:48 <f13> I further updated the scripts and we kicked off the rebuild.
18:25:11 <f13> ricky and I fixed another performance issue with cvs, and that enabled me to just do a single thread submission script
18:25:18 <f13> and more than overwhelmed the buildsystem with submissions
18:25:37 <f13> although there was some issues with my use of ssh sockets and some stalling of hte script, all the builds had been submitted by last night
18:25:46 <f13> and the buildsystem had about 3K pending this morning
18:25:52 <rdieter> woo
18:26:00 <f13> However, we've had to pause the rebuilds in order to install a security fix for python
18:26:07 <notting> oops.
18:26:11 <f13> which should happen soon as dgilmore noted.
18:26:19 <f13> just waiting on rhn sync
18:26:36 <f13> so I doubt we'll be done with all the builds by Tuesday, but Wed looks pretty good
18:26:43 <f13> We did start a day late waiting on glibc/gcc changes
18:27:09 <f13> so once the scripted builds end, I'll be looking at the failures and generating the lists of things that still need to build, as well as doing the mass tagging.
18:27:24 <f13> #action once rebuilds finish, lists will be generated of failed/needed rebuilds
18:27:31 <f13> #action f13 once rebuilds finish, lists will be generated of failed/needed rebuilds
18:27:44 <jwb> where are we with updates to f10/f11 rpm?
18:27:53 <notting> f11 is in updates-testing
18:28:06 <jwb> are we caring about f10?
18:28:15 <notting> panu said he was going to look at F-10 next, but he didn't appear to get to it before he left on friday
18:28:21 <jwb> ok
18:28:29 <notting> (and he's on vacation now)
18:28:41 <f13> oh boy
18:28:52 <f13> I talked with the QA folks about this a bit earlier today
18:29:03 <notting> so, we can poke ffesti/jnovy to pick that up
18:29:15 <f13> there is a problem where the rawhide rpm in the rebuild requires the new compression support to install
18:29:19 <f13> it also requires new glibc
18:29:34 <f13> so we can't just drop a old formatted new rpm build, as it would require the new glibc which requires the new compression
18:29:42 <f13> so some rawhide users may get caught in the lurch
18:30:09 * jwb can see the "gah!  broken!" vs. "rawhide eats babies" threads now
18:30:10 <f13> I do believe they are going to get the rawhide blog page updated with some of this information
18:30:11 <dgilmore> f13: gahh
18:30:13 <rdieter> eww, sounds familiar
18:30:25 <f13> yeah, i'ts like the md5sum thing, only worse
18:30:57 <jwb> so that's twice now we've hit a situation like this.  is there something we should be doing to not have this happen?
18:31:07 <rdieter> maybe need a special koji target with an older/compat glibc/rpm ?
18:31:11 <f13> at least this time it's happening before Alpha, so we won't be leading a bunch of people into a trap
18:31:11 <notting> jwb: don't build rpm in rawhide?
18:31:18 <drago01> jwb: like going back to no updates to rpm
18:31:31 <wwoods> might need a SOP for RPM format changes that break compat
18:31:35 <f13> there are reasons why rpm was stagnant for very long times
18:31:41 <jwb> wwoods, right
18:31:54 <notting> if you stage all changes in $prior-updates, it would sort of work
18:31:56 <f13> how long did we have an rpm in rawhide that supported the new compression format?
18:31:56 <notting> but be rather odd
18:32:10 <dgilmore> wwoods: we do because it also effects buildsystem, and lots of other things
18:32:24 <dgilmore> f13: a week
18:32:26 <notting> f13: rpm landed on monday
18:32:33 <dgilmore> maybe a bit longer  but not much
18:32:43 <f13> would we want longer than that?
18:33:01 <f13> actually, lets see if we can get a volunteer to own a SOP on rpm changes
18:33:18 <f13> I've got a bit much to take that on, is there anybody here that would be willing?
18:33:22 <notting> for *this specific* case, we could build a bumped f-11 version, and tag that into f-12
18:33:25 <f13> (if not, we can pitch it to the -devel list)
18:33:39 <dgilmore> f13: i will since it effects buildsysem among others
18:33:39 <f13> notting: why not just tag the f11-testing one ?
18:33:51 <wwoods> What about the general case? Is it acceptable to make changes that will result in rawhide being unable to be upgraded without manual intervention?
18:33:58 <f13> #action dgilmore will draft (with help) an SOP for managing rpm changes in rawhide.
18:33:58 <notting> f13: that would cause 4.7.1-1.fc11 to be newest over 4.7.1-1.fc12
18:34:00 <wwoods> If so - what's the required minimum update window?
18:34:16 <wwoods> like "If you don't upgrade rawhide at least once every week/two weeks/month stuff may break"
18:34:34 <f13> notting: for people that already have 4.7.1-1.fc12 that's fine.  For people who don't, they're going to be a lower n-v-r and update to the .f11 one
18:34:49 <f13> I don't think F12 would /ship/ like this, just be a crutch until after Alpha
18:35:09 <f13> wwoods: one week maybe 2 seems reasonable to me
18:35:25 <dgilmore> f13: id think 2 weeks
18:35:36 <notting> f13: *shrug*. wfm.
18:35:50 <notting> f13: we'd be breaking our own rawhide-never-goes-backwards rule
18:36:11 <wwoods> so I'd assume that the SOP should require double that (4 weeks) between the compatibility package hitting the repo (i.e. RPM with xz support) and the rebuild landing
18:36:14 <f13> notting: ah, good point.
18:36:32 <f13> wwoods: that's really long
18:36:41 <wwoods> er, wait, maybe not double. maybe 3 weeks. or two weeks and a day. or something.
18:36:44 <f13> and going to make feature freeze a lie
18:36:56 <wwoods> yeah obviously this case is grandfathered in
18:36:58 <f13> or we're going to have to start doing our rebuilds /after/ feature freeze
18:37:13 <f13> and adjusting schedules to cope with that
18:37:28 <wwoods> does a two-week window work, then?
18:38:17 <f13> dunno.
18:38:20 <dgilmore> f13: id say 2 weeks + 1 day
18:38:34 <f13> I'd say work with dgilmore to put together an SOP and present it to the community for feedback
18:38:36 <dgilmore> for when the rebuild content would land
18:38:40 <dgilmore> so next tuesday
18:38:55 <f13> it's too late to change anything for this rebuild
18:38:57 <dgilmore> wwoods: lets talk afte the meeting
18:38:59 <wwoods> sure
18:39:03 <f13> because packages are already landing that require the new rpm bits
18:39:42 <dgilmore> f13: right but the rpm that needs new glibc that needs new rpm has not yet landed right?
18:39:50 <dgilmore> f13: its in the rebuild chain?
18:40:08 <wwoods> in this case we might just need to muddle through without the SOP to avoid blowing the schedule
18:40:16 <wwoods> but surely we can learn from this
18:40:32 <f13> dgilmore: maybe?
18:41:00 <notting> dgilmore: no
18:41:07 <dgilmore> f13: i think we need to carefully land things with circular deps
18:41:09 <f13> yeah, latest rpm build doesn't require the new glibc
18:41:12 <notting> dgilmore: panu did a 4.7.1 rebuild in the middle/late week
18:41:20 <notting> f13: it does on x86. not on x86_64. (not sure why)
18:41:26 <f13> dgilmore: if by carefully you man "all at once".
18:41:34 <f13> notting: ergh.
18:41:40 <f13> notting: delay of build start
18:41:42 <dgilmore> notting: thats odd
18:41:54 <dgilmore> notting: koji makes sure that the same buildroot is used on all arches
18:41:57 <f13> the x86 build could have started well before the x86_64 build
18:42:00 <f13> oh
18:42:06 <dgilmore> f13: doesnt matter
18:42:07 <notting> x86 could have different symbol versions than x86-64
18:42:11 <f13> gotcha
18:42:24 <dgilmore> koji 1.3 uses repoid so they all use the same repo
18:42:53 <f13> we're getting off in the weeds here though
18:43:03 <f13> We've got an action item, I'd like to see some work happen outside the meeting.
18:43:09 <dgilmore> right
18:44:23 <f13> ok, lets move on to the next topic.
18:45:05 <f13> #topic No Frozen Rawhide
18:45:15 <f13> so yay, this got passed by FESCo
18:45:24 <f13> https://fedoraproject.org/wiki/No_Frozen_Rawhide_Proposal for the logs
18:45:50 <f13> I think the biggest blocker here is bodhi
18:46:07 <f13> which means I need to file a ticket in bodhi for the changes.
18:46:18 <f13> #action f13 to file a ticket with bodhi to get changes in to support no frozen rawhide
18:46:34 <lmacken> once the tickets exist, also make sure it's on my radar, and that I know it's a high priority :)
18:47:18 <f13> There is hope we can get it done by alpha freeze, but if we don't get it ready by alpha freeze, we punt to F13
18:47:22 <f13> lmacken: noted
18:47:47 <f13> If we do get it done in time, we'll have to adjust our schedules a bit too, so we'll need to send poelcat a 6-back of $beverage
18:49:19 <f13> other than that it'll just be some more compose tool work with cron jobs and such, and letting the mirrors know that changes are coming to layouts
18:49:23 <f13> and lots of wiki page changes.
18:49:58 <f13> anything else on this before we move on?
18:51:05 <wwoods> #link https://fedoraproject.org/wiki/No_Frozen_Rawhide_Proposal
18:51:05 <notting> f13: who has the action to do magic tag-fu on the f11 rpm?
18:51:11 <notting> f13: and are we bumping the release or not?
18:51:34 <f13> notting: good question.
18:51:46 <f13> #topic Rawhide fixup for rpm deps
18:52:00 <notting> so, the options are:
18:52:14 <notting> 1) tag existing F-11 package (and break 'rawhide never goes backwards' rule)
18:52:21 <notting> 2) build a bumped F-11 package, and tag that
18:52:57 <f13> we probably should do 2
18:52:59 <f13> so I'm +1 on 2
18:54:23 <notting> i can do that. just build into updates-candidate, tag for f12, but not push in bodhi for f-11?
18:54:36 <rdieter> sounds like a plan
18:55:31 <notting> ok, i'll do so. i'll e-mail the rpm team so i don't utterly confuse them
18:57:05 <f13> #agreed We will build a bumped .f11 package and tag it directly into rawhide (but not push it as an update in F11)
18:57:18 <f13> #action notting will build, tag, and inform the rpm team about the rpm changes.
18:57:45 <f13> alright, couple more topics.
18:57:50 <f13> #topic Fedora 12 Alpha
18:58:01 <f13> Feature freeze is tomorrow, but we're still not done with rebuilds.
18:58:18 <f13> and something is making compose images not show up, due to package mismatch?!  I've got to investigate that more after this meeting/lunch
18:58:27 <f13> #action f13 will investigate why we aren't getting installer images in rawhide
18:58:39 <f13> need to get that working before our test compose is due later this week
18:58:53 <f13> and our last topic
18:58:58 <f13> #topic Package Signing
18:59:10 <f13> I made great progress with mitr over the last week on sigul, our new rpm signing software.
18:59:45 <f13> I've got a test deployment of it here at my house in virt and have been able to successfully generate a new key and import an existing key, and sign + import + write out multiple builds with each key using sigul
19:00:17 <f13> I also re-write sign_unsigned as sigulsign_unsigned that is far more simple, far less code, and capable of taking in a list of builds or a tag and running sigul in batch mode
19:00:37 <f13> smooge and I are working on getting our new signing hardware in place and bridge guest prepared for deployment
19:00:50 <f13> It's my goal that we'll be able to use sigul and sigulsign_unsigned by the end of this week.
19:01:02 <smooge> yes.. I am trying to get the box to do what its supposed to do.
19:01:02 <f13> Then we can look at automating it in some ways
19:03:35 <f13> #open floor
19:03:37 <f13> whoops
19:03:40 <f13> #topic open floor
19:03:54 <f13> If there are no other topics to discuss, I'll close the meeting in 5
19:05:19 <notting> are we signing alpha?
19:06:06 <f13> yes
19:06:29 <f13> I can't guarantee signed packages /after/ alpha though
19:07:17 <f13> Alright, thanks all!  Until next week!
19:07:18 <f13> #endmeeting