18:18:30 <f13> #startmeeting Fedora Release Engineering
18:18:42 <f13> #topic Roll Call
18:18:49 * dgilmore is present
18:18:50 * notting is here
18:19:16 <f13> ping: notting jwb rdeiter dgilmore poelcat warren spot lmacken
18:19:23 * spot is around
18:19:29 * jwb is 1/2 here
18:19:39 * warren here
18:21:06 <f13> alright lets get started.
18:21:09 <f13> #topic old business
18:21:22 <f13> Reviewing the logs from last week there were a couple action items.
18:21:36 <f13> however I didn't see poelcat respond
18:21:53 <f13> the first action was for poelcat to add bodhi dates to the F12 schedule
18:24:53 <f13> Does not look like this has happened yet.
18:25:03 <warren> move on for now?
18:25:18 <f13> #action poelcat still needs to add bodhi dates to F12 schedule.
18:25:30 <f13> #nick poelcat
18:25:31 <f13> #action poelcat still needs to add bodhi dates to F12 schedule.
18:25:33 <f13> there we go
18:25:48 <f13> Next up was orphan clean up
18:26:00 <f13> action item was for me to file a releng ticket about purging the orphans.
18:26:17 <f13> I've done so, and sent the first round of lists to review to the community
18:26:25 <f13> a bunch have been picked up, I need to send the updated list out
18:26:51 <f13> #action f13 to send an updated list of orphans to be purged
18:27:07 <f13> Next was FAD follow up, which I had an action to review.
18:27:19 <f13> I failed, so I still need to do this
18:27:32 <f13> #action f13 will /really/ spend time on FAD this week.
18:28:05 <f13> There were topics about updates pushess, which I had an action item to file a ticket to back down gpg sig cleaning, which I forgot to do so that's still an open action item.
18:28:23 <f13> #action f13 still needs to file ticket about backing down how aggressive the gpg signed copy cleanup is in koji
18:28:35 <f13> and the last action was to talk about critical path this week.
18:28:47 <f13> that's the end of old business, lets start right in at critical path.
18:28:50 <f13> #topic Critical Path
18:28:58 <f13> skvidal: wwoods: this is your show now, are you around?
18:29:13 * skvidal reads up
18:29:29 <skvidal> okay
18:29:58 <skvidal> gist is that there are a set of pkgs which will require an extra step if the owner wants to update them
18:30:06 <skvidal> and/or change them in a time where we would normally be frozen
18:30:48 <skvidal> we've got the list of pkgs now - and once I add the solve out deps problem I can get the list of owners and we'll notify them
18:31:01 <skvidal> f13: is there another component of this which is special for releng?
18:31:20 <f13> well, the "extra step" would involve something with bodhi right?
18:31:20 <notting> skvidal: "is bodhi ready"?
18:31:38 <skvidal> I do not know of any bodhi changes - lmacken?
18:32:16 <notting> i thought it required some magic 'tag but no push' bodhi workflow
18:32:33 <skvidal> I believe it requires some
18:32:40 <skvidal> but I do not know the status of them
18:34:05 <f13> right, it would require bodhi work, and then releng/qa work on the side of twiddling whatever bit in the update request necessary to make it "go"
18:35:21 <skvidal> ok
18:38:00 <f13> lmacken: don't suppose you're around?
18:38:17 <f13> didn't think so
18:38:52 <f13> so maybe next week we can get a report from luke on what if any changes have gone into bodhi to support critical path packages.
18:39:00 <f13> #nick lmacken
18:39:15 <f13> #action lmacken to report next week on bodhi changes for Critical Path Packages
18:39:27 <poelcat> f13: http://poelstra.fedorapeople.org/schedules/f-12/f-12-releng-tasks.html #66 was supposed to address bodhi
18:39:43 <f13> so without bodhi support, can we still do some of the proposal, give extra special care to things in the critical path when freeze requests come through?
18:39:46 * poelcat in a conflicting meeting
18:40:00 <f13> poelcat: right you are, I'll note that later.
18:40:10 <poelcat> f13 anything else missing?
18:40:40 <poelcat> sorry for crashing current topic
18:40:44 <f13> I don't think so, that was just a follow up from last week
18:41:55 <lmacken> f13: yeah, I'm kind of around
18:42:43 <f13> so lets assume for a moment, that nothing else changes in our development cycle, and we're still using trac to handle freeze requests
18:43:09 <lmacken> no changes have gone into bodhi for the critical path stuff... is there a ticket somewhere that defines what the workflow needs to look like in bodhi?
18:43:13 <f13> how would we make good on Critical Path in this scenario?  We'd need to know if the package freeze break requested is a critical path package
18:43:30 <f13> and then we'd need to get an indication in the ticket that it has had a second look
18:43:44 <f13> lmacken: I don't think a ticket exists, nothing outside of the feature page itself
18:44:44 <dgilmore> f13: you would need some way to know its a critical package
18:44:58 <dgilmore> and soem way to sigify that its been triple checked
18:45:22 <f13> isn't that what I just said?
18:45:41 <notting> so, someone (tm) would have to do the check against crit path packages, and then just do manual acks in the ticket
18:46:18 <f13> here is what I'm thinking
18:46:47 <f13> if we're still using Trac for tag requests, I'd like to spend some effort and get "make tag-request' a target in the make system, that will do the trac filing for people
18:46:58 <f13> as part of that, it could check to see if the package in question is a critical path package so we'd know it in the ticket
18:47:18 <dgilmore> f13: thats doable
18:47:19 <f13> that would require getting offtrack packaged and installed on people's systems, as well as having some quick utility to check if a package is crit path or not
18:48:28 <f13> we'd also have to use said utility for the tickets that are manually filed
18:48:38 <f13> so auto filed tickets should have some signature that it was filed from the make system
18:49:28 <f13> I should probably work on offtrack, since the likelyhood of getting no frozen rawhide approved in the next couple weeks seems pretty low.
18:49:44 <dgilmore> f13: we could add offtrac as a requirement for bodhi-client or fedora-packager
18:50:05 <f13> #action f13 will work on getting offtrac packaged up in Fedora and a 'make tag-request' target added to the make system.  Also tied into checking if critical path package or not.
18:57:54 <f13> hrm, brb, gotta take care of a diaper
19:03:24 <f13> back.
19:03:35 <f13> wwoods: skvidal: anything else to discuss on Critical Path?
19:04:35 <skvidal> f13: I'd like to discuss you feeding your child to goats - but that's not critpath related, really.
19:04:47 <f13> LOL
19:05:46 <skvidal> http://www.flickr.com/photos/iamjessekeating/3736843791/ <--EVIDENCE
19:06:04 <f13> there were goat pellets in his hands
19:06:23 <f13> moving along.
19:06:36 <skvidal> sure
19:06:37 <f13> #topic No Frozen Rawhide
19:06:58 <f13> This is the big FAD proposal, with the most to change.
19:07:13 <f13> It seems to have generally good feedback from those I've talked to about it, and from the mailing list, although not much there
19:08:18 <f13> so lets look at where we are with our needs.
19:08:27 <f13> compose tools, those were supposed to get a lot faster to handle multiple composes in a day
19:08:37 <f13> we got better, but are we to the point were we can do multiple in a day?
19:09:21 <f13> last night's rawhide mash started at 02:17
19:09:53 <f13> Finished up at 08:57
19:10:00 <f13> nearly 7 hours
19:10:26 <f13> to do another one would put it at 14 hours
19:10:42 <dgilmore> you could gamble on 3 a day
19:10:49 <dgilmore> but 2 would be doable
19:11:00 * nirik somewhat jokingly wonders if we really need drpms for ppc64 (since it's not really a tree that should be used by end users much)
19:11:03 <dgilmore> f13: though with smaller changes how much quicker would it be?
19:11:08 <f13> yeah, but then we're not leaving very much time left for doing updates pushes and epel pushes
19:11:26 <f13> nirik: that's actually one request that's been filed, the ability to configure which arches get drpms
19:11:27 <notting> nirik: there's a bug on that
19:11:33 <dgilmore> f13: epel pushes are independent of the other two  but its all load on /mnt/koji
19:11:49 <f13> dgilmore: yeah, it's the load on /mnt/koji I wonder about
19:11:56 <nirik> yeah, I don't know how much it would help, but they seem not too required.
19:12:07 <f13> we were going to try running two composes at once on releng2 and releng-stage but I think our mirror issues have killed that effort
19:12:12 <f13> jwb: do you know where we are with that?
19:12:26 <dgilmore> f13: i think we have done updates and epel at the same time
19:13:00 <f13> updates right now are taking an artificially long time to finish due to restarts
19:13:06 * jwb reads scrollback
19:13:27 <jwb> f13, two composes at once?
19:13:51 <f13> jwb: yeah, we were going to test to see if we would get benefit of splitting composes across mutliple boxes
19:13:59 <f13> or if /mnt/koji would be too much of a single bottleneck
19:14:12 <jwb> f13, oh.  we never tried that because we're still fixing fallout from drpms
19:14:17 <jwb> among other things
19:14:50 <f13> right
19:14:54 <jwb> f13, also, releng1.stg can't talk to koji.  so that's sort of useless
19:14:59 <jwb> at least at the moment
19:15:06 <jwb> oh wait
19:15:08 <f13> which makes me worried that we wouldn't be able to produce multiple trees in a single day
19:15:13 <jwb> you said composes...
19:15:18 <ricky> What needs to happen on koji1.stg to make releng1.stg not useless?
19:15:24 <jwb> did you mean update pushes, or tree composes?
19:15:56 <f13> jwb: well, that kind of depends
19:16:08 <f13> jwb: we'd be doing one cron compose to produce rawhide based on the dist-rawhide tag
19:16:40 <f13> then we'd have to produce 2 other trees from tags managed by bodhi
19:16:50 <f13> and then we'd have to do the normal bodhi pushes
19:17:51 <f13> thinking it through, and thinking about where things land for the content to be the release
19:18:21 <f13> we might have to do a full Everything compose at Alpha, and that stays static, while we compose an updates-testing and updates like repo set to complement it.  So the content being composed is smaller each night
19:18:38 <f13> or we have to do a rawhide like everything tree each night, and then one addon repo that is the updates-testing
19:20:46 <f13> either way we'd be doing something close to 2x cron ran rawhide composes, and then whatever other bodhi pushes and epel pushes are necessary
19:22:03 <f13> we'd also need changes in bodhi for how to handle update requests for a pending release rather for something already released
19:22:11 <f13> and the push tools around that to know what to do (tag, but don't mash)
19:22:27 <f13> we have to fine tune the mirror layout
19:22:30 <f13> (see above)
19:22:44 <f13> Those are the real hard parts
19:22:53 <f13> and need to be figured out before we take it to FESCo
19:23:02 <f13> so who would like to work with me on these things this week?
19:23:36 <f13> #action f13 will try to get folks together to define missing parts of the no frozen rawhide proposal this week
19:24:19 <f13> anything else on this topic?
19:26:16 <f13> alright
19:26:25 <f13> #topic Fedora 12 Alpha
19:26:31 <f13> Couple of dates of interest
19:26:54 <f13> This Friday there is a bug blocker review day
19:27:15 <f13> next Tuesday is Feature Freeze, which is also the orphan blocking day
19:27:35 <f13> next Wed. we need to do a test compose and deliver it to QA for an initial round of testing
19:28:08 <f13> another blocker bug day that Friday
19:28:14 <f13> the following Tuesday is the alpha freeze
19:28:41 <f13> and the Thursday after that is when we're supposed to compose the Alpha release candidate, assuming that the blocker bugs are clear
19:29:02 <f13> I've got tickets filed in trac for most of these things
19:31:36 <f13> guess there is nothing left on those.
19:31:45 <f13> #topic open floor
19:31:56 <f13> if there is no new topic I'll close the meeting in 5 minutes
19:32:07 <notting> so, XZ in rpm, and x86 support -> mass rebuilds
19:32:13 <notting> x86 support changes are ready to go
19:32:38 <notting> XZ in rpm is not quite ready (need a new rpm build for infrastructure - working on it)
19:33:04 <f13> oh geez, mass rebuilds, forgot that
19:33:08 <f13> #topic Mass Rebuild
19:33:18 <notting> i should say, x86 support changes are live in koji now
19:33:20 <f13> even less time now to do
19:33:26 <notting> since friday
19:33:30 <f13> ok
19:33:37 <f13> what's the ETA on the XZ support?
19:34:26 <notting> getting the raw bits in... today? then it would be a matter of flipping the default in redhat-rpm-config &* sending out a large warning to the lists
19:34:47 <dgilmore> f13: we need builder support first
19:34:50 <f13> what's going to happen during the time where that config is enabled but we haven't rebuilt everything?
19:35:10 <jwb> notting, so... i'm assuming f11 RPM will not be able to cope with f12 RPMs after that flip
19:35:14 <notting> jwb: correct
19:35:19 <f13> I'm looking at https://fedoraproject.org/wiki/Fedora_11_Mass_Rebuild for how we did it for 11, we should duplicate that for 12 as close as possible.
19:35:26 <jwb> notting, does that sound a bit bad?
19:35:50 <notting> f13:  i'm not sure what you mean. XZ support in rpm requires macros to enable it; if we don't do that, nothing happens unless some wacko sets them by hand in their own .spec file
19:35:52 <f13> I'd hope there would be a planned F10/F11 update to rpm to handle the F12 content
19:35:58 <jwb> f13, right
19:36:04 <notting> yes. but i don't control that
19:36:28 <dgilmore> of course EL-5 users will be just as hosed as they are today
19:36:31 <f13> notting: what's going to happen when we enable it, but we haven't actually done the mass rebuild yet, so we have a few days worth of builds that use the new rpm but many that don't?
19:36:33 <f13> anything bad?
19:36:58 <notting> f13: they'll fall into the boat that jwb mentioned. aside from that, nothing too bad
19:37:00 <f13> dgilmore: it's easy enough to tell rpm to ignore file md5 or to generate srpms with the md5 file checksum.  Harder to ignore the compression
19:37:02 <dgilmore> f13: id hope rpm deals with each rpm just fine
19:37:06 <f13> nod
19:37:18 <notting> as noted on the feature page , xz does not break deltarpm, although it renders the deltas suboptimal
19:37:50 <f13> So if the stuff gets in today, and we start rebuilds on Friday
19:38:00 <f13> stuff will still be rebuilding by the 28th
19:38:04 <f13> which is the feature freeze
19:38:12 <notting> do we need to wait until friday?
19:38:41 <f13> last time we started on the 23rd after about 6 days warning
19:39:25 <f13> 4 days later the automated builds were done
19:39:34 <f13> with 500~ builds needed to be done manually
19:40:10 <f13> notting: we don't have to wait, but it just gives maintainers less time to opt-out, and it will cut into time that maintainers are scrambling to finish things up by feature freeze
19:40:35 <f13> plus our compose tools are going to go nuts when 2K packages change each day.  drpms will go through the roof
19:40:46 <notting> turn them off
19:40:48 <notting> see above :)
19:41:08 <f13> ok, we could do that
19:41:18 <f13> at some point we'll turn them back on though right?
19:41:33 <notting> yeah, after the mass rebuilds are done
19:41:44 <f13> guess that won't be too bad 'cause then it'll just be things changing after the mass rebuild.
19:41:47 <notting> but there's not a lot of sense in delta-ing across payload compression differences
19:41:49 <f13> so yeah, I like that idea.
19:42:13 <f13> mmcgrath: ping; did the CVS filesystem overhaul complete?
19:42:32 <mmcgrath> f13: yeah
19:42:33 <f13> mmcgrath: CVS locking was one of the biggest bottlenecks in last mass rebuild, is it better now do you think?
19:42:45 <mmcgrath> f13: I'd think so but it's worth testing for real
19:42:49 <f13> k
19:43:20 <f13> so if we started Thursday, we could potentially be done by Monday
19:43:32 <f13> of course, mirrors are just now trying to settle down from the outage
19:43:38 <f13> this isn't going to help
19:44:52 <notting> f13: so, i can clone the F11 page and do some adjustments
19:46:01 <f13> show of hands, how many people think we can/should do this?
19:46:23 <f13> notting: haha, I just erased a line asking you to do that, first I wanted to get some sort of consensus that we should push forward on it
19:46:45 * jwb raises 1/2 a hand
19:46:55 <f13> mmcgrath: I'd be interested in your opinion too.  How well do you think our infrastructure could handle a mass rebuild this weekend?
19:46:56 * notting raises a hand
19:48:13 <mmcgrath> f13: I believe it'd handle it well.
19:48:23 <mmcgrath> as for publishing to the mirrors that's different.
19:48:33 <f13> yeah, I'm considering that part of our infrastructure
19:48:43 <mmcgrath> Though we're better positioned now then we were this time last week.
19:49:37 <dgilmore> f13: i think we are in a fair state to do it this weeken
19:49:37 <dgilmore> d
19:49:56 * ricky would rather not have another factor to add to the mirror madness analysis, but ah well :-/
19:49:58 <f13> ok.
19:50:05 <mmcgrath> f13: do you have an estimate of how many files would change?
19:50:17 <dgilmore> mmcgrath: all of them in rawhide
19:50:24 <notting> what dgilmore said
19:50:26 <f13> mmcgrath: rough estimate would be every file under development/
19:51:20 <f13> #agreed We should attempt a mass rebuild driven by releng for arch and XZ support.
19:51:37 <f13> #action notting will copy the F11 mass rebuild wiki page for F12
19:52:12 <f13> Do folks feel that Thursday is a good starting day?
19:52:28 <f13> leaves a little time for folks to opt-out, and for mirrors to settle down, and for us to finish before the feature freeze
19:53:17 <jwb> mirrors settling down is a lie
19:53:23 <ricky> mmcgrath: That's currently 106377 files, for what that's worth.  About 1/6th or so of fedora-enchilada
19:53:26 <jwb> but the opt-out thing is fair
19:53:39 <mmcgrath> ricky: thanks
19:53:50 <jwb> ricky, minus drpms.  those won't get touched for now
19:53:58 <f13> jwb: you don't think mirrors will be in a better place by Thursday?
19:54:06 <notting> f13: so, i'm saying we'll have automated rebuilds done by July 28th, and any cleanups done by the 4th?
19:54:31 <jwb> f13, better, perhaps.  settled, no.  i have a push i'm going to start today hopefully
19:54:54 <ricky> jwb: Oh, I didn't know we had drpms for development
19:55:06 <jwb> ricky, we had them there first even :)
19:55:09 <f13> notting: that's a fair target.  I don't know that we've /ever/ had cleanups from mass rebuilds fully done
19:55:10 <ricky> Ah, nice
19:55:33 <ricky> The new number is 90662 then
19:55:57 <f13> #agreed Mass rebuilds will start Thursday the 23rd, to finish by Tuesday the 28th.  Cleanups will target completion by tuesday the 4th
19:56:21 <notting> please read over https://fedoraproject.org/wiki/Fedora_12_Mass_Rebuild
19:57:25 <f13> the XZ rpm payloads link is a dead link
19:57:33 <notting> reload!
19:57:41 <f13> bingo
19:57:58 <notting> i'll update the releng scripts with s/f11/f12/
19:58:16 <f13> Timeline looks odd
19:58:37 <f13> looks like it references the F11 stuff
19:58:42 <notting> cleared; that appeared to be a post-mortem section
20:00:42 <f13> notting: ok, looks good
20:00:46 <f13> ready for announcement
20:00:51 <notting> ok. i'll send mail to f-d-a
20:01:33 <f13> #action notting will announce the mass rebuild to fedora-devel-announce
20:01:39 <f13> anything else regarding this topic?
20:01:46 <f13> it's going to be a busy week next week (:
20:04:48 <f13> Ok, I think I'm going to call meeting.  Thanks for hanging in there, it was a long one.
20:04:58 <f13> #endmeeting