20:00:30 #startmeeting 20:00:30 Meeting started Wed Aug 8 20:00:30 2012 UTC. The chair is pwhalen. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:00:30 Useful Commands: #action #agreed #halp #info #idea #link #topic. 20:00:30 #chair pwhalen jonmasters bconoboy ctyler pbrobinson dgilmore 20:00:30 Current chairs: bconoboy ctyler dgilmore jonmasters pbrobinson pwhalen 20:00:37 .fas pwhalen 20:00:38 pwhalen: pwhalen 'Paul Whalen' 20:00:42 * maxam is here 20:00:44 .fas ctyler 20:00:45 ctyler: ctyler2621 'Christopher Tyler' - ctyler 'Chris Tyler' 20:00:46 .fas Frojoe 20:00:47 Frojoe: jacwang 'Jordan Cwang' - burningfool 'Jordan Cwang' 20:00:49 .fes blc@ 20:00:52 .fas djdelorie 20:00:53 .fas dmarlin 20:00:54 djdelorie: djdelorie 'DJ Delorie' 20:00:55 .fas blc@ 20:00:57 dmarlin: dmarlin 'David A. Marlin' 20:01:00 bconoboy: blc '' 20:01:05 .fas herrold 20:01:05 orc_fedo: herrold '' 20:01:11 .fas maxam 20:01:12 maxam: maxam '' - maxamaxim '' - maxamillion 'Adam Miller' 20:01:32 * pbrobinson wonders what's the need for the .fas bit? 20:01:38 #topic 1) F18 - Mass rebuild status update 20:02:12 * masta is here 20:02:19 so we're almost there! anything need mentioning here? 20:02:23 .fas parasense 20:02:24 masta: parasense '' 20:02:25 It's pretty much complete, there's a few blockers which is holding up some, there's likely a few stragglers but we're less than 500 builds missing 20:03:04 same packages: 11062 20:03:15 older: 446 20:03:32 .fas jsmith 20:03:32 jsmith: jsmith 'Jared Smith' - mikejsmith11 'Mike J. Smith ' 20:03:49 with the branch today we're also building f19 packages 20:04:08 EOF 20:04:13 what're the blocks to finishing f18 builds? 20:04:17 we had quite a few idle builders so likely not to slow k-s at all 20:04:36 pwhalen: completely un related 20:04:52 we're blocking on webkitgtk3 20:05:39 does someone want to take a look at that? 20:05:42 and I need to smack the 3.6 kernel a little bit as it's now starting to block some of the newer builds. One of the drivers is at fault, I was hoping if I left it a few days it would be fixed mainline, it's on my list to look at tonight 20:05:51 .fas jcapik 20:05:51 jcapik: jcapik 'Jaromír Cápík' 20:05:55 pwhalen: it needs upstream involvement 20:06:14 ok, any others? 20:06:57 that's it for the moment. There's likely some more minor ones that haven't bubbled to the top yet that I'm not aware of 20:07:06 #topic 2a) Mass rebuild - statistical analysis of time taken overall (ARM vs PA) 20:07:25 * djdelorie posted his scripts to http://djdelorie.fedorapeople.org/ 20:07:54 djdelorie, did you want to share your first round of analysis? 20:07:58 I have a bastard script which did some comparison over the f17 time frame and I was going post some stats over the weekend 20:08:05 these scripts showed us about 5x slower than PA in the f17 timeframe 20:08:18 5x slower than what? 20:08:37 if the current data is accurate (needs a second set of eyes), we're currently between 1.0 and 1.3x slower than PA 20:08:41 PA == i386 20:08:53 1.0 slower is the same speed? 20:08:57 we're expected to be a bit slower tho no? 20:09:00 that's a comparison of about 3800 packages, not including anything build on a guru or smarttop 20:09:03 for building the entire mass rebuild, on average per package, or what ? 20:09:08 unless we have a ton of huge arm machines available 20:09:09 yes, the data says we're as fast as PA 20:09:18 that can't be right 20:09:23 that's the "f18" tag from the koji's 20:09:40 djdelorie, so thats on a per package basis? 20:09:46 I'm willing to believe "that can't be right" too :-) 20:09:53 my scripts compare build times on a per-package basis, yes. 20:09:56 right, so you're taking an NVR and comparing the time it takes to build on PA vs ARM? 20:09:58 then average the per-package ratios 20:10:03 yes 20:10:18 takes a couple hours to run the extract, so I posted the extract files too 20:10:46 so is that done using just the NVRs from the mass rebuild? Mainline had some fairly large infra changes recently just before the mass rebuild 20:10:47 what I do *not* check is "days to complete the rebuild", which would include overhead and mucking about 20:11:14 the days to complete the rebuild would be completely useless data 20:11:18 er, it's "latest build" for everything with the f18 tag, for each of the four arch's 20:11:34 pbrobinson: perhaps not, it gives us a sense of overhead involved 20:11:54 djdelorie +1 20:11:54 ok, cool. Is there a nice web page or something with the output? 20:13:03 djdelorie: not really because koji-shadow for the last mass rebuild had to deal with crashes when I was a sleep, outages of both mainline and arm.koji and various other things which add a whole lot of useless data to the stats 20:13:23 the "output" is thus: 20:13:23 pkgcount 3800 20:13:23 360 1126 1.0 1.0 i386 20:13:24 300 1104 0.8 1.0 x86_64 20:13:24 420 1456 1.2 1.3 arm 20:13:24 360 1259 1.0 1.1 armhfp 20:13:26 well hes analyzing pkg build times tho 20:13:28 I'll post the graph... 20:13:33 and that adds time that is a lot more than a standard deviation 20:13:57 pbrobinson: that's the overhead I'm talking about 20:14:04 if we do well *despite* that, we're good :-) 20:14:30 #action djdelorie to post stats on build comparison - arm vs PA 20:14:31 if we're looking at providing stats for mainline promotion in terms of build times etc it's not useful data 20:15:01 but I'm looking forward to seeing pretty graphs and details for the raw NVR build times :-) 20:15:02 http://www.delorie.com/arm/koji-f18-times.png 20:15:15 maybe yes, maybe no. Some opponents only care about overall turnaround time 20:15:28 what are the axis? 20:15:38 oh sure, make me think... 20:16:07 Y is relative number of packages, and X is build time but I won't guarantee it's "minutes" 20:16:21 I was about to ask the same because I'm not sure how to read the picture 20:17:11 #link http://www.delorie.com/arm/koji-f18-times.png 20:17:45 #topic 2b) Mass rebuild - are we doing it right (koji-shadow vs mass queuing)? 20:17:49 is it possible to do a table of the packages with something like package-VR and the times with a % difference at the end so we could look at specific packages like qt or the kernel or something? 20:17:53 I think X is build time in minutes 20:18:00 djdelorie: so it's actually a histogram then, right? 20:18:06 ctyler: yes 20:18:22 pbrobinson: the scanning takes only seconds, download the two tarballs and play 20:18:27 djdelorie: so the relative position of the two "mountains" doesn't look like a 1.3 multiplier to me 20:18:30 pwhalen: yes we are because we need to build the same as mainline, and we don't do that with mass queueing 20:18:40 it's the extracting-from-koji that takes hours 20:18:43 #info on the link above, Y axis is number of packages, X axis is build time in minutes 20:18:52 we should be optimising koji-shadow so it more effect, more parallel and less crashy 20:19:03 pbrobinson, was the rebuild not queued in order? 20:19:24 ctyler: it's logarithmic, and the average ratio is the average of the ratios of build times, not the ratio of the sums of build times 20:19:41 i.e. the histogram doesn't show you the per-package correlation. 20:19:44 Data is messy :-) 20:20:13 pwhalen: that's fine if we're 100% up with mainline before it starts, but then if that was the case koji-shadow would run the same way and just queue it all up, which is what it did for a massive chunk but it was slow in doing it 20:21:07 why doesnt the script just monitor the koji tasks page and que the same pkgs? 20:21:14 for mainline that is 20:21:23 and just que the exact same in the same order here 20:21:25 #link http://djdelorie.fedorapeople.org/ - has koji stat scripts 20:21:46 that's what koji-shadow does in an ideal world, but arm is still not an ideal world 20:22:41 so, we need someone to look at rewriting the tool then 20:22:52 fossjon: because you also need the same underlying repository with the same identical NVRs as mainline to ensure we get the same builds which is the whole point of koji-shadow and what we need to be delivering if we ever wish to get promoted to primary 20:22:54 fossjon: it's not just build order that has to be considered, it's the versions of the dependencies used in PA. koji-shadow tracks this. a simple queuing script would not. 20:23:29 im assuming mainline tracks that, which is why copying them would be simple?? 20:23:35 thats all i was wondering tho 20:23:43 pwhalen: Not rewriting, but some big fixes would be good such that multiple instances wouldn't clobber each other. Also, we should be trying to get to primary so we don't need koji-shadow 20:24:05 s/big fixes/bug fixes/ 20:24:05 so we could end up with issues if say an arm package had a broken build for a package that bumped NVR we'd suddenly find that half the distro is built against the wrong package 20:24:32 i also thought that we were running repoclosure to tell us whats broken as well to tho 20:24:36 But koji-shadow with prefer-latest (or whatever that option is called) doesn't do exact NVR matching, so the whole premise of k-s is moot. And without prefer-latest, it's toast. 20:24:50 fossjon: you're assuming wrong because you're assuming a perfect world which has no issues that mainline doesn't have 20:25:48 ok its alright, was just wondering 20:26:06 i havent really looked at the script to see what it does in detail 20:26:17 ctyler: yes, but for example (and this was one we had just moments before mass rebuild) if a version of say libdb doesn't build it will block all of koji-shadow because it doesn't have the newer one to prefer, but if we just dump everything it it will build against the older one no matter what 20:27:22 the bottom line is that koji-shadow is the most disciplined approach we can take short of being primary... if we're trying to loosen that discipline we're moving away from PA. 20:27:24 pbrobinson: agreed in that case. I think there's a lot of work that could be done here though. 20:27:25 so say evolution was broken, which had a soname bump recently, it would be building everything that depends on it against the older soname 20:27:26 * masta would like to read the script, look for these ways to improve scale 20:28:09 so, k-s it is.. for now.. shall we move on? 20:28:27 ctyler: there's a lot of work that can be done to improve koji-shadow and that would help us in the speed terms considerably and the rest would then become a moot point 20:28:37 pwhalen: yes 20:28:55 #link https://fedorahosted.org/koji/browser/util/koji-shadow 20:29:00 masta: it's shipped in the koji packages 20:29:02 #topic 3) Defining release criteria for F18 20:29:30 pwhalen: different to what we discussed the last time we discussed this? 20:29:47 or are we all just going to repeat again? 20:29:47 I started to edit our previously defined release criteria, its linked at the bottom of the main page 20:29:55 link? 20:30:12 http://fedoraproject.org/wiki/Architectures/ARM/Fedora_18_Alpha_Release_Criteria 20:30:25 #link http://fedoraproject.org/wiki/Architectures/ARM/Fedora_18_Alpha_Release_Criteria 20:31:03 didn't we add highbank to the supported boards for alpha+? 20:31:04 there were no changes to the f18 release criteria that affect us (that I saw) 20:31:05 what's the plan for following this? vfads? 20:31:54 I think the alpha date for PA is the 28th 20:32:08 sounds about right now we've branched 20:32:17 * adamw is around for any useful input he can give 20:32:17 so, shortly thereafter 20:32:46 pbrobinson, yes, highbank needs to be added, thanks for reminding me 20:32:47 so what are we currently blocking on? What's the status of media-creator images? 20:33:13 (dmarlin) 20:33:31 there's some issues with the 3.6 kernel. I was going to have a first stab at those over the weekend now we have rc1 out there might be some hope of compiling stuff 20:34:14 bconoboy: we have pushed anaconda and lorax changes upstream and they are included in master (to be F18)... 20:34:15 * pbrobinson wonders if adamw ever got his ARM presents running F-17? 20:34:31 pbrobinson: what issues exactly? 20:34:46 bconoboy: but we are using kickstart installs, so the F18 versions will not run without a text ui, and that is missing 20:35:08 pbrobinson: replied out-of-meeting 20:35:29 jcapik: look at the koji kernel logs and see for yourself, sorry while I do remember a lot of random crap I don't remember kernel failure errors ;-) 20:35:56 bconoboy: we can build images running the F17 versions, but have only tested on highbank and trimslice... 20:36:02 pbrobinson: ok ..... will look later 20:36:04 #info arm specific F18 anaconda changes are upstream 20:36:24 #info last of text mode installer option breaks anaconda/livemedia creator on arm 20:36:29 oops 20:36:29 bconoboy: no uboot support is in anaconda yet, so we have to handle the boot.scr, etc. from kickstart %post 20:36:31 dmarlin: so how far away are we from producing qemu and pandaboard images that way? 20:36:39 #undo 20:36:39 Removing item from minutes: 20:36:46 #info lack of text mode installer option breaks anaconda/livemedia creator on arm 20:36:51 anaconda team's working on a text mode installer for newUI now... 20:36:58 #info uboot support not yet in anaconda 20:37:09 pbrobinson: not sure about qemu, but panda images pose a problem due to partitioning 20:37:15 #info kickstart %post can be used for uboot setup 20:37:45 adamw: it's not clear that it will be included during f18. it's a real risk. 20:38:05 #link http://www.redhat.com/archives/anaconda-devel-list/2012-August/msg00003.html 20:38:07 pbrobinson: I sent email describing the current situation and options. please reply with suggestions 20:38:16 #info text installer feature may not make f18- this is a problem 20:38:33 so, if I can ask you all to take a look at the release criteria and forward feedback to list. I'll be fixing things like missing links (thanks adamw) and adding based on feedback 20:38:36 dmarlin: I saw your email, not had a chance to read, absorb and respond. It's on the todo for the weekend 20:38:51 pbrobinson: thanks 20:39:18 I think we need to aim to get at least kickstart working at the very least so images can be built for alpha. 20:39:48 I suggest we revisit the time to do a vfad closer to the 28th, when we know where we stand 20:39:58 I'm surprised the text options don't install given the amount of installs that RH must support over serial console on x86 servers 20:40:32 Is VNC install a viable alternative to text on ARM? 20:40:33 the idea is that you can use VNC. 20:40:52 good question 20:41:05 I don't see why that wouldn't work just as well on ARM as x86 20:41:28 ctyler: it may... currently untested 20:41:34 adamw: that wouldn't work in a lot of the govt secure install environments where there's a requirement of a completely isolated network for the install. 20:41:37 livemedia-creator uses VNC by default, aiui. 20:41:38 ctyler: but it won't work for livemedia-creator 20:42:00 adamw: I think only is using virt 20:42:10 only if using virt 20:42:14 ah, right. 20:42:21 what does virt have to do with it? 20:42:30 bconoboy: livemedia-creator has two modes, which confuses things 20:42:46 I thought livemedia-creator used kickstart files 20:42:46 one of them involves running an entire virtual machine, into which fedora is installed, from which installation the live image is created 20:43:02 pbrobinson: right - but it can install into a VM _or_ into a mock-ish environment, using that kickstart 20:43:05 is that boxgrinder? 20:43:12 pbrobinson: it does, but we have to specify --novirt 20:43:41 ah, weird, but presumably wonderful 20:43:56 dgilmore mentioned and issue about that on the list.... or somewhere 20:45:08 dmarlin: can we work with --novirt? 20:45:19 looking forward to using the image creation tool, kickstart, etc... 20:45:35 bconoboy: that's what we use, but only in text mode. --vnc needs virt 20:45:42 * adamw just pinged bcl, regarding livemedia-creator 20:46:06 dmarlin: Okay, thanks 20:46:13 adamw: tnx 20:46:31 hi bcl, this is blc ;-) 20:47:08 bcl: We're going over release criteria for fedora 18 arm and have a problem with no text mode being available 20:47:39 well, yeah :) 20:47:43 bcl: arm doesn't have virt. livemedia creator needs virt for vnc mode. with -novirt we need a text mode. 20:48:05 for anaconda installs vnc is untested on arm. may or may not work. 20:48:25 So... can we have some sort of minimal text mode option in f18? 20:48:41 or...any other possible solution to the conundrum? =) 20:48:56 bconoboy: totally non-interactive... kickstart driven 20:48:59 jlk and msivak are working on it. so the answer is maybe. 20:49:21 adamw: not really. 20:49:33 bcl: per dmarlni's comment, it doesn't even need to be interactive 20:49:59 (infact we'd rather it not be) 20:50:38 lmc kickstarts everything, none of it is interactive. vlc is just there so you can babysit while it installs. 20:50:54 currently it uses --script mode which is a non-interactive text mode. 20:51:06 howdy 20:51:11 As soon as we have a text mode in f18 I'll make sure it works with kickstarts. 20:51:24 hi jlk- we're talking about text mode installs for fedora arm systems on f18 20:51:34 oh goodie. 20:52:06 looking for the f18 prognosis, schedule, etc.... so we can build test images and do alpha testing. 20:53:01 jlk: to be clear, in the context of livemedia-creator 20:53:10 jlk: is some sort of minimal mechanism on the way in the near future? 20:53:22 hold a sec, got a phone call. 20:53:24 since we can't use lmc in virt mode on ARM, we need some kind of way to use it in non-virt mode for ARM. AIUI anyhow. 20:53:49 likewise some solution for anaconda if the vnc doesn't work 20:54:44 shall we come back to this at the end? we have one more item and running low on time 20:54:58 pwhalen: there doesn't seem to have been much discussion of release criteria either =) 20:55:10 or take it to #fedora-arm ;-) 20:55:30 or arm@ 20:55:31 not much, I just posted it so perhaps next week after its been digested we can relook at it 20:55:31 sorry, almost done with the call 20:55:35 * pbrobinson needs to eat dinner as it's now 10pm here. 20:55:44 pwhalen: last topic? 20:55:49 back 20:55:53 #topic 4) Raspberry Pi Remix Update 20:55:58 The Raspberry Pi Fedora Remix is in pretty good shape, but we need to improve performance. Realisticaly we need to be on par or faster than Raspbian to be taken seriously. 20:55:58 We've got rebuilds of the remix image package set for armv6 softfp-abi-with-vfp2 and hardfp running now. Barring surprises, we will have enough built by early next week to run performance tests. 20:55:58 My preference would be v6 softfp-abi-with-vfp2 so we can interoperate with armv5tel (and not have to rebuild the universe), but which direction we go will depend on the test results -- update next meeting :-) 20:56:03 ok, so text mode in F18 20:56:25 I'm /just now/ starting to code up a screen that will actually /do/ the install 20:56:25 whether it'll work, I don't know. 20:56:30 we'll likely have something in time for F18 final, with limited functionality 20:56:52 jlk: We're looking for something to do f18 alpha install testing. What are our options? 20:56:53 adding more functionality and fine tuning it should be easy with updates.img or later builds of anaconda 20:57:09 I'm missing some context, why isn't vnc viable? 20:57:18 jlk: vnc requires virt. no virt available. 20:57:25 i think the scenarios here have gotten confused 20:57:33 jlk: ARM is very limited. 20:57:33 our devices have neither X nor virt support 20:57:38 yeah, I'm very confused, in what way does vnc require virt? 20:58:08 bconoboy: for an actual interactive installation from media created with pungi, or whatever, i think vnc ought to be viable 20:58:14 so for now you should be able to use livecd-creator in the meantime. 20:58:18 the problem with vnc is in the case of using livemedia-creator to build live images 20:58:43 adamw: vnc is just a way to watch the install. it is still virt. 20:59:19 Hey guys, can we follow up in #fedora-arm? 20:59:33 who which what? 20:59:36 our meeting time slot is up... 20:59:38 bcl/jlk 20:59:44 ah. 21:00:53 pwhalen: Sorry 'bout that... 21:00:59 any last items before we wrap? 21:01:00 np 21:01:08 djdelorie: don't we have 2 hours booked? 21:01:21 #endmeeting