21:00:17 <pwhalen> #startmeeting Fedora ARM weekly status meeting
21:00:17 <zodbot> Meeting started Wed Feb 13 21:00:17 2013 UTC.  The chair is pwhalen. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:17 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
21:00:17 <pwhalen> #chair pwhalen jonmasters bconoboy ctyler pbrobinson dgilmore
21:00:18 <zodbot> Current chairs: bconoboy ctyler dgilmore jonmasters pbrobinson pwhalen
21:00:22 <ctyler> .fas chris@tylers.info
21:00:22 <zodbot> ctyler: ctyler 'Chris Tyler' <chris@tylers.info>
21:00:22 * ctyler tips hat, sits down with Tim Horton's double-double
21:00:23 <pwhalen> good afternoon all
21:00:25 * pbrobinson waves
21:00:28 <pwhalen> .fas pwhalen
21:00:28 <zodbot> pwhalen: pwhalen 'Paul Whalen' <pwhalen@redhat.com>
21:00:28 <pbrobinson> good evening
21:00:34 <bconoboy> .fas blc@
21:00:37 <zodbot> bconoboy: blc '' <blc@redhat.com>
21:00:38 <jcapik> .fas jcapik
21:00:43 <zodbot> jcapik: jcapik 'Jaromír Cápík' <jcapik@redhat.com>
21:01:29 * nirik is around
21:01:37 <dmarlin> .fas dmarlin
21:01:37 <zodbot> dmarlin: dmarlin 'David A. Marlin' <dmarlin@redhat.com>
21:01:40 * j_dulaney waves
21:01:57 <pwhalen> #topic 0) Status of ACTION items from our previous meeting
21:02:14 <pwhalen> #info COMPLETE - bconoboy to provide pwhalen with JIT-enabled webkitgtk3
21:02:21 * j_dulaney will have to cut out early, in around 30 minutes
21:02:25 <pwhalen> #info tested webkitgtk3-1.11.5-1.fc19 on gmail, other javascript heavy sites, no crashes
21:02:49 <pbrobinson> I'll be re-enabling that soon in rawhide to enable wider testing
21:02:56 <pwhalen> sounds good, thanks
21:03:07 <pwhalen> #info INPROGRESS - jonmasters still on the hook for mongodb and an llvm update
21:03:18 <pwhalen> #info test builds of mongdb have some issues
21:03:43 <pwhalen> jonmasters, you spoke of this briefly yesterday, not sure if you had anything else to add?
21:04:04 * masta looks in
21:04:09 <pbrobinson> llvm is being updated to 3.2 and there looks to be some HF fixes in there
21:04:28 <pbrobinson> there will be a build pushed to mainline soon
21:04:57 <pbrobinson> and we've got a scratch build that at least builds on ARM and we'll test further then and it might need further polish
21:05:11 <pbrobinson> juhp is dealing with that
21:06:01 <pwhalen> #info juhp looking into llvm issues, testing sratch build
21:06:24 <pwhalen> #info COMPLETE - masta to test new ltrace package with potential arm fixes - latest build working
21:06:34 <pbrobinson> jcm still has mongo on his list.... he has to do something ;-)
21:06:45 <pwhalen> masta, I tested it out, worked okay on a quick glance
21:06:50 <bconoboy> awesome masta
21:07:13 <pwhalen> #info INPROGRESS - pwhalen & pbrobinson to continue work on broken package list
21:07:18 <masta> not all the test work, but ltrace does work
21:07:26 <masta> the build should pass
21:08:06 * masta notes the tests always pass how they are treated in spec file
21:08:18 <pwhalen> Peter and I spoke about getting this list together earlier, needs some updating
21:08:26 <pbrobinson> we'll know more for the broken package list once we've got the mass rebuild done
21:08:34 <bconoboy> or not done as the case may be
21:08:56 <pbrobinson> bconoboy: quiet in the cheap seats ;-)
21:09:00 <bconoboy> ;-)
21:09:15 <pwhalen> #info COMPLETE - pwhalen to organize wiki vfad on monday - all mentions of f17 scrubbed, some minor editing of the main page
21:09:40 <pwhalen> #info INPROGRESS - pwhalen to work on adding 'secret decoder ring' document content to the wiki
21:09:53 <masta> haha
21:09:54 <pbrobinson> that's good news
21:10:08 <pwhalen> this last item I would like some help with, I'll attempt to flesh it out a little, but could use some help as there is a fair amount there
21:10:25 <pbrobinson> can jsmith assist with that given he's been driving that?
21:10:55 <jonmasters> .fas jonmasters
21:10:55 <zodbot> jonmasters: jcm 'Jon Masters' <jonathan@jonmasters.org>
21:10:55 <bconoboy> looks like he's not in here right now
21:10:57 <jonmasters> note on mongodb: I did the atomics thing, had a couple problems with buildroots. Will fix that and post the patch
21:11:22 <pbrobinson> jonmasters: will you also work to get it upstreamed?
21:11:28 * jsmith apologizes for being late
21:11:36 <bconoboy> jsmith: <pwhalen> #info INPROGRESS - pwhalen to work on adding 'secret decoder ring' document content to the wiki
21:11:41 <bconoboy> jsmith: <pwhalen> this last item I would like some help with, I'll attempt to flesh it out a little, but could use some help as there is a fair amount there
21:12:07 <jonmasters> pbrobinson: yes, indeed
21:12:25 <jonmasters> pbrobinson: I do wonder what the other arches will want to do...
21:12:33 <jonmasters> pbrobinson: only has atomic code for x86 and arm now
21:13:05 <jsmith> bconoboy: I have requested a fedorahosted instance for the guide, and will work to convert it from wiki into an official Fedora guide, when it's ready for that
21:13:07 <jonmasters> pbrobinson: some silly glibc headers issue prevented the test package building yesterday. I'll fix it now I've got time - was in an all day meeting yesterday while I was looking
21:13:15 <masta> exclude those other arch until atomics are done?
21:14:05 <pbrobinson> other arches are their issue to deal with how they wish to, we need to focus on ARM
21:14:08 <bconoboy> jsmith: I think pwhalen is looking for help with content
21:14:25 <jsmith> bconoboy: I can help with some of that, as I learn myself :-)
21:14:27 <jonmasters> pbrobinson: sure, I was just remarking aloud. I'm planning to do nothing about anyone else :)
21:15:20 <pwhalen> well to be fair I was hoping we could all look at making some contribution. Many hands make light work
21:15:25 <pbrobinson> jonmasters: great, covered, we're on the docs stuff at the moment
21:15:29 <jonmasters> jsmith, pwhalen: I'm going to introduce you to Anne Mascarin, who will be joining #fedora-arm soon. She's a Red Hat ARM Partner Manager and I'd like the opportunity to help her figure out some of the secret decoder ring stuff. Please work with her if she appears
21:15:45 <jsmith> jonmasters: Absolutely!  The more the merrier!
21:15:52 <pwhalen> #link "Secret Decoder Ring" - http://piratepad.net/L1jH2TLCBO
21:16:43 <masta> cool!
21:17:20 <pwhalen> next?
21:17:39 <bconoboy> y
21:17:42 <pwhalen> #topic 1) Problem packages
21:18:26 <bconoboy> There's an ICE in gcc 4.8 for some packages (https://bugzilla.redhat.com/show_bug.cgi?id=910153)
21:18:43 <pbrobinson> we're not looking too bad. The gcc issue that BC mentions
21:18:46 <bconoboy> It's been fixed upstream- Jakub will be doing a rebuild pretty soon (within the next day or two) that will pull it in
21:18:55 <jonmasters> goo
21:18:57 <jonmasters> good
21:19:21 <bconoboy> pbrobinson: It seems like we're hovering between 1250-1500 missing builds in f19- is this all createrepo stuff?
21:19:37 <jonmasters> pbrobinson: on problem packages, I will be devoting time during the rest of this week to those on my list as well as 4.7 and 4.8 kernel
21:19:42 <pbrobinson> most of the failures I'm seeing is due to the repo issues.
21:19:59 <jonmasters> is there still a transient NFS build root failure?
21:20:13 <jonmasters> I might have seen that again myself in scratch builds
21:20:37 <bconoboy> jonmasters: If you suffered from dozens of packages not being available to fulfill dependencies, it's probably the same thing
21:20:40 <pbrobinson> basically we'll get a core package (say util-linux) that a lot of packages are dependent on and we'll end up where basically nothing builds so it blocks just about everything. So I end up doing a lot of manual builds and re-kicking koji-shadow and away we go for a bit
21:21:01 <jonmasters> bconoboy: indeed, that
21:21:47 <pbrobinson> but in other packages
21:21:51 <bconoboy> jonmasters: it might be a good thing to coordinate with pbrobinson on the issue you're seeing
21:21:58 <pbrobinson> we have issues again with eclipse which I'm working on
21:22:09 <jonmasters> bconoboy: I'll do that if it keeps happening
21:22:09 <pbrobinson> java has issues but that should be fixed in mainline now
21:22:16 <jonmasters> won't bug Peter unless it's persistent
21:22:19 <pbrobinson> the issue with "at" has been fixed
21:22:30 <bconoboy> #link gcc ICE bug https://bugzilla.redhat.com/show_bug.cgi?id=910153 will be resolved in next couple days
21:22:43 <bconoboy> #action pbrobinson working on eclipse issue
21:22:48 <pbrobinson> jonmasters: do feel free to ping me on an issue if I can quickly point you in the right direction
21:22:57 <jonmasters> ok
21:23:10 <bconoboy> #info "at" has been fixed (autoconf didn't know to add -lrt)
21:23:21 <pbrobinson> glibc issue has been fixed
21:23:31 <pbrobinson> as has the binutils issue
21:23:55 <bconoboy> #link glibc ld.so.cache failure is now resolved:  https://bugzilla.redhat.com/show_bug.cgi?id=905184
21:24:05 <bconoboy> what's the binutils issue?
21:24:18 <pbrobinson> there's a few other issues but I believe they're transient and need to retest to work out the exact state
21:25:11 <pbrobinson> #link binutils issue fixed https://bugzilla.redhat.com/show_bug.cgi?id=892261
21:25:31 <bconoboy> cool.
21:25:45 <bconoboy> any other problem packages?
21:26:02 <bconoboy> I think we're largely waiting on the move to start aggressively tackling the missing package count
21:26:10 <pbrobinson> oh and firefox 18 is fixed and should be done on all releases RSN :)
21:26:26 * pbrobinson has been busier than I remember :)
21:26:39 <pbrobinson> yes, correct
21:26:42 <bconoboy> #info firefox 18 fixed
21:27:00 <bconoboy> pwhalen: better move on.  pbrobinson has fixed hundreds of packages ;-)
21:27:15 <pwhalen> #topic 2) Kernel updates in F17, F18 causing issues
21:27:50 <pbrobinson> the point I would like to make strongly here is that I need decent bug reports. All I've really had to date is "it's broken"
21:28:03 <pwhalen> we've had a number of people report issues with 3.7 kernels in f17
21:28:13 <masta> I'd say two things the audit_syscall and the dtb, what else?
21:28:14 <bconoboy> pbrobinson: Are you looking for BZ-class reports?
21:28:30 <pbrobinson> I'm quite happy for the reports to be to the mailing list
21:28:35 <pwhalen> pbrobinson, I've pointed you to fpaste's each time..
21:28:44 <pbrobinson> but at the moment most of the reports are it's broken
21:28:54 <bconoboy> #info If a kernel update fails to boot, please file a BZ or minimally mail arm@lists.fedoraproject.org
21:29:01 <pbrobinson> pwhalen: I've still not got one for tegra I believe
21:29:20 <pbrobinson> and if it could be to the mailing list where I can tag them to follow up
21:30:06 <pwhalen> pbrobinson, sure
21:30:41 <bconoboy> pbrobinson: I think some of us wonder if 3.7 should even make it down to 3.6, considering for instance that we didn't have enablers for DTBs in f17 on trimslice or vexpress
21:31:07 <pbrobinson> bconoboy: what do you mean 3.7 make it down to 3.6?
21:31:14 <bconoboy> s/3.6/f17/
21:31:31 <pbrobinson> well people have the option to exclude it, we need to keep up with mainline
21:31:52 <bconoboy> is there any way we can hold it back?
21:31:56 <pbrobinson> if we don't it screws up koji-shadow and god knows what
21:32:18 <bconoboy> we could not tag it?
21:32:21 <pbrobinson> and people tend to love to update to the latest shiny and then bitch when it doesn't work
21:32:55 <pwhalen> dgilmore runs a sync script that will retag it iiuc
21:33:06 <pbrobinson> and I thought we'd been testing the 3.7 kernels. It would be useful to know where it broke on what arches
21:33:40 <pbrobinson> eg we know from 3.7.5 -> 3.7.6 there's a kirkwood breakage from the email today
21:33:42 <bconoboy> pwhalen: does your kernel tracking page include this?
21:33:42 <jonmasters> on the kernel front, I am going to work with pwhalen - he and I planned that yesterday
21:33:56 <bconoboy> pwhalen: (by this I mean updates to f17 kernel)
21:33:59 <jonmasters> we'll test upgrading some F17 systems (well he will) and I'll dig out whatever is broken
21:34:02 <pbrobinson> jonmasters: please include me in that
21:34:06 <jonmasters> pbrobinson: ok
21:34:16 <pwhalen> I've not been looking at F17 kernels for some time
21:34:23 <jonmasters> pwhalen: feel free to start the test upgrades on F17 and get the data
21:34:32 <pwhalen> jonmasters, will do
21:34:43 <jonmasters> pwhalen: yea, understood, but let's get that going. You make a matrix, I'll poke 3.8 while you do that, then we'll look at the data on 3.7
21:34:48 <pbrobinson> pwhalen: I suspect we'll see very similar issues on f18 because the configs for the kernel between the two are basically the same
21:34:52 <jonmasters> (both for F17 and F18)
21:35:00 <jonmasters> actually, maybe I should look at 3.7 on F18 first?
21:35:04 <pbrobinson> yep
21:35:09 <pbrobinson> that's a good start
21:35:10 <bconoboy> #action pwhalen, jonmasters, pbrobinson to work on kernel update compatibility coverage
21:35:11 <jonmasters> pbrobinson: which is higher priority? 3.7 on F18, right?
21:35:30 <jonmasters> pbrobinson: then 3.8 while pwhalen continues compiling F17 data, then 3.7 on F17?
21:35:38 <jonmasters> pbrobinson: feel free to suggest a priority
21:35:39 <dmarlin> jonmasters: would you even expect the 3.7 kernel to work on ARM F17, with stock u-boot, etc. ?
21:35:46 <pbrobinson> I know there's an issue with pandaES and a 3.7 kernel but as I've basically not been home for 3 weeks it makes it hard for me to test all the devices I have
21:36:04 <dmarlin> jonmasters: when it doesn't on F18
21:36:08 <jonmasters> pbrobinson: indeed. I have a PandaBoard ES that was ok with 3.7 on F18 but you mean F17
21:36:11 <pbrobinson> yes, lets focus f18 3.7 first
21:36:20 <jonmasters> pbrobinson: ?
21:36:21 <pbrobinson> as I believe we'll see the same problems on f17
21:36:52 <pbrobinson> jonmasters: I don't have any f17 any more on x86 or ARM :)
21:36:54 <jonmasters> first thing is I want to go over the data we have and see what exactly is actually broken, where :) that's my first priority with each of these issues
21:37:07 <jonmasters> pbrobinson: ok, I'll start going over what we have - will keep you in the loop
21:37:39 <jonmasters> dmarlin: I expect F17 3.7 updates to work, yea. I know there are a couple of known issues, but it's not all broken
21:37:44 <pbrobinson> so I think if we test the latest kernel on each of the devices on F-18, see what breaks between what NVRs
21:37:50 <jonmasters> yep
21:37:53 <bconoboy> jonmasters: even on trimslice and versatile express?
21:38:06 <dmarlin> jonmasters: and panda
21:38:11 <pbrobinson> once we have that data I think we can do the same this works/ then this breaks on F17 and get the same results
21:38:11 <dmarlin> jonmasters: and beagle
21:38:38 <bconoboy> pwhalen: have a link to your kernel test page handy?
21:38:43 <pbrobinson> dmarlin: I have a local test I think fixes beagle, I just need to get some face time with my device to test
21:38:48 <pwhalen> http://fedoraproject.org/wiki/Architectures/ARM/Quality_Assurance/Kernel_Testing
21:38:57 <jonmasters> bconoboy: let's get data. People keep saying "it's broken". But I'm not sure I like that answer. There is an issue with dtb on vexpress, and there is a dtb issue on tegra, but it's not all broken
21:39:02 <bconoboy> #link Current kernel test information is at http://fedoraproject.org/wiki/Architectures/ARM/Quality_Assurance/Kernel_Testing
21:39:10 <jonmasters> bconoboy: if F17 3.7 is otherwise different than F18 then we need to sync configs
21:39:35 <pbrobinson> pwhalen: can we clear out old data on that page please, we really don't need 3.5/3.6 any more as it's becoming very hard to read and cross reference
21:39:36 <bconoboy> jonmasters: For F18 we have data per the above link.  For F17 I'm concerned about trimslice and vexpress particularly because there is no DTB handling in F17.
21:40:04 <dmarlin> pbrobinson: 3.6 is the only known working version for many boards right now
21:40:25 <pwhalen> pbrobinson, sure I can move it to an archive
21:40:27 <dmarlin> pbrobinson: and if we are still building them
21:40:47 <pbrobinson> dmarlin: that's fine, keep the latest version that shipped with F-18 gold and clear the rest out, we don't need 3.6rc release for example
21:40:57 <dmarlin> pbrobinson: agreed
21:41:10 <pbrobinson> bconoboy: jonmasters: we have dtb files in the 3.7 F-17 kernels post abou 3.7.5
21:41:20 <pbrobinson> bconoboy: jonmasters: what else do we need to deal with that?
21:41:29 <jonmasters> bconoboy: I understand your concern. I'm a little perplexed by "no DTB handling". For vexpress, there is no automatic kernel upgrade path anyway and it's simple enough to address. Tegra is an issue
21:41:35 <pbrobinson> bconoboy: jonmasters: should basically just do the same as F-18 shouldn't we?
21:42:03 <bconoboy> pbrobinson: That may be, but boot-vexpress in f17 doesn't know what to do with them, nor does the boot.scr on trimslice in f17 know to load the dtb.  We at least need to document what users need to do differently.
21:42:17 <jonmasters> we need data. Rather than debate at length, Paul will compile updated matrices and I will look at the problem stuff for 3.7 and 3.8 on F17 and F18
21:42:35 <pwhalen> sounds good
21:42:54 <pbrobinson> jonmasters: bconoboy: there is no difference that I'm aware of between the 3.7 kernels in F-17 and F-18 that's why I was saying they should be the same for testing. Only difference would be the differences on mainline changes
21:42:58 <jonmasters> bconoboy: understood. I need to followup on the vexpress dtb stuff anyway as it's just plain busted (because everyone is focused on the F15 shiny)
21:43:04 <jonmasters> pbrobinson: +1
21:43:27 <jonmasters> pbrobinson: yea, it's just bootloader, etc. that might differ. Otherwise the configs ought to the equivalent and the kernels basically the same
21:43:32 <jonmasters> but hey, we'll look
21:43:46 <pbrobinson> jonmasters: F15 shiny? You mean A15 shiny?
21:44:23 * ctyler wondered too
21:44:24 <jonmasters> did I write F15? Sorry
21:44:28 <jonmasters> I meant A15
21:44:29 <pwhalen> last kernel issue, we have not yet sync'd out the fixed kernel to F18-updates (3.7 which includes the pkexec fix)
21:44:34 <jonmasters> I literally thought I typed A15!
21:44:35 <jonmasters> :)
21:44:46 <bconoboy> such is the power of Fedora
21:44:46 <pbrobinson> pwhalen: yes we have
21:45:02 <jonmasters> vexpress now has three memory models. It has an A15 model now as well that is distracting everyone who likes shiny balls
21:45:04 <pwhalen> its not here - http://dl.fedoraproject.org/pub/fedora-secondary/updates/18/armhfp/
21:45:09 <pbrobinson> pwhalen: the 3.7.6 kernel that was being reported as broken has the fix
21:45:36 <pbrobinson> pwhalen: Isn't it http://dl.fedoraproject.org/pub/fedora-secondary/updates/18/armhfp/kernel-3.7.6-201.fc18.armv7hl.rpm ?
21:45:38 <jonmasters> so I doubt anyone is really poking at the older A9 model (which is not as old as a the "old old" A9). You know how much I look forward to UEFI/ACPI? So so so much :)
21:45:56 <pwhalen> pbrobinson, no its kernel-3.7.6-202.fc18
21:46:12 <pwhalen> 201 got built after, therefor was listed as the latest package
21:46:31 * ctyler listening to ALS talk about ACPI etc in LCA13 LEG promo hangout
21:46:45 <pbrobinson> ctyler: completely unrelated to the meeting
21:47:45 <jonmasters> ok, so maybe that's enough on the kernel. Paul and I have it
21:48:00 * pbrobinson feels left ou
21:48:07 <jonmasters> pwhalen: if you could for each target test with/without dtb explicitly and record status please tell us
21:48:14 <jonmasters> pbrobinson: well you're implicit :)
21:48:32 <pwhalen> pwhalen, will do
21:48:34 <pbrobinson> :)
21:49:00 <dmarlin> I'm sure pwhalen wouldn't mind if pbrobinson wanted to split the task with him
21:49:11 <dmarlin> so many boards, so little time
21:49:53 <pbrobinson> dmarlin: I actually have a weekend at home and maybe, shockingly, a week where I can actually do some testing on different boards rather than the one I put in my case for the week
21:50:06 <dmarlin> cool
21:50:21 <pwhalen> next then?
21:50:50 <pbrobinson> pwhalen: you have the agenda :-P
21:51:38 <pwhalen> well, we all do
21:51:41 <pwhalen> #topic 3) Mass rebuild - Adding additional HSV builders, status update
21:51:47 <bconoboy> nirik: you here?
21:51:51 * nirik can update everyone on phx2 stuff.
21:51:52 <nirik> yeah. ;)
21:51:59 <bconoboy> dgilmore: you here?
21:52:05 <bconoboy> nirik: by all means :-)
21:52:10 <nirik> so, we have new chassis's installed, racked. I have power and serial on them all.
21:52:17 <nirik> I have mac addresses from all the SOCs
21:52:23 <nirik> we are lacking IP address space sadly.
21:52:45 <nirik> However, I think I can get us 48 ips in the net they are currently on possibly.
21:52:56 <nirik> would getting 24 of them up now be enough ?
21:53:05 <pbrobinson> 24 would be a great start
21:53:18 <nirik> then we could add more when we get ip space.
21:53:26 <jonmasters> 24 would rock
21:53:30 <pbrobinson> the more the merrier but if it's 24 to begin with we can cope
21:53:31 <nirik> which I keep thinking will happen any minute and never does.
21:53:40 <bconoboy> nirik: Yeha, I think that would be enough to start.  Maybe 22 hfp, 2 sfp?
21:53:53 <nirik> whatever ratio you folks want.
21:53:57 <pbrobinson> or 20 hfp and 4 sfp
21:53:59 * jonmasters thinks more sfp than that
21:54:03 <jonmasters> yea, +1 peter
21:54:09 <bconoboy> sure, 20/4
21:54:19 <jonmasters> nirik: we are also keen to spread over the physical boxes
21:54:30 <jonmasters> nirik: so perhaps don't just assign all IPs to one box
21:54:42 <pbrobinson> ultimately sfp is just f18/f17 updates so if they're a bit slower and have to wait for builders for a week or two so be it
21:54:44 <jonmasters> nirik: if you could spread across the chassis, it would increase resiliancy
21:54:44 <nirik> ah, ok. Any reasons?
21:54:48 <nirik> true.
21:54:52 <nirik> sure, easy enough to do
21:55:07 <jonmasters> nirik: thanks. We don't expect trouble, but you know, not all the eggs in one basket, etc.
21:55:12 <bconoboy> nirik: Will these be temporary IPs?
21:55:16 <pbrobinson> nirik: when do you think we'll get the rest of the IP space?
21:55:34 <nirik> bconoboy: yeah... and then we will move them to new space when we have it... hopefully that won't be a problem?
21:55:38 <jonmasters> nirik: so far, worst we have had is one energycore on one card going bad, out of many cards in a chassis, so not exactly a big deal, but you know...
21:55:42 <pbrobinson> ultimately if its a week or so I don't care about spreading and if we have one chassis for a week or two I think we could cope
21:55:45 <nirik> pbrobinson: I keep bugging network folks, but no idea.
21:56:08 <bconoboy> nirik: not really a problem- normally I'd advocate spreading across multiple boxes but it seems better to have the hodgepodge consolidated on one console
21:56:14 <bconoboy> er, one chassis
21:56:21 <jonmasters> Out of interest, do we have plans to migrate to IPv6 for that side of things?
21:56:23 <bconoboy> doesn't really matter though- either is fine
21:56:30 <pbrobinson> and if we have it on the one chassis it means nirik can move the other chassis as needed and then we can flip and he can re-ip the first chassis
21:56:37 <jonmasters> Or do we have enough v4 space in Fedora that it's not an issue
21:56:38 <jonmasters> ?
21:56:52 <bconoboy> pbrobinson: exactly
21:57:10 <nirik> pbrobinson: sure. true
21:57:11 * fossjon wonders how many ips x86 uses
21:57:14 <pbrobinson> jonmasters: please can we deal with something like that later.... I just want it working and don't needed extra complications
21:57:14 <jonmasters> I expect we'll be adding another hundred builders this time next year
21:57:18 <nirik> jonmasters: I know of no ipv6 plans there.
21:57:20 <fossjon> are we way above them?
21:57:22 <jonmasters> pbrobinson: sure, np
21:57:32 <bconoboy> nirik: Okay, so bottom line is we can get 24 systems up and running. That is awesome.  Timeframe?
21:57:36 <jonmasters> pbrobinson: sorry, I didn't mean to do the IPv6 discussion...thanks for steering this back on track
21:57:51 <nirik> these are private networks to be clear... I just can't make up one tho, since it may well be used somewhere else internally, causing untold horror
21:57:57 <pbrobinson> jonmasters: we just need focus :)
21:58:03 <jonmasters> indeed so
21:58:07 <bconoboy> #info Nirik has scavenged enough IPs to bring up 24 build systems
21:58:09 <nirik> bconoboy: I am scavenging ip's from the qa people as we speak. ;)
21:58:19 <nirik> hopefully I can get them installed later today
21:58:31 <bconoboy> Okay, so if we have 24 builders in PHX I don't think we need to explore the HSV builder alternative
21:58:34 <ctyler> How much downtime do we need for final sync and db transfer?
21:58:34 <pbrobinson> nirik: You are a champ!
21:58:35 <jonmasters> ok, so nirik is bringing up 24, he'll make an executive decision on timing with respect to spread and duration of IPs and we'll have 24 up soon
21:58:54 <bconoboy> #info Builders may be provisioned as early as later today
21:59:06 <jonmasters> nirik: please ping us if you need any assistance, I know you will
21:59:10 <bconoboy> Earlier today dgilmore said that migrating koji would take about 48 hours
21:59:16 <pbrobinson> dgilmore estimates two days but I think it might be less as we'll have koji offline and give it the full bandwidth and IOPS possible
21:59:27 <nirik> sure, I will. Also, feel free to ping me anytime for status.
21:59:46 <bconoboy> pbrobinson: Is there anything critical that we can't wait 48 hours on getting built?
21:59:52 <pbrobinson> so it won't be competing but the main problem appears to be the raw bandwidth between PHX and Seneca
22:00:26 <pbrobinson> bconoboy: hell no, I just want it moved now (as I'm sure ctyler does) so everything can wait.
22:00:53 <nirik> it might be faster too if it's not active.
22:00:54 <ctyler> I think we've been syncing daily, but there's been a lot of output lately because of the rebuild
22:01:13 <jonmasters> yea, we don't need anything /that/ badly. Scratch builds etc. can always be done in mock on local systems if we are working on something (like mongodb!)
22:01:17 <pbrobinson> bconoboy: ultimately it's a bit like the RAID rebuild, we take koji offline to get it done quicker and just take the hit and move on
22:01:17 <bconoboy> #info Koji migration will take up to 48 hours to complete.  Now is a good time to start.
22:01:25 <ctyler> Should we disable the builders here now and let them finish their current jobs?
22:01:38 <ctyler> s/here//
22:01:40 * jonmasters suggests doing when nirik is read
22:01:42 <jonmasters> ready
22:01:44 <bconoboy> I think we need dgilmore buy-in here since he's doing the work
22:01:58 <pbrobinson> ctyler: there has been no rebuild on ARM yet because it only kicked off today on mainline on a separate tag so we've not even started on ARM yet
22:02:02 <jonmasters> there might be a delay, need Dennis, etc. Just disable when they're ready. We can always have builds restart later
22:02:22 <pbrobinson> dgilmore nirik and I can coordinate with the koji stuff
22:02:24 <ctyler> Ok, there's been a lot of non-rebuild builds then!
22:02:49 <bconoboy> #action DGilmore, nirik, and pbrobinson to coordinate koji move (Hopefully later today)
22:03:18 <nirik> sorry for all the delays everyone. ;(
22:04:07 <jonmasters> nirik: oh you're a trooper
22:04:12 <pbrobinson> nirik: no issues, you can buy the first round of beers next time we meet up, then I'll buy the next 3 because of the work you've done to make it happen :)
22:04:20 <nirik> ha.
22:04:25 <pwhalen> we're in ot, anything else for the rebuild/move?
22:04:26 <jonmasters> and I'll buy a few rounds too!
22:04:33 <jonmasters> and then nirik will be on the floor reeling :)
22:04:57 <pwhalen> #topic 4) Open Floor
22:05:42 <pwhalen> any other topics to be discussed?
22:06:22 <pwhalen> thanks all!
22:06:24 <pwhalen> #endmeeting