16:01:25 <adamw> #startmeeting Fedora QA Meeting
16:01:25 <zodbot> Meeting started Mon Nov  7 16:01:25 2011 UTC.  The chair is adamw. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:25 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:01:28 <adamw> #meetingname fedora-qa
16:01:28 <zodbot> The meeting name has been set to 'fedora-qa'
16:01:52 * jskladan tips his hat
16:01:52 <adamw> #chair j_dulaney
16:01:52 <zodbot> Current chairs: adamw j_dulaney
16:01:56 <adamw> morning folks, who's around?
16:02:01 <adamw> #topic roll call
16:02:02 * tflink is present
16:02:04 * Cerlyn is here
16:02:06 * brunowolff is
16:02:07 * pschindl is here
16:02:10 * kparal is not here
16:02:17 * j_dulaney is here
16:02:22 <jskladan> kparal: still away? :)
16:02:38 * kparal is partying with hot chicks in Carribean
16:02:48 <kparal> or whatever is the spelling
16:02:49 <j_dulaney> It would seem that brunowolff just _is_
16:03:10 <adamw> bruno, indisputably, is.
16:03:17 <tflink> kparal: and you're still on IRC? /me wonders about you
16:03:42 <kparal> oh yes, fedora qa meetings are my weakness
16:03:48 * j_dulaney wondered about kparal already
16:04:38 * adamw feels bad for the hot chicks
16:05:25 <adamw> #topic Fedora 16 release planning
16:05:33 * kparal converting all hot chicks to using Fedora
16:05:39 <adamw> alright, this is just a kinda general f16 topic
16:05:43 <adamw> make sure we tie up any loose ends
16:05:59 <adamw> thanks again to everyone for their great work on validation, of course
16:06:39 <adamw> we do have the livecd-tools updates to get out for f14, f15 and f16: https://lists.fedoraproject.org/pipermail/test-announce/2011-November/000340.html
16:06:56 <adamw> i also need to get with hughsie about fixing f14's preupgrade
16:07:02 <adamw> and we have common bugs to do
16:07:13 <adamw> can anyone think of anything else?
16:07:22 * j_dulaney will test the livecd-tools for F15
16:07:24 <tflink> commonbugs?
16:08:21 <kparal> already mentioned
16:08:41 <tflink> huh, I missed it
16:08:43 <tflink> oh well
16:08:58 <adamw> you're fired!
16:09:02 <kparal> :D
16:09:05 <adamw> man, you're starting early this week
16:09:39 <brunowolff> I was on the phone at work.
16:09:43 <tflink> adamw: I see the weekly firings are going to continue past F16 going gold
16:09:59 <Cerlyn> But with all the layoffs, who is going to test Fedora?
16:10:00 <adamw> weekly? you're _lucky_ if it's only weekly.
16:10:07 <adamw> Cerlyn: you are!
16:10:10 <kparal> adamw, of course
16:10:18 <kparal> no sleep at all
16:10:21 <adamw> hehe
16:10:25 <tflink> adamw: well, there was a 2 week span where I didn't get fired
16:10:30 <j_dulaney> adamw:  I'm sure I can get rbergeron or jsmith to have you fired
16:10:33 <adamw> man, i must've been slacking
16:10:45 <adamw> anyhoo
16:11:08 <adamw> the commonbugs list is pretty long: https://bit.ly/fedora-commonbugs-proposed
16:11:17 <adamw> so if people can pitch in and help clear it that'd be great
16:11:48 <adamw> other than that  i guess we're good on this topic, yay - finally f16 is more or less behind us
16:12:04 <adamw> oh, missed one thing - the retrospective
16:12:49 <adamw> the retrospective is at https://fedoraproject.org/wiki/Fedora_16_QA_Retrospective : just as a reminder, it's where you can put thoughts about the f16 cycle, stuff that went well and stuff that didn't go so well, ideas for how to do things better next time
16:13:03 <adamw> some time soon i'll go through and pull out action items from it and turn them into trac tickets, and summarize it for the list
16:13:26 <adamw> so if you have any thoughts about how to make life less stressful next time, add them on there!
16:14:16 <adamw> alrighty, moving on...
16:14:21 <adamw> #topic proven tester process
16:15:11 <adamw> so this one is interesting. at their meeting last week, fesco more or less decided they want to abandon the proven testers process
16:15:17 <adamw> you can see the meeting log at http://meetbot.fedoraproject.org/fedora-meeting/2011-10-31/fedora-qa.2011-10-31-15.03.log.html
16:16:14 <adamw> mjg notified devel list, too: https://lists.fedoraproject.org/pipermail/devel/2011-November/158881.html
16:16:31 <adamw> they did not actually directly notify QA in any way, but never mind.
16:16:46 * nirik would like to interject. ;)
16:16:50 <adamw> interject away!
16:16:54 <nirik> we didn't decide anything. ;)
16:16:56 <red_alert> that first link goes to a QA meeting log, not to fesco :)
16:17:05 <adamw> sigh, my link skills suck
16:17:06 <adamw> one more try
16:17:12 <nirik> we wanted to wait a week for feedback from all groups... including qa. ;)
16:17:16 <adamw> http://meetbot.fedoraproject.org/fedora-meeting/2011-10-31/fesco.2011-10-31-17.00.log.html
16:17:46 <nirik> The thought was that proventesters were not that helpfull, and we should continue critpath and all just with no proventesters... s/proventester/logged in karma submitter/
16:17:49 <adamw> nirik: ah, okay. of course, it helps to get feedback from qa if you actually tell qa. =)
16:17:58 <kparal> it means proventester group would be cancelled with no replacement, do I read that correctly?
16:18:00 <nirik> sorry... we've all been busy. ;(
16:18:33 <nirik> kparal: well, replaced by 'anyone with a fas account' I guess...
16:18:49 <kparal> that means no replacement, same process as for all the other packages
16:18:55 <adamw> kparal: as nirik says, we'd essentially keep the current process except proventesters wouldn't exist and we'd just accept karma from any logged-in user as if it were pt karma as far as the current scoring goes
16:19:07 <adamw> kparal: critpath would still need higher total karma to be approved - 2 vs 1
16:19:24 <kparal> do we accept karma from anonymous users for non-critpath?
16:19:29 <kparal> I mean no FAS
16:19:31 <nirik> The thought was that proventesters haven't really made that much difference, so it seems like overhead to have to manage it and try and get people to become them.
16:19:46 <nirik> kparal: nope. anon still doesn't count.
16:19:56 <kparal> ok
16:19:58 <adamw> kparal: no, we don't use the numeric karma from anon users for any package (critpath or non-critpath)
16:20:14 <Cerlyn> Is there any proof one way or another that proventesters actually test things in a proven way that justifies trying to train people?
16:20:25 <adamw> nirik: my only worry is as posted to the list: two looks like a very small number but it is not zero, and this is a game where the difference between two and zero could be a substantial one
16:20:27 <nirik> so, I don't think this is something we need to do instantly... if everyone would like more time to look at stats and mull it over thats cool with me. ;)
16:20:37 <red_alert> proventesters are not specially trained :)
16:20:41 <j_dulaney> Cerlyn:  I try to do so
16:20:47 <kparal> Cerlyn: they confirmed they read the guidelines. which might be more than ordinary testers
16:20:48 <nirik> Cerlyn: the devel list post from mjg59 had some stats...
16:21:35 <kparal> #link http://lists.fedoraproject.org/pipermail/test/2011-October/104084.html
16:21:50 <adamw> there's also a (rather informal) process  whereby people who submit updates and get dumb feedback from proventesters can come and complain to us about it, and the pt gets educated. that's happened a few times.
16:22:10 <nirik> true.
16:22:25 <adamw> nirik: i also think there have been cases where we _could have used_ proventester karma to prevent some updates going out
16:22:25 <nirik> although we could also educate 'normal' testers. ;)
16:22:31 <adamw> but didn't, because negative karma is currently handled poorly
16:22:42 <nirik> yeah, could be...
16:23:21 <j_dulaney> adamw:  negatice karma could certainly be improved
16:23:34 <j_dulaney> Even if we keep proventestor
16:23:39 <j_dulaney> +s
16:23:44 <adamw> nirik: there've been a couple of cases where updates have gone out in cases like '+1 proventester, +1 regular user, +1 regular user, -1 proventester, SUBMITTED TO STABLE!, -1 proven tester, -1 proven tester, -1 regular user, PUSHED TO STABLE!'
16:24:03 <adamw> j_dulaney: yeah, to a degree they're separate issues, but if we had better handling of negative karma it may have changed the pt stats.
16:24:19 <red_alert> crit path packages actually should have a thorough test guide which proventesters can follow - right now they are just normal testers that *try* to take things a bit more serious but often lack the detail knowledge nethertheless
16:24:23 <adamw> there was a proposal, for e.g., to disallow any update with negative karma from a proven tester from going through
16:24:43 <adamw> red_alert: yeah, we've been working on the test guide side of things for a while, but it's one of the victims of limited resources
16:25:26 <j_dulaney> adamw:  That would be a good way to handle things
16:25:27 <adamw> going back to negative karma - and there's also the possibility of blocking any update which is submitted after passing the criteria but, before being pushed, drops to failing the criteria
16:25:55 <j_dulaney> Indeed
16:25:55 <adamw> so i think it might be worth considering ways of improving the handling of negative karma before deciding to dump the pt concept. especially since it's something that would be hard to get back
16:26:24 <adamw> if we ever cancel it we need to be damn sure it's okay to cancel it, because if we cancel proventesters then six months later put it back in, people might feel like they were being jerked around.
16:26:37 <nirik> yeah, not sure how much can be done in the current 1.x bodhi framework, but perhaps.
16:26:48 <tflink> I think that we're missing part of the why this was proposed, though
16:26:52 <red_alert> the process as its done right now doesn't work (i.e. is no improvement to not having pts) but having pts with a better process would surely add to the overall quality
16:26:53 <nirik> sure, I don't think we want to be hasty...
16:27:18 <tflink> ie maintainers getting frustrated with their packages stuck in testing due to lack of proventester action
16:27:23 <nirik> even though proventesters is trival to join, it's a hoop... so many people just don't bother.
16:27:42 <adamw> tflink: yeah, it's important to consider that
16:27:52 <Cerlyn> tflink: I think its more a lack of karma for packages in general, although I haven't seen the stats
16:28:13 <nirik> well, it's mostly the critpath ones...
16:28:18 <adamw> Cerlyn: nah, the proposals to 'revise' the proventester process always come from people who are frustrated that their updates take a long time to clear it
16:28:21 <nirik> because non critpath maintainers can push after a while.
16:28:38 <j_dulaney> tflink:  I know that during the latter half of the release process, I tend to focus more on the new release and less on testing updates
16:29:13 <tflink> so if we object to getting rid of the proventester process, do we have any solutions to the root of their complaints?
16:29:16 <j_dulaney> For instance, right now I don't have a box with F15
16:29:29 <j_dulaney> Although I'll be reinstalling it here shortly on my test box
16:29:33 <tflink> I'm guessing here, but I assume that there wouldn't be so much of a push if the delays weren't there
16:29:43 <adamw> tflink: fair point
16:29:46 <nirik> well, to some extent, the 'allowing to push after 2 weeks with no - karma' would help
16:30:07 <adamw> fesco was considering that for a while, and our position was basically 'we'll get back to you after f16', aside from the pt meetings nirik's been running
16:30:32 <j_dulaney> Which I haven't been able to make due to class at that time
16:30:48 <nirik> (which haven't really been all that well attended. ;) We did get a few things moving, but not much...
16:30:52 <brunowolff> We could still keep the proven tester group for a while even if we relax the karma requirements.
16:31:24 <tflink> nirik: the timing wasn't the greatest with the crazy that F16 release brought
16:31:33 <nirik> true.
16:31:39 <nirik> brunowolff: yep.
16:32:15 <adamw> so i guess we don't have an entirely concrete response to the proposal - we have concerns but recognize the reasons people don't like the process
16:32:38 <brunowolff> Maybe down the road we will count proven tester votes more strongly, but still enough enough normal karma to OK an update.
16:32:49 <adamw> oh, one other thought on the stats
16:32:57 <nirik> perhaps everyone could air their concerns on the devel list thread...
16:33:05 <nirik> and we can see where we end up next week?
16:33:13 <adamw> i assume the stats made the assumption that the proventester feedback on any update would still have been present but treated it no different from regular feedback
16:33:38 <adamw> so, that's not necessarily a safe assumption: proventesters may feel a stronger 'responsibility to test', and if you cancel the process, they might stop doing so
16:33:38 <nirik> yeah, although you would have to ask mjg59 for sure.
16:33:41 <adamw> but that's hard to gauge.
16:33:45 <adamw> nirik: sounds good
16:33:51 <nirik> yeah, very hard to quantify
16:35:00 <j_dulaney> I know that until about Beta of a release, I make it part of my daily ritual to pull from updates testing and put everything through its paces
16:35:15 <j_dulaney> I know not how many others do that.
16:35:30 <adamw> i think there were some stats on total pt feedback from luke recently too
16:35:36 * j_dulaney does not have everything that is in critpath installed, however
16:35:38 <red_alert> I would propose to have a QA/proventester meeting during FUDCon Blacksburg and try to come up with a beter pt-process there
16:35:55 <adamw> it would certainly be a nice thing to do at a fudcon if enough interested parties were there, for sure
16:35:59 <j_dulaney> red_alert:  Not a bad idea
16:35:59 <adamw> if people would wait that long
16:36:09 <nirik> yeah, it's a while from now... but possibly
16:36:28 <red_alert> getting rid of the pt-process doesn't sound very time-sensitive, though
16:36:45 <mjg59> It's not. But satisfying the concerns of maintainers is important.
16:37:17 <mjg59> The risk isn't so much individual packages. If people feel that the critpath process is unreasonable then the risk is that there's a backlash against the entire testing process.
16:37:19 <red_alert> maybe deactivate the process in the meantime, then?
16:37:22 <adamw> mjg59: as noted, the two week delay might sate the baying hordes for now =)
16:37:28 <Cerlyn> Is there a need for this to be voted upon at the fesco meeting following this one?
16:37:43 <adamw> mjg59: er, the two-week timeout thing for critpath
16:37:50 <mjg59> adamw: The two week delay is an admission of absolute failure
16:38:07 <mjg59> The process needs to work without that
16:38:30 <adamw> i didn't say otherwise, i said it might avoid a major backlash, which is a different question.
16:38:49 <red_alert> add the two week delay admission until we had the chance to fix the process, then? because right now it seems to be total failure until fixed :)
16:38:56 <adamw> i don't really want to re-open the whole debate about whether it's a good thing in this meeting, we have other topics to cov.er
16:39:11 <mjg59> Looking through the stats there are actually more cases of proventesters inappropriately blocking a package than there are of the pt process preventing bugs getting through
16:39:42 <mjg59> Which isn't to slight any of the individual proventesters involved
16:40:02 <mjg59> It just means it's questionable as to whether the requirement makes an overall positive difference to the quality of our updates
16:40:11 <tflink> mjg59: by inappropriately block, do you mean by not providing karma or providing negative karma where it wasn't warranted?
16:40:11 <adamw> yes, we already covered that at the start of the topic, 20 minutes ago
16:40:25 <adamw> are we going to get anywhere new, here, or should we move on?
16:40:38 <mjg59> tflink: Providing negative karma due to either an unrelated issue or a misunderstanding of the package
16:41:08 <j_dulaney> adamw:  I reckon moving on might be a good thing
16:42:00 <adamw> moving along then - we clearly haven't reached a complete conclusion on this topic yet, and we're not going to do it in the next five minutes
16:42:17 <adamw> as nirik suggested, please everyone with concerns, post your thoughts to the devel list thread, and fesco will consider them
16:42:52 <adamw> #agreed group has various perspectives on the proven tester issue, we will respond to the devel list thread and follow developments with fesco
16:43:14 <adamw> #topic fedora 17 pre-planning: anaconda GUI rewrite
16:43:25 <adamw> so, you know what the end of the f16 cycle means - time to start worrying about f17!
16:43:33 <tflink> yay!
16:43:41 * kparal expected some pause
16:43:46 <kparal> trip to hawaii, etc
16:43:54 <kparal> nevermind
16:44:08 <adamw> kparal: i thought you were there already?
16:44:48 <red_alert> finally! I've been waiting for the f17 cycle ever since the last readiness meeting's end ;D
16:44:49 <kparal> for the rest of you
16:44:57 <adamw> for anyone who doesn't know, one of the major features planned for fedora 17 is a complete re-design of the anaconda GUI - not just a cosmetic change to how it looks, but the actual workflow is to be completely changed
16:45:01 <adamw> red_alert: :)
16:45:18 <kparal> adamw: how realistic is that is will be done in f17?
16:45:22 <kk4ewt> adamw, *crossing fingers*
16:45:24 <kparal> it seems like a really big task
16:45:38 <red_alert> kparal: depends on the number of slips we accept? ;)
16:45:40 <adamw> kparal: very, they're already coding it.
16:45:44 <adamw> red_alert: correct answer ;)
16:45:52 <kparal> ok
16:46:06 <j_dulaney> My wondering is how soon we can start testing
16:46:10 <adamw> so, obviously we want to try and avoid f16-style crazy stress scenarios here, as far as possible
16:46:41 * j_dulaney is thinking that as soon as it starts hitting Rawhide nightlies, testing will begin for him
16:46:52 <adamw> i've thought for a while we should work directly with the anaconda team to get involved through the whole process and get testing done as early and often as possible
16:47:14 <tflink> +1
16:47:17 <adamw> i'm happy to work on that, or if anyone else would like to, feel free :)
16:47:46 * tflink wonders if it would be better to start with one point of contact
16:48:04 <adamw> yeah, i did say 'or'
16:48:16 <tflink> I can't imagine that it'll help if multiple people start asking them about their testing plans
16:48:31 <red_alert> don't we just need to know which specific nightlies they consider worth of testing and then report bugs, that's it? :)
16:48:32 <tflink> adamw: true, just thought that I would mention it.
16:49:11 <tflink> red_alert: maybe, depends on how we want to proceed and what the anaconda team already has planned
16:49:14 <red_alert> maybe create a special tracker bug for it, too...or go with F17Blocker already
16:49:36 <brunowolff> When I can try installing with only 512 MB of memory, I'll be looking at that.
16:49:48 <j_dulaney> Like I said, as soon as their stuff starts hitting nightlies, test it
16:49:49 * kparal notes we have some bugs already that can be assigned to F17Blocker. creating it now would help
16:50:06 <adamw> red_alert: it helps to know when they're planning to land changes, what's expected to work, and probably give them input into use cases they may not have considered, on the basis of the test matrix
16:50:20 <adamw> kparal: i thought i already had, but someone mentioned earlier that they weren't there, i'll do it in a bit
16:50:31 <tflink> and make sure that we're covering all the changes
16:50:43 <tflink> our current test matrix doesn't explicitly cover all the gui stuff
16:50:57 <adamw> kparal: though anyone caan do it, again - process is documented at https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Trackers
16:51:02 <j_dulaney> adamw:  I'll be having a bunch of free time starting in a month, so I can do the connecting
16:51:10 <adamw> starting in a month is a month too late ;)
16:51:26 * j_dulaney can start now
16:51:54 <j_dulaney> The only real blocker of my time will be finals the first week of December
16:52:30 <adamw> i think it might be best to have someone who's definitely going to have lots of time for it, which is why i was gonna volunteer
16:53:25 <j_dulaney> True
16:53:40 * nirik looks for more things for adamw to do since he has lots of time. ;)
16:53:50 <adamw> nirik: i don't now ;)
16:53:59 <nirik> too late. rats. ;)
16:54:05 <adamw> #action adamw to co-ordinate with anaconda team on f17 re-write test planning
16:54:05 <j_dulaney> Especially since he's firing everyone
16:54:08 <adamw> missed your spot!
16:54:16 <adamw> j_dulaney: oh right, i didn't fire nirik yet!
16:54:21 <red_alert> I
16:54:53 <adamw> so, tflink already had a few ideas that might help with testing - want to re-air them here tflink?
16:55:16 <red_alert> I'll monitor adamw's sleeping hours and tell nirik when I spot that he's got time available...i.e. as soon as he sleeps over 4h/night ;)
16:55:20 <tflink> adamw: I can or we can wait to see what the anaconda team has planned
16:56:39 <red_alert> adamw: ah, please make sure to get their thoughts on having some anaconda/QA hackfest during fudcon...might be important for my sponsorship request ;)
16:56:50 <tflink> the first was to make sure that we always have install isos available for testing
16:56:56 <adamw> red_alert: ooh, yup.
16:57:15 <j_dulaney> tflink:  Pull in rel-eng for that?
16:57:42 <tflink> which has two parts, keeping a somewhat up-to-date rawhide tree that is stable so that we can test anaconda without worrying about the rest of rawhide
16:58:35 <tflink> and the second, which is making it easier to build test isos with custom package sets for testing
16:58:58 <kk4ewt> adamw,  so you want hackfest space a Fudcon What days
16:59:05 * j_dulaney wonders if we could just test new Anaconda with F16 sicne we know that's stable?
16:59:20 <tflink> j_dulaney: that might be another option
16:59:29 <adamw> kk4ewt: we'll deal with that later
17:00:05 <kk4ewt> :)
17:00:19 <red_alert> using the new anaconda with F16 will make upgrade testing rather invalid, though
17:00:27 <j_dulaney> tflink:  Scientific method
17:00:33 <j_dulaney> Change only one variable
17:00:44 <tflink> the other different idea is going to be more of a question - would the availability of hardware/VMs affect the amount of testing done on new anaconda?
17:00:44 <j_dulaney> And this is a huge variable to change
17:01:06 <tflink> yeah, agreed on that one. upgrade testing could be separate
17:01:38 <tflink> but I'm not sure about all the implications of trying to test F17 anaconda w/ F16
17:01:44 <adamw> tflink: might elaborate on your second idea a bit. only quickly. =)
17:01:46 <j_dulaney> tflink:  if I can get my hands on a larger hardrive for the new laptop, I'll have hardware to test on (and do VM testing, too
17:02:37 <tflink> my idea was to offer test VMs to people who want to test. Testers would be able to access the GUI installer through a VNC-ish interface
17:03:24 <adamw> question is, do we have anyone who's dying to help test but has no access to test hardware or a VM?
17:03:28 <adamw> or is that...not the case?
17:03:49 * j_dulaney does not really have hardware right now
17:03:58 <kparal> I believe that use case may be pretty valid
17:04:07 <tflink> adamw: a better way to put it would be: would access to more VMs increase the amount of testing that people would do?
17:04:10 <j_dulaney> I've got a laptop capable of it, but I don't have a HD and can't afford one
17:04:27 <t8m> We are supposed to have FESCo meeting here. Will the Fedora QA meeting end soon?
17:04:43 <mmaslano> adamw: could you end your meeting please?
17:04:53 <adamw> mmaslano: yeah, sorry, been trying :(
17:05:08 <adamw> let's end this and move over to #fedora-meeting-1 to conclude
17:05:14 <tflink> k
17:05:20 <adamw> why does this happen, btw? didn't we used to have 2 hours between the meetings?
17:05:23 <brunowolff> As sort of a joint issue, is FESCO going to comment on their election guidelines since j_dulaney is a nominee who doesn't meet the letter of the policy?
17:05:39 <j_dulaney> So I should stick around?
17:05:42 <adamw> #agreed meeting to conclude in #fedora-meeting-1
17:05:42 <t8m> adamw, we have the start of the meeting UTC based
17:05:46 <adamw> #endmeeting