16:00:17 <jlaska> #startmeeting Fedora QA Meeting
16:00:18 <zodbot> Meeting started Mon Mar 22 16:00:17 2010 UTC.  The chair is jlaska. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:19 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:20 <jlaska> #meetingname fedora-qa
16:00:21 <zodbot> The meeting name has been set to 'fedora-qa'
16:00:28 <jlaska> #topic Gathering
16:00:49 * kparal appears
16:00:50 <jlaska> alright folks, who do we have around for the QA meeting?
16:01:16 <jlaska> kparal: jskladan howdy gang
16:01:32 <adamw> morning
16:01:44 <jlaska> adamw: heyo
16:01:46 * jskladan likes meeting time
16:02:17 * Southern_Gentlem 
16:02:40 <jlaska> Southern_Gentlem: hey there!
16:02:45 * jlaska waits a few more minutes
16:05:34 <jlaska> Oxf13: said he'd be out for the meeting
16:05:40 <jlaska> is wwoods around?
16:06:04 <jskladan> haven't seen him (or at least he did not respond to my ping)
16:06:39 <jlaska> okay, well ... let's get moving, I don't want to keep you folks longer than we need to
16:06:52 <jlaska> I'll be walking through the proposed agenda at - http://lists.fedoraproject.org/pipermail/test/2010-March/089489.html
16:06:59 <jlaska> #topic Previous meeting follow-up
16:07:12 <jlaska> First up ...
16:07:17 <jlaska> #info adamw to check-in with wwoods on tooling needs for priv esc. test
16:07:27 <adamw> well, I sent an email
16:07:31 <adamw> he hasn't replied yet :)
16:07:33 <jlaska> I saw adam reached out on this, wwoods || adamw  have you guys had a  chance to talk further?
16:08:28 <jlaska> okay ... I spoke to wwoods on Friday and he was putting a lot of brain cycles into depcheck ... I'll keep this on for follow-up next week
16:08:32 <adamw> okay
16:08:51 <jlaska> #action wwoods and adamw to discuss tooling needs for priv esc. test
16:09:14 <jlaska> This next topic is related to our agenda today, so we don't need to dive too deep on it
16:09:18 <jlaska> #info maxamillion seeking input from the Mentors group for guidance on mentor responsibilities
16:09:41 * jlaska seeing if maxamillion is around on #fedora-qa
16:10:16 <maxamillion> sorry I'm late
16:10:23 <jlaska> maxamillion: hey there, no worries
16:10:52 <adamw> hey maxa
16:10:58 <jlaska> maxamillion: just following up on your check-in with the Mentors group for guidance on mentor responsibilities
16:11:36 <jlaska> I think this will touch on a larger topic I was hoping we could discuss later on, so we can move discussion to that point if you like
16:11:37 <maxamillion> jlaska: lemme find the archive link .... I got 1 response that I'd like to put up for discussion
16:11:44 <maxamillion> yeah, lets do that
16:12:03 <jlaska> maxamillion: okay ...
16:12:30 <jlaska> #info got 1 response and will discuss later during QA meeting
16:12:38 <maxamillion> its probably going to be a long discussion so I think the "previous meeting follow-up" might not be the best spot to bring it up :)
16:12:45 <jlaska> maxamillion: heh, okay :)
16:12:52 * Viking-Ice half in half out..
16:13:08 <jlaska> Viking-Ice: howdy
16:13:16 <jlaska> #topic F13 Test Status
16:13:44 <jlaska> just a quick update on where things are with F13 Beta
16:14:03 <jlaska> We had the pre-beta acceptance drop last week - https://fedoraproject.org/wiki/Test_Results:Fedora_13_Pre-Beta_Acceptance_Test_1
16:14:12 <jlaska> nothing horribly broken detected from that test run
16:14:34 <jlaska> #info Oxf13 is working with dlehman to identify an anaconda build to deliver for the Beta 'test compose'
16:15:01 <jlaska> Once that's available ... we'll update rhe's announcement to the list
16:15:14 <jlaska> Folks are encouraged to post test results to https://fedoraproject.org/wiki/Category:Fedora_13_Beta_TC_Test_Results
16:15:34 <jlaska> in the upcoming test news front ...
16:16:05 <jlaska> #info assuming positive test results - Beta 'release candidate' planned for this Thursday (2010-03-25)
16:16:31 <jlaska> alongside the planned F13 testing, just a reminder on the test day front ...
16:16:57 <jlaska> thanks to all for participating in the gnome-disk-utility test day last week (https://fedoraproject.org/wiki/Test_Day:2010-03-18_Palimpsest)
16:16:59 * wwoods here
16:17:02 * maxamillion downloads the latest nightly build to fire up some tests
16:17:10 <jlaska> wwoods: hey there
16:17:15 <jlaska> this week we have ...
16:17:18 <adamw> yep, disk test day went well
16:17:21 <adamw> i'll send a recap to the list soon
16:17:27 <adamw> 12 bug reports, 2 already fixed
16:17:41 <jlaska> adamw: sweet, thanks ... I was really stoked at the improved UI
16:17:50 <kparal> yes, it looks great
16:17:52 <jlaska> kudos to davidz
16:18:21 <jlaska> #info Thursday (2010-03-25) we have a Printing test day - https://fedoraproject.org/wiki/Test_Day:2010-03-25_Printing
16:18:52 <jlaska> I don't think anyone from the QA team is involved in organizing this event?  Mostly twaugh driving here?
16:19:36 <wwoods> it'd be pretty cool if we had the Test Day SOPs so well-defined that QA involvement wasn't required to run a Test Day
16:20:21 * maxamillion would be willing to help but doesn't actually own a printer so thinks it would be a situational mismatch
16:20:23 <jlaska> wwoods: sarcasm? :)
16:21:18 <adamw> jlaska: yeah, I haven't done anything on it yet, and the page looks good
16:21:26 <jlaska> the wiki looks fairly well defined for the printing event, I might just check-in with twaugh to see if they need anything from us
16:21:30 <adamw> kudos to twaugh
16:21:48 <jlaska> right on!
16:22:12 <jlaska> alright, next week we have 2 test days ... hosted by QA
16:22:23 <jlaska> #info 2010-03-30 - SSSD by Default - https://fedoraproject.org/wiki/Test_Day:2010-03-30_SSSDByDefault
16:22:33 <jlaska> #info 2010-04-01 - ABRT - https://fedoraproject.org/wiki/Test_Day:2010-04-01_ABRT
16:22:47 <wwoods> jlaska: no, not at all - is that already the case? because awesome
16:23:09 <jlaska> Both have have a QA and devel pair established, so I believe they are moving forward nicely
16:23:19 <jlaska> wwoods: right on ... pass it along ... https://fedoraproject.org/wiki/QA/SOP_Test_Day_management
16:23:30 <jlaska> that's all I have for F-13 testing ...
16:23:38 <jlaska> it's just full of test runs and test days
16:23:40 <wwoods> oh that's awesome!
16:23:41 <jlaska> did I miss anything?
16:24:02 <wwoods> I'll send notes / make edits if there's anything missing. anyway, please continue
16:24:08 <jlaska> wwoods: kudos to adamw for that
16:24:35 <jlaska> okay ... moving on to next topic ...
16:24:49 <jlaska> #topic Updates testing brainstorm
16:25:16 <jlaska> I was hoping to spend time today discussing what we want (or think we need) out of an updates testing workflow
16:25:30 <jlaska> so much of our time is spent testing the branched release
16:26:06 <jlaska> but with the increased exposure of using bodhi for branched (and already released updates), I would like to start collecting ideas so we can document a roadmap/plan
16:26:26 <jlaska> there are a lot of efforts underway here already ...
16:26:41 <jlaska> kparal has an acceptance test plan in draft -- https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_acceptance_test_plan
16:27:07 <jlaska> maxamillion has been putting out feelers for defining the 'proventesters' group -- http://lists.fedoraproject.org/pipermail/test/2010-March/088980.html
16:27:24 <jlaska> but right now ... what we document to our testers is https://fedoraproject.org/wiki/QA/Updates_Testing
16:27:30 <adamw> so these are the things I think we need: a policy/sop for the 'proventesters' group, and a guide to providing updates-testing feedback for a) branched and b) stable releases, explaining what actually should be tested and how feedback should be given
16:27:36 <maxamillion> and I got 1 reponse --> http://lists.fedoraproject.org/pipermail/mentors/2010-March/000177.html
16:27:41 <maxamillion> response even ...
16:28:08 <maxamillion> adamw: +1
16:28:09 <jlaska> #info we need a policy/sop for the 'proventesters' group
16:28:30 <jlaska> #info we need a guide to providing updates-testing feedback for branched + released explaining what should be tested and how to give feedback
16:28:52 <kparal> we can leverage the fedora-easy-karma script
16:29:03 <adamw> kparal: have a 'usage' message in it?
16:29:23 <kparal> adamw: sorry?
16:29:34 <jlaska> related to your 'how to give feedback' ... lmacken and wwoods have talked about providing a wiki structure for testers (and contributors) to expand on that provides test instructions on a per-package basis
16:29:35 <adamw> kparal: just wondering how you meant to 'leverage' it
16:29:43 <kparal> We can document how best to use it
16:29:58 <kparal> and mention it in the guide in the first place :)
16:30:02 <adamw> jlaska: that sounds rather like the 'how_to_debug' pages?
16:30:19 <jlaska> adamw: similar, in that it's on the wiki and has a structure around the pages
16:30:59 <jlaska> what does it mean to test an update?
16:30:59 <adamw> okay
16:31:07 <jlaska> does that mean you've run through a documented series of tests
16:31:12 <jlaska> or just validated the listed bugs are resolved?
16:31:17 <jlaska> or something else?
16:31:24 <kparal> good point
16:31:31 <adamw> yes, that's what I was concerned with
16:32:04 <adamw> i think we need general guidance on that which applies to all packages; having specific instructions for some is also good, but the priority should be to define exactly what the acceptance testing is supposed to check
16:32:46 <jlaska> adamw: great point ... so continue to lock down the details drafted in https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_acceptance_test_plan ?
16:32:48 <adamw> do you think we should work that up ourselves, or go back to fesco for guidance?
16:33:04 <maxamillion> I think that's hard to write down because updates generally do one of two things: either it fixes a previous bug or adds an enhancement
16:33:16 <jlaska> #info we need general guidance on that which applies to all packages; having specific instructions for some is also good, but the priority should be to define exactly what the acceptance testing is supposed to check
16:33:21 <adamw> jlaska: sort of, but I think that's mostly focused on automated tests right now?
16:33:25 <maxamillion> and in the event of a bug fix, you need to test that the bug is fixed but also test that there aren't any regressions
16:33:35 <adamw> maxamillion: there can be multiple criteria for different types of updates, it doesn't need to be one-size-fits-all
16:33:41 <jlaska> adamw: yeah, just the tests required to accept the update for additional testing
16:33:52 <adamw> maxamillion: just needs to be clear enough that you can always tell what you should be testing for any given update
16:34:00 <wwoods> adamw: that's the *acceptance* plan
16:34:01 <maxamillion> adamw: true, maybe I just don't entirely understand how to capture that into a procedure
16:34:15 <adamw> jlaska: right, i'm thinking more about what people should be looking for at the bodhi stage. we can certainly put the criteria into that document, though.
16:34:27 <jlaska> adamw: I'm with ya
16:34:29 <wwoods> acceptance plans are (kind of by definition) simple and fast
16:34:33 <wwoods> which lends them to automation
16:34:41 <adamw> wwoods: well, the point is 'acceptance into what'
16:34:47 <wwoods> actual testing.
16:34:51 <adamw> right
16:34:53 <jlaska> what I'm hoping to drill out here is what happens after we've accepted the update
16:35:01 <adamw> so either we expand the scope of that document or we have two
16:35:05 <jlaska> wwoods: adamw: exactly, just what you guys are talking about
16:35:12 <wwoods> the point of an Acceptance Test Plan is to quickly decide whether or not the thing is even *testable*
16:35:17 <adamw> i don't think that question is super-important, though, the important thing is to nail down all the actual information
16:35:24 <adamw> then we can look at how exactly to arrange it
16:35:33 <jlaska> sure, let's get more ideas ...
16:35:54 <wwoods> and once something passes the acceptance testing, it's accepted for actual testing, and then moves on to another (separate) test plan
16:36:10 <adamw> wwoods: sure, so if you want it to stay as an acceptance plan, we need a separate document for what we're discussing now (defining what people should look at when deciding whether to mark an update as good or bad)
16:36:31 <adamw> for now i was just thinking about hashing out the actual criteria on a mailing list thread
16:36:35 <wwoods> in short: acceptance tests are designed to be the first-line test in a series of test plans. they're simple, very automatable, and not intended to be exhaustive.
16:36:37 <adamw> we can write it down all fancy-like later =)
16:36:48 <wwoods> I think you're missing the point I'm trying to make here
16:36:54 <adamw> i may be. what is it?
16:37:09 <jlaska> does it help for us to continue engaging in discussion to detail the different types of updates (e.g.  https://fedoraproject.org/wiki/Stable_Release_Updates_Proposal) ?
16:37:28 * jlaska holds question until wwoods is done
16:37:30 <wwoods> we need a separate document for what we're discussing now, because an Acceptance Test Plan is, *by definition*, not supposed to be the sole test plean
16:37:33 <wwoods> err test plan
16:37:53 <adamw> wwoods: sure. I got that. that's why I said that if we want it to be an acceptance plan, it can't cover this stuff and we need a new one.
16:38:27 <jlaska> so does it help to think of this discussion as just a "Package Update test plan"
16:38:32 <wwoods> an Acceptance Test Plan is always, by definition, the first in a series of test plans, so anything beyond the scope of "can this thing actually be tested properly" should be moved into subsequent plan(s)
16:38:50 <adamw> if we're thinking in formal QA terms, sure, that sounds about right
16:39:00 <kparal> I'm lost now. are we talking about acceptance test plan in general or about the work the proventesters will be doing (mainly critical path stuff)?
16:39:07 <jlaska> I think we all agree on that point, but I don't want to get bogged down in what final document this takes form in
16:39:23 <wwoods> the basic existence of an Acceptance Plan implies the intent to create more separate plans.
16:39:26 <adamw> kparal: that's a good point, i think we're talking about both, and you're right that under the fesco-approved policy they'll be distinct
16:39:28 <jlaska> kparal: I'd like ideas around what testing needs to occur for updates in general
16:39:48 <maxamillion> ok .... so: AcceptancePlan -> UpdatePlan -> Testing -> Karma -> CheckedForStable -> Bodhi
16:39:51 <maxamillion> ?
16:40:00 <wwoods> I just wanted to make sure we're all clear on this point, 'cuz it keeps coming up
16:40:12 <wwoods> but yeah. point belabored.
16:40:13 <adamw> maxamillion: er, the UpdatePlan is how you do the Testing, and how can it go from Karma to Bodhi? :)
16:40:54 <maxamillion> adamw: yeah, basically that's the piece of the puzzle we need to sort out .... which is more or less the root of this conversation and what we need outlined for the ProvenTester group
16:40:56 <adamw> erm, so can we reboot? do we want to talk about exactly what candidate updates should be tested for in this meeting, or do we just want to agree that's what we need to define and do it on the list?
16:41:11 <jlaska> adamw: thank you, the later was my intent
16:41:14 <maxamillion> adamw: at least that's how I understand it
16:41:28 <jlaska> #info focus for this meeting - agree that's what we need to define and do it on the list
16:41:44 <jlaska> so right now ... I've got that we need to ...
16:41:47 <adamw> i'm happy to write up a draft to get the ball rolling on discussion for that
16:41:53 <adamw> if anyone would like that
16:41:54 <jlaska> create proventesters group
16:42:20 <jlaska> maxamillion: you've been looking into supporting details around mentoring people into this group?
16:42:45 <adamw> maxamillion: are we expecting another draft of the proventesters policy/sop proposal?
16:43:14 * jskladan has to go - see you around tomorrow, gang!
16:43:23 <jlaska> jskladan: see you tomorrow :)
16:43:45 <maxamillion> jlaska: yes, but I haven't been able to find much ... I only got one response to my query with the mentors and I didn't feel it was the information I was hoping for
16:44:25 <maxamillion> adamw: yeah, I'm going to need to touch things up and write up a proposal for how the mentor program will be handled, but I worry that will spawn into a rather large topic of its own
16:44:58 <adamw> oh fun! i like topics.
16:45:10 <jlaska> as adamw pointed out, I think we can work the details of that out of the meeting ... unless there are specific roadblocks you needed to walk through here?
16:45:47 <maxamillion> nope, none in particular
16:46:20 <jlaska> alright ... so we've got defining 'proventesters' group, and drafting a policy/sop around detailing the group
16:46:25 <adamw> so should we take an action item for maxamillion to provide the next draft of the provenpackagers end of things, and me to provide a draft for the update test plan end of things?
16:46:25 <jlaska> what's after that?
16:46:47 * jlaska recommends excessive use of #action and #info
16:47:01 <maxamillion> who has control of meetbot?
16:47:12 <jlaska> I think anyone can add #action and #info tags, no?
16:47:18 <Southern_Gentlem> yes
16:47:54 <maxamillion> #action maxamillion will work on second draft of provenpackagers proposal
16:48:08 <maxamillion> #action adamw will provide a draft for the update test plan
16:48:20 <jlaska> okay, so I'm still trying to wrap my brain around the big picture ...
16:48:23 * maxamillion doesn't think meetbot picked those up ....
16:48:32 <maxamillion> doesn't meetbot normally echo the command or something?
16:48:37 <jlaska> not for those
16:48:41 <maxamillion> oh ok
16:48:49 <maxamillion> :)
16:49:09 <jlaska> alright, so we have proventesters and an update test plan ... are those the only 2 things standing in the way of having a defining updates testing workflow?
16:49:22 <jlaska> are there other things we need on the radar, perhaps further down the line?
16:49:33 <jlaska> s/defining/defined/
16:49:43 <maxamillion> not that I can think of ... but I'm sure there's something I missed from my inbox or I'm just simply out of the loop on
16:49:59 <maxamillion> or maybe I'm not and that'd kick hind parts :)
16:50:23 <adamw> i think that's enough to work on for now anyway
16:50:29 <maxamillion> agreed
16:50:40 <adamw> once we get those done there's some joining-up to do in the wiki but i can't think of anything else major
16:50:48 <maxamillion> +1
16:50:56 <jlaska> okay
16:51:06 <maxamillion> I have to run, have to grab a bite to eat before a meeting
16:51:17 <jlaska> adamw: I'll need to bug you after meeting to get a sense for hte scope of your test plan ... you might be knocking out a lot of stuff with just that
16:51:23 <jlaska> maxamillion: okay, thanks for joining
16:51:36 <maxamillion> jlaska: always happy to be here :)
16:52:10 <jlaska> wwoods: kparal: any other points not raised that you'd like to consider as eventual tasks?
16:52:41 <kparal> not currently
16:54:14 <jlaska> wwoods: anything else on your radar for this topic?
16:55:11 <jlaska> alright, in the interest of time ... let's move on to open-discussion
16:55:42 <jlaska> #topic open discussion - change meeting time?
16:56:25 <jlaska> once Europe changes their clocks for daylight savings, this meeting will be a bit late for kparal and jskladan I believe
16:56:40 <kparal> well, 6 pm
16:57:15 <jlaska> should I send out a http://whenisgood.net/ for a new meeting time
16:57:19 <kparal> there is this proposal to change the meeting time one hour back during the summer time (15.00 UTC) and revert it back once the winter time comes (16.00 UTC)
16:57:32 <kparal> that way the meeting time would be always the same for all of us
16:57:54 <kparal> but it would require announcing changes every half a year
16:57:55 <jlaska> UTC would change, but the localtime would remain the same
16:58:03 <adamw> i don't really mind
16:58:04 <kparal> jlaska: thanks for clarification
16:58:54 <jlaska> adamw: so that puts it @ 8am pacific?
16:59:47 <jlaska> alright, I'll follow-up to the list, but I don't have objections to that ... I don't think we're going to confuse lots of people with that.  The UTC / DST change already confuses people
16:59:52 <kparal> I think adamw is farthest in the west of all of us?
17:00:05 <jlaska> #action jlaska to post meeting time change request to test@l.fp.org
17:00:13 <jlaska> #topic open discussion - <your topic here>
17:00:13 <Viking-Ice> Except for me since i'm on 0 GMT
17:00:14 <adamw> yeah, 8am. 8am's fine
17:00:34 <jlaska> Viking-Ice: oh true
17:00:44 <jlaska> anything else not discuss that people would like to bring up?
17:00:54 <Viking-Ice> + check with 0xf13 think he has always had problem with the meeting time
17:01:06 <jlaska> Viking-Ice: will do, thanks
17:01:48 <jlaska> I'll close out the meeting in another minute, unless there are other topics raised
17:02:51 <jlaska> alrighty ... that'll do it for today then
17:02:59 <jlaska> thanks everyone, I'll follow-up with minutes to the list
17:03:03 <jlaska> #endmeeting