15:00:11 <adamw> #startmeeting Fedora QA meeting
15:00:11 <zodbot> Meeting started Mon Jun 15 15:00:11 2015 UTC.  The chair is adamw. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:11 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:13 <jjmcd> tnx
15:00:15 <adamw> #meetingname fedora-qa
15:00:15 <zodbot> The meeting name has been set to 'fedora-qa'
15:00:21 <adamw> #topic Roll call
15:00:26 * roshi is here
15:00:29 <adamw> ahoyhoy folks, who's around for some QA meetiness?
15:00:30 <randomuser> wb, adamw
15:00:37 * satellit listening
15:00:38 <adamw> hi randomuser
15:01:57 * tflink is here, wrapping up the qadevel meeting
15:03:26 * kparal is here
15:03:32 <danofsatx> .hello dmossor
15:03:33 <zodbot> danofsatx: dmossor 'Dan Mossor' <danofsatx@gmail.com>
15:03:41 <danofsatx> greetings, folks.
15:03:55 <adamw> hi tflink, kparal, dan
15:04:25 * FranciscoD sits in a corner to watch :)
15:05:11 <adamw> this room is shaped like an egg.
15:05:18 * adamw watches franciscod cease to exist in a puff of logic
15:05:24 <FranciscoD> :o
15:06:03 <adamw> qa-devel meeting done yet?
15:06:06 <adamw> oh, yes.
15:06:08 <adamw> let's go then!
15:06:20 <adamw> #topic Previous meeting follow-up
15:06:30 <adamw> I dug out the minutes from last time, doesn't look like there were any action items
15:06:38 <adamw> #info there appear to be no action items from last time
15:06:44 <adamw> was there anything else that requires follow-up?
15:07:39 <roshi> I can't think of anything
15:07:59 <adamw> okely dokely
15:08:10 <adamw> #info there's nothing requiring follow-up from 2015-06-01 meeting
15:08:17 <adamw> #topic Fedora 23 schedule / status / planning
15:08:35 <adamw> #info the current F23 schedule is available at https://fedorapeople.org/groups/schedule/f-23/f-23-quality-tasks.html
15:09:10 <danofsatx> action adamw: create action items
15:09:33 <adamw> as it stands, we have the first blocker meeting scheduled 07-01, Alpha TC1 07-14, Alpha freeze 07-28, and go/no-go 08-06
15:09:39 * adamw disappears in a puff of recursion
15:09:46 <tflink> wow, that soon?
15:09:48 <tflink> oof
15:10:04 <adamw> i think this is another short cycle to try and get back on october releases
15:10:09 <danofsatx> I won't be at that meeting. I'll be in the middle of the Carribean ;)
15:10:10 <adamw> Final release is scheduled for 10-27
15:10:42 <tflink> danofsatx: i wish I could use that excuse :)
15:11:08 <roshi> tflink: I'm pretty sure that was danofsatx's way of inviting us
15:11:22 <danofsatx> my FIL paid for it. I can't afford it :/
15:11:23 <tflink> did that get moved up or is it that we usually slip until after US thanksgiving?
15:11:43 <adamw> so, i've been kinda kicking around with dgilmore some ideas about changing the alpha/beta/final process, but we want to discuss that in more detail at flock and it looks like we'll be well into f23 by then
15:12:14 <roshi> so we might change it up for F24, then
15:12:25 <adamw> tflink: we slipped gradually backwards over i think 17-20
15:12:35 <roshi> could we get a tl:dr of what you guys have discussed?
15:12:37 <adamw> we kept slipping a few weeks then scheduling another 6 months
15:12:44 <adamw> roshi: if the idea holds up, yeah
15:13:13 <adamw> roshi: basically my thought is that the whole idea of alpha/beta milestone releases is a bit antiquated these days
15:13:27 <adamw> we put a lot of effort into them and then mostly people just go and install TCs or nightlies
15:13:27 <tflink> roshi: I'm game for a trip to the Caribbean :)
15:13:35 <dgilmore> adamw: :) dropping alpha would be awesome
15:13:53 <adamw> so i'd like to look into the possibility of a more nightly-based process
15:13:55 <dgilmore> we should have nightly TC composes for F23
15:14:16 <adamw> call 'em TCs or whatever, but just focus on continual improvement on the nightlies
15:14:17 <dgilmore> in releng we are working to make branched and rawhide look just like a TC
15:14:26 <roshi> similar to what we did between F21 and F22, but with the branched tree instead of rawhide, then?
15:14:30 * tflink would be interested in seeing how much traffic the alpha composes see after alpha release (knowing that the stats from dl.fp.o aren't the whole picture)
15:14:37 <tflink> i suspect that it's minimal
15:15:06 <adamw> roshi: well, what we did for 21 and 22 was do more nightly testing before alpha, but once we hit alpha we did more or less the same as before
15:15:17 <roshi> right
15:15:24 <adamw> my rough idea would be to just not do alpha or beta at all, and rework release validation around the nightlies
15:15:53 <roshi> well us going through all the blockers every meeting fits that pretty well
15:15:58 <adamw> there's a lot of questions of course, like how do we make sure the nightlies are at a reasonable minimal level of quality throughout, or at least major showstoppers are quickly fixed, without the alpha/beta blocker process
15:16:00 <roshi> and then we'd just have "blockers"
15:16:02 <adamw> but we can look at it at least
15:16:07 <adamw> right
15:16:24 <FranciscoD> the one thing I can think of that removing alpha/beta will affect would be the "press"
15:16:27 <adamw> obviously it's kind of a CI-ish model so we'd want it tied into taskotron/openqa testing somehow
15:16:30 <randomuser> milestone releases have marketing value
15:16:37 <FranciscoD> as in, no we have alpha/beta announcements and so on
15:16:38 * randomuser highfives FranciscoD
15:16:43 <adamw> FranciscoD: yeah, that's another point, alpha/beta have PR value too
15:16:44 <FranciscoD> lol
15:16:45 <tflink> adamw: taskotron can fire stuff based on composes
15:16:45 <roshi> that was my thought as well
15:16:50 <adamw> so, there's lots to think about there
15:16:52 <FranciscoD> randomuser: you stole my sentence :P
15:17:04 <tflink> hopefully we'll have a better beaker system before too much longer
15:17:11 * tflink doesn't know how openqa schedules stuff
15:17:23 <dgilmore> adamw: tflink: down the magical rod that sees into the future, I want to fire off composes when pieces that are in it change
15:17:31 <dgilmore> so we do a CI based compose
15:17:39 <adamw> tflink: dumbly, of course! basically you can run a cron job that'll look and see if it has something new to test. :)
15:17:51 <tflink> fair enough
15:17:56 <dgilmore> which would then go through a bunch of automated testing before going elsewhere
15:18:02 <FranciscoD> the marketing problem shouldn't be too difficult to handle though, stuff can be labelled alpha/beta even if the milestones aren't being used for QA
15:18:07 <tflink> dgilmore: what do you want/need from us to make that happen?
15:18:20 <tflink> the compose parts, not the automated testing parts :)
15:18:25 <adamw> FranciscoD: yeah, the good old tried and true PR strategy of 'lying' :)
15:18:32 * tflink has a decent idea of what's needed there
15:18:34 <dgilmore> tflink: we need to know what is actually in the pieces to know when to trigger composes
15:18:57 <dgilmore> tflink: then we need tooling to dynamically know what to make
15:19:04 <dgilmore> so we only make things that will be changed
15:19:26 <tflink> dgilmore: one thought would be to use taskotron to analyze what's changed and submit signals to fire off a compose
15:19:35 <tflink> but we can discuss that outside the meeting, if it makes sense
15:19:37 * adamw notes we're a bit off the f23 topic but doesn't want to stop the train as it's a useful discussion
15:19:44 <adamw> i'll fix it in the #info later, we can get back to 23 discussion in a bit
15:19:46 * tflink isn't sure what all has changed in releng plans recently
15:19:54 <dgilmore> tflink:  lets chat later
15:19:57 <adamw> dgilmore: how's work coming along on reducing compose times?
15:20:12 <dgilmore> adamw: for f23  we should have a nightly compose that looks like a TC/RC
15:20:20 <dgilmore> at least that is what releng is working towards
15:20:28 <adamw> dgilmore: that'd be great if we can get that soon
15:20:45 <dgilmore> adamw: its getting there, slower than we want, but we are making progress
15:20:46 <tflink> dgilmore: sounds like a plan
15:21:44 <adamw> #info we have some thoughts on overhauling the release validation process to be more nightly-based and CI-ish and removing or downplaying the significance of Alpha and Beta releases, but these are probably F24 or later stuff with the shortish F23 cycle
15:22:02 <adamw> #info releng is working on having full TC/RC-style nightly composes during F23 cycle (not just lives and boot.iso)
15:22:35 <adamw> so for F23 - do we want to move up the first blocker review meeting, carrying on the idea of 'do it early and often'?
15:22:53 <adamw> there are 2 proposed alpha blockers atm
15:23:08 <adamw> we could do a quick meeting next week to start the process off
15:23:14 <tflink> "want"? not so much, "would it be a good idea?", probably :)
15:24:27 <adamw> don't everyone jump for joy all at once :)
15:24:42 * roshi will send the announce
15:25:41 * tflink plants feet firmly on floor, will do no more jumping for joy
15:26:09 <adamw> #info we'll do the first blocker review meeting on 06-22 to get out in front of the proposed blockers
15:26:44 <adamw> otherwise i think we're more or less in shape for 23, right? all the tools and things seem to be pretty much ticking over
15:26:56 <adamw> can anyone think of any outstanding test case / criteria issues?
15:27:46 <roshi> kinda related: what are our expectations for mid-release updated images?
15:28:17 <adamw> ah, yeah, that's another active thing
15:28:27 <adamw> if it's gonna happen in f23 cycle we certainly need to discuss it, thanks roshi
15:28:42 <adamw> it's still at the 'being kicked around in fesco' stage right now, right?
15:28:55 <roshi> np - I suspect we'll need to have some criteria discussion with it
15:28:59 <roshi> afaik, yeah
15:29:16 <roshi> I mean, there's been discussion about it since F21
15:29:22 <roshi> it was part of the Grand Plan
15:30:25 <danofsatx> Are we talking Alpha criteria, or all of it?
15:30:27 <roshi> https://fedorahosted.org/fesco/ticket/1444
15:30:40 <roshi> danofsatx: for an updated release image, I'd say all of it
15:30:46 <danofsatx> hmmm....
15:30:57 <roshi> what good is a new released image with beta and final blockers on it?
15:30:57 * danofsatx comes up with a plan
15:31:24 <roshi> Final has to have no blockers of any kind, so an updated image/media/whathaveyou would too
15:31:32 <danofsatx> I was speaking specifically in regards to the criteria itself, not whether the images met the criteria
15:32:00 <adamw> #info roshi notes that there is a current discussion around shipping post-release respins for some images: https://fedorahosted.org/fesco/ticket/1444 . This obviously has significant QA implications
15:32:19 <danofsatx> I was thinking of going over the criteria with a fine tooth comb to determine what is still relevant, and what, if anything, was missing.
15:32:34 <adamw> danofsatx: well, i'd say at present all we have is a higher-level question than that: how do we ensure respun media are of acceptable quality?
15:32:57 <adamw> danofsatx: or were you going back to my earlier 'can anyone think of any criteria issues?' question?
15:33:05 <danofsatx> I was going back
15:33:09 <danofsatx> sorry, I'm distracted
15:33:31 * danofsatx puts on headphones to ignore the office
15:33:50 <adamw> danofsatx: that's totally fine, sorry, just got wires crossed :)
15:33:56 <FranciscoD> there are already updated respins, but they're not officially done by fedora infra
15:34:18 <roshi> or tested by us, afaik
15:34:20 <FranciscoD> yeah
15:34:26 <FranciscoD> Updated Live isos: http://tinyurl.com/Live-respins
15:34:30 <adamw> danofsatx: i was meaning the whole of the criteria. it's pretty much always a good idea to go through the whole set and see if you find anything that's outdated or incorrect or missing
15:34:31 * satellit soas is not part of respins though
15:34:40 <FranciscoD> been in the #fedora topic for quite a long time now
15:34:41 <adamw> danofsatx: i try to do it every so often but it's always a good idea
15:34:50 <adamw> danofsatx: so if you want to do that, more power to you
15:34:59 <adamw> anything you see, send out to the list as a proposal
15:35:02 <danofsatx> I know you do, but it seems a Herculean effort for one busy person
15:35:06 * tflink thought that one idea was to get the WGs to do most of the testing
15:35:11 <adamw> FranciscoD: right, the current live respins are unofficial and we don't do any testing on them
15:35:17 <FranciscoD> yeah
15:35:22 <tflink> for respins, anyways
15:35:42 <adamw> #info existing live respins - http://tinyurl.com/Live-respins - are semi- (or quarter-?) official and have no official QA process
15:35:50 <FranciscoD> I don't reckon they'll need too much testing - we don't push radical changes to stable fedora release anymore.
15:36:01 <FranciscoD> But yes, one has to install and test them, so that is quite a bit of work
15:36:13 <adamw> eh. there are kernel updates and dracut updates to name just two...
15:36:15 <tflink> another question is how to handle breakage
15:36:21 <adamw> right
15:36:37 <tflink> for example (not so likely but an example) say some update breaks anaconda's ability to install the respins
15:36:49 <adamw> there's the 'obvious' idea of just re-running the whole existing validation process for respun images...which obviously would probably work fine, but seems like a lot of work
15:37:26 <tflink> does that mean the anaconda devs are "on the hook" to fix it, does the update at fault get reverted or some other method for mitigation?
15:37:38 <adamw> then there's a whole spectrum of options between that and 'say we don't have time to do anything and it's up to WGs to test them or just kick them out the door'
15:37:45 <roshi> that's why in fesco ticket #1444 I suggested that the WGs do the testing, and bring their results to QA to sign off on
15:37:48 <adamw> right, that's another point
15:38:04 <FranciscoD> hrm, but then the WGs need their own "QA" teams?
15:38:04 <adamw> though seems like it might not be ours to decide (tflink's note)
15:38:22 <tflink> FranciscoD: they should already have folks willing to help test
15:38:26 <FranciscoD> or will QA work with WGs?
15:38:34 <tflink> the issue in my mind is that the big issue is personpower
15:38:44 <roshi> FranciscoD: from a QA standpoint, I would think QA can do best effort help
15:38:57 <FranciscoD> they probably do, but those folks will need to get up to speed with the QA validation process and things
15:39:00 <FranciscoD> ++
15:39:06 <roshi> since we generally have our hands full with the current release
15:39:15 <tflink> if the WGs want respins, they need to help :)
15:39:39 <adamw> is the respin idea coming out of WGs?
15:40:04 <roshi> afaict, QA could really care less in our day to day lives if there are updated images - because we're always working on the next thing
15:40:10 <adamw> the ticket is from dennis, and it sounds like it's with his releng hat on
15:40:10 <roshi> adamw: yeah, it is
15:40:13 <FranciscoD> won't making netinstallers primary sort of take care of this? Normal testing continues, and people get newer packages?
15:40:24 <adamw> FranciscoD: the obvious candidates for respins are the cloud imagesd
15:40:32 <roshi> my understanding is that it's cloud that wanted first, then atomic and whatnot
15:40:36 <adamw> which is kinda outside that scope
15:40:45 <FranciscoD> Ah, OK. Makes sense
15:41:10 <roshi> and there was talk from the workstation WG about wanting respun live images as well
15:41:45 <tflink> cloud images don't worry me as much since they're on the simpler side of deliverables
15:41:51 <FranciscoD> if one WG wants them, everyone will want them too :)
15:41:56 <adamw> so, this has been mostly a kick-around discussion so far
15:42:12 <adamw> do we want to take any definite points into the fesco discussion at this point, or do we want to wait for the proposal to take a bit more shape?
15:42:13 * tflink notes that the cloud WG doesn't exactly have the best track record for testing stuff
15:42:32 <roshi> adamw: yeah - it's been kicked around for a long time
15:42:43 <roshi> it's getting better :)
15:42:46 <tflink> trusting them to do the testing just because they say they will ... may not turn out so well
15:42:53 <roshi> they're even coming to blocker review meetings
15:43:05 <roshi> well, if no testing happens, no updated image
15:43:05 * tflink will believe it when he sees it
15:43:10 <roshi> seems easy enough to me
15:43:18 <tflink> yeah, that's what i was getting at
15:43:25 <adamw> and there's that automated testing thing for cloud images too
15:43:25 <roshi> a "must be this tall to ride" kinda thing
15:44:00 * satellit are netinstall tests = to repins for DE's?
15:44:05 <adamw> the only thing that concerns me about a WG-based approach is that it's kinda inconsistent; do we wind up with a future where some chunks of fedora QA work run  through us and some run through WGs and the border between the two makes no logical sense?
15:44:08 <tflink> don't commit to "yes we're OK with respun images on X date" without a "without testing done by a certain date, no respins"
15:44:27 <adamw> as things stand we're supposed to be 'the' fedora QA organization and in theory if we need more resources they should just magically appear and route through us, i think
15:44:35 <tflink> adamw: do you think we have the spare cycles to commit to respin testing?
15:44:41 <adamw> but on a practical level, requiring the WGs to bring the resources makes sense
15:44:53 <adamw> it depends on what the meaning of the word 'we' is ;)
15:45:17 <roshi> adamw: due to a lack of personpower, having the WGs do testing on updated images makes sense I think, so we can still focus on the next release
15:45:46 <roshi> I'd love it where the WGs assigned people to work as their QA monkeys with us
15:45:51 <adamw> ok, so let me take a couple of notes
15:46:49 <tflink> to address both concerns, maybe set the bars for the required testing for respins without committing time from folks who are usually doing Fn+1 work?
15:46:55 <adamw> #info we have a consensus that respins shouldn't be released without some kind of testing commitment in place, and the WGs responsible for the media to be respun need to help provide the resources to do the testing and demonstrate that it's actually occurred
15:47:01 <tflink> or some less akward/divisive way of phrasing that
15:47:07 <adamw> does that sound OK?
15:47:08 <tflink> that -> what I said
15:47:26 <roshi> adamw: sounds good to me
15:47:28 <tflink> yeah, that works
15:47:50 <adamw> ok, i'll try and summarize this for the fesco ticket and we can do more concrete discussion when the fesco proposal is more concrete
15:47:56 <roshi> sgtm
15:49:10 <adamw> so let's move on quick...
15:49:14 <adamw> #topic Fedora 22 retrospective
15:49:44 <adamw> I didn't really have much for this topic and we don't have much time, but does anyone have any reflections on the 22 cycle? any thoughts on how the things we changed worked out?
15:50:24 <adamw> i thought the idea to try and get blockers found and discussed earlier worked out pretty well overall, seems like there was less drama with anaconda team and we got the release done with only 1 week slip overall
15:50:39 <roshi> ^^ that
15:50:47 <roshi> I thought the blocker process went really well
15:51:01 <roshi> we almost never went over time, the meetings were generally shorter
15:51:20 <adamw> we did still wind up with some fairly long meetings near the end of the cycle, but i think it was better overall
15:51:29 <adamw> don't think we had any 6-hours-over-two-days epics
15:51:41 <roshi> not that I recall
15:52:29 <roshi> I think openQA helped as well
15:52:43 <roshi> how's the packaging of it going? need extra hands for it?
15:52:48 <danofsatx> I'm still slightly disgruntled about the KDE xcb error that got practically ignored.
15:53:40 <adamw> which bug is that, dan?
15:53:45 <danofsatx> hang on...
15:54:03 <danofsatx> .bug 1193742
15:54:06 <zodbot> danofsatx: Bug 1193742 (intel) Plasma5 Freezes (often due to notifications while locked), xcb/dri3 deadlock? - https://bugzilla.redhat.com/1193742
15:54:30 <adamw> roshi: it's not going atm, still in post-vacation-clearout mode. but it's not super difficult. i do have an os-autoinst package that builds; that was about as far as i got before my vacation. didn't check if it actually works as i don't know how to make os-autoinst do something useful outside the openqa framework :)
15:54:45 <adamw> roshi: http://paste.fedoraproject.org/232219/34383679
15:55:58 * roshi will take a look
15:56:32 <adamw> danofsatx: hum, afaict that one never really hit any of the release validation processes, right?
15:56:33 <adamw> it'
15:56:42 <adamw> doesn't look like it was proposed as a blocker or FE at any point
15:56:51 <danofsatx> It was.
15:57:56 <adamw> ah, it was proposed as https://bugzilla.redhat.com/show_bug.cgi?id=1217844
15:57:59 <roshi> that's what I was just looking at
15:58:14 <danofsatx> that's the other one I was looking for
15:58:31 <adamw> seems like it was punted on 05-11 and then never re-discussed?
15:58:36 <adamw> at least from the comments
15:59:25 <roshi> yeah
15:59:37 <danofsatx> I was just dissapointed in the process where a release-blocking desktop locking up did not qualify as a blocker.
15:59:41 * roshi wonders how this got missed in other meetings?
16:00:41 <danofsatx> I apparently wasn't eloquent (or pushy) enough to get this acted on, either by QA in general, the KDE Sig, or upstream.
16:01:02 <adamw> danofsatx: it looks more like a process breakdown
16:01:08 <roshi> looks like in comment #15 the blocker tracker got removed
16:01:10 <adamw> danofsatx: though the KDE sig does seem to be actively working on it
16:01:26 <adamw> ah, roshi's right
16:01:26 <roshi> so it never showed up in the irc list after that
16:01:27 <danofsatx> yeah, rdieter is driving it.
16:01:56 <roshi> so we never discussed it again because it had gone into stealth mode
16:01:58 <adamw> danofsatx: so the problem is that when doing the secretarialization you took it off the proposed blocker list, so it didn't show up for subsequent meetings
16:02:11 <adamw> probably my fault for not noticing when i reviewed the secretarialization :( sorry
16:02:28 <danofsatx> and my fault for screwing it up (again)
16:02:33 <adamw> ah well, we live and learn :)
16:02:50 * danofsatx vows to never volunteer again (NAVY - Never Again Volunteer Yourself)
16:03:06 <roshi> that seems a bit harsh :p
16:03:32 <roshi> this is a case where a respun live image could make sense
16:03:34 <adamw> but accurate :P
16:03:43 <adamw> potentially i guess yeah
16:03:54 <adamw> but live respins seem like the hardest case to me as you kinda wanna re-do all the validation
16:04:03 <adamw> they're desktops and they're installers
16:04:11 <adamw> anyhow, we're getting over time...
16:04:20 <roshi> I'm kinda working from the assumption that a respin of anything will need a whole run of the matrix
16:05:41 * roshi has nothing else to add
16:06:16 <adamw> #agreed F22 process tweaks like more nightly validation, earlier blocker review and openQA all worked pretty well, we'll continue with them for F23
16:06:56 <adamw> #info https://bugzilla.redhat.com/show_bug.cgi?id=1217844 was a victim of a secretarialization mistake during blocker review, let's remember to double-check our work for F23 :)
16:07:01 <adamw> #topic Open floor
16:07:07 <adamw> anyone have anything for open floor, quickly?
16:07:27 <roshi> just this: great work on F22, folks :)
16:07:55 <adamw> yep! thanks for all the work on f22, everyone!
16:08:51 <danofsatx> nothing here.
16:08:59 <adamw> see roshi's heroes of fedora posts: part 1 is up at https://roshi.fedorapeople.org/horoes-of-fedora-hof-fedora-22-part-1.html
16:08:59 * danofsatx votes for close.
16:09:14 <roshi> part 2 and 3 are forthcoming
16:09:17 <adamw> awesome job satellit on all the validation tests
16:09:31 <adamw> roshi: i note that at present it's actually a Horoes of Fedora post. :P
16:09:40 <roshi> eh
16:09:43 * roshi fixes it
16:10:14 * satellit_e :)
16:10:30 <adamw> ok, thanks again for coming everyone
16:10:36 * adamw sets fuse
16:10:37 <roshi> gotta fire my copy editor...
16:10:47 * danofsatx is looking for a job
16:11:31 <roshi> https://roshi.fedorapeople.org/heroes-of-fedora-hof-fedora-22-part-1.html
16:12:42 <adamw> danofsatx: roshi offers competitive pay and benefits, if by 'competitive' you mean 'none' :P
16:12:53 <adamw> call it an internship!
16:13:03 <roshi> haha
16:13:14 <adamw> #endmeeting