01:01:10 <nirik> #startmeeting How to test Fedora Updates 01:01:10 <zodbot> Meeting started Thu Apr 29 01:01:10 2010 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 01:01:10 <nirik> #meetingname how-to-test-fedora-updates 01:01:10 <nirik> #topic Introduction 01:01:11 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 01:01:14 <zodbot> The meeting name has been set to 'how-to-test-fedora-updates' 01:01:24 <nirik> Welcome everyone. 01:01:41 <nirik> I'm going to be talking about updates and testing in fedora tonight. 01:01:57 <nirik> please do feel free to ask questions or clarifications as we go. 01:02:21 <nirik> I have things split up into 3 high level topics... 01:02:43 <nirik> where we are now with testing of updates in fedora, where we should be soon, and where we hope to be someday. 01:03:23 <nirik> With what we go over tonight you should be able to dive in testing updates now, or help with new efforts down the road. 01:03:57 <nirik> So, first off... some high level questions: 01:04:13 <nirik> Why do we want to test updates? 01:04:45 <nirik> No matter how well an upstream project puts together things, they are unlikely to have the same setup as Fedora. Same compilers, libraries, etc. 01:04:57 <nirik> also, they are unlikely to have the vast number of people using things that we have in fedora. 01:05:28 <nirik> So, testing something to make sure it still works, or fixes a problem it claims to fix is vital. This allows us to not push this update out to everyone until it's really ready. 01:05:51 <nirik> Next one might ask: What are we testing here? 01:06:06 <nirik> It's the unit that we have... packages or groups of packages. 01:06:25 <nirik> This might be a bugfix for a reported bug, a new upstream update, or a brand new package. 01:06:50 <nirik> Finally, the question of How to test a package update comes to mind. 01:07:24 <nirik> We will go over that more inn detail a bit later... but often it's best to use the package as you would normally day to day. Or target testing a specific ifx. 01:07:27 <nirik> fix rather. 01:07:44 <nirik> Any questions on high level goals here? 01:08:11 <nirik> #topic Current Testing Framework 01:08:19 <nirik> So, where are we right now. 01:08:47 <nirik> Let me talk a bit about package updates and how they are created first: 01:09:06 <nirik> 1. A package maintainer checks in a update or patch or fix or new package into fedora's CVS repo. 01:09:23 <nirik> 2. The package is built in the Fedora build system: http://koji.fedoraproject.org/ 01:09:45 <nirik> At this point you can go to the buildsystem and download the package and install it if you wanted to. 01:10:02 <nirik> It would not be signed as coming from the Fedora project, and you would have to manually download and use it. 01:10:35 <nirik> 3. The package maintainer creates an update using the newly built package in Fedora's update system: http://admin.fedoraproject.org/updates/ 01:11:01 <nirik> When this update request is created the maintainer can specifiy things about it: 01:11:13 <nirik> - If it's a group of packages that should be updated together. 01:11:26 <nirik> - If there's any bugs that this specifically fixes and a link to that bug. 01:11:49 <nirik> - If this update should go to the updates-testing repository, or directly to the updates repository. 01:12:14 <nirik> - If going to updates-testing if it should require a specific amount of "karma" to promote to the updates repository. 01:12:23 <fenris02> what stage does autoqa happen? 2.5 ? 01:12:26 <nirik> - If it's a security related bug 01:12:43 <nirik> fenris02: yeah, will talk about that in the next section about where we are hoping to be soon with updates. ;) 01:12:58 <daemoen> why would an update skip over updates-testing and be straight to updates ? 01:13:05 <makfinsky> What exactly does "karma" mean? 01:13:25 <nirik> daemoen: good question. If the bug was a security issue or a very serious one perhaps. It's up to the maintainer currently. 01:13:38 <nirik> makfinsky: will be talking about that in just a minute. ;) 01:13:51 <kmeyer> daemoen: new packages for stable releases, potentially 01:13:53 <nirik> basically it's a "vote", 0, +1 or -1 about the update. 01:14:14 <daemoen> karma i would assume then refers to the integrity/validation of the packages reliability, so we trust it to fix it, it can now reach updates, instead of unreliably pushing a bad patch? 01:14:36 <nirik> daemoen: yep. Thats where testing of the update comes in. 01:15:13 <nirik> 4. The update is now in 'pending' state. The next time folks from release engineering sign and push updates, it will move to updates-testing or updates and mirror out. 01:15:14 <daemoen> so karma is simply put: the way of validating the qa process for a given package. gotcha. 01:15:36 <nirik> yes. It's crude, and planned for improvement, but it's what we have now. ;) 01:16:36 <nirik> 5. If the update is going to updates-testing it will wait there until either the maintainer requests it to go to stable, or it reaches +karma that the maintainer set for that. 01:16:58 <nirik> the default value is +3 karma, but the maintainer can change it to whatever they like as long as it's a positive integer. 01:17:08 <fenris02> how does a link between the admin.fp.o page and bz get made? manual only by maintainer? 01:17:36 <makfinsky> How does the karma increase? Who can increase or decrease the karma points? 01:17:43 <nirik> fenris02: it's added to the update by the maintainer. Maintainers can edit updates and add bugs later... but it has to be the same maintainer that submitted it editing it. (Or an admin) 01:17:49 <makfinsky> Or vote, as it were. 01:17:52 <fenris02> makfinsky, sign in with your FAS account, and you can +/-1 it. 01:18:06 <nirik> makfinsky: Anyone with a Fedora Account system account can login and add karma. 01:18:10 <makfinsky> Oh! 01:18:35 <nirik> so when adding karma to an update, some things to consider: 01:18:47 <nirik> - You should have installed/used/tested it. 01:18:49 <fenris02> anon users are still disqualified from the "+3 to push" auto-rule, right? 01:19:01 <nirik> - If you can confirm a specific bug is fixed, note that. 01:19:16 <nirik> fenris02: yep. Anonymous users can comment, but not add any karma. 01:19:49 <nirik> - If you are giving a -1 to a update, you might consider filing a bug or noting on a bug that it still happens and isn't fixed by this update as claimed. 01:21:04 <Kyril> nirik : Does the updates get removed automatically _if_ it reach -3 ? 01:21:28 <nirik> Kyril: excellent question. Yes, this is also settable by the maintainer, and is -3 by default. 01:21:38 <nirik> it will be 'unpushed' in that case, and disappear from updates-testing. 01:21:48 <nirik> note that this does not matter for stable updates. They always stay. 01:21:58 <nirik> and adding karma to stable updates doesn't change anything. 01:22:21 <fenris02> why would an admin.fp.o page exist for an update that was pushed directly to released updates? 01:22:49 <nirik> fenris02: it's still an update, so it has an update page. This can note security issues, bugs fixed, etc. 01:23:19 <nirik> Any further questions on karma or that setup? 01:24:13 <makfinsky> Good here. Learned something new! 01:24:26 <nirik> ok, some notes on what to do when you are going to test updates-testing packages: 01:24:26 <Kyril> Same ;) 01:24:41 <nirik> - If you can test the specific bug listed with the update, thats always great. 01:25:05 <nirik> Often this is not possible, as it requres specific hardware, or a setup you don't have or you can't easily see how to duplicate it. 01:25:20 <nirik> Occasionally the bug has a test case, which is great. 01:25:29 <fenris02> is there anything linking your FAS account to your smolt page(s) ? 01:25:50 <nirik> fenris02: not currently I don't think. Might be nice to add. 01:26:01 <Kyril> fenris02 : I don't think so 01:26:13 <fenris02> may help to isolate what specific hw sets trigger bugs (like kernel drivers) 01:26:14 <nirik> - If you can test the update and it's something you use day to day thats great... give it a work out and see if anything shows up. 01:26:30 <nirik> This is often what you have to do when it's a version update... just make sure it's still working as you expect. 01:27:10 <nirik> - You can currently test for things like broken dependencies and scriptlets and such. (AutoQA which we will get to in a bit here should take over that soon hopefully) 01:27:49 <nirik> So, the process is a bit fuzzy currently on what you should add +1 or -1 for. 01:29:08 <nirik> It's worth noting that you can get updates-testing updates several ways: You can just enable the updates-testing repo and apply all updates or you can selectively apply them as you see/want to test things. 01:29:34 <nirik> note that since updates-testing is testing updates you can occasionally get broken updates. So be ready if you go this route. 01:30:04 <nirik> ok, any questions on general testing notes? 01:30:19 <Kyril> Mhm, the updates system leave alot of things at the discretion of the pkg. maintainer 01:30:35 <Kyril> not for me ;) 01:30:39 <kmeyer> nirik: As far as choosing what to test, where does one start? 01:30:50 <kmeyer> the full updates-testing feed is probably a bit much for most people 01:30:55 <nirik> Kyril: currenly, yes. 01:31:23 <nirik> kmeyer: well, you can selectively run 'yum --enablerepo=updates-testing check-update' and see whats there against your installed packages. 01:31:33 <nirik> then you can 'yum --enablerepo=updates-testing update foobar' 01:31:38 <nirik> and test just the 'foobar' package. 01:32:31 <nirik> ok, finally I want to talk just a bit about some tools to integrate with bodhi (the updates system) 01:32:45 <nirik> You can of course login and use the web interface to provide your karma... 01:32:59 <kmeyer> nirik: oh, neat. Cool. 01:33:05 <nirik> There is also a command line 'bodhi' client. 01:33:21 <nirik> (it's in the 'bodhi-client' package) 01:33:33 <Kyril> nirik : really ? that's cool :) 01:33:33 <nirik> Maintainers can use this to submit, edit, change updates. 01:33:49 <nirik> Testers can use this to provide karma if they want. 01:34:10 <nirik> There is also another package thats very handy for testing feedback: fedora-easy-karma 01:34:18 * fenris02 installs 01:34:19 * makfinsky is installing bodhi rpms now... 01:34:49 <nirik> fedora-easy-karma will look at your installed packages and see which are from updates-testing and you haven't yet commented on, then it will go thru them one by one and let you add karma/comments. 01:35:12 <nirik> It's very handy for folks who do have updates-testing enabled... otherwise you can't remember which packages you need to comment on. 01:35:30 <nirik> I usually run it ever few days and mass add karma for things I have used in the last few days without problems. 01:35:31 <fenris02> "package-cleanup --orphans" if you didnt enable the whole repo. 01:36:00 <Kyril> nirik : could you give us an exemple of how to use fedora-easy-karma ? 01:36:01 <kmeyer> fenris02: sorry, what's that? 01:36:12 <Kyril> --fas-username 01:36:22 <nirik> Kyril: you install the package and run it... let me paste an example. 01:36:55 <fenris02> kmeyer, for example, "yum update selinux\* --enablerepo=updates-testing" will only enable the testing repo for that run. it will show up as an "orphan" package because it came from a repo you normally have disabled. 01:37:16 <nirik> fenris02: are you sure. I think it knows it came from there... 01:37:17 <kmeyer> I get it, I was curious what "package-cleanup --orphans" did 01:37:31 <fenris02> nirik, it's how i found my list ... might have changed after f12 01:38:13 <nirik> fenris02: 'yumdb info packageyoudidthatwith' 01:38:15 <fenris02> nirik, yumdb should still properly report where it came from though 01:38:22 <nirik> yeah. 01:38:30 <fenris02> package-cleanup appears to ignore that 01:39:01 <fenris02> "feature" perhaps. not sure, but i like it the way it works right now. 01:39:05 <Kyril> nirik : If i'm using updates-testing system wide, should i give some karma to the package that _still_ work 01:39:20 <Kyril> even if i didn't had the issue that the update fixed 01:39:20 <nirik> Kyril: yes. 01:39:23 <Kyril> (regression?) 01:39:31 <nirik> http://fpaste.org/U4NW/ 01:39:38 <nirik> this is an example output from fedora-easy-karma 01:39:53 <nirik> perhaps I should flood the channel with it so it's in logs. 01:40:00 <nirik> Hold your hats. :) 01:40:18 <nirik> ================================================================================ 01:40:18 <nirik> emacs-23.1.94-2.fc13 01:40:18 <nirik> ================================================================================ 01:40:19 <nirik> Update ID: FEDORA-2010-6269 01:40:19 <nirik> Release: Fedora 13 01:40:19 <nirik> Status: testing 01:40:21 <nirik> Type: bugfix 01:40:23 <nirik> Karma: 1 01:40:25 <nirik> Bugs: https://bugzilla.redhat.com/578272 - CVE-2010-0825 emacs, xemacs: Race condition by moving message from user's inbox into user's Rmail file [Fedora all] 01:40:27 <nirik> : https://bugzilla.redhat.com/578267 - CVE-2010-0825 emacs, xemacs: Race condition by moving message from user's inbox into user's Rmail file, when movemail setgid enabled 01:40:29 <nirik> Notes: - Add patch to fix race condition in movemail (tracked as 01:40:32 <nirik> : CVE-2010-0825). It is not a security flaw on Fedora, as setgid bit 01:40:34 <nirik> : is not set on the movemail in Fedora emacs packages. 01:40:36 <nirik> : - Fix Ctrl-C, Ctrl-b in nxml-mode. 01:40:38 <nirik> Submitter: jgu 01:40:40 <nirik> Submitted: 2010-04-01 16:13:46 01:40:42 <nirik> Comments: bodhi - 2010-04-09 04:30:48 (karma 0) 01:40:45 <nirik> This update has been pushed to testing 01:40:47 <nirik> pfrields - 2010-04-16 17:58:19 (karma 1) 01:40:49 <nirik> C-c C-b appears to work in nxml-mode. No other regressions noted to 01:40:53 <nirik> date 01:40:54 <nirik> https://admin.fedoraproject.org/updates/F13/FEDORA-2010-6269 01:40:56 <nirik> inst. RPMS: emacs-common-23.1.94-2.fc13.x86_64 - Emacs common files 01:41:00 <nirik> : emacs-23.1.94-2.fc13.x86_64 - GNU Emacs text editor 01:41:00 <nirik> Comment? -1/0/1 ->karma, other -> skip> 01:41:02 <nirik> so, you can see all kinds of info from this update. 01:41:07 <nirik> it's already gotten one +1 karma 01:41:19 * nirik hopes he didn't flood off. 01:41:20 <fenris02> nirik, if you put '1' or '-1' does it secondarily prompt your for why? 01:41:31 <nirik> yep. It asks for your comment then. 01:41:45 <fenris02> neat! 01:41:46 <nirik> and will log you in to your account if you haven't logged in yet. 01:41:53 <kmeyer> neat. 01:42:02 <Kyril> nirik : thanks i was wondering if i did something wrong but it turns out it take an instant to get fedora-easy-karma started :) 01:42:14 <nirik> so, with this update, if you were an emacs user and things were working fine and/or you could test the specific bugs you could +1 karma it. 01:42:33 <nirik> or if it didn't install right, or broke things, or you could confirm one of the bugs not fixed, you could -1 it. 01:42:47 <nirik> 0 karma votes are to allow you to place a comment without saying you tested it at all. 01:42:52 <fenris02> hm. no ~/.config-type file for bodhi-client? that's not ideal to have your user/pass in the environment 01:42:57 <nirik> perhaps to ask the maintainer something or note a possible issue. 01:43:36 <nirik> fenris02: I think it saves info in ~/.fedora/.fedora-session, but I am not sure. 01:43:47 * fenris02 looks 01:44:07 <Kyril> nirik : Do you know if the password is being saved somewhere with fedora-easy-karma ? 01:44:12 <Kyril> (by default) 01:44:17 <Kyril> I haven't looked at the options yet 01:44:31 <nirik> Kyril: no, I think it saves a session cookie, but thats it. You will have to enter your password if your session has timed out. 01:44:50 <nirik> any other questions about the current setup for updates? 01:45:49 <nirik> ok, lets talk quickly about what tomorrow holds, and then the great handwavy future. ;) 01:45:56 <nirik> #topic Tomorrow's Testing Framework 01:46:09 <nirik> by tomorrow here I mean "soon", not litterally tomorrow. ;) 01:46:28 <nirik> One of the big things that will hopefully land before too long is AutoQA. 01:46:37 <nirik> Some of the testing we do currently can be automated. 01:46:54 <nirik> Things like dependency checks, or does the package install right, or does it have some broken scriptlet, etc. 01:47:04 <nirik> https://fedoraproject.org/wiki/AutoQA 01:47:12 <nirik> is working to setup a framework for these tests. 01:47:45 <nirik> Probibly to start it will be simple things like conflicts, deps, etc... but down the road each package could have tests they would like to run. 01:48:02 <nirik> we could check all kinds of things that are possible to be checked in an automated way. 01:48:40 <nirik> I think it's planned that this would hook into several places in the current setup: 01:48:47 <nirik> - when the update is submitted 01:48:57 <nirik> - when the update is going to be pushed to updates-testing 01:49:04 <nirik> - when the update is going to be pushed to updates. 01:49:25 <nirik> Some of the tests would be advisory, some would be blockers. 01:49:48 <nirik> It's great stuff. anyone interested in it should get involved. ;) 01:50:07 <nirik> Any AutoQA questions? 01:50:34 <makfinsky> Yeah, when will we get 48 hours in a day? 01:50:44 <kmeyer> Hah! That was my question :P 01:50:49 <makfinsky> So much cool stuff to work on! 01:50:56 <nirik> yeah, no doubt. ;) 01:50:59 <igndenok> agree 01:51:14 <nirik> Anyhow, another short term thing coming is some changes to the current setup for updates: 01:51:28 <nirik> https://fedoraproject.org/wiki/Package_update_acceptance_criteria 01:51:28 <nirik> https://fedoraproject.org/wiki/QA:Package_Update_Acceptance_Test_Plan 01:51:42 <nirik> Basically this adds several things: 01:51:45 <nirik> - AutoQA 01:51:50 <makfinsky> Well, I do have one question - how much of the current process will change with autoqa? 01:52:01 <makfinsky> Will there still be karma, comments? 01:52:04 <nirik> - A set of Critical path / important updates. 01:52:09 <fenris02> hopefully, autoqa happens before karma imho 01:52:10 <nirik> makfinsky: yes. 01:52:48 <nirik> I don't know if it will just add karma, or just drop the update if it fails. 01:53:04 <nirik> but it will work with the existing bodhi setup. 01:53:35 <nirik> So, critical path are packages that are "important". They merit more testing. 01:53:54 <fenris02> dbus for example 01:54:06 <nirik> There will be a group called 'proventesters' of test folks who have shown that they know what they are doing. 01:54:21 <nirik> criticial packages will need +1 from one of them and also other people. 01:54:56 <nirik> - Finally this will add a requirement that packages that don't reach their karma will have to spend one week in updates-testing. 01:55:08 <nirik> This means maintainers will not be able to push directly out to stable. 01:55:29 <nirik> So, these changes should land somewhat soon. 01:56:17 <autopsy> Sounds good. 01:56:17 <nirik> any questions on those? 01:56:23 <N3LRX> What if it is a security update that requires immediate attention. Will those updates be delayed a week by this system? 01:57:07 <nirik> N3LRX: they would be delayed until they received sufficent karma. 01:57:20 <nirik> but with a pool of proventesters it should be pretty easy to get that. 01:57:46 <nirik> They could even get tested/karma while they were in pending state and then get pushed direct to stable. 01:58:09 <fenris02> how does one join the *testers group(s) ? 01:58:19 <nirik> https://fedoraproject.org/wiki/QA/JoinProvenTesters:Draft 01:58:33 <nirik> (this has since been approved, but I don't think it's been moved to the approved space yet) 01:59:01 <nirik> anyone with a fas account can comment / provide karma now. You don't need to be in any group. 01:59:14 <kmeyer> nirik: do we have any statistics on how many uncommon / "leaf" packages ever reach their karma before being pushed? 01:59:15 <nirik> to get into proventesters you need a mentor and then they promote you. 01:59:27 <nirik> kmeyer: sadly not off hand. 01:59:42 <kmeyer> also: under this system, can one simply set the desired karma at "0"? 01:59:45 <nirik> under this new setup, they could go to stable after a week in testing. 01:59:55 <nirik> kmeyer: no. You could set it to 1 though. 02:00:22 <kmeyer> are updates pushes immediate nowadays? does it still require manual intervention by an infra person? 02:00:59 <nirik> kmeyer: once a day weekdays. 02:01:07 <nirik> it does require rel-eng to sign the packages. 02:01:07 <kmeyer> (a week becomes a week and a half or two weeks if someone has to do it manually, at least historically) 02:01:17 <kmeyer> ah, ok 02:01:21 <nirik> updates have been pushing out every weekday for a long time now. 02:01:44 <nirik> so at most it could delay you 2-3 days... if you submitted something friday night and didn't get pushed until monday. 02:01:57 <kmeyer> ok, cool 02:02:04 <kmeyer> yeah, my memory of that was fuzzy at best 02:02:04 <nirik> ok, we are over time, but I wanted to touch just a tad on handwavy furture. 02:02:11 <nirik> #topic The great Future 02:02:19 <nirik> Someday we are looking to: 02:02:45 <nirik> - Replace the simple +1/-1/0 karma with more. So you could note regressions, or other things. 02:03:16 <nirik> - Have test plans for every package. This would allow you to install an update, go to a test plan page and be able to really test some specific app. 02:04:00 <nirik> - Have more guidence to maintainers as to when to push updates in the first place or when not to. Currently everything is pretty much up to the maintainer, but each maintainer has somewhat different ideas, so things appear muddled sometimes to users. 02:04:09 <nirik> - Tons more autoqa testing. 02:04:36 <nirik> Lots of things to strive for there. ;) 02:04:43 <nirik> #topic Q&A 02:04:55 <nirik> any general questions? 02:06:02 <nirik> ok, thanks for coming everyone. 02:06:07 <nirik> I hope it was informative... 02:06:07 <makfinsky> Thanks nirik! 02:06:10 <fenris02> thank you nirik ! 02:06:13 <kmeyer> test plans sound great 02:06:15 <nirik> and that all of you will go test some packages now. ;) 02:06:15 <igndenok> thanks nirik :) 02:06:17 <kmeyer> thanks for the lesson ;) 02:06:17 <N3LRX> Thanks nirik, great class 02:06:18 <makfinsky> Got lots to read through and start learning. 02:06:23 <kmeyer> nirik: already did! 02:06:35 <nirik> kmeyer: excellent! 02:06:37 <kmeyer> :) 02:06:49 <nirik> If anyone has further questions on this, #fedora-qa is the channel to be. :) 02:06:54 <silentusr> nirik: Thanks for everything! :) 02:07:07 <nirik> #endmeeting