13:30:53 <bkm> #startmeeting Testing bleeding edge kernels
13:30:53 <zodbot> Meeting started Wed Aug  3 13:30:53 2016 UTC.  The chair is bkm. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:30:53 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
13:30:53 <zodbot> The meeting name has been set to 'testing_bleeding_edge_kernels'
13:31:02 <bkm> speaker Paul Moore
13:31:10 <michalrud> #meetingname flock2016
13:31:10 <zodbot> The meeting name has been set to 'flock2016'
13:31:20 <bkm> #topic introduction
13:31:41 <bkm> pmoore@redhat.com or paul@paul-moore.com
13:32:00 <bkm> SELinux, audit, labeled networking kernel maintainer
13:32:13 <bkm> Created and maintain libseccomp project
13:32:43 <bkm> Talk about kernel testing problems and improvements
13:33:30 <bkm> How do we make leading edge changes accessible?
13:33:42 <bkm> #topic kernel development cycle
13:34:17 <bkm> * Create patch
13:34:32 <bkm> * get it reviewed and changes made
13:34:47 <bkm> * Patches merged into a subsystem next branch
13:35:09 <bkm> this varies by subsystem
13:35:53 <bkm> * Linus pulls the next branches into tree
13:36:19 <bkm> 2 week merge window after each kernel is released
13:37:07 <bkm> Approximately 8 week of testing before release
13:37:42 <bkm> #topic Kernel testing today
13:38:32 <bkm> * Kernel releases packaged for Fedora Rawhide
13:39:38 <bkm> * linux-next repository merges 'next' branches daily
13:39:59 <bkm> * Primarily focused on exposing merge conflicts
13:40:43 <bkm> * Coverage unclear
13:41:01 <bkm> * General user testing rare
13:41:27 <bkm> * As such does not find subsystem interaction problems
13:41:41 <bkm> #topic Problems with current approach
13:42:22 <bkm> Less than 8 weeks to find and solve problems
13:44:00 <bkm> 'Next' branches based on stable/old code
13:44:50 <bkm> Software development problem
13:45:25 <bkm> No way to easily test subsystem interaction
13:46:03 <bkm> #topic Kernel testing aspirations
13:46:36 <bkm> Regular testing of the latest upstream developments
13:47:40 <bkm> Make test kernels more accessible to users - not all users are comfortable building kernels, but can test out rpms
13:48:06 <bkm> Needs to be cheap, easy and portable
13:49:05 <bkm> * Automated build and test processes
13:49:50 <bkm> #topic continuous integration as a solution
13:50:25 <bkm> Connect development with test and deployment
13:50:56 <bkm> Constant stream of new, verified releases
13:51:12 <bkm> #topic Continuous integration problems
13:51:38 <bkm> Many CI frameworks require infrastructure
13:52:51 <bkm> Does not appear well suited to kernel testing
13:53:07 <bkm> #topic Simplified CI for kernel developers
13:54:27 <bkm> Create/find a set of easily run regression tests: self contained, can isolate individual tests
13:54:50 <bkm> Automate patching of Fedora kernel source rpm
13:54:57 <bkm> Automate building
13:55:12 <bkm> Automate ROM distribution to public repository
13:55:50 <bkm> Automate testing via virtual machines
13:56:13 <bkm> #topic Progress so far
13:56:38 <bkm> Useable test suite for SELinux and audit
13:57:01 <bkm> * updated selinux-testsuite project
13:57:16 <bkm> * created audit-testsuite since no suitable existing one
13:57:27 <bkm> Codes on github
13:57:51 <bkm> Fedora kernel patching automation
13:58:04 <bkm> Copr repository for kernel build and distribution
13:58:26 <bkm> Highly recommend Copr
13:58:41 <bkm> * Weekly builds and test runs for the past year
13:58:55 <bkm> #topic Lessons learned
13:59:15 <bkm> Regular regression testing works - if you do it!
13:59:42 <bkm> * Catches problems early
13:59:55 <bkm> * Minimizes time looking through git log
14:00:33 <bkm> * Less stress during merge window and RC cycle
14:01:15 <bkm> Automated patching works better than expected
14:01:29 <bkm> * Looking at automating further
14:01:59 <bkm> Copr great tool, but has reliability issues
14:02:12 <bkm> * Expect this to improve
14:02:19 <bkm> #topic Future improvements
14:02:39 <bkm> Automate more
14:02:47 <bkm> * VM test execution
14:03:02 <bkm> * Automatic build and test triggers
14:03:26 <bkm> Increase test coverage
14:04:03 <bkm> * Tests need to be easily and quickly run
14:04:28 <bkm> * Not comprehensive tests, aim to catch simple mistakes early
14:04:52 <bkm> Additional security subsystems beyond SELinux and audit
14:05:12 <bkm> #topic PRoject links
14:05:41 <bkm> #link https://github.com/SELinuxProject/selinux-testsuite
14:06:03 <bkm> #link https://github.com/linux-audit/audit-testsuite
14:06:13 <bkm> Bash scripts
14:06:44 <bkm> #link https://github.com/pcmoore/copr-pkg_scripts
14:07:21 <bkm> #link https://copr.fedoraincloud.org/coprs/pcmoore/kernel-secnext
14:07:59 <bkm> #topic Questions
14:08:05 <bkm> Blog post coming
14:08:40 <bkm> #topic Example installing and running
14:09:08 <bkm> see readme at pcmore
14:09:37 <bkm> Install and configure Copr client - makes hosting repository
14:10:02 <bkm> Clone github repository pcmoore/copr-pkg
14:11:19 <bkm> Create and configure new Copr project directory
14:11:31 <bkm> Clone the upstream project and add any remotes
14:11:43 <bkm> Clone the fedora package repository
14:12:33 <bkm> Welcomes contributions - email, pull requests etc
14:12:55 <bkm> Then run scripts
14:13:06 <bkm> * Generate patches from upstream
14:13:34 <bkm> * Create a patched SRPM and submit to Copr
14:14:15 <bkm> Read the documentation, more configuration options there
14:15:28 <bkm> #topic Questions
14:16:08 <bkm> Want to extend to architectures other than x86_64
14:17:20 <bkm> Power and ARM also of interest
14:17:41 <bkm> Could use Koji as well
14:17:53 <bkm> #endmeeting