15:00:55 #startmeeting 15:00:55 Meeting started Thu Oct 22 15:00:55 2009 UTC. The chair is mchua. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:55 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:01:02 * rwmjones has no idea what this involves ... 15:01:03 Ah, I love automatic logging. 15:01:33 hey, markmc! 15:01:39 hey mchua 15:01:48 huh, maybe we did get the right time after all, rwmjones. 15:01:58 nah, I just pinged 'em 15:02:11 * markmc was just as confused as everyone else 15:02:17 cdub can't make it for an hour 15:02:20 bloody timezones 15:02:37 Ok. I plan on sticking around for both this hour and the next to catch both times. 15:02:47 Let's start the first round, though. 15:02:51 cool 15:02:56 ok 15:02:58 (Anyone else coming right now?) 15:03:10 I just pinged danpb 15:03:10 For background: this interview is for the F12 feature profiles 15:03:16 #link https://fedoraproject.org/wiki/Category:F12_in-depth_features 15:03:18 lutter, rwmjones and me so far AFAICT 15:03:29 you can see an example here to get an idea of what we're shooting for 15:03:33 #link https://fedoraproject.org/wiki/SystemTap_in_Fedora_12 15:03:48 mchua, we're mainly virt people 15:03:49 howdy 15:04:09 rwmjones: Ayup. This interview is for the roundup of virt features in F12 15:04:13 #link https://fedoraproject.org/wiki/Virtualization_improvements_in_Fedora_12 15:04:38 So why don't we start with everyone introducing themselves briefly, and giving a sentence or two about what they do, and what virt features they worked on for F12? 15:05:02 rwmjones, I'm a software engineer at Red Hat, and I am working on http://libguestfs.org/ 15:05:08 (Don't worry about inter-conversation threading; I'll clean up the transcript later so the order makes sense) 15:05:44 libguestfs is a set of tools which you can use to examine and modify virtual machine images from outside (ie. from the host) 15:06:19 so for example if you had an unbootable guest, you could try to fix it by doing: virt-edit myguest /boot/grub/grub.conf 15:06:22 I'm an engineer at Red Hat, joined from Sun nearly 6 years ago. Previously worked on GNOME desktop related stuff, but have been working on virtualization for the past few years. 15:06:36 rwmjones: what would sysadmins have to do to fix that before libguestfs arrived? 15:06:41 mchua: David Lutterkort, software engineer at Red Hat, worked on http://fedorahosted.org/netcf (for the Network Interface Mgmt feature), in the past worked on ovirt and some of the virt-install tools 15:07:13 mchua, that's really tricky ... it was sort of possible using tools like kpartx and loopback mounts, but it was dangerous stuff, hard and you had to be root 15:07:13 markmc: Awesome. And you've been doing a lot of KVM work for F12 - can you tell us a little about that? 15:07:16 For Fedora 12, I worked on the NIC Hotplug and Stable Guest ABI features, along with packaging, bug triaging and general shepherding of all the other virt bits 15:07:30 mchua: besides that, work some on http://deltacloud.org/, and http://augeas.net/ 15:08:01 I work upstream on both qemu and libvirt, but at lot of my time is taken up by Fedora work these days. 15:08:02 now there's no root commands needed, and it's organized as nice little command line tools for each task with proper manual pages 15:08:21 lutter, markmc: can you explain the features that you worked on a bit, and why they make the lives of sysadmins easier? 15:09:11 I'd point people to the home page -- http://libguestfs.org/ -- to see lots of examples, and documentation 15:09:13 Okay, the NIC hotplug feature - the ability to add a new virtual NIC while the guest is running - was a pretty obviously missing feature from our KVM support previously. 15:09:51 The problem we had with implementing it, is that libvirt is responsible for configuring the virtual NIC and passes a file descriptor to the qemu process when it starts it. 15:10:12 mchua: Network Interface Mgmt lets sysadmins set up fairly complex network configurations (e.g. a bridge with a bond enslaved) through a simple description of the config, using the libvirt API; in the past, that required initimate knowledge of ifcfg-* files and a lot of nailbiting. Having an API also means that such setups can be done by programs (e.g., centralized virt mgmt software or virt-manager) 15:10:27 rwmjones: That's fantastic. How do libguestfs capabilities in Fedora compare with how a sysadmin might do the same thing on other, non-Linux (or linux-but-on-another-distribution) platforms? Are there other similar tools? 15:10:35 That's much harder to do when the guest is already running. So, most of the work involved some scary UNIX voodoo to allow passing that file descriptor between two running processes. 15:11:03 As for use cases, people often want to add and remove hardware from their guests without re-starting them. 15:11:15 You might want to add a guest to a new network, for example. 15:11:15 danpb_ltop: are you here for the virt interviews too? 15:11:53 mchua, we've worked with Guido Gunther from Debian on getting a parts of libguestfs packaged up for Debian. On Windows, Microsoft offer something called DiscUtils.Net which is similar but not nearly as powerful. So I'm confident Fedora is well ahead of everyone here. 15:12:22 mchua, was that interview you did for f11 published anywhere? would be good to link to it from https://fedoraproject.org/wiki/Virtualization/History 15:13:05 lutter: Awesome. If I'm understanding you right, this means that now sysadmins can automate complex custom network configurations for VMs? 15:13:25 Now, the Stable Guest ABI feature is really quite boring, but is about preparing KVM so that we can maintain compatibility across new releases. 15:13:47 markmc: I'm sure sysadmins appreciate that too. :) 15:13:47 The idea is that if you are running a Fedora 12 KVM host and you install a new host with Fedora 13 ... 15:14:19 markmc: I didn't do the F11 interview, but that's a good idea - I'll find it when we're done and do that. 15:14:19 ... you might like to migrate your running guests from the Fedora 12 host to the Fedora 13 host, without re-starting them. 15:14:30 #action mchua find F11 virt interview, link to https://fedoraproject.org/wiki/Virtualization/History 15:14:55 mchua: complex network configs on the host, generally ... a common request is 'how do I share a physical NIC between various VM's'; in the past, you had to manually go and edit ifcfg-* files. libvirt now has an API and XML description to make that setup much easier. The backend for the libvirt interface API is netcf, which is independent of virtualization, so you could use that to setup network configs in your VM's 15:15:04 Now, as we add new features to qemu in Fedora 13, we might end up 'upgrading' the virtual machine's hardware. 15:15:22 We might, for example, emulate a new chipset by default or add a new default NIC. 15:15:26 lutter: Ahhh, okay - thanks for the clarification. 15:15:51 The Stable Guest ABI feature means that when you migrate to the Fedora 13 host, the hardware emulated by qemu will remain the same for that guest. 15:15:54 lutter: How does this compare to how people would set up host network configs on other platforms? 15:16:19 mchua: right now this is exposed in the libvirt API; we're working (well, Cole Robinson is working) on exposing that in virt-manager so that people can say 'use this physical NIC for all my VM's' with one click 15:16:22 As you can imagine, if you change around the hardware under a running guest, the guest may get seriously confused. 15:16:34 rwmjones: Do you want to talk about the guestfish interface a bit? 15:17:04 But it's not just about live migration - if you upgrade your host and restart your guest, not all guest OSes will like if you've changed around the hardware. 15:17:18 mchua: there you either have to manually edit the network configs, which generally is only really possible for humans, not programs, or rely on the very dodgy, never-quite-right Xen networking scripts 15:17:29 Windows for example, with significant enough changes to the hardware, will require you to re-validate your license. 15:17:31 * mchua realizes this channel is currently a confusing hodgepodge of parallel conversations. Will be pulling back to get a broader virt picture with everyone right after the "descripts of individual features" threads finish up. 15:17:35 mchua, sure ... guestfish is one of the ways to get access to the libguestfs features, for use from shell scripts. The basic usage is to do: 15:17:51 We want to avoid that happening when you upgrade your Fedora host. 15:17:51 guestfish -i yourguest # where yourguest is some guest name known by libvirt 15:18:00 mchua, from my POV, it's working fine so far :-) 15:18:09 * markmc goes back and reads what the others have been saying 15:18:22 and that gives you a shell where you can list files in the guest, edit them, look in directories, find out what LVs the guest has (or create new ones) ... literally 200 commands 15:18:42 that's all documented here: http://libguestfs.org/guestfish.1.html 15:19:09 rwmjones: Wow. That documentation is gorgeous. 15:19:12 and if you run out of ideas, we have some "recipes" you can try out with guestfish: http://libguestfs.org/recipes.html 15:20:03 mchua, we've certainly all been put to shame by rwmjones docs :) 15:20:32 the pwoer of OCaml ;) 15:21:01 lutter: Is there a place where our readers can go to find out more about how to use the libvirt API? How do folks try these features out? 15:21:07 mchua, found it : http://jaboutboul.blogspot.com/2009/05/fedora-11-virtualization-reality.html 15:21:12 markmc: thanks! 15:21:25 Ah, Jack did the interview, coolness. 15:21:30 #link http://jaboutboul.blogspot.com/2009/05/fedora-11-virtualization-reality.html 15:21:33 (for meetbot) 15:22:02 mchua: there's a small amount of docs on the netcf site (I have to add more) and libvirt.org has API docs for the various virInterface* calls 15:22:03 lutter: I see instructions on how to test at https://fedoraproject.org/wiki/Features/Network_Interface_Management#How_To_Test 15:22:06 * mchua nods 15:22:28 lutter: Is there a place folks should be watching to see things go up as the F12 GA date approaches? 15:22:59 mchua: there's also a blog post somebody else wrote on netcf http://linux-kvm.com/content/netcf-silver-bullet-network-configuration 15:23:06 Nice! 15:23:31 * mchua would like to do a screencast walkthrough exercising all these cool new virt features at some point 15:23:37 (which is one reason why I'm asking for doc links right now) 15:23:40 mchua: besides bz ? ;) 15:24:01 lutter: *grin* what components should we be keeping track of? 15:24:30 mchua: I don't know of a good central place where this gets summarized, though FWN has been pretty good reporting about virt features. Besides that, watching the individual projects is everybody's best bet 15:24:42 lutter, rwmjones, markmc: in a moment, I'd like to pull back and have the three of you talk with each other about how virt in Fedora has progressed in the past few releases. 15:25:16 mchua, one sec - I'll cover gpxe and qcow2 featurehs 15:25:24 mchua, I would say that in Fedora 6 which is where I really started off with Fedora, it was quite primitive and unfriendly, although we did have virt-manager which has always been a nice tool 15:25:24 markmc: ok. 15:25:25 mchua, the feature owners aren't here 15:25:38 mchua: libvirt, libguestfs, virt-install, virt-manager are the most important ones from a user's POV 15:25:38 * mchua nods 15:25:51 lutter: the user typically being a sysadmin? 15:25:55 (in this case) 15:25:57 okay, the gPXE feature is about replacing the boot ROMs used by qemu for PXE booting with newer versions, basically 15:26:14 etherboot was the name of the project previously, but it's now called gPXE 15:26:30 mchua: virt-manager is definitely for end users, not just sysadmins; virt-install somewhere in the middle, the others get fairly technical 15:26:30 markmc: thanks - I wasn't sure who'd be showing up for the 2nd interview session, but more coverage is always good 15:26:53 rwmjones: What was the F6 virt experience like? 15:27:22 mchua, here's a guestfish example ... making a backup of /home from a Debian guest: 15:27:30 # guestfish -i --ro Debian5x64 15:27:31 Welcome to guestfish, the libguestfs filesystem interactive shell for 15:27:31 editing virtual machine filesystems. 15:27:31 Type: 'help' for help with commands 15:27:31 'quit' to quit the shell 15:27:31 > cat /etc/debian_version 15:27:31 It's important that we made the switch to gPXE because all future upstream development (new features, bug fixes) will go into gPXE instead of etherboot. 15:27:33 squeeze/sid 15:27:35 > tgz-out /home home.tar.gz 15:27:47 lutter: What would be a use-case for an end-user using virt-manager? (I'm guessing there will be users reading this interview who may not have tried out virt stuff before, but who might read this and go "ooh, hey..." and try it out.) 15:28:28 the qcow2 performance feature was about taking a cold hard look at the qcow2 file format and fixing an major bottlenecks 15:28:33 basically, we see qcow2 as a very useful format for virtual machine images 15:28:45 mchua: try out rawhide w/o the risk of breaking your current system 15:29:15 e.g. the size of qcow2 files is determined by the amount of disk space used by the guest, not the entire size of the virtual disk we're presenting to the guest 15:29:30 i.e. the images should be smaller on disk, even if you copy them between hosts 15:29:37 mchua: of course, that goes for any $OS ... in general, virt-manager is a graphical user interface to most/all virt features 15:29:40 mchua, Fedora 6 -> 12 .. it's a story of everything improving dramatically. It's not really that there are big new features eg. we have virt-manager back in 6, but modern virt-manager is just far better. 15:29:45 also, qcow2 supports a "copy on write" feature 15:30:01 markmc: (to backtrack a bit) why the switch from etherboot? (From what I've read, it sounds like the switch was actually requested by the etherboot upstream, in part.) 15:30:01 whereby you can base multiple guest images from the one base image 15:30:27 and I've been trying to work on making it better for sysadmins who want to automate things, hence libguestfs is very shell-script / automation-friendly 15:30:32 so you can reduce disk space further by installing one guest image, creating multiple qcow2 images backed by the first image 15:30:40 and yet, the guest can still write to their disks 15:30:59 so, in summary, we want more people to use qcow2, but they couldn't because the performance was poor 15:31:23 Kevin Wolf put serious effort in upstream to iron out those kinks and obtain a serious speedup 15:31:29 figures are in a table on the feature page 15:31:59 mchua, yes, the etherboot project is no more; it is deprecated in favor of gPXE 15:31:59 lutter: Ok - imagine I'm a new Fedora user, I've just installed F12, love it, want to get a preview of rawhide so I can see what's coming for F13. What do I need to install/run to get rawhide running in a VM? (If that process is quick and painless enough to put in a few "try this!" lines mid-interview.) 15:32:39 lutter: (I realize this is a pretty basic question, but I'd like to get virt used by as many folks as possible so that hopefully we'll have some of those folks going deeper and trying out the tools you've made) 15:32:45 mchua, but they're not completely identical, so there was some significant work involved ... done by Glauber Costa (our Brazilian joker) and Matt Domsch from Dell (AFAIR) 15:33:01 markmc: is gPXE being used by other OSes and distros too? 15:33:06 yeah, it was Matt Domsch 15:33:09 mchua: lemme dig around 15:33:29 mchua, it may be used by other distros, I'm not 100% sure about that 15:33:51 mchua, I'd imagine we're slightly ahead of the curve on this - upstream qemu is still using etherboot images 15:33:57 rwmjones: So one area of improvement between F6 virt and F12 virt is "F12 virt is far more automatable and shell-script friendly." 15:34:08 mchua: lvcreate -n F13Rawhide -L 10G vg_yourhost; virt-install -v -n F13Rawhide --accelerate -r 512 -f /dev/vg_yourhost/F13Rawhide -c /tmp/Fedora-13-netinst.iso 15:34:19 mchua, yeah I'd say that's true 15:34:35 rwmjones: "It's not really that there are big new features... but [features are] just far better" - so you can do the same things, more or less, just much faster (in terms of sysadmin-headache-time needed)? 15:34:39 rwmjones, hmm, no - I'd point people at virt-manager 15:34:56 lutter: thanks 15:35:01 mchua, go to Applications -> System Tools -> Virtual Machine Manager 15:35:10 yeah virt-manager will be easier ... 15:35:44 mchua, (well, first install the 'Virtualization' group in Add/Remove Software) 15:35:53 mchua, then click on New VM 15:36:10 mchua, well there are a lot of big new features behind the scenes (KVM, KSM, virtio ...). It's not clear how apparent they'll be to end users, but it will just all work better and faster. 15:36:10 mchua, choose a name for the guest, choose network install 15:36:26 lutter: ^^ (I think we've got it, no worries) 15:36:55 mchua, and then add a URL like http://download.fedoraproject.org/pub/fedora/linux/releases/12/Fedora/x86_64/os/ 15:37:12 mchua, after that, the instructions in the wizard should be fairly self explanatory 15:37:19 mchua: yeah, what markmc said 15:37:22 * mchua will make a video for the "how to try out virt" procedure in the next week or two 15:37:43 mchua, there's a story behind virt-df (http://libguestfs.org/virt-df.1.html). When I used to manage a bunch of virtual machines at my previous job, it was the tool that I wanted. It didn't exist, so at Red Hat, I wrote it. 15:37:57 mchua, the big change between F6 and F12 is that we've switched from Xen to KVM 15:38:34 mchua, but because all our work is based on the libvirt abstraction layer, the tools used in F6 for using Xen should be familiar to people using KVM in F-12 15:38:37 lutter: (feel free to chime in with any thoughts on how virt's changed from F6 to F12) 15:38:58 mchua, we've also put a significant emphasis on improving security over the last number of releases 15:39:13 mchua, danpb has more details on the security efforts in his F-11 interview 15:39:13 mchua, yeah ... someone on F6 who was using virt-manager or "virsh list", will be using exactly the same commands in F12, even though the hypervisor is completely different 15:39:23 * mchua nods 15:39:27 mchua, and he'll also have more details wrt. the VirtPrivileges feature 15:40:00 * mchua will look. All this content is likely to get split up across several things - we'll have a lot of virt marketing for F12 15:40:05 - feature profile(s) 15:40:09 - screencasts of features 15:40:19 mchua: libvirt, and therefore the whole virt tool stack now manages a much broader area of virt related aspects, not just VM lifecycles 15:40:21 - walkthroughs/docs of before & after comparisons 15:40:33 - virt-through-the-fedora-releases overview/retrospective 15:41:08 mchua, lutter has a good point - we now have tools for e.g. managing networking and storage 15:41:14 mchua, we also have much better support for remotely managing virtualization hosts 15:41:41 mchua, e.g. you can point virt-manager at a host, create a guest on that host, create storage for the guest, configure the network etc. 15:42:12 mchua: the tools are now a prety solid basis for datacenter virt management software like ovirt and RHEV-M 15:42:18 mchua, wrt. fedora virt changing over the years, we're also pushing very hard to adopt new virtualization hardware features introduced by vendors 15:42:40 mchua, so, for example, in F-11 we introduced VT-d support and in F-12 we're introducing SR-IOV support 15:43:01 mchua, and KVM itself is based on Intel and AMD hardware virtualization 15:43:09 mchua, also EPT/NPT support 15:43:38 mchua, so yeah, we're definitely leading the field in terms of shipping support for new hardware features 15:44:00 mchua, e.g. AFAIK no-one else (not even other hypervisor vendors) are yet shipping SR-IOV support 15:44:06 ... 15:44:17 mchua, okay, for the other features, I think we'll have: 15:44:31 cdub do KSM, huge pages and SR-IOV 15:44:34 yeah, Fedora is very likely the first place where you see a lot of new hardware virt features supported in OSS, mostly since so many upstream maintainers/developers for virt-related stuff work at RH and generally push their work to Fedora 'by default' .. spin that any way you want to avoid a distro war ;) 15:44:49 danpb do VirtPrivileges, VirtStorage and libvirt TCK 15:44:56 mchua, and then that'd be all the features covered 15:45:13 All while maintaining a consistent, familiar interface - as rwmjones pointed out, folks using virt-manager and virsh on F6 are still using the same commands. Though now they also have the option to use additional tools like guestfish to script the process (so, alternative-but-even-easier interface). 15:45:30 markmc: awesome. thanks! 15:45:40 mchua: we also added the capability to deploy and build appliances (through virt-install/virt-image and the thincrust project) 15:46:13 rwmjones_, markmc, lutter: two last questions I wanted to toss out: (1) what's coming up for virt in f13 and the future? and (2) what do you folks do for fun when you're not hacking on virt stuff? 15:46:16 mchua, oh, "Cloud" 15:46:27 (And then that's all I had - and then whatever else you'd like to talk about.) 15:46:27 mchua, none of us said that yet, how silly of us 15:46:31 cloud, cloud, cloud 15:46:35 * markmc gets it in a few times 15:46:37 for good effect 15:46:40 mchua, yeah the outlook is cloudy 15:46:56 * mchua chuckles 15:47:16 haha .. yeah ... everybody watch deltacloud.org 15:47:21 mchua, fedora based cloud project : http://deltacloud.org/ 15:47:25 lutter, :) 15:47:42 mchua, https://fedoraproject.org/wiki/Category:F13_Virt_Features 15:47:52 * mchua grins. We'll keep an eye on cloud for F13. 15:47:56 * rwmjones_ trolls OCaml features to C programmers 15:48:05 mchua, VHostNet is maybe the most exciting there so far 15:48:05 mchua: and virt datacenter mgmt along the lines of ovirt 15:48:11 mchua, we'll be adding more feature pages as time goes on 15:48:30 mchua, VHostNet is about handling virtio networking in the kernel, rather than in the qemu process 15:48:48 mchua, so network traffic goes straight from the guest to the kernel out to the network 15:48:55 rwmjones_, lutter, markmc: Whoa. Documentation and project webpages and a list of F13 features and *everything.* You folks are awesome. 15:48:58 mchua, without ever being diverted through the qemu process 15:49:17 mchua, Red Hat's Michael Tsirkin is busy getting that feature into the 2.6.33 kernel 15:49:28 * lutter blames markmc and rwmjones and a bunch of other people 15:49:53 mchua, also, the VNCResourceTunnel means we'll get sound from guests again, which would be nice :) 15:50:15 mchua, not sure how to spin that so it doesn't sound like "uh, we suck at audio, we're going to try a little harder" though :) 15:50:34 mchua: there's a much bigger group within RH working on all these virt features .. might be worth a mention; it's far from being just us 3 or 5 15:50:38 markmc: we'll figure something out :) 15:50:49 lutter: absolutely. who else should we be giving a shout-out to? 15:50:58 lutter: I have the folks listed on https://fedoraproject.org/wiki/Virtualization_improvements_in_Fedora_12 so far 15:51:11 mchua, chris lalancette too 15:51:12 mchua, lutter's dead right - there's a huge list of people working upstream on KVM and libvirt etc. 15:51:34 mchua, I might send you a full list later, rather than forget people now 15:51:36 markmc: might be worth underscoring how many of them are at RH 15:51:40 * mchua notes it might be worth doing a profile on virt upstreams at some point 15:52:06 * mchua notes that "at some point" is likely to come after GA day, looking at current to-do list 15:52:08 lutter, oh, I meant "a huge list of Red Hat" people 15:52:19 mchua: off the top of my head, Danial Veillard, Matt Booth and Laine Stump should be on that list .. also a long list of qemu/kvm/kernel hackers that markmc has a better overview of 15:52:48 mchua: Cole Robinson (virt-install and virt-manager) 15:52:49 mchua, Avi Kivity, Gerd Hoffman, Christoph Hellwig, ... 15:52:59 dammit, I'm just going to forget people if I try and list here 15:53:08 lutter: thanks! Yeah, upstreams don't get nearly enough recognition, imo... should spotlight them too 15:53:14 yeah, that's the danger with these lists 15:53:14 yeah, not forgetting the $100M investment in qumranet, now Red Hat 15:53:35 * mchua nods. Not going to treat these lists as complete, just as potential starting spots to find out more 15:54:11 mchua: if you want to plug virtual appliances, Bryan Kearney, Joey Boggs and David Huff are to blame for thincrust 15:54:34 lutter, markmc, rwmjones: Thanks - this is all awesome stuff. I know y'all have a ton of work to do, and really appreciate you taking the time to come and fill us in. 15:54:38 lutter: noted! 15:54:57 np 15:55:01 lutter, markmc, rwmjones: last question - when you're not hacking on virt stuff, what do you do for fun? 15:55:03 mchua: np .. was fun 15:55:13 * rwmjones trolls OCaml features to C programmers ... 15:55:18 mchua: hacking on non-virt stuff ? ;) 15:55:28 rwmjones: *grin* got that. :) 15:55:30 mchua: I have two little kids that take up most of my free time 15:55:52 mchua, I live in Dublin, Ireland with my wife. Close to the sea and mountains, so I race sailing dinghys, run, hike and generally try and avoid computers as much as possible :) 15:56:14 sounds like y'all have the good life :) 15:56:17 mchua, what lutter said, was fun and easy 15:56:26 * rwmjones cooks the best pizza of anyone I know 15:56:31 mchua, introduce yourself too btw! we haven't met :) 15:57:18 * mchua is a new Red Hatter on the Community Architecture team, running Fedora Marketing (...this is also my first Fedora release, and I had to look up "Marketing" on wikipedia after Jack and Max asked me to step in... long story) 15:57:36 * mchua is an education geek; came from the One Laptop Per Child's engineering crew in a prior life 15:57:58 and will be at FUDCon in Toronto in 1.5 months if any of you will be there. 15:58:13 mchua, cool stuff, welcome to Red Hat :) 15:58:25 thanks! 15:58:36 mchua, don't think any of us are making it, but Cole Robinson and Justin Forbes will be there 15:58:39 I think we're pretty much set, unless there's anything you folks want to chime in on 15:59:03 markmc: cool - I'll make sure to catch up with them in person there, then 15:59:06 mchua, hopefully danpb_ltop and cdub will chime in about their features 15:59:24 Yep. I'll be hanging out in this channel for another hour or so in the hopes of catching them. 15:59:52 thanks lutter, rwmjones, markmc! 16:01:36 cdub, mchua 16:01:39 mchua, cdub 16:01:48 mchua: hi 16:01:57 cdub, so, I said you'd cover KSM, huge pages and SR-IOV ... okay ? 16:02:24 hey, cdub. Gimme 4 minutes to clean up some notes and I'll be right back... 16:02:32 markmc: ok, works for me 16:02:43 mchua: np 16:03:05 cdub: if you want to start by introducing yourself and giving an overview of those features, I'm logging the channel right now ;) 16:03:08 * mchua brb 16:03:15 cdub, rwmjones, lutter and I just had a case of verbal spew for the past hour 16:04:01 fun ;-) 16:05:27 cdub: talk amongst yourself ;) 16:05:49 mchua: my name is Chris Wright. I'm a kernel hacker working at Red Hat on virtualization, specifically KVM 16:06:25 * mchua back and listening 16:06:56 We're continually improving the virtualization infrastructure and Fedora 12 has a nice long list of virtualization specific features as a result 16:07:01 For those reading along who aren't familiar with KVM, that's http://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine. 16:07:17 right, thanks 16:08:09 Our goals are to improve the efficiency of KVM so that there is very little cost associated with running an OS in a virtual machine compared with bare metal 16:09:10 and to improve the density that we can acheive when consolidating multiple guest OS's to a single physical host. 16:09:43 mchua: hello, what do you want to talk about ? 16:09:55 hey, danpb_ltop! 16:10:11 danpb_ltop: the game plan is to do a round of introductions on who you are and what you're working on 16:10:23 One of the features we added for F12 is called KSM, which is about improving density, i.e. the number of VMs we can run on a since host 16:10:35 danpb_ltop: and then cdub was going to talk about huge page backed memory, KSM, and SR-IOV 16:10:53 danpb_ltop: which leaves VirtPrivileges, VirtStorage and libvirt TCK for you to explain 16:10:57 danpb_ltop: sound good? 16:11:14 And the other 2 I planned to talk about, hugepages and SR-IOV, are about improving the efficiency of the VM. 16:11:26 * mchua nods. 16:11:49 cdub: let's start with KSM. 16:12:19 mchua: alright, KSM...really cool feature that addresses one of the bottlenecks in virtualization. 16:12:42 #link https://fedoraproject.org/wiki/Features/KSM 16:13:07 A modern computer has lots of cores, but memory is still relatively expensive 16:13:29 That's Kernel SamePage Merging (or the Korean Service Medal, or the Kothagudem School of Mines, but we're talking about the first one here. ;) 16:13:44 mchua: ok 16:13:45 So you can run out of memory for all the virtual machines you may want to run on a box, despite the fact that you've got CPU power to spare 16:13:47 * mchua nods and sits back to listen 16:14:21 danpb_ltop: I reckon we'll just go through the features in order, so feel free to chime in on the KSM/hugepages/SR-IOV discussion, and then when we switch to your 3, you get to drive. 16:14:44 mchua: heh, right. The acronym started life with a different translation, but same underlying meaning -- used to mean Kernel Shared Memory ;-) 16:16:00 KSM, at it's core, simply scans regions of physical memory, looking for duplicate contents. 16:17:11 And when it finds 2 pages of memory with identical contents, it collapses them to a single page. 16:17:52 So this has the effect of compressing memory utilization. 16:18:16 Now, when you consider this in a virtual machine context, you can see how this can be really useful. 16:18:57 If the virtual machines are running similar OSes, they'll contain some of the same memory just for the kernels and programs running in the OS. 16:19:43 So when we launch a VM, we register the memory associated with the VM to KSM, and let KSM scan away in the background. 16:20:10 cdub: Do you have a rough idea of the range of how much memory (%?) this might typically save? 16:20:15 You can actually watch as your free memory shrinks when you start a new VM, and then slowly grows back as KSM fins pages to merge. 16:20:17 (for a particular type of use case, etc.) 16:20:30 Ooh, I should get some screen captures of that. 16:20:50 mchua: good question, it's very workload dependent, I don't have a number right off the top of my head. 16:20:52 #action mchua get screencaps of free-memory-over-time for KSM section 16:21:29 mchua: but one thing to keep in mind that's interesting here, is that some OSes opportunistically write zeros to their free memory 16:21:47 cdub: Ok. Is there a way to find out? (I'm happy to do the legwork needed to get that comparison if there's a quick descript of what I should do / look to compare.) 16:21:59 this has the effect from the KVM point of view of making they hypervisor believe that the memory is in use (it's been written to) 16:22:31 but it's actually free, awaiting an allocation. With KSM, we'll find thousands of these pages, and collapse then to a single page. 16:22:32 cdub: Interesting. So it can't tell the free memory is free. 16:22:44 Nifty. 16:23:11 cdub: should we switch gears and talk about huge pages and SR-IOV for a bit? 16:23:16 mchua: right, the hypervisor doesn't have all the information, so it can only tell when a page is used, not unused 16:23:37 cdub: Oh - wait, before we do that... is there anything like KSM outside Fedora? 16:23:48 similar tools in other OSes, etc? 16:23:59 mchua: sure, one last thing on KSM...there are statistics you can view in /sys/kernel/mm/ksm/ 16:24:05 * mchua nods 16:24:28 mchua: actually, good question, reminds me of something I wanted to point out... 16:24:50 mchua: yes, there's at least one other OS that has a feature like this, ESX (VMware). 16:25:00 mchua: but in Fedora, KSM is not exclusive to VMs 16:25:15 mchua: KSM works w/ any program that registers it's memory as "mergeable" 16:25:52 mchua: so even regular programs running in Fedora could benefit, and some number crunchers at CERN have used this to improve their own application's memory usage 16:26:18 mchua: so that they can run more apps, and do more number crunching w/ the same hardware 16:26:54 Nice! 16:27:03 mchua: so that's KSM, shall we move on to hugepages and SR-IOV? 16:27:12 cdub: I was just about to suggest the same. 16:27:16 cool 16:27:28 danpb_ltop: Feel free to chime in any time; we'll get to your features in just a moment. :) 16:27:30 ok, moving from density to efficiency... 16:28:20 hugepages, another feature added to F12, give the user the ability to run a VM backed by huge pages 16:28:38 normally, when we run a VM, we simply malloc() the memory for the guest OS. 16:29:15 this means that the memory for the guest will be allocated in page sized chunks, 4k to be specific 16:30:12 the hypervisor needs to be able to translate between the guests view of physical memory and the hosts view of physical memory 16:31:10 when the host is using large pages, like 2M pages, or huge pages, to allocate guest memory, those translations become cheaper (there are fewer to do) 16:31:46 and, if the guest actually wants to use huge pages (database servers like to do this, as do some Java workloads) 16:32:01 the translation cost goes down further 16:32:29 ultimately, with hug pages we've seen double digit percentage improvement for some of those workloads 16:33:33 in fact, w/out backing the guest's memory w/ huge pages on the host, when a guest asked for a huge page...we lied. so this is really nice to fix 16:34:09 "we lied" --> "we said 'here is a huge page' that wasn't actually a huge page"? 16:34:32 exactly, which is not very nice since the reason the guest asked for a huge page was to improve it's own performance 16:35:48 * mchua nods. 16:36:06 SR-IOV real quick, and then on to danpb_ltop's features? 16:36:15 alright 16:36:35 SR-IOV is another attempt to improve guest VM efficiency 16:36:52 the I/O path for a guest is traditionally tough to virtualize 16:37:28 danpb_ltop, cdub: I'd also like to have the two of you talk together a bit on how virt has progressed overall between F6 and F12, where virt is headed for F13 and beyond, and then some non-virt stuff about yourselves so our readers get to know you a little better, just so you know what's coming up. 16:37:35 * mchua listens to SR-IOV 16:39:07 a typical VM has a NIC and a storage device that may be either emulated devices (this is the most expensive, but least likely to require any new drivers in the OS), like an emulated realtek NIC, or a virtual device which requires a special device driver, but doesn't need to emulate anything, it knows that it's just a virtual path to the hypervisor's I/O subsystem 16:40:20 in both of those cases, there is a fair amount of CPU involved in processing the I/O request. the emulated device has the most overhead, but even a virtual device (like KVM's virtio devices) have to copy data around, and cause expensive exits from the VM to the hypervisor 16:41:39 SR-IOV is an attempt by the industry to move virtualization out of the CPU (hardware virtualization extensions that allow KVM to work at all), past the chipset (like an IOMMU that allows memory isolation when a guest is talking directly to a physical device), and into the I/O devices 16:42:27 So this is something that's happening outside of just Fedora, too. 16:42:33 So, it requires newer hardware, specifically, the CPU, IOMMU and an SR-IOV capable card (there aren't a lot of these on the market yet, so Fedora is really on the leading edge) 16:42:52 right, SR-IOV is a PCI standard 16:43:20 An SR-IOV capable card allows you to effectively virtualize the I/O hardware 16:44:02 so rather than having a single physical E1000 NIC that you must share with each VM via some indirection (the emulated or virutal device I mentioned above) 16:44:35 you get a single physical NIC that you can allocate multiple virtual instances of 16:45:14 So now you can allocate some resource from the SR-IOV NIC (called a Virtual Function, or VF) 16:45:23 that shows up as a PCI device, just like real hardware 16:45:51 and w/ the existing ability in Fedora to do PCI device assignment to a guest, you can assign that VF directly to the guest 16:46:26 that means the guest is communicating directly to the hardware, it's running the same device driver you'd run on the hypervisor 16:46:35 * mchua nods 16:46:36 and this really shortens the I/O path. 16:46:55 With this you can effectively acheive bare metal I/O performance from a guest 16:47:13 IOW, the I/O bottle neck is removed 16:48:10 Nice. 16:48:22 Yeah, KVM in F12 is looking really good 16:48:31 danpb_ltop: ready to go? 16:49:13 cdub, danpb_ltop: I don't want to keep you folks waiting, so what we might do is have the two of you talk together about F6-->F12 virt improvements and what's coming down the virt pipeline for future releases together 16:49:44 mchua: I'm fine idling waiting for danpb_ltop to finish too 16:49:44 mchua: either way 16:49:44 cdub: and have you at some point throw in a "when I'm not hacking on virt stuff, I... " 16:50:09 and then cdub can run off, and I'll go through danpb_ltop's intro and those 3 features with him 16:50:26 cdub: while we're waiting, what do you do for fun aside from virt hackin'? 16:50:33 cdub: (and how did you get started doing it?) 16:51:23 mchua: well, I got started hacking on virt stuff because I was interested in security and isolation. The thing that I found really exciting was that this was an area where hardware was rapidly evolving 16:51:51 mchua: so, as a kernel hacker, it was really fun to work on software that's adpating to these new hardware features. 16:53:07 wow, well F6 to F12 is a huge amount of time - almost 3 years worth of Fedora releases 16:53:18 mchua: I was convinced that the virt stuff which I originally learned about in the context of Trusted Computing, had some other more useful benefits...that was probably 6 years ago, wow 16:54:12 Way back in F6 all focus was on Xen and making it easy to manage, this was the first release where we had apps like virt-manager available 16:54:59 mchua: as for fun...I've got two small children...so, sleep! ;-) Heh, that or hangin' w/ my kids, or riding my bike 16:55:27 and the first to introduce graphical installation for guests, so it really set the foundations for future work 16:55:35 cdub: Mmm, biking. :) 16:55:46 * mchua switches gears to F6-->F12 convo 16:55:55 mchua: markmc recently told me my bike has bling! never been associated w/ bling before ;-) 16:56:04 * mchua listens to danpb_ltop, waits for cdub to chime in on F6-->F12 too. 16:56:09 in Fedora 7, we added a very early release of KVM, along with support in libvirt + virt-mnanager for KVM+QEMU 16:56:25 but Xen was still the primary virt platform at that time 16:57:03 Fedora 8 focused on stepping up the security capabilites of the management toolchain 16:57:12 And it was a lot of effort to forward port Xen all along that path 16:57:34 by introducing support for securely using libvirt from a remote host using TLS/SSL or SSH tunnel, with similar capability added to the VNC server 16:58:17 yes, as chris says there was always a massive effort going on in the background from F5 right through F8 on forward porting the old Xen kernel trees to something uptodate 16:59:13 by F8 though, KVM was really gaining ground and was genuinely usable so many Fedora users had switch from Xen to KVM already at that time 17:00:02 in Fedora 9, we finally stopped trying to forward port the old Xen kernel trees, and switched to only support paravirt-ops based kernels from LKML upstream 17:00:30 this meant dropping support for Xen has a virtualization host platform, so from Fedora 9 onwards we only supported Xen as a guest 17:00:54 That was a tough decision to make. But reality was the forward porting just got too hard 17:01:05 fortunately KVM was in great shape by then, and hardware virtualization support was pretty widely available 17:02:05 Fedora 9 also introduce more security features, such as support for SASL which allows use of Kerberos authentication for libvirt and PolicyKit for local desktop authentication 17:02:36 Right, that was the key change. You could find new laptops, desktops, servers all w/ hardware virt support 17:03:13 libvirt work to provide APIs for managing storage allowed Fedora 10 to introduce full remote provisioning in virt-manage 17:04:14 and then finally F11 we added SASL support to the VNC server, comparable to that we'd done for libvirt in F10 17:04:45 * mchua enjoying this whirlwind tour through the ages 17:04:45 so both libvirt & VNC can now integrate with pretty much any commonly found authentication services 17:05:43 the most important feature in F11 though was the introduction of sVirt 17:06:18 which is integration between libvirt and SELinux to provide security protection between virtual machines running on the same host 17:06:35 (previously SELinux had merely protected the host from VMs, but not VMs from each other) 17:07:06 Yeah, also a nice way to show the benefit of KVM being a part of Linux 17:08:20 yep, it avoids having to reinvent all these concepts in a separate hypervisor 17:09:01 cdub,danpb_ltop: And I think that brings us to F12 and the features we've just covered / are about to cover. Sweet. 17:09:03 there are soo many important & useful features available in Linux we want to take advantage of that you really don't want to start having to re-invent them all 17:09:10 * mchua nods 17:09:26 all this stuff is summarized in our history page https://fedoraproject.org/wiki/Virtualization/History 17:09:34 danpb_ltop, cdub: can you talk a little bit about where you see virt work headed in the future, for Fedora N where N > 12? 17:09:43 * mchua notes that and will link to virt history wiki page heavily 17:10:10 danpb_ltop, cdub: cloud sounded like a big thing to watch. 17:10:13 deltacloud, specifically 17:10:21 It's nice to look back and see how in a few years we've gone from no virt support in Fedora to what you see in F12 17:11:03 mchua: one thing to underscore in that F6->F12 is the importance of libvirt 17:11:04 mchua: you can thing of deltacloud as doing for clouds, what libvirt did for hypervisors 17:11:13 exactly 17:11:26 libvirt made it possible to write applications against a simple, standard, stable API regardless of the underlying hypervisor technology 17:11:47 so the success of libvirt in isolating tools from the underlying hypervisor is just waht deltacloud is for cloud management 17:12:09 the fact that we now have libvirt support for Xen, KVM, QEMU, OpenVZ, VMWare ESX, VMWare GSX, LXC (native containers), IBM Power Hypervisor & OpenNebula 17:12:52 shows just how much people like the idea of libvirt - all of those drivers except for Xen & KVm were started by libvirt community members 17:13:25 libvirt made it possible to move from F5 w/ paravivrt only Xen to Xen w/ HVM to KVM, w/out having to keep rewriting the magnement tools (they did evolve, like danpb_ltop mentioned ;-) 17:13:29 deltacloud is aiming todo the same for cloud providers so you can write one app targetting any service and avoid being locked into proprietary cloud mgmt APis 17:14:05 oh, add VirtualBox to that list for libvirt - mustn't forget one ! 17:14:16 mchua: idea being you can manage multiple clouds from single tool, and even support moving from one cloud to another 17:14:48 Nice! 17:15:05 mchua: of course, we'll keep working on the infrastructure too 17:15:33 mchua: continually improving the efficiency of the hypervisor, the managability of the hypervisor, etc. 17:15:41 there's still plenty of work to be done for non-cloud related virt of course 17:16:08 in the management tools we really want to polish the desktop virt usage scenario 17:16:34 we've tended to focus more on server virt, so there's some things that aren't so nice to use for the single desktop case 17:16:57 you can see the start of this with the major design overhaul of virt-manager UI in F12 17:17:33 there's always more work to be done with security features too 17:17:55 another thing we should see is better work on the remote desktop 17:18:07 previously introduced sVirt allows us to protect VMs from each other, but all VMs still had more or less the same policy rules 17:18:29 we want to start making this more tunable so you can easily customize policy for individual VMs 17:18:50 for example, if running a Windows desktop, you might give it a policy that blocks all network traffic on port 25 17:18:56 to prevent it being turned into a spam botnet 17:19:38 or just want to restrict what VMs on a host are allowed to communicate with each other 17:20:02 the whole way we manage VM networking is being reviewed as well 17:20:51 fine grained access control over the libvirt APIs is also another thing we'd like todo 17:21:00 just seeing new patches on the libvirt dev list to try and create new APIs for managing the rules surrounding a VMs network interface 17:21:24 so you can determine who can manage each VM & what operations they can perform, etc 17:22:02 anyway, shall we get back to the F12 features 17:22:49 danpb_ltop: Yep. 17:23:03 (05:10:53 PM) mchua: danpb_ltop: which leaves VirtPrivileges, VirtStorage and libvirt TCK for you to explain 17:23:10 so taking them in that order 17:23:10 cdub: I think I have everything I need from you - you're welcome to stick around, of course! but if you have to run, we're good. ;) 17:23:15 danpb_ltop: thanks! 17:23:24 mchua: cool, thanks 17:23:48 VirtPrivileges is yet another feature focusing on security (you've noticed that's a common theme in virt work :-) 17:24:20 libvirt has two modes of running virtual machines 17:24:56 what we call our 'system' instance, is a per-host instance that runs maximum privileges for accessing storage / networking / etc 17:25:10 this was primarily intended for server virtualization scenarios 17:25:37 and then what we call our 'session' instance, is a per-user instance that runs with the same privileges as the user connecting to it 17:26:11 this was intended for desktop virtualization, although it has not been really used much yet because it is hard to provide useful networking connectivity with it 17:26:39 For the VirtPrivileges feature we wanted to improve security by reducing the privileges of the QEMU/KVM process 17:26:53 but without sacrificing the functionality available 17:27:12 so we now have QEMU running as a dedicated 'qemu' user account and group, instead of 'root' 17:27:29 and libvirt manages permissions on resources that are assigned to QEMU, such as its disks 17:27:52 one of the hard things was being able to maintain full network connectivity 17:28:08 so we had to work with QEMU developers to provide a new way to hotplug network cards 17:28:38 where libvirt sets up a "TAP" device and then passes it across to an already running QEMU process with a little UNIX blackmagic 17:28:58 so this all improved security of the libvirt "system" instance 17:29:12 to make the 'session' instance more useful, we also changed the KVM setup so that 17:29:29 any user on the system can access /dev/kvm and thus run hardware accelerated virtual machines 17:30:34 once we figure out how to provide better network connectivity to unprivileged virtual machines the 'session' instance of libvirt will finally be useful for desktop virt and address alot of long standing bugs/RFEs people have had 17:31:06 Moving onto the 'VirtStorage' feature 17:31:23 quite a few releases back we introduced storage management APIs into libvirt 17:31:30 Does VirtPrivileges intersect with any of the other virt feature work being done with F12? I know there's been some network interface dev going on, etc. 17:31:47 at the time we supported local disks, LVM, file based storage, iSCSI in the storage APis 17:32:08 mchua: yes the network interface dev work was related to it, allowing us to hotplug network interfaces to running VMs 17:32:38 the VirtStorage feature, extends our existing storage APis to now support SCSI FibreChannel adapters 17:33:02 so you can discover what SCSI adapters you have, and what LUNs they are exporting to the host 17:33:34 there is some fairly new technology called "NPIV" which allows one physical SCSI host adapter to be used 17:33:48 to create many virtual host adapters, each with their own set of LUNs 17:33:48 mchua, quick question, where do you publish these interviews when you've edited them together? 17:34:09 so work was also done to allow libvirt to create / delete virtual host adapters when NPIV is supported 17:34:35 the idea behind NPIV is that you might have one virtual SCSI host adapter associated with each VM 17:35:01 and so instead of having to expose all SCSI luns to all hosts 17:35:06 rwmjones: I'm going to be doing the editing on the Fedora wiki and it'll temporarily live there, but we'll also publish it on Fedora Insight once that goes live. 17:35:17 you only need to expose the virtual SCSI host adapter to the host on which the VM is currently running 17:35:42 rwmjones: https://fedoraproject.org/wiki/Fedora_Insight, the publictest is http://publictest6.fedoraproject.org/zikula/ and it's almost ready to go staging --> production. 17:35:43 this makes management of storage much more flexible, an secure 17:36:08 mchua, cool - I'd not heard of that site before, but it looks excellent 17:36:32 finally the libvirt TCK 17:37:54 rwmjones: it's just a centralized place to publish all the Fedora marketing materials we already generate (but currently scatter across multiple blogs / wiki pages / etc) 17:38:06 readers may or may not be aware of the Java TCK which is a huge test suite that people who write Java JRE/JDKs have to run & pass to ensure compliance with the java specification 17:38:08 that's a very very good idea 17:38:37 with libvirt we've had some ups & downs on the quality front and as we gained support for more & more APis & hypervisors 17:38:53 * mchua pulls up supporting materials to explain TCK 17:38:55 it was becoming increasingly hard to ensure the new libvirt releases were off the quality people expect 17:38:57 #link http://jcp.org/en/resources/tdk 17:39:08 #link http://en.wikipedia.org/wiki/Technology_Compatibility_Kit 17:39:15 (the wikipedia article == more useful resource) 17:39:26 so we decided to build what we call the 'libvirt TCK' (libvirt Technology Compatability Kit) 17:39:43 the idea being that we write a huge set of tests covering all aspects of libvirt APIs 17:40:03 which we can then run against each hypervisor libvirt supports to ensure everything is working as it is expected to 17:40:31 this not only finds bugs in libvirt, but also helps identify bugs in new releases of the underlying hypervisor/virtualization platform 17:40:44 or in the way an OS distributor built / packaged them 17:40:59 danpb_ltop: how is that kind of QA being carried out now (or before the libvirt TCK came around?) 17:41:12 this is quite a new bit of work and we've only got a handful of test cases built into it so far 17:41:40 but it has already allowed us to identify & fix alot of bugs before releasing which would have otherwise caused regressions for users 17:42:21 mchua: well there's testing by upstream libvirt developers, testing by OS packagers / distributors and testing by end users (eg in a Fedora test day, or even of the final releases) 17:42:40 the libvirt TCK is primarily targetted at upstream libvirt, and OS distributors 17:43:06 upstream libvirt community wants to make sure they don't release something which stupid bugs in it 17:43:16 * mchua nods 17:43:25 and OS distributors want to make sure they've built & packaged everything, and then when they update to the latest KVM / Xen / whatever 17:43:38 that they are not going to cause regressions in libvirt or applications using libvirt 17:43:52 above all we want to catch as many bugs as possible before they get to end-users 17:45:16 its got fairly minimal testing coverage for F12, but come F13 we want to have all important core functionality automatically tested 17:45:35 danpb_ltop: What can our readers to to help out with this testing? 17:45:44 (or to try out any of these features and send feedback, really?) 17:46:45 well had a Virtualization Test Day a few weeks back now, but if interested in doing testing 17:47:01 keep an eye out for future test days during the course of Fedora 13 development 17:47:18 joining the fedora-virt mailing list is a good way to get involved in Fedora virtualization work 17:47:57 or if they have development experiance, then the various upstream communities always have plenty of need for help 17:49:27 * mchua nods - thanks! 17:49:57 danpb_ltop: I think we're almost done - anything else on those three features (or any others) you'd like to call out/explain/plug? 17:52:35 danpb_ltop, V2V? 17:54:51 danpb_ltop: I'd also like to get a sentence or two of introduction from you (since we missed that at the beginning) and a couple things you do for fun outside of virt-hackin' 17:54:58 and then we'll be done. 17:55:07 danpb_ltop: Thanks for being so patient - I know this took longer than expected. 17:55:56 rwmjones: there's no V2V stuff in F12 AFAIK 17:56:27 didn't matt add it? anyhow, doesn't matter 17:58:37 mchua: I've worked on Red Hat for quite a long time now, must be more than 7 years, with the last 3 focusing on virtualization 17:58:58 i originally got involved in the virtualization team by writing the virt-manager application 17:59:21 but since then Cole Robinson has taken the lead on that development, and I'm spending most time on lower level areas 17:59:51 probably 80% libvirt, and the rest related things like QEMU / KVM / VNC 18:00:13 danpb_ltop: Why do dev work for those things in Fedora? 18:00:15 its a good mixture of upstream work, Fedora work and RHEL work 18:00:42 well upstream libvirt is on an approx monthly schedule 18:01:43 Fedora has a short 6 month schedule, which means Fedora is a great place to get early exposure to real users 18:02:16 a new libvirt release ends up in Fedora rawhide almost always the same day 18:02:46 and Fedora stable releases are only a couple of releases behind latest 18:03:18 it works out well for us as libvirt community developers, and for users who always want the latest stuff 18:04:11 Sweet. And then eventually that work finds its way over to RHEL? 18:05:50 yep, periodically it works its way into RHEL, but on a much longer timescale 18:06:13 since RHEL has more prolonged testing / quality control cycles before release than you'd get with Fedora 18:07:08 * mchua references http://www.youtube.com/watch?v=xu81frqUtlc, Paul's video 18:07:22 danpb_ltop: Thanks. And what do you do when you're not hacking on virt? 18:07:57 err, sleep 18:08:16 Sleep is good stuff. 18:08:32 nah, seriously i spend quite alot of time on photography 18:08:40 * mchua notes "sleep" seems to be one of the first "what do I do in my free time" responses from virt hackers... 18:08:53 mchua: They all have guest instances working while they snooze. 18:09:28 oh, nice! seems like we've got a pretty good set of hobbyist photographers at RH 18:09:34 danpb_ltop, dwa, mizmo, etc 18:10:15 danpb_ltop: is there a gallery you'd like to share with folks, just for fun? 18:10:21 (totally optional) 18:11:07 stickster: btw, I'm going to be cleaning this up en route to Toronto, do you want it tomorrow or sometime over the weekend or Monday or at some later date? 18:11:29 stickster: I have *way* more than enough info now to make multiple marketing shiny things from this 18:12:10 mchua: Monday would be fine -- I'll be mostly out of commission FAD-ing this weekend 18:12:24 mchua: Might want to let f-mktg-l know, Kara will see it there too 18:12:33 hah, they can google for it ! 18:12:46 danpb_ltop: maybe I will ;) 18:12:58 danpb_ltop: anything else? otherwise I think we're done - thanks for all your time (and patience!) 18:13:08 * mchua waves at mthompson 18:15:14 think that's all 18:15:55 danpb_ltop: We're all set, then. Thanks for your time! 18:16:00 * mchua will try to find a way to streamline this process in the future - I think yours was the one that went most over, because of the staggered scheduling. 18:16:22 * rbergeron yawns 18:16:29 * mchua closes out logs 18:16:31 #endmeeting